Find Jobs
Hire Freelancers

data project

$30-250 USD

Cerrado
Publicado hace casi 2 años

$30-250 USD

Pagado a la entrega
In this project, you will develop an Oozie workflow to process and analyze a large volume of flight data. • Instructions: 1. Form a project team of four students (including yourself). 2. Install Hadoop/Oozie on your AWS VMs. 3. Download the Airline On-time Performance data set (flight data set) from the period of October 1987 to April 2008 on the following website: [login to view URL]:10.7910/DVN/HG7NV7 4. Design, implement and run an Oozie workflow to find out a. the 3 airlines with the highest and lowest probability, respectively, of being on schedule; b. the 3 airports with the longest and shortest average taxi time per flight (both in and out), respectively; and c. the most common reason for flight cancellations. • Requirements: 1. Your workflow must contain at least three MapReduce jobs that run in fully distributed mode. 2. Run your workflow to analyze the entire data set (total 22 years from 1987 to 2008) at one time on two VMs first and then gradually increase the system scale to the maximum allowed number of VMs for at least 5 increment steps, and measure each corresponding workflow execution time. 3. Run your workflow to analyze the data in a progressive manner with an increment of 1 year, i.e. the first year (1987), the first 2 years (1987-1988), the first 3 years (1987-1989), …, and the total of 22 years (1987-2008), on the maximum allowed number of VMs, and measure each corresponding workflow execution time. • Submission (all in a zipped file: [login to view URL]): 1. A [login to view URL] text file that lists all the commands you used to run your code and produce the required results in a fully distributed mode 2. An [login to view URL] text file that stores the final results from all the runs 3. The source code of your MapReduce programs (including the JAR files) and any other programs you might have developed and included in the workflow 4. The Oozie workflow XML file 5. A project report in PDF that includes: a. A diagram that shows the structure of your Oozie workflow b. A detailed description of the algorithm you designed to solve each of the problems c. A performance measurement plot that compares the workflow execution time in response to an increasing number of VMs used for processing the entire data set (22 years) and an in-depth discussion on the observed performance comparison results d. A performance measurement plot that compares the workflow execution time in response to an increasing data size (from 1 year to 22 years) and an in-depth discussion on the observed performance comparison results
ID del proyecto: 33638624

Información sobre el proyecto

6 propuestas
Proyecto remoto
Activo hace 2 años

¿Buscas ganar dinero?

Beneficios de presentar ofertas en Freelancer

Fija tu plazo y presupuesto
Cobra por tu trabajo
Describe tu propuesta
Es gratis registrarse y presentar ofertas en los trabajos
6 freelancers están ofertando un promedio de $144 USD por este trabajo
Avatar del usuario
Yo! I am an expert PHP, laravel, codeignter programmer with skills including Database Administration, Data Processing, Data Analysis, Data Science and Data Analytics. Please send a message to discuss more about this project. Thanks
$155 USD en 7 días
4,5 (6 comentarios)
4,1
4,1
Avatar del usuario
Hi, I have +5 years of experience dealing with machine learning algorithms and worked on multiple projects in this field, I absolutely can do your project as you like. Please contact me to discuss more. Have a nice day
$140 USD en 7 días
5,0 (1 comentario)
0,2
0,2
Avatar del usuario
I am big data specialist. Contact me me please........................................... Regs Jinu
$140 USD en 7 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
Hi! My name is Mani, I am a student in my 3rd year at the University of St Andrews (ranked #1 in the UK this year) studying Computer Science. I have ample experience with Python, and especially with pandas and numpy, having done numerous data science projects in the past. I am familiar with Oozie, and would be more than happy to help you with this task!
$150 USD en 7 días
0,0 (0 comentarios)
0,0
0,0

Sobre este cliente

Bandera de EGYPT
Cairo, Egypt
4,9
39
Forma de pago verificada
Miembro desde oct 25, 2018

Verificación del cliente

¡Gracias! Te hemos enviado un enlace para reclamar tu crédito gratuito.
Algo salió mal al enviar tu correo electrónico. Por favor, intenta de nuevo.
Usuarios registrados Total de empleos publicados
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Cargando visualización previa
Permiso concedido para Geolocalización.
Tu sesión de acceso ha expirado y has sido desconectado. Por favor, inica sesión nuevamente.