A Big Data Hadoop and Spark project for absolute beginners

Цена: $29.99

This course will prepare you for a real world Data Engineer role !

Get started with Big Data quickly leveraging free cloud cluster and solving a real world use case! Learn Hadoop, Hive , Spark (both Python and Scala) с нуля!

Learn to code Spark Scala & PySpark like a real world developer. Understand real world coding best practices, Ведение журнала, обработка ошибок , configuration management using both Scala and Python.

Проект

A bank is launching a new credit card and wants to identify prospects it can target in its marketing campaign.

It has received prospect data from various internal and 3rd party sources. The data has various issues such as missing or unknown values in certain fields. The data needs to be cleansed before any kind of analysis can be done.

Since the data is in huge volume with billions of records, the bank has asked you to use Big Data Hadoop and Spark technology to cleanse, transform and analyze this data.

Что вы узнаете :

Большие данные, Hadoop concepts
How to create a free Hadoop and Spark cluster using Google Dataproc
Hadoop hands-on – HDFS, Hive
Основы Python
PySpark RDD – руки вверх
PySpark SQL, DataFrame – руки вверх
Project work using PySpark and Hive
Scala basics
Spark Scala DataFrame
Project work using Spark Scala
Spark Scala Real world coding framework and development using Winutil, Maven and IntelliJ.
Python Spark Hadoop Hive coding framework and development using PyCharm
Building a data pipeline using Hive , PostgreSQL, Spark
логирование , error handling and unit testing of PySpark and Spark Scala applications
Spark Scala Structured Streaming
Applying spark transformation on data stored in AWS S3 using Glue and viewing data using Athena

Загрузите и установите MX OPC-сервер :

Some basic programming skills
Some knowledge of SQL queries