Realtime Data Streaming | End To End Data Engineering Project

In this video, you will be building a real-time data streaming pipeline, covering each phase from data ingestion to processing and finally storage. We'll utilize a powerful stack of tools and technologies, including Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra—all neatly containerized using Docker.

Created by

Yusuf Ganiyu

01:27:48 Hours

(1 Reviews)

English

Last updated

Tue, 02-Apr-2024

Course description

Unlock the world of real-time data streaming with this comprehensive tutorial.

Starting with data ingestion, you'll be guided through each phase, utilizing a robust stack of tools and technologies. Harness the power of Apache Airflow for setting up your data pipeline, stream data with Kafka, synchronize with Zookeeper, process it using Apache Spark, and store your findings in Cassandra and PostgreSQL.

One of the highlights of this tutorial is the containerization of the entire data engineering environment using Docker, ensuring a smooth and integrated experience.

With hands-on examples, expert insights, and a plethora of resources, this video is your definitive guide to mastering the art of real-time data streaming.

What will i learn?

Data Ingestion: This is the first step in the data pipeline where you’ll learn how to collect and import data from various sources into a single system for further processing.
Apache Airflow: You’ll learn how to use Apache Airflow, an open-source platform to programmatically author, schedule, and monitor workflows. It will be used to set up and manage your data pipeline.
Kafka: Kafka is a distributed streaming platform. You’ll learn how to use Kafka to stream data in real-time from different sources.
Zookeeper: Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. You’ll learn how to synchronize your data with Zookeeper.
Apache Spark: Apache Spark is a unified analytics engine for large-scale data processing. You’ll learn how to process your data using Spark, performing operations such as filtering and aggregation.
Cassandra and PostgreSQL: These are database systems where you’ll store your processed data. You’ll learn how to interact with these databases, perform CRUD operations, and optimize your queries for performance.

Requirements

Basic knowledge of Python Programming Language
Basic Knowledge of Apache Spark
Basic Knowledge of Apache Airflow
Basic Knowledge of Docker

Realtime Streaming Project | End to End Data Engineering
01:27:48

Yusuf Ganiyu

View profile Follow

john ameh

21-Nov-2023

5

£14.99

£24.99

Lectures

1

Skill level

Beginner

Expiry period

Lifetime

Certificate

Yes

Remove from cart Add to cart Buy now

Top categories