S25 Data Engineering Program
Course Instructor
Sumit Mittal
/CourseBundles(54431)/3813647-S-25_Course_Card.png)
Course Overview
Schedule of Classes
Course Curriculum
17 Subjects
Welcome Session
1 Learning Materials
Welcome Session
Welcome Session
Video
40:28
Week 1: Big Data - The Big Picture
1 Exercises • 10 Learning Materials
Big Data Fundamentals
Introduction to Big Data
Video
32:40
Hadoop Overview
Video
30:37
Cloud and its Advantages
Video
20:29
Understanding Apache Spark at a High-Level
Video
15:8
Database Vs Data Warehouse Vs Data Lake
Video
24:40
Big Data - The Big Picture
Video
39:45
Hadoop Distributed File System - HDFS Architecture
Video
17:34
Role of Data Engineers
Video
22:35
Running Notes & Summary Document
Week 1 Running Notes
PDF
Week 1 Summary Document
PDF
Week-1 Quiz
Big Data - The Big Picture
Exercise
Week 2: Distributed Storage and Data Lake
1 Exercises • 14 Learning Materials
Datalake Storage & Getting Started with the Labs
HDFS Overview
Video
38:15
About Practice Labs
Video
31:5
Linux Commands - cd & ls
Video
45:53
More Linux Commands
Video
46:27
HDFS Commands
Video
43:53
More About Practice Lab
Video
17:4
HDFS Vs Cloud Data Lake
Video
15:3
Getting Started with Distributed Processing
Video
42:55
Running Notes & Summary Document
Week 2 Running Notes
PDF
Week 2 Summary Document
PDF
Steps to SSH in Windows
PDF
FAQs
Week 2 : Frequently Asked Questions
PDF
Week 2: Quiz
Distributed Storage and Data Lake
Exercise
Week 2: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Week 3: Distributed Processing Fundamentals
1 Exercises • 18 Learning Materials
Downloadable Resources
Distributed Processing - Program Jar Files (Executing MapReduce Jar )
PDF
Week 3 : Reference Jupyter Notebooks
ZIP
Downloadable : Practice Datasets
More About Distributed Processing
Distributed Processing Continuation
Video
18:13
Changing the Number of Reducers
Video
30:23
Use Case 1 - Sensor Data Example
Video
31:51
Real-time Industry Use Case of Distributed Processing
Video
9:58
Distributed Computing Demo
Video
37:24
Distributed Computing Demo Continuation
Video
34:48
Apache Spark
Getting to Know Apache Spark
Video
31:51
Apache Spark Vs Databricks
Video
23:58
Spark Execution Plan
Video
33:22
Word Count Example in Apache Spark
Video
40:5
Running Notes & Summary Document
Week 3 Running Notes
PDF
Summary Document
PDF
FAQs
Week 3 : Frequently Asked Questions
PDF
Week 3: Quiz
Distributed Processing Fundamentals
Exercise
Week 3: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Week 4: Apache Spark Core APIs
1 Exercises • 17 Learning Materials
Downloadable Resources
Datasets
Week 4 : Reference Jupyter Notebooks
ZIP
Distributed Processing - Pyspark In-Depth
Python Basics
Video
25:44
Spark Usecase 1 - Orders Data
Video
45:50
Spark Core APIs - RDD
Video
26:51
More on Spark Parallelize
Video
27:15
More Spark Transformations
Video
25:22
Spark DAG Visualization | reduce Vs reduceByKey
Video
14:37
reduceByKey Vs groupByKey
Video
45:38
Spark Join
Video
24:9
Broadcast Joins
Video
25:9
Repartition Vs Coalesce
Video
21:26
Cache
Video
21:7
Running Notes & Summary Document
Running Notes
PDF
Summary Document
PDF
Week 4: Quiz
Apache Spark Core APIs
Exercise
Week 4: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Week 5: park APIs - Dataframes & Spark SQL
1 Exercises • 15 Learning Materials
Downloadable Resources
Week 5 : Reference Jupyter Notebooks
ZIP
Higher Level APIs in Apache Spark
Higher Level APIs - Dataframes & Spark SQL
Video
22:43
Understanding Dataframes
Video
26:56
More about Dataframe Reader
Video
34:13
Database Creation : Setting Configuration Properties
PDF
Introducing Spark SQL
Video
26:33
Spark SQL - Managed vs External Tables
Video
24:2
Use Case - Dataframes & Spark SQL
Video
38:3
Getting Started with Spark Optimizations
Video
5:36
Spark Executors
Video
22:55
A Little More on Spark Executors
Video
18:25
Running Notes & Summary Document
Running Notes
PDF
Summary Document
PDF
Week 5: Quiz
Spark APIs - Dataframes & Spark SQL
Exercise
Week 5: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Week 6: Spark Dataframe Transformations
1 Exercises • 15 Learning Materials
Downloadable Resources
Week 6 : Reference Jupyter Notebooks
ZIP
Spark Transformations
Recap of the Concepts Learnt
Video
15:26
Schema Enforcement
Video
32:31
How to Deal with Date Type
Video
31:19
Read Modes
Video
14:30
Different ways of Dataframe Creation
Video
40:25
Converting RDD to Dataframe
Video
18:28
Nested Schema
Video
19:56
Dataframe Transformations | select Vs selectExpr
Video
30:51
Removal of Duplicates from Dataframe
Video
11:50
Spark Session in Detail
Video
52:13
Running Notes & Summary Document
Week 6 Running Notes
PDF
Summary Document
PDF
Week 6: Quiz
Spark Dataframe Transformations
Exercise
Week 6: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Week 7: Apache Spark Caching In-Depth
1 Exercises • 18 Learning Materials
Downloadable Resources
Accessing Spark UI
Week 7 : Reference Jupyter Notebooks
ZIP
Distributed Processing - Pyspark In-Depth
Spark UI
Video
16:48
Accessing Spark UI
Steps for Accessing Spark UI
PDF
Understanding Cache & Persist
Video
23:25
Cache Practicals
Video
57:17
More on Cache
Video
39:57
Parsed | Analyzed | Optimized Logical Plan
Video
9:12
Cache - InMemory Table Cache | Node Local & Process Local
Video
24:49
Caching Spark Table
Video
34:46
Spark Catalog, Managed & External Tables
Video
34:2
Cache Performance
Video
30:00
Understanding Persist
Video
24:58
Running Notes & Summary Document
Week 7 Running Notes
PDF
Summary Document
PDF
Week 7: Quiz
Apache Spark Caching In-Depth
Exercise
Week 7: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Week 8: Apache Spark Architecture
1 Exercises • 16 Learning Materials
Downloadable Resources
Week 8 : Reference Jupyter Notebooks
ZIP
Spark Architecture | Aggregate & Window Functions
Spark On YARN Architecture
Video
37:54
More on Spark Architecture
Video
26:33
Ways of Accessing Columns in PySpark
Video
24:56
Simple Aggregate Functions
Video
26:4
Grouping Aggregations
Video
15:50
Windowing Aggregations
Video
16:6
Understanding Rank, Dense Rank & Row Number
Video
37:23
Understanding Lead and Lag Functions
Video
14:56
Analyzing Log Files
Video
21:40
Continuation of Analyzing Log Files
Video
18:58
Optimization - Pivot Table
Video
12:40
Running Notes & Summary Document
Week 8 Running Notes
PDF
Summary Document
PDF
Week 8: Quiz
Apache Spark Architecture
Exercise
Week 8: Assignment
Week 8 Assignment
PDF
Week 8 Assignment Solutions
Week 8 : Top Best Solutions
ZIP
Week 9: Apache Spark Internals
1 Exercises • 19 Learning Materials
Downloadable Resources
Week 9 : Reference Jupyter Notebooks
ZIP
Program File - Spark Submit
ZIP
Spark Internals & Dataframe Writer API
Dataframe Writer API
Video
40:15
PartitionBy Clause
Video
26:32
More on Partition's Performance Benefits
Video
37:22
Understanding Bucketing and its Performance Gains
Video
26:18
Accessing Spark UI using Databricks Community Edition
Video
24:54
Spark Internals
Video
30:4
Continuation of Spark Internals
Video
39:00
Disabling Dynamic Executor Allocation
Video
21:27
Spark-Submit at a High-Level
Video
42:4
Evaluating the Initial Partitions in a Dataframe
Video
23:41
Calculating the Initial Number of Partitions for a Single Non-Splitable file
Video
31:5
Calculating the Initial Number Of Partitions for Multiple Files
Video
28:9
Running Notes & Summary Document
Week 9 Running Notes
PDF
Summary Document
PDF
Week 9: Quiz
Apache Spark Internals
Exercise
Week 9: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Assignment Solution Program files
ZIP
Week 10: Apache Spark Optimizations 1
1 Exercises • 15 Learning Materials
Downloadable Resources
Week 10 : Reference Jupyter Notebooks
ZIP
PySpark Optimizations
Internals of groupBy
Video
25:37
Normal Join Vs Broadcast Join
Video
18:31
More on Broadcast Join
Video
16:25
Different types of Joins
Video
16:14
Partition Skew
Video
25:39
3 Use-cases & Better Optimizations
Video
19:7
Adaptive Query Execution (AQE)
Video
39:1
More on Join Types
Video
26:10
Join Strategies
Video
37:43
Optimizing Join of 2 Large Tables - Bucketing
Video
29:53
Running Notes & Summary Document
Week 10 Running Notes
PDF
Summary Document
PDF
Week 10: Quiz
Apache Spark Optimizations 1
Exercise
Week 10: Assignment
Weekly Assignment
PDF
Assignment Solution
PDF
Saturday Live Session Recordings
10 Learning Materials
Recordings of Saturday Live Sessions
8th Feb 2025 - Success Story Session
Video
49:1
15th Feb 2025 - Getting Started with AI for Data Engineers
Video
54:29
22nd Feb 2025 - Career Guidance Session
Video
54:28
Success story of Nehal Jaiswal 1st March 2025
Video
48:12
Live Session by Sumit Sir- Career Guidance - 15th March 2025
Video
1:23:13
Live Session by Sumit Sir- Career Guidance - 22nd March 2025
Video
1:28:2
How to get more interview calls ( in India and Outside India)-29 march 2025
Video
57:49
Gen AI webinar 5th April
Video
1:3:16
Live Session by Sumit Sir - Career Guidance - 12th April 2025
Video
1:21:25
Saturday Live Session - LinkedIn Profile Building - 26th April 2025
Video
1:1:27
Week 11: Apache Spark Optimizations 2
1 Exercises • 16 Learning Materials
Downloadable Resources
Reference Jupyter Notebooks
ZIP
Continuation of PySpark Optimizations
Memory Management in Apache Spark
Video
1:8:15
Sort Aggregate Vs Hash Aggregate
Video
20:32
Continuation of Sort Vs Hash Aggregate
Video
20:27
Apache Spark Logical and Physical Plans
Video
32:1
Catalyst Optimizer
Video
10:44
File Formats and Compression Techniques
Row Based and Column Based File Formats
Video
22:56
Continuation of File Formats
Video
37:37
Specialized File Formats - Parquet | Avro | ORC
Video
26:36
Continuation of Specialized File Formats
Video
50:46
Schema Evolution
Video
27:19
Compression Techniques
Video
35:17
Running Notes & Summary Document
Running Notes
PDF
Summary Document
PDF
Week 11: Quiz
Apache Spark Optimizations 2
Exercise
Week 11: Assignment
Weekly Assignment
PDF
Assignment Solution
ZIP
Week 12: Apache Spark Project Part 1
1 Exercises • 15 Learning Materials
Downloadable Resources : Project Datasets and Code
Datasets for the Lending Club Project
Lending Club Project Code : Jupyter Notebooks
ZIP
Lending Club Data Dictionary
ZIP
Lending Club Project
Key Elements of a Project
Video
19:13
Example Problem Statements
Video
39:37
Agile Methodology
Video
28:32
More on Agile Methodology
Video
29:22
Lending Club Project Introduction
Video
43:32
Lending Club Project Continuation
Video
42:56
Lending Club Project: Data Cleaning Session 1
Video
59:29
Lending Club Project: Data Cleaning Session 2
Video
39:42
Lending Club Project: Data Cleaning Session 3
Video
35:13
Lending Club Project: Data Cleaning Session 4
Video
18:29
Running Notes & Summary Document
Week 12 Running Notes
PDF
Summary Document
PDF
Quiz
Apache Spark Project Part 1
Exercise
Week 13: Apache Spark Project Part 2
1 Exercises • 22 Learning Materials
Downloadable Resources
Datasets for the Lending Club Project
Lending Club Project Code : Jupyter Notebooks
ZIP
Practice Datasets - Project Structuring
ZIP
Project Structuring - Code
PDF
Pytest Code File
PDF
Log4j Code
ZIP
Lending Club Project Part 2
Understanding Loan Score Calculation Logic
Video
25:23
Processing - Permanent Table Creation on Cleaned Data
Video
24:39
Access Patterns - Quick Access(Old Data) & Slow Access(New Data)
Video
23:16
Criteria for Loan Score Calculation
Video
18:9
Identifying the Bad Data
Video
30:33
Segregating the Identified Bad Data from the Normal Data
Video
15:21
Processing and Storing the Final Loan Score
Video
39:49
Project Structuring
Virtual Environments | Python | Pip
Video
43:2
Project Structuring & Execution
Video
55:34
Virtual Environment Setup (Python)
PDF
Unit Testing
Identifying and Writing Unit Test Cases | Fixture | Teardown - Yield
Video
38:52
Fixture to Check if the Calculated Results match Expected Results | Markers
Video
28:7
Parameterized Generic Test Cases
Video
15:50
Logging Level in Apache Spark
Implementing Logging Level in Apache Spark
Video
27:49
Running Notes & Summary Document
Week 13 Running Notes
PDF
Summary Document
PDF
Quiz
Apache Spark Project Part 2
Exercise
Week 14: GIT GITHUB & CICD
1 Exercises • 21 Learning Materials
Downloadable Resources
Retail Analysis Project Dataset
ZIP
Git & GitHub
Overview of Git & GitHub
Video
27:40
Git Installation | GitHub Account Creation | Visual Studio Installation
Video
19:17
Git & Visual Studio Code Installation
PDF
Important Git Commands | Scenario 1 - Project Creation through GitHub (Remote)
Video
33:59
Scenario 2 - Project Creation through Git (Local)
Video
23:7
Branches in Git
Video
26:43
Reverting back to the Previous Code Base
Video
21:4
Scenario 3 : Working on Existing Project (Fork Command)
Video
40:5
Git Stash Command
Video
19:9
Handling Merge Conflicts
Video
12:26
Continuous Integration & Continuous Deployment - CICD
Branching Strategy & Stages of CICD
Video
25:35
Deploying and Configuring Jenkins Server
Video
16:7
Branching Structure
Video
11:43
Jenkins Configurations
Video
16:3
Creating Sample Jenkins Pipeline
Video
16:54
Build | Test | Package & Deploy - Jenkins Pipeline for Project
Video
12:39
Continuation of Jenkins Pipeline for Project
Video
11:1
Document: Deploying & Configuring Jenkins Server
PDF
Running Notes & Summary Document
Week 14 Running Notes
PDF
Summary Document
PDF
Week 14: Quiz
GIT GITHUB & CICD
Exercise
Week 15: Apache Hive
1 Exercises • 15 Learning Materials
Apache Hive
Introduction to Apache Hive
Video
27:36
Apache Hive Practical
Video
23:45
Apache Hive Tables - Managed & External
Video
36:4
Apache Hive External Table
Video
28:3
Hive Optimizations - Partitioning
Video
30:5
Hive Optimizations - Bucketing
Video
19:15
Hive Join Optimizations
Video
37:57
Hive Join Optimizations Continuation
Video
22:23
Hive Transactional Tables | ACID Properties
Video
17:31
Hive Transactional Table Practical
Video
28:27
Insert Only Transactional Table with ACID Properties
Video
20:43
Spark-Hive Integration
Video
13:2
Hive MSCK Repair
Video
17:14
Running Notes & Summary Document
Running Notes
PDF
Summary Document
PDF
Week 15: Quiz
Apache Hive
Exercise
Course Instructor
Sumit Mittal
125 Courses • 370412 Students
By clicking on Continue, I accept the Terms & Conditions,
Privacy Policy & Refund Policy