This repository is the ultimate guide for mastering advanced Spark Performance Tuning and Optimization concepts and for anyone preparing for Data Engineering Interviews involving Spark. Additionally, this repository serves as a reference for all the code snippets used in my Spark Performance Tuning Playlist on YouTube. The goal of the playlist and the accompanying code snippets is to make complex concepts in Apache Spark easy to understand, while also developing a deep understanding of how things work under the hood.
| Concept | YouTube Link | Code |
|---|---|---|
| Spark Query Plans |  |
 |
| Spark DAGs | |
|
| Spark Memory Management | |
|
| Spark Executor Tuning | |
|
| Shuffle Partitions | |
|
| Data Partitioning | |
|
| Bucketing | |
|
| Caching | |
|
| Data Skew | |
|
| Salting | |
|
| AQE & Broadcast Joins | |
|
| Dynamic Partition Pruning | |
|
For any questions or feedback, feel free to reach out:
- Afaque Ahmad - LinkedIn
- GitHub Issues - Open an Issue