Three billion people are doing the exact same thing at the same time. Posting. Scrolling. Messaging. Watching. And somehow the whole thing just works. No crashes. No waiting. No downtime.

The answer is not more servers. It is not a bigger database. It is a fundamental design decision that most engineers never think about until their system is already on fire.

Meta treats reads and writes as two completely separate problems.

In most systems, reads and writes share the same path. The same database handles both. Which means when one gets busy, the other suffers. Your users are trying to load their feed while your pipeline is ingesting millions of new events at the same time. They are fighting over the same resources. And under load, everybody loses.

Most engineers respond to this by adding memory, scaling horizontally, or upgrading their database tier. None of that fixes the actual problem. Because the actual problem is architectural, not operational.

Meta solved it by building two completely different systems from scratch.

LOGDEVICE: BUILT FOR WRITES ONLY

LogDevice is Meta’s distributed log storage system. It was designed with one purpose: ingest data at massive speed without ever slowing down.

Every like, every message, every video view, every backend sensor ping across three billion users - LogDevice takes it all in. It uses a log-structured approach which means it writes data sequentially rather than randomly jumping around the disk. Sequential writes are dramatically faster than random writes. That is not an accident. That is a deliberate design choice made specifically to maximize write throughput.

LogDevice does not care about reads. It was never designed to serve reads efficiently. That is the point. By giving up on reads entirely, it becomes extraordinarily good at the one thing it was built for.

Most engineers designing their first production system try to pick one database that handles everything. LogDevice is the proof that this instinct, while understandable, is wrong at scale.

ROCKSDB: BUILT FOR READS WITH SURGICAL PRECISION

RocksDB started as Google’s LevelDB. Meta took it, rebuilt it, and open sourced it in 2013. Today it powers systems at Facebook, LinkedIn, Yahoo, Twitter, and hundreds of other companies running at scale.

The reason Meta built RocksDB instead of using an existing solution is the same reason they built LogDevice. Nothing on the market gave them the control they needed.

RocksDB is an embeddable key-value store that lets you tune read and write performance independently at the instance level. This is the part most engineers miss.

You can deploy one RocksDB instance configured entirely for fast point lookups optimized for the read patterns of a news feed where you need to retrieve a specific user’s data in milliseconds. You deploy another instance configured for high write throughput - optimized for the ingestion patterns of an analytics pipeline processing billions of events. Same underlying technology. Completely different configurations. Completely different jobs.

They never compete for the same resources because they were never meant to run the same workload.

RocksDB also uses a data structure called an LSM tree - Log Structured Merge tree which batches writes in memory and flushes them to disk in sorted order. This makes writes fast and keeps related data physically close together on disk. When you request data, the disk has to seek less to find it. Less seeking means faster reads. Meta takes this even further by pre-arranging the most frequently accessed bytes so they are physically adjacent on disk. The result is a feed that loads in milliseconds regardless of how many people are using it simultaneously.

WHY THIS MATTERS FOR YOUR SYSTEM RIGHT NOW

You are probably not building for three billion users. But the principle applies at every scale.

If you have a system that slows down under load, the first question to ask is not what hardware do I need. The question is are my reads and writes competing for the same resources.

Leave a comment

I have seen this exact problem at companies with 50 engineers and companies with 5,000. A shared database handling both analytical queries and transactional writes. A single Kafka consumer group processing both real-time and batch workloads. One pipeline serving five different use cases with completely different performance requirements.

The symptom is always the same. Things work fine until load increases. Then everything degrades together because everything is coupled together.

The fix is always the same too. Separate the concern. Define the job. Build for that job specifically.

LogDevice does not try to be RocksDB. RocksDB does not try to be LogDevice. And Facebook never goes down.

Here is the three step framework I apply before designing any new data system:

Step one. Write down every read pattern your system needs to support. How frequently. What latency is acceptable. What the data shape looks like.

Step two. Write down every write pattern separately. How much volume. How fast does it need to land. What consistency guarantees do you need.

Step three. Ask honestly whether one system can serve both patterns without compromising either. If the answer is no, you already know what to do.

Separate the concern first. Then optimize. That is how you build something that survives contact with real traffic.

If you found this valuable, Thursday’s paid newsletter goes even deeper.

Not subscribed yet? Now is a good time.

I am breaking down the exact career moves that separate engineers who understand systems from engineers who just operate them. The difference in compensation between those two groups at companies like Meta is not small.

Thursday 5:30pm. Paid subscribers only.

See you Thursday.

— Avantika

Hi everyone,

Before we begin, I want to share a quick, honest note and sincere apologies to all my readers.

I’ve been quieter than usual over the last few months. I was dealing with some personal and health-related challenges and needed to step back briefly. Thank you for your patience, messages, and continued support it truly means more than you know.

I’m back now, and it felt right to restart with a topic that sits at the very core of data engineering one that has quietly shaped almost every system I’ve worked on.

Table of Contents

• What is SQL in Data Engineering?

• Why SQL is Crucial for Data Engineering

• SQL for ETL vs. ELT Pipelines

• Essential SQL for Data Engineers

• SQL in Modern Data Engineering Tools

• Best Practices for Writing SQL in Data Engineering

• Future of SQL in Data Engineering

• Conclusion

• FAQs

Prefer listening over reading?

Are you on your way to work or heading back home?
Starting a run, folding laundry, or just taking a quiet break?

If reading feels like too much right now, I’ve got you.

I recorded an audio version of this newsletter so you can listen while you move through your day. Same ideas, same depth just in a format that fits real life.

Plug in your headphones, press play, and let SQL make sense in the background while you take care of everything else.

Introduction

Structured Query Language (SQL) remains the foundation of data engineering, enabling data professionals to design, build, and maintain scalable data pipelines. Despite the rise of modern technologies like Apache Spark and NoSQL databases, SQL’s declarative syntax and universal adoption make it indispensable in real-world data engineering workflows.

In this piece, I’ll walk you through how SQL shows up in real data engineering work, what actually matters in practice, and why it continues to be one of the most valuable skills you can invest in as a data engineer.

What is SQL in Data Engineering?

Meta story: Early in my career, I believed mastering tools would make me a great data engineer. Spark, Airflow, Kafka I chased them all. What actually made my work reliable wasn’t a tool. It was the moment I truly understood SQL as a way of thinking: describing what the data should look like, not how to move every row. That shift changed how I designed pipelines forever.

At its core, SQL (Structured Query Language) is the language we use to talk to data stored in relational systems to ask questions, shape answers, and turn raw records into something meaningful.

In data engineering, SQL is used to:

• Ingest raw data

• Clean and validate datasets

• Transform data into analytics-ready models

• Load data into warehouses and lakes

SQL acts as the linchpin of both ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) pipelines, making it the backbone of modern data platforms.

Why SQL is Crucial for Data Engineering

Project moment: On one production pipeline, we had Python transformations, custom logic, and retries everywhere and still broke SLAs weekly. The fix wasn’t a rewrite. It was replacing fragmented logic with clear, well-structured SQL. Fewer lines. Fewer bugs. More trust.

1. Data Extraction

SQL makes it surprisingly easy to pull data from structured systems like PostgreSQL, MySQL, and Oracle, and even from modern platforms that support SQL-style querying such as BigQuery and Redshift.

2. Data Transformation

Data engineers rely on SQL for cleansing, aggregation, and normalization. Features like Common Table Expressions (CTEs), window functions, and subqueries allow complex transformations to remain readable and maintainable.

3. Data Loading

In practice, SQL-powered pipelines move data into warehouses and lakes in a way that keeps analytics teams productive and downstream systems stable.

4. Data Integration

By joining datasets across multiple systems, SQL helps engineers create unified data models that power reliable reporting and decision-making.

5. Performance Optimization

Modern SQL engines such as Apache Hive, Presto, and Spark SQL provide query optimization capabilities that reduce execution time, improve resource utilization, and scale analytics workloads.

SQL for ETL vs. ELT Pipelines

Career insight: I’ve seen engineers struggle not because they chose ETL or ELT, but because they didn’t understand where SQL belongs in the system. Once you see SQL as a first-class layer not an afterthought architectural decisions become simpler.

SQL sits at the center of both approaches. In recent years, ELT has become more common simply because cloud warehouses make large-scale transformations easier and cheaper to run in parallel.

Essential SQL for Data Engineers

Interview reality: Almost every senior data engineering interview I’ve seen comes down to this section. Not syntax trivia but whether you can express business logic clearly, safely, and efficiently in SQL.

1. Window Functions

These are used when you need running totals, rankings, or comparisons across groups without losing row-level detail.

SELECT
customer_id,
order_date,
SUM(order_amount) OVER (
PARTITION BY customer_id
ORDER BY order_date
) AS cumulative_sales
FROM orders;

2. Common Table Expressions (CTEs)

CTEs make complex logic easier to read, reason about, and safely modify over time.

WITH recent_orders AS (
SELECT order_id, customer_id, order_date
FROM orders
WHERE order_date > '2026-01-01'
)
SELECT * FROM recent_orders;

3. Joins

Most real-world datasets only make sense once multiple tables are joined together this is where SQL earns its keep.

SELECT customers.name, orders.order_id
FROM customers
JOIN orders
ON customers.customer_id = orders.customer_id;

4. Indexes and Query Optimization

Indexes improve read performance, while query planners and EXPLAIN statements help identify bottlenecks in large-scale systems.

5. Data Partitioning

Partitioning large tables improves performance in distributed systems such as Hive and BigQuery by limiting the amount of data scanned.

SQL in Modern Data Engineering Tools

Meta observation: Tools change faster than job titles. What stays constant is SQL acting as the common language across platforms the one skill that transfers cleanly when stacks evolve.

ToolPurposeSQL RoleApache HiveData warehousing on HadoopHiveQL for querying HDFSApache Spark SQLDistributed data processingSQL on DataFramesGoogle BigQueryServerless data warehouseStandard SQLAWS RedshiftCloud data warehousePostgreSQL-like SQLSnowflakeCloud data platformANSI SQLdbtData transformationSQL-based modeling

Best Practices for Writing SQL in Data Engineering

Production lesson: If someone else can’t understand your SQL six months later including you it will eventually cost time, trust, or money. Readability is not optional in production systems.

1. Use CTEs for complex logic
   Break queries into logical steps to improve readability and maintainability.

2. Avoid SELECT *
   Explicitly select required columns to reduce data scanning and improve performance.

3. Leverage indexes and partitioning
   Use clustering, partition keys, and indexes to optimize large datasets.

4. Monitor query performance
   Analyze execution plans using EXPLAIN statements to identify inefficiencies.

5. Follow data governance standards
   Ensure compliance with organizational policies around data security, privacy, and access control.

Future of SQL in Data Engineering

Forward-looking thought: SQL isn’t competing with new paradigms it’s absorbing them. Streaming, federated queries, and data mesh architectures are all bending toward SQL as the shared interface.

Despite the growth of NoSQL and distributed systems, SQL’s declarative nature ensures its continued relevance.

Key trends shaping the future:

• SQL on streaming data using platforms like Apache Flink and ksqlDB

• Federated queries enabling cross-platform data access

• SQL in data mesh architectures as a shared querying layer across decentralized domains

SQL is not being replaced it is evolving alongside modern data architectures.

Conclusion

Closing reflection: Every time data systems fail, the root cause is rarely “bad data.” It’s unclear logic. SQL, when written well, makes intent explicit and that’s why it continues to matter.

SQL is far more than just a querying language. It is the backbone of data engineering, powering data ingestion, transformation, integration, and analytics at scale.

As tools and platforms evolve, SQL’s clarity, expressiveness, and adaptability ensure it remains an essential skill for data engineers, data scientists, and analytics professionals.

FAQs

Is SQL used in data engineering?
Yes. SQL is fundamental to data engineering and is used extensively for data extraction, transformation, loading, validation, and modeling.

How do I become a SQL data engineer?
Build a strong foundation in SQL and database systems, practice query optimization, learn data modeling, and gain hands-on experience with modern cloud data warehouses. Complement SQL with Python for automation and orchestration.

Is SQL still relevant in 2026?
Absolutely. SQL remains one of the most in-demand skills due to its deep integration with cloud platforms, analytics tools, and modern data stacks.

Is Python and SQL enough for data engineering?
They form a strong foundation, but data engineers also benefit from learning data orchestration tools, distributed systems, and cloud platforms.

Should data engineers know SQL?
Yes. SQL is essential for building reliable data pipelines, modeling data, and ensuring data quality.

What are some of the best SQL courses for data engineers?

• PostgreSQL for Everybody (Coursera)

• SQL Fundamentals (Dataquest)

• The Ultimate MySQL Bootcamp (Udemy)

• Complete SQL Mastery (CodeWithMosh)

• Advanced SQL for Data Engineering (Udemy)

🔗 Follow Avantikka Penumarty on LinkedIn

🔒 Want to turn this into real skill?
Access interactive SQL flashcards, quizzes, and AI-powered explanations in the paid section below.

Take This From Insight to Instinct

Reading builds understanding.
Practice builds confidence.

If SQL plays a role in your job, interviews, or long-term growth, passive reading isn’t enough.

I’ve created an interactive learning extension to this newsletter designed to help you think in SQL, not just recognize syntax.

Inside the paid section, you’ll get:

• 🧠 Flashcards to reinforce core SQL concepts and mental models

• 🧪 Scenario-based quizzes that mirror real data engineering decisions

• 🤖 AI-powered explanations that walk you through the why, not just the answer

• 🏗️ Applied reasoning you can reuse in production systems and interviews

This is how you move from:

“I’ve read this”
to
“I can apply this under pressure.”

Read more

“Airflow is legacy!”
“Dagster is the future!”
“Prefect is so clean!”

But you’re not here for debate. You’re here to ship pipelines, learn fast, and get hired.

So let’s get real:
Which tool should you pick — for your project, for your portfolio, or for your resume?

This guide won’t compare every feature.
It’ll show you how to think like a system builder, not a tool fangirl.

1. When to Use Airflow

Use Airflow when:

• You’re applying to big tech or mature data teams

• You want to show you understand orchestration, retries, and scheduling

• You need strong scheduling and visibility (UI, logs, retries)

Why it works for your resume:

“Familiar with industry-standard orchestration using Airflow, including DAG design, sensors, retry logic, and alerting.”

Why it’s great for teaching yourself the fundamentals:
Airflow makes you learn how pipelines actually run: tasks, triggers, failures, dependencies. It exposes orchestration in raw form.

Where it’s weaker:

• Not great for data scientists or notebooks

• Heavier to deploy without Docker

• Harder to reason about with dynamic workflows

2. When to Use Prefect

Use Prefect when:

• You’re building a lightweight project

• You don’t want to mess with Airflow configs

• You like writing clean Python with decorators

Why it works for solo projects:

• It’s Pythonic and elegant

• Easy to get up and running

• Great docs, fast dev feedback loop

Why it works for data science/analytics engineers:

• You can orchestrate model training, dbt, API tasks quickly

• You don’t need to explain DAGs to non-engineers

Where it shines on a resume:

“Used Prefect to orchestrate model training and dbt transformations in a low-latency ML workflow.”

Where it’s weaker:

• Less recognized by recruiters

• Smaller community

• Not ideal for heavy-duty, multi-team orchestration

3. When to Use Dagster

Use Dagster when:

• You care about type safety, IO contracts, and observability

• You want to model your data pipeline like software

• You’re building a data platform, not just a DAG

What makes Dagster interesting:

• Built-in concepts like assets, software-defined pipelines

• First-class support for data lineage and testing

• Powerful for collaborative teams that want engineer-level visibility

On your resume:

“Designed asset-aware DAGs in Dagster to enforce data lineage, retry logic, and schema-aware contracts.”

Where Dagster may not be right:

• Overkill for simple projects

• Less intuitive for beginners

• Still evolving rapidly — may have rough edges

What Your Tool Choice Says About You

If you picked...You’re signaling...Airflow“I understand production-scale systems.”Prefect“I move fast and iterate cleanly in Python.”Dagster“I think in contracts, lineage, and scale.”

There is no “best” tool — only the one that aligns with:

• The story you want to tell

• The type of work you want to do

• The types of teams you want to join

The Hidden Skill Behind Tool Choice: Narrative Alignment

The tool you use reflects how you frame problems.

Interviewers aren’t just looking at which orchestrator you picked — they’re evaluating whether your mental model matches the role.

• Use Airflow if you want to tell a story about scale, resilience, and legacy integration

• Use Prefect if you want to show you’re experiment-driven, lean, and fast-moving

• Use Dagster if you want to prove you're a system thinker who designs for complexity and traceability

This is less about syntax, more about signal. Make your choice reflect your intended audience.

What Top Companies Use (And Why It Matters)

If you're applying somewhere — match your project tool with what that company runs in production (or wants to).

How to Frame This on Your Resume

Don’t write:

“Used Airflow for DAG orchestration.”

Write:

“Designed idempotent, retry-aware data pipelines using Airflow for scalable ETL orchestration across S3, Spark, and Snowflake — with SLA tracking and alerting.”

Or:

“Built fast, modular Prefect flows to orchestrate data science pipelines with resume-to-dashboard visibility in under 15 minutes per run.”

Or:

“Architected lineage-aware pipelines using Dagster assets and config mapping to reduce failure recovery time by 80% across multiple data teams.”

The tool isn’t the flex. The system design behind it is.

Thinking Like a Hiring Manager

If I’m hiring:

• Airflow tells me you’ve worked with mature pipelines and understand operational burden.

• Prefect tells me you can build fast, are probably solo or hybrid (data + product), and can ship.

• Dagster tells me you write pipelines like software — clean, testable, typed — and are comfortable with architecture decisions.

If you’re junior? Pick one and go deep.
If you’re senior? Know when to choose which — and explain it.

Final Advice

Don’t just pick the tool that’s hyped.
Pick the one that matches how you want to think and explain your work.

What you build is one thing.
How you talk about it is what gets you hired.

“Yeah, I’ve used Airflow.”

But interviewers can tell within 60 seconds whether you actually understand it — or just ran someone else’s DAG.

This post is not a checklist of features.
It’s a breakdown of the top Airflow mistakes that break pipelines, burn teams, and ruin interviews — and how to avoid them.

Let’s make you sound like someone who’s deployed DAGs in production, not just built toy examples.

Mistake #1: Treating Airflow Like a Script Runner

If your tasks are running huge pandas transformations, calling 15 APIs, and returning massive objects between tasks, you're doing it wrong.

Airflow is not Spark. Not dbt. Not a transformation engine. It’s the orchestrator — the conductor of the data workflow.

Do this instead:

• Keep tasks modular

• Push heavy lifting into external jobs (Spark, SQL, cloud ETL)

• Use Airflow to manage dependencies, not business logic

How to say this in interviews:

“I used Airflow to orchestrate pipeline steps, but offloaded heavy data processing to Snowflake and Spark jobs to keep the DAGs lean and observable.”

Mistake #2: Ignoring Retry Logic and Failure Handling

Most junior engineers write DAGs that work… when nothing goes wrong.

But in production:

• APIs time out

• S3 files get delayed

• Database connections drop

Best practices:

• Always configure retries, retry_delay, and on_failure_callback

• Log why each task failed (don’t just rely on the default log dump)

• Use idempotent task design — so retries don’t break downstream logic

Pro-level tip: Add exponential backoff and SLA alerts.

Upgrade to Annual

Read more

Data arrives late.
APIs fail.
Files drop into S3 at random.
And your pipeline has to wait, trigger, or backfill — not just run on a timer.

Today, we’re digging into how scheduling really works — and how you should answer when interviewers ask:

“How do you schedule and trigger your Airflow DAGs?”

First: Understand the Two Types of Runs

1. Scheduled Runs
   You tell Airflow to run a DAG every X time:

   • Every hour, day, week, etc.

   • Use cases: batch ETL, daily reporting, metrics updates

2. Triggered Runs
   Airflow runs the DAG when something happens:

• A file lands in S3

• An upstream DAG completes

• An API returns a signal

Elite Bonus Drop Coming:
This Thursday, I’ll walk through Sensors, ExternalTask dependency patterns, and DAG chaining in a real multi-DAG system.

This is where your interview answers start sounding like a Staff Data Engineer.

UPGRADE TO ANNUAL MEMBERSHIP

Read more

Today’s goal?
You’ll build your first working DAG — start to finish — that does something useful and gets you closer to job-ready.

If you can complete this, you’ll be able to walk into any interview and say:

“Yes, I’ve built production-style pipelines in Airflow.”

Let’s do this.

Why This Matters

Anyone can Google "What is Airflow?"
You’re here to build with it — and speak about it like an engineer who’s done it in prod.

Today’s drop isn’t theory.
You’ll create your first working DAG — a mini pipeline that does something useful, reliable, and interview-worthy.

You’ll leave this newsletter with:

• A runnable DAG

• A deployable GitHub project

• A STAR-based story you can use in interviews

• And clarity about what makes DAGs production-grade

Upgrade & Learn in Detail!

What You’re Building

The business use case is real:
A team receives a messy CSV of order data every few hours. Right now it’s being cleaned manually. You’ve been tasked with automating this.

Here’s what your DAG will do:

• Extract: Read CSV

• Transform: Drop nulls, clean types

• Load: Write to Snowflake

• Retry: If anything breaks

• Alert: Slack notifications

• Schedule: Every 6 hours

Your Airflow Starter Pack

Step 1: Install Airflow (If not done yet)

pip install apache-airflow
airflow db init
airflow webserver --port 8080
airflow scheduler

Open Airflow UI at localhost:8080 and create your user.

Read more

Not the kind you can meditate through —
The kind that breaks your data pipeline at 2AM while your dashboards go blank and your PM starts Slacking you with “???”

That’s why Airflow exists.

Airflow doesn’t just help you schedule things.
It’s the brain of your data pipelines — the system that says:
“Hey, this job failed, let’s retry.”
“Wait, don’t run until the upstream finished.”
“Log it, alert it, and move on.”

And yet, for most people, Airflow seems intimidating.

Today, we’ll break it down — no fluff, no buzzwords.
By the end of this, you’ll understand what Airflow really does, how DAGs work, and how to confidently talk about orchestration in any interview.

What Airflow Actually Does

Forget the textbook.

Here’s what Airflow really handles:

• Running your ETL jobs in the right order

• Retrying failed jobs automatically

• Monitoring status of every task

• Sending alerts if something breaks

• Triggering tasks on a schedule or based on dependencies

That’s orchestration. It’s not the “doing” — it’s the directing.

Airflow isn’t processing your data. It’s making sure the process happens correctly.

How DAGs Work (Without the Jargon)

A DAG = Directed Acyclic Graph = A set of tasks that:

1. Have a clear start and end

2. Never loop back on themselves

3. Run in a specific order

Think of it like a checklist for your pipeline:

• Step 1: Pull data from S3

• Step 2: Transform using Spark

• Step 3: Load to Snowflake

• Step 4: Run quality checks

• Step 5: Refresh dashboard

Each of these is a task.
The entire thing is a DAG.

In Airflow, these steps are defined in Python. You set dependencies like this:

extract_task >> transform_task >> load_task

UPGRADE TO ELITE/ANNUAL MEMBERSHIP

Read more

You walk into a Data Engineering interview. They ask:

“Can you walk me through an ETL pipeline you’ve built?”

Seems basic, right?
But here’s the catch: they’re not looking for just a tool dump.
They’re trying to reverse-engineer your thinking.

You’ve now built ETL pipelines, cleaned real data, and understood how batch vs streaming works.

Today’s goal is simple. In Zero2DataEngineer breakdown, we’ll decode how ETL interview questions are framed, what they’re secretly testing, and how to structure your answers like a pro — even if you’ve never worked at FAANG.

Whether you’re applying for a Data Engineer, Analytics Engineer, or even Backend-heavy role — ETL questions will show up.

Here’s how to answer them like someone who’s done it before.

How ETL Interview Questions Are Really Framed

They won’t ask:

“What is ETL?”

They’ll ask:

• How would you design a pipeline to load millions of rows from an external source?

• What happens if your load step fails halfway through?

• How do you make your pipeline idempotent?

• When do you use batch vs real-time ingestion?

The trick is to answer like a system thinker, not just a coder.

Subscribe now

Interview Answer Formula (Reframe Your Thinking)

Use this ETL STAR + Stack Formula when answering:

Situation: What was the use case?
Tool stack: Which tools did you choose and why?
Architecture: Show the pipeline stages.
Resilience: How did you handle failures, alerts, schema drift?
+ Stack Justification: Why this combo (Airflow + S3 + Spark etc.)?

Pro tip: If you haven’t built one end-to-end yet, use this:

“Here’s how I would design it for a [use case].”
Then walk them through your design — intelligently, not hypothetically.

Master These Real ETL Questions Before Your Next Interview

Q1. What are some common challenges in ETL pipelines?
1. Handling bad records
2. Schema changes
3. Late-arriving data
4. Dependencies & retries

Sample Answer:

“One of the biggest challenges I’ve faced is handling schema drift — especially when upstream sources silently change a column name or data type. I’ve built schema validation into the extraction step using Great Expectations and version control through Glue Catalog.

I’ve also handled bad records by logging and quarantining them into a separate S3 bucket with alerting. For late-arriving data, I design pipelines to be idempotent — using UPSERT logic or late data windows. And for dependencies, I always make sure DAGs have proper depends_on_past, retry logic, and failure alerts configured.”

UPGRADE TO ANNUAL

Read more

Now the question is:
How often should your pipeline run?
And more importantly… should it run in batches — or in real time?

Let’s break it down.

What Is a Batch Pipeline?

A batch pipeline runs on a schedule.
It pulls a large chunk of data at once — typically daily, hourly, or weekly.

Think:

• Nightly revenue reports

• Weekly customer churn rollups

• Monthly sales summaries

It’s like picking up laundry every Sunday.
No need to track every sock in real time — just do one large pickup.

Upgrade to Annual

When to Use Batch

• Your data changes slowly (e.g., payments, orders)

• You’re running reports, not alerts

• You want to keep cloud costs low

• You need high data completeness over speed

Read more

This week, we’re shifting from data structure to data movement.
Because understanding SQL is table stakes — but knowing how raw data becomes trustworthy, usable insights?

That’s where ETL comes in.

What the Hell Is ETL?

ETL stands for:

• Extract – pull raw data from source systems

• Transform – clean, reformat, validate, and reshape

• Load – push structured data into a final destination

That’s it — in theory.

But in practice, ETL is messy, strategic, and essential.

Let’s Make It Real: The Restaurant Analogy

You're running a restaurant.

• Extract = ordering raw ingredients from vendors

• Transform = chopping, cooking, seasoning, plating

• Load = serving the final dish to the customer

You wouldn’t send a customer raw flour and onions.
That’s what skipping ETL looks like in a data system.

UPGRADE TO ANNUAL

Read more

You’ve learned SQL.
You’ve learned schema design.
You now understand indexing, ACID, joins, and denormalization.

But here’s the part nobody tells you:

SQL isn’t just a query language.
It’s a way of thinking about data.

The best data engineers? They don’t “run SQL.”
They design systems that make SQL sing.

The Layer Cake of Modern Data

Let’s pull back the curtain. When you're working in a company like Stripe, Airbnb, or Netflix, your data system isn’t just tables and dashboards.

It’s a layered architecture, and SQL flows through it all:

1. Raw Layer – messy, fast, unfiltered (S3, GCS)

2. Staging Layer – cleaned, typed, deduped

3. Warehouse Layer – modeled, relational, denormalized

4. Semantic Layer – Looker, Tableau, PowerBI

5. Delivery Layer – Dashboards, APIs, notebooks

SQL is the thread that weaves it all together — whether you write it, generate it with dbt, or visualize it through tools.

So What’s the Real Job of SQL?

UPGRADE TO ANNUAL

Read more

In data engineering, pipelines break, APIs fail, jobs timeout.
But the data itself?

That still needs to be correct.

Today we talk about how databases keep your world from falling apart — even during chaos.

What is a Transaction?

A transaction is a single unit of work in the database.

• Transfer money? Transaction.

• Insert a row? Transaction.

• Update 3 tables in sequence? Still a transaction.

A good system either completes the transaction entirely or rolls everything back — no half-baked updates.

The ACID Model (Explained Like You’re On-Call)

Why this matters:
You don’t want to deduct money from a customer… but fail to generate the order.
You also don’t want two people editing the same record and overwriting each other’s data.

Real-Life Scenario: E-Commerce Checkout

Imagine a customer places an order. Here’s what happens:

1. Add a row in orders

2. Deduct stock from inventory

3. Charge the customer in payments

4. Send confirmation email

Without transactions, if step 3 fails after step 1 & 2 succeed…

• Inventory is gone

• No payment

• No order shipped

• No email sent

Now your support team is flooded.

With transactions, all 4 steps are wrapped in one block. If payment fails, nothing is saved.

SQL in Action

BEGIN;

INSERT INTO orders (...) VALUES (...);
UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 123;
INSERT INTO payments (...) VALUES (...);

COMMIT;

If anything fails before COMMIT, Postgres rolls it all back. No manual cleanup. No missing rows. No silent data corruption.

Interview Angle: How to Stand Out

UPGRADE TO ANNUAL MEMBERSHIP

Read more

Because all the cleanest SQL in the world means nothing…
If your query takes 17 minutes and your dashboard cries blood.

Why Indexing Matters

Imagine searching for a contact in your phone:

• No Index: Scroll, scroll, scroll… manually.

• With Index: Type “S” → boom, “Sara” shows up instantly.

That’s what indexes do to your database.
They help the engine locate what you need — without scanning every row.

Why Indexing Exists

Imagine walking into a bookstore with no signage.
No sections, no labels. Just 100,000 books in a pile.

That’s your database without an index.
Even the simplest query becomes a full table scan — the database flips through every row, like a librarian on Red Bull.

With an index?
You give the DB a map. It jumps to exactly where the answer lives.

The Two Most Common Index Types

Deep Dive: B-Tree vs Hash Index

Interview tip: Most RDBMS default to B-Tree, so unless otherwise needed, go with that.

Real World Example: The Wrong Index Costs Real Money

When I worked on a loyalty points table with 30M+ rows,
a teammate created an index on user_id.

Problem?
All queries filtered on created_at That index was useless.

We swapped it to:

CREATE INDEX idx_created_at ON loyalty_points (created_at);

Result: Dashboard load time dropped from 38s → 2s.
That tiny change saved $600/month in Snowflake compute credits.

Real-World: How One Index Saved a Failing Job

At a logistics startup, we had a shipment tracking job that queried a 60M row tracking_events table every 5 minutes.

We thought it was clean:

SELECT * FROM tracking_events WHERE status = 'delivered';

But it took 45 seconds.

The problem? No index on status.

Worse — status had only 3 values (in_transit, delivered, failed) → low cardinality.

So indexing that didn’t help. Instead, we looked at event_time, which had high uniqueness and was used in the WHERE clause:

CREATE INDEX idx_event_time ON tracking_events (event_time);

Query dropped to 2.1 seconds.

And that’s when we learned:

Index the column that filters the MOST rows — and does so selectively.

UPGRADE TO ANNUAL

Read more

Today, we go deeper into how schema design can either streamline your data pipeline… or turn it into a spaghetti monster. we unpack how your database schema shapes everything — speed, cost, logic, scalability.

And no, it’s not just theory.

Good schema = fast queries, fewer bugs, and easier handovers.
Bad schema = JOIN hell, duplication drama, and angry dashboards.

What’s the Difference?

Normalization Forms (Real-World Style)

1NF – First Normal Form

• Rule: Atomic columns, no repeating groups

• Think of it like: Every drawer in your closet holds only one type of item

• Example:
  Bad: Phone Numbers = 123, 456
  Good: Separate rows per number

2NF – Second Normal Form

• Rule: 1NF + all columns depend on entire primary key

• Think of it like: No room for half-relevant details

• Example: Don't include Student Name in a table where the primary key is (Student_ID, Course_ID)

3NF – Third Normal Form

• Rule: 2NF + no transitive dependencies

• Think of it like: No gossip. Columns shouldn't depend on other non-key columns.

• Example: Move Department_Location to a separate Departments table instead of placing it in Employees.

Real Life Scenarios

Fintech Use Case: Normalized Schema (OLTP)

Let’s say you're working at a fintech startup on a banking system. You separate:

• Customer Table → updated rarely

• Accounts Table → moderate frequency

• Transactions Table → updated constantly

Why this works: Different update rates + data integrity + modular ETL pipelinesYour transactional system handles:

• Customers

• Accounts

• Transactions

Instead of storing everything in one big table, you normalize:

• One table for customer details

• Another for account info

• A third for transactions

Why?
Because each table changes at different speeds and scales. Transactions update constantly. Customers, rarely. This improves storage, indexing, and update reliability.

UPGRADE TO ANNUAL SUBSCRIPTION

Read more

This week we leave behind raw querying and step into schema thinking.

This is where you stop being a query-runner and start becoming a real engineer.

Here’s what we’ll cover:

• How relational databases actually work

• The power of constraints and keys

• When to choose OLTP vs OLAP

• How broken schema design breaks pipelines

• And how to crush RDBMS interview questions

What Even Is a Relational Database?

It’s a system that stores data in tables — with clearly defined relationships between them.

Think:

• users

• orders

• products

• payments

Each table is connected using primary and foreign keys, ensuring data consistency, referential integrity, and scalable querying.

Real Example: Food Delivery App

You’re building for DoorDash, Swiggy, or Zomato.

Here’s your schema:

• users(user_id, name, location)

• restaurants(restaurant_id, name, cuisine)

• orders(order_id, user_id, restaurant_id, created_at)

• order_items(order_item_id, order_id, item_name, price)

Now someone deletes a restaurant record.

If you didn’t use foreign key constraints, your orders table is now pointing to... nothing.
Your BI dashboard breaks. Refunds misfire. Customers rage tweet.

This is why RDBMS still matters — it protects you from yourself.

Schema Mistakes I’ve Lived Through

Mistake #1: I once stored prices in 3 different places — product table, invoice table, and analytics table.
Marketing ran a 10% off promo.
One table got updated. Two didn’t.
Revenue numbers were off. Refunds had to be manually processed. Chaos.

Fix: I created a normalized schema where pricing logic sat in one place, and every other table referenced it via foreign key. Revenue dashboards started matching Stripe. Everyone slept better.

UPGRADE TO ANNUAL

Read more

This isn’t a wrap-up.
This is a transformation checkpoint.

By now, you’ve learned:

• How to clean data with pandas

• How to call APIs and normalize JSON

• How to handle files like a production engineer

Today is about how to:

• Debug like a pro

• Structure your folders and logs

• Turn those learnings into 3 real portfolio projects

Because employers don’t hire people who know Python —
They hire people who used it to build something real.

What DEs Actually Do with Python

DE-Style Script Structure

Here’s how I organize every serious Python script:

project/
├── data/
│   ├── raw/
│   ├── processed/
│   └── archive/
├── logs/
│   └── run_2024-04-26.log
├── scripts/
│   └── clean_users.py
└── config/
    └── secrets.env

This structure tells future-you:

• What was cleaned

• When it ran

• Where it went

• What config powered it

  UPGRADE TO ANNUAL

Read more

A real data engineer doesn’t say:

“Let me just load this CSV and print the head.”

They say:

“Let’s structure this folder, validate file size, and ingest in chunks with logging.”

Today’s lesson:
Python for production-grade file handling.

Files You’ll Actually Handle in the Wild

You’re not reading files for fun.
You’re turning them into clean, usable, repeatable pipeline inputs.

Real Example: Chunk Reading a Large CSV

import pandas as pd

chunks = pd.read_csv("big_sales.csv", chunksize=50000)

for i, chunk in enumerate(chunks):
    cleaned = chunk.dropna(subset=["customer_id"])
    cleaned.to_csv(f"cleaned_sales_part_{i}.csv", index=False)

This is how DEs handle:

• Large files without crashing memory

• Splitting clean output into batches

• Ensuring pipeline resiliency

Read more

Most companies don’t give you tidy CSVs.
They give you access to an API.

Your job?

• Authenticate

• Call it

• Handle errors

• Parse the JSON

• Normalize it into something that fits your pipeline

Today’s lesson will teach you how to go from “what’s an API?” to “here’s how I ingest Stripe, Notion, or any public data feed.”

Why APIs Matter for Data Engineers

UPGRADE TO ANNUAL

Read more

Data cleaning isn’t sexy.
But neither is debugging a broken dashboard at 3AM.

This lesson is about:

• Detecting messes before they cause problems

• Structuring your cleanup scripts for clarity

• Building confidence in your data before it ever hits SQL

Because you can’t query your way out of a dirty dataset.

What Messy Data Actually Looks Like

You won’t just see typos or NULLs.

You’ll face:

• Extra columns named Unnamed: 12

• Date formats like "2023/01/07" next to "07-01-23"

• Country codes like IN, india, INDIA, and null

• Duplicated rows from multi-system exports

• Random empty string values ("") that silently break joins

Your job isn’t just to “fix it.”

Your job is to make the next step downstream bulletproof.

Real Example: Cleaning Before Upload

UPGRADE TO ANNUAL

Read more

Most Python tutorials teach you how to build apps, games, or complicated backend systems.

That’s great — if you want to be a software engineer.

But as a Data Engineer?

You need Python to:

• Clean messy data

• Move data between systems

• Automate boring tasks

• Talk to APIs, files, and databases

Today, we're flipping your mindset:
Python isn't for coding. Python is for data movement.

✅ How Data Engineers Actually Use Python

You’re not building websites.
You’re building bridges between messy raw data and clean, usable data.

Real Example: Simple Python Data Cleaning

Imagine you have a messy CSV of customers:

customer_id, signup_date, country
123, 2024-01-10, us
124, NULL, uk
125, 2025-06-01, ca
126, 2024-11-05, null

Your job?

• Remove NULL signup dates

• Standardize country codes to uppercase

• Save the clean output

import pandas as pd

df = pd.read_csv('customers.csv')

# Remove rows where signup_date is NULL
df = df.dropna(subset=['signup_date'])

# Standardize country codes
df['country'] = df['country'].str.upper()

# Save clean version
df.to_csv('customers_clean.csv', index=False)

5 lines of Python → pipeline-ready data.

UPGRADE TO ANNUAL MEMBERSHIP

Read more