Three billion people are doing the exact same thing at the same time. Posting. Scrolling. Messaging. Watching. And somehow the whole thing just works. No crashes. No waiting. No downtime.

The answer is not more servers. It is not a bigger database. It is a fundamental design decision that most engineers never think about until their system is already on fire.

Meta treats reads and writes as two completely separate problems.

In most systems, reads and writes share the same path. The same database handles both. Which means when one gets busy, the other suffers. Your users are trying to load their feed while your pipeline is ingesting millions of new events at the same time. They are fighting over the same resources. And under load, everybody loses.

Most engineers respond to this by adding memory, scaling horizontally, or upgrading their database tier. None of that fixes the actual problem. Because the actual problem is architectural, not operational.

Meta solved it by building two completely different systems from scratch.

LOGDEVICE: BUILT FOR WRITES ONLY

LogDevice is Meta’s distributed log storage system. It was designed with one purpose: ingest data at massive speed without ever slowing down.

Every like, every message, every video view, every backend sensor ping across three billion users - LogDevice takes it all in. It uses a log-structured approach which means it writes data sequentially rather than randomly jumping around the disk. Sequential writes are dramatically faster than random writes. That is not an accident. That is a deliberate design choice made specifically to maximize write throughput.

LogDevice does not care about reads. It was never designed to serve reads efficiently. That is the point. By giving up on reads entirely, it becomes extraordinarily good at the one thing it was built for.

Most engineers designing their first production system try to pick one database that handles everything. LogDevice is the proof that this instinct, while understandable, is wrong at scale.

ROCKSDB: BUILT FOR READS WITH SURGICAL PRECISION

RocksDB started as Google’s LevelDB. Meta took it, rebuilt it, and open sourced it in 2013. Today it powers systems at Facebook, LinkedIn, Yahoo, Twitter, and hundreds of other companies running at scale.

The reason Meta built RocksDB instead of using an existing solution is the same reason they built LogDevice. Nothing on the market gave them the control they needed.

RocksDB is an embeddable key-value store that lets you tune read and write performance independently at the instance level. This is the part most engineers miss.

You can deploy one RocksDB instance configured entirely for fast point lookups optimized for the read patterns of a news feed where you need to retrieve a specific user’s data in milliseconds. You deploy another instance configured for high write throughput - optimized for the ingestion patterns of an analytics pipeline processing billions of events. Same underlying technology. Completely different configurations. Completely different jobs.

They never compete for the same resources because they were never meant to run the same workload.

RocksDB also uses a data structure called an LSM tree - Log Structured Merge tree which batches writes in memory and flushes them to disk in sorted order. This makes writes fast and keeps related data physically close together on disk. When you request data, the disk has to seek less to find it. Less seeking means faster reads. Meta takes this even further by pre-arranging the most frequently accessed bytes so they are physically adjacent on disk. The result is a feed that loads in milliseconds regardless of how many people are using it simultaneously.

WHY THIS MATTERS FOR YOUR SYSTEM RIGHT NOW

You are probably not building for three billion users. But the principle applies at every scale.

If you have a system that slows down under load, the first question to ask is not what hardware do I need. The question is are my reads and writes competing for the same resources.

Leave a comment

I have seen this exact problem at companies with 50 engineers and companies with 5,000. A shared database handling both analytical queries and transactional writes. A single Kafka consumer group processing both real-time and batch workloads. One pipeline serving five different use cases with completely different performance requirements.

The symptom is always the same. Things work fine until load increases. Then everything degrades together because everything is coupled together.

The fix is always the same too. Separate the concern. Define the job. Build for that job specifically.

LogDevice does not try to be RocksDB. RocksDB does not try to be LogDevice. And Facebook never goes down.

Here is the three step framework I apply before designing any new data system:

Step one. Write down every read pattern your system needs to support. How frequently. What latency is acceptable. What the data shape looks like.

Step two. Write down every write pattern separately. How much volume. How fast does it need to land. What consistency guarantees do you need.

Step three. Ask honestly whether one system can serve both patterns without compromising either. If the answer is no, you already know what to do.

Separate the concern first. Then optimize. That is how you build something that survives contact with real traffic.

If you found this valuable, Thursday’s paid newsletter goes even deeper.

Not subscribed yet? Now is a good time.

I am breaking down the exact career moves that separate engineers who understand systems from engineers who just operate them. The difference in compensation between those two groups at companies like Meta is not small.

Thursday 5:30pm. Paid subscribers only.

See you Thursday.

— Avantika

It was my first week at Meta. I had just been handed access to one of the most complex data pipelines I had ever seen. A DAG running on 40 trillion events a day. Likes. Messages. Video views. Sensor pings from devices most people don’t even know exist.

I sat there thinking, I have built pipelines before. Real ones. For Fortune 500 clients. I know Spark. I know SQL. I know how to ship. I’ve got this.

I did not have this.

The first thing that humbled me was not the complexity of the code. It was how wrong my assumptions were.

Before Meta, I assumed deduplication was a solved problem. You write the logic once. It works. Done. At 10 million rows that is true. At 40 trillion events, I was generating duplicates that lived quietly in production for weeks before anyone noticed. And by the time we caught it, half the company was downstream of that bad data.

The code was not wrong. My assumption was wrong.

The second assumption that broke me was around retries. In most systems, if a job fails you retry it. Simple. Safe. Standard practice. At Meta scale, a retry meant potentially processing the same event twice. Which meant potentially double charging a million users. Which meant a P0 incident at 2am with half the engineering org on a call.

I had never once thought about idempotency as a design requirement. At scale it is not a nice to have. It is the difference between a working system and a crisis.

The third assumption was around SLAs. I assumed if a job had a 4 hour SLA and ran in 2 hours in staging, we were fine. Until the cluster was hot. Until three other high priority jobs were competing for the same resources. Until my 2 hour job was at hour 6 and my SLA was breached and I was explaining to my manager why downstream dashboards were empty.

Here is what I learned from all of this. The engineers who survive at that level are not the smartest ones in the room. They are the ones who documented every assumption their system made and then intentionally tried to break each one. Not in production. Not after an incident. Before it ever went live.

That is the skill nobody teaches you. Not in bootcamps. Not in certifications. Not in any course I have ever taken. You learn it by being on the wrong side of an incident and spending 6 hours in a war room tracing back to the assumption you made three months ago that seemed totally reasonable at the time.

Or you learn it here. Before it costs you a night of sleep.

The framework below is the exact checklist I run before any pipeline goes to production. The same one I wish someone handed me in my first week at Meta.

Unlock the Full Framework

THE ASSUMPTION AUDIT: HOW TO BREAK YOUR PIPELINE BEFORE IT BREAKS YOU

After years of building at scale, I now run every pipeline through five assumption categories before it goes live. Not because I am paranoid. Because every incident I have ever been part of traced back to exactly one of these five.

Read more

New here? 👋 Hi. I am Avantika.

I spent 5 years at Meta building data systems for ARVR -Reality Labs, Marketplace and Consumer Connectivity. I then moved on to build platforms for Walmart and Marriott, processing half a billion events a day. Yes, half a billion. Before my morning coffee.

But I have also been the person who Googled “what is a slowly changing dimension” at 11pm the night before an interview. Zero connections in tech. Broke into MAANG the hard way. Both versions of me write this newsletter — the one who learned everything the hard way, and the one who finally knows enough to make it easier for you.

Let me tell you what nobody says out loud……

Most engineers who fail to break into Data Engineering did not fail because they lacked talent. They failed because they spent months learning tools instead of learning how to think. They memorized syntax without understanding the problem the syntax was solving. They chased certificates while the engineers getting hired were building things.

In 2026, the bar has shifted. Companies are not impressed by a list of tools on your resume. They are impressed by engineers who understand data at a conceptual level, write clean purposeful code, and can walk into a system they have never seen before and figure it out.

Because honestly, when AI can write the code in seconds, memorizing syntax is no longer your competitive advantage. What AI cannot do is understand why a pipeline is failing at 3am, make a judgment call on a schema design that has to last five years, or walk into a broken system and reason through it from first principles. That is what companies are paying $150K for in 2026. Not your ability to remember a function name. Your ability to think.

This roadmap is not about tools. It is about building the mental models that make everything else click.

Did you know most Data Engineers who land $150K roles cannot list every feature of every tool they use? But they can explain exactly why data moves the way it does, where it breaks, and how to fix it. That is the real skill.

And the engineers who move into staff and principal and manager roles? They are almost always the most technical people in the room. But not because they memorized the most.

Because they mastered the one skill nobody talks about enough: being the human in the room. The judgment. The context. The ability to sit across from a stakeholder and translate ambiguity into architecture. The instinct that comes from having actually broken things and fixed them.

In the fastest moving AI era we have ever seen, that is not a soft skill. That is your sharpest competitive edge.

Week 1: Learn to Think in SQL

SQL is not a query language. It is a way of reasoning about data relationships.

Before you write a single line of code, understand what you are actually asking the database to do. Every JOIN is a question about how two sets of information relate to each other. Every window function is a question about context, how does this row relate to the rows around it? Every CTE is a way of breaking a complex question into smaller, readable pieces.

Level 1: Foundation How data is stored and why. SELECT, WHERE, GROUP BY, ORDER BY. INNER, LEFT, RIGHT and FULL JOINs. Aggregations: COUNT, SUM, AVG, MIN, MAX. Filtering with HAVING versus WHERE and why the difference matters.

Level 2: Intermediate CTEs and subqueries. When to use one over the other and why it affects readability. Window functions: ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD. Running totals and moving averages. Multi-table joins and handling NULL values intentionally.

Level 3: Advanced Query optimization. How a database engine reads your query, what an execution plan actually tells you, and where queries die. Indexing strategy and when indexes hurt more than they help. Writing SQL for ETL transformations, not just reporting. Schema design decisions and how they affect every query written against them.

What to build: take one messy real-world dataset, something with missing values, duplicates, and inconsistent formats, and write SQL that cleans it, transforms it, and answers three business questions from it. Do not move on until you can explain every line of code out loud.

The concept that changes everything: query optimization. Understanding why a query is slow teaches you more about how databases actually work than any course ever will.

Week 2: Learn to Think in Python

Python in Data Engineering is not about knowing Pandas. It is about understanding how data flows through a system.

An ETL pipeline is just three ideas: get data from somewhere, change its shape, put it somewhere else. Everything else is implementation detail. Once that concept clicks, the code almost writes itself.

Level 1: Foundation Python data types and when to use each one. Reading and writing files: CSV, JSON, TXT. Functions, loops, and error handling. Connecting to a database and running a query from Python.

Level 2: Intermediate Data manipulation with Pandas. Merging, reshaping, filtering and cleaning dataframes. Writing reusable ETL functions. Working with APIs and parsing responses. File format differences: CSV versus JSON versus Parquet and when each one is the right choice.

Level 3: Advanced Building a full ETL pipeline from scratch. Scheduling and automating jobs. Handling failures gracefully with logging and alerting. Writing Python that a team can maintain, not just code that runs once. Performance optimization when your dataset stops fitting in memory.

What to build: a Python script that pulls data from a public API, transforms it into a clean structured format, and loads it into a local database. No frameworks. No shortcuts. Just you, the data, and the logic.

The concept that changes everything: understanding file formats. Why does Parquet exist? What problem does it solve that CSV does not? Engineers who understand the why behind format choices make better decisions at every level of a system.

Week 3: Learn to Think About Scale

Here is where most people get confused. Cloud platforms are not the skill. Scale is the skill.

The question is never “how do I use AWS?” The question is “what happens to my pipeline when the data is 100 times bigger than it is today?” Cloud platforms are just where you go to find out.

Level 1: Foundation What cloud infrastructure actually is and why it exists. Object storage versus databases versus data warehouses. The difference between batch processing and streaming and when each one is the right choice. Cost awareness: why architectural decisions have a dollar amount attached to them.

Level 2: Intermediate Designing pipelines that separate storage from compute. Serverless functions and when to use them. Infrastructure as code: why writing infrastructure in code is the same discipline as writing application code. Monitoring pipelines and knowing when something is wrong before users tell you.

Level 3: Advanced Designing for failure. What happens when a pipeline breaks halfway through and how idempotency protects you. Data partitioning strategies and how they affect query performance downstream. The tradeoffs between real-time and near-real-time architectures and the cost of each.

What to build: take the ETL pipeline from the previous section and think through what would break at 10x the data volume. Then redesign it to handle that load. The redesign does not need to be deployed anywhere. The thinking is the exercise.

The concept that changes everything: serverless architecture. Understanding when to let infrastructure scale automatically versus when to control it manually is a decision every senior engineer makes constantly.

Week 4: Learn to Think About Distributed Systems

Apache Spark is not a tool you learn. It is a mental model you develop.

The core idea: some problems are too big for one machine. Distributed computing is the art of breaking a big problem into smaller problems, solving them in parallel, and assembling the results. Once that concept is clear, the code is just syntax.

Level 1: Foundation Why distributed computing exists and what problems it actually solves. The difference between a single-node and a multi-node system. How data gets split across machines and reassembled. What a shuffle is and why it is the most expensive operation in distributed computing.

Level 2: Intermediate DataFrames in a distributed context versus a single-machine context. Transformations versus actions and why the distinction changes how you write code. Partitioning strategies and how the wrong one quietly destroys performance. Reading and writing large datasets efficiently.

Level 3: Advanced Performance tuning from first principles. Understanding data skew and why it kills parallel execution. Caching strategy and when holding data in memory helps versus hurts. Debugging a distributed job that fails silently on only some partitions. Designing pipelines that stay performant as data volumes grow unpredictably.

What to build: take a large dataset, at least a few million rows, and write a transformation job that would be painfully slow in regular Python. Then think through how distributing that work across multiple machines changes the execution. Understand the tradeoffs between partitioning strategies before worrying about which cluster to run it on.

The concept that changes everything: data locality. Where the data lives relative to where the computation happens determines everything about performance. This is the insight that separates engineers who tune systems from engineers who just run them.

Four Weeks. One Shift. Everything Changes.

Four mental models. SQL thinking, Python thinking, scale thinking, distributed thinking. That is the entire curriculum of a $15K bootcamp. You just got it in 30 days by understanding the concepts not memorizing the commands.

Now go build something that breaks. That is where the real learning starts.

Drop your biggest conceptual challenge in the comments. Not “I do not know which tool to learn.” The real question underneath that. That is where the work actually starts.

Leave a comment

That is all for today.

If this helped, forward it to a friend or colleague who is figuring out their next move in tech. This newsletter costs less than your daily coffee and it might be the thing that gets them unstuck.

Follow me on LinkedIn and Instagram @avantika.tech for daily AI and Data Engineering content.

See you tomorrow.

— Avantika Penumarty

Share Avantikka_Penumarty

I need to tell you about the biggest mistake I almost made in my career.

It was 2021. I was scrolling through LinkedIn (procrastinating, as usual) when I saw it:

“Data Engineer – Series B Startup – $180K base”

My heart raced.
This was it. The role I’d been working toward.

Then I opened the job description.

The Requirements That Almost Stopped Me

The list felt intimidating:

• 5+ years of data engineering experience
  (I had 3 years in automation + 2 years as a data analyst)

• Spark expert
  (I’d used it in a few projects, but expert?)

• Data engineering background required
  (I came from automation engineering)

• CS degree preferred
  (I had one, plus a Master’s in Engineering Management and still doubted myself)

I stared at the screen for 20 minutes.

Then I closed the tab.

“I’m not qualified.”

The Voice of Self-Sabotage

For three weeks, that job posting haunted me.

Every morning, I’d see it in my saved jobs. Every evening, I’d tell myself:

• “I need more Spark experience first”

• “They want someone from a pure DE background”

• “I should wait until I’m more ready”

• “What if they figure out I’m not qualified?”

Here’s what I didn’t know at the time:

I was already qualified. I just couldn’t see it.

What Changed Everything

On a random Tuesday, I was venting to a former colleague about my job search frustration.

Me: “I keep finding roles that want 5+ years of pure data engineering experience. I have automation and analyst background, but that doesn’t count, right?”

Him: “Wait, you built those automation pipelines that processed millions of events per day, right?”

Me: “Yeah, but that’s not data engineering—”

Him: “That IS data engineering. You built data pipelines. You designed data models. You worked with stakeholders on data requirements. What do you think data engineering is?”

That conversation changed my perspective completely.

The Truth About “Qualifications”

I went back and looked at my actual experience with fresh eyes:

From Automation Engineering, I Had:

• Built pipelines processing 2M+ events daily

• Designed monitoring and alerting systems

• Debugged production issues under pressure

• Understood reliability and system design

From Data Analyst Work, I Had:

• Mastered SQL (wrote hundreds of complex queries)

• Understood stakeholder needs deeply

• Built data models for business analytics

• Created dashboards and reports

From My Education, I Had:

• Master’s in Computer Science

• Master’s in Engineering Management

• Strong fundamentals in algorithms and systems

I wasn’t missing data engineering experience. I was just calling it different names.

How I Reframed My Background

Instead of apologizing for my “non-traditional” path, I rewrote my story:

What I Used to Say:

“I’m trying to transition into data engineering. I don’t have direct DE experience, but I’m a fast learner.”

What I Started Saying:

“I’ve built data pipelines processing millions of events in my automation work. I bring SQL expertise from my analyst background and system design thinking from my engineering management studies. Here are three projects that demonstrate my DE skills...”

The skills were always there. I just needed to own them.

I Applied. Here’s What Happened.

I finally clicked “Apply” on that job posting (it had been reposted after 3 weeks - they were still looking).

The interview came two days later.

Interviewer: “Tell me about your data engineering experience.”

Old me would have said: “Well, I don’t have traditional DE experience, but...”

What I actually said: “I’ve built automation pipelines that process 2 million events daily, designed data models for business analytics, and worked extensively with SQL for data transformation. Let me walk you through my most complex project...”

I got the offer two weeks later.

$185K base + equity + remote flexibility.

More importantly? I loved the work. My “non-traditional” background turned out to be an advantage. I brought perspectives the team needed.

Why This Matters for You

If you’re reading this and thinking “but my background is different too,” that’s exactly my point.

Your background is probably MORE relevant than you think.

Coming from Software Engineering? You understand production systems, testing, and code quality - things many DEs struggle with.

Coming from Data Analysis? You understand stakeholder needs and data modeling - the “why” behind the pipelines.

Coming from DevOps/SRE? You understand reliability, monitoring, and infrastructure - critical for production data systems.

Coming from QA/Testing? You understand data quality, edge cases, and validation - increasingly important in DE.

“Non-traditional” is your advantage, not your weakness.

The Real Requirements

Here’s what companies ACTUALLY need (vs. what they write in job descriptions):

Job Description Says:

• 5+ years data engineering experience

• Expert in Spark, Airflow, dbt

• CS degree required

• Big data experience

What They Actually Need:

• Can you solve data problems?

• Can you learn new tools quickly?

• Can you communicate with stakeholders?

• Can you ship working pipelines?

If you can do the work, the specific background doesn’t matter as much as you think.

How to Position Your “Non-Traditional” Background

Map Your Transferable Skills

Your Experience → DE Skill → How to Frame It

Built automation scripts → Pipeline development → “Built automated data pipelines processing X events/day”

SQL for reporting → Data transformation → “Designed and optimized SQL queries for data transformation”

API integrations → Data ingestion → “Integrated data from 10+ external APIs”

Monitoring systems → Data observability → “Implemented monitoring and alerting for data quality”

Reframe, don’t apologize.

Build 1-2 Targeted Projects

Pick projects that fill your biggest gaps:

If you lack pipeline experience: Build an end-to-end ETL pipeline with Airflow + dbt

If you lack big data experience: Process a large public dataset with Spark

If you lack cloud experience: Deploy a pipeline on AWS/GCP free tier

Time investment: 2-4 weekends per project

ROI: “Here’s a project I built” beats “I’m trying to learn” every time

Apply Even If “Underqualified”

My rule: If you have 60% of the requirements, apply.

Why?

• Job descriptions are wish lists, not requirements

• They’re often written by recruiters who don’t know the role

• Teams are frequently flexible on specific requirements

• Worst case? They say no. You lose nothing.

Let THEM decide if you’re qualified. Stop deciding for them.

The Action Plan

If you’re in a similar position, here’s what to do this week:

Monday: Audit Your Skills (30 minutes) List everything you’ve built or done that involves data, even tangentially.

Tuesday: Rewrite Your Resume (1 hour) Reframe your experience using data engineering language.

Wednesday: Build Your First Project (Start it) Pick one weekend project that addresses your biggest gap.

Thursday-Friday: Apply to 5 Jobs (1 hour) Apply to roles where you have 60%+ of requirements.

Weekend: Ship Your Project Get something working and on GitHub.

Next Monday: Apply to 5 More Jobs Iterate weekly.

Don’t wait until you’re “ready.” You’re already more ready than you think.

The Hard Truth

Three months after I got that job, they hired someone else to the team.

Want to know their background?

Marketing analytics.

She had even less “traditional” DE experience than me. But she was brilliant, motivated, and brought a perspective the team needed.

That’s when I realized: The “perfect” candidate doesn’t exist. Teams need diverse backgrounds.

What’s Holding You Back?

I shared my story on LinkedIn yesterday, and within an hour, I got dozens of messages:

“This is exactly my situation. I have X experience but job wants Y.”

So let me ask you:

What’s actually stopping you from applying?

• Is it missing skills? (You probably have transferable ones)

• Is it the job description? (It’s a wish list, not a requirement)

• Is it imposter syndrome? (Everyone has it, even with 2 Master’s degrees)

The only thing that will definitely stop you from getting a DE job is not applying.

Your Next Step

Here’s what I want you to do after reading this:

1. Find 3 job postings you’ve been avoiding because you feel “not qualified”

2. Make a list of why you think you’re not qualified

3. Reframe each objection as a transferable skill or learnable gap

4. Apply to at least 1 of them this week

If you’re serious about making this transition, I’ve created some resources to help:

If you’re serious about making this transition, I’ve created some resources to help.

What this resource includes

• A Data Engineering Career Readiness Assessment to help you honestly evaluate where you stand today.

• A curated list of 40 hands-on data engineering projects, organized by difficulty, so you know exactly what to build next.

• Clear guidance on what each project demonstrates, how long it takes, and how to talk about it in interviews.

• Practical frameworks to translate non-traditional experience into data engineering language recruiters understand.

• An interview-focused approach so you’re not just building projects, but learning how to explain decisions, tradeoffs, and impact.

Access Link Here

One Last Thing

The person who got that job I almost didn’t apply for?

Me.

But it almost wasn’t.

I almost let imposter syndrome make that decision for me.

Don’t let your doubt decide your future.

That voice saying “not yet” is a liar.

You’re ready. Or close enough.

Apply anyway.

See you next week,

Avantikka Penumarty

P.S. What job are you going to apply for this week? Hit reply and tell me. I want to know.

P.P.S. If this resonated with you, forward it to someone else who needs to hear it. We all need permission to believe in ourselves sometimes.

Follow my journey:
LinkedIn: @Avantikka_Penumarty
Twitter: @avantikka_penumarty
Instagram: @avantikka.penumarty

Not subscribed yet? Subscribe here for weekly data engineering career advice.
Want to chat? Reply to this email. I read every response.

Hi everyone,

Before we begin, I want to share a quick, honest note and sincere apologies to all my readers.

I’ve been quieter than usual over the last few months. I was dealing with some personal and health-related challenges and needed to step back briefly. Thank you for your patience, messages, and continued support it truly means more than you know.

I’m back now, and it felt right to restart with a topic that sits at the very core of data engineering one that has quietly shaped almost every system I’ve worked on.

Table of Contents

• What is SQL in Data Engineering?

• Why SQL is Crucial for Data Engineering

• SQL for ETL vs. ELT Pipelines

• Essential SQL for Data Engineers

• SQL in Modern Data Engineering Tools

• Best Practices for Writing SQL in Data Engineering

• Future of SQL in Data Engineering

• Conclusion

• FAQs

Prefer listening over reading?

Are you on your way to work or heading back home?
Starting a run, folding laundry, or just taking a quiet break?

If reading feels like too much right now, I’ve got you.

I recorded an audio version of this newsletter so you can listen while you move through your day. Same ideas, same depth just in a format that fits real life.

Plug in your headphones, press play, and let SQL make sense in the background while you take care of everything else.

Introduction

Structured Query Language (SQL) remains the foundation of data engineering, enabling data professionals to design, build, and maintain scalable data pipelines. Despite the rise of modern technologies like Apache Spark and NoSQL databases, SQL’s declarative syntax and universal adoption make it indispensable in real-world data engineering workflows.

In this piece, I’ll walk you through how SQL shows up in real data engineering work, what actually matters in practice, and why it continues to be one of the most valuable skills you can invest in as a data engineer.

What is SQL in Data Engineering?

Meta story: Early in my career, I believed mastering tools would make me a great data engineer. Spark, Airflow, Kafka I chased them all. What actually made my work reliable wasn’t a tool. It was the moment I truly understood SQL as a way of thinking: describing what the data should look like, not how to move every row. That shift changed how I designed pipelines forever.

At its core, SQL (Structured Query Language) is the language we use to talk to data stored in relational systems to ask questions, shape answers, and turn raw records into something meaningful.

In data engineering, SQL is used to:

• Ingest raw data

• Clean and validate datasets

• Transform data into analytics-ready models

• Load data into warehouses and lakes

SQL acts as the linchpin of both ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) pipelines, making it the backbone of modern data platforms.

Why SQL is Crucial for Data Engineering

Project moment: On one production pipeline, we had Python transformations, custom logic, and retries everywhere and still broke SLAs weekly. The fix wasn’t a rewrite. It was replacing fragmented logic with clear, well-structured SQL. Fewer lines. Fewer bugs. More trust.

1. Data Extraction

SQL makes it surprisingly easy to pull data from structured systems like PostgreSQL, MySQL, and Oracle, and even from modern platforms that support SQL-style querying such as BigQuery and Redshift.

2. Data Transformation

Data engineers rely on SQL for cleansing, aggregation, and normalization. Features like Common Table Expressions (CTEs), window functions, and subqueries allow complex transformations to remain readable and maintainable.

3. Data Loading

In practice, SQL-powered pipelines move data into warehouses and lakes in a way that keeps analytics teams productive and downstream systems stable.

4. Data Integration

By joining datasets across multiple systems, SQL helps engineers create unified data models that power reliable reporting and decision-making.

5. Performance Optimization

Modern SQL engines such as Apache Hive, Presto, and Spark SQL provide query optimization capabilities that reduce execution time, improve resource utilization, and scale analytics workloads.

SQL for ETL vs. ELT Pipelines

Career insight: I’ve seen engineers struggle not because they chose ETL or ELT, but because they didn’t understand where SQL belongs in the system. Once you see SQL as a first-class layer not an afterthought architectural decisions become simpler.

SQL sits at the center of both approaches. In recent years, ELT has become more common simply because cloud warehouses make large-scale transformations easier and cheaper to run in parallel.

Essential SQL for Data Engineers

Interview reality: Almost every senior data engineering interview I’ve seen comes down to this section. Not syntax trivia but whether you can express business logic clearly, safely, and efficiently in SQL.

1. Window Functions

These are used when you need running totals, rankings, or comparisons across groups without losing row-level detail.

SELECT
customer_id,
order_date,
SUM(order_amount) OVER (
PARTITION BY customer_id
ORDER BY order_date
) AS cumulative_sales
FROM orders;

2. Common Table Expressions (CTEs)

CTEs make complex logic easier to read, reason about, and safely modify over time.

WITH recent_orders AS (
SELECT order_id, customer_id, order_date
FROM orders
WHERE order_date > '2026-01-01'
)
SELECT * FROM recent_orders;

3. Joins

Most real-world datasets only make sense once multiple tables are joined together this is where SQL earns its keep.

SELECT customers.name, orders.order_id
FROM customers
JOIN orders
ON customers.customer_id = orders.customer_id;

4. Indexes and Query Optimization

Indexes improve read performance, while query planners and EXPLAIN statements help identify bottlenecks in large-scale systems.

5. Data Partitioning

Partitioning large tables improves performance in distributed systems such as Hive and BigQuery by limiting the amount of data scanned.

SQL in Modern Data Engineering Tools

Meta observation: Tools change faster than job titles. What stays constant is SQL acting as the common language across platforms the one skill that transfers cleanly when stacks evolve.

ToolPurposeSQL RoleApache HiveData warehousing on HadoopHiveQL for querying HDFSApache Spark SQLDistributed data processingSQL on DataFramesGoogle BigQueryServerless data warehouseStandard SQLAWS RedshiftCloud data warehousePostgreSQL-like SQLSnowflakeCloud data platformANSI SQLdbtData transformationSQL-based modeling

Best Practices for Writing SQL in Data Engineering

Production lesson: If someone else can’t understand your SQL six months later including you it will eventually cost time, trust, or money. Readability is not optional in production systems.

1. Use CTEs for complex logic
   Break queries into logical steps to improve readability and maintainability.

2. Avoid SELECT *
   Explicitly select required columns to reduce data scanning and improve performance.

3. Leverage indexes and partitioning
   Use clustering, partition keys, and indexes to optimize large datasets.

4. Monitor query performance
   Analyze execution plans using EXPLAIN statements to identify inefficiencies.

5. Follow data governance standards
   Ensure compliance with organizational policies around data security, privacy, and access control.

Future of SQL in Data Engineering

Forward-looking thought: SQL isn’t competing with new paradigms it’s absorbing them. Streaming, federated queries, and data mesh architectures are all bending toward SQL as the shared interface.

Despite the growth of NoSQL and distributed systems, SQL’s declarative nature ensures its continued relevance.

Key trends shaping the future:

• SQL on streaming data using platforms like Apache Flink and ksqlDB

• Federated queries enabling cross-platform data access

• SQL in data mesh architectures as a shared querying layer across decentralized domains

SQL is not being replaced it is evolving alongside modern data architectures.

Conclusion

Closing reflection: Every time data systems fail, the root cause is rarely “bad data.” It’s unclear logic. SQL, when written well, makes intent explicit and that’s why it continues to matter.

SQL is far more than just a querying language. It is the backbone of data engineering, powering data ingestion, transformation, integration, and analytics at scale.

As tools and platforms evolve, SQL’s clarity, expressiveness, and adaptability ensure it remains an essential skill for data engineers, data scientists, and analytics professionals.

FAQs

Is SQL used in data engineering?
Yes. SQL is fundamental to data engineering and is used extensively for data extraction, transformation, loading, validation, and modeling.

How do I become a SQL data engineer?
Build a strong foundation in SQL and database systems, practice query optimization, learn data modeling, and gain hands-on experience with modern cloud data warehouses. Complement SQL with Python for automation and orchestration.

Is SQL still relevant in 2026?
Absolutely. SQL remains one of the most in-demand skills due to its deep integration with cloud platforms, analytics tools, and modern data stacks.

Is Python and SQL enough for data engineering?
They form a strong foundation, but data engineers also benefit from learning data orchestration tools, distributed systems, and cloud platforms.

Should data engineers know SQL?
Yes. SQL is essential for building reliable data pipelines, modeling data, and ensuring data quality.

What are some of the best SQL courses for data engineers?

• PostgreSQL for Everybody (Coursera)

• SQL Fundamentals (Dataquest)

• The Ultimate MySQL Bootcamp (Udemy)

• Complete SQL Mastery (CodeWithMosh)

• Advanced SQL for Data Engineering (Udemy)

🔗 Follow Avantikka Penumarty on LinkedIn

🔒 Want to turn this into real skill?
Access interactive SQL flashcards, quizzes, and AI-powered explanations in the paid section below.

Take This From Insight to Instinct

Reading builds understanding.
Practice builds confidence.

If SQL plays a role in your job, interviews, or long-term growth, passive reading isn’t enough.

I’ve created an interactive learning extension to this newsletter designed to help you think in SQL, not just recognize syntax.

Inside the paid section, you’ll get:

• 🧠 Flashcards to reinforce core SQL concepts and mental models

• 🧪 Scenario-based quizzes that mirror real data engineering decisions

• 🤖 AI-powered explanations that walk you through the why, not just the answer

• 🏗️ Applied reasoning you can reuse in production systems and interviews

This is how you move from:

“I’ve read this”
to
“I can apply this under pressure.”

Read more

Hi there,

You’re here because you want more than just theory. You want a roadmap that actually works.

This isn’t fluff. I built this guide based on what I wish someone gave me when I was starting out — real skills, real proof, and real steps that lead to interviews and offers. No gatekeeping. No unnecessary jargon.

I’m not here to sell you a bootcamp. I’m here to show you that you can build production-level data systems — even without a CS degree, fancy title, or 5+ years of experience.

If you're serious about making the jump into DE, everything you need to start is in this email.

Let’s begin.

If you're a new grad trying to break into Data Engineering, you're not alone.

Data Engineering is one of the fastest-growing roles in tech today — but also one of the most misunderstood. Most beginners assume they just need to learn SQL and Python. But here's the hard truth:

Knowing SELECT * FROM table_name is not enough.

Thousands of grads take bootcamps, binge YouTube tutorials, and still never land interviews. Why? Because they have no real-world proof that they can build or maintain data systems.

And in this field, that proof matters more than any degree or certificate.

So let’s fix that.

JOIN A COMMUNITY OF 10K+ Data Engineers

Why Data Engineering?

It’s one of the most practical, future-proof, and underrated roles in tech today.

Perfect for those who:

• Enjoy backend more than frontend

• Love building real systems over tweaking models

• Want high pay, low ego, and fewer LeetCode puzzles

But DE is also one of the most misunderstood roles.

New grads think they just need SQL + Python.
Others spend months collecting certifications.
But neither gets interviews.

Why? Because they’re missing the one thing that matters most:

Proof that you can build and run data systems.

That proof > any degree, certificate, or keyword-stuffed resume.

What Do Data Engineers Actually Do at Work?

This is what you'll be expected to do in a real DE role:

• Build pipelines that ingest, clean, and transform raw data

• Maintain data quality, integrity, and freshness

• Work with analysts, ML teams, and backend engineers

• Manage cloud resources (like S3, Snowflake, BigQuery)

• Monitor systems, set up alerts, and debug failures

• Optimize queries, improve performance, and own uptime

Think of yourself as the plumber of the data world. You build and maintain the pipes that make everything else possible.

Here’s What You Actually Need to Land Your First DE Role:

✅ Go beyond basic SQL.
Learn how to use CTEs, window functions, GROUPING SETS, and indexes. Hiring managers want to see that you can write performant queries, not just run simple reports.

1. Build ETL pipelines.
   Get hands-on with Airflow and dbt. Learn to extract messy data, clean it up, and load it into warehouses. Use cloud tools like AWS S3 or GCP BigQuery to simulate real infrastructure.

2. Understand data modeling.
   Study how companies design scalable data systems. Know what a Star Schema is. Learn dimensional modeling. Explore how tools like Snowflake and Redshift structure data for analytics at scale.

3. Publish real projects.
   Build 1–2 end-to-end data pipelines using public datasets. Document them clearly on GitHub. A great portfolio can often do more than a great resume — especially when you're just starting out.

And what you don’t need:

❌ A CS degree
❌ 5+ years of experience
❌ Perfect code

You just need to prove that you understand how data flows through a system — and that you can make it work in production, not just in a classroom.

New Grad to Data Engineer: The Real Roadmap

Month 1 – Build Your Foundations (SQL + Python)

This month is NOT just about “learning basics.” It’s about mastering fundamentals that you’ll actually use in interviews and pipelines.

SQL:

• Learn SELECT, WHERE, GROUP BY, ORDER BY

• Go deep into Window Functions: ROW_NUMBER, RANK, LAG, LEAD

• Practice CTEs, Subqueries, and Indexing Basics

• Learn performance tuning (EXPLAIN plans, indexes)

👉 Resources: LeetCode (SQL), Mode Analytics SQL Tutorial, OneCompiler

Python:

• Learn data structures: Lists, Dicts, Tuples

• Write basic ETL scripts using pandas

• Practice reading/writing files (CSV, JSON, API calls)

• Understand exception handling and basic functions

👉 Resources: Dataquest, Jupyter Notebook, Kaggle Datasets

My DE Prep Scheduler (SQL + Python)

Here’s how I structured my own study plan when I first applied for Data Engineering roles. No fluff. Just habits that stuck.

Week 1 – SQL Core + Python Setup

Goal: Build confidence with basic SQL and get Python environment ready.

Mon–Tue:

• SQL: SELECT, WHERE, GROUP BY, ORDER BY

• Resource: Mode SQL Tutorial + OneCompiler (hands-on)

• Python: Install Python, set up Jupyter Notebook or VS Code

Wed–Thu:

• Python: Data types, loops, functions, lists & dicts

• SQL: Write queries using GROUP BY and filters on public datasets (Kaggle or LeetCode)

Friday:

• Mini Project: Analyze a CSV file using pandas and filter top rows by condition

• Output: Save your script and query result snapshot

Week 2 – SQL Intermediate + ETL in Python

Goal: Get hands-on with intermediate SQL and write your first ETL script.

Mon–Tue:

• SQL: CTEs, Subqueries (solve 3 problems/day on LeetCode)

• Python: Read/write CSV + JSON files, understand APIs with requests

Wed–Thu:

• Build: ETL Script (Extract COVID API → Clean → Write to local file or SQLite)

• SQL: Join practice + filtering with aliases and subqueries

Friday:

• GitHub Upload: Push your ETL script with README

• Bonus: Create a 1-pager explaining what your script does

Week 3 – Window Functions + Real Data Handling

Goal: Use advanced SQL and clean real-world data.

Mon–Tue:

• SQL: ROW_NUMBER, RANK, LAG, LEAD (focus on order + partition logic)

• Python: Use pandas to clean messy real-world data (missing values, data types)

Wed–Thu:

• Practice: NYC Taxi or Netflix dataset — clean with pandas, summarize with SQL

• SQL: Practice use cases like top 3 per category, running totals

Friday:

• Output: Save your cleaned dataset, sample queries, and explain what insights you found

Week 4 – Optimization + Portfolio Building

Goal: Wrap Month 1 with real proof of work.

Mon–Tue:

• SQL: Learn EXPLAIN plan, indexing, query performance basics

• Python: Add logging, exception handling to your ETL script

Wed–Thu:

• Portfolio Time: Write a full README for your project

• Push code + screenshots to GitHub

Friday:

• Reflection:

  • What did you learn?

  • What would you do differently?

  • What’s the next dataset you want to try?

Tools I Used (And Still Recommend):

• SQL Practice: LeetCode SQL, Mode, OneCompiler

• Python IDE: Jupyter Notebook, VS Code

• Data: Kaggle, NYC OpenData, COVID API

• Version Control: GitHub

• ETL Stack: pandas + SQLite (perfect beginner combo)

Month 2 – Learn ETL + Orchestration (Real Project Work Begins)

This month is about translating your learning into real pipelines. You’ll now build actual systems that move and transform data.

Choose 1–2 projects below based on your tool comfort (local/cloud/dbt/Airflow).
The goal is not to learn everything — it’s to complete one pipeline end-to-end and publish it on GitHub.

Project 1: Local CSV to SQLite ETL (Beginner-Friendly, No Cloud)

Stack: Python, pandas, SQLite, cron job (or Airflow optional)

Problem Statement:
Build a pipeline that pulls NYC Taxi data (CSV), cleans it using pandas, and stores it in a local SQLite DB.

Steps:

1. Download public CSV dataset

2. Clean & transform with pandas (fix datatypes, nulls, etc.)

3. Load into a local SQLite DB

4. Schedule pipeline using cron or basic Airflow DAG

Outcome:

• Simple file-based ETL

• Lightweight & fully local

• Teaches end-to-end scripting

Project 2: API to Cloud Warehouse with Airflow

Stack: Python, Airflow, AWS S3, BigQuery or Snowflake

Problem Statement:
Pull daily COVID-19 data from a public API, store raw files in S3, process data, and load it into a warehouse.

Steps:

1. Extract from API (requests + JSON)

2. Save raw JSON/CSV to S3

3. Clean data in pandas or dbt

4. Load to Snowflake or BigQuery

5. Orchestrate everything with Airflow DAG

Outcome:

• Full cloud-native ETL pipeline

• Shows data ingestion + orchestration

• Great proof of cloud skillset

Project 3: Local CSV to dbt + DuckDB (No Cloud, SQL Focus)

Stack: dbt, DuckDB, CSV, Jinja, SQL

Problem Statement:
Use dbt to build a transformation pipeline that models ecommerce order data from a local CSV.

Steps:

1. Load CSV into DuckDB (acts like a local warehouse)

2. Create staging and mart models with dbt

3. Apply SQL transformations using dbt

4. Generate documentation & DAG visualizations

Outcome:

• Teaches modeling & transformations

• Easy to run locally

• Helps learn dbt structure and SQL best practices

Project 4: Reddit Data Pipeline with Python + MongoDB

Stack: Python, PRAW API, MongoDB, Airflow (optional)

Problem Statement:
Extract posts from a subreddit using Reddit’s API and store them into a NoSQL database.

Steps:

1. Authenticate with Reddit using PRAW

2. Extract posts & comments

3. Clean and process text

4. Store into MongoDB

5. Optional: schedule using Airflow

Outcome:

• Exposure to unstructured data

• Real-world use of APIs + NoSQL

• Fun and engaging project for resumes

Project 5: Batch to Analytics Dashboard (SQL + Streamlit)

Stack: pandas, SQLite, Streamlit, Matplotlib, SQL

Problem Statement:
Ingest historical sales data and build an analytics dashboard to track trends.

Steps:

1. Ingest CSV files weekly into SQLite

2. Use pandas/SQL to analyze key metrics (revenue, retention, cohort)

3. Build a live dashboard with Streamlit

4. Optional: Automate ingestion with cron job

Outcome:

• Combines data engineering with dashboarding

• Useful for end-user reporting

• Makes you stand out for DE/BI hybrid roles

Tip for All Projects:

Each project should include:

• README.md (with overview, stack, steps, and diagram)

• Pipeline code/scripts/notebooks

• Screenshot of output or working dashboard

• Optional: Loom video explaining your pipeline

How to Talk About Projects in Interviews

Most candidates freeze here. Don’t.

Here’s how to prep for interview questions about your projects:

• Why did you choose this dataset?

• What challenges did you face while cleaning or transforming it?

• What trade-offs did you make in designing the pipeline?

• How would you scale this pipeline for daily use?

• What would you improve if given more time?

Have clear answers. Show thought process. That’s how you stand out.

Common Traps to Avoid

Avoid these if you want to land a job faster:

• Spending 3 months making your README “aesthetic”

• Overbuilding with 6 AWS tools before writing your first ETL script

• Learning SQL, Python, and Airflow in isolation — instead of connecting them in one project

• Binge-watching tutorials without building a single pipeline

• Applying to 100 jobs with no GitHub proof of work

One Last Thought:

If you're stuck, it’s not because you're not smart enough. It's because no one ever gave you the real playbook.

This newsletter is that playbook.
If it helped you, forward it to a friend who's trying to get into tech. You never know whose life it might change.

–
Avantikka Penumarty
Ex-META | Snr. Data Engineer | Founder, Zero to Data Engineer
zero2dataengineer.substack.com

“Airflow is legacy!”
“Dagster is the future!”
“Prefect is so clean!”

But you’re not here for debate. You’re here to ship pipelines, learn fast, and get hired.

So let’s get real:
Which tool should you pick — for your project, for your portfolio, or for your resume?

This guide won’t compare every feature.
It’ll show you how to think like a system builder, not a tool fangirl.

1. When to Use Airflow

Use Airflow when:

• You’re applying to big tech or mature data teams

• You want to show you understand orchestration, retries, and scheduling

• You need strong scheduling and visibility (UI, logs, retries)

Why it works for your resume:

“Familiar with industry-standard orchestration using Airflow, including DAG design, sensors, retry logic, and alerting.”

Why it’s great for teaching yourself the fundamentals:
Airflow makes you learn how pipelines actually run: tasks, triggers, failures, dependencies. It exposes orchestration in raw form.

Where it’s weaker:

• Not great for data scientists or notebooks

• Heavier to deploy without Docker

• Harder to reason about with dynamic workflows

2. When to Use Prefect

Use Prefect when:

• You’re building a lightweight project

• You don’t want to mess with Airflow configs

• You like writing clean Python with decorators

Why it works for solo projects:

• It’s Pythonic and elegant

• Easy to get up and running

• Great docs, fast dev feedback loop

Why it works for data science/analytics engineers:

• You can orchestrate model training, dbt, API tasks quickly

• You don’t need to explain DAGs to non-engineers

Where it shines on a resume:

“Used Prefect to orchestrate model training and dbt transformations in a low-latency ML workflow.”

Where it’s weaker:

• Less recognized by recruiters

• Smaller community

• Not ideal for heavy-duty, multi-team orchestration

3. When to Use Dagster

Use Dagster when:

• You care about type safety, IO contracts, and observability

• You want to model your data pipeline like software

• You’re building a data platform, not just a DAG

What makes Dagster interesting:

• Built-in concepts like assets, software-defined pipelines

• First-class support for data lineage and testing

• Powerful for collaborative teams that want engineer-level visibility

On your resume:

“Designed asset-aware DAGs in Dagster to enforce data lineage, retry logic, and schema-aware contracts.”

Where Dagster may not be right:

• Overkill for simple projects

• Less intuitive for beginners

• Still evolving rapidly — may have rough edges

What Your Tool Choice Says About You

If you picked...You’re signaling...Airflow“I understand production-scale systems.”Prefect“I move fast and iterate cleanly in Python.”Dagster“I think in contracts, lineage, and scale.”

There is no “best” tool — only the one that aligns with:

• The story you want to tell

• The type of work you want to do

• The types of teams you want to join

The Hidden Skill Behind Tool Choice: Narrative Alignment

The tool you use reflects how you frame problems.

Interviewers aren’t just looking at which orchestrator you picked — they’re evaluating whether your mental model matches the role.

• Use Airflow if you want to tell a story about scale, resilience, and legacy integration

• Use Prefect if you want to show you’re experiment-driven, lean, and fast-moving

• Use Dagster if you want to prove you're a system thinker who designs for complexity and traceability

This is less about syntax, more about signal. Make your choice reflect your intended audience.

What Top Companies Use (And Why It Matters)

If you're applying somewhere — match your project tool with what that company runs in production (or wants to).

How to Frame This on Your Resume

Don’t write:

“Used Airflow for DAG orchestration.”

Write:

“Designed idempotent, retry-aware data pipelines using Airflow for scalable ETL orchestration across S3, Spark, and Snowflake — with SLA tracking and alerting.”

Or:

“Built fast, modular Prefect flows to orchestrate data science pipelines with resume-to-dashboard visibility in under 15 minutes per run.”

Or:

“Architected lineage-aware pipelines using Dagster assets and config mapping to reduce failure recovery time by 80% across multiple data teams.”

The tool isn’t the flex. The system design behind it is.

Thinking Like a Hiring Manager

If I’m hiring:

• Airflow tells me you’ve worked with mature pipelines and understand operational burden.

• Prefect tells me you can build fast, are probably solo or hybrid (data + product), and can ship.

• Dagster tells me you write pipelines like software — clean, testable, typed — and are comfortable with architecture decisions.

If you’re junior? Pick one and go deep.
If you’re senior? Know when to choose which — and explain it.

Final Advice

Don’t just pick the tool that’s hyped.
Pick the one that matches how you want to think and explain your work.

What you build is one thing.
How you talk about it is what gets you hired.

“Yeah, I’ve used Airflow.”

But interviewers can tell within 60 seconds whether you actually understand it — or just ran someone else’s DAG.

This post is not a checklist of features.
It’s a breakdown of the top Airflow mistakes that break pipelines, burn teams, and ruin interviews — and how to avoid them.

Let’s make you sound like someone who’s deployed DAGs in production, not just built toy examples.

Mistake #1: Treating Airflow Like a Script Runner

If your tasks are running huge pandas transformations, calling 15 APIs, and returning massive objects between tasks, you're doing it wrong.

Airflow is not Spark. Not dbt. Not a transformation engine. It’s the orchestrator — the conductor of the data workflow.

Do this instead:

• Keep tasks modular

• Push heavy lifting into external jobs (Spark, SQL, cloud ETL)

• Use Airflow to manage dependencies, not business logic

How to say this in interviews:

“I used Airflow to orchestrate pipeline steps, but offloaded heavy data processing to Snowflake and Spark jobs to keep the DAGs lean and observable.”

Mistake #2: Ignoring Retry Logic and Failure Handling

Most junior engineers write DAGs that work… when nothing goes wrong.

But in production:

• APIs time out

• S3 files get delayed

• Database connections drop

Best practices:

• Always configure retries, retry_delay, and on_failure_callback

• Log why each task failed (don’t just rely on the default log dump)

• Use idempotent task design — so retries don’t break downstream logic

Pro-level tip: Add exponential backoff and SLA alerts.

Upgrade to Annual

Read more

You get the metaphors.
You’ve seen the screenshots.

But let’s be real:
Until you write a DAG yourself, Airflow will always feel like someone else’s tool.

Today’s Elite drop is different.
You’re not reading about DAGs — you’re building one. From scratch. With your hands. With real code.

This walkthrough is intentionally barebones.
Because clarity is power.
You’ll leave this with a runnable pipeline and confidence to answer:

“Yes, I’ve built DAGs before.”

What You’re Building

You’ll create a DAG that:

• Ingests a CSV file

• Cleans it using Python

• Loads the clean data to a Snowflake table

• Runs every 6 hours

• Sends a Slack alert if something fails

Simple. But production-grade.

UPGRADE TO ELITE ACCELERATOR

Step 1: Set Up Your Environment

If you don’t already have Airflow:

pip install apache-airflow

Then initialize Airflow:

airflow db init
airflow users create \
  --username admin \
  --firstname Firstname \
  --lastname Lastname \
  --role Admin \
  --email your@email.com
airflow webserver --port 8080
airflow scheduler

You now have Airflow running locally.

Read more

Data arrives late.
APIs fail.
Files drop into S3 at random.
And your pipeline has to wait, trigger, or backfill — not just run on a timer.

Today, we’re digging into how scheduling really works — and how you should answer when interviewers ask:

“How do you schedule and trigger your Airflow DAGs?”

First: Understand the Two Types of Runs

1. Scheduled Runs
   You tell Airflow to run a DAG every X time:

   • Every hour, day, week, etc.

   • Use cases: batch ETL, daily reporting, metrics updates

2. Triggered Runs
   Airflow runs the DAG when something happens:

• A file lands in S3

• An upstream DAG completes

• An API returns a signal

Elite Bonus Drop Coming:
This Thursday, I’ll walk through Sensors, ExternalTask dependency patterns, and DAG chaining in a real multi-DAG system.

This is where your interview answers start sounding like a Staff Data Engineer.

UPGRADE TO ANNUAL MEMBERSHIP

Read more

Today’s goal?
You’ll build your first working DAG — start to finish — that does something useful and gets you closer to job-ready.

If you can complete this, you’ll be able to walk into any interview and say:

“Yes, I’ve built production-style pipelines in Airflow.”

Let’s do this.

Why This Matters

Anyone can Google "What is Airflow?"
You’re here to build with it — and speak about it like an engineer who’s done it in prod.

Today’s drop isn’t theory.
You’ll create your first working DAG — a mini pipeline that does something useful, reliable, and interview-worthy.

You’ll leave this newsletter with:

• A runnable DAG

• A deployable GitHub project

• A STAR-based story you can use in interviews

• And clarity about what makes DAGs production-grade

Upgrade & Learn in Detail!

What You’re Building

The business use case is real:
A team receives a messy CSV of order data every few hours. Right now it’s being cleaned manually. You’ve been tasked with automating this.

Here’s what your DAG will do:

• Extract: Read CSV

• Transform: Drop nulls, clean types

• Load: Write to Snowflake

• Retry: If anything breaks

• Alert: Slack notifications

• Schedule: Every 6 hours

Your Airflow Starter Pack

Step 1: Install Airflow (If not done yet)

pip install apache-airflow
airflow db init
airflow webserver --port 8080
airflow scheduler

Open Airflow UI at localhost:8080 and create your user.

Read more

Not the kind you can meditate through —
The kind that breaks your data pipeline at 2AM while your dashboards go blank and your PM starts Slacking you with “???”

That’s why Airflow exists.

Airflow doesn’t just help you schedule things.
It’s the brain of your data pipelines — the system that says:
“Hey, this job failed, let’s retry.”
“Wait, don’t run until the upstream finished.”
“Log it, alert it, and move on.”

And yet, for most people, Airflow seems intimidating.

Today, we’ll break it down — no fluff, no buzzwords.
By the end of this, you’ll understand what Airflow really does, how DAGs work, and how to confidently talk about orchestration in any interview.

What Airflow Actually Does

Forget the textbook.

Here’s what Airflow really handles:

• Running your ETL jobs in the right order

• Retrying failed jobs automatically

• Monitoring status of every task

• Sending alerts if something breaks

• Triggering tasks on a schedule or based on dependencies

That’s orchestration. It’s not the “doing” — it’s the directing.

Airflow isn’t processing your data. It’s making sure the process happens correctly.

How DAGs Work (Without the Jargon)

A DAG = Directed Acyclic Graph = A set of tasks that:

1. Have a clear start and end

2. Never loop back on themselves

3. Run in a specific order

Think of it like a checklist for your pipeline:

• Step 1: Pull data from S3

• Step 2: Transform using Spark

• Step 3: Load to Snowflake

• Step 4: Run quality checks

• Step 5: Refresh dashboard

Each of these is a task.
The entire thing is a DAG.

In Airflow, these steps are defined in Python. You set dependencies like this:

extract_task >> transform_task >> load_task

UPGRADE TO ELITE/ANNUAL MEMBERSHIP

Read more

You walk into a Data Engineering interview. They ask:

“Can you walk me through an ETL pipeline you’ve built?”

Seems basic, right?
But here’s the catch: they’re not looking for just a tool dump.
They’re trying to reverse-engineer your thinking.

You’ve now built ETL pipelines, cleaned real data, and understood how batch vs streaming works.

Today’s goal is simple. In Zero2DataEngineer breakdown, we’ll decode how ETL interview questions are framed, what they’re secretly testing, and how to structure your answers like a pro — even if you’ve never worked at FAANG.

Whether you’re applying for a Data Engineer, Analytics Engineer, or even Backend-heavy role — ETL questions will show up.

Here’s how to answer them like someone who’s done it before.

How ETL Interview Questions Are Really Framed

They won’t ask:

“What is ETL?”

They’ll ask:

• How would you design a pipeline to load millions of rows from an external source?

• What happens if your load step fails halfway through?

• How do you make your pipeline idempotent?

• When do you use batch vs real-time ingestion?

The trick is to answer like a system thinker, not just a coder.

Subscribe now

Interview Answer Formula (Reframe Your Thinking)

Use this ETL STAR + Stack Formula when answering:

Situation: What was the use case?
Tool stack: Which tools did you choose and why?
Architecture: Show the pipeline stages.
Resilience: How did you handle failures, alerts, schema drift?
+ Stack Justification: Why this combo (Airflow + S3 + Spark etc.)?

Pro tip: If you haven’t built one end-to-end yet, use this:

“Here’s how I would design it for a [use case].”
Then walk them through your design — intelligently, not hypothetically.

Master These Real ETL Questions Before Your Next Interview

Q1. What are some common challenges in ETL pipelines?
1. Handling bad records
2. Schema changes
3. Late-arriving data
4. Dependencies & retries

Sample Answer:

“One of the biggest challenges I’ve faced is handling schema drift — especially when upstream sources silently change a column name or data type. I’ve built schema validation into the extraction step using Great Expectations and version control through Glue Catalog.

I’ve also handled bad records by logging and quarantining them into a separate S3 bucket with alerting. For late-arriving data, I design pipelines to be idempotent — using UPSERT logic or late data windows. And for dependencies, I always make sure DAGs have proper depends_on_past, retry logic, and failure alerts configured.”

UPGRADE TO ANNUAL

Read more

Now the question is:
How often should your pipeline run?
And more importantly… should it run in batches — or in real time?

Let’s break it down.

What Is a Batch Pipeline?

A batch pipeline runs on a schedule.
It pulls a large chunk of data at once — typically daily, hourly, or weekly.

Think:

• Nightly revenue reports

• Weekly customer churn rollups

• Monthly sales summaries

It’s like picking up laundry every Sunday.
No need to track every sock in real time — just do one large pickup.

Upgrade to Annual

When to Use Batch

• Your data changes slowly (e.g., payments, orders)

• You’re running reports, not alerts

• You want to keep cloud costs low

• You need high data completeness over speed

Read more

You’ve written your first full ETL pipeline.
But here’s the part that defines your value as a data engineer:

Can you reshape raw data into something useful?

That’s where transformation lives — and where good engineers become great ones.

What Data Transformation Really Means

It’s more than .fillna() or .dropna().
It’s about making messy, inconsistent data make sense — across teams, systems, and use cases.

Think of transformation as translation:
You’re turning raw, ambiguous data into structured, meaningful signals.

Real-World Scenario: Cleaning Order Data

You get a CSV dump from a vendor with these columns:

Your tasks:

• Normalize dates to standard format

• Strip $ and convert to float

• Unify state values (CA, Calif. → CA)

• Deduplicate based on key columns

• Group purchases by user

Sample Code Walkthrough

import pandas as pd

df = pd.read_csv("vendor_orders.csv")

# Standardize column names
df.columns = [c.strip().lower().replace(" ", "_") for c in df.columns]

# Parse dates and coerce invalid entries
df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")

# Clean monetary values
df["value"] = df["value"].replace("[\$,]", "", regex=True).astype(float)

# Normalize state values
df["state"] = df["state"].replace({"Calif.": "CA"})

# Deduplicate
df = df.drop_duplicates(subset=["order_id", "user_name"])

# Group and summarize
summary = df.groupby("user_name")["value"].sum().reset_index()

Read more

We’re not just talking about it.
We’re building it.

The Setup

You’ve been given a CSV file of product orders.
It’s messy. It has:

• Blank rows

• Wrong column types

• Duplicate users

• Date inconsistencies

Your job?

• Extract the data

• Transform it for cleanliness and insight

• Load it into a structured database

Step 1: Extract (Read the Data)

Let’s use Python’s pandas library to ingest the file.

import pandas as pd

df = pd.read_csv("orders_raw.csv")
print(df.head())

Scan the data. Understand it. Ask:

• Are column names clean?

• Are dates formatted?

• Are there obvious outliers or NULLs?

  UPGRADE TO ANNUAL

Read more

This week, we’re shifting from data structure to data movement.
Because understanding SQL is table stakes — but knowing how raw data becomes trustworthy, usable insights?

That’s where ETL comes in.

What the Hell Is ETL?

ETL stands for:

• Extract – pull raw data from source systems

• Transform – clean, reformat, validate, and reshape

• Load – push structured data into a final destination

That’s it — in theory.

But in practice, ETL is messy, strategic, and essential.

Let’s Make It Real: The Restaurant Analogy

You're running a restaurant.

• Extract = ordering raw ingredients from vendors

• Transform = chopping, cooking, seasoning, plating

• Load = serving the final dish to the customer

You wouldn’t send a customer raw flour and onions.
That’s what skipping ETL looks like in a data system.

UPGRADE TO ANNUAL

Read more

You’ve learned SQL.
You’ve learned schema design.
You now understand indexing, ACID, joins, and denormalization.

But here’s the part nobody tells you:

SQL isn’t just a query language.
It’s a way of thinking about data.

The best data engineers? They don’t “run SQL.”
They design systems that make SQL sing.

The Layer Cake of Modern Data

Let’s pull back the curtain. When you're working in a company like Stripe, Airbnb, or Netflix, your data system isn’t just tables and dashboards.

It’s a layered architecture, and SQL flows through it all:

1. Raw Layer – messy, fast, unfiltered (S3, GCS)

2. Staging Layer – cleaned, typed, deduped

3. Warehouse Layer – modeled, relational, denormalized

4. Semantic Layer – Looker, Tableau, PowerBI

5. Delivery Layer – Dashboards, APIs, notebooks

SQL is the thread that weaves it all together — whether you write it, generate it with dbt, or visualize it through tools.

So What’s the Real Job of SQL?

UPGRADE TO ANNUAL

Read more

In data engineering, pipelines break, APIs fail, jobs timeout.
But the data itself?

That still needs to be correct.

Today we talk about how databases keep your world from falling apart — even during chaos.

What is a Transaction?

A transaction is a single unit of work in the database.

• Transfer money? Transaction.

• Insert a row? Transaction.

• Update 3 tables in sequence? Still a transaction.

A good system either completes the transaction entirely or rolls everything back — no half-baked updates.

The ACID Model (Explained Like You’re On-Call)

Why this matters:
You don’t want to deduct money from a customer… but fail to generate the order.
You also don’t want two people editing the same record and overwriting each other’s data.

Real-Life Scenario: E-Commerce Checkout

Imagine a customer places an order. Here’s what happens:

1. Add a row in orders

2. Deduct stock from inventory

3. Charge the customer in payments

4. Send confirmation email

Without transactions, if step 3 fails after step 1 & 2 succeed…

• Inventory is gone

• No payment

• No order shipped

• No email sent

Now your support team is flooded.

With transactions, all 4 steps are wrapped in one block. If payment fails, nothing is saved.

SQL in Action

BEGIN;

INSERT INTO orders (...) VALUES (...);
UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 123;
INSERT INTO payments (...) VALUES (...);

COMMIT;

If anything fails before COMMIT, Postgres rolls it all back. No manual cleanup. No missing rows. No silent data corruption.

Interview Angle: How to Stand Out

UPGRADE TO ANNUAL MEMBERSHIP

Read more

Because all the cleanest SQL in the world means nothing…
If your query takes 17 minutes and your dashboard cries blood.

Why Indexing Matters

Imagine searching for a contact in your phone:

• No Index: Scroll, scroll, scroll… manually.

• With Index: Type “S” → boom, “Sara” shows up instantly.

That’s what indexes do to your database.
They help the engine locate what you need — without scanning every row.

Why Indexing Exists

Imagine walking into a bookstore with no signage.
No sections, no labels. Just 100,000 books in a pile.

That’s your database without an index.
Even the simplest query becomes a full table scan — the database flips through every row, like a librarian on Red Bull.

With an index?
You give the DB a map. It jumps to exactly where the answer lives.

The Two Most Common Index Types

Deep Dive: B-Tree vs Hash Index

Interview tip: Most RDBMS default to B-Tree, so unless otherwise needed, go with that.

Real World Example: The Wrong Index Costs Real Money

When I worked on a loyalty points table with 30M+ rows,
a teammate created an index on user_id.

Problem?
All queries filtered on created_at That index was useless.

We swapped it to:

CREATE INDEX idx_created_at ON loyalty_points (created_at);

Result: Dashboard load time dropped from 38s → 2s.
That tiny change saved $600/month in Snowflake compute credits.

Real-World: How One Index Saved a Failing Job

At a logistics startup, we had a shipment tracking job that queried a 60M row tracking_events table every 5 minutes.

We thought it was clean:

SELECT * FROM tracking_events WHERE status = 'delivered';

But it took 45 seconds.

The problem? No index on status.

Worse — status had only 3 values (in_transit, delivered, failed) → low cardinality.

So indexing that didn’t help. Instead, we looked at event_time, which had high uniqueness and was used in the WHERE clause:

CREATE INDEX idx_event_time ON tracking_events (event_time);

Query dropped to 2.1 seconds.

And that’s when we learned:

Index the column that filters the MOST rows — and does so selectively.

UPGRADE TO ANNUAL

Read more