Most data engineering roadmaps give you a list of 40 tools and leave you to figure out what comes first.
You don't need 40 tools.
Start with four things: SQL, Python, a data warehouse, and an orchestrator. Then build from there.
The order matters.
If you're preparing for a data engineering interview, Intervue.io connects you with senior data engineers for one-on-one mock interviews, including SQL problems, pipeline design questions, and detailed feedback.
What Does a Data Engineer Actually Do?
Data engineers build the systems that move data from where it's generated to where it's needed.
An analyst might need clean data for a dashboard. A data scientist might need structured data for a model. The data engineer builds and maintains the pipelines that make that possible.
In practice, that means extracting data from databases and APIs, cleaning and transforming it, loading it into a warehouse, scheduling pipelines, monitoring failures, and fixing things when they break.
And things will break.
A pipeline might fail at 2 AM. A query might slow down after the data volume jumps. A new source might need to be integrated into an existing system.
That's the work you're preparing for.
The Data Engineer Roadmap
1. SQL (6–8 weeks)
Start with SQL.
SQL shows up throughout data engineering work and is a major part of technical interviews.
Begin with:
- SELECT
- WHERE
- GROUP BY
- ORDER BY
- JOIN
Get comfortable enough that you can write these without constantly checking the syntax.
Then move to window functions. Focus on RANK, DENSE_RANK, ROW_NUMBER, LAG, LEAD, and PARTITION BY.
After that, learn CTEs. You'll use them to break complicated queries into smaller, readable pieces.
Finally, spend some time on query performance. Understand indexes, hash joins, nested loop joins, and EXPLAIN ANALYZE.
You don't need to become a database administrator. You do need to know why a query is slow and what you can do about it.
PostgreSQL is a good place to practise.
2. Python for Data Engineering (4–6 weeks)
Once SQL feels comfortable, move to Python.
Focus on the parts you'll use for data work:
- Reading and writing CSV, JSON, and Parquet
- Data manipulation with pandas
- Writing functions for cleaning and validation
- Working with REST APIs
- Error handling
- Writing scripts that can run without manual intervention
Real data is messy.
Files have encoding issues. APIs return unexpected formats. A column that looks like a number might arrive as a string.
Learn how to handle that stuff early. You'll run into it a lot.
You can leave the deeper machine learning libraries for later.
3. Data Modeling and Warehousing (4–5 weeks)
Now you're getting into the part that starts to feel distinctly like data engineering.
Data modeling is about deciding how data should be organised so people can query it and make sense of it.
Start with star schemas and Kimball's approach to dimensional modeling.
Then pick one warehouse and learn it properly.
Snowflake, BigQuery, and Redshift are common choices. The right one depends on the companies you're targeting.
Go beyond basic queries. Learn partitioning, clustering, query planning, and how warehouse design affects cost and performance.
This is also a good time to learn dbt.
You'll use dbt to build, test, document, and version-control SQL transformations.
4. Airflow and Pipeline Orchestration (3–4 weeks)
Say you have a Python script that pulls data, transforms it, and loads it into your warehouse.
Now someone asks you to run it every morning at 6 AM, retry it when it fails, alert the team if it keeps failing, and make sure dependent tasks run in the right order.
That's orchestration.
Start with Airflow.
Learn how to write DAGs, schedule tasks, define dependencies, configure retries, set up alerts, and debug failed runs.
Once you understand Airflow, other orchestration tools become easier to pick up.
5. Spark and Big Data (4–6 weeks)
Pandas is great until the data gets too big for your machine.
That's where Spark comes in.
Start with PySpark and learn how to read data from cloud storage, work with DataFrames, apply transformations, and write the results back to storage or a warehouse.
Then dig into the areas that come up in interviews:
- Partitioning
- Transformations vs. actions
- Slow Spark jobs
- Broadcast joins
- Performance tuning
If you're targeting teams that use Spark heavily, Databricks is worth learning too.
6. Kafka and Streaming (3–4 weeks)
Batch pipelines process data on a schedule.
Streaming pipelines process data as it arrives.
Kafka is widely used for this kind of work.
For junior roles, you can usually pick this up after the core pipeline skills. For mid-level and senior roles, streaming becomes more relevant, especially for systems handling real-time events.
Learn topics, partitions, consumer groups, at-least-once delivery, exactly-once semantics, and how Kafka works with Spark Streaming or Flink.
7. One Cloud Platform (4–6 weeks)
Pick one cloud platform and go deep.
Trying to learn AWS, GCP, and Azure at the same time usually gives you a shallow understanding of all three.
For AWS, focus on S3, Glue, Redshift, Lambda, and IAM.
For GCP, focus on BigQuery, Cloud Storage, Dataflow, and Pub/Sub.
For Azure, focus on Azure Data Factory, Synapse Analytics, and Blob Storage.
Pick based on the companies you're applying to.
What Do Data Engineering Interviews Test?
SQL is a big one.
Expect questions around window functions, joins, data manipulation, and query performance.
Python can come up too, especially for data manipulation and scripting.
Then there's system design.
At mid and senior levels, you might be asked to design a pipeline that processes 100 million events a day. You need to think through ingestion, storage, transformations, failures, scaling, and monitoring while answering follow-up questions along the way.
That's where knowing the concepts and being able to use them under pressure become two different things.
You might know how a window function works and still freeze when you have to write one in a plain editor while explaining your thinking.
Practise that part too.
Intervue.io offers one-on-one data engineering mock interviews with engineers who have worked on production pipelines.
Data Engineer Roadmap FAQs
What should I learn first for data engineering?
Start with SQL.
Get comfortable with joins and aggregations, then move into window functions, CTEs, and query performance.
How long does it take to become a data engineer?
From scratch, roughly 8–12 months of consistent learning and project work is a reasonable range.
With strong SQL experience, such as a data analyst background, the transition can be shorter. Software engineers with Python experience may need less time because they already have programming fundamentals.
Do I need to learn every cloud platform?
No.
Pick one and learn it properly. AWS, GCP, and Azure all have data engineering services and jobs.
Is a CS degree required?
A CS degree isn't the only path into data engineering.
You need to be able to demonstrate that you can do the work. Projects, practical experience, cloud skills, SQL, and Python all help show that.
What should I learn after SQL and Python?
A practical sequence is:
SQL → Python → Data Modeling → Data Warehouse → dbt → Airflow → Spark → Kafka → Cloud
You don't need to master every tool before applying for jobs. Build depth as you move through the path.
