Skip to content
getdataengineeringjobs

How To Become A Data Engineer and Land Your First Job

A data engineering roadmap built on 4,342 open jobs: the skills employers ask for, what to learn first, and where the junior and entry-level jobs really are.

By Adyl · · 19 min read

In January 2021 I wrote How To Become a Data Engineer, a list of the skills and resources I thought mattered. Five years later, much of it still holds: SQL didn’t die, Python is everywhere, and distributed systems are still the hard part. But the tools changed (dbt, Databricks, lakehouse table formats), the cloud went from “nice to have” to the default, and AI showed up in a quarter of job descriptions.

This time I don’t have to guess. This board collects data engineering jobs from company career pages, applicant tracking systems and job boards, and records what each one asks for. So this guide is built on data: 4,342 open data engineering jobs at 1,363 companies, as of 15 November 2026.

TL;DR

  • Learn SQL and Python first. 82% of jobs ask for at least one of them, and half ask for both.
  • Then add one engine, one orchestrator and one cloud. Spark (35% of jobs), Airflow (29%) and AWS (40%) are the most common, with dbt (27%) close behind.
  • The entry level is narrow. Only 6% of open jobs are junior roles or internships; 56% are senior or above. Plan for that instead of being surprised by it.
  • Junior roles are mostly on-site. 17% of junior and intern jobs are remote, against 29% overall.
  • The common side doors are consultancies, analytics engineering and internal moves from analyst or backend roles.
  • Build one end-to-end project that looks like real work rather than ten tutorials.

About the data

Every number below comes from the jobs that were open on this board on 15 November 2026. Each job is read by a language model that pulls out the seniority, technologies, languages, cloud platforms, location and salary, and leaves a field empty when the description doesn’t say. A few caveats:

  • Shares are of all 4,342 open jobs, unless I say otherwise. A technology at 20% means one in five job descriptions mentions it.
  • “Not specified” is common. Seniority is missing for 16% of jobs, and most descriptions say nothing about visa sponsorship.
  • The board leans to the US and Europe. 35% of jobs with a known location are in the US.
  • It is a snapshot, not a trend. The board started collecting in October 2026, so I can’t say what grew or shrank yet. The live numbers are on the stats page.

What a data engineer actually does

Data engineers build and run the systems that move data from where it is produced to where it is used: ingestion from applications and third-party APIs, storage in warehouses and lakes, transformations, and the reliability and governance around all of it. Analysts and data scientists stand on top of that work. If the pipeline is late, broken or wrong, everything above it is too.

In practice, “data engineer” covers three fairly different jobs. Of the open jobs on the board:

Kind of role Share What the work looks like
Core data engineering 56% Pipelines, ingestion, batch and streaming processing, warehouse loading
Data infrastructure 26% Platforms other data teams build on: Kafka, Spark clusters, Kubernetes, Terraform, storage formats
Analytics engineering 15% Modelling clean, tested tables in the warehouse with SQL and dbt, close to analysts and the business

Open data engineering jobs by kind of role: core data engineering 56% (pipelines, ingestion, batch and streaming processing), data infrastructure 26% (platforms other data teams build on), analytics engineering 15% (modelled, tested warehouse tables with SQL and dbt), not classified 4%

The remaining 4% couldn’t be classified. The three need different emphasis. Analytics engineering rewards strong SQL, data modelling and an understanding of the business. Data infrastructure is closer to backend and platform engineering. Core data engineering sits in the middle and is the largest market, so it is the safest default target.

The hard truth about entry-level jobs

Here is the seniority of every open job on the board:

Seniority Jobs Share
Intern 61 1.4%
Junior 199 4.6%
Mid-level 791 18.2%
Senior 1,820 41.9%
Lead, staff, principal 598 13.8%
Manager, director and above 194 4.5%
Not specified 679 15.6%

Open data engineering jobs by seniority: intern 1.4% (61 jobs), junior 4.6% (199), mid-level 18.2% (791), senior 41.9% (1,820), lead, staff and principal 13.8% (598), manager, director and above 4.5% (194), not specified 15.6% (679)

Only 260 jobs, 6%, are explicitly junior or internships. Senior and above is 56%. Employers mostly hire data engineers who have already broken production pipelines on someone else’s budget.

The experience requirements tell the same story. Taking the first “N years” each description mentions (a rough measure), the median ask is:

  • 2 years for junior roles
  • 3 years for mid-level roles
  • 5 years for senior roles
  • 8 years for staff roles

Median years of experience asked for, by seniority: junior 2 years (126 jobs stating a number), mid-level 3 (571), senior 5 (1,185), lead 7 (143), staff 8 (172), principal 10 (72)

So even “junior” often means “a year or two of something relevant”. That doesn’t have to be a data engineering title. It can be experience as an analyst, a backend developer or a database administrator.

Two more things to plan for:

  • Junior roles are less often remote. 20% of junior jobs and 8% of internships are remote, against 29% of all jobs and 35% of senior ones. Companies want new engineers close to the people who will teach them. If you’re starting out, a hybrid job in a city you can reach is more realistic than “fully remote, anywhere”.
  • Visa sponsorship is rare and mostly unstated. Only 360 jobs mention it either way, and 116 of those offer it. That’s under 3% of all jobs. If you need sponsorship, your first data job is more likely to be in a country where you can already work.

Share of open jobs that are remote, by seniority: intern 8% (5 of 61), junior 20% (39 of 199), mid-level 28% (219 of 791), senior 35% (635 of 1,820), lead 28% (72 of 253), staff 38% (90 of 237), principal 31% (34 of 108), all open jobs 29% (1,270 of 4,342)

None of this means it is impossible. It means you shouldn’t aim only at job titles that say “Junior Data Engineer”. I cover the other ways in below.

The skills employers ask for

SQL and Python are not optional

Language Share of jobs
Python 70.6%
SQL 61.3%
Java 13.7%
Scala 10.7%
Go 6.8%

Share of open jobs mentioning each programming language: Python 70.6%, SQL 61.3%, Java 13.7%, Scala 10.7%, Go 6.8%, C++ 3.7%, TypeScript 3.6%, Rust 2.9%. 82% ask for Python or SQL, 50% ask for both, and 76% of junior and intern jobs ask for Python

Python and SQL together dominate: 82% of jobs ask for at least one, 50% ask for both. In junior jobs it is even starker: Python appears in 76% of them and SQL in 63%.

In 2021 I predicted we’d “stick to SQL for another decade or two”. I’m comfortable repeating that. And being able to write SQL isn’t enough: you need to be good at it.

  • Joins of every kind, and what happens to row counts when keys aren’t unique
  • Aggregations, GROUP BY / HAVING, and NULL semantics
  • Window functions (ROW_NUMBER, LAG, running totals, deduplication)
  • CTEs, and turning a messy question into readable steps
  • Reading a query plan, and knowing what an index is and when it helps

For Python, you need more than notebooks. Learn to write a small, tested package: functions and modules, virtual environments and dependencies (I now use uv for everything), reading from APIs with retries, working with files and dataframes, logging and pytest. Data engineering is software engineering whose product is data.

Java and Scala still appear in about one job in four combined, mostly in streaming and infrastructure roles. They’re worth learning later, not first.

Spark: still the engine to learn

The most-mentioned technologies across all open jobs:

Technology Share of jobs
Apache Spark 35.2%
Apache Airflow 29.2%
dbt 27.2%
Databricks 24.3%
Apache Kafka 20.7%
Snowflake 20.0%
Kubernetes 18.1%
Git 16.7%
Terraform 16.1%
Docker 13.4%
BigQuery 13.2%
Power BI 11.3%
PostgreSQL 10.5%

Share of open jobs mentioning each technology, all jobs vs. junior and intern jobs: Spark 35% vs 31%, Airflow 29% vs 21%, dbt 27% vs 18%, Databricks 24% vs 16%, Kafka 21% vs 15%, Snowflake 20% vs 14%, Kubernetes 18% vs 12%, Git 17% vs 18%, Terraform 16% vs 9%, Docker 13% vs 11%, BigQuery 13% vs 7%, Power BI 11% vs 12%, PostgreSQL 10% vs 14%

Spark is in more than a third of all jobs and is the top technology for junior roles too (31%). With Databricks at 24% (it is built on Spark), learning Spark well covers a large part of the market. “Well” means understanding why a job is slow, not memorising the DataFrame API: partitions, shuffles, skew, broadcast joins, and how the work is split into stages and tasks.

Hadoop is down to 7% and Hive to 3%. You don’t need to learn the old stack unless a specific employer runs it.

Orchestration: Airflow first

Airflow is in 29% of jobs, against 6% for Dagster and 3.5% for Prefect. Learn Airflow: DAGs, scheduling, retries, backfills, idempotent tasks, and why you should never put heavy processing inside the scheduler. The concepts carry over to the newer tools, which are better designed in several ways.

The modern warehouse: dbt plus one platform

The data stack has settled around a few cloud platforms with a transformation layer on top:

  • dbt (27%) is the standard way to write and test SQL transformations. If you’re aiming at analytics engineering, it is essential.
  • Snowflake (20%), Databricks (24%) and BigQuery (13%) are the main warehouses and lakehouses. Pick one; the ideas transfer.
  • Open table formats are growing in importance: Delta Lake is in 8% of jobs and Apache Iceberg in 6%. Know what they solve (ACID transactions and schema evolution on files in object storage).

Underneath all of these is data modelling: dimensional models, facts and dimensions, slowly changing dimensions, grain. It is mentioned explicitly in 27% of descriptions and implied by most of the rest. It is also the skill bootcamps most often skip.

Cloud: pick one and go deep

56% of jobs name a cloud platform:

Cloud Share of jobs
AWS 39.9%
Azure 30.5%
GCP 27.0%

Share of open jobs naming each cloud platform, all jobs vs. junior and intern jobs: AWS 40% vs 32%, Azure 30% vs 22%, Google Cloud 27% vs 18%. 56% of all jobs name at least one

My 2021 ranking (AWS, then Azure, then GCP) still holds, but the gap is smaller than people think. Azure is strong in large enterprises and consultancies (Azure Data Factory is in 8.5% of jobs, Microsoft Fabric in 5%). Choose based on where you want to work. If you have no preference, choose AWS.

You don’t need every service. For your cloud, learn object storage (S3, ADLS or GCS), its managed warehouse, its identity and permission model (IAM is where beginners lose days), and how to keep costs under control.

Streaming: one in five jobs

Kafka is in 21% of jobs and Flink in 7.5%; “streaming” appears in 22% of descriptions. Streaming is rarely a first job, but understanding the log (topics, partitions, offsets, consumer groups, delivery guarantees) makes you a better engineer even for batch work. Jay Kreps’s essay The Log is still the best introduction.

Infrastructure and engineering practice

Kubernetes (18%), Terraform (16%), Docker (13%) and Git (17%, and assumed by nearly everyone else) show how far the job has moved towards software engineering. At a minimum, be comfortable with Git branches and pull requests, write a Dockerfile, and read a CI pipeline. Terraform and Kubernetes can wait until your first job, unless you’re aiming at data infrastructure.

The skills that don’t show up as tags

Some of the most frequent requirements aren’t technologies at all:

Share of open jobs whose description mentions each topic: communication 53%, data quality 36%, a degree 29%, data modelling 27%, LLMs, generative AI or RAG 24%, streaming 22%

  • Communication is mentioned in 53% of descriptions. You’ll explain to analysts why a number changed and to engineers why their schema change broke your pipeline.
  • Data quality is in 36%. Tests, contracts, monitoring, and the habit of asking “how would I know if this is wrong?”
  • Data modelling is in 27%, as above.

What about AI?

24% of job descriptions mention LLMs, generative AI, RAG, vector databases or AI agents. In most of them, the data engineer’s job is to feed AI systems: clean, documented, well-governed data, document pipelines, embeddings, and evaluation datasets. You don’t need to become an ML engineer. But knowing how a retrieval pipeline works, and having used an LLM API in a project, is now a real plus.

AI coding assistants also raise the bar for juniors: writing boilerplate is no longer a differentiator. Understanding why a pipeline is correct, fast and cheap is.

A learning roadmap

This is the order I’d learn things in, starting from zero programming experience. If you already work as an analyst or developer, skip what you know. The timings assume steady part-time study.

A learning roadmap in seven steps: 1. SQL, 1–2 months; 2. Python, 2–3 months; 3. Data modelling, 1 month; 4. A cloud and a warehouse, 1–2 months; 5. dbt and Airflow, 1–2 months; 6. Spark, 1–2 months; 7. Fundamentals (Docker, Linux, database internals, distributed systems), ongoing

1. SQL (1–2 months). Install PostgreSQL and work with a real dataset. Solve SQL problems until window functions feel natural.

2. Python (2–3 months). Write scripts that call an API, save results to files and load them into Postgres. Add tests. Learn Git along the way and put everything on GitHub.

3. Data modelling and warehousing (1 month). Read Kimball. Design a star schema for a business you understand. Learn the difference between OLTP and OLAP, and why columnar storage exists.

4. A cloud and a warehouse (1–2 months). Create a free-tier account on one cloud. Load data into object storage and query it from Snowflake, BigQuery or Databricks.

5. dbt and orchestration (1–2 months). Rebuild your transformations in dbt with tests and documentation. Schedule the whole pipeline with Airflow.

6. Spark (1–2 months). Process a dataset that doesn’t comfortably fit in pandas. Learn to read the Spark UI.

7. Fundamentals, continuously. Docker, Linux, networking basics, how databases store data, and distributed systems. These are what turn a tool user into an engineer, and they are what senior interviews test.

That’s roughly a year. It’s a long time, but it is what the market data says employers expect.

Build one project that looks like real work

Recruiters have seen thousands of Titanic notebooks. One realistic, end-to-end project does more than ten tutorials. It should have:

  • A real, changing source. A public API that updates daily (transport, weather, energy prices, public procurement, GitHub events), not a static CSV.
  • Incremental ingestion. Load only new data, handle late and duplicate records, and be safe to re-run (idempotent).
  • Raw and modelled layers. Land raw data in object storage, then build staging and mart models with dbt.
  • Orchestration. A scheduled Airflow DAG with retries and alerting.
  • Data quality tests. dbt tests or similar, plus at least one check that would catch a silent upstream change.
  • Infrastructure as code for bonus points: a Docker Compose file anyone can run, or Terraform for the cloud resources.
  • A README that explains decisions. Why this schema, what breaks at 100 times the volume, what it costs to run per month. This is what you’ll talk about in interviews.

Pick a domain connected to the industry you want to work in. 15% of open jobs are at consulting firms; after that come AI/ML companies, SaaS, fintech, healthcare and banking. A fintech project in a fintech interview is a strong opening.

How to land the first job

Use the side doors

With so few junior titles, most people get into data engineering sideways:

  • Move internally. If you’re an analyst, take over the pipelines your team depends on. If you’re a backend engineer, volunteer for the event tracking or reporting database. In 2021 I wrote that backend engineers are the strongest candidates for the move, and that is still true. An internal move skips the “2 years of experience” filter entirely.
  • Consultancies. Consulting is the largest single industry on the board, and consultancies hire graduates in cohorts and train them. Of the junior and intern jobs whose employer’s industry is known, 41% are at consulting firms, against 32% of all jobs. France stands out: it has the second-most junior jobs on the board after the US, many of them graduate roles at consultancies (“Junior Data Engineer”, “Data and AI Engineer Consultant – Recent Graduate”).
  • Analytics engineering. It is 15% of the market, sits closest to analysts, and needs SQL and dbt more than distributed systems. It is a good first step towards core data engineering.
  • Internships and graduate programmes. There are few, but they are the most direct route. Large companies post next summer’s internships in the autumn, so apply early.
  • Adjacent titles. Junior work often hides behind names like “Data Support Engineer”, “Data Product Engineer” or “Software Engineer, Data”. Search by skills, not just by title.

Open junior and intern jobs. By industry, share of the 133 with a known industry: consulting 41%, automotive 10%, fintech 7%, food tech 6%, insurance 5% (consulting is 32% of all open jobs with a known industry). By country: United States 68 jobs, France 29, Netherlands 13, Singapore 12, Canada 11, Brazil 10

On this board you can filter for junior and intern roles, filter by technology, and save a search to get new matches by email.

Read the description, then tailor your CV

  • Mirror the job’s stack where you honestly can. If the description says Airflow, dbt and Snowflake, those words should appear in your CV, next to what you built with them.
  • Lead with your project and its results (“loads 2 million rows a day incrementally, rebuilds in 4 minutes, costs $3 a month”), not with a list of tools.
  • One page. Link to GitHub, and make sure the repository’s README is the first thing a reviewer sees.
  • Apply early. The first applicants usually get the most attention, so set up alerts rather than checking once a month.

Prepare for the interview

A typical data engineering interview loop has:

  1. SQL, live: joins, aggregations, window functions, deduplication. Practise on LeetCode or a similar site until it’s routine.
  2. Python coding: data manipulation and simple algorithms. You won’t need dynamic programming, but you will need dictionaries, sets and complexity basics.
  3. Data modelling: “design the tables for a ride-hailing app” or similar. Talk about grain, facts and dimensions, and how history is kept.
  4. Pipeline or system design, from mid-level up: batch or streaming, how to handle late data, backfills, failures and costs.
  5. Behavioural questions: a time you found bad data, a time you disagreed with a stakeholder. Remember that communication is in more than half of all job descriptions.

What it pays

About 16% of open jobs publish a salary, most of them in the US. These are the annual salaries stated in US jobs posted since May 2026, in US dollars: the median of each range’s midpoint, and the middle half of those midpoints.

Seniority (US) Jobs with salary Middle half Median
Junior 25 $138k–173k $155,000
Mid-level 96 $134k–188k $157,500
Senior 201 $165k–230k $196,000
Staff 59 $223k–403k $267,000

US salaries by seniority, annual pay stated in US jobs posted since May 2026. Junior (25 jobs): middle half $138k–173k, median $155k. Mid-level (96): $134k–188k, median $157.5k. Senior (201): $165k–230k, median $196k. Staff (59): $223k–403k, median $267k

Treat the junior figure with care: there are only 25 such jobs, and the companies that publish junior salaries are disproportionately large US tech firms, which is why it sits so close to mid-level. Outside the US, and in most of the jobs that don’t publish a salary, the numbers are considerably lower. The point stands, though: in the US, a data engineer’s salary is excellent even early in a career, and it grows steeply with seniority. For more detail, see What Data Engineers Get Paid.

Resources

These are the resources I’d use today, most of them free:

SQL - Mode’s SQL Tutorial: a gentle start, from basic to advanced. - Use The Index, Luke!: how indexes and query plans really work. - Modern SQL: window functions and the features most tutorials skip.

Data engineering - Fundamentals of Data Engineering by Joe Reis and Matt Housley: the best overview of the whole lifecycle. - Data Engineering Zoomcamp by DataTalks.Club: a free, project-based course covering Docker, Terraform, a warehouse, dbt, Spark and Kafka. - dbt Learn: the official free dbt courses. - Introduction to Apache Airflow: my own introduction to Airflow.

Data modelling - The Data Warehouse Toolkit by Ralph Kimball and Margy Ross: decades old and still the reference for dimensional modelling.

Fundamentals - Designing Data-Intensive Applications by Martin Kleppmann: if you read one book on this list, read this one. - CMU 15-445 Database Systems with Andy Pavlo: how databases work inside, with lectures on YouTube. - MIT’s distributed systems course (6.5840, formerly 6.824): free lectures and labs. - The Log: What every software engineer should know about real-time data’s unifying abstraction by Jay Kreps.

Certifications can help get past CV filters, especially at consultancies and enterprise employers. The cloud providers’ data engineer certifications (AWS, Azure and Google Cloud each have one) are the most recognised. Treat them as a structured syllabus, not a substitute for a project.

Final words

In 2021 I ended with “practice makes perfect”, and that is still the advice. The market data adds a warning: there are far fewer junior jobs than senior ones, and the bar for the first job is higher than tutorials suggest. Prepare for that. Master SQL and Python, learn one tool of each kind, build one project that looks like real work, and use the side doors: internal moves, consultancies, analytics engineering and internships.

When you’re ready, start with the junior and intern roles on the board, and check the stats page to see what employers are asking for this week. Good luck!

The weekly Digest

The week's most transparent data engineering jobs in your inbox once a week.

Once a week, a few new jobs that state their salary, location and stack, plus links to the rest. No open tracking; clicks are counted per issue, not per person. Unsubscribe with one click.

More from the blog