Data Engineering Pipeline Orchestration Interview Questions
Reviewed by Mark Dickie · Last updated
5 questions on this page you can answer and see marked on the spot. Go to the first one
Pipeline orchestration is the coordination of data tasks into scheduled, dependency-aware workflows that move and transform data from source to destination on time and in the right order. For a data engineering interview, you should be able to explain how DAG-based schedulers work, why idempotency matters for re-runs, and how retry and backfill strategies keep pipelines reliable when upstream systems fail or arrive late. Expect questions on Airflow concepts (operators, sensors, XCom, task state), comparison with Dagster and Prefect, and how you would design a pipeline that handles late-arriving data, partial failures, and SLA monitoring.
What does a pipeline orchestration interview test?
Interviewers want to see that you can reason about the full lifecycle of a data pipeline, not just write a single SQL transform. The core areas break down as follows:
| Concept | What interviewers probe | Common tools |
|---|---|---|
| DAG design | Task dependencies, branching, dynamic task generation | Airflow, Dagster |
| Scheduling | Cron expressions, sensors, catchup vs backfill | Airflow, Prefect |
| Reliability | Retries, exponential backoff, dead-letter handling | All major orchestrators |
| Idempotency | Safe re-runs, deduplication, checkpointing | dbt, Spark, custom code |
| Observability | SLA misses, alerting, lineage tracking | Airflow callbacks, OpenLineage |
How do you design an orchestration pipeline that survives failures?
A strong interview answer walks through the decisions in order:
- Define the DAG topology: identify which tasks can run in parallel and which have hard data dependencies.
- Choose operators or assets that match your compute layer — e.g. KubernetesPodOperator for isolated jobs, PythonOperator for in-process transforms.
- Set retry policies with exponential backoff so transient failures self-heal without manual intervention.
- Make every task idempotent so that a backfill or re-run produces the same output regardless of how many times it executes.
- Add SLA rules and alerting (email, Slack, PagerDuty) so missed deadlines surface before downstream consumers notice.
Airflow vs Dagster vs Prefect: what should I know?
Airflow remains the most common orchestrator in production and the one most interviewers default to. Dagster introduces the asset-based model where you define data assets and their dependencies rather than tasks, which changes how you think about lineage and freshness. Prefect leans into a Pythonic, decorator-driven API with dynamic workflows and easier local development. You do not need deep expertise in all three, but you should be able to explain the conceptual difference between task-based and asset-based orchestration and when each approach fits.
What are the most common orchestration mistakes candidates make?
A frequent error is treating the orchestrator as a compute engine — running heavy Spark or Pandas work inside a scheduler worker instead of delegating to a cluster. Another is ignoring idempotency: if a task appends rows on every run, a retry or backfill corrupts the target table. Know how to use partitioning, overwrite modes, and merge logic so re-runs are safe. Finally, candidates often under-configure observability; being able to describe how you would catch an SLA miss and trace the failure back through lineage shows operational maturity that interviewers look for.
Key facts
- Tarmac has 15 Data Engineering interview questions on this topic, 10 of them on this page, at difficulty 1–5 of 5.
- Tarmac tracked 1,538 job postings asking for Data Engineering in September 2026.
- Roles asking for Data Engineering advertise a median base salary of £75,000, across 448 job postings as of September 2026.
- Tarmac last reviewed these Data Engineering interview questions on 19 October 2026.
At a glance
| Questions | 10 shown · 15 in the bank |
|---|---|
| Difficulty | 1–5 of 5 |
| Formats | Multiple choice, Ordering, Multiple answer, Code output, True / false, Find the bug, Short answer, Fill in the blank, Coding exercise, Flashcard |
| Interactive | 1 run your code against tests, in the app |
What you'll review
- dag based orchestration
Practice questions
Try one before you open the answer. Pick an option and press Check; it's marked on the spot.
In a DAG-based pipeline orchestrator such as Apache Airflow, what must be true of the graph of task dependencies?#
Options
Show answer
A DAG-based orchestrator requires its task dependency graph to have no cycles. A directed acyclic graph lets the scheduler compute a valid run order, because no task waits on itself through a chain of dependencies. Multiple root tasks and tasks with fan-in or fan-out are allowed.
DAG stands for directed acyclic graph. The orchestrator uses the dependency edges to compute a valid execution order, and that is only possible if following the edges never returns to a starting task. Tasks may have multiple roots, fan-in and fan-out, so the graph does not need to be a tree. Requiring a complete graph or single upstream and downstream links would be wrong, since the graph only needs the ordering constraints the pipeline actually has.
A daily Airflow DAG has four tasks wired as a single linear chain: each task declares the previous one as an upstream dependency. Put the tasks in the order the scheduler will run them within one DAG run.#
Put these in order
Show answer
In a linear Airflow DAG, tasks run in dependency order: extract first, then load into staging, then transform into the fact table, and finally publish. Each task starts only after its upstream task succeeds, so the chain runs strictly in sequence.
The tasks form a linear dependency chain: each task lists the previous one as upstream, so a task becomes eligible only after its upstream task succeeds. The extract must run first because it produces the data that gets loaded into staging. The load must come before the transform, which reads from staging. The transform must finish before the publish step, which exposes the fact table. Since every adjacent pair has a strict upstream-to-downstream relationship, there is exactly one valid order.
In Apache Airflow (2.x), which of the following statements about DAG-based orchestration are true? Select all that apply.#
Options
Pick every one that applies.
Show answer
In Apache Airflow, tasks with no dependency path between them can run concurrently, limited by executor slots and pools. By default, a task waits until all of its direct upstream tasks have succeeded. Cycles are rejected because the scheduler needs an acyclic graph to determine a valid execution order.
A DAG is a directed acyclic graph, so the scheduler needs a valid topological order; a cycle is rejected at parse time regardless of retry settings. Tasks that have no path between them in the graph are independent and may run in parallel, limited by executor slots and pools. The default trigger rule, all_success, makes a task wait until every direct upstream task has succeeded. Retrying a failed task only re-runs that task instance, not the whole DAG, so the statement about re-running every task is false.
A small DAG orchestrator runs tasks using Kahn's algorithm. Each task starts with an in-degree equal to its number of upstream dependencies. The ready list is kept sorted, and when a task finishes, its downstream tasks are visited in sorted order. What does the following program print?#
deps = {
"extract": [],
"transform": ["extract"],
"load": ["transform"],
"audit": ["extract"],
"report": ["load", "audit"],
}
def run_order(deps):
indegree = {t: len(d) for t, d in deps.items()}
children = {t: [] for t in deps}
for t, ds in deps.items():
for d in ds:
children[d].append(t)
ready = sorted(t for t, n in indegree.items() if n == 0)
order = []
while ready:
task = ready.pop(0)
order.append(task)
for c in sorted(children[task]):
indegree[c] -= 1
if indegree[c] == 0:
ready.append(c)
ready.sort()
return order
print(" -> ".join(run_order(deps)))Show answer
extract -> audit -> transform -> load -> report
Only extract has no upstream dependencies, so it runs first. Finishing it lowers the in-degree of transform and audit to zero; the sorted ready list is [audit, transform], so audit runs next. Finishing audit lowers report from 2 to 1 but does not release it. transform then runs and releases load, which runs next, and finishing load finally releases report. The printed order is therefore extract -> audit -> transform -> load -> report.
In Apache Airflow 2.x, a task that uses the default trigger rule all_success is not executed when one of its upstream tasks ends in the failed state after exhausting its retries.#
Options
Show answer
True. In Apache Airflow 2.x, a task with the default all_success trigger rule does not run when an upstream task has failed after exhausting its retries. The scheduler marks the downstream task upstream_failed instead, which prevents incomplete or bad data from flowing into later steps.
Airflow's default trigger rule for a task is all_success, which requires every direct upstream task to have succeeded before the task runs. When an upstream task fails after its retries are used up, the downstream task cannot satisfy that rule, so the scheduler marks it upstream_failed and does not execute it. Other trigger rules such as all_done or one_failed would change this behavior, but the question concerns the default.
This Airflow 2.x DAG loads one daily partition of orders. It is scheduled @daily with catchup=True, and operators are supplied the run's context. A data engineer reports that backfilling the last 30 days writes every historical run into today's partition, and rerunning a past date changes its data. Which option identifies the bug?#
1 from datetime import datetime
2
3 def load_partition(**context):
4 run_date = datetime.now().date().isoformat()
5 rows = extract_orders(run_date)
6 write_partition("orders", run_date, rows)
7
8 with DAG(dag_id="orders_daily", start_date=datetime(2024, 1, 1),
9 schedule="@daily", catchup=True) as dag:
10 load = PythonOperator(task_id="load", python_callable=load_partition)Options
Show answer
Line 4: datetime.now() returns the wall-clock time at execution, so backfills and reruns use today's date instead of the run's logical date (context["ds"]).
The run date is derived from datetime.now() on line 4, which is the time the task actually executes. During a backfill or a rerun, the task executes today, so every historical run extracts and writes today's data into the same partition. The fix is to use the logical date supplied by the context (ds, or context["ds"]), which is fixed per DAG run and makes each run idempotent. The other options are wrong: datetime is correctly imported from the datetime module, the preset @daily schedule is valid, and run_date being an ISO string is the expected type for a partition key.
An Airflow 2.x DAG runs daily with catchup=True. One task in it is declared with depends_on_past=True. Explain how this flag changes when a task instance for run date D is allowed to start, and describe one concrete operational consequence of it when a single earlier run fails and you later backfill a month of dates.#
Show answer
With depends_on_past=True, the task instance for run date D is not eligible to run until the same task's instance for the previous run date has reached success. Scheduling is therefore serialized along the time axis for that task, even though the DAG runs themselves are independent. The consequence: if one earlier day's task fails and is not cleared or marked success, every later date's instance of that task stays waiting (not running), so a backfill over a month stalls at the first failed day and the remaining dates cannot make progress until that failure is resolved.
The flag adds an inter-run dependency: a task instance for date D waits for the same task's instance for the prior scheduled date to succeed. Because of this chain, one failed instance blocks every subsequent instance of that task, so a backfill of many dates cannot proceed past the failed date until an operator clears or marks that instance successful. The answer must name the cross-run dependency (not just the within-run upstream/downstream dependencies) and the stall/blocking consequence during backfill.
In Apache Airflow 2.x, a DAG has the dependencies extract >> transform >> load and [extract, transform, load] >> cleanup. The cleanup task must run once every upstream task has reached a terminal state, whether that state is success, failed, or skipped. Set its trigger rule to _____ (give the trigger rule's string value).#
Show answer
In Apache Airflow 2.x, a DAG has the dependencies extract >> transform >> load and [extract, transform, load] >> cleanup. The cleanup task must run once every upstream task has reached a terminal state, whether that state is success, failed, or skipped. Set its trigger rule to all_done (give the trigger rule's string value).
Airflow's default trigger rule is all_success, which would skip cleanup as soon as any upstream task fails, so cleanup would not run during the failure cases it is meant to handle. The all_done trigger rule makes a task run once all of its direct upstream tasks have finished in any state (success, failed, or skipped), which is exactly the cleanup behavior required. The dependency list is what defines the upstream set, so the trigger rule alone determines when cleanup runs.
A pipeline DAG has tasks, each with a non-negative integer duration, and directed edges [upstream, downstream] meaning downstream cannot start until upstream has finished. Assume unlimited parallelism: every task starts as soon as all of its upstream tasks have finished.#
Starter code
def critical_path_length(durations, edges):
# TODO: compute the longest weighted path; return -1 if the graph has a cycle
return max(durations.values())
Your solution must pass
- simple chain extract-transform-load
- independent tasks take the longest one
This one is written and run, not read. Solve it in the app and your code is executed against these tests and the hidden ones.
In Apache Airflow 2.2+ (the data interval model), a DAG has schedule='0 2 * * *' (daily at 02:00 UTC). For the DAG run covering the data interval 2024-06-01T02:00 to 2024-06-02T02:00: what is its logical_date, and when does the scheduler typically start that run?#
Show answer
logical_date = 2024-06-01T02:00 (equal to data_interval_start). The scheduler starts the run at or after data_interval_end, i.e. 2024-06-02T02:00, once the interval has closed.
Since Airflow 2.2, each scheduled DAG run is defined by a data interval [data_interval_start, data_interval_end). The run's logical_date (the old execution_date) is set to data_interval_start, not to the moment the run executes. Because the interval must be complete before its data can be processed, the scheduler only creates and starts the run once the cron schedule fires at data_interval_end, which here is 2024-06-02T02:00 UTC. So the run labelled 2024-06-01 actually executes after the 2024-06-01 day's data has closed, which is why a daily job's logical date appears one period 'behind' its wall-clock start time.
Related interview questions
Job market
See data-engineering salaries and hiring demand from live job postings.
The other 5 questions
This page shows 10 and marks what you pick. That's as far as a page can go. A free account opens the other 5 and keeps every answer. What you miss comes back until it's right: after a day, then at longer gaps.
Free · the whole bank · 100 marked answers per 30 days · written feedback on the paid plan