Apache Airflow Interview Questions

@daily is a cron preset equivalent to 0 0 * * * (midnight every day). Airflow schedules a run at the end of each data interval, not the start: the run whose interval is 2026-01-01 00:00 → 2026-01-02 00:00 is only triggered once that interval has closed, i.e. just after midnight on 2026-01-02. This 'run after the interval ends' model is the single most common source of confusion for newcomers, who expect a @daily job to fire at the start of the day. The logical_date (formerly execution_date) of that first run is the interval start, 2026-01-01, which is why the templated {{ ds }} reads as the day the data belongs to rather than the day the task ran. Option c is wrong because unpausing a DAG with catchup enabled backfills from start_date rather than running 'now'.

retries sets how many times Airflow re-attempts a task instance after it fails before marking it failed for good — a per-task (or default_args) integer, so retries=3 allows three additional attempts. retry_delay (a timedelta) is the wait between attempts. By default that delay is constant; setting retry_exponential_backoff=True makes the wait grow after each failure (roughly doubling, capped by max_retry_delay), which is the standard pattern for transient failures against rate-limited or temporarily-overloaded external systems — backing off avoids hammering a struggling dependency. Retries only help when tasks are idempotent: a re-run must be safe to repeat, otherwise automatic retries can compound the damage of a partial first attempt.

The >> (right-bitshift) operator is overloaded in Airflow to mean 'set downstream'. Using a list on either side fans the dependency out or in: a >> [b, c] makes both b and c downstream of a, so they become eligible to run concurrently once a succeeds; [b, c] >> d makes d downstream of both, so by the default all_success trigger rule d waits for both to complete successfully. The result is a diamond. Option b describes a linear chain (a >> b >> c >> d), which is a different graph. The list syntax is purely sugar over repeated set_downstream calls and is the idiomatic way to express fan-out/fan-in without writing each edge by hand.

The clean mental model is a three-layer separation: Connections store the credentials/endpoint (in the metadata DB, keyed by conn_id), Hooks are the reusable client that reads a Connection and talks to the external system, and Operators are the DAG-level task units that often delegate to Hooks. The interview-critical detail is that Hooks pull credentials from Connections rather than hard-coding them, which is how Airflow keeps secrets out of DAG source. Calling a Hook directly inside a custom task is the idiomatic escape hatch when no off-the-shelf operator fits.

A task instance moves through distinct states driven by the separation between the scheduler and the executor. It starts with no status while waiting on upstream dependencies; once the scheduler sees those dependencies satisfied (and pool/concurrency limits allow), it marks the instance scheduled. The scheduler then hands it to the executor, which queues it; a worker eventually picks it up and the instance goes running; on a clean exit it becomes success, which is what unblocks downstream tasks. Understanding this pipeline matters operationally: a task stuck in scheduled usually points to exhausted pool slots or max_active_tasks, while one stuck in queued points to the executor or workers (e.g. no Celery workers available, or a Kubernetes pod that can't be scheduled). Knowing which component owns which state is how you debug a 'why isn't my task running' incident.

Sensors poll for a condition and have two classic modes. In poke mode (the default) the sensor sits in a running task slot for its entire wait, re-checking on poke_interval — simple, but many concurrent poke sensors can exhaust worker slots and deadlock the pool, so option d is false. In reschedule mode the sensor releases its slot between checks and is re-queued each interval, trading a little scheduling overhead for far better slot economy at scale (a). A timeout is essential so a sensor that will never be satisfied fails (and can alert) rather than hanging indefinitely (c). The modern option is deferrable sensors: they hand off the wait to the lightweight triggerer process via asyncio, freeing the worker entirely and resuming only when the trigger fires (e) — the most slot-efficient choice for long waits.

True. catchup controls whether the scheduler 'fills in' the schedule intervals that elapsed between start_date and the present. With catchup=True, unpausing a DAG whose start_date is two weeks back triggers a run for each missed interval (subject to max_active_runs), which is great for backfilling historical data but a notorious foot-gun: people set a months-old start_date, unpause, and accidentally launch hundreds of runs that hammer a database or external API. The defensive defaults are to set catchup=False so only the most recent interval runs going forward, and/or pin max_active_runs to throttle concurrency. Note that catchup=False does not run 'all missed intervals as one' — it simply skips them and starts from the latest completed interval. To replay a specific historical window deliberately, you use airflow dags backfill rather than relying on catchup.

The TaskFlow API (@task) makes XCom passing implicit: a task's return value is pushed to XCom, and passing make() as the argument to show() wires up the dependency and pulls that value back as the function argument at run time. So show receives the actual dict {"rows": 42} and payload["rows"] prints 42. Option c is the pre-TaskFlow mental model — with classic operators you'd push/pull XCom manually via ti.xcom_push/xcom_pull, but the decorator handles it for you. Option d misreads @task as async; it is an ordinary synchronous Python callable wrapped as an operator. The key insight is that calling a TaskFlow function in the DAG body does not execute it inline — it registers a task and returns an XComArg placeholder that resolves to the real value only when the task runs.

The bug is a violation of idempotency, the single most important property of an Airflow task. Airflow will run a task more than once for the same logical_date — on retry after a failure, on a manual re-run, or during a backfill — and a task must produce the same end state each time. This INSERT ... SELECT blindly appends the day's rows, so the second execution duplicates them. The fix is to make the write idempotent for the partition keyed by ds: delete-then-insert that day's partition, use INSERT OVERWRITE/MERGE/upsert, or write to an ds-scoped location you replace wholesale. Option b is wrong — ds is a valid context key (and execution_date is the deprecated name); option c misreads a safe parameterized query as injection; option d is wrong because **context is the standard way a task receives the runtime context. Designing every task to be safely re-runnable is what makes retries and backfills trustworthy.

logical_date marks the data interval a run owns (its start), while the task may execute much later — typically after the interval closes. The interview point is that tasks should be written against logical_date ({{ ds }}, {{ data_interval_start }}) instead of datetime.now(), because that decouples what data a run processes from when it happens to run. That decoupling is precisely what lets retries, manual re-runs, and backfills all process the identical window and stay idempotent. A strong answer connects logical_date to reproducibility/backfill safety; a weak one just defines the term without explaining why now() is dangerous.