Databricks Interview Questions — Practice Quiz with Real Questions

The traditional pattern was two-tier: a cheap data lake (object storage, any file format) held raw data, and a separate data warehouse held a curated copy for BI, because only the warehouse offered ACID transactions, schema enforcement, and fast SQL. That split meant paying to store and pipeline the same data twice, and the copies could drift out of sync. Databricks' lakehouse architecture uses Delta Lake as a transactional storage layer directly on top of cheap object storage (S3/ADLS/GCS), so a single copy of the data gets warehouse-grade ACID guarantees, schema enforcement, and governance (via Unity Catalog) while remaining open Parquet files usable by any engine — which is why it's pitched as 'one copy of data, many workloads' (BI, data engineering, ML) rather than a lake feeding a warehouse.

DBUs are the lever that makes cluster-type choice a real cost decision rather than just an operational one: the same instance type costs a different DBU rate depending on whether it's running as all-purpose, jobs, or SQL warehouse compute, which is why teams deliberately route scheduled production workloads to the cheaper job-cluster or serverless-SQL rate instead of leaving them on a shared interactive cluster.

Cluster autoscaling in Databricks dynamically adjusts the number of worker nodes in a cluster — scaling up when demand increases and scaling down when nodes are underutilized — so you do not have to manually pick a fixed cluster size.

Job clusters are ephemeral — Databricks creates them for a specific job run and terminates them when the run completes. All-purpose (interactive) clusters, by contrast, stay alive after work finishes so users can continue interacting with notebooks; they stop only when manually terminated or after the configured idle autotermination period elapses.

Databricks pricing distinguishes between all-purpose and job DBU rates. All-purpose clusters are priced at a higher per-DBU rate because they support interactive, multi-user sessions. Job clusters are priced lower per DBU since they serve single-owner automated workloads and terminate when the job finishes.

Unity Catalog's object hierarchy is metastore → catalog → schema → tables/views/volumes/models/functions. The catalog is the first layer of organization beneath the metastore and is the natural place to draw a hard isolation boundary — most commonly one catalog per environment (dev/staging/prod) or per business domain — with schemas nested inside a catalog grouping related tables the way a database groups tables in a traditional RDBMS. That three-level catalog.schema.table naming is what lets Unity Catalog apply consistent grants, lineage, and audit logging at any of the three levels, and lets the same schema name (e.g. sales) exist independently in both the dev and prod catalogs without colliding.

True. The default value of the delta.deletedFileRetentionDuration table property is 'interval 7 days' (168 hours), and Delta Lake's built-in retention-duration check actively blocks a VACUUM call from using a shorter interval unless that safety check is explicitly disabled. That default exists because old files can still be needed by a concurrent long-running reader, a still-in-flight time-travel query, or a streaming job that hasn't caught up yet — vacuuming too aggressively can pull the floor out from under any of those.

Data has to exist in a source system before anything ingests it, so source comes first. Bronze ingests it with as little transformation as possible specifically so it can be reprocessed later without re-hitting the source, which is why cleaning and deduplication happen one stage later, in silver, not during initial ingestion. Gold only makes sense once silver has already conformed and validated the data — aggregating raw, unvalidated bronze records straight into gold would bake in whatever data-quality problems silver exists to catch. Consumption comes last because BI tools and ML training are built to read the already-aggregated, read-optimized gold layer, not to repeat gold's aggregation logic themselves.

Nothing happens until something starts the run, so a trigger has to come first. Because a job cluster is ephemeral rather than already running, provisioning happens next, before any task can execute. Tasks then run according to the job's dependency graph — a downstream task can't start before the upstream task it depends on finishes. Only once every task in the DAG has completed does the cluster have nothing left to do, which is when it terminates automatically to stop accruing DBU cost. Run history, logs, and alerting are populated from what happened during execution, so they're only available to review after the run — and its cluster — are done.

Autoscaling in Databricks requires you to set both a minimum and maximum worker count, and it is supported on both all-purpose and job clusters — so options a and b are correct. The minimum worker count must be at least 1, so a cluster cannot scale down to zero nodes; option c is wrong. Databricks does not offer a single universal autoscaling mode — available modes and their behavior differ by cloud provider and cluster type, so option d is also wrong.

A DBU is the fundamental billing unit for Databricks compute. The total compute cost is driven by the instance type's DBU rate, the number of hours the cluster runs, and the workload-specific pricing tier (All-Purpose, Jobs, Photon, etc.). Cloud infrastructure costs (the underlying VMs) are billed separately by the cloud provider.

Delta Lake's actual mechanism is a transaction log of JSON/Parquet commit entries layered over standard Parquet data files: that log is what gives ACID transactions (a), lets you time-travel to a prior committed version (b), and lets the engine check incoming writes against the recorded schema, either enforcing it strictly or evolving it when explicitly allowed (c). It is not a proprietary binary format — the underlying files are still open Parquet, and Delta Lake is an open-source, Linux Foundation project readable by any engine with a Delta connector, so (d) is a common but false 'vendor lock-in' misconception. And Delta Lake has no automatic row-uniqueness guarantee (e): duplicates are entirely possible unless a pipeline explicitly enforces them with a MERGE upsert or a check constraint — there's no implicit primary key.

Job clusters exist for the lifetime of one job run: they spin up automatically when the run starts, so there's no pre-provisioning idle cost, and they terminate automatically the moment the run finishes (a); Databricks' optimized autoscaling used by job clusters can shrink the cluster after around 40 seconds of underutilization, a much shorter window than the interactive idle timeouts typical of all-purpose clusters (b), which are intentionally kept up and shared for exploratory notebook work by multiple users until someone stops them or the idle timeout fires (d). The DBU rate runs the other way from (c): jobs compute is billed at a lower rate than all-purpose compute for equivalent instance types, precisely because all-purpose adds interactive/collaboration overhead — so all-purpose being cheaper is a real but false assumption teams sometimes make. And (e) is false: a job task can absolutely run a notebook on a job cluster; notebook tasks are one of the most common task types in Databricks Workflows/Lakeflow Jobs.

False. The default trigger behavior is micro-batch processing: Spark runs the next micro-batch as soon as the previous one finishes, but it is still batching records together, not handling them one at a time. Continuous, record-at-a-time processing is a distinct execution mode you have to opt into explicitly with .trigger(continuous = "1 second"), and it's far more restrictive — it supports only a limited subset of operations (no most aggregations, for example) and remains an experimental mode, so the overwhelming majority of Structured Streaming pipelines, including everything running with .trigger(availableNow = True) for incremental batch-style ingestion, run on the default micro-batch engine.

The medallion architecture is a data-quality staging convention, not a Databricks-specific feature — bronze preserves raw source fidelity for reprocessability, silver conforms and cleans it into a trustworthy shared representation, and gold shapes it for a specific consumer (BI, ML features, a reporting mart). The reason it matters in interviews and on the job alike: without the layering, a schema change or a bad transformation upstream forces a full re-extraction from the source system to fix, and there's no clean boundary for who owns what quality guarantee. With bronze preserved, silver and gold can always be rebuilt from data already sitting in the lakehouse.

MLflow's value is giving every stage of the ML lifecycle — experimentation, packaging, and deployment governance — a consistent, tool-agnostic interface, so a model trained by one engineer can be evaluated, versioned, and promoted by someone else without reverse-engineering how it was built. Registering models in Unity Catalog rather than the legacy workspace-scoped registry is the current recommended path specifically because it puts model artifacts under the same access-control and lineage machinery as the tables that fed them, closing a governance gap that used to exist between data and models.

Enhanced autoscaling is designed to react quickly to increased load by adding nodes in exponentially growing steps, while it scales down conservatively—removing one node at a time—to avoid prematurely shedding capacity that may still be needed. Standard autoscaling (Azure) behaves differently and more gradually in both directions.

Driver DBUs = 1.5 × 7 × 1.3 = 13.65. Worker DBUs = 2.0 × 4 × 7 × 1.3 = 72.8. Total DBUs = 13.65 + 72.8 = 86.45. Cost = 86.45 × $0.55/DBU = 47.5475, which rounds to 47.55. The Photon multiplier of 1.3 is applied to both driver and worker DBU consumption before multiplying by the per-DBU rate.

Total DBUs = (1.0 + 1.5 × 4) × 8 = 7.0 × 8 = 56.0. All-Purpose cost = 56.0 × $0.55 = $30.80. Jobs Compute cost = 56.0 × $0.15 = $8.40. Difference = $30.80 − $8.40 = $22.40. This illustrates the cost premium of running interactive All-Purpose Compute versus scheduling the same workload as a job, which is a key Databricks cost-optimization decision.

A checkpoint has to persist across runs to do its job: it's how Auto Loader (and Structured Streaming generally) tracks which source files it has already processed, so a restart resumes instead of starting over. /tmp on a Databricks cluster is local disk on that specific VM, and since this pipeline runs on a job cluster that's provisioned fresh and torn down every run, that local checkpoint directory never survives to the next run — every run looks like the first run ever, so every previously-ingested file gets read and appended to bronze.orders again, silently duplicating rows. The fix is a checkpoint location on durable cloud storage the next run's cluster can also see (a DBFS/Unity Catalog volume path or a cloud storage URI), not local disk. .trigger(availableNow=True) is valid and simply means 'process everything currently available, then stop' (b is wrong), cloudFiles (Auto Loader) is the correct, current API for incremental file ingestion including JSON (c is wrong), and checkpointing is required for exactly-once source tracking on any streaming read, aggregating or not (d is wrong).

The core argument for job clusters is cost and blast-radius isolation: they bill at a lower DBU rate, terminate automatically so there's no idle spend, and one job's failure or resource spike can't take down other people's work on a shared always-on cluster. What's given up is start-up latency (a job cluster provisions from cold on every run, unlike an already-warm shared cluster) and some operational simplicity when many small jobs each spin up separately rather than sharing one cluster's resources — which is exactly the kind of tradeoff a senior engineer is expected to weigh explicitly rather than defaulting to 'always-on is simpler.'

Enhanced Autoscaling leverages Spark's graceful decommissioning feature. When the autoscaler selects a node for removal, it marks the executor for decommission, which prevents new tasks from being assigned. In-flight tasks are allowed to complete, and shuffle blocks are migrated to surviving executors before the node is actually terminated. Standard Autoscaling lacks this mechanism, so removing nodes that hold shuffle data causes those files to be lost, triggering expensive stage recomputation. This is why Enhanced Autoscaling can scale down more aggressively without risking data loss or job failures.

VACUUM deletes data files that are no longer part of the current table version, once they're older than the configured retention window — but older historical versions (and any query still reading them) depend on exactly those files staying in place until that window passes. Line 1 turns off Delta Lake's built-in guard against unsafe short retention, and line 2 then immediately deletes anything untracked by the current version, with zero grace period. That's why the downstream reconciliation query on line 3, reading version 42, starts intermittently throwing a file-not-found error: whether it fails depends on the race between when VACUUM runs and when that read happens. VERSION AS OF is valid syntax and VACUUM does accept RETAIN n HOURS, so (b) and (c) misdiagnose the syntax; and Databricks' own documentation is explicit that shortening the retention interval below the 7-day default is unsafe for exactly this reason, so (d) is false — VACUUM absolutely can, and does, remove files a concurrent reader or an old snapshot still needs once you disable the safety check.

In the Databricks DBU cost model, the DBU dollar rate ($/DBU) is identical whether the underlying cloud instances are spot or on-demand. Spot instance savings come exclusively from the cloud provider's discounted infrastructure (VM) pricing — the per-DBU charge from Databricks does not change. The buggy line dbu_dollar_rate = dbu_dollar_rate * (1 - spot_infra_discount) incorrectly reduces the DBU dollar rate, which would understate the Databricks portion of the cost. The correct approach is to compute DBU cost at the full DBU rate and apply the spot discount only to the separate infrastructure cost line item. Option (a) is wrong because the driver node does consume DBUs. Option (c) is wrong because DBU rates are billed per hour (prorated per second), not solely per second. Option (d) is wrong because spot instances have no effect on DBU consumption or rate at all.

Delta Lake's OCC protocol begins by recording the current table version as the transaction's read snapshot (b). The transaction then performs its reads and writes, tracking file-level actions and read predicates for later conflict detection (c). Before committing, the transaction checks whether its starting version is still the latest (e). If a newer commit has landed, the transaction runs conflict resolution against the intervening commits to detect read-write or write-write conflicts (a). If no conflict is found — or there were no intervening commits — the transaction atomically commits its changes as the next version in the delta log (d).