Data Engineering Interview Questions: Data Lakes, Lakehouses & Medallion Architecture

A data lakehouse merges the data lake and the data warehouse. From the data lake it takes open-format, low-cost, scalable storage for structured, semi-structured, and unstructured data; from the data warehouse it takes ACID transactions, schema enforcement, and optimized query performance. The OLTP database and key-value store pairing, the message queue and stream processor pairing, and the data mart and operational data store pairing are not the paradigms a lakehouse is built from.

In the medallion architecture pattern, the Bronze layer is the first landing zone: it ingests data from source systems in its raw, untransformed form, preserving the original schema and payload for traceability. The Silver layer applies cleansing, deduplication, and conforming, and the Gold layer curates business-level aggregates and reporting-ready tables, so neither the Silver layer nor the Gold layer is the raw landing zone.

The medallion architecture names its three layers Bronze (raw landing zone), Silver (cleansed and conformed), and Gold (curated, business-level aggregates). Filling the blanks in that order gives Bronze, Silver, Gold, which runs from least to most refined.

In a lakehouse, the data files sit on cheap object storage as Parquet or similar files, and object storage alone offers no multi-file transactions. What adds ACID guarantees is a transactional metadata layer such as Delta Lake, Apache Iceberg, or Apache Hudi, which maintains commit logs, file-version tracking, and snapshot isolation so that readers and writers are isolated and commits are atomic. A columnar file format like Parquet only defines how bytes are laid out within a file, replication only provides durability, and a query optimizer only chooses execution plans, so none of those three provides transactions.

Object storage is immutable at the object level and provides no multi-object transactional guarantees, and Parquet is only a file format. The component that provides ACID behavior is a transactional metadata layer (Delta Lake, Apache Iceberg, or Apache Hudi) that keeps a commit log and snapshot metadata mapping each table version to its set of data files. New commits are added atomically, which gives snapshot isolation and rollback. Neither the storage layer nor the Parquet format alone provides these properties.

Bronze stores raw data as landed from the source with minimal transformation. Silver applies cleansing, deduplication, schema conformance, and integration across sources. Gold holds curated, business-level aggregates and metrics. The Silver layer's defining job is to produce a clean, conformed, enterprise-ready dataset, not raw landing data or final business aggregates.

In a medallion architecture, data flows from Bronze (raw landing in original format) to Silver (cleansed, deduplicated, conformed, and joined with dimensions) to Gold (aggregated business-level metrics). Raw clickstream events landed from Kafka are Bronze, deduplicated and type-cast records joined with a user dimension are Silver, and daily active user counts by region are Gold. The order therefore runs raw events, then silver records, then daily active user counts.

The medallion architecture (popularized by Databricks and Delta Lake) uses three named tiers in order of increasing quality: Bronze holds raw landing data with minimal transformation, Silver holds data that has been cleansed, deduplicated, and conformed to a common schema, and Gold holds curated business-level aggregates for analytics and BI. Each successive layer increases data quality and business readiness.

Open lakehouse table formats do not rely on a separate lock manager or row-level locking inside Parquet files. Instead, they maintain a metadata layer (Delta Lake's transaction log, Iceberg's catalog + manifest lists, Hudi's timeline) that records which data files make up each table snapshot. Writers perform atomic commits—often a single compare-and-swap operation on a metadata pointer—so that reads always see a consistent snapshot and concurrent writes are serialized, giving ACID guarantees without changing the underlying object store.

In a medallion architecture, data flows from bronze (raw landing of source data in original format) through silver (schema enforcement and deduplication, then conformed joins across sources with quality checks) to gold (business-level aggregations and curated reporting tables). Raw data must be ingested before it can be deduplicated and schema-enforced; deduplicated records must be conformed and joined before they can be aggregated into business-ready reporting tables. This bronze → silver → gold sequence is the core design principle of the medallion pattern.

Silver is the refinement layer between raw ingest and business consumption. It takes the raw data landed in Bronze and applies deduplication, schema conformance, data quality rules, and master-data alignment to produce a cleansed, trusted, and reusable dataset. Landing data in its original format with minimal transformation is the Bronze role. Building business aggregates and dimensional models for reporting is the Gold role, which builds on Silver. Access control and catalog metadata are governance concerns, not a medallion layer's responsibility.

The medallion architecture defines three layers: Bronze (raw ingest, minimal transformation, source fidelity), Silver (cleansed, deduplicated, schema-conformed, master-data aligned), and Gold (business-level aggregates, dimensional models, curated datasets for analytics and ML). Data flows Bronze to Silver to Gold, with each layer adding progressively more structure, quality, and business logic.

The formats rely on optimistic concurrency control: writers stage data files without blocking each other, then compete in an atomic commit to the metadata log, where exactly one commit wins per table version and the others retry after re-validating against the latest state. The atomic-commit guarantee depends on the storage layer offering put-if-absent or atomic rename semantics. Because metadata tracks conflicts at file, partition, or predicate granularity rather than per row, writers touching different rows in the same file collide and one must retry, which a row-locking warehouse would allow. That is the key trade-off: cheap non-blocking writes and simple scaling for batch and append workloads, at the cost of retry overhead and possible aborts under contention on overlapping files.

The Bronze layer is an immutable landing zone whose purpose is to capture raw source data with minimal transformation. Append-only writes preserve the full CDC event history for point-in-time replay, audit, and reprocessing. Allowing the schema to evolve (for example with mergeSchema) lets Bronze absorb additive upstream changes without blocking ingestion. Retaining out-of-contract and malformed records in raw form gives full-fidelity auditability and lets you reprocess from Bronze if downstream logic changes. Enforcing NOT NULL and referential integrity constraints belongs to the Silver layer, where data is cleansed, deduplicated, and conformed; applying them in Bronze would reject records at ingestion and break the capture-everything guarantee that makes Bronze a reliable replay source.

The bug is on line 7, deduped = bronze_df.dropDuplicates(["id"]). dropDuplicates keeps one row per id but does not guarantee which one, and it does not look at updated_at. When an entity has both an INSERT and a later UPDATE in the same batch, the retained row may be the older INSERT, and the MERGE then writes that stale state into Silver. The fix is to rank events per id by updated_at descending with row_number() over a window partitioned by id, keep rank 1, and merge only that row. The MERGE logic on lines 9 to 13 is correct once it receives the latest event per id.

The medallion architecture is a progressive refinement pipeline. Bronze is a raw ingestion layer that keeps source fidelity so data can be replayed and audited. Silver applies cleansing, deduplication, standardization, and conformance to produce a trusted enterprise dataset with consistent schemas. Gold reshapes data for specific analytical or operational consumers, often aggregating or denormalizing it for performance and business readability. Across the layers, quality and structure rise while data stays traceable: Delta Lake's transaction log provides ACID commits, versioning, and time travel, which support auditing and reprocessing of earlier layers.

The Bronze layer's role in a medallion architecture is to land source data as a faithful, unmodified copy for auditability and replay. Line 7 calls dropna(), which removes every row containing a null before the Bronze Delta table is written on line 9, so rows present in the source are permanently lost before they ever reach storage. Null handling and type conformance are Silver-layer cleansing concerns, which the pipeline already performs on lines 12–14. The other lines are appropriate: lines 3 reads the raw source, line 9 writes Bronze unmodified, lines 12–14 cleanse in Silver, line 15 writes Silver, line 17 aggregates for Gold, and line 18 writes Gold.

Delta Lake data files are immutable, so MERGE never edits rows inside an existing Parquet file. It writes new files holding the updated rows, in arrival order, and marks the old versions as removed in the transaction log. Those new files are not clustered on customer_id, so their min/max ranges cover most of the key space and cannot be skipped for a point lookup. As upserts accumulate, the share of well-clustered files shrinks and pruning degrades. Liquid Clustering addresses this by maintaining clustering keys on the table, and OPTIMIZE then clusters the newly written files incrementally rather than re-sorting the whole table. The other statements are wrong: MERGE does not rewrite files in place, ANALYZE TABLE only refreshes table statistics and does not change file layout, VACUUM only deletes unreferenced files and does not alter the clustering of active files, and Z-ORDER works on any sortable column, so cardinality is not the cause.

The root cause is that Gold's incremental logic relies on a timestamp high-water-mark over Silver's updated_at. Silver's MERGE is idempotent, so a replayed Bronze batch leaves Silver correct, and because updated_at comes from the source event time, the replayed rows carry timestamps at or before last_gold_ts. Gold's next run selects only rows with updated_at greater than last_gold_ts, so it skips those keys and the aggregate stays stale. The inconsistency is therefore a property of Gold's change-detection design: Gold is not a deterministic function of Silver's current state but of the sequence of changes it has already consumed.