Apache Spark Interview Questions

This distinction underlies almost everything else about Spark's performance model: because transformations only build up a plan rather than executing immediately, Spark can see and optimize the entire chain before any of it runs, and can pipeline compatible steps into fewer physical stages. Forgetting that transformations are lazy is a frequent source of confusion for engineers new to Spark, who sometimes expect a .filter() call alone to have already 'done something' before any action has been called.

Catalyst's pipeline begins with Analysis, where the Analyzer uses the catalog to turn an unresolved logical plan (containing 'UnresolvedRelation' and 'UnresolvedAttribute' nodes) into a resolved logical plan with concrete table and column references.

Catalyst's logical optimization phase is driven by a set of Rule objects that pattern-match on logical plan nodes and transform them. These rules (e.g., ConstantFolding, PushDownPredicate) are applied repeatedly until a fixed point is reached.

map is a narrow transformation: each output partition is derived from a single input partition, so Spark can compute it locally without moving data across executors. Wide transformations such as groupByKey or reduceByKey, by contrast, require a shuffle because an output partition depends on data from many input partitions.

An RDD's transformations are arbitrary user functions (a lambda passed to .map(), for instance) — Spark has no way to see inside that lambda, so it can only schedule and pipeline the stages, not optimize the logic itself. A DataFrame, in contrast, is built from a declarative, schema-aware API (select, filter, groupBy, expressed as an internal logical plan), which the Catalyst optimizer can analyze and rewrite before any of it runs — pushing filters down to the data source, pruning columns nobody reads, choosing a join strategy based on estimated sizes, and generating optimized JVM bytecode via whole-stage code generation (Tungsten). That's why equivalent DataFrame code is routinely faster than the RDD version doing the 'same' work by hand. RDDs still run on current Spark versions and are not deprecated (c is false), and both APIs execute distributed across the cluster's executors, not just the driver (b and d are both false) — the driver only coordinates and, for actions like collect(), receives results.

If Spark executed map, then filter, then select as three separate, immediately-run steps, each one would have to fully materialize its output before the next could start — reading and writing an intermediate result to memory/disk at every stage. By instead recording each transformation as a node in a lazy logical plan (a DAG) and only executing anything when an action forces a result, Spark gets to see the entire chain at once: the Catalyst optimizer can then rewrite that whole plan (e.g. pushing a filter below a join so less data flows through the expensive join), and the physical execution can fuse multiple narrow transformations that don't require a shuffle into a single pipelined stage, avoiding unnecessary materialization entirely. This is a live, load-bearing part of how Spark achieves its performance, not a legacy artifact (d is false), has nothing to do with JAR size (b), and Spark chains an arbitrary number of transformations in one lazy plan, not just one (c).

False. filter is a transformation, and transformations in Spark are lazy: calling df.filter(...) only adds a node to the DataFrame's logical plan describing 'filter this data by this predicate' — it does not touch the underlying data at all. Nothing actually scans, reads, or filters any data until an action (.show(), .count(), .collect(), .write(...), etc.) forces the accumulated logical plan to be optimized and executed. This is what allows Catalyst to see the entire chain of transformations before running any of it and optimize across the whole plan, rather than executing each step in isolation the moment it's called.

Catalyst is Spark SQL's query optimization framework: it takes the unresolved logical plan built from DataFrame/SQL operations, resolves references against the schema (the 'Analysis' phase), applies a rule-based optimizer to the resolved logical plan (predicate pushdown, constant folding, projection pruning, and more), then generates one or more candidate physical plans and picks among them (informed by statistics where available), before handing the chosen physical plan to Tungsten for whole-stage code generation and actual execution. It's specifically the optimization/planning layer, not serialization (b, which is closer to what Kryo/Tungsten's encoders handle), not an automatic caching mechanism (c — Spark never auto-persists a DataFrame; caching is always an explicit .cache()/.persist() call), and not cluster resource management (d, which is the job of the cluster manager — YARN, Kubernetes, or Spark's own standalone scheduler).

RDDs are opaque collections of objects with no schema information, so Catalyst cannot inspect their structure to optimize queries and Tungsten cannot apply its off-heap memory management or code generation. DataFrames are untyped but schema-aware (named columns with types), enabling full Catalyst + Tungsten optimization. Datasets (Scala/Java only) add compile-time type safety via encoders while retaining the same optimization pipeline. Python and R only have RDDs and DataFrames — no Datasets API.

Transformations in Spark are lazy: they only record metadata and build a lineage DAG (a). Actions such as collect() trigger the scheduler to execute all pending transformations in the lineage (b). Spark does NOT auto-cache intermediates — the developer must explicitly call persist() or cache() (c is false). Because lineage is preserved, a lost partition can be recomputed from its parent partitions via the DAG (d).

This flashcard tests a mid-level engineer's grasp of the three core Spark abstractions: the unoptimized, schema-less RDD; the schema-aware, Catalyst-optimized DataFrame; and the type-safe Dataset that bridges both via Encoders. Knowing that DataFrame is an alias for Dataset[Row] in Spark 2+ is the detail that separates surface-level recall from genuine understanding.

distinct, sortBy, filter, and map are lazy transformations that only build a lineage; take, count, and reduce are actions that each trigger independent execution of that lineage. For a: the original list is [3, 1, 4, 1, 5, 9, 2, 6]; distinct removes the duplicate 1, yielding {1, 2, 3, 4, 5, 6, 9}; sortBy arranges them as [1, 2, 3, 4, 5, 6, 9]; take(3) gives [1, 2, 3]. For b: filter(lambda x: x > 3) keeps [4, 5, 9, 6], so count = 4. For c: map(lambda x: x * 2) doubles each element of the original list → [6, 2, 8, 2, 10, 18, 4, 12]; reduce sums them → 6+2+8+2+10+18+4+12 = 62. The print statement outputs [1, 2, 3] 4 62.

All DataFrame operations before the prints are transformations—no data is processed until an action runs. step1 filters to rows with v > 1: ("b",2), ("c",3), ("a",4). step2 groups by k and sums v: a→4, b→2, c→3. step3 orders by k ascending: a, b, c. The first action step3.count() returns 3. The second action step3.collect() returns rows [(a,4),(b,2),(c,3)], yielding ['a4','b2','c3']. The third action step1.count() re-evaluates the filter lineage independently and returns 3.

Wide transformations are, by definition, exactly the operations whose output partition depends on data spread across multiple input partitions, which forces a shuffle to co-locate matching keys (a). reduceByKey's map-side combine is the textbook reason it beats groupByKey for aggregation: partial sums (or whatever the reduce function combines) are computed locally before anything crosses the network, so the shuffle itself moves far less data, whereas groupByKey ships every individual value across first and only combines afterward (b). coalesce() is specifically designed to reduce partition count cheaply by merging partitions already local to the same executors when possible, avoiding a full shuffle, while repartition() (used to either increase or decrease partition count) always performs a full shuffle to redistribute data evenly (c). spark.sql.shuffle.partitions genuinely defaults to 200 and directly controls the number of partitions produced by shuffle operations in DataFrame/SQL code (d). But shuffles are triggered by many operations beyond an explicit .repartition() call — groupBy, join, distinct, and orderBy all trigger one as a side effect of the operation itself, so (e)'s claim that only .repartition() can cause a shuffle is false.

.collect() moves the entire result set into the driver's single JVM heap, so a result that was perfectly fine sharded across dozens of executors can blow past the driver's memory the moment it's collected — a very common real-world OOM (a). Python UDFs cross the JVM/Python process boundary for every row, paying serialization cost each way, and because the UDF's logic is opaque Python bytecode, Catalyst can't push predicates through it or otherwise optimize around it the way it can with a built-in expr()-based function (b). Because Spark transformations are lazy, an uncached DataFrame reused in multiple downstream branches (say, two different aggregations off the same expensive joined result) is recomputed in full for each one, unless it's explicitly .cache()d or .persist()d after the expensive part (c). And failing to get a broadcast join for a large-vs-small join means Spark falls back to shuffling the large side across the network just to co-locate it with the small side's data, when broadcasting the small table to every executor (automatic below spark.sql.autoBroadcastJoinThreshold, or forced with a broadcast hint) avoids that shuffle entirely (e). spark.sql.shuffle.partitions is a tunable configuration property, not a hard architectural limit — 200 is only its default, and it's routinely tuned up or down based on data volume and cluster size, so (d) is false.

True. Since Spark 1.6's unified memory management model, spark.memory.fraction (default 0.6 of the heap after subtracting reserved memory) is a single pool shared between execution and storage, rather than two hard-partitioned regions as in the older static memory manager. Either side can borrow free space from the other under memory pressure — if storage isn't using its share, execution can use it for a large shuffle or sort, and vice versa — with one asymmetry: execution memory can forcibly evict cached (storage) blocks if it genuinely needs the space and none is free, since a computation failing outright is worse than losing a cache that can be recomputed, but storage cannot forcibly evict blocks that are actively part of a running task's execution memory. This borrowing is exactly why cranking cache usage too high can start causing tasks to spill or slow down — the two are competing for the same underlying pool, not isolated from each other.

Predicate pushdown and partition pruning both depend on Catalyst being able to understand the filter expression well enough to rewrite the physical scan — recognizing 'this is country == 'US'' lets it skip reading partitions whose directory-level value isn't US at all, and skip decoding column chunks a native Spark SQL filter (df.filter(df.country == "US")) makes fully visible to the optimizer. A Python UDF is opaque: Catalyst has no way to know what is_us computes, only that it's some black-box function to run per row, so it cannot push anything down or prune anything — it has to materialize and deserialize every row (paying the JVM↔Python serialization cost per row too) and evaluate the UDF against each one after the fact. Rewriting line 7 as df.filter(df.country == "US") — a native Spark SQL expression, not a UDF — restores both partition pruning and predicate pushdown. Partition pruning genuinely does work for native filters on partitioned Parquet tables (b is false), the output file format on line 8 has no bearing on how the input is read (c), and 'always reads every partition regardless of the filter' is exactly the bug this specific UDF introduces, not baseline Spark behavior (d).

This tests whether an engineer understands the mechanism, not just the buzzword: predicate pushdown only works when the optimizer can see and understand the filter well enough to hand it to the storage layer, and that visibility is exactly what a UDF destroys by being an arbitrary black-box function. It's a very common, subtle source of real production slowdowns — code that looks correct and passes tests, but silently reads 10-100x more data than necessary because a filter got expressed as a UDF instead of a native expression — which is why it's a strong senior-level probe.

Nothing can be optimized before Spark even knows what a column reference points to, so analysis — resolving names against the actual schema — has to happen before any rewriting. Only a schema-resolved plan can be safely rewritten by rule-based optimizations like predicate pushdown, since pushing a filter down requires knowing exactly which physical column and data source it refers to, which is why logical optimization comes after analysis, not before. Physical planning only makes sense once the logical plan is already in its final, optimized form — there's no point generating physical alternatives for a plan that's about to be rewritten anyway. Code generation needs one concrete, selected physical plan to compile against, and actual execution across the cluster can only happen once real generated code exists to run — so each phase is a strict prerequisite for the one after it.

A standard shuffle join partitions both sides by the join key's hash so that matching keys land on the same reducer, and every row with user_id = -1 — roughly 800 million of the 2 billion — is one such key. All of that data is forced onto a single partition/task no matter how many total shuffle partitions exist, while the other 199 tasks split the remaining, much smaller portion of relatively evenly-distributed keys. That's exactly why 199 finish quickly and one becomes an enormous straggler: it isn't slow because joins are inherently slow, it's disproportionately large because of the skewed key distribution, which is precisely what 'data skew' means. Because AQE (which has a runtime skew-join optimization that can automatically split an oversized partition) is explicitly disabled here for reproducibility, none of that automatic mitigation is available, and a manual fix is needed — the standard techniques are salting the skewed key (appending a random suffix to user_id = -1 on both sides and joining on the composite key, then aggregating away the salt afterward) or isolating and broadcasting/handling the skewed key's rows separately from the rest of the join. File format (b, c) has no bearing on a single-task straggler caused by key distribution — a CSV read/write would still hit the identical skew on line 4 — and a 100×+ runtime disparity on one task driven by a known, massively over-represented sentinel value is not 'normal variance' that resolves itself (d).

Data skew is one of the most-asked Spark performance topics in interviews precisely because it's a real, recurring production issue that's invisible from correctness testing — the query returns the right answer, it's just catastrophically slow — and diagnosing it from symptoms (a straggler task in the Spark UI, an executor OOM on one task while the rest are fine) is a genuine on-the-job skill. Salting and AQE's automatic skew-join splitting are the two standard, complementary answers: salting is the manual technique that works everywhere including older Spark versions or when AQE is disabled for reproducibility, while AQE's runtime detection is the modern default that needs no code change but requires accurate runtime shuffle statistics to trigger.

Spark 3.x executes every Dataset/DataFrame action through the same fixed Catalyst pipeline. The API calls first produce an unresolved logical plan (a). The Analyzer then binds unresolved attributes against the catalog to yield the analyzed logical plan (b). Catalyst's logical optimizer rewrites that plan with rule-based and cost-based optimizations to produce the optimized logical plan (c). The SparkPlanner then enumerates physical strategies and picks one, yielding the physical plan (d). Finally, the physical plan is lowered into an RDD DAG — with Tungsten whole-stage code generation fusing adjacent operators into generated Java bytecode — and submitted to the DAGScheduler for execution (e). This ordering is strict: analysis cannot run before a logical plan exists, logical optimization cannot run before attributes are resolved, physical planning requires an optimized logical plan, and RDD DAG generation requires a concrete physical plan. The key insight tying this to the RDD-vs-DataFrame-vs-Dataset topic is that Dataset and DataFrame are higher-level declarative abstractions whose user code never directly constructs RDDs; instead, Catalyst and Tungsten compile the declarative query down through these phases into an RDD DAG that the execution engine ultimately runs.

Three shuffles create three ShuffleMapStages plus one ResultStage = four stages. Boundaries are drawn at ShuffleDependency (wide) edges; narrow dependencies stay pipelined within a stage. Lazy evaluation means the lineage is only metadata until an action calls runJob, at which point the full DAG is visible and the scheduler can optimally partition it.

The Catalyst pipeline is a strict, linear chain. The Parser first produces an Unresolved Logical Plan (a) from the Dataset expression. The Analyzer then uses the catalog to resolve relations and columns, producing the Analyzed Logical Plan (b). The Catalyst optimizer applies rule-based rewrites to yield the Optimized Logical Plan (c). The SparkPlanner then converts this into a Physical Plan (d), selecting among physical strategies. Finally, whole-stage Tungsten code generation compiles the chosen physical plan into JVM bytecode that runs as RDD transformations (e). This ordering is fixed by Spark's QueryExecution sequence: analyzed → optimizedExecutedPlan → executedPlan, and the sub-steps (parse → analyze → optimize → plan → prepare/execute) are architecturally sequential with no reordering or parallelism. RDDs sit at the bottom of this stack as the execution substrate, which is why Dataset and DataFrame benefit from Catalyst optimization that RDDs bypass entirely.

In Spark's RDD API, the map transformation always sets the resulting RDD's partitioner to None, even when the input RDD has an explicit partitioner and the map function preserves the key. Spark makes no assumptions about whether map changes the key. Only mapValues and flatMapValues—which are contractually guaranteed to leave keys untouched—preserve the partitioner. On line 7, partitioned.map { case (k, v) => (k, v * 2) } produces an RDD with partitioner = None, even though the underlying data is physically still hash-partitioned by key into 200 partitions. Consequently, the join on line 14 cannot recognize transformed as co-partitioned with lookup and triggers a full shuffle of transformed's data. The fix is to replace map with mapValues(v => v * 2), which preserves the HashPartitioner(200) and makes the join shuffle-free.