RAG & Vector Search Interview Questions — AI Engineering Practice

HNSW and IVF are both approximate nearest-neighbor (ANN) structures: HNSW's greedy graph traversal can settle into a local optimum and miss a true neighbor reachable only through a path it didn't explore, and IVF only searches the nprobe clusters closest to the query, which can exclude a true neighbor sitting just across a cluster boundary. Both trade a small, tunable amount of recall (often 90-99%+ at reasonable settings) for search that's orders of magnitude faster than brute force at scale. Only a flat/brute-force index, which compares the query against every vector, is guaranteed exact — and that's exactly the cost ANN indexes exist to avoid.

RAG stands for Retrieval-Augmented Generation. The retrieval step fetches relevant passages from an external knowledge source (e.g., a vector database) and includes them in the prompt context so the model can ground its answer in up-to-date, source-specific information.

Cosine similarity measures the angle between vectors, so it is invariant to vector length — useful because embedding magnitude often carries little semantic meaning. When embeddings are L2-normalized, cosine similarity, dot product, and (inverse) Euclidean distance rank results identically, which is why many vector stores normalize on ingest and then use a plain dot product internally.

HNSW's graph-of-graphs structure is what most production vector databases (Qdrant, Weaviate, Pinecone, pgvector's HNSW mode, FAISS) default to or offer, because it consistently lands near the top of ANN recall/latency benchmarks and is comparatively simple to operate: no offline clustering pass and reasonable behavior even as the dataset grows.

The two foundational preprocessing steps in a RAG pipeline are chunking (dividing documents into manageable passages) and embedding (converting each passage into a dense vector). These vectors are what the vector database indexes and compares against the query vector at retrieval time.

The dot product of [3,4,0] and [3,0,0] is 33 + 40 + 0*0 = 9. The norm of the doc vector is sqrt(3²+4²+0²) = sqrt(25) = 5, and the norm of the query vector is sqrt(3²+0²+0²) = 3. Cosine similarity = 9 / (5 * 3) = 9/15 = 0.6. Rounding to 4 decimal places gives 0.6.

Cosine similarity is dot(a, b) / (||a|| * ||b||). The dot product is 1*2 + 2*0 + 2*1 = 4; ||a|| = sqrt(1+4+4) = 3 and ||b|| = sqrt(4+0+1) = sqrt(5) ≈ 2.2360679.... So 4 / (3 * 2.236...) ≈ 0.59628..., which rounds to 0.5963. This is the core ranking primitive behind vector search — the actual embeddings just have hundreds or thousands of dimensions, but the arithmetic is identical.

Embedding maps text → tokens → a single fixed-length vector that positions the text in a high-dimensional space, where proximity (cosine similarity / nearest-neighbour distance) approximates semantic similarity. Indexing stores those vectors; search embeds the query identically and finds the closest ones. The incompatibility is the key gotcha: each model trains its own space — different dimensionality and arbitrary, model-specific axes — so a query vector from one model has no meaningful geometric relationship to documents embedded by another. Always embed queries with the exact model + version used to build the index.

Ingestion runs first — documents are chunked, each chunk is embedded, and the vectors plus metadata are stored in the index. At query time you retrieve the most similar chunks for the question, then generate the answer with that context injected into the prompt. (A reranking step often sits between retrieve and generate, but the ingest-then-serve order is fixed.)

HNSW (Hierarchical Navigable Small World) builds a proximity graph in layers: the top layer has very few nodes and long-range edges for fast coarse traversal, and each layer below is progressively denser, down to a bottom layer containing every vector. A query starts at the top layer and greedily walks toward the nearest node, dropping down a layer at each local optimum, which gives roughly logarithmic search cost without scanning the whole dataset. Option (b) describes locality-sensitive hashing (LSH), a different ANN family that buckets by hash rather than traversing a graph. Option (c) describes a B-tree, which is built for one-dimensional ordered lookups and breaks down in high-dimensional space because there is no single sort key that preserves proximity. Option (d) describes a flat/brute-force index — exact, but linear in the number of vectors, which is exactly what HNSW exists to avoid at scale.

Euclidean distance earns its place when magnitude is meaningful (a) — normalizing away magnitude on purpose throws away a real signal your model actually encoded. It also matters when the index's internal machinery is built around L2 (b): several ANN structures and quantization schemes (e.g. product quantization) are formulated to minimize squared L2 error directly, so mixing in a different metric at query time can be inconsistent with what was optimized at index time. And it matters for consistency with other L2-native algorithms downstream (e) — if you're already clustering the same vectors with k-means, which minimizes within-cluster squared Euclidean distance, using a different metric for retrieval means "close" means two different things in the same system. (c) is false as a categorical claim: on L2-normalized vectors, dot product (which cosine reduces to) is typically the cheapest comparison, not L2. (d) is simply untrue — cosine similarity, like any dot-product-based metric, runs efficiently on GPUs; it is not a hardware limitation.

Both ef_search (HNSW) and nprobe (IVF) are query-time knobs, not index-build-time ones: ef_search controls how many candidates HNSW keeps in its search frontier while traversing the graph, and nprobe controls how many IVF clusters get scanned. Raising either explores more of the index per query — more graph neighbors visited, or more clusters searched — which recovers true nearest neighbors the search would otherwise have missed, at the direct cost of more distance computations and higher latency per query. Because they're read-time parameters, they can be tuned per query or per workload (e.g. higher ef_search for a batch/offline job, lower for a latency-sensitive live endpoint) against the same already-built index.

Line 9 embeds the query with text-embedding-3-small while the documents were indexed with text-embedding-3-large (line 1/4). Different embedding models produce vectors in different, incompatible spaces — they have different dimensionalities and geometry, so cosine similarity between a small-model query vector and large-model document vectors is meaningless. The query must be embedded with the exact same model (and version) used to build the index. The fix is to reuse DOC_MODEL on line 9.

IVF only searches inside the nprobe clusters closest to the query, so any true nearest neighbor sitting in cluster #2 or #3 (by centroid distance) is invisible to the search when nprobe = 1. That happens routinely for vectors near a cluster boundary, since k-means partitions space with hard boundaries that don't line up with any individual query's actual nearest neighbors. Raising nprobe searches more clusters and recovers those neighbors, at the cost of scanning more vectors and higher latency — this is IVF's core recall/latency knob, and it's a runtime query parameter, not something requiring a rebuild. Option (b) is wrong because k-means cluster boundaries are approximate partitions, not a guarantee that a vector's true nearest neighbors share its cluster. Option (c) invents behavior IVF doesn't have — a low nprobe degrades recall silently, it doesn't trigger a fallback. Option (d) confuses a query-time parameter with an insert-time one; nprobe has no effect on indexing/insert throughput.

Cosine similarity is dot(a, b) / (||a|| * ||b||). Once every vector has ||v|| = 1, that denominator is always 1, so cosine similarity collapses to a plain dot product — the two metrics rank candidates identically. That's why it's common practice to normalize vectors once at ingest time and then configure the index's metric as dot product (inner product), which is a cheaper single multiply-accumulate than cosine's extra division and norm lookups at every comparison. Option (b) is backwards: dot product is always defined, it's just numerically equal to cosine here. Option (c) is a category error — neither metric is inherently "more relevant"; correctness depends on whether the vectors are normalized and matches the model the embeddings were trained/evaluated with. Option (d) has the relationship backwards: normalizing is exactly what makes the two metrics converge, not diverge.

Each stored vector's memory cost and each distance computation's cost scale with dimension count, so cutting 1536 dimensions to 256 shrinks the index roughly 6x and speeds up every similarity comparison in proportion — a meaningful win at billion-vector scale, where memory is often the binding constraint. The cost is representational: a narrower vector has less room to separate fine-grained semantic distinctions, so recall/quality typically degrades somewhat, and how much depends on whether the smaller model (or truncated Matryoshka prefix) was actually trained to preserve the semantics you need in fewer dimensions. Option (b) is false — both memory and the cost of each distance computation scale with dimensionality, independent of vector count. Option (c) overgeneralizes a real but narrow effect (some dimensionality reduction can denoise) into a universal law; past a point, cutting dimensions removes real signal, not just noise. Option (d) is also false: computing a distance is itself O(dimensions) work per comparison, so wider vectors make every graph traversal step more expensive, not free.

cand_a = [5, 0] points in the exact same direction as query = [1, 0] but has 5x the magnitude, while cand_b = [1, 1] points in a different direction but is close by raw distance. Euclidean: dist(query, cand_a) = sqrt((1-5)^2 + 0^2) = 4, dist(query, cand_b) = sqrt(0^2 + (0-1)^2) = 1 — cand_b is closer (1 < 4), so nearest_euclidean = "B". Cosine: cos(query, cand_a) = 5 / (1 * 5) = 1.0 (identical direction, maximal similarity), cos(query, cand_b) = 1 / (1 * sqrt(2)) ≈ 0.707 — cand_a wins (1.0 > 0.707), so nearest_cosine = "A". The two metrics disagree because cand_a's large magnitude penalizes it under Euclidean distance but is invisible to cosine similarity, which only measures angle. This is exactly why the choice of distance metric isn't cosmetic: on unnormalized embeddings where magnitude carries no intended meaning, Euclidean distance can rank a semantically-aligned but larger-magnitude vector as farther away than a smaller, differently-directed one.

Line 3 queries the ANN index for only the top 5 candidates without the status = "published" filter, and line 4 then filters that fixed-size, already-final pool afterward (classic post-filtering). If most of the 5 nearest vectors happen to be unpublished, there's nothing left after filtering — the code never goes back to fetch more candidates. The fix is either to over-fetch a much larger unfiltered pool before filtering (e.g. topK: 200 before line-4's filter) so enough published matches survive to reach 5, or, better, to pass the filter into the index query itself (index.query(queryVector, { topK: 5, filter: { status: "published" } })) if the store supports native pre-filtering, so the ANN search only considers eligible vectors in the first place.

The core insight is that dense and sparse retrieval fail in complementary ways: dense embeddings generalize across paraphrases but blur exact identifiers, while sparse/BM25 nails exact terms but has no notion of meaning. Hybrid search runs both and combines the result lists rather than picking one. Reciprocal Rank Fusion is the standard combiner precisely because it sidesteps the score-normalization problem — cosine similarity and BM25 scores live on unrelated scales, so summing raw scores is meaningless, but summing 1/(k + rank) per list only needs each list's internal ordering, which is always comparable. A strong answer should land on: two complementary failure modes, running both retrievers, and a rank-based (not score-based) fusion method.

This is the standard two-stage retrieval architecture, and the reasoning an interviewer is probing for is the bi-encoder/cross-encoder distinction: bi-encoders trade precision for the ability to precompute and index document vectors once, which is required to search at scale; cross-encoders trade scalability for precision by letting the query and document attend to each other, which is only affordable over a small shortlist. A strong answer explains why the cross-encoder can't just replace the ANN stage outright (it has no fixed per-document representation to index) rather than just asserting that reranking 'improves quality.'

This is one of the most common production vector-search bugs: a team adds a metadata filter, sees good recall in testing against a broad filter, then watches recall silently collapse for a narrow one — because a fixed-size post-filtered pool starves once the filter's selectivity is high enough that few of the top candidates pass it.

The query is embedded first, matching the index's embedding model. Dense and sparse search run in parallel against the same query — one over vectors, one over terms — producing two independently-ranked candidate lists. Those lists are fused into a single ranking, most commonly with Reciprocal Rank Fusion, which combines rank positions rather than requiring the two very differently-scaled scores to be normalized against each other. Only then does the (expensive) cross-encoder reranker run, and only over that fused shortlist rather than the whole corpus, since it can't be precomputed or indexed. Finally the pipeline truncates to the top-k passages that actually fit the model's context budget.

ef_search (called ef in FAISS) is the parameter that controls how many candidates HNSW explores in the dynamic list during each layer traversal at query time. Increasing it widens the search frontier, improving recall at the cost of more distance computations and higher latency. ef_construction affects index build quality and the resulting graph structure, but once the index is built it does not change query-time behavior. M (max connections per node per layer) also affects graph connectivity but is set at build time; while a low M can limit achievable recall, the parameter you tune at query time to trade recall for latency is ef_search. ml (the level normalization factor) controls the exponential probability distribution for layer assignment and is not a query-time tunable.

Post-filtering a fixed top-20 pool against a filter that only 2% of the corpus passes means the expected number of surviving candidates is about 20 * 0.02 = 0.4 — recall collapses because there's structurally almost nothing left to return, independent of how good the underlying ANN search is. The direct fix is to make the pre-filter pool large enough relative to the filter's selectivity that the expected survivor count comfortably exceeds k (here, a few thousand candidates for a 2%-selective filter and k=10), or to use pre-filtering / filtered ANN search if the index supports traversing the graph while respecting the filter directly, which avoids over-fetching altogether. Option (b) treats this as a plain recall/latency knob, but a mild ef_search/nprobe bump does not multiply the survivor count by the ~50x needed to compensate for a 2%-selective filter on a fixed-size pool — the shortfall is structural, not a small recall gap. Options (c) and (d) address unrelated properties (metric choice, vector width) that have no bearing on why a fixed unfiltered pool starves under a selective filter.

This rubric evaluates whether the candidate can architect a RAG system that addresses the five hardest production concerns in a legal-domain setting: citation provenance, pre-retrieval access control, document versioning, retrieval quality at scale, and cross-lingual/multi-jurisdiction handling. Each criterion tests a distinct architectural decision; an answer that omits any one leaves a critical production gap. The sample answer satisfies all five: hierarchical chunking with paragraph metadata (c1), metadata pre-filtering at the vector store (c2), version increment with tombstoning and query-time filtering (c3), hybrid search + cross-encoder re-ranking + jurisdiction sharding + HNSW parameter tuning (c4), and multilingual embeddings with parallel multi-jurisdiction routing (c5).