Embeddings interview questions — AI Engineering practice quiz
Reviewed by Mark Dickie · Last updated
Embeddings are dense vector representations of text, images, or other data, produced by a model so that semantically similar inputs land near each other in a high-dimensional space. For an AI Engineering interview, you should know how embeddings are generated from transformer encoder layers, what distance metrics (cosine, dot product, Euclidean) reveal about similarity, and how dimensionality trades off between expressiveness and storage cost. Expect questions on chunking strategies, retrieval-augmented generation pipelines, and common failure modes like anisotropy or domain mismatch between the embedding model and your corpus.
What does an embeddings interview question test?
Most questions fall into a few buckets: the math behind vector similarity, the mechanics of embedding models, and the engineering decisions around building retrieval systems. The table below maps the core areas you should prepare.
| Area | What gets asked | Example focus |
|---|---|---|
| Vector math | Cosine vs. dot product vs. L2 distance | When does normalization matter? |
| Model architecture | How encoder layers produce embeddings | Pooling strategies (CLS, mean, max) |
| Dimensionality | Choosing embedding dimensions | Trade-off between storage and recall |
| Retrieval pipeline | Chunking, indexing, ANN search | HNSW vs. IVF, chunk size tuning |
| Evaluation | Measuring retrieval quality | Recall@k, NDCG, MRR |
| Failure modes | Anisotropy, domain shift, hubness | Why do dissimilar items sometimes score high? |
How should I prepare for embeddings questions?
- Work through the math by hand: compute cosine similarity between two small vectors, and verify what happens when you normalize them first.
- Compare two embedding models on the same corpus (for example,
text-embedding-3-smallvs.all-MiniLM-L6-v2) and measure how their nearest-neighbor rankings differ. - Build a minimal RAG pipeline with a vector store so you can talk through chunking, indexing, and retrieval end to end.
- Study the evaluation metrics that hiring teams actually use — recall@k and NDCG come up frequently because they connect embedding quality to downstream task performance.
- Read at least one paper on embedding failure modes (anisotropy in pretrained models is a common topic) so you can discuss limitations, not just the happy path.
The quiz below draws from real interview questions on these topics. Work through it to find gaps before you walk into the room.
Key facts
- Tarmac has 26 AI Engineering interview questions on this topic, 25 of them on this page, at difficulty 1–5 of 5.
- Tarmac tracked 4,175 job postings asking for AI Engineering in August 2026.
- Roles asking for AI Engineering advertise a median base salary of US$180,000, across 757 job postings as of August 2026.
- Tarmac last reviewed these AI Engineering interview questions on 31 August 2026.
At a glance
| Questions | 25 shown · 26 in the bank |
|---|---|
| Difficulty | 1–5 of 5 |
| Formats | Multiple answer, Fill in the blank, Multiple choice, Short answer, Flashcard, Design exercise, True / false, Code output, Find the bug, Ordering |
What you'll review
- embeddings
- llm foundations
- rag basics
- vector search
Practice questions
Try one before you open the answer. Pick an option and press Check; it's marked on the spot.
AI Engineering/llm-foundations/embeddings
Which of the following statements are true about text embeddings used in LLM applications? Select all that apply.#
Options
Pick every one that applies.
Show answer
Text embeddings represent text as vectors of real numbers and enable similarity comparisons between texts using distance metrics such as cosine similarity. They do not store original text verbatim, nor do they guarantee identical vectors for similar inputs — similar texts produce similar but distinct vectors.
Text embeddings encode text as dense vectors of real numbers (option a), and these vectors can be compared using distance metrics like cosine similarity to measure semantic similarity between texts (option c). Option b is false because embeddings are compressed numerical representations, not verbatim text stores. Option d is false because even very similar inputs produce different (though close) vectors, not identical ones.
AI Engineering/llm-foundations/embeddings
Embeddings are frequently compared by measuring the _____ between their vectors, which captures the angle between two vectors while being insensitive to their magnitudes.#
Show answer
Embeddings are frequently compared by measuring the cosine similarity between their vectors, which captures the angle between two vectors while being insensitive to their magnitudes.
Cosine similarity measures the cosine of the angle between two vectors, producing a value in [-1, 1]. Because it only considers direction (not magnitude), it is the standard metric for comparing embedding similarity, where the semantic meaning of a text is encoded in the direction of its vector rather than its length.
AI Engineering/llm-foundations
During autoregressive text generation, what does a decoder-only LLM (e.g., GPT-style) explicitly output at each step that is then fed back as input for the next step?#
Options
Show answer
An autoregressive (decoder-only) LLM outputs a probability distribution over its entire vocabulary at each step. One token is sampled from that distribution, appended to the running context, and fed back in so the model can produce the next-step distribution — repeating until a stop condition is met.
A decoder-only LLM applies a softmax over its vocabulary head at the final position, producing a probability distribution. The sampler picks one token from that distribution, appends it to the context, and the model re-runs (or caches forward through) to produce the next-step distribution. It does not output a single embedding, generate tokens in parallel, or emit a loss value during inference.
AI Engineering/llm-foundations
Modern decoder-only LLMs such as GPT and Llama are built on the Transformer architecture. The core mechanism that lets every token attend to all previous tokens in the sequence — computing weighted relationships without recurrence — is called _____.#
Show answer
Modern decoder-only LLMs such as GPT and Llama are built on the Transformer architecture. The core mechanism that lets every token attend to all previous tokens in the sequence — computing weighted relationships without recurrence — is called self-attention.
Self-attention (often implemented as multi-head scaled dot-product attention) is the defining mechanism of the Transformer. It computes a weighted sum of value vectors for every token pair, enabling each position to incorporate information from all other positions in a single forward pass — replacing the sequential recurrence of RNNs/LSTMs.
AI Engineering/rag/rag-basics
What is retrieval-augmented generation (RAG) and what problem does it solve over just prompting the model?#
Show answer
RAG retrieves relevant documents from an external knowledge source (usually via a vector search over embeddings) and injects them into the prompt as context, so the model answers grounded in that retrieved data. It solves the problem that a model's parametric knowledge is frozen at training time and can hallucinate — RAG lets you supply fresh, private, or domain-specific facts at query time without retraining, and lets you cite sources.
RAG decouples knowledge from weights: the model stays general, while a retrieval layer (embed the query, search a vector store, pass the top hits as context) supplies current and proprietary facts. The big wins are freshness, the ability to cite sources, and reduced hallucination — but retrieval quality becomes the bottleneck, so chunking, the embedding model, and reranking matter as much as the LLM itself.
AI Engineering/rag/vector-search
Vector search ranks documents by similarity to the query embedding. The most common metric, which measures the angle between two vectors and ignores their magnitude, is _____ similarity.#
Show answer
Vector search ranks documents by similarity to the query embedding. The most common metric, which measures the angle between two vectors and ignores their magnitude, is cosine similarity.
Cosine similarity measures the angle between vectors, so it is invariant to vector length — useful because embedding magnitude often carries little semantic meaning. When embeddings are L2-normalized, cosine similarity, dot product, and (inverse) Euclidean distance rank results identically, which is why many vector stores normalize on ingest and then use a plain dot product internally.
AI Engineering/llm-foundations/embeddings
In the context of LLM embeddings, why is cosine similarity often preferred over raw dot product when comparing two embedding vectors?#
Show answer
Cosine similarity measures the angle between two vectors, so it is invariant to each vector's magnitude. Because LLM embeddings are often not length-normalized, dot product magnitude can vary with vector norm and produce misleading similarity scores. Normalizing the vectors and using cosine similarity (or equivalently the dot product of unit vectors) gives a stable, scale-independent measure of semantic closeness.
This flashcard tests understanding of a foundational concept in embedding-based retrieval: cosine similarity is magnitude-invariant, making it more robust for comparing embeddings whose norms may vary due to the model or lack of normalization.
AI Engineering/llm-foundations/embeddings
Modern text embedding models (e.g., OpenAI text-embedding-3-small, sentence-transformers all-MiniLM-L6-v2) typically output vectors that are L2-normalized to unit length. For two such unit-length embedding vectors u and v, which statement is correct?#
Options
Show answer
When embedding vectors are L2-normalized to unit length, the dot product equals the cosine similarity exactly, because the cosine formula (u·v)/(‖u‖·‖v‖) has a denominator of 1. Both range from −1 to 1. This is why many vector databases let you use either inner-product or cosine distance interchangeably when the stored embeddings are pre-normalized.
When both vectors are L2-normalized, ‖u‖ = ‖v‖ = 1. The cosine similarity formula is cos(u, v) = (u·v) / (‖u‖·‖v‖). With unit-length vectors the denominator becomes 1, so the dot product equals the cosine similarity exactly. Because each component can be negative, the dot product (and thus cosine similarity) ranges from −1 (opposite direction) to 1 (identical direction). Option (b) is wrong because dot product and Euclidean distance are different functions even for unit vectors. Option (c) is wrong because the range is −1 to 1, not 0 to 1. Option (d) is wrong because ‖u‖·‖v‖ = 1 for unit vectors, not 2.
AI Engineering/llm-foundations/embeddings
You are building a customer-support knowledge-base search for a SaaS company. Users type natural-language questions into a help widget, and you want to retrieve the most relevant articles from a corpus of ~50,000 support docs. Design an end-to-end embedding-based semantic retrieval system. Cover: how you embed the query and documents, how you store and index the vectors for fast retrieval, how you turn retrieved vectors into a ranked list of article results, and one concrete approach you would use to evaluate whether the embedding model is producing good results. Keep the design practical and name specific components.#
Show answer
I would use a pre-trained text embedding model (e.g., text-embedding-3-small or a Sentence-BERT model) to embed both the user's query and every support document into the same dense vector space. For long articles that exceed the model's context window, I would split each document into overlapping chunks (e.g., 512 tokens with 50-token overlap) and embed each chunk independently, keeping a reference back to the parent article ID.
The embeddings would be stored in a vector database such as pgvector (if we are already on PostgreSQL) or a managed service like Pinecone. Each stored vector record would include the chunk text, the parent document ID, the document title, and a URL, so that after similarity search we can return the full article to the user. The index would use approximate nearest neighbor (ANN) search (HNSW in pgvector/Pinecone) so that retrieval stays in the tens of milliseconds even at 50,000 vectors.
At query time I embed the user's question with the same model, then issue a top-k ANN query against the vector index using cosine similarity as the distance metric. The database returns the k most similar chunk vectors along with their metadata. I deduplicate by parent document ID (keeping the highest-scoring chunk per article) and return the resulting ranked list of articles to the help widget.
To evaluate the embedding model, I would build a small labeled dataset of 200–300 real user queries each annotated with the set of known-relevant article IDs (sourced from support-ticket resolutions or manual labeling). I would run each query through the retrieval pipeline and compute Recall@5 and nDCG@5 against the labeled ground truth. I would run the same evaluation on a BM25 keyword-search baseline and on at least one alternative embedding model, then select the configuration that maximizes Recall@5 while keeping latency acceptable.
This design exercise tests applied knowledge of embedding-based retrieval at a foundational level. The rubric covers the four core pillars a junior-to-mid AI engineer must address: (1) embedding both queries and documents in a shared space with proper chunking, (2) storing vectors in a vector database with metadata, (3) ranking by cosine similarity, and (4) evaluating with standard IR metrics against a labeled dataset and baseline. Difficulty 2 is appropriate because these are well-established, widely-documented practices with no deep ambiguity.
AI Engineering/llm-foundations
In the scaled dot-product attention used by transformer-based LLMs, the query–key dot product is divided by the square root of _____ before the softmax is applied. This scaling prevents the dot products from growing large in magnitude, which would otherwise push the softmax into regions with extremely small _____.#
Show answer
In the scaled dot-product attention used by transformer-based LLMs, the query–key dot product is divided by the square root of d_k before the softmax is applied. This scaling prevents the dot products from growing large in magnitude, which would otherwise push the softmax into regions with extremely small gradients.
Scaled dot-product attention computes Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. The factor √d_k — the square root of the key (head) dimension — counteracts the fact that dot products of high-dimensional vectors have large variance, which would push the softmax into saturation (outputs near 0 or 1) and consequently produce vanishingly small gradients during backpropagation, making training unstable.
AI Engineering/llm-foundations
In an autoregressive LLM that samples tokens via softmax over logits, what is the effect of setting the temperature parameter to a value approaching 0 (e.g., 0.01)?#
Options
Show answer
Setting temperature near 0 makes the softmax distribution increasingly peaked on the highest-logit token, which is effectively greedy (argmax) decoding. The model still responds differently to different prompts, but the sampled token becomes almost entirely deterministic.
Temperature divides each logit by T before softmax. As T → 0, the largest logit dominates exponentially more strongly, so the distribution concentrates virtually all probability mass on the single highest-scoring token — behavior equivalent to greedy (argmax) decoding. The model's output still depends on the prompt (ruling out option a); temperature 1 means no scaling (ruling out option c); and near-zero temperature makes output deterministic, not random (ruling out option d).
AI Engineering/llm-foundations/embeddings
In a semantic search system, you embed a query and compare it to document embeddings with cosine similarity. What does cosine similarity actually measure?#
Options
Show answer
Cosine similarity measures the angle between two vectors, ignoring their magnitudes. It is the dot product divided by the product of the magnitudes, so it depends only on direction, not length. That is why it suits embeddings: two passages on the same topic point the same way regardless of how long each text is. It is not Euclidean distance, shared-dimension counting, or raw token overlap.
Cosine similarity is the cosine of the angle between two vectors: it is the dot product divided by the product of the magnitudes, so it depends only on direction, not length. That is why it works well for embeddings — two passages about the same topic point the same way regardless of how long each text is. Euclidean distance (a) is a different metric that is sensitive to magnitude (though on length-normalized vectors the two ranking orders coincide). "Shared dimensions" (c) is not a real similarity measure for dense embeddings, whose dimensions are not independently interpretable. Token overlap (d) describes lexical/keyword matching (e.g. BM25), which is exactly what dense embeddings are meant to go beyond.
AI Engineering/llm-foundations/embeddings
High cosine similarity between two text embeddings reliably means the two texts have the same factual answer or intent.#
Options
Show answer
False. Cosine similarity measures topical and semantic proximity in the embedding space, not truth or intent equivalence. "How do I cancel my subscription?" and "How do I upgrade my subscription?" sit very close yet need opposite answers, and a negation like "the drug is safe" versus "the drug is not safe" can embed nearly identically. This is why retrieval pipelines add a reranker and evaluate with answer-grounded metrics rather than raw similarity alone.
Cosine similarity measures topical/semantic proximity in the embedding space, not truth or intent equivalence. "How do I cancel my subscription?" and "How do I upgrade my subscription?" sit very close together yet need opposite answers; a negation ("the drug is safe" vs "the drug is not safe") can also embed nearly identically. This is exactly why retrieval-augmented pipelines add a reranker and why you should evaluate retrieval with answer-grounded metrics, not raw similarity alone.
AI Engineering/llm-foundations/embeddings
This computes the cosine similarity between two embedding vectors and prints it rounded to 4 decimals. What does it print?#
import math
a = [1, 2, 2]
b = [2, 0, 1]
dot = sum(x * y for x, y in zip(a, b))
norm_a = math.sqrt(sum(x * x for x in a))
norm_b = math.sqrt(sum(x * x for x in b))
print(round(dot / (norm_a * norm_b), 4))Show answer
0.5963
Cosine similarity is dot(a, b) / (||a|| * ||b||). The dot product is 1*2 + 2*0 + 2*1 = 4; ||a|| = sqrt(1+4+4) = 3 and ||b|| = sqrt(4+0+1) = sqrt(5) ≈ 2.2360679.... So 4 / (3 * 2.236...) ≈ 0.59628..., which rounds to 0.5963. This is the core ranking primitive behind vector search — the actual embeddings just have hundreds or thousands of dimensions, but the arithmetic is identical.
AI Engineering/llm-foundations/embeddings
Walk through what happens to a piece of text when you 'embed' it for semantic search, and explain why a search index built with one model cannot be queried using a different model's output.#
Show answer
The text is tokenized, then run through an embedding model that maps it to a fixed-length vector of floating-point numbers — a point in a high-dimensional space where semantically similar texts land close together (measured by cosine similarity or distance). At index time you store each document's vector; at query time you embed the query the same way and find the nearest vectors. Two different models can't be mixed because each learns its own space: they often differ in dimensionality, and even at the same size the axes and geometry are arbitrary and model-specific, so a vector from model A has no meaningful distance to a vector from model B. The query must use the exact same model (and version) that built the index.
Embedding maps text → tokens → a single fixed-length vector that positions the text in a high-dimensional space, where proximity (cosine similarity / nearest-neighbour distance) approximates semantic similarity. Indexing stores those vectors; search embeds the query identically and finds the closest ones. The incompatibility is the key gotcha: each model trains its own space — different dimensionality and arbitrary, model-specific axes — so a query vector from one model has no meaningful geometric relationship to documents embedded by another. Always embed queries with the exact model + version used to build the index.
AI Engineering/llm-foundations
In a decoder-only transformer (e.g., GPT-style architecture), what is the primary purpose of causal (masked) self-attention during training?#
Options
Show answer
Causal masking in decoder-only transformers ensures each token attends only to itself and earlier positions, preserving the autoregressive property so predictions never depend on future tokens. It prevents a train/inference mismatch but does not reduce O(n²) attention complexity, address gradient issues, or serve as regularization.
Causal masking sets the attention scores for all future positions to negative infinity (zero after softmax), so each token at position i can only attend to positions 0 through i. This preserves the autoregressive property: during training, the model never 'sees the future,' which matches the left-to-right generation used at inference time. Without causal masking, there would be a train/inference mismatch. Causal masking does not change the O(n²) complexity of attention, does not address vanishing gradients (that is the role of residual connections and layer norm), and is not a form of regularization like dropout.
AI Engineering/llm-foundations
During autoregressive inference in transformer-based LLMs, the _____ cache stores the projected key and value tensors from all previously processed token positions, allowing each newly generated token to attend to the full context without re-running earlier tokens through the attention layers. This reduces the per-step cost of generating one additional token from O(n²) (full recompute) to O(n) in sequence length.#
Show answer
During autoregressive inference in transformer-based LLMs, the KV cache stores the projected key and value tensors from all previously processed token positions, allowing each newly generated token to attend to the full context without re-running earlier tokens through the attention layers. This reduces the per-step cost of generating one additional token from O(n²) (full recompute) to O(n) in sequence length.
The KV (key-value) cache is a fundamental optimization in autoregressive LLM inference. After each token is processed through the attention layers, its projected key and value vectors are cached. When the next token is generated, only the new token's query, key, and value need to be computed; the new query attends to the cached keys and values of all previous positions. This avoids recomputing attention over the entire prefix for every new token, reducing per-step complexity from O(n²) to O(n). The trade-off is memory: the cache grows linearly with sequence length and number of layers, which is a key consideration in serving LLMs at scale.
AI Engineering/llm-foundations/embeddings
Documents were indexed with one embedding model, but retrieval returns near-random results. Which line is the bug?#
const DOC_MODEL = "text-embedding-3-large";
async function indexDocs(docs: string[]) {
const res = await openai.embeddings.create({ model: DOC_MODEL, input: docs });
await store.upsert(res.data.map((d, i) => ({ id: i, vector: d.embedding, text: docs[i] })));
}
async function search(query: string) {
const res = await openai.embeddings.create({ model: "text-embedding-3-small", input: query });
return store.query(res.data[0].embedding, { topK: 5 });
}Show answer
The bug is on line 9.
Line 9 embeds the query with text-embedding-3-small while the documents were indexed with text-embedding-3-large (line 1/4). Different embedding models produce vectors in different, incompatible spaces — they have different dimensionalities and geometry, so cosine similarity between a small-model query vector and large-model document vectors is meaningless. The query must be embedded with the exact same model (and version) used to build the index. The fix is to reuse DOC_MODEL on line 9.
AI Engineering/llm-foundations/embeddings
The Python code below performs masked mean pooling over token embeddings and then computes cosine similarity to a query vector. Trace through it and determine the exact output printed to the console.#
import numpy as np
def masked_mean_pool(token_embeddings, attention_mask):
mask = attention_mask[:, None] # (T, 1)
summed = (token_embeddings * mask).sum(axis=0) # (D,)
count = mask.sum()
return summed / count
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
token_embs = np.array([[1.0, 0.0], [0.0, 2.0], [3.0, 0.0]])
attention = np.array([1, 1, 0])
pooled = masked_mean_pool(token_embs, attention)
sim = cosine_similarity(pooled, np.array([1.0, 1.0]))
print(round(float(sim), 4))Show answer
0.9487
The attention mask [1, 1, 0] zeroes out the third token embedding [3.0, 0.0] before summation. The masked sum is [1.0, 0.0] + [0.0, 2.0] + [0.0, 0.0] = [1.0, 2.0], and dividing by the valid token count (2) gives pooled = [0.5, 1.0]. The cosine similarity with [1.0, 1.0] is (0.5×1 + 1.0×1) / (√(0.25+1) × √(1+1)) = 1.5 / (√1.25 × √2) = 1.5 / 1.58114 ≈ 0.9487. The code prints round(float(sim), 4), which outputs 0.9487.
AI Engineering/llm-foundations/embeddings
A team uses raw (non-fine-tuned) BERT [CLS] token embeddings (768-dim) for semantic search. They observe that cosine similarities between semantically unrelated query–document pairs are consistently above 0.85, which degrades retrieval quality. Which property of raw transformer embeddings best explains this phenomenon?#
Options
Show answer
Raw BERT embeddings are anisotropic: they occupy a narrow cone in the vector space, which inflates pairwise cosine similarities even for unrelated sentences (Ethayarajh, 2019). This is why raw [CLS] vectors yield poor retrieval and why techniques like contrastive fine-tuning, whitening, or sentence-transformer pooling are needed to produce isotropic, discriminative sentence embeddings.
Ethayarajh (2019) and subsequent work showed that contextual embeddings from pre-trained transformers are strongly anisotropic: the representations occupy a narrow cone in the vector space, so cosine similarities are systematically high even for unrelated inputs. Option (a) is wrong because in high dimensions random vectors tend toward orthogonality (cosine → 0), not toward high similarity. Option (c) is wrong because [CLS] embeddings do have non-trivial variance; the issue is the directional concentration, not lack of variance. Option (d) is spurious—float32 rounding does not produce a systematic upward bias in similarity.
AI Engineering/llm-foundations
The code below counts the total trainable parameters in a small decoder-only transformer model. The model uses tied input/output embeddings (the output projection reuses the embedding weight matrix, so the embedding parameters are counted only once). Each transformer layer contains four attention projection matrices (Q, K, V, O) and a two-layer feed-forward network — no biases, no layer-norm parameters. Trace the code and determine the printed output.#
d_model = 512
d_ff = 2048
n_layers = 6
vocab_size = 10000
# Per-layer params (no biases, no layer-norm params)
# Attention: Q, K, V, O = 4 * d_model^2
# FFN: 2 * d_model * d_ff
per_layer = 4 * d_model * d_model + 2 * d_model * d_ff
# Embedding + output projection (tied weights -> counted once)
embedding = vocab_size * d_model
total = n_layers * per_layer + embedding
print(total)Show answer
23994368
Tracing the code step by step: d_model = 512, d_ff = 2048, n_layers = 6, vocab_size = 10000. Per layer, attention contributes 4 × 512 × 512 = 1,048,576 parameters and the FFN contributes 2 × 512 × 2048 = 2,097,152, so per_layer = 3,145,728. Across 6 layers that is 6 × 3,145,728 = 18,874,368. The tied embedding matrix (counted once because weights are shared between input embeddings and output projection) adds 10,000 × 512 = 5,120,000. total = 18,874,368 + 5,120,000 = 23,994,368, which is what print(total) outputs.
AI Engineering/llm-foundations
Rotary Position Embeddings (RoPE) are used in many modern decoder-only LLMs (e.g., LLaMA, Mistral). Which of the following statements about RoPE are correct? Select all that apply.#
Options
Pick every one that applies.
Show answer
The correct statements are that RoPE encodes relative position through position-dependent rotations applied to both query and key vectors, and that the resulting query–key dot product depends only on the relative distance between tokens. RoPE uses no learnable parameters — the rotations are fixed functions of position and a base frequency — and it is applied to both queries and keys, not queries alone.
Statement (a) is correct: RoPE multiplies each pair of dimensions in the query and key by a 2-D rotation whose angle is determined by the token's absolute position index, so the rotations themselves are position-dependent. Statement (b) is correct and is the defining property of RoPE — because the same rotation scheme is applied to both Q and K, the inner product <R_m·q, R_n·k> simplifies to a function of (m − n), the relative position, making the attention scores translation-invariant. Statement (c) is wrong: RoPE uses fixed rotation matrices derived from a base frequency θ (e.g., 10 000) and the position index; there are no learnable position-embedding parameters. Statement (d) is wrong: RoPE is applied to both queries and keys — rotating only one would not produce the relative-position property in the dot product.
AI Engineering/llm-foundations/embeddings
In a BERT-family transformer-based text embedding model (e.g., sentence-transformers/all-MiniLM-L6-v2), the pipeline that converts a raw input string into a single normalized embedding vector follows a strict, universally-agreed sequence of stages. Order the six stages below from first (operates on the raw text) to last (produces the final output vector).#
Put these in order
Show answer
The correct order is: Tokenization → Token embedding lookup → Positional encoding addition → Transformer encoder layers → Pooling → L2 normalization. Each stage depends on the output of the previous one: you need token IDs to look up embeddings, positional encodings to augment those embeddings, encoder layers to contextualize them, pooling to collapse the sequence into one vector, and finally normalization to produce a unit-length embedding for similarity search.
Each stage strictly depends on the output of the previous one, forming an uncontested total order. Tokenization (t1) must occur first because the embedding lookup requires token IDs. Token embedding lookup (t2) produces the initial vectors that positional encodings are added to (t3) — positional information is meaningless without a base embedding to augment. The transformer encoder layers (t4) consume the position-aware embeddings and produce contextualized representations; they cannot run on raw token IDs or pre-pooling vectors without positional context. Pooling (t5) must come after the encoder because it reduces the sequence of contextualized hidden states to a single vector — pooling before the encoder would destroy the positional and contextual information the attention layers need. Finally, L2 normalization (t6) operates on the single pooled vector to produce the unit-length embedding used for cosine similarity, so it must come last. This order is consistent across BERT-family sentence-embedding architectures.
AI Engineering/llm-foundations/embeddings
Mean-centering (subtracting the dataset mean from every embedding vector) is a core step in the whitening transformation used to combat the anisotropy problem in transformer embeddings, where learned representations occupy a narrow cone and pairwise cosine similarities are artificially inflated. The code below shows how mean-centering alone — without full whitening — changes the cosine similarity landscape for three toy 4-dimensional embeddings, computing all three unique pairwise cosine similarities both before and after centering. What is the exact output?#
import numpy as np
# Three embeddings in 4D space
a = np.array([1.0, 2.0, 3.0, 4.0])
b = np.array([2.0, 3.0, 4.0, 5.0])
c = np.array([4.0, 3.0, 2.0, 1.0])
# Compute the dataset mean
mean = (a + b + c) / 3.0
# Center the vectors
a_c = a - mean
b_c = b - mean
c_c = c - mean
# Cosine similarity before centering
cos_ab_before = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
cos_ac_before = np.dot(a, c) / (np.linalg.norm(a) * np.linalg.norm(c))
cos_bc_before = np.dot(b, c) / (np.linalg.norm(b) * np.linalg.norm(c))
# Cosine similarity after centering
cos_ab_after = np.dot(a_c, b_c) / (np.linalg.norm(a_c) * np.linalg.norm(b_c))
cos_ac_after = np.dot(a_c, c_c) / (np.linalg.norm(a_c) * np.linalg.norm(c_c))
cos_bc_after = np.dot(b_c, c_c) / (np.linalg.norm(b_c) * np.linalg.norm(c_c))
print(f"cos(a,b) before: {cos_ab_before:.4f}, after: {cos_ab_after:.4f}")
print(f"cos(a,c) before: {cos_ac_before:.4f}, after: {cos_ac_after:.4f}")
print(f"cos(b,c) before: {cos_bc_before:.4f}, after: {cos_bc_after:.4f}")Show answer
cos(a,b) before: 0.9938, after: 0.4082
cos(a,c) before: 0.6667, after: -0.8018
cos(b,c) before: 0.7454, after: -0.8729
The dataset mean is [7/3, 8/3, 3, 10/3]. Subtracting it removes the shared directional component — the analog of the dominant principal direction that causes anisotropy. The centered vectors are a_c=[-4/3, -2/3, 0, 2/3], b_c=[-1/3, 1/3, 1, 5/3], and c_c=[5/3, 1/3, -1, -7/3]. Before centering, all three pairs appear positively correlated: cos(a,b)=40/√1620≈0.9938, cos(a,c)=20/30=0.6667, and cos(b,c)=30/√1620≈0.7454, because every vector shares a large common additive component that inflates pairwise similarity. After centering, the shared component is stripped away: cos(a_c,b_c)=1/√6≈0.4082 (weakly similar), cos(a_c,c_c)=−3/√14≈−0.8018 (strongly anti-correlated), and cos(b_c,c_c)=−4/√21≈−0.8729 (even more strongly anti-correlated). This reveals that b and c are actually the most anti-correlated pair — a relationship completely masked by the shared mean. This illustrates why anisotropy inflates cosine similarities and why whitening (centering + decorrelation + rescaling) is effective: it exposes the true geometric relationships that the dominant shared direction was obscuring.
AI Engineering/llm-foundations
The code below implements a single head of causal (decoder-only) self-attention with numerical stability (max-subtraction before softmax). Trace the computation by hand and determine the exact value printed.#
import numpy as np
Q = np.array([[1, 0],
[0, 1],
[1, 1]])
K = np.array([[1, 0],
[0, 1],
[1, 1]])
V = np.array([[1, 2],
[3, 4],
[5, 6]])
d_k = 2
scores = Q @ K.T / np.sqrt(d_k)
# Causal mask: upper-triangular set to -1e9
mask = np.triu(np.ones((3, 3)), k=1) * -1e9
scores = scores + mask
# Numerically stable softmax
scores = scores - scores.max(axis=-1, keepdims=True)
exp_scores = np.exp(scores)
attn = exp_scores / exp_scores.sum(axis=-1, keepdims=True)
output = attn @ V
print(np.round(output[2].sum(), 4))Show answer
8.0208
We need output[2], the attention output for the third token (index 2), which can attend to all three positions under causal masking.
Step 1 — Raw dot products (Q[2] · K[j]): Q[2] = [1, 1]. • Q[2]·K[0] = 1·1 + 1·0 = 1 • Q[2]·K[1] = 1·0 + 1·1 = 1 • Q[2]·K[2] = 1·1 + 1·1 = 2
Step 2 — Scale by 1/√d_k = 1/√2: scores[2] = [0.70711, 0.70711, 1.41421]
Step 3 — Causal mask (row 2 is all zeros, no masking): No change.
Step 4 — Stable softmax (subtract max = 1.41421): Shifted: [−0.70711, −0.70711, 0] exp: [0.49307, 0.49307, 1.0] Sum = 1.98614 attn[2] = [0.24826, 0.24826, 0.50347]
Step 5 — Weighted sum of V: output[2] = 0.24826·[1,2] + 0.24826·[3,4] + 0.50347·[5,6] = [0.24826 + 0.74479 + 2.51736, 0.49653 + 0.99306 + 3.02083] = [3.51042, 4.51042]
Step 6 — Sum: 3.51042 + 4.51042 = 8.02083 → rounded to 4 decimals = 8.0208.
Related interview questions
Job market
See ai-engineering salaries and hiring demand from live job postings.
The other 1 question
This page shows 25 and marks what you pick. That's as far as a page can go. A free account opens the other 1 and keeps every answer. What you miss comes back until it's right: after a day, then at longer gaps.
Free · the whole bank · 100 marked answers per 30 days · written feedback on the paid plan