AI Engineering interview questions: LLM tokens & context windows

In a standard decoder-only LLM pipeline, raw text is first tokenized into subword pieces, then each token is converted to a numerical token ID via the vocabulary lookup, then each ID is mapped to a dense embedding vector, then positional encodings are added so the model can use order information, and finally the resulting vectors are fed into the transformer layers. This order is fixed: you cannot look up embeddings without IDs, and you cannot add positional encodings before embeddings exist.

The context window bounds the total tokens the model attends to: system prompt + retrieved context + conversation history + the tokens it generates. If you stuff the input close to the limit, you starve the output and risk truncated completions. Budget explicitly — reserve headroom for max_tokens of output, and remember that long outputs cost both latency and window space.

The string is 49 characters; ceil(49 / 4) = ceil(12.25) = 13. The ~4 chars/token rule is a quick estimate for budgeting context windows and cost, not an exact count — real tokenizers (BPE) split on sub-word units, so whitespace, punctuation, code, and non-English text all change the ratio. For anything that must be exact (e.g. enforcing a hard context limit) you should call the model's real tokenizer rather than rely on this heuristic.

The context window is the hard token budget for a single request, shared between the prompt (system + history + retrieved context + user message) and the completion. Overflowing it forces truncation or summarization of history, and because attention cost grows with sequence length, packing the window with marginal context also raises latency and price — so context is a resource to budget, not just fill.

An append-only transcript grows monotonically, and every iteration re-sends all of it, so a long run walks straight into the context-window limit — the request is rejected on validation, before any generation happens. That is why the failure looks sudden and iteration-dependent rather than input-dependent: nothing about iteration 26 is special except the cumulative size. The fixes are all about what you carry forward — summarise or compact older turns, clear old tool results, or keep detail out of the loop and retrieve it on demand. The distractors are the three things engineers commonly assume the provider is doing for them: system prompts do not expire (b), old tool results stay valid and are not garbage-collected for you (c), and there is no per-conversation tool-call ceiling — your own max_steps is the only one (d). Believing any of them leads to the same production bug: an agent that works in testing and dies on the long tasks that matter.

Statements (a), (b), and (d) are correct. Tokenization (e.g., BPE, WordPiece) runs as a preprocessing step that splits text into subword units and maps each to a vocabulary ID (a). The context window is the upper bound on how many input tokens the model can accept in one request (b). The input embedding layer is a lookup table where each vocabulary ID corresponds to exactly one fixed-size vector (d). Statement (c) is wrong because standard transformer self-attention has O(n²) complexity in sequence length, not linear—doubling the context more than doubles attention cost. Statement (e) is wrong because tokens are subword units, not whole words; a single word may span multiple tokens and a single token may encode partial words, punctuation, or whitespace.

Line 17 (while remaining and budget <= 0:) is the sole bug. The variable budget (line 13) is max_tokens - count_tokens(system['content']) — the token capacity available for non-system messages. The condition budget <= 0 is true only when the system prompt alone already exceeds max_tokens, so the loop body runs only in that edge case. In the common scenario the system prompt fits (budget > 0) but the remaining conversation still overflows the budget; the loop never executes, so the function returns the full conversation without dropping anything, violating the token limit. The condition should instead check whether the total token count of all remaining messages exceeds the available budget: while remaining and sum(count_tokens(m['content']) for m in remaining) > budget:. Line 18 (remaining.pop(0)) is correct: it removes the oldest non-system message (the first element of remaining), which is exactly the required behavior, and needs no change.

Long sessions are kept alive by managing what occupies the window: summarize/compact older turns into a compact running state while keeping the latest turns verbatim, and offload detail to an external memory the agent can retrieve when relevant (a). Dropping the system prompt and tool definitions (c) deletes the agent's instructions and capabilities — it will lose its task and its tools. Overflow is not gracefully ignored (d): exceeding the window either errors outright (most provider APIs reject the request) or, in some client/framework helpers, silently truncates the oldest tokens (often the goal/system prompt) — which is the failure you're trying to avoid. Temperature (b) does nothing about capacity.

Long-running agents survive by managing the window rather than enlarging it. Two complementary moves dominate: (1) compaction — replace older turns with a running summary, keeping recent turns verbatim; (2) externalize + retrieve — persist the full history to an outside store and pull back only what's relevant to the current step (RAG over your own transcript). Supporting tactics: prune/evict low-signal turns, and pin durable facts (goal, decisions, constraints) in a small always-present block. Simply 'use a model with a bigger context window' defers the problem and raises cost/latency; the win is compressing and selecting what the model actually needs to see.

This tests a staff-level understanding of how positional encoding strategy determines context-length behavior. ALiBi's relative-distance bias is position-agnostic by construction, enabling extrapolation; absolute embeddings are table-lookups with a fixed vocabulary of positions, creating a hard upper bound. RoPE sits between these two extremes — it is relative but periodic, which is why it requires position interpolation or NTK-aware scaling to extend beyond the pre-training length.