Kubernetes Architecture Interview Questions

True statements about the Kubernetes API server are: it is the only control plane component that reads from and writes to etcd directly; it validates and persists the desired state of Kubernetes objects; and it exposes the Kubernetes API over HTTP/HTTPS for clients like kubectl. The API server does not schedule pods onto nodes — that is the kube-scheduler's job.

The Kubernetes API server processes a create request in this order: authenticate the client, then authorize against RBAC, then run mutating admission controllers, then run validating admission controllers, and finally persist the object to etcd. Mutating controllers run before validating controllers so that any changes they make are visible to the validators that follow.

The API server first serializes an AdmissionReview request to JSON, then sends it as an HTTPS POST to the webhook endpoint, then receives the AdmissionReview response with a JSON patch, then applies the patch to the in-flight object, and finally proceeds to the next admission controller in the chain. Each mutating webhook in the chain repeats this same sequence.

Kubernetes persists the authoritative state of every object in etcd, a consistent distributed key-value store. The kube-apiserver is the only component that reads from and writes to etcd directly; every other component goes through the API server. Consequently an etcd backup is effectively a full cluster backup, and losing etcd without one means losing the cluster's desired state.

The kubelet is the node agent that runs on every worker node. It registers the node with the control plane, watches the API server for Pods assigned to its node, and drives the container runtime (via the CRI) to start, stop, and health-check those Pods' containers. It also reports node and Pod status back. It does not schedule Pods — it only runs the ones the scheduler placed on it.

etcd is the cluster's backing store: a consistent, highly-available distributed key-value store that holds the entire cluster state (every object and its desired/observed status). The kube-apiserver is the only component that reads from and writes to etcd; everything else goes through the API server. Because it's the single source of truth, a regular etcd backup is effectively a full-cluster backup.

The Kubernetes API server (kube-apiserver) is the central management entity of the control plane. It validates and configures API objects such as pods, services, and controllers, and it is the only control-plane component that communicates directly with the cluster's backing data store, etcd. Every other component — the scheduler, controller-manager, and each kubelet — reads and writes cluster state exclusively through REST calls to the API server, which in turn persists that state into etcd.

kube-proxy runs on every node and implements Service networking: it programs iptables or IPVS rules (or coordinates with the CNI) so that traffic sent to a Service's stable ClusterIP is load-balanced to one of its healthy backend Pods. It does not schedule Pods (the scheduler), pull images (the kubelet and runtime), or relay kubectl commands (those hit the API server directly).

I would create three separate webhook configurations, each scoped tightly to its resource type.

Resource requests (MutatingWebhookConfiguration): A mutating webhook scoped to pods with a namespaceSelector that excludes system namespaces. It defaults missing CPU/memory requests and limits to team-specific baseline values. Because it mutates the object, it must run in the mutating phase. I set its webhook name so it runs early in the mutating chain (before any other webhook that might read resource values).

TLS certificate validation (ValidatingWebhookConfiguration): A validating webhook scoped to ingresses that checks the referenced TLS secret or cert ARN against the ConfigMap of approved certificates. This is pure enforcement — no mutation — so it belongs in the validating phase. failurePolicy: Fail and timeoutSeconds: 3: if the webhook is unavailable, the Ingress creation is rejected rather than allowing an ingress with an unapproved certificate. The trade-off is that a webhook outage blocks all Ingress writes, which is acceptable because an unapproved TLS cert is a security violation.

Namespace labels (ValidatingWebhookConfiguration): A validating webhook scoped to namespaces that rejects Namespace creation or update if mandatory labels are missing. Again failurePolicy: Fail with a short timeout.

Ordering: MutatingWebhookConfiguration always runs before ValidatingWebhookConfiguration in the API server admission chain, so the resource-request defaults are applied before any validating webhook sees the Pod. Within the mutating phase, webhooks are ordered by name; I prefix names accordingly.

Scalability: Each webhook backend runs as a Deployment with HPA behind a ClusterIP Service. Because admission is synchronous — the API server blocks the original request until all webhooks return — webhook latency directly caps API throughput. I configure keep-alive connections between the API server and webhook backends, co-locate backends on nodes close to API servers, and set aggressive timeoutSeconds (2–3s) so a slow webhook degrades rather than hangs. I also instrument webhook latency histograms and alert if p99 exceeds 100ms.

The controller uses the client-go shared informer pattern: one SharedInformer per resource type (DataPipeline, Deployment, ConfigMap, Job).

List+Watch with resourceVersion: On startup each informer issues an initial List against the API server to seed its local store (cache). The List response includes a resourceVersion — an opaque string representing a point in the etcd revision history. The informer then opens a Watch from that resourceVersion, receiving ADDED/MODIFIED/DELETED events for all subsequent changes. The resourceVersion is the consistency token: the API server uses it to determine whether the client's view is current enough to stream incremental events.

Reconnect and 410 Gone: If the Watch connection drops or times out, the informer attempts to re-establish the Watch from the last-seen resourceVersion. If the API server returns HTTP 410 Gone — meaning the requested resourceVersion has been compacted away because too much history has elapsed — the informer performs a full re-list to get a fresh resourceVersion and reseeds the local store. This is critical: after a network partition lasting minutes, the etcd history compaction window may have passed, so resuming from the old resourceVersion would silently miss events. The 410 forces a safe re-list.

Level-triggered reconciliation: The informer's event handlers do not perform reconciliation directly. Instead, each handler enqueues the object's key (namespace/name) onto a work queue. The reconcile loop dequeues a key, reads the current desired state (from the DataPipeline cache) and current actual state (from the Deployment/ConfigMap/Job caches), and makes whatever changes are needed. Because reconciliation compares current state — not the event payload — the system is level-triggered. A missed event is corrected on the next resync or the next related event that enqueues the same key. Duplicated events are harmless because reconciliation is idempotent.

Restart recovery and resync: On controller restart, the in-memory cache is empty. Each informer starts fresh with a full List, rebuilding the cache from the API server's current state. No persistent local state is required — the API server is the source of truth. Additionally, the shared informer has a configurable resync period (e.g., 10 minutes) that periodically re-enqueues every cached object's key into the work queue. This acts as a safety net: even if an event was somehow missed (e.g., a brief watch gap that didn't trigger 410), the next resync re-examines the object and reconciles any drift.