DevOps CI/CD interview questions

A well-designed build artifact is immutable once published, produced from a specific reproducible source commit, and promoted unchanged across environments. It should NOT bundle environment-specific secrets or config; those are injected at runtime so the same artifact can be safely deployed everywhere.

Common build-artifact formats in CI/CD pipelines include Docker container images, compiled JAR or WAR files, and ZIP archives of binaries and assets. Git commit messages are source-control metadata, not build artifacts.

Pipeline-as-code means defining CI/CD pipeline configuration in version-controlled text files such as YAML or Groovy, rather than configuring it manually through a web UI. This lets teams review, branch, and reproduce their build and deployment workflows exactly like application code.

In a Docker CI/CD deployment, the steps proceed in strict dependency order: first clone the source repository, then build the image from the Dockerfile, push it to a container registry, pull it on the deployment target, and finally start a container from it. Each step requires the previous step's output — you cannot build without source, push without a build, pull without a push, or run without a local image.

The top-level key is on. For example, on: push means the workflow runs when a push event occurs.

Hypertext Transfer Protocol

False. Build artifacts like compiled binaries, JAR files, and Docker images should be stored in a dedicated artifact repository (e.g., Artifactory, Nexus, or a container registry), not committed to Git. Committing artifacts bloats the repository with large binary files, creates unnecessary merge conflicts, and duplicates data that can always be reproduced by rebuilding from source.

True. The same immutable build artifact that passed tests in staging should be promoted directly to production. Rebuilding from source for production risks producing a subtly different artifact due to changes in build environment, dependency versions, or transient state, which would break the guarantee that the tested software is exactly the shipped software.

A manual approval gate is a deployment checkpoint that requires explicit human approval before a build can be promoted to the next pipeline stage, adding a manual decision point to an otherwise automated release flow.

In CI/CD pipelines, a quality gate automatically blocks deployment when a measurable signal — such as test pass rate, code coverage, or static-analysis score — falls below the configured threshold.

Pipeline-as-code lets you version, review, and branch pipeline changes alongside application code. Because the pipeline definition lives in a committed file (e.g., Jenkinsfile or .gitlab-ci.yml), every modification goes through the same code-review and history workflow as source code, giving you traceability, reproducibility, and team collaboration that a web-UI configuration cannot match.

The canonical CI/CD pipeline stage order is Source → Build → Test → Deploy. The pipeline first checks out code from version control, then compiles and packages the artifact, then runs automated tests against that artifact, and finally releases the validated artifact to the target environment. Each stage depends on the output of the prior stage.

The test stage. GitLab CI/CD's built-in default stages are .pre, build, test, deploy, and .post, and a job with no stage keyword is placed in test.

Continuous delivery and continuous deployment both keep the codebase continuously releasable — every change is automatically built, tested, and proven deployable. The one difference is the production gate. Continuous delivery makes each change release-ready but a human approves the final push to production. Continuous deployment removes that gate: any change that passes the automated pipeline ships to production with no manual step. Continuous integration is the upstream practice both build on — frequent merges with automated testing on each commit.

The practice is called "build once, promote everywhere" (also referred to as promoting immutable artifacts or single-source artifact promotion). One key reason it is preferred is that it guarantees the exact binary tested in staging is the one deployed to production, eliminating drift caused by non-deterministic builds (e.g., differing dependency versions, toolchain updates, or environment-specific build flags) that could introduce defects between environments.

In a CI/CD pipeline, a canary gate samples live telemetry — such as error rate and p99 latency — for a fixed observation window after routing a small percentage of production traffic to the new release. The pipeline proceeds to full rollout only if every metric threshold stays within its defined limit throughout that window; otherwise the release is automatically rolled back.

A change to the shared library's default branch can modify every consuming pipeline with no commit in any application repository. Because library 'shared-pipeline-lib' specifies no pinned version, Jenkins resolves the library from its default branch at each build, so the library and the Jenkinsfile are not coupled to the same commit and the library's changes are reviewed in its own repo, not the application's.

The correct order is Source → Build → Package → Publish. A commit triggers the pipeline; source is compiled and linked into a binary; the binary is bundled with its runtime dependencies into a deployable artifact; and the packaged artifact is pushed to a container registry. Each stage depends on the direct output of the one before it, forming a strict prerequisite chain.

It re-deploys the exact previously-running artifact (app:git-2b7e08) with no rebuild, so the rollback is fast and reproducible — you get back the bytes that were known-good in prod

Gate architecture: Each pipeline has five stages with both team-configurable and org-mandated gates.

1. Build stage: Org-mandated gate — container image build + SAST scan + dependency vulnerability scan (fail on Critical CVEs). Team gate — unit tests with ≥80% coverage delta threshold.
2. Integration stage: Org-mandated gate — consumer-driven contract tests against the service's published API spec (Pact or equivalent). The gate fails if any registered consumer's contract is broken. Team gate — integration tests against a shared ephemeral environment.
3. Staging stage: Org-mandated gate — load test reaching 2× expected peak QPS; p99 latency must stay within SLO. Team gate — smoke tests and synthetic transaction checks.
4. Pre-prod (canary launch): The service is deployed to 5% of production traffic. Canary gate evaluates error rate (must not exceed baseline + 0.5 percentage points), p99 latency (must not exceed baseline × 1.2), and business KPI delta (e.g., checkout conversion rate must not drop more than 1% relative to control). If any metric breaches its threshold, the gate triggers an automatic rollback to the previous version. Metrics that are informative but not safety-critical (e.g., cache hit rate, memory usage trending up but within limits) trigger a hold-and-notify — deployment pauses at 5%, alerts the team, and waits for manual promotion or rollback.
5. Production (progressive rollout): After canary passes, traffic is ramped 25% → 50% → 100% with the same canary gate re-evaluated at each step. A failure at any step rolls back to the last known-good percentage.

Cross-service dependency gating: Every service publishes an OpenAPI/Protobuf spec in a central schema registry. Before a service can deploy to production, the gate runs backward-compatibility checks — if the new spec removes a required field or changes a type in a breaking way, the gate fails unless all downstream consumers have already deployed a compatible version. The dependency graph is maintained by a service catalog; the gate queries it to identify consumers. Consumer-driven contract tests run at the integration stage and must all pass; the gate blocks if any registered consumer's contract is broken by the new version.

Decentralization with enforced baseline: Gates are defined as code in a shared gate-library repository. Each team's pipeline imports a 'base-gate-profile' that contains the org-mandated gates (SAST, CVE scan, contract tests, load test, canary SLO gate). Teams can add team-specific gates on top but the base profile is enforced by a policy controller (OPA/Conftest) that validates every pipeline definition at commit time — if the base gates are missing or disabled, the pipeline config is rejected by the CI system and cannot run. This gives teams autonomy to extend while guaranteeing the safety floor.

Gate taxonomy and automation boundary:

Automated gates (no human action required, evidence captured automatically):

• Build gate: compilation + unit tests + coverage threshold (≥80%). Pass/fail is deterministic.
• Security gate: SAST, DAST, dependency CVE scan, container image scan. Fail on Critical/High.
• IaC gate: Terraform plan validation + policy-as-code check (no public S3 buckets, encryption required, etc.).
• Integration gate: contract tests + integration tests against a staging environment mirroring production config.
• Compliance gate: automated check that the deployment package matches the approved change ticket (e.g., Jira issue linked in the commit), and that all required automated gates have passed with their evidence stored.

Human-approval gate (the only manual step):

• Production release authorization: a designated approver (not the builder) reviews the compiled evidence package — test results, scan reports, change-ticket linkage, artifact hash — and explicitly approves promotion to production. This gate exists because SOX requires management authorization for changes to production financial systems, and a human must attest that the change was reviewed.

Separation of duties enforcement: The CI/CD platform enforces two disjoint RBAC roles: 'Deployer' (can trigger pipeline runs, merge code, push artifacts through staging) and 'Approver' (can authorize production promotion). The pipeline's production-promotion step checks the identity of the approver against the identity of the user who triggered the current pipeline run. If they match, the step is blocked and returns an error. Additionally, pull requests require at least one review from a member of the Approver group before merge, and the PR system prevents self-review. These are enforced by the platform (e.g., pipeline engine identity checks + SCM branch-protection rules), not by verbal policy.

Compliance evidence capture: Every gate — automated and manual — writes a structured evidence record to an immutable, append-only audit store (e.g., a dedicated evidence repository or a tamper-evident log service). Each record contains: gate name, gate version/config hash, input artifact hash, pass/fail result, detailed output (test report, scan findings), timestamp (UTC, from a trusted time source), and the identity that triggered or approved it. Records are correlated by a deployment ID that links every gate result to the specific artifact hash and change ticket. An auditor can query by deployment ID or date range and receive a complete, ordered timeline of every gate executed, its result, and who acted. The audit store uses hash-chaining (each record references the hash of the previous record) to make tampering detectable.

Bottleneck reduction: (1) The human approver never runs tests or scans manually — by the time approval is requested, all automated gates have passed and the evidence package is pre-assembled. The approver reviews a summary dashboard with green/red indicators and drill-down links, taking 2–5 minutes. (2) Production rollout uses canary delivery: the approver authorizes a 5% canary, automated health gates evaluate metrics for 15 minutes, and if they pass, the rollout auto-promotes to 100% without a second human approval. The human is in the loop once, not at every step. (3) For low-risk changes (documentation, non-functional config via feature flags), an expedited path auto-approves if the change classifier determines the diff touches only flag-config files — the compliance gate still records all evidence and the auto-approval decision with its rationale.

Breaking-change detection and expand-contract enforcement:

The gate includes a migration analyzer that parses each migration script (or diffs the before/after schema) and classifies every DDL operation:

• Breaking: DROP COLUMN, DROP TABLE, RENAME COLUMN, type narrowing (VARCHAR(255) → VARCHAR(50)), adding NOT NULL constraint without a default, removing a default.
• Non-breaking: ADD COLUMN (nullable or with default), ADD INDEX (using CONCURRENTLY), widening types, adding a table.

When the gate detects a breaking change, it checks the migration history and the currently-deployed application code version. For a DROP COLUMN, the gate requires that: (a) a prior migration added the replacement column, (b) the currently-deployed code version no longer references the old column (verified by checking that the deployed artifact's schema-config or ORM mapping doesn't include it), and (c) at least one full deployment cycle has passed since the expand migration. If any condition is unmet, the gate blocks the migration with a message like 'Contract phase blocked: column X still referenced by deployed code version Y.' This enforces expand-then-contract by making the gate stateful — it tracks which expand migrations have been deployed and only permits the corresponding contract migration after code deployment is verified.

Production-scale performance validation:

The gate does not trust staging data volume. Instead, it runs the migration against a production-sized data clone — a logical replica of production created from the latest PITR snapshot, restored to a disposable compute instance. The migration is executed against this clone with instrumentation: the gate captures wall-clock time, lock-wait time, and whether any statement holds an AccessExclusiveLock for more than a configurable threshold (e.g., 5 seconds). If the migration exceeds the time threshold or acquires a blocking lock, the gate fails and suggests an alternative (e.g., use CREATE INDEX CONCURRENTLY, or batch the UPDATE). For very large tables, the gate also runs EXPLAIN ANALYZE on destructive statements to estimate row-scan costs. The clone is sized to match production's largest table row counts, closing the staging-vs-production gap.

Migration-application deployment ordering:

The gate enforces ordering through a deployment manifest that each service submits with its pipeline. The manifest declares the migration type (expand, contract, or none) and the code compatibility level (backward-compatible with old schema, requires new schema). The gate enforces:

1. If the migration is 'expand' (additive): migration runs first, then new code is deployed. Old code remains compatible because the migration is additive.
2. If the migration is 'contract' (removing old schema): the gate verifies the currently-deployed code is the backward-compatible version from the expand phase, then runs the contract migration, then (optionally) deploys code that assumes the old schema is gone.
3. If no migration: standard code deployment. The pipeline engine refuses to proceed if the manifest's declared ordering doesn't match the gate's classification — e.g., if a team labels a DROP COLUMN migration as 'expand,' the analyzer catches the mismatch and blocks.

Rollback safety:

The gate classifies each migration as reversible or irreversible at analysis time:

• Reversible: ADD COLUMN (nullable), ADD INDEX, CREATE TABLE. These can be rolled back with DROP, and no committed data is lost (the column was new).
• Irreversible: DROP COLUMN (data in that column is destroyed), DROP TABLE, TRUNCATE, type narrowing (data may be truncated).

Policy: When a deployment fails after a migration has been applied, the system checks the migration's reversibility classification. If reversible, the migration is rolled back and the previous code version is redeployed. If irreversible, the system rolls back the code to the previous version but leaves the schema change in place — rolling back the migration would cause data loss. The incident is flagged for a forward-fix: the team must fix and redeploy rather than revert the schema. The gate also requires that all contract-phase migrations are preceded by a verified backup or that the data being removed has been archived — the gate checks for an archive artifact (e.g., a dump of the column's data stored in object storage) before permitting an irreversible migration to proceed.

Use a content-addressable storage (CAS) model keyed on the hash of inputs (source + toolchain + env vars + transitive deps), not on branch/commit alone. Bazel and Buck2 exemplify this: the build graph is hermetically sealed (all inputs explicitly declared), outputs are stored by their content hash, and remote execution nodes share a CAS-backed cache. A cache hit on node A is therefore deterministic on node B because the hash captures the full input set—toolchain version, compiler flags, env var values, and dependency hashes—making locality irrelevant.

Reproducibility means the build is a pure function of declared inputs: the same inputs produce byte-identical output regardless of time, worker, or environment. Reproducibility is the prerequisite—without it, cache keys derived from inputs are unreliable. Hermeticity is the enforcement mechanism: all inputs are explicitly declared and ambient state (wall clock, network, env vars, filesystem layout) is sandboxed or forbidden. Together, hermeticity guarantees that the input hash captures everything, which makes reproducibility provable and cache keys trustworthy.

Path-based triggering via a service-to-directory mapping, dependency-graph traversal that finds services transitively depending on a changed shared library, and diffing against the merge-base are correct for selective pipeline execution. The claim that selective execution removes the need for integration testing before production is false, since it only shortens CI feedback time.