GitOps and Delivery

Core Principle

GitOps means desired cluster state is declarative, versioned, reviewed, and reconciled by controllers. The cluster should not depend on someone remembering the right kubectl command.

flowchart LR
  Change[Change request] --> PR[Pull request]
  PR --> CI[CI validation]
  CI --> Merge[Merge to environment branch]
  Merge --> GitOps[Argo CD or Flux]
  GitOps --> Cluster[Kubernetes cluster]
  Cluster --> Health[Health and drift status]
  Health -. feedback .-> PR

Argo CD

Argo CD is a Kubernetes controller that compares live cluster state to desired state in Git, reports drift as out-of-sync, and can sync changes automatically or manually.

Use it for:

• Application and platform component deployment.
• Multi-tenant app projects.
• Drift visibility.
• Manual gates for sensitive environments.
• Progressive promotion with related tools.

Flux

Flux keeps clusters in sync with configuration sources and can automate updates when new code or images are available. It is composable through controllers and works well for GitOps at fleet scale.

Use it for:

• Cluster bootstrap.
• Helm release management.
• Image update automation.
• SOPS-based secret flows.
• App and infrastructure dependencies.

Helm vs Kustomize

Tool	Good for	Risk
Helm	Packaged apps with values.	Template complexity and values drift.
Kustomize	Environment overlays and patching.	Overlay sprawl and hidden differences.

Staff answer:

I care less about Helm vs Kustomize than about reproducibility: pinned versions, rendered diff, policy checks, promotion path, rollback, and post-sync validation.

Progressive Delivery

For inference:

• Canary model/runtime changes.
• Use synthetic requests before real traffic.
• Gate on p99, error rate, queue time, GPU memory, and correctness checks.
• Roll back traffic first; clean up state second.
• Separate model artifact rollout from driver/runtime rollout when possible.

Tools you can mention:

• Argo Rollouts.
• Flagger.
• Gateway API/service mesh traffic splitting.
• Prometheus/OpenTelemetry metrics as rollout gates.

stateDiagram-v2
  [*] --> Deploy
  Deploy --> ValidateHealth
  ValidateHealth --> Shadow
  Shadow --> Canary
  Canary --> Promote: SLO and correctness pass
  Canary --> Rollback: SLO or correctness fail
  Promote --> Full
  Full --> [*]
  Rollback --> Previous
  Previous --> [*]

CI/CD Pipeline

For this NVIDIA role, speak in terms of GitHub Actions or GitLab CI building inference libraries and container-based software stacks:

1. Build library/runtime/container image with pinned toolchain and dependency lockfiles.
2. Run unit, integration, ABI/API, and compatibility tests.
3. Build or register immutable model and engine artifacts where applicable.
4. Generate SBOM, scan vulnerabilities, and sign artifacts.
5. Render Helm/Kustomize/Crossplane manifests.
6. Run policy checks for security, resource requests, GPU pool selectors, and ownership labels.
7. Produce a human-readable diff against the target environment.
8. Promote through dev, staging, canary, and production with environment-specific approvals.
9. GitOps controller syncs the desired state.
10. Run post-sync validation: health, synthetic inference, metrics, logs, and rollback readiness.
11. Shift traffic progressively and halt on SLO, correctness, GPU memory, or error-budget gates.

flowchart LR
  Source[Git change] --> CI[GitHub Actions or GitLab CI]
  CI --> Build[Build image and libraries]
  Build --> Test[Test and compatibility matrix]
  Test --> Supply[SBOM, scan, sign]
  Supply --> Render[Render manifests]
  Render --> Policy[Policy and diff]
  Policy --> GitOps[GitOps sync]
  GitOps --> Canary[Canary validation]
  Canary --> Promote{SLO and correctness pass}
  Promote -- yes --> Prod[Progressive production rollout]
  Promote -- no --> Rollback[Rollback or halt]

Test automation answer:

I would treat tests as release evidence, not only pass/fail. The pipeline should publish which model/runtime/GPU/driver matrix was exercised, synthetic inference latency, correctness samples, CVE status, rendered manifest diff, and post-deploy SLO gates.

Incident Scenario

Question: “Someone hotfixed production with kubectl and GitOps reverted it.”

Answer:

• Confirm impact.
• If hotfix is needed, commit it to source of truth or pause sync intentionally with approval.
• Avoid fighting the controller.
• Afterward, define break-glass process: who can pause sync, how changes are recorded, and how drift is reconciled.

Senior Close

GitOps is not a substitute for release engineering. It gives you reconciliation and auditability, but you still need rollout policy, health gates, dependency ordering, and a humane break-glass path.