Security and Policy

Baseline

For a senior DevOps/infrastructure automation role, security is part of reliability. A tool that can drain nodes, change routing, or update GPU drivers must have least privilege, auditability, and safe defaults.

flowchart TD
  Request[Automation request] --> Identity[Service identity]
  Identity --> RBAC[Scoped RBAC]
  RBAC --> Policy[Admission and policy checks]
  Policy --> Guardrails[Blast radius guardrails]
  Guardrails --> Action[Approved action]
  Action --> Audit[Audit event and metrics]
  Policy -- denied --> Deny[Reject with reason]

Kubernetes Security Areas

Area	What to know
RBAC	Least privilege roles for humans, automation, and controllers.
Admission control	Enforce production contracts before workloads land.
Pod Security Standards	Restricted/baseline/privileged posture by namespace and workload class.
Secrets	External secrets, KMS, rotation, no secrets in Git.
Network policy	Default deny where practical, explicit service communication.
Image security	Signed images, SBOMs, vulnerability scanning, trusted registries.
Runtime security	Seccomp, AppArmor/SELinux, non-root, read-only filesystems where possible.
Audit logs	Who changed what, when, and through which automation.

Policy Tools

• Native admission policies where sufficient.
• Kyverno for Kubernetes-native policy authoring and mutation/validation.
• OPA Gatekeeper for Rego-based policy.
• Sigstore/cosign for image signing workflows.
• SOPS/external-secrets for GitOps-friendly secret management.

CVE Remediation Workflow

Vulnerability response is release engineering under time pressure.

1. Identify affected artifact: base image, OS package, Python wheel, CUDA library, driver container, Helm chart, or sidecar.
2. Determine exploitability in context: exposed surface, privilege, network reachability, tenant boundary, and compensating controls.
3. Build patched artifact with pinned dependencies and SBOM/provenance.
4. Run compatibility tests for CUDA, TensorRT, Triton backend, container runtime, and target GPU SKU when relevant.
5. Canary with SLO, correctness, startup, GPU memory, and node health gates.
6. Roll out by environment/pool and keep rollback artifact available.
7. Record evidence for compliance: affected versions, fixed versions, rollout timestamps, exceptions, and owner signoff.

Senior answer:

I do not treat CVE remediation as “bump the image and hope.” For GPU infrastructure, patched drivers, CUDA libraries, and runtime images can change behavior, so the security fix still needs compatibility gates and rollback.

GPU-Specific Security

• Privileged access for GPU stack components should be isolated to system namespaces.
• Workloads should not get broad host access just because they need GPUs.
• MIG can help isolate tenants on supported hardware.
• Time-slicing/MPS may have weaker isolation; be explicit about tenant trust.
• Debug containers and nvidia-smi access should respect RBAC and incident workflow.

For workload identity federation, service account token risks, and multi-tenant isolation patterns, see Kubernetes Internals 2026.

Automation Permissions

For repair automation:

• Read nodes, pods, events, metrics.
• Cordon/drain only scoped node pools.
• Create events and audit records.
• Update labels/taints only in approved prefixes.
• No broad cluster-admin unless truly unavoidable.
• Separate recommendation mode from execution mode.

Interview Drill

Question: “Your automation needs permissions to reboot or drain GPU nodes. How do you secure it?”

Answer:

1. Scope RBAC to specific node pools and verbs.
2. Require dry-run plan and approval for high-risk actions.
3. Use service identity with short-lived credentials where possible.
4. Emit audit events for plan, approval, action, and result.
5. Add concurrency and capacity guardrails.
6. Keep break-glass separate from normal automation.
7. Test permissions in staging and verify denied paths.