Kubernetes Internals 2026

Scope

This page is the staff-level Kubernetes interview layer. The goal is to explain what happens below kubectl, why each component exists, how failures propagate, and how to debug without guessing.

Docs checked for this pass:

• Kubernetes v1.36 docs
• Admission webhook good practices
• Scheduling Framework
• Operating etcd clusters for Kubernetes
• Container Runtime Interface
• About cgroup v2
• Cluster Networking
• Topology Aware Routing
• Resource Quotas
• Multi-tenancy
• Service Accounts
• Operator pattern
• CSI developer documentation
• Cilium Service Mesh
• Istio ambient mode

The Core Mental Model

Kubernetes is a distributed control system:

• Users and controllers write desired state to the API server.
• Admission enforces policy before persistence.
• etcd durably stores cluster state.
• Watchers observe changes.
• Controllers reconcile current state toward desired state.
• The scheduler assigns unscheduled pods to nodes.
• Kubelet asks CRI/CNI/CSI/device plugins to create the actual workload.

flowchart TD
  Kubectl[kubectl or controller] --> API[API server]
  API --> Auth[Authentication and authorization]
  Auth --> Mutate[Mutating admission]
  Mutate --> Validate[Schema and validating admission]
  Validate --> Etcd[etcd commit]
  Etcd --> Watch[Watch cache and informers]
  Watch --> Scheduler[Scheduler]
  Watch --> Controllers[Controllers]
  Scheduler --> Bind[Pod binding]
  Bind --> Kubelet[Kubelet]
  Kubelet --> CRI[CRI runtime]
  Kubelet --> CNI[CNI plugin]
  Kubelet --> CSI[CSI driver]
  Kubelet --> Device[Device plugin]
  CRI --> Pod[Pod sandbox and containers]
  CNI --> PodNet[Pod network]
  CSI --> Volume[Mounted volume]
  Device --> GPU[GPU or accelerator allocation]

Interview line:

I think of Kubernetes as API-driven reconciliation. The API server and etcd provide the source of truth; controllers, scheduler, kubelet, CNI, CSI, and device plugins convert that truth into running processes, networks, volumes, and devices.

What Happens After kubectl apply

1. kubectl builds a REST request using kubeconfig, credentials, and target API group/version/resource.
2. The API server authenticates the identity.
3. Authorization checks RBAC or another authorizer.
4. Mutating admission webhooks may default or rewrite the object.
5. Built-in validation and OpenAPI/schema validation run.
6. Validating admission webhooks and policies accept or reject the final object.
7. The API server writes the object to etcd.
8. Watchers see the new revision through watches/informer caches.
9. Controllers react by creating or updating dependent objects.
10. If a pod has no nodeName, the scheduler evaluates it and binds it.
11. Kubelet on the selected node starts the pod through CRI, CNI, CSI, and device plugins.

sequenceDiagram
  participant User
  participant API as API server
  participant Etcd
  participant Controller
  participant Scheduler
  participant Kubelet

  User->>API: apply Deployment
  API->>API: authn, authz, mutation, validation
  API->>Etcd: persist desired state
  Etcd-->>Controller: watch event
  Controller->>API: create ReplicaSet and Pods
  API->>Etcd: persist Pods
  Etcd-->>Scheduler: watch unscheduled Pod
  Scheduler->>API: bind Pod to node
  Etcd-->>Kubelet: watch assigned Pod
  Kubelet->>Kubelet: CRI, CNI, CSI, devices, probes

Senior debugging:

• If kubectl apply is slow before persistence, inspect authn/authz/admission/API Priority and Fairness.
• If objects persist but nothing happens, inspect controllers and informer/watch lag.
• If pods remain pending, inspect scheduler filters, quotas, PVC binding, and device resources.
• If pods are assigned but stuck, inspect kubelet, CRI, CNI, CSI, image pulls, runtime hooks, and node pressure.

Kubelet Internals And Node Reconciliation

Kubelet is the node-level reconciler. It watches pods assigned to its node, prepares the sandbox, calls CRI/CNI/CSI/device plugins, runs probes, reports status, and enforces node-pressure decisions.

Area	What it does	Failure mode
Pod lifecycle sync loop	Reconciles desired pod specs against containers on the node.	Pod status lags, restart loops, stale runtime state.
PLEG	Detects runtime pod/container changes and feeds kubelet sync.	PLEG is not healthy; kubelet stops making progress on pod state.
CRI calls	Asks containerd/CRI-O to create sandboxes and containers.	Sandbox creation, image pull, runtime hook, or container start failures.
CNI invocation	Configures the pod sandbox network.	ContainerCreating with network setup errors, IPAM exhaustion, policy, or routing failures.
CSI mount path	Coordinates attach, mount, and projected volumes.	ContainerCreating due to attach/mount, secret/configmap projection, or permissions.
Device manager	Allocates extended resources from device plugins.	GPU requested but not visible, stale allocatable, plugin socket issues.
Probe workers	Execute startup, readiness, and liveness checks.	Good process marked unready, bad process kept in service, restart storms.
Eviction manager	Enforces memory, disk, inode, PID, and node-pressure thresholds.	Pods evicted even when app-level metrics look normal.
CPU/topology managers	Align CPU/device/NUMA allocation when configured.	Latency variance or poor accelerator locality.

flowchart TD
  Watch[Watch assigned pod] --> Sync[Kubelet sync loop]
  Sync --> Volumes[Prepare volumes and projected tokens]
  Sync --> Devices[Allocate devices]
  Sync --> Sandbox[Create pod sandbox through CRI]
  Sandbox --> CNI[Configure network through CNI]
  CNI --> Containers[Start init and app containers]
  Containers --> Probes[Run startup, readiness, liveness probes]
  Probes --> Status[Report pod and node status]
  Status --> Endpoints[Readiness drives EndpointSlice]
  Sync --> Eviction[Eviction manager watches node pressure]

Hard kubelet debugging:

• ContainerCreating is a phase, not a root cause. Split image pull, volume mount, sandbox, CNI, device allocation, and runtime hook.
• CrashLoopBackOff means the process started and exited. Use previous logs and exit code before blaming Kubernetes.
• Running does not mean serving. Readiness gates, sidecars, model warmup, and EndpointSlice propagation still matter.
• Node Ready can be true while a device plugin, CSI plugin, or CNI agent is broken.
• Kubelet status can lag when the runtime is overloaded or PLEG is unhealthy.

GPU-specific kubelet path:

1. Scheduler binds pod to a node with allocatable GPU.
2. Kubelet asks the device plugin for allocation.
3. Device plugin returns device IDs and environment/mount/runtime hints.
4. Runtime hook exposes GPU devices/libraries into the container.
5. Application initializes CUDA and loads model memory.

If any step fails, the symptom can be different: pending pod, ContainerCreating, container start error, app-level CUDA error, or runtime crash.

Etcd Tuning And Failure Modes

Etcd matters because API server responsiveness depends heavily on etcd health. Slow durable writes, leader churn, member network latency, database growth, and compaction pressure surface as slow creates, updates, leader election renewals, endpoint updates, and controller lag.

Know these terms:

Topic	Staff-level explanation
Raft consensus	Etcd commits writes through a leader and quorum. Losing quorum makes writes unavailable. Slow followers or unstable network paths can increase commit latency and leader elections.
Fsync latency	Every durable commit depends on disk behavior. Slow disks become API server latency.
Compaction	Removes old revision history no longer needed by watchers. If watchers fall behind past compacted revisions, they must relist.
Defragmentation	Reclaims database file space after compaction. It is operationally useful but must be done carefully, member by member.
Snapshot	Point-in-time backup of etcd state. A backup is not real until restore has been practiced.
Watch pressure	High object churn creates watch load on API servers and clients. Controllers that relist constantly can amplify control-plane load.
Alarm state	No-space or corruption alarms can block normal operation until the root issue and alarm are handled.
Member changes	Adding/removing members changes quorum math; sloppy automation can turn degradation into outage.

flowchart TD
  Write[API server write] --> Leader[etcd leader]
  Leader --> WAL[WAL and fsync]
  Leader --> Replicas[Replicate to followers]
  Replicas --> Quorum{Quorum ack}
  WAL --> Quorum
  Quorum --> Commit[Committed revision]
  Commit --> Watch[Watch delivery]
  Watch --> Controllers[Controllers react]

Incident: etcd latency is killing deploys.

1. Freeze nonessential deploys, autoscaling experiments, and noisy automation.
2. Check API server storage latency, etcd leader changes, member health, fsync latency, DB size, and network between members.
3. Check whether serving traffic is actually impacted or only operations are impaired.
4. Protect quorum. Do not restart multiple members casually.
5. If database size is high, plan compaction and member-by-member defrag using the cluster’s operational runbook.
6. If restore is required, restore from a known snapshot, validate revision behavior, and restart API servers/controllers in the tested order.
7. Prevent recurrence with disk latency alerts, object churn budgets, webhook/controller rate limits, and backup restore drills.

Advanced restore pitfalls:

• Restoring an old snapshot can confuse controllers and clients that observed newer revisions.
• Restoring with mismatched certificates, peer URLs, or member names can leave API servers unable to reconnect.
• Restoring only one member of a multi-member cluster incorrectly can break quorum expectations.
• A snapshot that was never restored in staging is only a hope, not a backup.
• After restore, validate API server health, controller convergence, node status updates, and critical add-ons before reopening deploy automation.

Strong answer:

Etcd is not just storage. It is the latency floor for Kubernetes writes and the source of watch revisions. I would protect quorum first, reduce write amplification, measure fsync and leader stability, then use compaction/defrag or restore only through a practiced runbook.

Admission Chain

Admission sits between authorization and persistence. It is powerful because it can enforce platform contracts, but dangerous because it is inline with API writes.

Admission type	Purpose	Risk
Mutating webhook	Defaults or rewrites the incoming object before validation.	Ordering and reinvocation surprises; mutation loops; hidden behavior.
Validating webhook	Accepts or rejects the final object.	Deployment outage if slow, unavailable, or scoped too broadly.
ValidatingAdmissionPolicy	Native policy path for many validation use cases.	Less flexible than arbitrary webhooks, but operationally simpler when sufficient.

Key details:

• Mutating admission runs before validating admission.
• Webhooks have timeout and failure policy behavior that directly affects API writes.
• failurePolicy: Fail protects safety but can turn webhook downtime into cluster write downtime.
• failurePolicy: Ignore improves availability but can allow unsafe objects during failure.
• Webhooks should use tight namespace/object selectors.
• Webhooks must avoid calling back into APIs that create dependency loops.
• Monitor webhook latency, rejection rate, timeout rate, and endpoint health.

flowchart LR
  Request[API write] --> Auth[Authn/Authz]
  Auth --> Mutating[Mutating admission]
  Mutating --> Validate[Schema validation]
  Validate --> Validating[Validating admission]
  Validating --> Store[Persist to etcd]
  Validating -- reject or timeout --> Fail[Request fails or ignores by policy]

Debug: a slow webhook is killing cluster throughput.

1. Confirm API server latency buckets split by verb/resource and admission metrics.
2. Identify webhook name, operation, resource, namespace selector, object selector, timeout, and failure policy.
3. Check webhook endpoints, TLS/cert validity, DNS, service routing, and pod saturation.
4. Look for broad scope: all pods, all namespaces, all updates, or status subresources.
5. Temporarily halt rollouts or scale webhook backends if safe.
6. Use a documented emergency patch only if the risk of blocked writes exceeds the risk of bypass.
7. Fix with narrow selectors, low timeout, capacity, dry-run tests, canary rollout, and SLO alerts.

Interview line:

A webhook is on the API server’s critical path. I debug it like a production dependency, not like a passive policy file.

Scheduler Extenders Vs Scheduling Framework Plugins

The default scheduler is a framework with extension points. Modern custom scheduling should usually use scheduling framework plugins when you need deep scheduler behavior.

Approach	How it works	Tradeoff
Scheduler extender	The scheduler calls an external HTTP service for filtering/prioritization decisions.	Easier separation, but network calls add latency and failure modes.
Scheduling framework plugin	Code runs inside the scheduler process at extension points such as QueueSort, PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, PreBind, Bind, PostBind.	Lower-latency and deeper integration, but requires scheduler build/config discipline.

flowchart LR
  QueueSort[QueueSort] --> PreFilter[PreFilter]
  PreFilter --> Filter[Filter]
  Filter --> PostFilter[PostFilter or preemption]
  Filter --> PreScore[PreScore]
  PreScore --> Score[Score]
  Score --> Reserve[Reserve]
  Reserve --> Permit[Permit]
  Permit --> PreBind[PreBind]
  PreBind --> Bind[Bind]
  Bind --> PostBind[PostBind]

GPU scheduling examples:

• Filter nodes without the required GPU SKU, MIG profile, topology, or driver/runtime label.
• Score nodes by least fragmentation, thermal headroom, or locality.
• Reserve scarce accelerator topology before binding.
• Permit pods for gang/batch semantics when partial placement is harmful.

Hard question:

Why not just use a scheduler extender?

Answer:

I would avoid an extender for high-throughput scheduling paths unless there is a strong reason. HTTP callbacks add latency, availability dependencies, and serialization overhead. A scheduling framework plugin is the better fit when the logic must participate directly in filter/score/reserve/bind semantics.

Topology-Aware Scheduling And Routing

Do not mix these up:

Concept	Decides	Example
Node affinity	Which nodes a pod can or should run on.	Require GPU SKU or zone.
Pod affinity/anti-affinity	Placement relative to other pods.	Separate replicas across nodes or zones.
Topology spread constraints	Evenly distribute pods across topology domains.	Spread inference replicas across zones.
Topology Manager	Kubelet-level alignment of CPU/device/NUMA resources.	Align CPU cores and GPU/NIC topology on a node.
Topology Aware Routing	Prefer endpoints close to the client when routing Service traffic.	Keep same-zone traffic local when endpoint distribution permits.

For AI inference:

• Placement topology affects latency, GPU locality, and blast radius.
• Routing topology affects cross-zone cost and p99.
• Storage topology affects PVC binding and attach behavior.
• Accelerator topology affects NVLink, PCIe, NUMA, and multi-GPU efficiency.

Senior answer:

I separate scheduling locality from routing locality. A pod being in the right zone does not guarantee traffic stays in-zone; EndpointSlice hints, Gateway/load balancer behavior, mesh policy, and readiness all matter.

Quotas, LimitRanges, Priority, And Preemption

At scale, quotas are admission-time control, not just billing metadata.

Control	What it does	Interview trap
ResourceQuota	Caps namespace resource consumption and object counts.	Burst scaling can fail at admission even if cluster nodes have capacity.
LimitRange	Defaults or bounds per-object requests/limits.	Defaults can accidentally make every pod request too much or too little.
PriorityClass	Defines pod priority for scheduling and preemption.	High priority does not bypass all constraints, quotas, PDBs, or shape mismatch.
Preemption	Scheduler may evict lower-priority pods to fit a higher-priority pod.	Victims may be found but fit can still fail due to topology, affinity, PVC, or GPU shape.

Debug: burst scale fails under quota.

1. Check events for exceeded quota or LimitRange errors.
2. Compare requested resources to quota hard/used values.
3. Check whether pending pods already consumed quota through admission.
4. Inspect HPA/KEDA/autoscaler behavior and whether it keeps submitting impossible pods.
5. Use priority and separate namespaces for critical serving vs opportunistic batch.
6. Prevent with quota headroom dashboards, per-workload classes, and admission messages humans can act on.

Strong answer:

Quota is part of the control plane write path. A cluster can have spare nodes and still reject pods because namespace policy says the tenant has exhausted its budget.

CNI Internals And Packet Tracing

Kubernetes requires every pod to have network connectivity, but the implementation belongs to the CNI plugin. Calico, Cilium, Flannel, and cloud CNIs differ materially.

Common Linux mechanics:

• Pod network namespace.
• Veth pair connecting pod namespace to host namespace or datapath.
• Routes for pod CIDRs or overlay tunnels.
• Service load balancing through iptables/IPVS/eBPF depending on implementation.
• NetworkPolicy enforcement through iptables, eBPF, or datapath-specific policy engines.
• Conntrack or conntrack-like state for NAT/load balancing paths.

Datapath modes to compare:

Mode	How traffic is usually handled	Debug angle
Flannel-style overlay	Simple overlay routes pod traffic across nodes, often VXLAN.	Check pod CIDR routes, overlay interface, MTU, and encapsulation.
Calico routed/BGP	Routes pod CIDRs directly or through BGP; policy depends on mode.	Check BGP sessions, routes, policy rules, IP pools, node-to-node mesh.
kube-proxy iptables	Services translate through iptables rules.	Inspect iptables-save; rule scale can make debugging noisy in large clusters.
kube-proxy IPVS	Services translate through IPVS virtual servers.	Inspect ipvsadm, backend health, and conntrack/NAT interactions.
Cilium eBPF	Services, policy, and observability use eBPF programs/maps.	Inspect Cilium status, identities, maps, Hubble flows, and proxy redirection.
Cloud CNI	Pods use provider-specific IP allocation and datapath behavior.	Inspect ENI/IP allocation, security groups, node IP capacity, provider controller.

flowchart LR
  App[Container process] --> PodNS[Pod network namespace]
  PodNS --> Veth[veth pair]
  Veth --> HostNS[Host network namespace]
  HostNS --> CNI[CNI datapath]
  CNI --> Service[Service LB: iptables, IPVS, or eBPF]
  Service --> RemoteNode[Remote node]
  RemoteNode --> RemoteVeth[Remote veth]
  RemoteVeth --> RemotePod[Destination pod]

Packet trace: pod A to pod B across nodes.

1. Resolve Service DNS to ClusterIP or connect directly to pod IP.
2. From source pod: confirm route, DNS, and local socket state.
3. On source node: capture veth/host interface and inspect CNI policy decision.
4. Check service translation path: iptables/IPVS/eBPF maps depending on CNI/kube-proxy mode.
5. Inspect conntrack state if NAT is involved.
6. On destination node: capture ingress interface and destination veth.
7. Confirm destination pod listener, readiness, sidecar/mesh interception, and NetworkPolicy.

Commands to have ready:

kubectl get pod -o wide
kubectl get svc,endpointslices
kubectl exec -it <pod> -- ip route
kubectl exec -it <pod> -- ss -tanp
tcpdump -i any host <pod-or-node-ip>
conntrack -S
iptables-save | grep <service-or-pod-ip>
bpftool map show
bpftrace -l 'tracepoint:syscalls:*connect*'

Staff answer:

I start with the logical path, then prove the physical datapath. With Cilium I inspect eBPF maps and Hubble flows; with Calico I expect routes, policy rules, and possibly BGP; with Flannel I expect a simpler overlay with less policy machinery.

eBPF For Kubernetes

eBPF lets the kernel run verified programs at specific hook points without loading arbitrary kernel modules. In Kubernetes, eBPF commonly shows up in networking, service load balancing, network policy, observability, security, and sometimes sidecar acceleration.

Know the building blocks:

eBPF concept	What it means in practice
Program	Verified bytecode attached to a kernel hook.
Verifier	Kernel safety checker that rejects unsafe programs before load.
Maps	Kernel-resident key/value state shared between eBPF programs and user space.
Tail calls	Jump from one eBPF program to another to compose larger datapaths.
Helper calls	Limited kernel helper functions exposed to eBPF programs.
CO-RE/BTF	Compile once, run everywhere style portability using kernel type info.
XDP	Very early packet hook at the NIC driver path. Useful for fast drop/load balancing.
TC	Traffic control hook, commonly used for pod/node networking paths.
Cgroup hooks	Enforce or observe socket and process behavior at cgroup boundaries.
Kprobes/tracepoints	Kernel instrumentation hooks for observability/debugging.
Uprobes	User-space function instrumentation.

flowchart TD
  Packet[Packet arrives] --> XDP[XDP hook]
  XDP --> TCIngress[TC ingress]
  TCIngress --> Stack[Kernel network stack]
  Stack --> Cgroup[Cgroup socket hooks]
  Cgroup --> App[Pod process]
  App --> Uprobe[Optional uprobes]
  Stack --> Trace[Tracepoints and kprobes]
  XDP --> Maps[eBPF maps]
  TCIngress --> Maps
  Cgroup --> Maps
  Maps --> Agent[User-space agent]
  Agent --> Metrics[Metrics, flow logs, policy state]

Kubernetes use cases:

• Service load balancing without kube-proxy iptables/IPVS.
• NetworkPolicy enforcement at pod/node boundaries.
• Pod-to-pod flow visibility without full packet capture everywhere.
• L7 visibility or policy when integrated with Envoy or another proxy path.
• DNS visibility and policy.
• Transparent encryption or identity-aware policy in some datapaths.
• Runtime security signals from syscalls, file access, process execs, and network connects.
• Low-overhead profiling and latency attribution.

Cilium-specific interview points:

• Cilium uses eBPF maps to represent service backends, identities, policy, and connection state.
• Hubble surfaces flow visibility from the datapath.
• Cilium can replace kube-proxy for Service load balancing.
• Cilium NetworkPolicy extends native NetworkPolicy with richer L3-L7 semantics.
• Cilium Service Mesh can combine eBPF L4 behavior with Envoy for L7.
• Debugging requires knowing whether a packet stayed in eBPF fast path or was redirected to a proxy.

Hard truth:

eBPF improves visibility and can reduce iptables complexity, but it does not delete distributed systems problems. You still debug DNS, MTU, conntrack-like state, policy identity, endpoint readiness, route propagation, and proxy behavior.

eBPF Debugging Flow

Use this when someone asks how you trace a packet in a Cilium/eBPF cluster.

1. Confirm endpoint identity and policy: source pod, destination pod, namespace, labels, security identity.
2. Check Cilium agent health and node datapath status.
3. Use Hubble or flow logs to see whether traffic is forwarded, dropped, denied by policy, or redirected.
4. Inspect Service translation: ClusterIP, backend selection, session affinity, topology behavior.
5. Inspect eBPF maps for services, endpoints, policy, and connection tracking if needed.
6. Fall back to tcpdump at pod veth and host interfaces to prove where packets disappear.
7. Check whether traffic enters Envoy or another L7 proxy path.
8. Correlate with DNS, endpoint readiness, and application listener state.

Commands and tools to recognize:

cilium status
cilium connectivity test
cilium monitor
hubble observe
bpftool prog show
bpftool map show
bpftool map dump pinned <path>
tc filter show dev <iface> ingress
ip link show
tcpdump -i any host <ip>

Common eBPF failure modes:

Failure	What it looks like	What to check
Agent unhealthy	Policies or services stop updating correctly.	Cilium agent status, logs, Kubernetes watches, API connectivity.
Map pressure	New flows/services/policies fail or behave inconsistently.	Map capacity metrics, service/backend count, policy scale.
Identity mismatch	Policy denies traffic that labels appear to allow.	Endpoint identity, label sync, policy selectors, namespace labels.
Proxy redirection issue	L3/L4 path works but L7 traffic fails.	Envoy config, listener state, certs, policy, timeout.
Kernel feature mismatch	Datapath feature fails on one node pool.	Kernel version, Cilium mode, enabled features, node image.
MTU/tunnel issue	Small packets work, large requests fail.	Encapsulation mode, path MTU, tcpdump, fragmentation symptoms.

Interview line:

For eBPF, I ask where the program is attached, what map state it reads, what identity/policy it applied, and whether user space successfully updated that state. That is more useful than saying “Cilium is broken.”

Service Mesh Data Plane

Service mesh is not free. It adds identity, mTLS, policy, telemetry, retries, traffic shaping, and L7 routing, but it can also add latency, CPU, memory, debugging complexity, and retry amplification.

Architecture	Data path	Tradeoff
Sidecar mesh	Each pod gets a local proxy such as Envoy.	Strong per-workload L7 control, but extra containers and hop overhead per pod.
Ambient or sidecarless mesh	Node-level or shared data-plane components handle L4 security, with optional waypoint/L7 proxies.	Less per-pod sidecar overhead, but different failure domains and debugging model.
Cilium service mesh	eBPF datapath for L3/L4 and Envoy for L7 features.	Tight network integration, but you must know when traffic enters proxy paths.

mTLS bootstrapping concepts:

• Workload identity comes from service account, trust domain, SPIFFE-like identity, or mesh identity mapping.
• Certificates are issued/rotated by the mesh control plane or CA integration.
• Proxies or dataplane agents enforce peer authentication and authorization.
• Failure modes include stale certs, clock skew, trust domain mismatch, policy denial, and waypoint/proxy saturation.

GPU inference cautions:

• L7 proxying can hurt streaming and long-lived responses.
• Retries can multiply expensive inference work.
• Timeouts must propagate from client to gateway to model server.
• Mesh telemetry cardinality can explode with tenant/model labels.

CSI Internals And Volume Lifecycle

CSI decouples Kubernetes from storage vendors through gRPC services.

CSI service	Typical responsibility
Controller service	Create/delete volumes, publish/unpublish volumes, snapshot/clone, attach/detach where supported.
Node service	Stage/unstage and publish/unpublish volumes on the node.
Identity service	Plugin identity and capability discovery.

Volume lifecycle:

flowchart TD
  PVC[PVC] --> Provision[External provisioner creates volume]
  Provision --> PV[PV bound]
  PV --> Schedule[Pod scheduling considers binding/topology]
  Schedule --> Attach[Attach via VolumeAttachment when needed]
  Attach --> Stage[NodeStageVolume]
  Stage --> Publish[NodePublishVolume into pod]
  Publish --> Running[Pod running]
  Running --> Unpublish[NodeUnpublishVolume]
  Unpublish --> Detach[Detach and cleanup]

Node failure race:

• The pod disappears from one node slowly or the node becomes unreachable.
• The volume may still be attached at the cloud/storage layer.
• The attach-detach controller and CSI external-attacher coordinate VolumeAttachment state.
• A replacement pod can be stuck because the volume is still attached to the dead node.
• Force detach may risk data corruption unless the storage system guarantees safety.

RWO vs RWX:

Mode	Meaning	Interview use
RWO	ReadWriteOnce: writable by one node at a time.	Common block volume for one writer; replacement after node failure can wait for detach.
RWX	ReadWriteMany: writable by many nodes.	Shared filesystems; useful for shared artifacts but can bottleneck or weaken isolation.
ROX	ReadOnlyMany: read-only from many nodes.	Safer shared model artifact pattern when artifacts are immutable.
RWOP	ReadWriteOncePod: writable by one pod.	Stronger single-pod writer semantics when supported.

CRI, Pod Sandboxes, Namespaces, And Cgroups

The kubelet does not directly run containers. It talks to the container runtime through CRI. The runtime creates a pod sandbox and containers using Linux primitives.

Primitive	What it isolates or controls
PID namespace	Process tree visibility.
Network namespace	Interfaces, routes, ports, conntrack view.
Mount namespace	Filesystem view.
IPC namespace	Shared memory and IPC.
UTS namespace	Hostname/domain identity.
User namespace	UID/GID mapping where enabled.
Cgroups	CPU, memory, IO, pids, and device resource accounting/enforcement.
Seccomp/AppArmor/SELinux	Syscall and mandatory access controls.

Pod sandbox details:

• The sandbox holds the shared pod-level namespaces, especially network.
• App containers join the pod sandbox.
• CNI configures networking for the sandbox.
• CSI mounts volumes before or during container start.
• Device plugins and runtime hooks expose GPUs or other devices.

Staff answer:

A pod is not a VM. It is a set of processes sharing selected namespaces and controlled by cgroups, with Kubernetes coordinating runtime, network, storage, and device setup.

Workload Identity And Federation

Kubernetes service accounts provide workload identity inside the cluster. Modern clusters often federate that identity to cloud IAM or external identity providers using projected tokens and OIDC trust.

Know the pattern:

1. Pod runs as a Kubernetes service account.
2. Kubelet mounts a projected, bounded service account token.
3. External IAM trusts the cluster issuer and token claims.
4. Workload exchanges the token for cloud credentials or calls a broker.
5. RBAC controls Kubernetes API access; cloud IAM controls external resource access.

Failure modes:

• Wrong service account on pod.
• Token audience mismatch.
• Expired or stale token.
• OIDC issuer or JWKS not reachable.
• Trust policy too broad.
• Namespace/name reuse grants unintended access.
• Secret fallback remains mounted and bypasses intended federation.

Senior answer:

I separate Kubernetes RBAC from external IAM. A service account can be tightly scoped inside Kubernetes but dangerously powerful in cloud IAM if federation claims are broad.

Controllers And Operators

Controllers are reconciliation loops. Operators encode domain-specific operations as controllers over custom resources.

Core mechanics:

• Informer watches API objects and maintains a local cache.
• Workqueue stores reconciliation keys.
• Reconcile reads desired and observed state, then converges.
• Status subresource reports observed state and conditions.
• OwnerReferences drive garbage collection of dependents.
• Finalizers block deletion until cleanup completes.
• Leader election avoids multiple active reconcilers when needed.
• Rate-limited requeues and exponential backoff prevent hot loops.

flowchart TD
  Watch[Informer watch] --> Cache[Local cache]
  Cache --> Queue[Rate-limited workqueue]
  Queue --> Reconcile[Reconcile key]
  Reconcile --> Desired[Read desired state]
  Reconcile --> Observed[Read observed state]
  Desired --> Diff[Compute diff]
  Observed --> Diff
  Diff --> Act[Create, update, delete, or no-op]
  Act --> Status[Update status and conditions]
  Act --> Requeue[Requeue on transient failure]

Finalizer trap:

A finalizer is not cleanup itself. It is a deletion gate. If the controller that removes it is broken, objects can be stuck terminating indefinitely.

Operator interview answer:

I would write an operator when the lifecycle needs domain knowledge: ordered rollout, health validation, backup/restore, failover, or external resource cleanup. I would not write one just to template YAML.

GitOps And Operator Patterns

GitOps and operators are both reconcilers. The failure mode is controller conflict.

Patterns:

• GitOps owns desired manifests.
• Operators own generated children and status.
• Humans avoid editing live objects that GitOps owns.
• Emergency changes are either committed to Git or done through an explicit sync pause.
• CRDs need versioning, conversion, backup, and rollback planning.

Conflict examples:

• GitOps keeps reverting manual hotfix.
• Helm chart renders a field that an operator mutates.
• Two controllers fight over labels/annotations.
• CRD upgrade removes a field still used by live resources.
• Finalizer blocks deletion after GitOps prunes the controller first.

Staff rule:

Every field should have one owner. If two reconcilers own the same field, drift is a symptom of design failure, not just operator noise.

Cascading Failures And Backoffs

Kubernetes failure modes often cascade through retries and controllers.

Examples:

• API server slow -> controllers retry -> more API load -> API server slower.
• Webhook slow -> rollouts block -> HPA creates more writes -> queue grows.
• DNS slow -> clients retry -> CoreDNS and gateway overload.
• Model server p99 high -> clients retry expensive requests -> GPU queues collapse.
• Node pressure -> evictions -> rescheduling -> image pulls/artifact downloads spike.
• CSI attach delay -> pods pending -> autoscaler adds nodes that do not fix attach bottleneck.

Mitigations:

• Use exponential backoff with jitter.
• Set client deadlines and retry budgets.
• Apply API Priority and Fairness for control-plane protection.
• Put circuit breakers at gateways and clients.
• Add queue limits and load shedding.
• Pause automation when control-plane health degrades.
• Prefer idempotent controllers and bounded reconciliation fan-out.

Observability For Kubernetes Internals

You need more than Prometheus and Grafana deployed. You need the right signals and labels.

Layer	Signals
API server	Request rate/latency by verb/resource, inflight requests, admission latency, APF queues, watch count, error rate.
Etcd	Leader changes, proposal failures, fsync latency, DB size, network latency, compaction/defrag history.
Scheduler	Pending queue, scheduling attempts, unschedulable reasons, plugin latency, preemption attempts.
Controller manager	Workqueue depth, reconcile latency, requeues, leader election, client-go throttling.
Kubelet	Pod startup latency, PLEG/runtime errors, image pull time, volume mount time, node pressure, probe failures.
CNI	Policy drops, flow logs, conntrack, DNS, service translation, eBPF map pressure where relevant.
CSI	Provision, attach, mount latency, VolumeAttachment state, node plugin errors.
Workloads	SLO, readiness, retries, saturation, queue depth, model/GPU metrics.

Troubleshooting rule:

Events tell you what Kubernetes tried. Metrics tell you how the system behaved. Logs explain local decisions. Traces connect the request path. You need all four for hard incidents.

Multi-Tenancy

Multi-tenancy is not just namespaces. It is identity, policy, resource fairness, network isolation, operational boundaries, and blast-radius control.

Area	Guardrail
Identity	Separate service accounts, cloud identities, and trust policies per tenant/workload class.
RBAC	Least privilege by namespace and API group. Avoid shared powerful automation identities.
Network	Default-deny NetworkPolicy where practical; explicit ingress/egress contracts.
Resources	ResourceQuota, LimitRange, PriorityClass, and workload-class-specific pools.
Admission	Enforce labels, owners, image policy, resource requests, GPU pool selectors, and security posture.
Runtime	Pod Security Standards, seccomp, non-root, read-only root filesystem where possible.
Data	Separate secrets, PVCs, artifact permissions, and external IAM scopes.
Observability	Tenant labels with bounded cardinality, per-tenant SLOs, and quota dashboards.

GPU-specific multi-tenancy:

• MIG can provide stronger hardware partitioning where supported.
• Time-slicing and MPS improve utilization but require trust and noisy-neighbor analysis.
• Shared model servers need tenant-level admission, fairness, and request shape limits.
• Debug access to GPU nodes should be audited and scoped.

Hard Interview Drills

What happens if etcd is slow but existing pods are serving?

Separate operations-plane impact from data-plane impact. Existing pods and Services may continue serving, but rollouts, scaling, endpoint changes, leader elections, and status updates degrade. I would freeze nonessential writes, check etcd quorum/fsync/leader changes, inspect API storage latency, and avoid making the control plane worse with restart storms.

How do you debug a webhook that blocks all pod creates?

Find the webhook from API server admission metrics/events, inspect failurePolicy, timeout, operations, resources, selectors, service endpoints, certs, DNS, and backend saturation. Mitigate by pausing rollouts, scaling/fixing the webhook, or using a pre-approved emergency scope/failure-policy patch if the availability risk exceeds the policy risk.

Why might a high-priority GPU pod not preempt successfully?

Preemption cannot solve every fit problem. The pod may require a specific GPU count on one node, MIG profile, zone, PVC topology, anti-affinity, or taint/toleration. PDBs may constrain victims. Quota can block admission before scheduling. A scheduler event saying preemption was attempted is not proof the pod can fit.

How do you trace external traffic to a pod?

Start at DNS/load balancer, then Gateway/Ingress, mesh or CNI datapath, Service, EndpointSlice, node, veth, pod listener, and readiness. At each boundary check config and evidence: route objects, endpoint contents, policy, flow logs, tcpdump, conntrack/eBPF maps, proxy access logs, and application logs.

Why etcd instead of a relational database?

Kubernetes needs a strongly consistent, highly available key-value store with watch semantics and revisioned state for distributed controllers. Etcd’s Raft-based consistency, compare-and-swap behavior, leases, and watches fit the API server/controller model. SQL features are less important than consistent object storage and watchable revisions.

How would you write a controller safely?

Use informers and a rate-limited workqueue, reconcile idempotently from observed to desired state, update status conditions, use finalizers only for required cleanup, avoid owning fields controlled by other reconcilers, bound external calls with deadlines, emit events, expose metrics, and test deletion, retry, and upgrade paths.

Staff-Level Close

The interview bar is not naming components. It is explaining causality:

Kubernetes stores desired state in etcd, exposes it through the API server, and relies on reconcilers to converge the world. When something fails, I identify which reconciliation loop is blocked, which dependency is slow, which policy rejected the change, and whether the data plane or only the control plane is impacted.