Kubernetes Skill

Three domains: debugging (pod triage, networking, resources), security (RBAC, pod hardening, network isolation, supply chain), and cobaltcore (KVM exporter, hypervisor metrics). Select by request signal, then follow the phases below.

Always specify -n <namespace> in every kubectl command. Use read-only commands to gather evidence before proposing changes.

---

Domain Selection

Signal	Domain
CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending	Debugging
Service unreachable, DNS failure, port-forward	Debugging (network)
CPU throttling, memory limit, disk pressure	Debugging (resources)
RBAC, permissions, roles, ServiceAccount	Security (access)
Pod hardening, container security, PodSecurity	Security (pods)
NetworkPolicy, default-deny, namespace isolation	Security (network)
Image signing, secrets, admission control	Security (supply chain)
KVM exporter, cobaltcore, hypervisor metrics	Cobaltcore

---

Phase 1: TRIAGE

Debugging Triage Flow

Follow this sequence for every pod or workload issue. Do not skip steps -- many failures are only visible in events and describe output, not in logs.

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
kubectl exec -it <pod-name> -n <namespace> -c <container-name> -- /bin/sh

Always check --previous logs for crashed containers before current logs -- restarting destroys them permanently.

Diagnosis routing:

Symptom	Action
CrashLoopBackOff, ImagePullBackOff, Pending, FailedScheduling	Check describe output for events, previous logs, image pull errors
Service unreachable, DNS failure	Check service endpoints, CoreDNS, NetworkPolicy below
CPU throttling, OOMKill, disk pressure	Check resource limits, requests vs actual, node capacity
"no endpoints available for service"	Compare svc selector with pod labels

Network debugging:

# Verify service has endpoints
kubectl get endpoints <service-name> -n <namespace>
# DNS lookup from inside cluster
kubectl run dns-debug --rm -it --restart=Never --image=busybox:1.36 -n <namespace> -- \
  nslookup <service-name>.<namespace>.svc.cluster.local
# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Port-forward for local testing
kubectl port-forward svc/<service-name> -n <namespace> 8080:80

Security Domain Selection

For security requests, provide concrete YAML manifests from the patterns below. Reference-backed specifics, not generic advice.

RBAC patterns: Prefer namespace-scoped Roles over ClusterRoles. Write exact verbs and resources. Create dedicated ServiceAccounts per workload. Set automountServiceAccountToken: false on pods that need no API access.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app-team
  name: deployment-reader
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

Pod security: Enforce PodSecurity labels at namespace level. All containers: runAsNonRoot: true, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, capabilities: drop: ["ALL"]. Use distroless base images. Pin image digests.

labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/warn: restricted

Network policies: Start with default-deny for ingress and egress. Add allow-list rules per service. Always allow DNS egress (UDP/TCP 53).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Cobaltcore Domain

Components: KVM Exporter (cobaltcore-dev/kvm-exporter). Load references/cobalt-kvm-exporter.md for architecture, metric catalogs, configuration, and deployment. Pair with go-patterns for code, prometheus-grafana-engineer for metrics.

---

Phase 2: DIAGNOSE / RESPOND

Debugging: Follow the triage flow. Gather evidence with read-only commands before proposing changes.

Security: Provide copy-paste-ready YAML using the patterns in Phase 1.

Cobaltcore: Use component-specific references for architecture, metrics, concurrency patterns, and testing.

---

Phase 3: VERIFY

• Debugging: Confirm the fix resolves the symptom with the same triage commands.
• Security: Validate against the PodSecurity standards and RBAC least-privilege patterns above.
• Cobaltcore: Verify against component test patterns in references/cobalt-testing-patterns.md.

---

Deep References

Signal	Reference	Content
KVM exporter architecture, metrics, config	references/cobalt-kvm-exporter.md	Full component reference (463 lines)
Cobaltcore concurrency, goroutine, semaphore	references/cobalt-concurrency-patterns.md	Go concurrency patterns (268 lines)
Cobaltcore testing, mock, Kind cluster	references/cobalt-testing-patterns.md	Testing strategies (271 lines)