33 investigation skills.

Diagnose CrashLoopBackOff — exit code discrimination (1, 127, 137, 139, 143), restart history, recent config changes, resource limits, image and entrypoint misconfigurations. Reads previous container logs, not current, since the container exits before writing.

CrashLoopBackOff · container keeps restarting · restart count

Diagnose OOMKilled — memory usage vs. limits, VPA recommendations, recent usage spikes, distinguishing limit misconfiguration from an actual leak. Kubernetes 1.36 sidecar-aware: accounts for sidecar containers separately.

OOMKilled · exit code 137 · memory · container killed

Diagnose Pending pods — node capacity, resource quotas, taints/tolerations, affinity rules, PVC binding. Also covers Dynamic Resource Allocation failures (Kubernetes 1.36) and admission policy rejections with drift attribution.

Pending pod · FailedScheduling · unschedulable

Structured evidence gathering for any Kubernetes alert or service degradation — workload status, events, recent logs, recent manifest changes, and metrics. The general-purpose entry point when no specific failure mode skill matches.

Any K8s alert · pod down · service degraded

Detect and diagnose deployment regressions — metrics degraded after a rollout, pre/post deploy comparison, rollback plan generation. Answers: did this deployment cause the problem, or was it already happening?

Metrics worse after deploy · latency increased · error rate up after rollout

Diagnose service connectivity failures — NetworkPolicy blocking, service endpoints, DNS resolution, Istio/Envoy proxy configuration. Covers the cases where a pod is running fine but can't be reached or can't reach its dependencies.

Service unreachable · DNS failure · connection refused · NetworkPolicy

Generate a GitOps PR to fix an identified Kubernetes issue — resolve ArgoCD/Flux ownership from annotations, generate a minimal patch, create the branch and PR with evidence summary and rollback plan. Never applies directly to the cluster.

Called by k8s-fixer after hypothesis is confirmed

Kubernetes security hardening checklist covering Pod Security Admission, network policies, RBAC least privilege, image signing, and secrets management. Covers the controls that are actually violated in production, not just the ones that look good in a benchmark.

/k8s-harden · /k8s-security-scan

Validate a runbook against live cluster state — tests commands, verifies resource names still exist, checks permissions, flags stale content and gaps. Produces a scored report on what's accurate, what's outdated, and what's missing.

/runbook-check · validate runbook

Step-by-step process for reviewing and tightening IAM and RBAC permissions to least privilege — identifies overly broad roles, wildcards, and permissions that haven't been used in 90 days.

/iam-review · RBAC audit · permission tightening

Change management checklist with risk rating, rollback plan, communication plan, approval tracking, and monitoring requirements. Ensures no production change goes out without a rollback path and a clear owner.

/change-review · production change planning

Protocol for investigating ALB 5xx errors — target health check status, access log analysis, WAF rule evaluation, SSL/TLS certificate issues, and backend connection pool exhaustion.

ALB 5xx · load balancer errors · 502 503 504

Protocol for triaging ECS task failures, service instability, and container errors — task stopped events, placement failures, container exit codes, service event history, and CloudWatch log correlation.

ECS task failure · service unhealthy · task stopped

Protocol for debugging Lambda errors including timeouts, OOM, cold starts, concurrency limits, and DLQ analysis. Correlates CloudWatch metrics with X-Ray traces and dead-letter queue depth.

Lambda timeout · Lambda OOM · cold start · Lambda error

Protocol for triaging RDS issues — connection exhaustion, CPU spikes, slow queries, storage fill alerts. Distinguishes between client-side connection pool misconfiguration and server-side resource exhaustion.

RDS CPU · RDS connections · slow queries · RDS storage

Protocol for investigating sudden AWS cost spikes — service breakdown, anomaly detection, resource attribution, and owner identification. Answers: what changed, when, and who owns the resource causing the spike.

AWS cost spike · unexpected billing · cost anomaly

Multi-window multi-burn-rate alert formula for SLOs — the math explained and PromQL generated. Covers why a single window misses slow burns and how the 1h/6h window pair catches both fast and slow error budget consumption.

/slo-design · burn rate alert · error budget alerting

Protocol for establishing RED/USE metric baselines for a service — what to measure, what healthy looks like, and how to know when you've crossed the threshold from 'degraded' to 'incident'. Starting point for any new service instrumentation.

New service observability setup · metrics baseline

Using distributed traces to find latency bottlenecks — span analysis, critical path identification, and bottleneck isolation across services. Covers Jaeger, Tempo, and vendor-neutral trace interpretation.

Distributed tracing · Jaeger · Tempo · latency bottleneck

Generic alert response structure covering orient, investigate, mitigate, escalate, and communicate phases. The five-step protocol for on-call engineers who are paged on something they haven't seen before.

On-call response · alert handling

Protocol for reviewing a Terraform plan output — identifies destroys, replacements, exposed secrets in plan values, blast radius, and naming violations. The structured checklist before any apply.

Terraform plan output pasted · /tf-plan-review

Protocol for detecting and documenting drift between Terraform state and live cloud resources — identifies resources modified out-of-band, resources missing from state, and attributes that have drifted from their managed values.

/tf-drift-check · state inconsistency

Step-by-step protocol for safely upgrading Terraform provider versions — identifies breaking changes, migration steps for affected resources, and validation gates before applying the upgraded configuration.

/tf-upgrade · provider version upgrade

Generic CI/CD failure triage protocol — test failures, build errors, deploy failures, and flaky test identification. Platform-agnostic: works with any CI system before pulling in the GitHub Actions or GitLab CI specialist.

CI failure · pipeline broken · build error

Protocol for reviewing a deployment pipeline for missing gates — tests, security scans, staging validation, and approval gates. Produces a scored report on what's present, what's missing, and what the risk is of going to production without it.

/pipeline-review · CD pipeline assessment

Checklist for feature flag deployments — flag existence in the target environment, targeting rules, kill switch configuration, and cleanup plan. Ensures no feature ships with a flag that has no owner and no off-ramp.

Feature flag · /feature-flag-check

Security incident response protocol — contain, investigate, eradicate, recover, and notify phases. Covers credential compromise, unauthorized access, and data exposure. Distinguishes between active attacker and historical breach.

Security incident · credential compromise · unauthorized access

Disaster recovery runbook template with RTO/RPO targets, failover steps, communication procedures, and verification checklist. Produces a runbook that can actually be followed at 3am under pressure, not just in a tabletop.

DR runbook · disaster recovery planning

Blameless postmortem template with timeline, contributing factors, impact assessment, and action items. The structure that turns an incident review into prevention tasks rather than a blame session.

/postmortem-template · incident review

Checklist for onboarding a new service to the platform — namespace, RBAC, monitoring, alerts, runbook, and on-call setup. Ensures every new service has an owner, an alert, and a runbook before it reaches production.

/platform-onboard · new service onboarding

SLO design guide — how to choose objectives, set targets, define error budgets, and establish review cadence. Covers why 'five nines' is usually the wrong target and how to pick an SLO that actually drives engineering decisions.

/slo-design · SLO definition · reliability targets

Simple capacity planning model using current usage, growth rate, and headroom requirements per service. Produces a month-by-month projection with the math visible, not a rough estimate from a gut feeling.

/capacity-plan · resource projection

Framework for identifying, measuring, and eliminating toil — distinguishes toil from overhead and engineering work, quantifies the cost of current manual processes, and produces prioritized automation candidates.

Toil reduction · automation opportunities · on-call load