Update to Control Plane

{
  "description": "Diagnoses unhealthy Control Plane workloads. Use when asked why a workload is crashing, not starting, OOMKilled, ImagePullBackOff, returning 502s, failing health checks, unreachable, or stuck deploying.",
  "included_files": [],
  "name": "workload-troubleshooting",
  "skill_md_contents": "---\nname: workload-troubleshooting\ndescription: \"Diagnoses unhealthy Control Plane workloads. Use when asked why a workload is crashing, not starting, OOMKilled, ImagePullBackOff, returning 502s, failing health checks, unreachable, or stuck deploying.\"\n---\n\n# Workload Troubleshooting\n\nMaps a symptom to its platform-specific cause and a fix the schema accepts. Diagnosis is read-only. Most failures trace to a Control Plane rule a generic engineer would not guess; the most common is OOMKilled.\n\n## Gather state\n\n`mcp__cpln__diagnose_workload` first: one call reads the deployments, events, and error log lines, matches them against this catalog, and returns the likely cause with its evidence. When it is not conclusive:\n\n- `list_deployments`, with `location` for one location in full detail.\n- `get_workload_events` (needs `?toolsets=full`) for event history past the 10 newest events `diagnose_workload` returns: image pulls, crashes, scheduling, probe failures, and `OOMKilled`.\n- `get_workload_logs`; the `_accesslog` container holds HTTP status codes and latency.\n- `query_metrics`, first without a `query` to list the metric names, for memory, CPU, and latency pressure.\n- `query_traces`, then again with one trace's `traceId`, for a slow request; needs tracing on the GVC (`metrics-observability` skill).\n- In-container inspection goes through the CLI (`cpln` skill).\n\n## Failure catalog\n\n### Out of memory (OOMKilled)\n\n`memory` is a hard cap per container, sidecars included: app, runtime, GC, and buffers together. Usual culprits: Java without `-Xmx`, Node without `--max-old-space-size`, Python loading large datasets. Capacity AI can downsize a spiky workload too far; set `minMemory` as a floor.\n\n**Fix:** measure with `query_metrics`, then raise `memory`, and raise `cpu` with it: memory in MiB stays within 8 times CPU in millicores, so the default `50m` caps memory at 400Mi.\n\n### Image pull failures (`ImagePullBackOff`, `ErrImagePull`)\n\nCheck the reference form, the architecture (`linux/amd64` on managed locations; BYOK allows more), and the pull secret: a private external registry needs a secret of type `docker`, `ecr`, or `gcp` in the GVC's `pullSecretLinks`, created with `create_secret` and `values: \"user\"`, attached with `update_gvc`. Org `//image/` images need none. Detail: `image` skill.\n\n### Secret access failures\n\nEmpty env vars or secret errors in events mean the identity, `reveal` policy, or reference is missing or wrong. `grant_workload_secret_access` fixes the identity and policy in one call; reference formats per type: `setup-secret` skill.\n\n### Healthy but 502 or 503: port mismatch\n\n- The spec port must match what the process listens on; compare the spec with the startup logs.\n- On serverless the runtime injects `PORT` and rejects a `PORT` env var that differs from the exposed port.\n- Serverless exposes exactly one port (TCP needs a dedicated load balancer); standard and stateful expose zero or more; cron serves nothing.\n- Blocked ports cannot be bound and are invalid for TCP probes: 8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000.\n\n### Firewall blocking traffic\n\nFix with `update_workload`: `external.inboundAllowCIDR` for inbound, `external.outboundAllowCIDR` or `outboundAllowHostname` for outbound (hostname rules reach only ports 80, 443, and 445), and `internal.inboundAllowType` or `allow_workload_access` for other workloads. Detail: `firewall-networking` skill.\n\n### Health-check failures\n\nServerless gets TCP readiness and liveness on the container port; standard, stateful, and cron get none. For a slow start raise `initialDelaySeconds` (0 to 600); for an over-eager probe raise `periodSeconds` (1 to 600) or `timeoutSeconds` (1 to 600, default 1); an HTTP path must return 200 to 399. A readiness failure takes the replica out of rotation and pauses the rollout; a liveness failure restarts it. An autoscaled workload without a real readiness probe takes traffic before it is ready and returns 502 on scale-up: add an `httpGet` readiness probe. Detail: `workload-security` skill.\n\n### Resources and Capacity AI\n\n- CPU and memory must fit the org quota; `maxScale` times the per-replica resources is enforced at scheduling. A quota error names its quota (a workload's total CPU or memory error means `cpu-per-workload` or `memory-per-workload`); `request_quota_increase` asks for a higher max when the user wants one.\n- Capacity AI is off with the `cpu` metric or multi-metric scaling; check the stored `spec.defaultOptions.capacityAI` first. On cron a new reservation lands at the next run.\n- Stateful sizing, unless Capacity AI is on in `defaultOptions` and in every `localOptions` entry: `minCpu` to `cpu` at most 4 to 1 and 4000m apart; `minMemory` to `memory` at most 4 to 1 and 4096Mi apart.\n- Ephemeral storage is 1 GB per CPU core, at least 1 GB; exceeding it replaces the replica.\n\n### Container will not start\n\n- UID 1337 is the mesh proxy's: a container running as it bypasses the sidecar. Set `runAsUser` to another UID from 1 to 65534; root is rejected.\n- Reserved container names: `istio-proxy`, `queue-proxy`, `istio-validation`, and any starting with `cpln-` or `debugger-`.\n- Env names starting with `CPLN_`, and `K_SERVICE`, `K_CONFIGURATION`, `K_REVISION`, are rejected; each value is at most 4096 characters.\n- `spec.defaultOptions.suspend: true` stops the workload: clear it, or `cpln workload start WORKLOAD --gvc GVC`.\n\n### Autoscaling misconfiguration\n\nServerless has no `latency` or multi-metric scaling; standard and stateful have no `concurrency`; cron has no autoscaling. Scale to zero: serverless with `rps` or `concurrency`; standard and stateful only with `metric: keda`; never cron. 502 on scale-up is the readiness probe above. Detail: `autoscaling-capacity` skill.\n\n### Failures during deploys or scale-down\n\n- If `sleep` is missing from any container, every container is killed at once with no drain; distroless images often lack it (`which sleep`). Add `sleep` or a custom `preStop`.\n- A failing custom `preStop` in any container kills all of them.\n- After the preStop (default `sleep 45`) the container gets SIGTERM, then SIGKILL once `terminationGracePeriodSeconds` (default 90; at most 900 without the `cpln/relaxGracePeriodMax` tag) elapses. An app that ignores SIGTERM drops requests.\n\nSequence and rollout options: `workload-security` skill.\n\n### Volume mount failures\n\n- Secret volumes need the identity and `reveal` policy.\n- Cloud volumes (S3, GCS, Azure Blob and Files) need an identity with cloud access and outbound firewall to the provider hosts (`*.amazonaws.com`, `*.googleapis.com`, `*.blob.core.windows.net`, `*.file.core.windows.net`, `*.azure.com`). Only identity auth works; embedded keys do not. Read-only except Azure Files.\n- `filesystemGroupId` defaults to 0 (root) when unset: set it from 1 to 65534 for a non-root app.\n\nVolume sets: `stateful-storage` skill.\n\n### Service-to-service failures\n\n- The target's internal firewall must admit the caller (`allow_workload_access`); listing a workload needs `view` on it.\n- Call `http://WORKLOAD.GVC.cpln.local:PORT` over plain HTTP. An omitted port means the target's first container port; only listed ports are reachable.\n- Cross-GVC calls may span locations and pay egress.\n\n### Dedicated load balancer and domains\n\n- TCP needs a dedicated load balancer on the GVC plus a Domain with a TCP port; default endpoints serve HTTP, HTTP2, and gRPC only.\n- Enabling or disabling a dedicated load balancer causes a brief DNS propagation outage and per-location charges.\n- Behind a custom domain, a serverless workload receives the canonical endpoint as `Host` (the custom domain arrives in `X-Forwarded-Host`); standard and stateful receive the custom domain.\n- The domain port protocol must match the container's: HTTP2 fronts HTTP2 or gRPC, HTTP fronts HTTP.\n\nDetail: `domain` and `ipset-load-balancing` skills.\n\n## Fix within the schema's limits\n\nA fix the validator rejects wastes a round trip. Before a resource or option change, check:\n\n- Memory in MiB at most 8 times CPU in millicores; CPU at least 25m; memory at least 32Mi; `minCpu` and `minMemory` never above `cpu` and `memory`.\n- `runAsUser` and `filesystemGroupId` from 1 to 65534.\n- `terminationGracePeriodSeconds` at most 900, unless the `cpln/relaxGracePeriodMax` tag is set.\n- `capacityAI` is rejected with `metric: cpu` and with a GPU container.\n- Standard and stateful `minScale: 0` needs `metric: keda`; cron and vm cannot scale to zero.\n- A metric outside the type's list is rejected.\n\nA restart without a spec change (fresh secret values, wedged replicas) is `restart_workload`. After a fix, `diagnose_workload` confirms the symptom cleared.\n"
}