Update to Control Plane

{
  "description": "Workload autoscaling and Capacity AI on Control Plane. Use when the user asks about scaling up/down, min/max replicas, scale-to-zero, concurrency/RPS/CPU/memory/latency scaling, KEDA, event-driven scaling, or right-sizing.",
  "included_files": [],
  "name": "autoscaling-capacity",
  "skill_md_contents": "---\nname: autoscaling-capacity\ndescription: \"Workload autoscaling and Capacity AI on Control Plane. Use when the user asks about scaling up/down, min/max replicas, scale-to-zero, concurrency/RPS/CPU/memory/latency scaling, KEDA, event-driven scaling, or right-sizing.\"\n---\n\n# Autoscaling & Capacity AI\n\nDeep skill for scaling and resource optimization. Everything scaling lives in **one block** — `spec.defaultOptions.autoscaling` (with `capacityAI` beside it); `spec.localOptions[]` overrides it per location. The platform keeps the chosen metric near but below `target`. For workload types, production defaults, and the spec shape, start with the **`workload`** skill.\n\n## Picking a metric\n\n| Metric | Scales on | Types | Notes |\n|---|---|---|---|\n| `concurrency` | avg in-flight requests per replica | **serverless only** (its default) | pair with `maxConcurrency` for a hard per-replica cap |\n| `rps` | requests per second per replica | all three | consistent-response-time HTTP |\n| `cpu` | % of allocated CPU | all three (stateful default) | `target` ≤ 100; conflicts with Capacity AI (below) |\n| `memory` | % of allocated memory | all three | `target` ≤ 100 |\n| `latency` | response time in **ms** at `metricPercentile` | standard / stateful | `p50` (default) / `p75` / `p99`; `target` is ms, not % |\n| `multi[]` | several metrics; highest replica count wins | standard / stateful | entries from `cpu` / `memory` / `rps` only, each at most once; **replaces** `metric` and top-level `target` |\n| `keda` | external / event-driven triggers | standard / stateful | GVC must enable KEDA first; `target` is rejected |\n| `disabled` | nothing — fixed at `minScale` | all | realized as min = max |\n\nIf `metric` is omitted, serverless defaults to `concurrency` and stateful to `cpu`. **Standard is the trap:** with Capacity AI on (its default) an omitted metric resolves to `disabled` — min = max, no autoscaling — so always name the metric on a standard workload that must scale. A metric invalid for the workload type is **rejected** (e.g. `concurrency` on standard).\n\n**The metric constrains the type — decide them together.** Type is chosen at creation and is immutable, so a metric-type mismatch is a *type* problem, not a metric problem. The most common case: concurrency-style scaling on a standard workload — the fix is to create the workload as **serverless** (concurrency lives only there) or use **`rps`** on standard (the closest equivalent), not to retry with the same pairing.\n\n## The autoscaling block\n\nSet with `mcp__cpln__create_workload` / `mcp__cpln__update_workload`, then verify with `mcp__cpln__list_deployments`. All fields:\n\n```yaml\nspec:\n  defaultOptions:\n    autoscaling:\n      metric: rps\n      target: 100             # default 95; integer 1-20000; ≤100 for cpu/memory; ms for latency\n      minScale: 2             # default 1; must be ≤ maxScale; 0 = scale-to-zero (rules below)\n      maxScale: 10            # default 5; no schema maximum\n      scaleToZeroDelay: 300   # 30-3600s, default 300\n      maxConcurrency: 0       # serverless only; 0-30000, default 0 = unlimited (excess queues)\n      metricPercentile: p99   # latency only: p50 (default) / p75 / p99\n    capacityAI: true\n```\n\n- **Per-location overrides:** `spec.localOptions[]` (the `defaultOptions` fields plus `location`), set with `localOptions` on `mcp__cpln__create_workload` or `mcp__cpln__update_workload`. `capacityAIUpdateMinutes` and `multiZone` are top-level fields on both tools and apply to every location; set them in an override only when one location needs a different value. An override **replaces** `defaultOptions` in its location and inherits nothing afterwards, so a later `defaultOptions` change skips every location that has one. `update_workload` merges by location, and locations not named keep their overrides:\n  - **Before writing:** read the workload with `mcp__cpln__get_resource` and tell the user which locations already have an override and what it holds. Use the GVC's own location names; an override for a location the GVC does not run in has no effect until it does.\n  - **A location with an override:** send only the fields that change; the rest of the stored override stays, and `autoscaling` merges key by key.\n  - **A location without one:** the new override starts as a copy of the current `defaultOptions` plus the fields sent; on `create_workload`, a copy of the `defaultOptions` the same call sets.\n  - **Back to `defaultOptions`:** `removeLocalOptions` with the location link. Naming a location in both `removeLocalOptions` and `localOptions` rebuilds its override from `defaultOptions` in one write, for example to drop a stored `multi` or `keda` block that a merge would keep.\n  - **After writing:** the result says how each override was built and its autoscaling now; check that it matches what the user asked.\n- **`scaleToZeroDelay` is dual-purpose:** on serverless it is the idle period before scaling to 0; on standard/stateful it sets the **scale-down stabilization window** (default 300s) — scale-up is immediate.\n\n### Multi-metric (standard/stateful)\n\n```yaml\nautoscaling:\n  minScale: 2\n  maxScale: 10\n  multi:\n    - metric: cpu\n      target: 80\n    - metric: memory\n      target: 80\n```\n\nEach entry is evaluated independently; the highest replica count wins. Only `cpu` / `memory` / `rps`, each at most once; targets go inside the entries (`metric`/`target` at the top level are rejected alongside `multi`). With `multi`, Capacity AI defaults to off.\n\n## Scale to zero\n\n- **Scale to zero by type:** serverless freely; standard and stateful only with `metric: keda` (anything else is rejected); cron never. Serverless reaches zero with `concurrency` or `rps`; `cpu` and `memory` ride an HPA that never drops to zero. The first request after idle pays a cold start.\n\n## KEDA (event-driven, standard/stateful)\n\n**1. Enable on the GVC first** — `mcp__cpln__update_gvc`:\n\n```yaml\nspec:\n  keda:\n    enabled: true                            # default false\n    identityLink: //gvc/GVC/identity/NAME    # optional: cloud/network access for the KEDA operator\n    secrets: [//secret/NAME]                 # optional: each becomes a TriggerAuthentication named after the secret\n```\n\n**2. Set the workload** — `metric: keda` plus raw [KEDA trigger specs](https://keda.sh/) (passed through as-is):\n\n```yaml\nautoscaling:\n  metric: keda          # target is rejected with keda\n  minScale: 0           # maps to KEDA minReplicaCount — this is how standard/stateful scale to zero\n  maxScale: 10\n  keda:\n    triggers:\n      - type: redis\n        metadata:\n          address: my-redis.my-gvc.cpln.local:6379\n          queueLength: '5'\n          passwordFromEnv: REDIS_PASSWORD\n```\n\n- Triggers needing auth reference a GVC-listed secret via `authenticationRef.name` (the TriggerAuthentication is named after the secret).\n- If the trigger source is a Control Plane workload, allow KEDA in the source's firewall: `internal.inboundAllowWorkload: [cpln://internal/keda]`.\n- Also supported: `keda.advanced.scalingModifiers` (custom formulas), `fallback`, `pollingInterval`, `cooldownPeriod`.\n- **Prometheus trigger**, to scale on any platform or custom metric: `type: prometheus` with `serverAddress: https://metrics.cpln.io:443/metrics/org/ORG`, a `query` (PromQL), `threshold`, and `authModes: bearer`, with `authenticationRef.name` naming a dictionary secret listed in `gvc.spec.keda.secrets` whose `bearerToken` key holds a service-account key with `readMetrics` (each key becomes a trigger parameter; the user enters the value in the Console). **Before wiring any trigger, confirm the signal resolves:** `mcp__cpln__query_metrics` without a `query` for real names (add `metric` for one metric's live label values), then with the PromQL as its `query` to run it; a never-resolving signal pins the workload at `minScale`. Custom app metrics come from the container `metrics` block (see **metrics-observability**).\n\n## Capacity AI\n\nRight-sizes each container's **reserved** resources (what you're billed for) from usage history, between the `minCpu`/`minMemory` floor and the `cpu`/`memory` ceiling. **Every workload type supports it.** Standard, serverless, and cron enable it by default.\n\n```yaml\nspec:\n  containers:\n    - name: app\n      cpu: '1000m'       # ceiling (and the fixed allocation when Capacity AI is off)\n      memory: '1Gi'      # ceiling\n      minCpu: '100m'     # floor\n      minMemory: '256Mi' # floor\n  defaultOptions:\n    capacityAI: true\n```\n\n- **With `metric: cpu`:** explicitly enabling Capacity AI is **rejected** (dynamic CPU allocation fights CPU-based scaling); left unset with `cpu` or `multi`, it silently defaults to **off**.\n- **GPU containers reject Capacity AI.**\n- Adjustments land **in place** on standard and stateful when the cluster supports pod resize (no restart; otherwise a rolling update); serverless rolls a new revision; **cron picks up the new reservation on its next execution** — a run already in flight is never resized. Throttle frequency with `capacityAIUpdateMinutes` (min 2; per location through `update_workload` `localOptions`, or on `defaultOptions` through `cpln apply`).\n- Idle floor is **25m** CPU, rising with memory at **1 millicore per 3 MiB**. A just-changed workload pauses adjustments while history rebuilds — apps that reserve resources at startup may not benefit.\n\n### Resource bounds (all types)\n\n- Floors and the memory to CPU ratio: `workload` skill. `minCpu` and `minMemory` never exceed `cpu` and `memory`.\n- **With Capacity AI off:** `cpu`/`memory` are the fixed allocation and `minCpu`/`minMemory` are ignored — except on **stateful**, where `minCpu`/`minMemory` become the static **reserved** request and `cpu`/`memory` stay the burst ceiling.\n- **Stateful `minCpu`/`minMemory` are bounded** unless Capacity AI is on in `defaultOptions` and in every `localOptions` entry: max/min ratio ≤ **4** AND gap ≤ **4000m** CPU / **4096Mi** memory.\n- **GPU:** `nvidia` model `t4` (quantity up to 4) or `a10g` (exactly 1); strict per-model CPU/memory minimums: fetch exact numbers with `mcp__cpln__get_resource` (`kind: workload`, `schema: true`).\n- **Cost:** billing follows reserved resources, so Capacity AI (or stateful `minCpu`) directly lowers cost.\n\n## Troubleshooting\n\n| Symptom | Check |\n|---|---|\n| Not scaling up | Does the signal exist? `mcp__cpln__query_metrics` without, then with, a `query`; check `maxScale`; check replica readiness via `mcp__cpln__list_deployments` |\n| Not scaling down | Standard/stateful stabilization window = `scaleToZeroDelay` (default 300s); check `minScale` |\n| Scale-to-zero not happening | Serverless needs `concurrency`/`rps`; standard/stateful need `metric: keda`; check `scaleToZeroDelay` |\n| KEDA not triggering | KEDA enabled on the GVC? Trigger auth secret listed in `gvc.spec.keda.secrets`? Source firewall allows `cpln://internal/keda`? |\n| Capacity AI not adjusting | Check the stored `spec.defaultOptions.capacityAI`; off with `metric: cpu` or `multi`; rejected with GPUs; a recent spec change pauses it; `capacityAIUpdateMinutes` throttle; on cron it only lands at the next execution |\n| Replicas stuck at `minScale` | The scaling metric never resolves — verify the PromQL/trigger returns data |\n"
}