Update to Control Plane

"---\nname: k8s-operator\ndescription: \"Manages Control Plane resources as Kubernetes CRDs. Use when the user asks about the Kubernetes operator, kubectl apply for Control Plane, CRDs, ArgoCD GitOps from a cluster, or exporting resources as K8s manifests.\"\n---\n\n# Kubernetes Operator\n\nThe operator (Helm chart `cpln-operator`) runs in any Kubernetes cluster and reconciles `cpln.io/v1` custom resources, plus labeled native Secrets, against the platform on a 30-second loop. Reach for it only when resources must live in Git and be reconciled from a cluster (ArgoCD/Flux); for direct provisioning prefer the typed MCP tools, and for pipeline-driven YAML prefer `cpln apply` (`gitops-cicd` skill). The recurring failure is manifest shape: `org`, `gvc`, and `description` sit at the **top level next to `spec`**, not inside it — author the `spec` block with `get_resource_schema`, or skip hand-writing entirely by exporting with `-o crd`.\n\n## Install\n\ncert-manager is a hard requirement (the chart ships a self-signed Issuer whose certificate backs the operator's mutating webhook), then the chart, then per-org auth:\n\n```bash\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml\nkubectl wait --for=condition=Available deployment --all -n cert-manager --timeout=300s\n\nhelm repo add cpln https://controlplane-com.github.io/k8s-operator\nhelm install cpln-operator cpln/cpln-operator -n controlplane --create-namespace\nkubectl get pods -n controlplane -l app=operator\n\ncpln operator install --serviceaccount k8s-operator --org ORG\n```\n\n`cpln operator install` configures **auth only** (the Helm step deploys the operator): it gets-or-creates the service account, adds it to a group (`--serviceaccount-group`, default `superusers` — any other group prints a warning), mints a **new key**, and applies a Secret named after the org in the `controlplane` namespace. Re-running with the same `--serviceaccount` is a no-op; a different name replaces the secret with a fresh key; an org-named secret the operator does not own aborts the install. `--export` prints the Secret YAML for Git instead of applying it — but still creates the service account and key on the platform.\n\nManual equivalent (one secret per org; multiple orgs = multiple secrets):\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: ORG                    # must equal the org name\n  namespace: controlplane\n  labels:\n    app.kubernetes.io/managed-by: cpln-operator   # required — the operator only sees labeled Secrets\ndata:\n  token: BASE64_KEY            # echo -n \"KEY\" | base64; the CLI also stamps a cpln/serviceaccount annotation it uses to detect reuse\n```\n\nTokens are cached in memory per org and never re-read — after rotating a key, `kubectl rollout restart deployment/operator -n controlplane`. Helm values of note: `env.MANAGE_KINDS` (comma list restricting which kinds get controllers), `env.RECONCILE_INTERVAL_SECONDS` (default 30), `env.CPLN_API_URL`.\n\n## CRD shape\n\n```yaml\napiVersion: cpln.io/v1\nkind: workload            # kind names are lowercase\nmetadata:\n  name: my-app            # cluster name; annotation cpln.io/name-replacement overrides the platform name\n  namespace: default\n  annotations:\n    cpln.io/resource-policy: keep   # optional: deleting this CR then leaves the platform resource intact\norg: ORG                  # required on every CR — there is no default org\ngvc: GVC                  # required for gvc-scoped kinds: workload, identity, volumeset\ndescription: my app       # top level, like tags\nspec:                     # exact platform spec\n  type: serverless\n  containers:\n    - name: main\n      image: nginx:latest\n      port: 80\n```\n\nOrg-scoped kinds: `agent`, `auditctx`, `cloudaccount`, `domain`, `group`, `gvc`, `ipset`, `location`, `org`, `policy`, `serviceaccount`. GVC-scoped: `workload`, `identity`, `volumeset`. Secrets are native v1 Secrets, not CRDs (a `cpln.io/v1 secret` CRD exists but is operator-internal — it holds sync status for native Secrets; never author it). An `mk8scluster` CRD ships but the platform API has no matching endpoint, so mk8s clusters cannot be operator-managed. Recommended layout: one namespace per GVC for gvc-scoped kinds, one per org for org-scoped kinds.\n\n## Secrets (native v1 Secrets)\n\n```yaml\napiVersion: v1\nkind: Secret\ntype: opaque               # the platform secret type, lowercase: opaque, aws, azure-connector, azure-sdk, dictionary, docker, ecr, gcp, keypair, nats-account, tls, userpass\nmetadata:\n  name: my-secret\n  namespace: default       # any namespace EXCEPT controlplane (everything there is skipped as operator config)\n  labels:\n    app.kubernetes.io/managed-by: cpln-operator   # required — unlabeled Secrets are invisible to the operator\n  annotations:\n    cpln.io/org: ORG       # required — selects the org and its auth secret\ndata:                      # keys mirror the platform secret's data object\n  payload: c2VjcmV0LXZhbHVl\n  encoding: cGxhaW4=       # opaque only: plain | base64\n```\n\nFor `azure-sdk`, `docker`, and `gcp` the platform payload is a single string — put it under one `value` key. Platform tags ride as `cpln.io/`-prefixed annotations. Each synced Secret gets a companion `secrets.cpln.io` CR (same name) carrying sync status and `cpln.io/sync-health-status` / `cpln.io/sync-health-message` annotations — check it when a Secret will not sync.\n\n## How sync behaves\n\n- A mutating webhook stamps every CR (and labeled Secret) with the `cpln.io/sync-protection` finalizer; namespaces labeled `skip-webhook: \"true\"` are exempt.\n- **Local edit (metadata.generation changed):** the operator PUTs the CR to the platform — local state wins.\n- **No local edit:** every cycle it pulls platform state into the CR, so console edits appear as CR changes. Under ArgoCD `selfHeal` that registers as drift, Argo restores the Git version, and the operator pushes it back — **Git wins over console edits**.\n- **Deleted on the platform:** the CR is deleted from the cluster (with `selfHeal`, Argo re-applies it and the operator re-creates the platform resource).\n- **CR deleted:** the platform resource is deleted too, unless annotated `cpln.io/resource-policy: keep`. Deletes blocked by dependent resources retry each cycle — delete children first.\n- Failures land in `status.operator.validationError` with the platform error, and retry with exponential backoff (capped at 30s). `status.phase` is `Ready`, `Pending`, `Unhealthy`, or `Suspended`; the platform's own status fields (endpoints, health) are merged into the CR `status`.\n- Workloads additionally stream live deployment state over WebSocket into read-only child CRs: `kubectl get deployments.cpln.io` (named `LOCATION.WORKLOAD`), plus `deploymentversions`, `containerstatuses`, and `jobexecutionstatuses` for cron; volumesets get `volumesetstatuslocations` and `persistentvolumestatuses`. Children are owner-referenced and garbage-collected with the parent.\n\n## Exporting existing resources\n\nCRD export is CLI/console-only — no MCP tool emits CRD YAML. Discover and inspect with `list_resources` / `get_resource`, then:\n\n```bash\ncpln workload get my-app --gvc GVC --org ORG -o crd > workload.yaml\ncpln gvc get -o crd --org ORG > all-gvcs.yaml        # no name = whole collection, --- separated\n```\n\nIn the console, every resource has Export, and the create flow has Preview, with a \"K8s CRD\" option. System fields are stripped and tags become annotations. **Secret export embeds the revealed payload** (base64-encoded, not encrypted) and needs the `reveal` permission — treat the output as sensitive.\n\n## ArgoCD\n\nWorks without special configuration: point an Application at a Git path of CRD manifests (or a Helm chart templating them). The chart patches the ArgoCD ConfigMap with per-kind health checks — CR `status.phase` and `validationError` surface as Argo health — when the `argocd` namespace exists at install time; if Argo came later, run `helm upgrade cpln-operator cpln/cpln-operator`.\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata: { name: cpln-resources, namespace: argocd }\nspec:\n  project: default\n  destination: { server: https://kubernetes.default.svc, namespace: NAMESPACE }\n  source: { repoURL: \"https://github.com/ORG/REPO.git\", path: cpln-crds, targetRevision: main }\n  syncPolicy:\n    automated:\n      prune: true      # manifest removed from Git = platform resource deleted (honor resource-policy keep)\n      selfHeal: true   # console edits revert to Git\n```\n\n## Uninstall\n\n`cpln operator uninstall --org ORG` removes only the auth secret. Before `helm uninstall cpln-operator -n controlplane`, decide the fate of synced resources: annotate CRs `cpln.io/resource-policy: keep` (or delete the CRs you want gone first) — once the operator is gone nothing clears the `cpln.io/sync-protection` finalizer, so leftover CRs stick in Terminating until you strip it (`kubectl patch ... -p '{\"metadata\":{\"finalizers\":null}}'`). Platform resources whose CRs were never deleted survive uninstall.\n\n## Verify\n\n- `kubectl get workloads -o yaml` — `status.phase: Ready` and no `status.operator.validationError`.\n- `cpln workload get my-app --gvc GVC --org ORG` — the platform side exists and matches.\n- `kubectl logs -n controlplane -l app=operator -f` — watch a sync round-trip.\n\n## Troubleshooting\n\n| Symptom | Cause and fix |\n|---|---|\n| Operator pod not starting, webhook TLS errors | cert-manager missing or certs not issued — `kubectl get pods -n cert-manager`, `kubectl get certificates -n controlplane` |\n| Log: \"unable to sync resources because the secret ORG could not be found\" | Auth secret missing, misnamed, or missing the `managed-by` label (the operator's cache is label-filtered) — re-run `cpln operator install` |\n| 401/403 errors after key rotation | The old token is cached — `kubectl rollout restart deployment/operator -n controlplane`; for 403s check the service account's group |\n| `validationError`: \"CRD resource has no org field\" / gvc-scoped kind has no gvc | Add top-level `org` (and `gvc`) — they are not defaulted and do not go in `spec` |\n| Secret never syncs, no error anywhere | Missing the label (invisible) or the `cpln.io/org` annotation, or it lives in the `controlplane` namespace (always skipped) — then check the companion `secrets.cpln.io` CR annotations |\n| CR stuck Terminating | Platform delete blocked by dependents (delete child resources first) or the operator is gone (strip the `cpln.io/sync-protection` finalizer) |\n| Console edits keep reverting | ArgoCD `selfHeal` working as designed — Git is the source of truth; change the manifest instead |\n| CRD validation errors on apply | `kubectl explain workload.spec` shows the schema the cluster accepts; regenerate the manifest with `-o crd` |\n| `mk8scluster` CR errors with 404 | Expected — the platform API has no mk8scluster path; manage mk8s via `mk8s-byok` instead |\n\n## Quick reference\n\n| Task | Command / tool |\n|---|---|\n| Author a CRD `spec` block | `get_resource_schema` (kind=workload, gvc, ...) |\n| Inspect resources before export | `list_resources` / `get_resource` |\n| Configure / remove operator auth | `cpln operator install -s SA --org ORG [--export]` / `cpln operator uninstall --org ORG` |\n| Export as CRD manifest | `cpln KIND get [NAME] [--gvc GVC] -o crd`, or console Export / Preview \"K8s CRD\" |\n| Restrict managed kinds | Helm value `env.MANAGE_KINDS: workload,volumeset` |\n\n### Related skills\n\n- **gitops-cicd** — pipelines with `CPLN_TOKEN` + `cpln apply`; choose it over the operator when no cluster-side reconciler is wanted.\n- **iac-terraform-pulumi** — the Terraform/Pulumi alternative for declarative management.\n- **mk8s-byok** — provisioning a Kubernetes cluster to host the operator (and managing mk8s itself).\n- **workload** — the primary skill for what goes inside a workload `spec`.\n\n## Documentation\n\n- [Kubernetes Operator Reference](https://docs.controlplane.com/core/kubernetes-operator.md)\n- [Operator Install Guide](https://docs.controlplane.com/guides/cli/cpln-operator.md)\n- [Operator source and issues](https://github.com/controlplane-com/k8s-operator)\n"