{"id":48142,"plugin_id":"plugins_6abffe30af5881919d4fc103b4e6c368","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-10-24T13:35:32.942Z","digest":"1cfabb1315af05a7e3c045db86e92637375fbefbb2b160841e23a02d165df396","against":null,"payload":{"description":"Use when the user wants to create, configure, scale, debug, observe, or roll out container workloads on DataRobot's Workload API. Triggers: deploy a container as a managed service; list/start/stop workloads; change replica counts or autoscaling; pick CPU/GPU compute bundles; inject DataRobot credentials as env vars; diagnose stuck/errored/crash-looping workloads (CrashLoopBackOff, ImagePullBackOff, OOMKilled, probe failures, exec format error); pull application logs/OpenTelemetry traces/metrics/request stats; create or iterate container artifacts; build images server-side; lock artifacts for production; zero-downtime rolling artifact replacement.","included_files":[{"relative_path":"references/code-to-workload.md","size_in_bytes":15585},{"relative_path":"references/common-error-patterns.md","size_in_bytes":7023},{"relative_path":"references/declarative-cli-deploy.md","size_in_bytes":5705},{"relative_path":"references/lifecycle-flows.md","size_in_bytes":7568},{"relative_path":"references/schema-reference.md","size_in_bytes":5955},{"relative_path":"references/status-vocabulary.md","size_in_bytes":4158},{"relative_path":"references/web-uis-behind-the-edge.md","size_in_bytes":14289},{"relative_path":"scripts/check_limits.py","size_in_bytes":2817},{"relative_path":"scripts/diagnose_workload.py","size_in_bytes":8872},{"relative_path":"scripts/wait_for_build.py","size_in_bytes":4190},{"relative_path":"scripts/wait_for_replacement.py","size_in_bytes":4070},{"relative_path":"scripts/wait_for_running.py","size_in_bytes":3056}],"name":"datarobot-workload-api","skill_md_contents":"---\nname: datarobot-workload-api\ndescription: >-\n  Use when the user wants to create, configure, scale, debug, observe, or roll\n  out container workloads on DataRobot's Workload API. Triggers: deploy a\n  container as a managed service; list/start/stop workloads; change replica\n  counts or autoscaling; pick CPU/GPU compute bundles; inject DataRobot\n  credentials as env vars; diagnose stuck/errored/crash-looping workloads\n  (CrashLoopBackOff, ImagePullBackOff, OOMKilled, probe failures, exec format\n  error); pull application logs/OpenTelemetry traces/metrics/request stats;\n  create or iterate container artifacts; build images server-side; lock\n  artifacts for production; zero-downtime rolling artifact replacement.\n---\n\n# DataRobot Workload API\n\nRuns containers as managed, autoscalable DataRobot services. Four jobs — pick by intent:\n\n1. **Create / configure / scale** — deploy a container; change replicas, resources, autoscaling, bundle; inject credentials\n2. **Diagnose** — workload stuck, errored, or crash-looping\n3. **Observe** — logs, traces, metrics, service stats for a running workload\n4. **Artifact lifecycle** — iterate drafts, build images, lock for production, roll out new versions\n\n## Prerequisites\n\nAuth: like `gh`. `dr auth login` (or existing `.env`/`~/.config/datarobot/drconfig.yaml`) persists credentials — no per-run env vars needed. Verify with `dr auth check`. On failure, run `datarobot-setup`.\n\n**Keep the CLI current** (like `datarobot-agent-assist`): `dr self update --force`. Never gate on a pinned version. If a subcommand still errors `unknown command` after update (seen with `config`/`up` — `references/declarative-cli-deploy.md`), use the raw-REST fallback below.\n\n`DATAROBOT_ENDPOINT` (must end `/api/v2`) and `DATAROBOT_API_TOKEN`: required as env vars only for raw REST (`scripts/`, `httpx`/`curl`) or CI — these bypass CLI auth. Header: `Authorization: Bearer ${DATAROBOT_API_TOKEN}`. Not in the `datarobot` Python SDK — call REST directly.\n\n**Transport:** examples use Python `httpx` (`pip install httpx`); `curl` or the `pulumi-datarobot` provider work too.\n\n## Bundled scripts\n\nPython in `scripts/`. Each uses `httpx`, reads `DATAROBOT_ENDPOINT` + `DATAROBOT_API_TOKEN`:\n\n- `wait_for_running.py <workload_id>` — poll until `running`; exit 2 on terminal failure, 3 on timeout\n- `diagnose_workload.py <workload_id>` — run the 5-step debug flow, print a diagnosis (`--json` for machine output)\n- `wait_for_build.py <artifact_id> <build_id>` — poll a server-side image build; dumps last 2KB of logs on `FAILED`\n- `wait_for_replacement.py <workload_id>` — poll a rolling replacement; handles the 404-when-cleared case\n- `check_limits.py` — print effective org scaling limits via `/account/info/`\n\n## Deeper docs in references/\n\nIn `references/`:\n\n- `status-vocabulary.md` — status enums and transitions\n- `common-error-patterns.md` — CrashLoopBackOff / ImagePullBackOff / OOMKilled / probe / exec-format\n- `schema-reference.md` — schemas, credential-type→key maps, spec quirks\n- `lifecycle-flows.md` — draft→lock→prod rules, replacement preconditions, redeploy matrix\n- `code-to-workload.md` — deploy from source: `dr` CLI, `codeRef`, Execution Environments\n- `declarative-cli-deploy.md` — `dr workload config`/`up`: one-command create/build/deploy\n- `web-uis-behind-the-edge.md` — web UI through the endpoint: prefix, auth gate, CSRF, WebSockets\n\n## OpenAPI spec is source of truth\n\nAt `${DATAROBOT_ENDPOINT}/openapi.yaml` (some instances omit Workload API paths — see `references/schema-reference.md`). **~5 MB — never dump whole.** Save once, slice with `yq` (or `print()` one key in Python):\n\n```bash\ncurl -sS \"${DATAROBOT_ENDPOINT}/openapi.yaml\" -o /tmp/wapi-spec.yaml\nyq '.components.schemas.CreateWorkloadRequest' /tmp/wapi-spec.yaml\n```\n\nWorkload paths key with `/api/v2/` — see `references/schema-reference.md`.\n\n---\n\n# 1. Create / configure / scale\n\n## Deploy a container (common case)\n\n```yaml\n# spec.yaml — JSON also accepted; spec is sent verbatim\nname: my-api-service\nimportance: low\nartifact:\n  name: my-api-service-artifact\n  spec:\n    type: service\n    containerGroups:\n      - name: default\n        containers:\n          - name: main\n            imageUri: ghcr.io/org/my-app:latest\n            port: 8000\n            primary: true\n            readinessProbe: {path: /readyz, port: 8000, initialDelaySeconds: 10}\n            livenessProbe: {path: /healthz, port: 8000, initialDelaySeconds: 30}\nruntime:\n  containerGroups:\n    - name: default          # must match artifact.spec.containerGroups[].name (above)\n      replicaCount: 1\n      containers:\n        - name: main\n          resourceAllocation: {cpu: 1, memory: \"512MB\"}\n```\n\n```bash\ndr workload create --spec-file spec.yaml         # 4xx: 400=schema/limit, 403=cap (run check_limits.py), 409=name conflict\ndr workload get <workload_id>                    # or `dr workload status` — poll until status=running\n```\n\nLifecycle one-liners: `dr workload {stop|start|delete|endpoint|list} <id>`.\n\nRaw fallback when CLI unavailable: `httpx.post(f\"{base}/workloads/\", headers=headers, json=spec)` + `r.raise_for_status()` + `r.json()[\"id\"]`; poll with `python scripts/wait_for_running.py <workload_id>`.\n\n**Critical gotchas:**\n\n- `importance`: `low`/`moderate`/`high`/`critical`. `type`: `service` (default) or `nim`. One container per group must be `primary: true`.\n- `cpu`: cores (float OK). `memory`: decimal string (`\"512MB\"`, units B/KB/MB/GB) or byte integer. No Kubernetes binary suffixes (`Mi`/`Gi`).\n- `port` must be `>= 1024`. Container must actually listen on it (image env var or entrypoint).\n- Image needs a **linux/amd64** manifest. Apple Silicon defaults to ARM64: crash-loops with `exec format error`. Build: `docker buildx build --platform linux/amd64,linux/arm64 -t <ref> --push .`.\n- Status: `submitted` → `provisioning` → `launching` → `running` (happy path); `updating` = rolling redeploy; `errored` recoverable; `failed`/`terminated` unrecoverable. Full table: `references/status-vocabulary.md`.\n\n## Web UI through the endpoint\n\nBrowser web app (UI + backend/API/WebSocket) via `dr workload endpoint <id>`, not headless: edge gateway serves it under a path prefix. **Strips prefix inbound** (no outbound rewrite; app must be sub-path aware). **Is the auth gate** (DataRobot login), **hijacks `Authorization`** (→ `401 {\"message\":\"Invalid API key\"}`, container never sees it). **Passes WebSockets through**. Pattern: base-path = prefix (from `WORKLOAD_ID`) + inbound shim, CSRF off, unauthenticated probe path. Disable app auth only with user sign-off. Detail: `references/web-uis-behind-the-edge.md`.\n\n## Update paths\n\n| User intent | Endpoint | Effect |\n|---|---|---|\n| Rename / redescribe / change importance | `PATCH /workloads/{id}/` | Metadata only — no restart |\n| Change replicas / resources / autoscaling on the same artifact | `PATCH /workloads/{id}/settings/` | Triggers rolling redeploy |\n| Deploy a different artifact (new image / version) | `POST /workloads/{id}/replacement/` | Rolling swap — see section 4 |\n\n## Replicas, resources, autoscaling\n\n`PATCH /workloads/{wid}/settings/`, full body. Use `replicaCount` OR `autoscaling`, not both. Read via `GET /workloads/{wid}/settings/` first, then PATCH back:\n\n```python\nhttpx.patch(\n    f\"{base}/workloads/{wid}/settings/\",\n    headers=headers,\n    json={\n        \"runtime\": {\n            \"containerGroups\": [\n                {\n                    \"name\": \"default\",\n                    \"replicaCount\": 3,\n                    \"containers\": [\n                        {\n                            \"name\": \"main\",\n                            \"resourceAllocation\": {\"cpu\": 2, \"memory\": \"1GB\"},\n                        }\n                    ],\n                    # OR: \"autoscaling\": {\"enabled\": True, \"policies\": [{\n                    #       \"scalingMetric\": \"cpuAverageUtilization\",\n                    #       \"target\": 70, \"minCount\": 1, \"maxCount\": 10}]}\n                }\n            ]\n        }\n    },\n)\n```\n\n`scalingMetric` values: `cpuAverageUtilization`, `httpRequestsConcurrency`, `gpuCacheUtilization`, `gpuRequestQueueDepth`, or a custom NIM metric. Settings updates are **rolling** — zero-downtime needs `replicaCount >= 2` (or autoscaling `minCount >= 2`).\n\n## Org-set scaling limits\n\nAdmin-set caps: `maxConcurrentWorkloads`, `maxWorkloadReplicas`. `0` = unlimited; users can't change them. Read via **`GET /account/info/`** → `{\"limits\": {\"maxConcurrentWorkloads\": N, \"maxWorkloadReplicas\": M}}` (or `python scripts/check_limits.py`). `/users/{uid}/` and `/organizations/{id}/` need Admin API access. Over limit: **HTTP 403** `{\"detail\": \"Requested replicas (N) exceeds the maximum allowed (M).\"}` — check limits first; propose the max, or flag admin help needed.\n\n## GPU/VRAM via compute bundle\n\n`resourceAllocation` accepts only `cpu`, `memory`, `gpu` (count) — no `gpuType`/`gpuMemory` field. For a GPU model/VRAM size: `GET /mlops/compute/bundles/` lists bundles (`cpu.small`, `gpu.l4.small`, `gpu.a10g.medium`); pass `\"resourceBundles\": [\"gpu.l4.small\"]` (a list, but only ONE bundle allowed) under the container group. A set bundle overrides `resourceAllocation` cpu/memory.\n\n## Credential injection — no hardcoded secrets\n\nDataRobot stores credentials centrally, injects into `environmentVars` by reference:\n\n```python\n\"environmentVars\": [\n    {\"name\": \"PLAIN_VAR\", \"value\": \"literal-value\"},\n    {\"source\": \"dr-credential\", \"name\": \"AWS_ACCESS_KEY_ID\",\n     \"drCredentialId\": \"<credential-id>\", \"key\": \"awsAccessKeyId\"},\n]\n```\n\nFlow: `GET /credentials/?limit=50` → note `credentialType` → look up valid `key` names in `references/schema-reference.md`.\n\n## Create from artifact\n\nProvide `artifactId` instead of an inline `artifact` block. `runtime`'s `containerGroups[].name`/`containers[].name` must match the artifact's.\n\n---\n\n# 2. Diagnose\n\n## Full diagnosis, one command\n\n```bash\npython scripts/diagnose_workload.py <workload_id>\n```\n\nRuns the 5 steps below, prints a report (status, logTail signals, flagged events, proton K8s detail, evidence, recommended fix, console URL). `--json` for machine output. Empty `Evidence`: pull application logs via section 3 — don't guess from status alone.\n\n## The 5-step flow\n\nScript does this; steps below for ambiguous output or one-off calls.\n\n1. **`GET /workloads/{id}/`** — `status`, `statusDetails.logTail` (~30 lines; scan for `error`/`exception`/`traceback`/`killed`/`permission denied`/`connection refused`), `statusDetails.conditions`. `statusDetails` is `null` during `submitted`/`provisioning` — guard for it.\n2. **`GET /workloads/{id}/events/`** — flag `type: Warning` or `reason` with `Failed`/`Error`/`Kill`/`OOM`; the last Warning before `errored` is usually the trigger.\n3. **`GET /workloads/{id}/protons/`** — pick `role: \"active\"` (or the `candidate` during a rolling replacement; else newest `createdAt`).\n4. **`GET /workloads/{id}/protons/{pid}/statusDetails/`** — `204` while initializing (not an error). Read `replicas[*].containers[*].status`+`restartCount` → `replicas[*].conditions[*]` (any `value:false`) → `overallStatus.summary`.\n5. **Application logs** — section 3.\n\nCommon patterns (`CrashLoopBackOff`, `ImagePullBackOff`, `OOMKilled`, probe/pending, `exec format error`) and fixes: `references/common-error-patterns.md`.\n\n## Diagnosis output\n\n```\nWorkload {id} — Diagnosis\n- Status: {current}\n- Root cause: {one sentence}\n- Evidence: {the specific logTail line, condition, container reason, or event}\n- Recommended fix: {next step — section 1 (settings), section 4 (artifact), or app code}\n- Console: https://app.datarobot.com/console-nextgen/workloads/{id}/overview\n```\n\n---\n\n# 3. Observe\n\n| Stream | Endpoint | Needs app instrumentation? |\n|---|---|---|\n| Logs | `/otel/workload/{id}/logs/` | No — auto from stdout/stderr |\n| Traces | `/otel/workload/{id}/traces/` | **Yes** (OTEL spans) |\n| Metrics | `/otel/workload/{id}/metrics/autocollectedValues/` | Partially |\n| Service stats | `/workloads/{id}/stats/` | No — DataRobot edge proxy |\n| Replacement history | `/workloads/{id}/history/` | No — platform |\n| Lifecycle events | `/workloads/{id}/events/` | No — platform |\n\nCheck `r.status_code` before `.json()`: 401 bad token, 404 not found, 429 rate-limited (exponential backoff). List endpoints accept `limit` + `offset`.\n\n## Logs\n\n```bash\ndr workload logs <wid> --level error --limit 100   # --follow streams; --output-format json\n```\n\n`--level` is a MINIMUM severity threshold, not an exact match. For substring filtering on the message body, or proton-scoped logs (proton IDs: section 2), drop to REST — `dr workload logs` lacks these filters:\n\n```python\nr = httpx.get(\n    f\"{base}/otel/workload/{wid}/logs/\",\n    headers=headers,\n    params=[\n        (\"searchKeys\", \"proton_id\"),\n        (\"searchValues\", pid),\n        (\"searchKeys\", \"level\"),\n        (\"searchValues\", \"error\"),\n    ],\n)\n```\n\n`searchKeys`/`searchValues` are positional parallel lists — pass a **list of tuples** to httpx (dict can't repeat keys). `includes=<substring>`: case-sensitive substring filter on the message body.\n\n## Traces\n\n```python\ntraces = httpx.get(f\"{base}/otel/workload/{wid}/traces/\", headers=headers).json()[\n    \"data\"\n]\n# summary: traceId, rootSpanName, rootServiceName, duration (NANOSECONDS), spansCount, errorSpansCount\ntrace_id = next(\n    (t[\"traceId\"] for t in traces if t.get(\"errorSpansCount\", 0) > 0),\n    traces[0][\"traceId\"],\n)\ntrace = httpx.get(\n    f\"{base}/otel/workload/{wid}/traces/{trace_id}/\", headers=headers\n).json()\n```\n\n> **`duration` is NANOSECONDS**, summaries and spans. Divide by 1,000,000 for ms before display. Empty `data`: app not instrumented — tell the user to wire up OTEL.\n\n## Metrics + service stats\n\nConvert before display: `bytes`→MB (`/1024**2`), `nanocores`→cores (`/1_000_000`), `percentage` already %.\n\n```python\nstats = httpx.get(f\"{base}/workloads/{wid}/stats/\", headers=headers).json()\n# {\"period\": {...}, \"metrics\": {totalRequests, serverErrors, userErrors, slowRequests,\n#   responseTime, requestsPerMinute, concurrentRequests, *ErrorRate}}. /workloads/stats/ = aggregate.\n```\n\n> **Destructive:** `DELETE /workloads/{id}/stats/?metricName=<name>` zeroes a metric's history — only on explicit request.\n\n## Output format\n\nLogs: `timestamp | level | message`, ERROR/CRITICAL first. Traces: table sorted by errors desc then recency. Metrics: apply unit conversion before display. Service stats one-liner: *\"`{totalRequests}` requests, `{totalErrorRate*100:.2f}%` errors, `{responseTime:.1f}` ms avg, `{requestsPerMinute}` req/min.\"* Empty data: state *why* (not running, not instrumented, empty window) — not just \"no data\".\n\n---\n\n# 4. Artifact lifecycle\n\n**Artifact**: immutable-after-lock definition of what a workload runs (image, port, env vars, probes). **Workload**: the running instance + its runtime (replicas, resources, autoscaling). Resources do NOT belong on the artifact.\n\n## Choose a redeploy path\n\nFind the running artifact (`workload[\"artifactId\"]`), check `artifact[\"status\"]`. A running workload does **not** auto-adopt a rebuild — redeploy is required.\n\n- **Same draft (C2W loop) — in-place edit or rebuild.** PATCH/rebuild the draft, roll with `PATCH /workloads/{id}/settings/`: re-send the runtime body (even unchanged — triggers rolling `202`, re-reads current spec + latest `COMPLETED` build). Zero-downtime at ≥2 replicas. (`POST /replacement/` onto the same draft also works.)\n- **Different / locked artifact.** `POST /replacement/` onto the other artifact ID. Locked in-place edit: clone → PATCH clone → lock → replace onto the clone.\n\n**Lock:** `dr artifact lock <id>` (= `PATCH /artifacts/{id}/ {\"status\":\"locked\"}`). **Promote** (`POST /workloads/{wid}/promote/`, 200) locks the running draft in place, no restart. Runtime-only changes (replicas/resources/autoscaling) → `PATCH /settings/`; a PATCH to the artifact affects live workloads only on redeploy.\n\nPreconditions (status-match, same-artifact rule) and the full redeploy matrix: `references/lifecycle-flows.md`.\n\n## Image source\n\nThe artifact's `imageUri` must point at a registry DataRobot can pull from (image-pull creds not yet accepted at workload creation). Two paths:\n\n1. **Bring your own image** — public registry or one the admin pre-configured. `docker buildx ... --platform linux/amd64`, push, set `imageUri`. Default flow.\n2. **Code-to-Workload (C2W)** — no local Docker / no public registry: `dr artifact code init` + `sync`, then `dr artifact build create` builds server-side, pushes to DataRobot's internal registry, and populates `imageUri`. Full flow: `references/code-to-workload.md`.\n\nPoll: `python scripts/wait_for_build.py <artifact_id> <build_id>`; only drafts build. **`imageUri` is build-managed** — never hand-PATCH it (`422` \"not permitted on this cluster\"), never PATCH the spec *mid-build* (whole-spec write clobbers the pending image → redeploys the old one). Edit spec before `build create` or after `COMPLETED`.\n\n> **C2W may still need `ENABLE_WORKLOAD_API_CONTAINERS=true`** (org-set; not re-verified this session).\n\n## Rolling artifact replacement\n\n```python\nhttpx.post(\n    f\"{base}/workloads/{wid}/replacement/\",\n    headers=headers,\n    json={\n        \"artifactId\": new_artifact_id,\n        \"strategy\": \"rolling\",  # only \"rolling\" supported\n        \"config\": {\"warmupDurationMinutes\": 2, \"keepOldVersionMinutes\": 5},  # optional\n        # \"runtime\": {...}  # optional; same shape as PATCH /settings/\n    },\n)\n```\n\nMonitor: `python scripts/wait_for_replacement.py <workload_id>`. Preconditions: status must match (draft↔draft/locked↔locked, else `400`); same-artifact replacement 422s for locked, works for drafts — for a draft roll without replacement, use `PATCH /settings/`. **Not idempotent** — a second `POST` queues another swap. `GET .../replacement/` `404` = none in progress. `DELETE` to cancel. Detail: `references/lifecycle-flows.md`.\n\n---\n\n## Related skills\n\n- `datarobot-setup` — install SDK, configure auth, set env vars\n- `datarobot-app-framework-cicd` — declarative artifact + workload management via Pulumi and CI/CD\n- `datarobot-external-agent-monitoring` — instrument arbitrary agent code with OTEL → DataRobot\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}