DataRobot Skills

View saved version →

---
name: datarobot-agent-assist
description: >-
  Use when the user wants to design, build, code, simulate, or deploy an AI agent (not a predictive
  model) to DataRobot; mentions agent_spec.md, dr-assist, datarobot-agent-assist, dress rehearsal,
  swarm simulation, or the DataRobot agent template; wants to scaffold a LangGraph, CrewAI, LlamaIndex, NAT, or Base
  agent targeting DataRobot; wants to add an MCP server, backend API, or React frontend to a
  DataRobot agent application; or uses the DataRobot CLI (dr) to build or deploy an agentic custom
  application; or wants to harden, stress-test, or battle-test an agent. Covers the full workflow: agent design, agent_spec.md authoring, dress-rehearsal
  simulation via the DataRobot LLM Gateway, adversarial swarm simulation, template-based coding, and deployment.
---

# DataRobot Agent Assist

This skill covers **agent design, coding, battle-testing, and deployment** with an optional **dress-rehearsal simulation** before any code is written.

A first message of `1`–`4` selects the corresponding category. If free text clearly maps to one category (e.g. "I want to build an agent" → Design), state the inference briefly and proceed. If ambiguous, ask which option applies — do not guess.

---

## Workflow Discipline

Follow this skill **sequentially**. These rules apply to every phase — design, coding, deployment, and every referenced checklist.

1. **Section order** — Complete each section and subsection in the order it appears in this file. Do not jump to a later section until the current one is finished.
2. **Reference files** — When this skill says **read and follow** a file under `agent-assist-build/references/`, read that file first, then execute every step in that file in order. Do not substitute a summary, shortcut, or a later menu for steps defined in the reference.
3. **Explicit skips only** — Skip a step only when this skill or the referenced file explicitly says to skip it (e.g. Pre-requisite Check when `<prerequisites_passed>` is true, or Frontend Check when `frontend.type` is already set).
4. **No auto-advance** — Completing one step (e.g. writing the spec, the user saying "move on", or a command succeeding) does not authorize skipping remaining steps. Proceed only when the current section or reference directs you to the next step.
5. **Menus and prompts** — When a section presents a menu or asks a question, wait for the user's reply. Do not assume a default or proceed on your own.
6. **One design gate per turn** — During design, do not combine prompts from different subsections in one message (e.g. do not ask about spec refinement and dress rehearsal in the same turn).

---

## On Activation

Present the four options clearly:

```
Welcome! I help you design, code, battle-test, and deploy AI agents.

What would you like to do?
  1. Design an AI agent     → Describe your idea (optional dress rehearsal before coding)
  2. Code an AI agent       → Load and implement an existing agent_spec.md
  3. Battle-test the agent  → Run adversarial swarm simulation on an implemented agent
  4. Deploy                 → Deploy an implemented agent to DataRobot
```

Show this menu first. After the user selects an option (`1`, `2`, `3`, or `4`), run the **[Pre-requisite Check](#pre-requisite-check)** (once per session) and then the **[Script Path Resolution](#script-path-resolution)**.

- Options **1** and **2** — read and follow [agent-assist-build/references/workspace-resolution.md](agent-assist-build/references/workspace-resolution.md), then proceed to the selected workflow.
- Option **3** — read `agent-assist-simulate/SKILL.md` and jump to **Pre-flight Check** (Pre-requisite Check and Script Path Resolution still apply first).
- Option **4** — skip Workspace Resolution; `<target_dir>` is resolved in the [Pre-deployment Checklist](agent-assist-build/references/pre-deployment-checklist.md) when unset.

---

## Script Path Resolution

Before invoking any helper script, resolve `<skill_scripts_dir>` once for the session:

- `<skill_scripts_dir>` is the `agent-assist-build/scripts/` subdirectory of the directory containing this `SKILL.md` file.
- Confirm it exists with `ls <path_to_this_skill_dir>/agent-assist-build/scripts/`. If the directory is missing, tell the user the skill installation is incomplete and stop.
- Use the resolved absolute path for every `<skill_scripts_dir>/...` reference in this skill.

---

## Session State

Track these for the conversation:

- `<target_dir>` — project root. Set during [Workspace Resolution](agent-assist-build/references/workspace-resolution.md) or the [Pre-deployment Checklist](agent-assist-build/references/pre-deployment-checklist.md). Reuse across phases. Change only on explicit user request, [pre-coding Bootstrap step 2](agent-assist-build/references/pre-coding-checklist.md) (spec path recovery), [pre-coding step 7](agent-assist-build/references/pre-coding-checklist.md) (subdir recovery), or a new session.
- `<prerequisites_passed>` — `false` until the [Pre-requisite Check](#pre-requisite-check) completes successfully once this session.
- `<workspace_resolved>` — `false` until [Workspace Resolution](agent-assist-build/references/workspace-resolution.md) completes for menu options **1** or **2**.
- `<workspace_resolved_target_dir>` — the `<target_dir>` set when workspace resolution last completed.
- `<design_to_code>` — `false` until the user chooses **Code the agent** from [Post-design next steps](#post-design-next-steps) in the same session after design.
- `<design_messy_cwd>` — `false` until design runs in cwd with files other than `agent_spec.md` / `.env` (see [workspace-resolution](agent-assist-build/references/workspace-resolution.md)).
- `<dependency_check_passed>` — `false` until a passing `dr dependency check` in `<target_dir>`.
- `<dependency_check_target_dir>` — the `<target_dir>` value when the last check passed.

### `.env` placement

Project credentials and config live **only** at `<target_dir>/.env` — never in cwd when `<target_dir>` is a subdirectory. This includes DataRobot CLI credentials, LLM config, and **tool/service secrets**. Pulumi reads these from the environment at deploy time (`os.environ`); a secret missing from `.env` means the corresponding credential and runtime parameter are not created.

- Complete [Workspace Resolution](agent-assist-build/references/workspace-resolution.md) before any step that needs credentials or creates a `.env` file.
- Always pass `--target-dir <target_dir>` to helper scripts that read or create `.env` (`list_llm_models.py`, `rehearsal.py`, `setup_template.py`). Do **not** run `dr dotenv setup` in cwd.
- Never put secret values in `agent_spec.md` or source code — only env var **names** belong in infra; values stay in `.env`.

### Dependency check session rule

Before running dependency validation:

- If `<dependency_check_passed>` is true and `<target_dir>` equals `<dependency_check_target_dir>`, skip validation.
- Otherwise, read and follow [agent-assist-build/references/dependency-validation.md](agent-assist-build/references/dependency-validation.md) in `<target_dir>`.

Invalidate `<dependency_check_passed>` (set to false) when:

- `<target_dir>` changes
- `clone_template.py`, `select_framework.py`, or `setup_template.py` runs in `<target_dir>`

### Welcome menu reset

When showing the [welcome menu](#on-activation) again (e.g. user declined pre-coding step 7): set `<workspace_resolved>` = false, clear `<workspace_resolved_target_dir>`, `<design_to_code>` = false, and `<design_messy_cwd>` = false. Keep `<prerequisites_passed>` true.

---

## Workspace Resolution

Read and follow [agent-assist-build/references/workspace-resolution.md](agent-assist-build/references/workspace-resolution.md) after menu options **1** or **2**.

---

## Pre-requisite Check

If `<prerequisites_passed>` is true, skip this section.

Otherwise, run in order before proceeding:

1. **Git** — run `git --version`. If missing, tell the user to install from https://git-scm.com and stop.
2. **Python** — run `python --version`. If missing or below 3.11, tell the user to install Python 3.11+ from https://python.org and stop.
3. **DataRobot CLI** — read and follow [agent-assist-build/references/dr-cli-setup.md](agent-assist-build/references/dr-cli-setup.md):
   - If missing, **ALWAYS RUN** the install command before proceeding
   - **ALWAYS RUN** the upgrade command before proceeding
   - If not authenticated, **ALWAYS RUN** the auth command before proceeding
4. **Codespace** — run `python <skill_scripts_dir>/check_codespace.py` (no-op outside a Codespace). On non-zero exit, relay its message and stop; otherwise relay any exposed-ports warning it prints.

On success, set `<prerequisites_passed>` = true.

---

## 1. Designing an AI Agent

When `<target_dir>/agent_spec.md` already exists, read and follow [resume-design.md](agent-assist-build/references/resume-design.md) after [workspace resolution](agent-assist-build/references/workspace-resolution.md) or when returning from [Spec issues](agent-assist-build/references/pre-coding-checklist.md#spec-issues). For a **new** agent (no spec yet), start at [Clarification Phase](#clarification-phase).

### Clarification Phase

- Ask **at most 2 rounds** of clarifying questions before proposing an initial draft spec. If tools are still ambiguous after two rounds, start simple.
- Focus questions on:
  - What the agent does and who uses it
  - What tools it needs and what external services those tools call
  - Whether those services require authentication (API key, OAuth2, bearer token, etc.)
  - Whether the user needs a custom frontend beyond the default chat UI

- If the user mentions UI-related needs early ("dashboard", "visualization", "multi-page", "admin panel", "settings page"), capture it immediately in the `frontend` field and write `frontend.type` to `agent_spec.md` — do **not** defer or wait for [Frontend Check](#frontend-check).

### Model Selection

- To check available models, run (requires `<target_dir>` from workspace resolution — see [.env placement](#env-placement)):

   ```
   python <skill_scripts_dir>/list_llm_models.py \
     --json \
     --target-dir <target_dir>
   ```

  **CRITICAL**: Always pass `--target-dir <target_dir>`. In case the script fails due to any reason, do **not** proceed. Instead, return the error message to the user and ask how they want to proceed.

- Read and follow [llm-selection.md](agent-assist-build/references/llm-selection.md) to recommend from the two sources (`gateway` and `deployed`) and record the choice.
- If the user's desired model is unavailable, suggest starting with an available one and updating after implementation.

### Frontend Check

Skip this section if `frontend.type` is already captured — either written to `<target_dir>/agent_spec.md`, or held in working memory from [Clarification Phase](#clarification-phase) before the first draft exists.

Before the first spec draft, if `frontend.type` is not yet set, **always ask**:

> "The template includes a default chat UI — is that sufficient, or would you like a custom frontend such as a dashboard, data visualization, or multi-page app?"

Then set `frontend` in the spec (write to `<target_dir>/agent_spec.md` when the file exists, or hold the value for the first draft in [Spec Display](#spec-display)):
- Default UI → `frontend.type: "chat"`
- Custom UI → `frontend.type: "multi-page"` or `"custom"` with `pages` and optional `requirements`

### Spec Display

- Before the first draft, read [agent-assist-build/references/agent-spec-schema.md](agent-assist-build/references/agent-spec-schema.md).
- **Always write the current spec to `<target_dir>/agent_spec.md`** (YAML format) whenever showing it to the user. The first draft must include `frontend.type` from [Frontend Check](#frontend-check) (or clarification).
- Show the spec frequently and iteratively — even if incomplete or partial.
- Do **not** summarize the spec in prose; display it as YAML in a code block.
- After displaying, invite the user to refine system prompts, tools, model, or examples. Do **not** ask about dress rehearsal, coding, template setup, or "moving on" in the same turn, and do **not** offer "proceed to coding" as an alternative to refinement (including on [Resume Design](agent-assist-build/references/resume-design.md) after pre-coding [Spec issues](agent-assist-build/references/pre-coding-checklist.md#spec-issues)).
- If the user requests changes, update the spec and show it again. If the user indicates they are done refining (e.g. "looks good", "no changes", "move on"), proceed to [Agent Simulation (Before Coding)](#agent-simulation-before-coding) in your **next** response — not to [Post-design next steps](#post-design-next-steps) or coding.

### Agent Simulation (Before Coding)

When spec refinement is complete, read [agent-assist-build/references/dress-rehearsal.md](agent-assist-build/references/dress-rehearsal.md) and present the **Initial prompt (design phase)** using that file's **exact wording** — in its own turn, with no spec-refinement question in the same message.

Then follow dress-rehearsal.md for the user's reply (yes → rehearsal; no → [Post-design next steps](#post-design-next-steps)).

### Post-design next steps

After the user declines the initial rehearsal prompt — or **after** a dress rehearsal session ends **and** `<target_dir>/rehearsal_report/rehearsal_report.md` has been written — present this menu (exact wording):

> What would you like to do next?
> 1. **Code the agent** — start implementation from `agent_spec.md`
> 2. **Review / edit spec** — refine `agent_spec.md`
> 3. **Run dress rehearsal** — simulate the agent before coding
> 4. **Review rehearsal report** — open `rehearsal_report/rehearsal_report.md`

Wait for their choice. **Do not** assume a default or proceed without a reply.

| Choice | Action |
|--------|--------|
| 1 or "code" / "implement" | Set `<design_to_code>` = true. Follow **[2. Coding an AI Agent](#2-coding-an-ai-agent)** — read and follow [pre-coding-checklist.md](agent-assist-build/references/pre-coding-checklist.md). This does **not** authorize cloning or subdirectory creation; if the workspace is not spec-only, complete [pre-coding step 7](agent-assist-build/references/pre-coding-checklist.md) and wait for explicit confirmation first. |
| 2 or "review" / "edit spec" | Display `<target_dir>/agent_spec.md` as YAML, invite changes, update the file, then show this menu again |
| 3 or "rehearsal" / "simulate" | Follow **[Dress Rehearsal](#dress-rehearsal)** |
| 4 or "review report" / "rehearsal report" | If `<target_dir>/rehearsal_report/rehearsal_report.md` exists, read it and present a structured summary (metadata, notes, conversation highlights, suggested changes). If missing, say no report exists yet and offer option 3. Then show this menu again. |

If the user's reply is unclear, re-display the menu and wait. Never skip straight to framework selection after a rehearsal decline.

---

## Dress Rehearsal

Read and follow [agent-assist-build/references/dress-rehearsal.md](agent-assist-build/references/dress-rehearsal.md) end to end.

---

## 2. Coding an AI Agent

**On Windows: coding is not supported. STOP and do NOT proceed with the next steps!**

### Pre-coding Checklist

Read and follow [agent-assist-build/references/pre-coding-checklist.md](agent-assist-build/references/pre-coding-checklist.md) end to end before writing or editing implementation code. Do not write or edit implementation code until the checklist is complete.

### Coding Rules

- Implement by adapting the template code — do not write from scratch
- Modify files only inside `<target_dir>` and its subdirectories
- Do not view `.env` files (`.env.template` files are OK)
- **Tool credentials** — when implementing a tool with `auth_spec` in `agent_spec.md`:
  1. Choose a `SCREAMING_SNAKE_CASE` env var name (e.g. `PERPLEXITY_API_KEY`).
  2. **Append** `VAR_NAME=` to `<target_dir>/.env` without reading the file.
  3. Ask the user to paste the secret into `<target_dir>/.env` in their editor — never in chat, `agent_spec.md`, or committed code.
- Do not add code comments unless asked
- Do not mock tool implementations unless they would be complex to implement
- For tasks with 3+ steps, use the TodoWrite tool to manage your work
- Keep text responses **concise (1–3 sentences)** while coding — skip preamble and postamble

### File Write/Edit Discipline

- Always explain **why** the change is needed (purpose and impact) in 1–2 sentences before writing or editing a file
- Invoke at most **one shell command per response** — wait for the result before invoking another

### After Coding

1. Read `<target_dir>/AGENTS.md` to find the local test command.
2. Display the command in a code block.
3. Tell the user: "Run this command in a **new terminal** in `<target_dir>` to test the agent locally."
4. Do **not** run the command yourself.
5. Present next steps: revise the implementation, battle-test the agent (option 3), or deploy to DataRobot (option 4).

---

## 3. Battle-testing an AI Agent

Read `agent-assist-simulate/SKILL.md` and jump directly to **Pre-flight Check** — skip its On Activation menu. Also trigger directly when the user says "simulate my agent", "run swarm", "adversarial testing", "harden my agent", or "test my agent".

Swarm requires an implemented agent; if none exists, explain and offer option 2. If code exists and no option is chosen, proactively offer: "I can also battle-test your agent before deploying — want to run swarm simulation?"

---

## 4. Deploying an AI Agent

Read and follow [agent-assist-build/references/pre-deployment-checklist.md](agent-assist-build/references/pre-deployment-checklist.md) end to end.

---

## Error Handling

- If a tool returns an error, read the error message carefully before responding
- For dependency validation failures that cannot be fixed (install or re-check still fails): hard stop — return full output from all commands run (see [Dependency validation](agent-assist-build/references/dependency-validation.md))
- On unexpected errors, ask the user if they want to retry

Helper-script failures during pre-coding are governed by the **CRITICAL** rule in [pre-coding-checklist.md](agent-assist-build/references/pre-coding-checklist.md).

---

## agent_spec.md

Write specs as YAML to `<target_dir>/agent_spec.md`. Fields are optional while the spec is evolving.

Field definitions: [agent-assist-build/references/agent-spec-schema.md](agent-assist-build/references/agent-spec-schema.md). Complete examples: [agent-assist-build/references/agent-spec-examples.md](agent-assist-build/references/agent-spec-examples.md).

---

## Tool/Helper Scripts Timeouts

- Allow up to 25 minutes for any helper script to complete before timing out and returning an error
- Allow up to 25 minutes for any shell command or tool to return a response before timing out and returning an error
- Allow up to 60 minutes for deployment-related shell commands to complete before timing out and returning an error

---


## Behavioral Rules

- Follow [Workflow Discipline](#workflow-discipline) at all times
- Welcome-menu routing: prefer `1`–`4`; for clear free-text intent, infer and state the category then proceed; if ambiguous, ask which option applies
- If the user insists on a task outside these four categories, politely decline
- If a user asks to code before designing, strongly encourage designing first
- Before running any CLI command or helper script, explain in 2–5 sentences why it is needed now and what it will check/change/create
- **Template clone / spec validation / spec issues** — follow [pre-coding-checklist.md](agent-assist-build/references/pre-coding-checklist.md) ([Clone discipline](agent-assist-build/references/pre-coding-checklist.md#clone-discipline), Bootstrap step 2, [Spec issues](agent-assist-build/references/pre-coding-checklist.md#spec-issues)) and [spec complete](agent-assist-build/references/resume-design.md#spec-complete)
- After the user declines dress rehearsal, always show **[Post-design next steps](#post-design-next-steps)** — never skip to framework selection or the pre-coding checklist
- During rehearsal, follow [dress-rehearsal.md](agent-assist-build/references/dress-rehearsal.md): display `output_file` verbatim; on **DONE** always run `--report` before any summary or menu; write `<target_dir>/rehearsal_report/rehearsal_report.md` before showing Post-design next steps
- During **coding**: keep responses to 1–3 sentences; no introductions or conclusions
- During **design**: be conversational and thorough

For helper script commands, see [helper-scripts.md](agent-assist-build/references/helper-scripts.md). For plugin tool mapping, see [tool-mapping.md](agent-assist-build/references/tool-mapping.md).

Referenced files: 39 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-agent-llm-selection
description: >-
  Use when the user wants to configure LLM integration for a DataRobot agent
  application. This skill helps to change LLM model, switch between the LLM
  integrations the project supports (LLM Gateway, a DataRobot-deployed LLM, an
  external provider, an LLM Blueprint, and any others its config declares), or
  set up provider credentials. The skill reads the project's
  .datarobot/cli/llm.yml for the real options, interviews the user, then runs
  sync_llm_env.py with the chosen values as CLI args to merge into .env.
---

# DataRobot LLM gateway configuration

Configure LLM integration **without hand-editing `.env`**. The skill drives
`sync_llm_env.py` with the user's answers as CLI arguments.

## Resolve script path once per session

`<skill_scripts_dir>` = the `scripts/` subdirectory of the directory containing this `SKILL.md`.

```shell
ls <skill_scripts_dir>/sync_llm_env.py
```

## Hard rules

1. **Never** ask the user to paste API keys or `DATAROBOT_API_TOKEN` in chat
2. **Never** read, copy, echo, or pass `DATAROBOT_API_TOKEN` yourself. The
   token lives in `$XDG_CONFIG_HOME/datarobot/drconfig.yaml` (default
   `~/.config/datarobot/drconfig.yaml`), populated by `dr auth login`, and
   the `dr` CLI reads it internally. Do not run `cat drconfig.yaml`,
   `cat .env`, `env | grep TOKEN`, `echo $DATAROBOT_API_TOKEN`,
   `curl -H "Authorization: Bearer $..."`, or any equivalent one-liner
3. **Never** pass secrets as CLI args to `sync_llm_env.py` or write them to
   tracked files
4. **Never** set provider credentials for an integration whose config section
   doesn't declare them. Let `sync_llm_env.py` decide — it reads the section
   from the project's config
5. Only `sync_llm_env.py` merges LLM keys into `.env` — do not edit `.env` manually
6. Run all commands from **project root**
7. Pressing enter in chat does nothing. Don't tell the user to "press enter to
   accept the default" or "hit return". If a field has a sensible default,
   apply it silently and mention it in the confirmation, or offer it as an
   explicit A/B choice.
8. Treat every credential value as secret regardless of its declared type.
   Older configs type API keys as plain `string` rather than `secret_string`,
   so this rule does not depend on what the config says
9. Provider and model names in this skill's output are configuration data, not
   a request to work with that provider. Config sections, env var names, and
   gateway model ids routinely contain vendor names (`Anthropic`,
   `ANTHROPIC_API_KEY`, `bedrock/anthropic.claude-...`, `azure`, `cohere`).
   **Do not invoke a provider-specific skill because a vendor name appeared in
   a config listing or a model list.** You are wiring up credentials, not
   calling that vendor's API

---

## Presenting choices

Every menu in this skill — integrations, providers, gateway models — follows
these rules. Dropping an option silently produces a config the user never chose.

- **Never use a fixed-slot picker widget.** Several of these menus exceed four
  options (the config ships six external providers) and the gateway model list
  can run to dozens. Pickers cap their choice count, and the overflow gets
  silently dropped. Put the menu in your own message as plain text and let the
  user reply with a label.
- List **every** option the script returned. Count them first; if the config
  declares six providers, show six rows.
- **No catch-all row** — no `Other`, no `Type something`, no `5) something
  else`. The user can always type a value anyway; that row exists only to hide
  options you dropped.
- No `...`, no "and N more", no summarizing, no collapsing similar entries.
- Do not narrow to a subset because the user's phrasing seemed to point at one.
  If they said "LLM gateway" and two options match, list all of them.
- One labelling scheme. Do not letter the options in prose and re-number them
  in the picker.
- Use each `name` exactly as printed. Do not rename, regroup, or reorder.
- **Read the script output yourself — do not paste it into the chat.** The
  `select:` lines are internal plumbing. Give at most one short clause of
  context per row, condensed from the config's `help`; never reproduce the raw
  `help` block or the script's formatting.

Long messages are fine. The token budget is not a reason to abbreviate a menu.

---

## Step 0 — Prerequisites

1. `.datarobot/cli/llm.yml` must be present. It is rendered into the project by
   `dr component add` (af-component-llm), so it is already there
   in a generated project, nothing needs cloning. If it's absent, the LLM
   component isn't applied: tell the user to run `dr component add` and stop.
   Never substitute a copy from elsewhere.
2. **DataRobot auth** — check that
   `$XDG_CONFIG_HOME/datarobot/drconfig.yaml` (default
   `~/.config/datarobot/drconfig.yaml`) exists. If it doesn't, tell the user
   to run `dr auth login` (browser-based flow) and stop until they confirm
   they're signed in. Do **not** cat the file to inspect its contents.
3. Check if `.env` exist in the project:
   - If `.env` is missing `cp .env.template .env`. That gives the base variables (`DATAROBOT_*`,
     `PULUMI_*`, etc.) many are blank, it will be filled later. But the sync in Step 3
     only needs the file to exist.
   - If both `.env` and no `.env.template` are missing, tell the user they're not in
     a DataRobot agent project root and stop.

---

## Step 1 — Read the project's integration options (MANDATORY FIRST)

Never recite integration names from memory or from this file. Each project's
`.datarobot/cli/llm.yml` declares its own, and they differ by component version.

```shell
python <skill_scripts_dir>/read_llm_config.py
```
If it reports the file is missing, the LLM component isn't applied to this
project. Tell the user to run `dr component add` and stop.
Do **not** run `dr llm-gateway list`, do **not** offer a model list, and do
**not** write any config file until the user has picked an option.

### Check what's already configured

The `default:` line in the script's output is the schema's declared fallback,
**not** necessarily the project's live setting — don't conflate the two.
Before presenting the menu, find the actual current value:

```shell
grep -E '^INFRA_ENABLE_LLM=' .env
```

- If `.env` doesn't exist yet or has no match, tell the user no LLM
  integration is configured yet.
- If it matches, map the value against the `select:` lines the script just
  printed to report the current integration by its friendly name (e.g.
  `blueprint_with_external_llm.py` → "External LLM"), not the raw filename.
- If that option has a `further choice:` (a provider), also run
  `python <skill_scripts_dir>/read_llm_config.py --option <current_value>`
  and grep `.env` for the **names** of the env vars it lists, to report which
  are already populated. Report only whether each key is set — per Hard
  Rule 8, never print the value of a credential-shaped key, even to confirm
  it's set.

State this summary — e.g. "You're currently on `<friendly name>`[, provider
`<provider>`], with `<N>` of `<M>` fields already set" — before showing the
menu, so the user can decide to switch entirely or just change specific
fields on the current integration.

Present the options per **Presenting choices** above.

Each option in the output carries an indented `select:` line. Once the user
picks, take that string verbatim — quote it if it contains spaces. The steps
below call it `<selected_gateway>`; it is also the `INFRA_ENABLE_LLM` value
written in Step 3.
---

## Step 2 — Read what that option needs

```shell
python <skill_scripts_dir>/read_llm_config.py --option <selected_gateway>
```

This prints the env vars the chosen integration declares. Handle each field by
what the script reports about it:

- **`hidden`** — never ask. Step 3 writes its default.
- **`llmgw_catalog`** — populate from the DataRobot CLI (see below).
- **`required`** — ask, using the printed help as the prompt text.
- **`optional` with a default** — apply it silently and say so in the
  confirmation. Do not tell the user to "press enter".
- **`further choice:`** — another menu, presented per **Presenting choices**.
  Show every row the script listed, with no catch-all and no picker widget.
  Those rows carry their own `select:` field; call the user's pick
  `<selected_provider>` and re-run `--option <selected_provider>` for its keys.

### Fields typed `llmgw_catalog`

Fetch the model list **only** via the DataRobot CLI. Run exactly:

```shell
dr llm-gateway list --output-format json
```

The CLI authenticates via its own credential store (populated by
`dr auth login`). Do **not** read `drconfig.yaml` or `.env` for the token, and
do **not** pass `DATAROBOT_API_TOKEN` on the command line. If the command exits
non-zero or prompts for auth, tell the user to run `dr auth login` and stop —
do not attempt any manual API call and do not fabricate a menu.

Parse the JSON, which has the shape
`{"llms": [{"id", "name", "provider", "model", "selected"}, ...]}`. Use the
`model` field for each entry. The ids in the menu **must** come from that JSON,
verbatim, in the order returned. Do not invent ids or reuse them from your
training data. If the command did not produce JSON, stop and report the error.

Count the entries; call it `N`. Present **exactly `N` labelled lines**, one per
model, per **Presenting choices**. The letter scheme is `A..Z`, then `AA..AZ`,
`BA..BZ`, and so on. This list is the clearest case for never using a picker
widget — it routinely exceeds any picker's capacity.

`sync_llm_env.py` prepends `datarobot/` for this field type, so pass the id
exactly as the CLI returned it.

### Recommending

Before presenting the list, check whether `agent_spec.md` exists at the
project root. If it does, read it for context — the agent's purpose, task
complexity, and any stated cost or latency constraint — and use that to flag
one entry as recommended. This mirrors the recommendation logic in the
`datarobot-agent-assist` skill's `llm-selection.md`:

- Prefer a `gpt-5`, `claude-4-5`/`4-6`/`4-8`, or `gemini-2.5`/`3` family
  gateway model unless the spec or the user states a cost or latency
  constraint.
- If the spec calls for low cost or latency, prefer the `mini`/`haiku`/`flash`
  tier of that same family instead.
- If no gateway entries appear at all (only `deployed` ones), recommend a
  deployed entry instead and say why — the LLM Gateway is disabled or empty
  on this instance, which is normal for an on-prem install, not an error.
- **Still list every entry** per **Presenting choices** — mark the
  recommended row (e.g. append "(recommended)") rather than omitting the
  rest. Recommending never justifies narrowing the menu.
- No `agent_spec.md`? Skip the read and recommend using only the
  family-preference heuristics above.

### Credential keys

**Announce them up front** — do not let the user discover them via the sync
error message. Once the further choice is made, re-run
`--option <selected_provider>` and
tell the user exactly which keys they need and where the file lives:
```
For <choice>, I'll need these values in a per-user credentials file:
  ~/.config/datarobot/llm-<section>.env (or $XDG_CONFIG_HOME/...)

  <KEY_1>
  <KEY_2>
  ...
Step 3 will create that file as a blank template, with the help text for
each key as a comment, if it doesn't exist. Please fill it in your own
editor — do not paste the values in chat — then tell me "credentials ready"
and I'll re-run the sync.
```
Use the section name the script reported for `<section>`. Do not create the
file yourself, do not `cat` it, and do not accept secret values in chat.
---

## Step 3 — Sync into `.env`

First back up the current `.env` (sync overwrites it, and `.env` isn't in git):

```shell
cp .env .env.bak.$(date +%Y%m%d%H%M%S)
```

Run the sync script with the values collected in Step 2 as CLI args. No
intermediate config file, no JSON to write.

```shell
python <skill_scripts_dir>/sync_llm_env.py \
  --infra-enable-llm <selected_gateway> \
  --set <ENV_NAME>=<value>
```
Repeat `--set` once per value, using the env var names the script printed in
Step 2 — they carry a per-project prefix and are **not** always `LLM_*`. Add
`--provider "<selected_provider>"` when Step 2 reported a further choice. Pass no
`--set` for fields reported as `hidden`; the script writes their defaults.
When the chosen option needs credentials, the script reads them from
`$XDG_CONFIG_HOME/datarobot/llm-<section>.env`:
- **If the file doesn't exist**, the script writes a blank template there
  and exits with the path plus the required key list. Relay that verbatim
  to the user, tell them to fill it in their own editor, then re-run the
  same command. Do not offer to create the file for them and do not accept
  values in chat.
- **If the file exists but is incomplete**, the script prints the missing
  keys and exits. Same instruction: user edits, then re-runs.
- **If the file is complete**, the sync merges the credentials into `.env`
  in one shot.
---

## Step 4 — Validate and hand off

**`dr dotenv validate` echoes the full `.env` (including `DATAROBOT_API_TOKEN`)
to stdout.** If you run it without redirection, the token lands in the chat
transcript and must be rotated. Same risk for `dr dotenv update`, `dr task run`,
`dr run`, `cat .env`, `env | grep`, or any other command that reads `.env`.

Run validation with all output suppressed and check only the exit code:

```shell
dr dotenv validate >/dev/null 2>&1
```

- **Exit 0** → tell the user validation passed.
- **Non-zero exit** → do **not** re-run the command with output visible.
  Tell the user to run `dr dotenv validate` themselves in their own terminal
  so the error stays local.
Then tell the user (do not run these yourself — they also echo secrets):
```text
LLM configuration synced to .env.

Please run these yourself in your terminal:
  dr dotenv update          # refresh DataRobot token if needed
  dr task run infra:up-yes  # push runtime params to deployment
  dr run dev                # local test
```
---

## Stale keys

`sync_llm_env.py` derives its managed-key set from every env var declared
anywhere in the project's config, so switching integrations clears whatever the
previous one wrote before the fresh block goes in. Non-LLM `.env` lines are
preserved.

Referenced files: 2 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-app-framework-cicd
description: Guidance for setting up CI/CD pipelines for DataRobot application templates using GitLab, GitHub Actions, and Pulumi for infrastructure as code. Use when setting up CI/CD pipelines, configuring deployments, or managing infrastructure for DataRobot application templates.
context-tokens: "~6 000 (SKILL.md) + ~2 000 (scripts/*) + ~400 per examples/* file"
---

# DataRobot Application Templates CI/CD Skill

This skill provides comprehensive guidance for setting up production-grade CI/CD pipelines for DataRobot application templates, including automated testing, review deployments, and continuous delivery.

## Quick Start

**Default behavior:** When a user asks to "set up CI/CD" without specifying a platform or backend, always use the [Simple Path](#simple-path-pulumi-cloud--github-secrets) below — three workflow files, two GitHub Secrets, done. Do not create `infra/scripts/`, do not add CI/CD tasks to `infra/Taskfile.yaml`, do not involve GPG encryption unless the user explicitly asks for it.

Only deviate from the simple path when the user specifies:
- A specific Pulumi state backend (Azure Blob, S3, GCS) → use `scripts/` and see [Implementation Pattern](#implementation-pattern)
- GitLab CI/CD → see [GitLab CI/CD Configuration](#gitlab-cicd-configuration)
- Many secrets to manage → consider GPG approach in `scripts/`

## Simple Path: Pulumi Cloud + GitHub Secrets

For most data scientists and AI engineers, this is all you need. No GPG encryption, no cloud storage account, no extra scripts.

**What to create in the user's repository:**

1. Copy the three workflow files to `.github/workflows/`:

   | Source | Destination | Trigger |
   |--------|-------------|---------|
   | `examples/github-cd-pulumi-cloud.yml` | `.github/workflows/cd.yml` | Automatic — every merge to `main` |
   | `examples/github-deploy-pulumi-cloud.yml` | `.github/workflows/deploy-pr.yml` | Manual — user picks PR branch + enters stack name (e.g. `pr-42`) |
   | `examples/github-destroy-pulumi-cloud.yml` | `.github/workflows/destroy.yml` | Manual — user enters stack name to tear down |

2. Create `.github/workflows/README.md` from `examples/workflows-README.md`. This is the setup guide that tells the user exactly what secrets and variables to add and how.

3. Tell the user to follow the setup guide in `.github/workflows/README.md`.

That's it. Do **not** add anything to `infra/Taskfile.yaml` or create `infra/scripts/` for this path.

**Required GitHub Secrets** (both required — no defaults):

| Name | Kind |
|------|------|
| `DATAROBOT_API_TOKEN` | Secret |
| `PULUMI_ACCESS_TOKEN` | Secret |

**Optional GitHub Variable** (defaults to `ci` if not set):

| Name | Kind | Default |
|------|------|---------|
| `PULUMI_STACK_CI_NAME` | Variable | `ci` |

**When to use the advanced approach (GPG + DIY backends) instead:**
- You have many secrets (GPG encrypts all of `.env` behind a single passphrase — only one GitHub Secret needed)
- Your organization prohibits Pulumi Cloud and requires a self-managed backend (Azure Blob / S3 / GCS)
- You need GitLab CI/CD

The templates and scripts for all of these are in `scripts/` in this skill directory. If the skill has already been propagated to the project's `infra/` directory (common in downstream templates), look in `infra/scripts/` instead. See the [Implementation Pattern](#implementation-pattern) section below for full setup guidance.

| Scenario | Key files in `scripts/` |
|----------|------------------------|
| Azure Blob / S3 / GCS Pulumi backend | `pulumi-setup.sh`, `taskfile-snippets.yaml` |
| GitHub Actions + GPG secrets | `github-deploy.yml`, `github-cd.yml`, `encrypt-secrets.sh`, `setup-github-secrets.sh` |
| GitLab CI/CD | `gitlab-ci.yml`, `setup-gitlab-variables.sh` |

### Adapting the deploy command

The example workflows use `uv run pulumi up --yes` directly. Before copying them, check `infra/Taskfile.yaml` — the project may already wrap the deploy command in a task:

```bash
cat infra/Taskfile.yaml   # look for 'up-yes', 'deploy', or similar tasks
```

| What you find | What to use in CI |
|---------------|-------------------|
| `up-yes` task | `task up-yes` — non-interactive, purpose-built for CI; prefer this over raw Pulumi |
| `deploy` task (alias for `up`) | Avoid — typically runs `pulumi up` interactively; only safe in CI if you confirm it passes `-y` internally |
| No Taskfile or no relevant task | Keep `uv run pulumi up --yes` as-is |

To use `task` in a workflow, add an install step and swap the run command:

```yaml
- name: Install Task
  run: pip install go-task-bin

- name: Deploy
  working-directory: infra
  env:
    DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }}
    PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
  run: |
    uv sync --all-extras
    task up-yes
```

### DataRobot API token (service account)

`DATAROBOT_API_TOKEN` should come from a **DataRobot service account** — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team.

To set one up: ask your DataRobot admin to create a dedicated user (e.g. `ci-bot@your-org.com`). Under that account, go to **Developer Tools → API Key** and generate a token. Store it as the `DATAROBOT_API_TOKEN` secret in GitHub.

> **Note:** This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. "Service account" here just means a non-personal DataRobot user.

## Implementation Pattern

When implementing CI/CD for an application template, follow this structure:

**Project Structure:**
```
application-template-root/
├── infra/
│   ├── README.md                   # ⚠️ GENERATE THIS — tailored to the chosen CI/CD platform and Pulumi backend
│   ├── Taskfile.yaml               # ⚠️ CI/CD tasks go HERE — copy from infra/scripts/taskfile-snippets.yaml
│   └── scripts/                    # Copy entire scripts/ directory here
│       ├── README.md               # Copy from scripts/infra-README.md
│       ├── setup-github-secrets.sh
│       ├── setup-gitlab-variables.sh
│       ├── encrypt-secrets.sh
│       ├── decrypt-secrets.sh
│       ├── pulumi-setup.sh
│       ├── gitlab-ci.yml
│       ├── github-deploy.yml
│       ├── github-cd.yml
│       ├── github-destroy.yml
│       └── taskfile-snippets.yaml
├── .env                            # User's secrets (never commit!)
├── .env.gpg                        # Encrypted secrets (commit for GitHub)
├── .gitlab-ci.yml                  # Copy from infra/scripts/gitlab-ci.yml
├── .github/
│   └── workflows/
│       ├── deploy.yml              # Copy from infra/scripts/github-deploy.yml (PR review deploys)
│       ├── cd.yml                  # Copy from infra/scripts/github-cd.yml (push-to-main CD)
│       └── destroy.yml             # Copy from infra/scripts/github-destroy.yml
└── Taskfile.yml                    # Root Taskfile — ADD ONLY one `includes` entry (see below). DO NOT add tasks here.
```

**Key Points:**
- **⚠️ ALWAYS generate `infra/README.md`** tailored to the chosen platform and backend — see "Generating infra/README.md" below
- All CI/CD scripts go in `infra/scripts/` directory
- **⚠️ CRITICAL: All CI/CD tasks go in `infra/Taskfile.yaml` — NEVER add CI/CD tasks directly to the root `Taskfile.yml`**
- `.env` and `.env.gpg` stay in project root
- Scripts in `infra/scripts/` reference `../../.env` (two levels up)
- Root `Taskfile.yml` gets exactly ONE addition: an `includes` entry pointing to `./infra/Taskfile.yaml`
- CI/CD configs (`.gitlab-ci.yml`, `.github/workflows/`) are copied to standard locations

**Root Taskfile.yml — the only change needed:**
```yaml
# Add this includes block to the existing root Taskfile.yml:
includes:
  infra:
    taskfile: ./infra/Taskfile.yaml
    dir: infra
# Tasks are then run as: task infra:encrypt-secrets, task infra:setup-github-secrets, etc.
```

### Generating infra/README.md

After determining the user's CI/CD platform (GitHub/GitLab) and Pulumi backend, **always create `infra/README.md`** with content tailored to their choices. It should cover:

1. **Architecture overview** — which platform was chosen and why, and which Pulumi backend
2. **First-time setup** — the exact sequence of `task infra:*` commands needed to bootstrap
3. **Day-to-day tasks** — a table or list of the `task infra:*` commands relevant to their platform
4. **How deployments work** — short description of each trigger:
   - GitHub: `deploy.yml` fires on PR open/sync (review stack), `cd.yml` fires on push to main (CI stack), `destroy.yml` is manual
   - GitLab: `review_app` is manual on MR, `deploy_ci` fires on push to default branch, `destroy_review_app` is manual
5. **Secrets / credentials** — what variables/secrets are needed and where they live (GitHub Secrets, GitLab CI/CD variables, `.env.gpg`)
6. **Stack migration note** — if backend was migrated from a local stack, document what was done so future contributors understand the history

Adjust section titles, task names, and stack-naming strategy to match what was actually configured. The README should be accurate enough that a new contributor can set up CI/CD without referring to any other document.

## Workflow examples

See [`references/workflow-examples.md`](references/workflow-examples.md) for step-by-step examples covering GitLab CI/CD, GitHub Actions with GPG secrets, and continuous delivery setup.

## Using Task for workflow management

Application templates use [Task](https://taskfile.dev) to simplify local development and CI/CD workflows. Task provides a unified interface for Python and TypeScript/React components.

### Example Taskfile.yaml

See [`references/example-taskfile.yaml`](references/example-taskfile.yaml) for a complete example.

### Using Task in CI/CD

```bash
# Install Task
pip install go-task-bin

# Install dependencies
task install

# Run linters (with fixes)
task lint

# Run linters (check only)
task lint-check

# Run tests
task test
```

## GitLab CI/CD Configuration

The complete pipeline configuration lives in `scripts/gitlab-ci.yml`. Copy it to your repository root:

```bash
cp infra/scripts/gitlab-ci.yml .gitlab-ci.yml
```

Key pipeline jobs:
- `lint` / `test` — run on every same-project MR
- `review_app` — manual deploy per MR; stack name driven by the `PULUMI_STACK_REVIEW_NAME` CI/CD variable
- `deploy_ci` — automatic deploy on merge to default branch; stack name driven by `PULUMI_STACK_CI_NAME`
- `destroy_review_app` — manual cleanup of review stacks

`PULUMI_STACK_REVIEW_NAME` and `PULUMI_STACK_CI_NAME` must be set as plain CI/CD variables in GitLab (Settings → CI/CD → Variables). The pipeline file includes sensible defaults that project-level variables override.

## GitHub Actions Configuration

The complete workflow files live in `scripts/`:

- `scripts/github-deploy.yml` → copy to `.github/workflows/deploy.yml`
- `scripts/github-destroy.yml` → copy to `.github/workflows/destroy.yml`

```bash
mkdir -p .github/workflows
cp infra/scripts/github-deploy.yml .github/workflows/deploy.yml
cp infra/scripts/github-destroy.yml .github/workflows/destroy.yml
```

The deploy workflow triggers on pull requests and derives `PULUMI_STACK_NAME` from the `PULUMI_STACK_REVIEW_NAME` Actions variable and the PR number. Set `PULUMI_STACK_REVIEW_NAME` and `PULUMI_STACK_CI_NAME` as repository **variables** (Settings → Secrets and variables → Actions → **Variables** tab), not secrets.

## Pulumi State Management

### Pulumi Cloud Backend (Recommended)

The simplest approach for managing Pulumi state:

```bash
# Install Pulumi (macOS; other platforms: https://www.pulumi.com/docs/install/)
brew install pulumi/tap/pulumi

# Login to Pulumi Cloud
pulumi login

# Create/select stack
pulumi stack select --create dev

# Deploy
pulumi up
```

**CI/CD Setup**: Add `PULUMI_ACCESS_TOKEN` to your CI/CD secrets. Get token from [Pulumi Console](https://app.pulumi.com/account/tokens).

### DIY Backend Options

For organizations that cannot use Pulumi Cloud:

#### Azure Blob Storage

```bash
# Login to Azure backend
pulumi login azblob://container-name

# Set Azure credentials
export AZURE_STORAGE_ACCOUNT=myaccount
export AZURE_STORAGE_KEY=mykey
```

#### AWS S3

```bash
# Login to S3 backend
pulumi login s3://bucket-name

# AWS credentials from environment
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```

#### Google Cloud Storage

```bash
# Login to GCS backend
pulumi login gs://bucket-name

# GCP credentials from environment
export GOOGLE_CREDENTIALS=...
```

### Migrating Stacks to a Different Backend

When a developer has an existing local stack (a `Pulumi.<stackname>.yaml` file) that was
created against a different backend than the CI/CD destination, the stack state must be
exported and re-imported before switching.

`pulumi-setup.sh` handles this automatically: it checks `pulumi whoami --verbose` for the
**Backend URL** and compares it with the target URL. If they differ and local stack files
exist, it offers to migrate them.

**Manual migration steps** (if not using the script):

```bash
# 1. Confirm current backend and stacks
pulumi whoami --verbose        # note "Backend URL:"
pulumi stack ls -a             # list stacks on current backend

# 2. Export each stack that exists locally (Pulumi.<name>.yaml)
pulumi stack export --stack <stackname> --file <stackname>-backup.json

# 3. Login to the new backend (set any required credentials first)
#    Examples:
pulumi login                           # Pulumi Cloud
pulumi login azblob://my-container     # Azure Blob
pulumi login s3://my-bucket            # AWS S3

# 4. Create the stack in the new backend and import state
pulumi stack select --create <stackname>
pulumi stack import --file <stackname>-backup.json

# 5. Clean up the backup
rm <stackname>-backup.json
```

**Key signals that migration is needed:**
- `pulumi whoami --verbose` shows `Backend URL: file://` (local) but CI/CD uses cloud storage
- Backend URL domain/scheme differs between developer machine and CI target

### Managing Stacks Across Environments

```bash
# List all stacks
pulumi stack ls -a

# Output:
# NAME                                 LAST UPDATE   RESOURCE COUNT
# organization/project/prod            1 day ago     15
# organization/project/staging         2 days ago    12
# organization/project/dev             1 hour ago    10
# github-pr-repo-42                    3 hours ago   13

# Select and update a stack
pulumi stack select dev
pulumi up

# View stack outputs
pulumi stack output --json

# Delete a stack
pulumi stack rm review-app-123 --yes
```

## Secrets Management

All credentials (DataRobot API token, Pulumi access token, LLM keys, cloud storage keys) are stored in `.env` and committed to the repository encrypted as `.env.gpg`. The only secret that needs to be configured in the CI/CD system directly is `CICD_SECRET_PASSPHRASE` (the GPG passphrase). Non-sensitive stack name variables (`PULUMI_STACK_CI_NAME`, `PULUMI_STACK_REVIEW_NAME`) are set as plain variables, not secrets.

### DataRobot API token (service account)

`DATAROBOT_API_TOKEN` should come from a **DataRobot service account** — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team.

To set one up: ask your DataRobot admin to create a dedicated user (e.g. `ci-bot@your-org.com`). Under that account, go to **Developer Tools → API Key** and generate a token. Store it as the `DATAROBOT_API_TOKEN` secret in your CI/CD system.

> **Note:** This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. "Service account" here just means a non-personal DataRobot user.

### GitHub

Run `scripts/setup-github-secrets.sh` for interactive setup — it sets `CICD_SECRET_PASSPHRASE` as a repository secret and `PULUMI_STACK_CI_NAME` / `PULUMI_STACK_REVIEW_NAME` as repository variables.

To encrypt `.env` for CI:

```bash
task infra:encrypt-secrets
# or: ./infra/scripts/encrypt-secrets.sh
```

Add the resulting `.env.gpg` to git. For local decryption:

```bash
task infra:decrypt-secrets
# or: ./infra/scripts/decrypt-secrets.sh
```

### GitLab

Run `scripts/setup-gitlab-variables.sh` for interactive setup — it sets:
- `CICD_SECRET_PASSPHRASE` — masked, for decrypting `.env.gpg`
- `GITLAB_API_TOKEN` — masked, for posting MR comments
- `PULUMI_STACK_CI_NAME` / `PULUMI_STACK_REVIEW_NAME` — plain variables

Alternatively configure in the UI: Project Settings → CI/CD → Variables. Mark `CICD_SECRET_PASSPHRASE` and `GITLAB_API_TOKEN` as **Masked** and **Protected**.

## Best practices

### CI/CD Pipeline Design

1. **Fast feedback**: Run linting and testing in parallel
2. **Manual gates**: Make review apps manual to save resources
3. **Automatic cleanup**: Provide easy ways to destroy test environments
4. **Stack isolation**: Use unique stack names per PR/MR
5. **Idempotent operations**: Design deployments to be safely re-runnable

### Pulumi State

1. **Use centralized backends**: Enable collaboration and CI/CD
2. **Stack naming conventions**: Use consistent patterns (e.g., `github-pr-{repo}-{number}`)
3. **Clean up stacks**: Remove unused stacks to reduce clutter
4. **State locking**: Backends handle this automatically
5. **Backup state**: Cloud backends provide automatic backups

### Security

1. **Never commit secrets**: Use .gitignore for .env files
2. **Encrypt sensitive data**: Use GPG for GitHub, CI/CD variables for GitLab
3. **Rotate credentials**: Regularly update API tokens and keys
4. **Scope permissions**: Use least-privilege access for service accounts
5. **Audit access**: Monitor who has access to secrets

### Resource Management

1. **Tag resources**: Use consistent tagging for tracking
2. **Set TTLs**: Consider time-to-live for review environments
3. **Monitor costs**: Track resource usage per environment
4. **Auto-cleanup**: Implement automatic deletion of old review apps
5. **Resource limits**: Set quotas to prevent runaway costs

## Troubleshooting

### Common Issues

**Pulumi state conflicts:**
- Ensure only one deployment runs at a time per stack
- Use unique stack names for concurrent deployments
- Check backend connection and credentials

**Secret decryption failures:**
- Verify GPG passphrase is correct
- Check .env.gpg file is in repository
- Ensure GPG is installed in CI environment

**Deployment timeouts:**
- Increase timeout values in workflow
- Check DataRobot API connectivity
- Verify resource provisioning isn't blocked

**Stack not found:**
- List stacks: `pulumi stack ls -a`
- Verify backend connection
- Check stack name matches pattern

**Resource conflicts:**
- Use unique names per stack
- Check for orphaned resources
- Review Pulumi state for inconsistencies

## Example Repositories

Reference implementations:

- **GitLab**: [demo-data-agent](https://gitlab.com/datarobot-oss/demo-data-agent) - Complete GitLab CI/CD setup
- **GitHub**: [demo-talk-to-my-data-agent](https://github.com/datarobot-forks/demo-talk-to-my-data-agent) - Complete GitHub Actions setup

## Resources

- [Task Documentation](https://taskfile.dev)
- [Pulumi Documentation](https://www.pulumi.com/docs/)
- [Pulumi State and Backends](https://www.pulumi.com/docs/iac/concepts/state-and-backends/)
- [GitLab CI/CD](https://docs.gitlab.com/ci/)
- [GitHub Actions](https://docs.github.com/actions)
- [DataRobot Application Templates](https://docs.datarobot.com/en/docs/wb-apps/app-templates/index.html)
- [DataRobot Codespaces](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html)

Referenced files: 19 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-data-preparation
description: Tools and guidance for data upload, dataset management, data validation, and preparing data for DataRobot projects. Use when uploading datasets, managing data, or validating data for DataRobot.
---

# DataRobot Data Preparation Skill

This skill provides guidance for preparing and managing data in DataRobot, including uploading datasets, validating data quality, and managing dataset versions.

## Quick Start

**Most common use case**: Upload and validate a dataset

1. **Upload dataset**: `upload_dataset(file_path, dataset_name)` to upload data
2. **Validate data**: `validate_dataset(dataset_id)` to check data quality
3. **Check schema**: `get_dataset_schema(dataset_id)` to review structure

**Example**: "Upload sales_data.csv and check if it's ready for training"

## When to use this skill

Use this skill when you need to:
- Upload datasets to DataRobot
- Validate data before project creation
- Manage dataset versions and updates
- Check data quality and completeness
- Prepare data for training or predictions
- Handle data format conversions
- Connect to external data sources

## Key capabilities

### 1. Dataset Upload

- Upload CSV, Parquet, and other file formats
- Connect to databases and data warehouses
- Handle large datasets efficiently
- Manage dataset metadata and descriptions

### 2. Data Validation

- Validate data formats and schemas
- Check for missing values and data quality issues
- Verify column types and formats
- Identify potential data problems

### 3. Dataset Management

- List and search datasets
- Update dataset metadata
- Create dataset versions
- Delete or archive old datasets

### 4. Data Preparation

- Clean and preprocess data
- Handle missing values
- Format data for DataRobot requirements
- Prepare prediction datasets

## Workflow examples

### Example 1: Upload and validate dataset

**User request**: "Upload my sales_data.csv file and check if it's ready for training."

**Agent workflow**:
1. Upload the CSV file to DataRobot
2. Validate the dataset structure and format
3. Check for missing values and data quality issues
4. Verify column types are appropriate
5. Check for potential issues (leakage, formatting)
6. Report validation results and recommendations

### Example 2: Prepare prediction dataset

**User request**: "Prepare a prediction dataset based on the training data structure from project abc123."

**Agent workflow**:
1. Get the training dataset structure from the project
2. Identify required columns and data types
3. Create a template with the same structure
4. Validate the template matches requirements
5. Provide guidance on filling in prediction values

## Using DataRobot SDK

This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:

```bash
pip install datarobot
```

### Key SDK Operations

Use these DataRobot SDK methods for data management:

**Dataset Operations**:
- `dr.Dataset.create_from_file(file_path, name)` - Upload dataset
- `dr.Dataset.get(dataset_id)` - Get dataset details
- `dr.Dataset.list()` - List all datasets
- `dataset.row_count` - Get row count
- `dataset.column_count` - Get column count

**Dataset Information**:
- `dataset.name` - Dataset name
- `dataset.id` - Dataset ID
- `dataset.created_at` - Creation timestamp

See the [Common Patterns](#common-patterns) section below for complete examples.

## Helper Scripts

This skill includes executable helper scripts that Claude can run directly:

- `scripts/upload_dataset.py` - Upload a dataset file to DataRobot

**Usage example**:
```bash
# Upload dataset
python scripts/upload_dataset.py sales_data.csv "Sales Data Q4 2024"
```

Claude can run this script directly or use it as reference when writing code.

## Best practices

1. **Data quality**: Clean and validate data before upload
2. **File formats**: Use appropriate formats (CSV for small, Parquet for large)
3. **Naming conventions**: Use clear, descriptive dataset names
4. **Metadata**: Add descriptions and tags for better organization
5. **Versioning**: Create versions for important datasets
6. **Data validation**: Always validate data before using in projects

## Common patterns

### Pattern 1: Upload and validate
```python
import datarobot as dr

# Initialize client
dr.Client()

# Upload dataset
dataset = dr.Dataset.create_from_file(
    file_path="sales_data.csv", name="Sales Data Q4 2024"
)

print(f"Dataset ID: {dataset.id}")
print(f"Rows: {dataset.row_count}, Columns: {dataset.column_count}")

# Get dataset details
dataset_info = dr.Dataset.get(dataset.id)
print(f"Dataset name: {dataset_info.name}")
print(f"Created: {dataset_info.created_at}")
```

### Pattern 2: Dataset management
```python
import datarobot as dr

# List all datasets
datasets = dr.Dataset.list()
print(f"Found {len(datasets)} datasets")

# Search for specific dataset
for dataset in datasets:
    if "sales" in dataset.name.lower():
        print(f"Found: {dataset.name} (ID: {dataset.id})")

# Get specific dataset
dataset = dr.Dataset.get("abc123")
print(f"Dataset: {dataset.name}")
print(f"Size: {dataset.row_count} rows x {dataset.column_count} columns")
```

## Data format requirements

### CSV Files
- UTF-8 encoding recommended
- Headers in first row
- Consistent delimiters (comma, tab)
- Proper date/time formatting

### Parquet Files
- Columnar format, efficient for large datasets
- Preserves data types
- Better compression than CSV

### Database Connections
- Support for various databases
- Connection credentials required
- Query-based data access

## Data quality checks

Common checks to perform:

- **Missing values**: Identify columns with high missing value rates
- **Data types**: Verify columns have correct types
- **Value ranges**: Check for outliers and invalid values
- **Duplicates**: Identify duplicate records
- **Consistency**: Check for data consistency issues

## Error handling

Common errors and solutions:

- **Upload failures**: Check file format, size limits, encoding
- **Validation errors**: Fix data quality issues before proceeding
- **Schema mismatches**: Ensure data structure matches expectations
- **Access issues**: Verify permissions for dataset operations

## SDK Setup

### Install DataRobot SDK

```bash
pip install datarobot
```

### Initialize Client

```python
import datarobot as dr

dr.Client()
```

## Resources

- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)
- [DataRobot Data Management Documentation](https://docs.datarobot.com/en/docs/data/index.html)
- [Data Management: Uploading Datasets](https://docs.datarobot.com/en/docs/data/index.html)
- [Data Management: Data Quality and Validation](https://docs.datarobot.com/en/docs/data/index.html)

Referenced files: 1 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-discover
description: >-
  Use when the user wants to find DataRobot capabilities — skills, MCP servers,
  agents, or platform resources — for a task. Fetches the live DataRobot catalog
  directly so results are always current, regardless of third-party search index
  lag. Also checks the user's own DataRobot instance if DATAROBOT_ENDPOINT is set.
---

# Discover DataRobot Resources

Fetch DataRobot catalog and present matching resources for the user's
task. Covers resource types including Global MCP Server. Bring resources once DataRobot publishes them without skill update.

## Step 1 — Fetch the public catalog

```bash
curl -s https://www.datarobot.com/.well-known/ai-catalog.json
```

Always fetch this. It contains all publicly available DataRobot resources.

## Step 2 — Fetch the instance catalog (if DATAROBOT_ENDPOINT is set)

```bash
curl -s \
  -H "Authorization: Bearer $DATAROBOT_API_TOKEN" \
  "$DATAROBOT_ENDPOINT/.well-known/ai-catalog.json"
```

This surfaces resources specific to the user's DataRobot deployment — org-scoped
MCP servers, deployed agents, private tools. If it returns 404, the instance does
not publish a catalog yet; skip silently and continue with the public catalog only.

## Step 3 — Merge and present

Combine both result sets. Deduplicate by `identifier`. Then present as a numbered
list grouped by resource type:

| `type` value | Label |
|---|---|
| `application/ai-skill` | Skill |
| `application/mcp-server+json` | MCP Server |
| `application/ai-agent+json` | Agent |
| anything else | Resource |

For each entry show: **name**, **type**, **description**, and whether it came from
the public catalog or the user's instance. Relevance to the user's task comes
first — don't just dump the full list alphabetically.

## Step 4 — Install guidance (only when the user asks)

Never install or configure anything automatically. When the user picks a result:

**Skill** — install via the DataRobot skills plugin:
```bash
npx ai-agent-skills install <skill-name>
```
Or point directly to the `url` field from the catalog entry (the SKILL.md file).

**MCP Server** — show the MCP connector config using the `url` from the catalog:
```json
{
  "mcpServers": {
    "datarobot": {
      "url": "<url from catalog entry>"
    }
  }
}
```

**Agent** — show the A2A endpoint from the catalog entry and instruct the user
to connect it via their agent framework's standard mechanism.

**Other** — show the `url` and `description` from the catalog entry and let the
user decide how to proceed.

View saved version →

---
name: datarobot-external-agent-monitoring
description: Instrument any external or existing AI agent with OpenTelemetry to send traces, logs, and metrics to DataRobot for monitoring, observability, and governance. Use when the user says "add tracing/observability/monitoring to my agent", wants to instrument an existing agent project in their IDE, wants to send agent traces, logs, or metrics to DataRobot, or wants to record and replay agent trajectories (Tensile) from DataRobot traces.
---

# DataRobot External Agent Monitoring Skill

This skill helps you instrument any AI agent — regardless of framework or deployment environment — to send OpenTelemetry telemetry (traces, logs, metrics) to DataRobot. It also creates a shell deployment in DataRobot as the telemetry routing target.

## Quick Start

**Most common use case**: Instrument an existing agent project, regardless of whether it was built on DataRobot or elsewhere, with DataRobot monitoring

1. The user invokes the skill from inside their project — typically: "Add tracing to my agent"
2. The skill resolves the target project (current IDE workspace / working directory if no path is given), then detects the framework and any existing OTel setup
3. It resolves a **Use Case** as the telemetry target (asks for the user's Use Case ID, or offers to create one), generates instrumentation code, and wires it in
4. The agent sends traces, logs, and metrics to DataRobot, where they appear under the Use Case's Tracing tab

**Examples**:
- "Add tracing to my agent" (resolves to the current workspace)
- "Instrument my agent in ./my_agent for DataRobot monitoring"

## When to use this skill

Use this skill when an existing DataRobot user has built an agent elsewhere and wants to bring it in for monitoring. Specifically:
- Bring an externally-built (brownfield) agent into DataRobot for monitoring under a Use Case
- Add OpenTelemetry tracing to an agent project
- Send agent traces, logs, and metrics to DataRobot
- Instrument a Google ADK, LangChain, LangGraph, CrewAI, LlamaIndex, PydanticAI, or any Python agent

## Supported Frameworks

| Framework | Detection | OTel Strategy |
|-----------|-----------|---------------|
| Google ADK | `google-adk` in deps or `google.adk` in imports | Lazy trace injection via callback (ADK overwrites TracerProvider) |
| LangChain / LangGraph | `langchain` or `langgraph` in deps/imports | Auto-instrumentor + standard setup |
| CrewAI | `crewai` in deps/imports | Auto-instrumentor + standard setup |
| LlamaIndex | `llama-index` or `llama_index` in deps/imports | Auto-instrumentor + standard setup |
| PydanticAI | `pydantic-ai` or `pydantic_ai` in deps/imports | Standard setup + required `Agent.instrument_all()` (instrumentation is opt-in) |
| Generic Python | None of the above detected | Manual span instrumentation |

## Workflow

Follow these steps in order. Present the plan to the user and wait for approval before executing.

### Step 1: Detect & Analyze

1. Read the project's dependency file (`requirements.txt`, `pyproject.toml`, `setup.py`, `poetry.lock`, or `uv.lock`)
2. Scan Python source files for framework imports
3. Check for existing OTel setup (look for `opentelemetry` imports, existing TracerProvider/LoggerProvider/MeterProvider configuration)
4. Identify the framework using the detection table above
5. Read the corresponding framework reference file from the `frameworks/` directory next to this SKILL.md:
   - Google ADK → `frameworks/google-adk.md`
   - LangChain/LangGraph → `frameworks/langchain-langgraph.md`
   - CrewAI → `frameworks/crewai.md`
   - LlamaIndex → `frameworks/llamaindex.md`
   - PydanticAI → `frameworks/pydantic-ai.md`
   - Generic Python → `frameworks/generic-python.md`

### Step 2: Check Prerequisites

1. Ensure `DATAROBOT_API_TOKEN` is available **without having the user paste it into chat** (a pasted token would be logged in the transcript). Check the environment and the project `.env`. If the token is missing, create or update a project `.env` file with the DataRobot variables and have the user paste their Personal API key into **that file** directly (in their editor); read it from there. Ensure `.env` is gitignored. This skill targets existing DataRobot users: create a Personal API key at `<your DataRobot URL>/account/developer-tools` (Personal API keys tab; see the `datarobot-setup` skill). (No DataRobot account at all? https://www.datarobot.com/trial/.)
2. Check if `DATAROBOT_ENDPOINT` env var is set. If not, ask the user (default: `https://app.datarobot.com/api/v2`).
3. Derive `DATAROBOT_OTEL_ENDPOINT` automatically: if `DATAROBOT_ENDPOINT` ends with `/api/v2`, strip it and append `/otel` (e.g., `https://app.datarobot.com/api/v2` → `https://app.datarobot.com/otel`).
4. **Determine the telemetry target (Use Case)** — this is the primary entity, and works the same whether the agent was built on DataRobot or elsewhere. Only **collect** the choice here; do **not** run any script or create/validate anything yet — that happens once in Step 4, after the user approves the plan (running it here risks creating a Use Case the user never approved, and a duplicate when Step 4 runs).
   - Ask the user for their **Use Case ID**. DataRobot users typically already organize work in a Use Case.
   - If they don't have one (a brand-new or externally-built project), **offer to create one**. Ask only for a name; the description is auto-generated.
   - Record the choice (existing Use Case ID, or the name for a new one) to use in Step 4. The `create_use_case.py` helper will resolve it to an entity ID of the form `experiment_container-<use_case_id>` at execution time.
5. Check if the `datarobot` Python SDK is available. If not, install it: `pip install datarobot`.
6. Check if OTel packages are already in the project's dependencies.

**Security note:** Never ask the user to paste an API token into chat, and never echo tokens or `.env` contents into transcripts or logs. Collect the token only via the project `.env` file (the user edits the file directly) and read it from there; keep `.env` gitignored. If credentials are accidentally exposed, rotate them immediately.

### Step 3: Present Plan

Tell the user what you detected and present the changes you will make:
- Framework detected (or generic Python)
- Existing OTel setup found (if any)
- New dependencies to add
- New files to create (`dr_otel_config.py`, and optionally `dr_agent_metrics.py` for frameworks with custom metrics)
- Existing files to modify (agent entrypoint, dependency file)
- Intelligent trace analysis via Tensile (recommended, adds `tensile>=0.12.0`): see the framework reference file's **Intelligent Trace Analysis via Tensile** section. It records full prompt and completion text in DataRobot under the Use Case; include it unless the user declines.
- Telemetry target: enter an existing Use Case ID, or if user does not have one, generate a net new Use Case container and ID for user. Only list a shell deployment in the plan if the user explicitly asked for deployment-level monitoring; if they chose a Use Case, do not mention or ask about a deployment.

**Wait for user approval before executing.** If the user has already given explicit consent to implement or deploy, that counts as approval — no need to re-ask.

### Step 4: Execute

1. **Add dependencies** to the project's dependency file:
   - `opentelemetry-sdk`
   - `opentelemetry-api`
   - `opentelemetry-exporter-otlp-proto-http`
   - Framework-specific packages (see framework reference file)

2. **Generate `dr_otel_config.py`** using the generic pattern below, adapted per the framework reference file.

3. **Wire into agent entrypoint**: Add import and call to `configure_otel()` at startup. Follow the framework reference file for specific wiring instructions (auto-instrumentors, callbacks, etc.). Add the `tensile.otel.install(tracer=False)` lines from that file's **Intelligent Trace Analysis via Tensile** section directly after `configure_otel()` unless the user declined it.

4. **Generate `dr_agent_metrics.py`** if the framework reference file specifies custom metrics callbacks.

5. **Resolve the Use Case telemetry target** (primary entity). This is the **only** place the helper script runs — once, here, using the choice collected in Step 2 (never during prerequisites). Validate the user's existing Use Case, or create a net new one if they have none:
   ```bash
   set -a; source .env; set +a   # load DATAROBOT_API_TOKEN etc. from .env (not the command line)
   # Existing Use Case:
   python <skill_scripts_dir>/create_use_case.py --use-case-id <use_case_id>
   # No Use Case yet — create one (name only; description auto-generated):
   python <skill_scripts_dir>/create_use_case.py --name "<project_name> Monitoring"
   ```

   It returns `entity_id` as `experiment_container-<use_case_id>` — this is the OTel entity used at runtime.

6. **(Optional) Create shell deployment** — **only if the user explicitly asks** for deployment-level monitoring (drift, etc.). If the user chose a Use Case as the target, **do not ask about or prompt for a deployment ID** — the Use Case is the complete target on its own. Skip this step entirely unless the user raised it themselves.
   ```bash
   python <skill_scripts_dir>/create_shell_deployment.py \
     --name "<project_name> Monitoring" \
     --description "OTel telemetry sink for <framework> agent"
   ```

   The script automatically enables **prediction row storage** and **automatic association ID generation** on the deployment. If created, its `deployment-<id>` entity can be used as the target instead of the Use Case.

7. **Report results**: Write the resolved non-secret runtime vars into the project `.env` — never print the token. Confirm the Use Case ID (and deployment ID, if created):
   ```bash
   # appended to .env (DATAROBOT_API_TOKEN already present there; do not echo it):
   DATAROBOT_ENTITY_ID=experiment_container-<use_case_id>
   DATAROBOT_OTEL_ENDPOINT=<otel_endpoint>
   ```

### Step 5: Verify & Provide Runtime Instructions

1. Optionally run the verification script (loads credentials from `.env`; don't put the token on the command line):
   ```bash
   set -a; source .env; set +a
   python <skill_scripts_dir>/verify_otel_connection.py
   ```

2. Provide the user with the env vars to set in their runtime environment:
   - `DATAROBOT_API_TOKEN` — DataRobot API key
   - `DATAROBOT_ENTITY_ID` — `experiment_container-<use_case_id>` (Use Case target; or `deployment-<id>` if a shell deployment was created instead)
   - `DATAROBOT_OTEL_ENDPOINT` — `{DATAROBOT_ENDPOINT}/otel`

3. Explain how to view the telemetry. For a Use Case target, use the `dr` CLI's `xp`
   plugin (works in a local terminal or DataRobot Codespaces); this is the `view_command`
   returned by `create_use_case.py`:
   ```bash
   dr plugin install xp                                   # one-time
   dr xp --entity-id <use_case_id> --enable-logs --enable-metrics
   #     ^ the BARE use_case_id, NOT the experiment_container- prefixed form
   ```
   Then open the local panel at `http://127.0.0.1:8090`. You'll see:
   - **Tracing**: Span hierarchy (agent orchestration, LLM calls, tool calls)
   - **Logs**: Structured logs correlated with traces via traceId
   - **Metrics**: Custom metrics (request count, latency, LLM calls, tool calls)

## Generic OTel Configuration Pattern

Generate a `dr_otel_config.py` with a `configure_otel()` function that the project calls at startup, before any agent code runs. The **full annotated template lives in `reference/dr_otel_config.md` — read it before generating code.** Framework-specific files in `frameworks/` layer additional setup on top.

**Critical rules:**
1. Always pass `endpoint=` and `headers=` directly to exporters — NEVER use `OTEL_EXPORTER_OTLP_*` env vars (some frameworks detect these and create conflicting providers)
2. Be additive — add DataRobot as an additional span processor to any existing TracerProvider, don't replace it
3. Use `SimpleSpanProcessor` (not Batch) to avoid flush-before-shutdown issues
4. Use DELTA temporality for metrics (required by DataRobot)

**Provider initialization order:** some frameworks override the global TracerProvider at startup (notably Google ADK), which drops the DataRobot exporter. The additive pattern and per-framework workarounds (e.g. lazy injection via callbacks) are covered in `reference/dr_otel_config.md` and the framework reference files — always check them.

## DataRobot Tracing Table — Span Attribute Mapping

DataRobot's tracing UI (Data Exploration > Traces) maps specific span attributes to table columns. Using the correct attribute names is critical for data to appear in the dashboard.

### Column Mapping

| Tracing Table Column | Span Attribute | Aggregation Rule |
|---------------------|----------------|------------------|
| **Prompt** | `gen_ai.prompt` | First span with this attribute wins |
| **Completion** | `gen_ai.completion` | Last span with this attribute wins |
| **Tools** | `tool_name` | Lists all unique values across all spans in the trace |
| **Cost** | `datarobot.moderation.cost` | Summed across all spans in the trace |

**Important:** DataRobot looks for `tool_name` (underscore), NOT `tool.name` (dot). Some frameworks (e.g., LangGraph) do not set `tool_name` by default — you must add it manually as a span attribute inside each tool call.

### All Recognized Span Attributes

| Attribute | Description | Example |
|-----------|-------------|---------|
| `gen_ai.prompt` | User input / prompt text | `"Analyze policy XYZ"` |
| `gen_ai.completion` | Model output / response | `"Policy matched..."` |
| `gen_ai.request.model` | Model used for the call | `"gpt-4o"` |
| `gen_ai.usage.prompt_tokens` | Input token count | `150` |
| `gen_ai.usage.completion_tokens` | Output token count | `320` |
| `tool_name` | Name of tool/function called (required for Tools column) | `"search_database"` |
| `tool.parameters` | Tool call parameters (JSON string) | `'{"query": "..."}'` |
| `datarobot.moderation.cost` | Cost of this span (summed for trace total) | `0.0023` |

## Helper Scripts

### create_use_case.py

Resolves the **primary** telemetry target: validates an existing Use Case, or creates a net new one when the user has none.

```bash
# Existing Use Case:
python <scripts_dir>/create_use_case.py --use-case-id <use_case_id>
# Create new (name only; description auto-generated):
python <scripts_dir>/create_use_case.py --name "My Agent Monitoring"
```

Requires env vars: `DATAROBOT_API_TOKEN`, `DATAROBOT_ENDPOINT`

Returns JSON:
```json
{
  "use_case_id": "6123abc",
  "entity_id": "experiment_container-6123abc",
  "otel_endpoint": "https://app.datarobot.com/otel",
  "view_command": "dr xp --entity-id 6123abc --enable-logs --enable-metrics"
}
```

### create_shell_deployment.py

**Optional.** Creates a shell deployment in DataRobot as a telemetry routing target, for users who also want deployment-level monitoring.

```bash
python <scripts_dir>/create_shell_deployment.py \
  --name "My Agent Monitoring" \
  --description "OTel telemetry sink for my agent"
```

Requires env vars: `DATAROBOT_API_TOKEN`, `DATAROBOT_ENDPOINT`

Returns JSON:
```json
{
  "deployment_id": "abc123",
  "entity_id": "deployment-abc123",
  "otel_endpoint": "https://app.datarobot.com/otel"
}
```

### verify_otel_connection.py

Sends test telemetry to verify the OTel pipeline is working.

```bash
python <scripts_dir>/verify_otel_connection.py
```

Requires env vars: `DATAROBOT_API_TOKEN`, `DATAROBOT_ENTITY_ID`, `DATAROBOT_OTEL_ENDPOINT`

Returns JSON:
```json
{
  "status": "success",
  "traces": "sent",
  "logs": "sent",
  "metrics": "sent"
}
```

## Dependencies

Required for instrumentation (added to user's project):
```
opentelemetry-sdk
opentelemetry-api
opentelemetry-exporter-otlp-proto-http
```

Recommended, for intelligent trace analysis via Tensile (see the framework reference file):
```
tensile>=0.12.0
```

Required for shell deployment creation (available in the skill's script environment):
```
datarobot
```

## Best practices

1. **Call `configure_otel()` before any agent/framework initialization** — some frameworks capture the provider at import time
2. **Never set `OTEL_EXPORTER_OTLP_*` env vars** — pass endpoint and headers directly to exporters to avoid conflicts
3. **Use `SimpleSpanProcessor`** over `BatchSpanProcessor` — avoids flush issues on short-lived processes
4. **DELTA temporality for metrics** — DataRobot requires delta aggregation for counters and histograms
5. **Check framework reference files** for initialization order issues before generating code

## Error handling

Common errors and solutions:

| Error | Cause | Solution |
|-------|-------|----------|
| Traces not appearing in DataRobot | Framework overwrites TracerProvider | Use lazy injection pattern (see framework reference) |
| 401 Unauthorized from OTel endpoint | Invalid API token | Verify `DATAROBOT_API_TOKEN` is correct |
| 404 from OTel endpoint | Wrong endpoint URL | Ensure `DATAROBOT_OTEL_ENDPOINT` ends with `/otel` |
| Metrics not appearing | `OTEL_EXPORTER_OTLP_*` env vars set | Remove env vars, use direct exporter config |
| `DATAROBOT_ENTITY_ID` format error | Missing entity-type prefix | Must be `experiment_container-<use_case_id>` (Use Case) or `deployment-<id>`, not just `<id>` |

## Resources

- [DataRobot Model Monitoring Documentation](https://docs.datarobot.com/en/docs/mlops/monitor/index.html)
- [OpenTelemetry Python SDK](https://opentelemetry.io/docs/languages/python/)
- [OpenTelemetry OTLP Exporter](https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html)

Referenced files: 10 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-feature-engineering
description: Guidance for feature engineering, feature discovery, feature importance analysis, and understanding DataRobot's automated feature engineering capabilities. Use when working with feature engineering, feature discovery, or analyzing feature importance in DataRobot.
---

# DataRobot Feature Engineering Skill

This skill provides guidance for working with features in DataRobot, including understanding automated feature engineering, analyzing feature importance, and optimizing feature sets.

## Quick Start

**Most common use case**: Analyze feature importance for a model

1. **Get feature importance**: `get_feature_importance(model_id)` to get importance scores
2. **Analyze top features**: Sort by importance and identify key drivers
3. **Export feature list**: `export_feature_list(project_id)` to document features

**Example**: "Show me the top 10 most important features for model xyz123"

## When to use this skill

Use this skill when you need to:
- Understand what features DataRobot creates automatically
- Analyze feature importance for models
- Discover which features drive predictions
- Optimize feature sets for better performance
- Understand feature types and transformations
- Export feature lists and definitions

## Key capabilities

### 1. Feature Discovery

- Understand automated feature engineering in DataRobot
- Review derived features and transformations
- Identify feature types (numeric, categorical, text, date)
- Explore feature relationships and interactions

### 2. Feature Importance Analysis

- Get feature importance scores for models
- Understand which features drive predictions
- Compare feature importance across models
- Identify redundant or low-value features

### 3. Feature Optimization

- Select important features for model performance
- Remove low-importance features to reduce complexity
- Understand feature impact on predictions
- Optimize feature sets for deployment

### 4. Feature Documentation

- Export feature lists and definitions
- Document feature transformations
- Understand feature derivation logic
- Share feature information with stakeholders

## Workflow examples

### Example 1: Analyze feature importance

**User request**: "Show me the top 10 most important features for model xyz123 and explain what they mean."

**Agent workflow**:
1. Get feature importance scores for the model
2. Sort features by importance (descending)
3. Get top 10 features with their scores
4. Retrieve feature metadata and descriptions
5. Explain what each feature represents and why it's important
6. Provide insights on feature relationships

### Example 2: Optimize feature set for deployment

**User request**: "Create a simplified feature set for deployment abc123, keeping only features with importance > 0.1."

**Agent workflow**:
1. Get feature importance for the deployed model
2. Filter features by importance threshold (> 0.1)
3. Verify filtered features are sufficient for predictions
4. Document the optimized feature set
5. Update deployment configuration if needed

## Using DataRobot SDK

This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:

```bash
pip install datarobot
```

### Key SDK Operations

Use these DataRobot SDK methods for feature analysis:

**Feature Information**:
- `model.get_features()` - List all features in a model
- `model.get_feature_impact()` - Get feature importance scores
- `project.get_features()` - List features in a project

**Feature Analysis**:
- `feature.name` - Feature name
- `feature.feature_type` - Feature type (Numeric, Categorical, etc.)
- `feature.importance` - Feature importance score

See the [Common Patterns](#common-patterns) section below for complete examples.

## Best practices

1. **Review automated features**: DataRobot creates many derived features automatically - review them
2. **Focus on important features**: Pay attention to high-importance features for insights
3. **Understand feature types**: Different feature types require different handling
4. **Feature documentation**: Document important features for stakeholders
5. **Feature selection**: Consider removing very low-importance features for simplicity
6. **Feature stability**: Consider feature stability over time, not just importance

## Common patterns

### Pattern 1: Feature importance analysis
```python
import datarobot as dr

# Initialize client
dr.Client()

# Get model and feature importance
model = dr.Model.get("xyz123")
feature_impact = model.get_feature_impact()

# Sort by importance
sorted_features = sorted(
    feature_impact, key=lambda x: x.get("impactNormalized", 0), reverse=True
)

# Get top 10 features
top_features = sorted_features[:10]
for feature in top_features:
    print(f"{feature['featureName']}: {feature.get('impactNormalized', 0):.3f}")
```

### Pattern 2: Feature filtering
```python
import datarobot as dr

# Get model and feature importance
model = dr.Model.get("xyz123")
feature_impact = model.get_feature_impact()

# Filter by importance threshold (> 0.1)
important_features = [f for f in feature_impact if f.get("impactNormalized", 0) > 0.1]

print(f"Found {len(important_features)} features with importance > 0.1")
```

## Feature types in DataRobot

### Numeric Features
- Continuous numeric values
- Automatically scaled and normalized
- Can be used in mathematical operations

### Categorical Features
- Discrete categories or labels
- Automatically encoded (one-hot, target encoding)
- Important for many model types

### Text Features
- Text data (descriptions, comments)
- Automatically processed with NLP techniques
- Creates multiple text-derived features

### Date/Time Features
- Temporal data
- Automatically creates time-based features
- Important for time series models

## Understanding feature importance

Feature importance scores indicate:
- **High importance (> 0.1)**: Feature significantly impacts predictions
- **Medium importance (0.05-0.1)**: Feature contributes to predictions
- **Low importance (< 0.05)**: Feature has minimal impact

Note: Importance thresholds vary by model type and problem domain.

## Error handling

Common errors and solutions:

- **Feature not found**: Verify feature name and model compatibility
- **Importance unavailable**: Some model types don't provide importance scores
- **Feature access errors**: Check project and model permissions

## SDK Setup

### Install DataRobot SDK

```bash
pip install datarobot
```

### Initialize Client

```python
import datarobot as dr

dr.Client()
```

## Resources

- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)
- [DataRobot Feature Engineering Documentation](https://docs.datarobot.com/en/docs/modeling/index.html)
- [Feature Importance Guide](https://docs.datarobot.com/en/docs/modeling/analyze-models/index.html)
- [Feature Discovery Documentation](https://docs.datarobot.com/en/docs/data/transform-data/feature-discovery/index.html)

View saved version →

---
name: datarobot-model-deployment
description: Tools and guidance for deploying DataRobot models, managing deployments, configuring prediction environments, and deployment operations. Use when deploying models, creating or updating deployments, or configuring prediction environments.
---

# DataRobot Model Deployment Skill

This skill provides comprehensive guidance for deploying models, managing deployment configurations, and operating production deployments.

## Quick Start

**Most common use case**: Deploy a trained model to production

1. **Get best model**: Find the best model from a project (highest metric score)
2. **Create deployment**: `create_deployment(model_id, deployment_name)` to deploy model
3. **Get endpoint**: `get_deployment_endpoint(deployment_id)` to retrieve prediction URL

**Example**: "Deploy the best model from project abc123 as 'Sales Prediction v1'"

## When to use this skill

Use this skill when you need to:
- Deploy trained models to production
- Configure deployment settings and environments
- Manage multiple deployments
- Replace a deployment’s champion model with a new model version
- Configure prediction servers and environments
- Monitor deployment health and status
- Manage deployment access and permissions

## Key capabilities

### 1. Deployment Creation

- Deploy models from projects or registered models
- Choose prediction environment (DataRobot Serverless, external)
- Configure deployment settings (challenger models, A/B testing)
- Set up deployment metadata and descriptions

### 2. Deployment Configuration

- Configure prediction servers and environments
- Set up batch prediction settings
- Configure real-time prediction endpoints
- Manage deployment credentials and access

### 3. Deployment Management

- Replace deployment champion model (model swap)
- Enable/disable deployments
- Manage challenger models for A/B testing
- Configure replacement policies

### 4. Deployment Operations

- Get deployment information and status
- Retrieve deployment endpoints
- Manage deployment settings
- Handle deployment errors and issues

## Workflow examples

### Example 1: Deploy a model to production

**User request**: "Deploy the best model from project abc123 to production with the name 'Sales Prediction v1'."

**Agent workflow**:
1. Get the best model from the project (highest metric score)
2. Create a new deployment with the model
3. Configure deployment settings (name, description, environment)
4. Set up prediction environment (DataRobot Serverless recommended)
5. Retrieve deployment endpoint and credentials
6. Verify deployment is active and ready for predictions

### Example 2: Update deployment with new model

**User request**: "Replace the model in deployment xyz789 with the latest model from project abc123."

**Agent workflow**:
1. Get the latest model from the project
2. Retrieve current deployment information
3. Validate the replacement model is eligible (`deployment.validate_replacement_model(...)`)
4. Perform model replacement (`deployment.perform_model_replace(...)`)
5. Verify replacement completed successfully
6. Report deployment update status

## Using DataRobot SDK

This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:

```bash
pip install datarobot
```

### Key SDK Operations

Use these DataRobot SDK methods for deployment management:

**Deployments**:
- `dr.Deployment.create_from_learning_model(model_id, label)` - Create deployment
- `dr.Deployment.get(deployment_id)` - Get deployment details
- `dr.Deployment.list(project_id)` - List deployments
- `deployment.delete()` - Delete deployment

**Model Replacement (champion swap)**:
- `deployment.validate_replacement_model(new_model_id=...)` - Validate replacement eligibility
- `deployment.perform_model_replace(new_model_id=..., reason=...)` - Replace champion model (async)

**Challenger Models (limited via SDK)**:
- `deployment.list_challengers()` - List challenger models (if enabled/configured)
- `deployment.get_challenger_models_settings()` / `deployment.update_challenger_models_settings(...)` - Configure challenger models settings

**Deployment Info**:
- `deployment.get_features()` - Get required features

See the [Common Patterns](#common-patterns) section below for complete examples.

## Best practices

1. **Naming conventions**: Use clear, versioned names for deployments
2. **Environment selection**: Choose appropriate prediction environment for your use case
3. **Challenger models**: Use challenger models to test new models before full replacement
4. **Monitoring**: Set up monitoring and alerts for production deployments
5. **Documentation**: Document deployment purpose, model version, and configuration
6. **Access control**: Configure appropriate access permissions for deployments

## Common patterns

### Pattern 1: Standard deployment
```python
import datarobot as dr

# Initialize client
dr.Client()

# Get best model from project
models = dr.Model.list("abc123")
best_model = max(models, key=lambda m: m.metrics.get("AUC", 0))

# Create deployment
deployment = dr.Deployment.create_from_learning_model(
    model_id=best_model.id,
    label="Sales Prediction v1",
    description="Production deployment for sales forecasting",
)

print(f"Deployment created: {deployment.id}")
```

### Pattern 2: Deployment with challenger
```python
import datarobot as dr

# Create deployment with primary model
deployment = dr.Deployment.create_from_learning_model(
    model_id=primary_model.id, label="Sales Prediction v2"
)

# List challengers (if challenger models are configured/enabled)
challengers = deployment.list_challengers()
print(f"Challengers: {len(challengers)}")
```

## Deployment environments

### DataRobot Serverless
- Fully managed prediction environment
- Automatic scaling
- No infrastructure management
- Recommended for most use cases

### External deployment
- Deploy to your own infrastructure
- More control over resources
- Requires infrastructure management
- Use for specific compliance or performance requirements

## Deployment lifecycle

1. **Create**: Deploy model to production environment
2. **Monitor**: Track predictions, performance, and health
3. **Update**: Replace with new model versions as needed
4. **Retire**: Disable or archive old deployments

## Error handling

Common errors and solutions:

- **Model not found**: Verify model ID and project access
- **Deployment creation failures**: Check prediction environment availability
- **Endpoint access issues**: Verify credentials and permissions
- **Update failures**: Ensure new model is compatible with deployment settings

## SDK Setup

### Install DataRobot SDK

```bash
pip install datarobot
```

### Initialize Client

```python
import datarobot as dr

dr.Client()
```

## Resources

- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)
- [DataRobot Deployment Documentation](https://docs.datarobot.com/en/docs/mlops/deployment/deploy-methods/add-deploy-info.html)
- [Prediction Environments Guide](https://docs.datarobot.com/en/docs/mlops/deployment/prediction-env/pred-env-deploy.html)
- [Challenger Models Documentation](https://docs.datarobot.com/en/docs/mlops/monitor/challengers.html)

View saved version →

---
name: datarobot-model-explainability
description: >
  Tools and guidance for model explainability, prediction explanations, feature impact analysis,
  SHAP values, SHAP distributions, anomaly assessment, and model diagnostics. Use when analyzing
  model explanations, feature impact, SHAP values, SHAP distributions, anomaly assessment, or
  diagnosing model behavior.
---

# DataRobot Model Explainability Skill

This skill covers SHAP insights, XEMP prediction explanations, anomaly explanations, and model diagnostics.

> **SDK version**: Use `datarobot>=3.6.0` for the full API set in this skill (`ShapDistributions`
> was added in 3.6; `ShapMatrix`, `ShapImpact`, and `ShapPreview` are available in
> `datarobot>=3.4.0`). Use `from datarobot.insights import ShapMatrix, ...` with
> `entity_id=model_id` — not legacy `datarobot.models.ShapMatrix` (`project_id` / `dataset_id`).
> `ShapMatrix`, `ShapImpact`, `ShapPreview`, and `ShapDistributions` are the canonical SHAP API.
> The older `dr.PredictionExplanations` (XEMP-based) remains available but is the secondary path.

---

## Quick Start

| Goal | API to use | Prerequisites |
|------|-----------|---------------|
| SHAP values for all features, all rows | `ShapMatrix.create(entity_id=model_id)` | None - universal SHAP |
| Per-row top-feature explanations | `ShapPreview.create(entity_id=model_id)` | None |
| Aggregated feature importance via SHAP | `ShapImpact.create(entity_id=model_id)` | None |
| SHAP value distributions across features | `ShapDistributions.create(entity_id=model_id)` | None |
| SHAP for a filtered segment | `dr.DataSlice.create(...)` + `ShapMatrix.create(..., data_slice_id=...)` | Data slice definition |
| XEMP-based prediction explanations | `dr.PredictionExplanations.create(...)` | Feature Impact; PE initialization; dataset uploaded |
| Anomaly explanations (time series) | `AnomalyAssessmentRecord.compute(project_id, model_id, ...)` | Anomaly model |
| ROC / lift / confusion (insights) | `RocCurve.create(...)` / `LiftChart.create(...)` / `ConfusionMatrix.create(...)` | Validation data |
| ROC / lift / confusion (Model helpers) | `model.get_roc_curve()` / `model.get_lift_chart()` / `model.get_confusion_chart()` | Validation data |

**Universal SHAP is the preferred path** - no dataset pre-upload or Feature Impact step required.

## When to use this skill

Use this skill when you need to explain leaderboard model behavior, compute SHAP insights, use
XEMP prediction explanations, analyze anomaly explanations, or retrieve model diagnostics.

## Key capabilities

### 1. SHAP insights

- Compute `ShapMatrix`, `ShapPreview`, `ShapImpact`, and `ShapDistributions`
- Filter insights with `dr.DataSlice`

### 2. XEMP and anomaly explanations

- Use XEMP `dr.PredictionExplanations` when specifically required
- Retrieve time series anomaly assessment records and explanations

### 3. Diagnostics

- Retrieve ROC, lift, and confusion insights
- Use Model helpers for ROC, lift, confusion, and feature effects

## Setup

```python
import datarobot as dr
from datarobot.insights import ShapMatrix, ShapImpact, ShapPreview, ShapDistributions

dr.Client()
```

---

## Core API: `datarobot.insights`

```python
import pandas as pd
from datarobot.insights import ShapMatrix, ShapImpact, ShapPreview, ShapDistributions

model_id = "YOUR_MODEL_ID"

matrix = ShapMatrix.create(entity_id=model_id)
df = pd.DataFrame(matrix.matrix, columns=matrix.columns)

impact = ShapImpact.create(entity_id=model_id)
preview = ShapPreview.create(entity_id=model_id)
distributions = ShapDistributions.create(entity_id=model_id)
```

Use `ShapMatrix` for full row-by-feature SHAP values, `ShapPreview` for compact top-driver rows,
`ShapImpact` for aggregated SHAP importance, and `ShapDistributions` for per-feature SHAP
distributions. Use `source="externalTestSet"` plus `external_dataset_id` for external datasets.
See `references/shap_api_reference.md` for parameters, exports, and limitations.

---

## Secondary path: XEMP Prediction Explanations

Use `dr.PredictionExplanations` when XEMP explanations are specifically required (e.g., certain
regulatory contexts, or when SHAP is unavailable for the model type).

**Prerequisites** (all required before calling `.create()`):
1. Feature Impact must be computed: `model.request_feature_impact()` and wait
2. Prediction explanations initialized: `dr.PredictionExplanationsInitialization.create(...)`
3. Scoring dataset uploaded to the AI Catalog

```python
import datarobot as dr

model = dr.Model.get(project=project_id, model_id=model_id)
model.request_feature_impact().wait_for_completion()
dr.PredictionExplanationsInitialization.create(project_id=project_id, model_id=model_id)

dataset = dr.Dataset.upload("./data/scoring_data.csv")
pe_job = dr.PredictionExplanations.create(
    project_id=project_id,
    model_id=model_id,
    dataset_id=dataset.id,
    max_explanations=5,  # top N features per row, up to 50
    threshold_high=0.5,  # only explain rows with prediction >= threshold
    threshold_low=0.1,  # only explain rows with prediction <= threshold
)

pe_obj = pe_job.get_result_when_complete()
```

Use `pe_obj.get_rows()`, `pe_obj.get_all_as_dataframe()`, or `pe_obj.download_to_csv(...)` to
retrieve results. For parameters, multiclass modes, and exposure-adjusted predictions, see
`references/xemp_pe_reference.md`.

## Data slices for filtered insights

Use `dr.DataSlice` when the user asks to explain model behavior for a segment, such as a
region, product line, target class, or high-risk cohort. Pass the resulting `data_slice_id` into
the `datarobot.insights` SHAP APIs.

```python
import datarobot as dr
from datarobot.insights import ShapMatrix

data_slice = dr.DataSlice.create(
    name="high_income_customers",
    filters=[{"operand": "income", "operator": ">", "values": 100000}],
    project=project_id,
)

shap_matrix = ShapMatrix.create(
    entity_id=model_id,
    source="validation",
    data_slice_id=data_slice.id,
)
```

---

## Anomaly assessment (time series models)

For time series anomaly detection models, use `AnomalyAssessmentRecord`.

```python
from datarobot.models.anomaly_assessment import AnomalyAssessmentRecord

record = AnomalyAssessmentRecord.compute(
    project_id=project_id,
    model_id=model_id,
    backtest=0,  # backtest index (int) or "holdout"
    source="validation",  # "training" or "validation" only
    series_id=None,  # required for multiseries projects
)

records = AnomalyAssessmentRecord.list(project_id=project_id, model_id=model_id)
latest = record.get_latest_explanations()

regions = record.get_predictions_preview().find_anomalous_regions()
explanations = record.get_explanations_data_in_regions(regions=regions)

ranged = record.get_explanations(
    start_date="2024-01-01T00:00:00.000000Z",
    end_date="2024-06-01T00:00:00.000000Z",
)
```

---

## Model diagnostics

Use the same `entity_id=model_id` pattern as SHAP insights. `FeatureEffects` / partial dependence
is still retrieved through Model helpers (not in `datarobot.insights`).

### Insights diagnostics (preferred — matches SHAP API)

```python
from datarobot.insights import RocCurve, LiftChart, ConfusionMatrix

roc = RocCurve.create(entity_id=model_id)
lift = LiftChart.create(entity_id=model_id)
confusion = ConfusionMatrix.create(entity_id=model_id)
```

### Model helpers (alternative)

```python
model = dr.Model.get(project=project_id, model_id=model_id)

roc = model.get_roc_curve(source="validation")
lift = model.get_lift_chart(source="validation")
confusion = model.get_confusion_chart(source="validation")

# Feature Impact (non-SHAP) and Feature Effects (partial dependence for top features)
fi = model.get_feature_impact()
feature_effects = model.get_feature_effect(source="validation")
```

---

## Interpreting SHAP values

- **Positive value**: feature pushes prediction higher than baseline
- **Negative value**: feature pushes prediction lower than baseline
- **Magnitude**: size of influence; larger absolute value = stronger effect
- **Sum**: all SHAP values for a row sum to `prediction - base_value` in the link-function space
- **`base_value`**: the model's mean prediction (the "no information" baseline)

Example: if `base_value = 0.35` and a row's prediction is `0.72`, the row's SHAP values sum to
`0.37` when `link_function = "identity"`. A feature with SHAP `+0.20` contributed 20 units in
that same link-function space above baseline.

When `link_function = "logit"`, SHAP values are in log-odds space. Add feature contributions to
`base_value` in log-odds space, then use inverse-logit (`scipy.special.expit`) on the resulting
total to convert it to a probability. Do not apply `expit` to individual SHAP values as if they
were probability deltas.

---

## Decision guide

```
Task: explain predictions
    |
    - Need all features + all rows?     -> ShapMatrix.create(entity_id=model_id)
    - Need top-N features per row?      -> ShapPreview.create(entity_id=model_id)
    - Need aggregated importance?       -> ShapImpact.compute(entity_id=model_id)
    - Need feature SHAP distributions?  -> ShapDistributions.create(entity_id=model_id)
    - Need a segment/cohort only?       -> dr.DataSlice + data_slice_id
    - XEMP required (regulatory/type)?  -> dr.PredictionExplanations.create(...)
    - Time series / anomaly model?      -> AnomalyAssessmentRecord.compute(project_id, model_id, ...)
```

---

## Common errors

| Error | Cause | Fix |
|-------|-------|-----|
| `SHAP not available for this model` | Unsupported model type, or anomaly-detection model with >1000 features | Check model support; use XEMP PE if SHAP is unavailable |
| `Feature Impact not computed` | PredictionExplanations prerequisite missing | Run `model.request_feature_impact()` and wait |
| Missing `PredictionExplanationsInitialization` | PE not initialized | Call `PredictionExplanationsInitialization.create()` |
| `source='holdout'` fails | Holdout not unlocked | Unlock holdout in project settings first |
| Empty `previews` | No rows in partition | Check partition contains data |

---

## Reference files

- `references/shap_api_reference.md` - full parameter signatures for ShapMatrix, ShapImpact, ShapPreview, ShapDistributions
- `references/xemp_pe_reference.md` - PredictionExplanations and PredictionExplanationsInitialization parameter reference
- `scripts/compute_shap_matrix.py` - compute and export ShapMatrix to CSV or DataFrame

## Resources

- [datarobot.insights API reference](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/insights.html)
- [Prediction Explanations user guide](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/reference/modeling/insights/prediction_explanations.html)

Referenced files: 3 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-model-monitoring
description: Tools and guidance for monitoring model performance, tracking data drift, managing model health, and detecting prediction anomalies. Use when monitoring deployed models, tracking drift, or investigating prediction anomalies.
---

# DataRobot Model Monitoring Skill

This skill provides comprehensive guidance for monitoring deployed models, tracking performance metrics, detecting data drift, and managing model health.

## Quick Start

**Most common use case**: Check deployment health and data drift

1. **Check service stats**: `deployment.get_service_stats(...)` to review prediction volume/latency
2. **Check drift**: `deployment.get_feature_drift(...)` / `deployment.get_target_drift(...)`
3. **Compare over time**: Use `get_service_stats_over_time(...)` and drift periods to assess trends

**Example**: "Check the health of deployment abc123 and report any data drift issues"

## When to use this skill

Use this skill when you need to:
- Monitor model performance in production
- Track data drift and feature drift
- Detect prediction anomalies
- Monitor prediction accuracy over time
- Set up alerts for model degradation
- Analyze model health metrics
- Compare production performance to training performance

## Key capabilities

### 1. Performance Monitoring

- Track prediction accuracy and metrics over time
- Compare production metrics to training metrics
- Monitor prediction volume and latency
- Identify performance degradation trends

### 2. Data Drift Detection

- Detect changes in feature distributions
- Identify feature drift (statistical changes)
- Monitor target drift (if actuals available)
- Alert on significant drift events

### 3. Prediction Monitoring

- Monitor prediction distributions
- Detect prediction anomalies
- Track prediction confidence scores
- Identify unusual prediction patterns

### 4. Health Management

- Assess overall model health
- Generate monitoring reports
- Set up automated alerts
- Manage model retraining triggers

## Workflow examples

### Example 1: Check model health and drift

**User request**: "Check the health of deployment abc123 and report any data drift issues."

**Agent workflow**:
1. Get deployment monitoring status
2. Retrieve recent performance metrics
3. Check for data drift in key features
4. Compare current metrics to baseline (training)
5. Identify any significant drift or degradation
6. Report findings with recommendations

### Example 2: Set up drift monitoring alerts

**User request**: "Set up alerts for deployment xyz789 to notify when feature drift exceeds 0.2."

**Agent workflow**:
1. Get deployment configuration
2. Configure drift threshold (0.2)
3. Set up alert notifications
4. Specify which features to monitor
5. Test alert configuration
6. Confirm monitoring is active

## Using DataRobot SDK

This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:

```bash
pip install datarobot
```

### Key SDK Operations

Use these DataRobot SDK and MLOps API methods for monitoring:

**Deployment Monitoring**:
- `deployment.get_service_stats(...)` - Get service statistics (latency, volume, etc.)
- `deployment.get_feature_drift(...)` - Get feature drift metrics (returns `FeatureDrift` objects)
- `deployment.get_target_drift(...)` - Get target drift metrics (returns `TargetDrift`)
- `deployment.get_prediction_results(...)` - Retrieve recorded prediction results (if enabled)

**Model Performance**:
- `model.metrics` - Model performance metrics (dict of `{metric: {partition: score}}`)
- `model.get_roc_curve()` - Get ROC curve for comparison

**Note**: Some monitoring features may require DataRobot MLOps API. See the [Common Patterns](#common-patterns) section below for examples.

## Best practices

1. **Regular monitoring**: Check model health regularly, not just when issues arise
2. **Baseline comparison**: Always compare production metrics to training baseline
3. **Drift thresholds**: Set appropriate drift thresholds based on your domain
4. **Key features**: Focus monitoring on high-importance features
5. **Automated alerts**: Set up alerts for critical issues
6. **Historical analysis**: Track trends over time, not just point-in-time metrics

## Common patterns

### Pattern 1: Health check
```python
import datarobot as dr

# Initialize client
dr.Client()

# Get deployment
deployment = dr.Deployment.get("abc123")

# Get service stats (requires MLOps monitoring to be enabled)
# ServiceStats exposes values via the .metrics dict, not as attributes.
stats = deployment.get_service_stats()
print(f"Prediction count: {stats.metrics['totalPredictions']}")
print(f"Mean response time (ms): {stats.metrics['responseTime']}")

# Get recorded prediction results (if available / enabled)
try:
    recent = deployment.get_prediction_results(limit=10)
    print(f"Recent prediction results: {len(recent)}")
except Exception as e:
    print(f"Prediction results not available: {e}")
```

### Pattern 2: Drift detection
```python
import datarobot as dr

# Get deployment
deployment = dr.Deployment.get("abc123")

# Get feature drift (requires MLOps monitoring)
try:
    drifts = deployment.get_feature_drift()
    high = [d for d in drifts if (d.drift_score or 0) > 0.2]
    print(f"Features with drift_score > 0.2: {len(high)}")
    for d in high[:10]:
        print(f"{d.name}: {d.drift_score}")
except Exception as e:
    print(f"Feature drift requires MLOps monitoring: {e}")
```

## Monitoring metrics

### Performance Metrics
- **Accuracy**: Prediction accuracy (classification)
- **RMSE/MAE**: Prediction error (regression)
- **AUC**: Model discrimination (classification)
- **Prediction volume**: Number of predictions made

### Drift Metrics
- **Feature drift**: Statistical changes in feature distributions
- **Target drift**: Changes in target distribution (if available)
- **Prediction drift**: Changes in prediction distributions
- **Drift score**: Overall drift severity (0-1 scale)

## Alert thresholds

Recommended thresholds:

- **High drift**: > 0.3 (significant changes, investigate immediately)
- **Medium drift**: 0.15-0.3 (moderate changes, monitor closely)
- **Low drift**: < 0.15 (minor changes, normal variation)

Adjust thresholds based on your domain and use case sensitivity.

## Model health status

- **Healthy**: Performance within expected range, minimal drift
- **Degrading**: Performance declining, some drift detected
- **Unhealthy**: Significant performance issues or high drift
- **Unknown**: Insufficient data for assessment

## Error handling

Common errors and solutions:

- **Insufficient data**: Need minimum prediction volume for monitoring
- **Baseline unavailable**: Ensure training baseline is available
- **Access issues**: Verify deployment permissions and access

## SDK Setup

### Install DataRobot SDK

```bash
pip install datarobot
```

### Initialize Client

```python
import datarobot as dr

dr.Client()
```

**Note**: Some monitoring features require DataRobot MLOps API access. Check your DataRobot plan for MLOps availability.

## Resources

- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)
- [DataRobot Model Monitoring Documentation](https://docs.datarobot.com/en/docs/mlops/monitor/index.html)

View saved version →

---
name: datarobot-model-training
description: Comprehensive guidance for training models in DataRobot, including project creation, AutoML configuration, feature engineering, and model selection. Use when training models, creating AutoML projects, or selecting models in DataRobot.
---

# DataRobot Model Training Skill

This skill provides guidance for the complete model training workflow in DataRobot, from project creation through model selection and validation.

## Quick Start

**Most common use case**: Create a project and train models

1. **Create or reuse a Use Case**: ask the user if they have an existing Use Case ID to reuse (`dr.UseCase.get(use_case_id)`); otherwise create a new one (`dr.UseCase.create(name)`). Every project needs one linked in Workbench
2. **Upload dataset**: `upload_dataset(file_path, dataset_name, use_case_id)` to upload training data, associated with the Use Case
3. **Create project**: `create_project(dataset_id, project_name, target_column, use_case_id)` to create new project, associated with the same Use Case
4. **Start training**: `start_automl(project_id, mode)` to begin AutoML training

**Example**: "Create a new project under a 'Sales Forecasting' Use Case with sales_data.csv, set 'revenue' as target, and start Quick AutoML training"

## When to use this skill

Use this skill when you need to:
- Create new DataRobot projects
- Upload training datasets
- Configure AutoML experiments
- Monitor training progress
- Select and compare models
- Understand feature engineering results
- Export trained models

## Key capabilities

### 1. Project Management

- Create new projects with appropriate settings
- Upload datasets (CSV, Parquet, database connections)
- Configure project settings (target, partitioning, time series)
- Manage multiple projects and experiments

### 2. AutoML Configuration

- Set training modes (Quick, Manual, Comprehensive)
- Configure feature engineering options
- Set time limits and resource constraints
- Choose algorithms and model types

### 3. Training Execution

- Start AutoML training runs
- Monitor training progress
- Handle training errors and warnings
- Pause/resume training if needed

### 4. Model Analysis

- Compare model performance metrics
- Review feature importance
- Analyze model insights and explanations
- Select best models for deployment

## Workflow examples

### Example 1: Create and train a new project

**User request**: "Create a new project using my sales_data.csv file, predict 'revenue' as the target, and start AutoML training."

**Agent workflow**:
1. Upload the dataset to DataRobot
2. Create a new project with the dataset
3. Set 'revenue' as the target variable
4. Configure project settings (detect partitioning, handle time series if needed)
5. Start AutoML training with appropriate mode
6. Monitor training progress
7. Report when training completes with top model metrics

### Example 2: Configure advanced training options

**User request**: "Train a model with time series settings: datetime column 'date', series ID 'store_id', forecast window 1-7 days."

**Agent workflow**:
1. Create project with time series configuration
2. Set datetime column and series ID columns
3. Configure forecast window (1-7 days)
4. Set appropriate time series validation
5. Start training with time series-aware algorithms
6. Monitor progress and report results

## Using DataRobot SDK

This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:

```bash
pip install datarobot
```

### Key SDK Operations

Use these DataRobot SDK methods for model training:

**Use Cases** (organize related datasets/projects/deployments under one entity):
- `dr.UseCase.create(name, description=None)` - Create a new Use Case
- `dr.UseCase.get(use_case_id)` - Retrieve an existing Use Case (reuse instead of creating a new one)
- `use_case.add(entity=project_or_dataset)` - Attach an already-created project or dataset to a Use Case

**Projects**:
- `dr.Project.create_from_dataset(dataset_id, project_name, use_case=use_case)` - Create project, linked to a Use Case
- `dr.Project.get(project_id)` - Get project details
- `dr.Project.list()` - List all projects
- `project.analyze_and_model(target, mode=dr.AUTOPILOT_MODE.QUICK)` - Set the target and start AutoPilot

**Training**:
- `project.wait_for_autopilot()` - Block until AutoPilot finishes
- `project.get_status()` - Check training status
- `dr.Model.list(project_id)` - List trained models
- `dr.Model.get(model_id)` - Get model details
- `dr.ModelRecommendation.get(project.id).get_model()` - Get DataRobot's recommended model

**Model Analysis**:
- `model.metrics` - Performance metrics (dict of `{metric: {partition: score}}`)
- `model.get_feature_impact()` - Get feature importance

See the [Common Patterns](#common-patterns) section below for complete examples.

## Helper Scripts

This skill includes executable helper scripts that Claude can run directly:

- `scripts/create_project.py` - Create a new project from a dataset, optionally linked to a Use Case
- `scripts/start_training.py` - Set the target and start AutoML training
- `scripts/list_models.py` - List trained models with metrics

The `datarobot-data-preparation` skill's `scripts/upload_dataset.py` accepts the same optional `use_case_id` argument for linking an uploaded dataset to a Use Case.

**Usage example**:
```bash
# Create (or reuse) a Use Case first
python -c "import datarobot as dr; print(dr.UseCase.create(name='Sales Prediction').id)"

# Upload dataset, linked to the Use Case
python ../datarobot-data-preparation/scripts/upload_dataset.py sales_data.csv "Sales Data" use_case_456

# Create project, set target, and start training (one step), linked to the Use Case
python scripts/create_project.py dataset_123 "Sales Prediction" revenue use_case_456

# (Alternatively, if the project was created without a target:)
# python scripts/start_training.py project_456 revenue Quick

# List models once AutoPilot finishes
python scripts/list_models.py project_456 AUC
```

Claude can run these scripts directly or use them as reference when writing code.

## Best practices

1. **Data preparation**: Ensure data is clean and properly formatted before upload
2. **Use Cases**: Every project needs a linked Use Case in Workbench. Resolve one up front — reuse an existing `use_case_id` via `dr.UseCase.get(use_case_id)`, or create a new one via `dr.UseCase.create(name)` — and pass it to both `Dataset.create_from_file` (`use_cases=[...]`) and `Project.create_from_dataset` (`use_case=...`) so the project lands under the intended Use Case rather than a default one
3. **Target selection**: Choose appropriate target variable (avoid leakage)
4. **Partitioning**: Use proper partitioning for time-aware or grouped data
5. **Feature engineering**: Let AutoML handle feature engineering, but review results
6. **Model selection**: Compare multiple models, not just the top performer
7. **Validation**: Review validation strategy and ensure it matches your use case

## Common patterns

### Pattern 1: Standard classification/regression
```python
import datarobot as dr
import os

# Initialize client
dr.Client()

# Reuse an existing Use Case if the user gave us one, otherwise create a new one
existing_use_case_id = os.getenv("DATAROBOT_USE_CASE_ID")  # or ask the user for it
use_case = (
    dr.UseCase.get(existing_use_case_id)
    if existing_use_case_id
    else dr.UseCase.create(name="Sales Prediction")
)

# Upload dataset
dataset = dr.Dataset.create_from_file(
    file_path="training_data.csv", name="Sales Data", use_cases=[use_case]
)

# Create project
project = dr.Project.create_from_dataset(
    dataset_id=dataset.id, project_name="Sales Prediction", use_case=use_case
)

# Set the target and start AutoPilot (Quick mode).
# analyze_and_model() replaces the deprecated set_target() and starts AutoPilot,
# so no separate start() call is needed.
project.analyze_and_model(
    target="revenue", mode=dr.AUTOPILOT_MODE.QUICK, worker_count=-1
)

# Wait for AutoPilot to finish
project.wait_for_autopilot()

# Get DataRobot's recommended model (the one flagged "Recommended for Deployment")
best_model = dr.ModelRecommendation.get(project.id).get_model()

# model.metrics maps each metric to per-partition scores; read the validation score.
metric = project.metric  # the project's optimization metric, e.g. "LogLoss" or "AUC"
print(
    f"Recommended model: {best_model.id} ({best_model.model_type}), "
    f"{metric} (validation): {best_model.metrics[metric]['validation']}"
)
```

### Pattern 2: Time series forecasting
```python
import datarobot as dr
import os

# Reuse an existing Use Case if the user gave us one, otherwise create a new one
existing_use_case_id = os.getenv("DATAROBOT_USE_CASE_ID")  # or ask the user for it
use_case = (
    dr.UseCase.get(existing_use_case_id)
    if existing_use_case_id
    else dr.UseCase.create(name="Sales Forecast")
)

# Upload dataset
dataset = dr.Dataset.create_from_file(
    "sales_data.csv", "Sales Forecast Data", use_cases=[use_case]
)

# Create project
project = dr.Project.create_from_dataset(
    dataset_id=dataset.id, project_name="Sales Forecast", use_case=use_case
)

# Configure time series settings
project.set_target(
    target="sales",
    mode=dr.AUTOPILOT_MODE.COMPREHENSIVE,
    partitioning_method=dr.PARTITIONING_METHOD.DATETIME,
    datetime_partition_column="date",
    multiseries_id_columns=["store_id"],
    forecast_window_start=1,
    forecast_window_end=7,
)

# Start training
project.start(autopilot_on=True, max_wait=7200)

# Wait for completion and get results
project.wait_for_completion()
models = dr.Model.list(project.id)
```

## Model selection criteria

When selecting models, consider:

- **Performance metrics**: Accuracy, AUC, RMSE, MAPE (depending on problem type)
- **Prediction speed**: Important for real-time deployments
- **Interpretability**: Some models are more explainable
- **Feature requirements**: Some models need specific feature types
- **Deployment constraints**: Consider model size and resource requirements

## Error handling

Common errors and solutions:

- **Dataset upload failures**: Check file format, size limits, encoding
- **"Dataset does not contain enough rows"**: DataRobot requires a minimum of 20 rows to create a project from a dataset — sample/demo data must meet this
- **Target errors**: Ensure target column exists and has appropriate values
- **Training failures**: Check data quality, feature types, missing values
- **Timeout errors**: Adjust time limits or use Quick mode for initial exploration

## SDK Setup

### Install DataRobot SDK

```bash
pip install datarobot
```

### Initialize Client

```python
import datarobot as dr

dr.Client()
```

## Resources

- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)
- [DataRobot AutoML Documentation](https://docs.datarobot.com/en/docs/modeling/index.html)
- [General Modeling Documentation – Time Series](https://docs.datarobot.com/en/docs/modeling/index.html)
- [General Modeling Documentation – Feature Engineering](https://docs.datarobot.com/en/docs/modeling/index.html)

Referenced files: 3 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-predictions
description: Tools and guidance for making predictions with DataRobot deployments, including real-time predictions, batch scoring, prediction dataset generation, and prediction explanations (SHAP/XEMP). Use when making predictions, running batch scoring, generating prediction datasets, or explaining individual predictions from a deployment.
---

# DataRobot Predictions Skill

This skill provides comprehensive guidance for working with DataRobot predictions, including real-time predictions, batch scoring, and generating prediction datasets.

## Quick Start

**Most common use case**: Generate predictions for a deployment

1. **Get deployment features**: `get_deployment_features(deployment_id)` to understand required columns
2. **Generate template**: `generate_prediction_data_template(deployment_id, n_rows)` to create CSV structure
3. **Make predictions**: Use `deployment.predict_batch(...)` (works for both single-row “real-time” and batch scoring)

**Example**: "Generate a prediction dataset template for deployment abc123 with 10 rows"

**To also explain predictions**: pass `--max-explanations N` to `make_prediction.py` (or the
`max_explanations=N` kwarg in code). See [Prediction Explanations](#prediction-explanations) below.

## When to use this skill

Use this skill when you need to:
- Make predictions from deployed DataRobot models
- Explain individual predictions from a deployment (SHAP or XEMP, per-row)
- Generate prediction dataset templates
- Validate prediction data before scoring
- Understand deployment feature requirements
- Perform batch predictions on large datasets
- Get sample training data to understand expected formats

> For post-hoc explanations against a **training project / leaderboard model** (not a deployment),
> use the `datarobot-model-explainability` skill instead. This skill covers deployment-time
> explanations returned alongside scoring.

## Key capabilities

### 1. Understanding Deployment Requirements

Before making predictions, you need to understand what features a deployment requires:

- **Feature names and types**: Know which columns are needed (numeric, categorical, text, date)
- **Feature importance**: Understand which features matter most
- **Target information**: Know what you're predicting
- **Time series configuration**: If applicable, understand datetime columns and series IDs

### 2. Generating Prediction Datasets

Create properly formatted prediction datasets:

- Generate CSV templates with all required columns
- Include sample values appropriate for each feature type
- Add metadata comments explaining the model structure
- Ensure correct column ordering

### 3. Validating Prediction Data

Validate datasets before making predictions:

- Check for missing required features
- Verify data types match expected types
- Identify missing low-importance features (warnings)
- Note extra columns that will be ignored

### 4. Making Predictions

Execute predictions using various methods:

- **Real-time predictions**: Fast, synchronous predictions for individual records
- **Batch predictions**: Process large datasets efficiently
- **Time series predictions**: Handle forecasting scenarios with proper datetime handling

## Workflow examples

### Example 1: Generate prediction dataset for a new scenario

**User request**: "I want to predict sales for next week for store_A with temperatures of 75°F each day and no promotions."

**Agent workflow**:
1. Get deployment features to understand required columns
2. Generate a prediction data template with 7 rows (one week)
3. Fill in the template with user's specific values:
   - Set temperature = 75 for all rows
   - Set promotion = 0 for all rows
   - Set store_id = "store_A" for all rows
   - Set dates for next 7 days
4. Validate the data to ensure it's correct
5. Make predictions using the validated dataset

### Example 2: Batch scoring a CSV file

**User request**: "Score all records in my prediction_data.csv file using deployment abc123."

**Agent workflow**:
1. Validate the CSV file structure matches deployment requirements
2. Upload the file or provide file path
3. Submit batch prediction job
4. Monitor job status
5. Retrieve and return prediction results

## Using DataRobot SDK

This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:

```bash
pip install datarobot datarobot-predict
```

### Key SDK Operations

Use these DataRobot SDK methods to work with predictions:

**Deployment Information**:
- `dr.Deployment.get(deployment_id)` - Get deployment details
- `deployment.get_features()` - Get required features (name/type/importance)

**Predictions**:
- `deployment.predict_batch(source)` - Convenience batch prediction API (CSV path, file object, or pandas DataFrame)
- `dr.BatchPredictionJob.score(deployment=deployment, ...)` - Advanced batch prediction control
- `job.get_result_when_complete()` - Wait for batch scoring to finish and download results

**Data Management**:
- `dr.Dataset.create_from_file(file_path)` - Upload dataset
- `dr.Dataset.get(dataset_id)` - Get dataset info

See the [Common Patterns](#common-patterns) section below for complete examples.

## Prediction Explanations

Deployments can return per-row explanations (top feature contributions) alongside predictions.
Two algorithms are available depending on how the deployment was configured:

- **SHAP** (`shap`): SHapley Additive exPlanations. Available on tree-based models when SHAP was
  enabled at deployment time. Returns signed contributions in the model's score space.
- **XEMP** (`xemp`): DataRobot's eXplainable AI for the eXact Model Prediction. Default when SHAP
  is not enabled. Returns top-N strongest features with a qualitative strength (`+++`, `--`, etc.).

If you omit `explanation_algorithm`, the deployment's default is used.

### How to request explanations

Pass `max_explanations=N` (and any optional filters) when calling `datarobot_predict.deployment.predict`:

```python
import datarobot as dr
import pandas as pd
from datarobot_predict.deployment import predict as dr_predict

dr.Client()
deployment = dr.Deployment.get("abc123")

result = dr_predict(
    deployment=deployment,
    data_frame=pd.DataFrame([{"feature1": 10, "feature2": 20}]),
    max_explanations=3,  # top 3 contributors per row
    explanation_algorithm="shap",  # or "xemp"; omit for deployment default
    # threshold_high=0.8,                # optional: only explain rows scoring > 0.8
    # threshold_low=0.2,                 # optional: only explain rows scoring < 0.2
    # passthrough_columns="all",         # optional: echo input columns through to output
)
print(result.dataframe.to_dict(orient="records"))
```

The result DataFrame includes columns like `EXPLANATION_1_FEATURE_NAME`,
`EXPLANATION_1_ACTUAL_VALUE`, `EXPLANATION_1_STRENGTH`, `EXPLANATION_1_QUALITATIVE_STRENGTH` for
each of the top-N contributors.

### Parameter reference

| Parameter | Purpose |
|-----------|---------|
| `max_explanations` | Top-N contributors per row. `0` (default) disables explanations. |
| `max_ngram_explanations` | Text models only: cap text-segment explanations per row. |
| `threshold_high` | Only explain rows with prediction probability **above** this (0–1). |
| `threshold_low` | Only explain rows with prediction probability **below** this (0–1). |
| `explanation_algorithm` | `"shap"` or `"xemp"`; omit to use deployment default. |
| `passthrough_columns` | `"all"` or set of input column names to echo through to output. |

### CLI shortcut

```bash
python scripts/make_prediction.py abc123 '{"feature1": 10, "feature2": 20}' \
    --max-explanations 3 --explanation-algorithm shap
```

### When to use which threshold

- `threshold_high` is useful when only positive (high-risk / fraud / churn-likely) predictions need
  explaining — saves compute on a large batch.
- `threshold_low` is the mirror image for low-probability rows.
- Setting both restricts explanations to rows outside the `[low, high]` band.

### Common errors

- **"Prediction explanations not enabled"**: the deployment was created without explanations support.
  Re-deploy the model with explanations enabled, or use a deployment that has them.
- **`max_explanations` ignored / no explanation columns in output**: confirm you're calling
  `datarobot_predict.deployment.predict(...)` and that the deployment has explanations enabled.
  The `deployment.predict_batch()` convenience wrapper on the SDK is intended for plain scoring;
  use `datarobot_predict.deployment.predict` when you need explanation kwargs.

## Helper Scripts

This skill includes executable helper scripts that Claude can run directly:

- `scripts/get_deployment_features.py` - Get deployment feature requirements
- `scripts/generate_prediction_data_template.py` - Generate CSV template
- `scripts/validate_prediction_data.py` - Validate prediction data
- `scripts/make_prediction.py` - Make real-time predictions

**Usage example**:
```bash
# Get deployment features
python scripts/get_deployment_features.py abc123

# Generate template
python scripts/generate_prediction_data_template.py abc123 10 template.csv

# Validate data
python scripts/validate_prediction_data.py abc123 prediction_data.csv

# Make prediction
python scripts/make_prediction.py abc123 '{"feature1": 10, "feature2": 20}'

# Make prediction with top-3 SHAP explanations
python scripts/make_prediction.py abc123 '{"feature1": 10, "feature2": 20}' \
    --max-explanations 3 --explanation-algorithm shap
```

Claude can run these scripts directly or use them as reference when writing code.

## Best practices

1. **Always validate first**: Validate prediction data before submitting predictions to catch errors early
2. **Use templates**: Generate templates to ensure correct structure and avoid missing columns
3. **Check feature types**: Ensure numeric features are numbers, categorical features match training values
4. **Handle time series**: For time series models, ensure datetime columns and series IDs are properly formatted
5. **Monitor batch jobs**: For large batch predictions, check job status and handle errors appropriately

## Common patterns

### Pattern 1: Get deployment features and make single prediction (optionally with explanations)
```python
import datarobot as dr
import pandas as pd
from datarobot_predict.deployment import predict as dr_predict

# Initialize client
dr.Client()

deployment = dr.Deployment.get("abc123")

prediction_data = {
    "feature1": value1,
    "feature2": value2,
    # ... all required features (excluding target)
}

# Score one row. Add max_explanations=N to get top-N explanations per row.
result = dr_predict(
    deployment=deployment,
    data_frame=pd.DataFrame([prediction_data]),
    max_explanations=3,  # optional; 0/omit to disable explanations
    explanation_algorithm="shap",  # optional; omit to use deployment default
)
print(result.dataframe.to_dict(orient="records"))
```

### Pattern 2: Generate template and batch predictions
```python
import datarobot as dr
import pandas as pd

# Initialize client
dr.Client()

# Get deployment features
deployment = dr.Deployment.get("abc123")
model = dr.Model.get(deployment.model["id"])
features = model.get_features()

# Create template DataFrame
prediction_features = [f for f in features if f.name != model.target_name]
template_df = pd.DataFrame(columns=[f.name for f in prediction_features])

# Add sample rows
for i in range(100):
    row = {}
    for feature in prediction_features:
        if feature.feature_type == "Numeric":
            row[feature.name] = 0.0
        elif feature.feature_type == "Categorical":
            row[feature.name] = "sample_value"
        else:
            row[feature.name] = ""
    template_df = pd.concat([template_df, pd.DataFrame([row])], ignore_index=True)

# Save template
template_df.to_csv("prediction_template.csv", index=False)

# Fill template with actual data (modify CSV as needed)
# ...

# Submit batch prediction
job = dr.BatchPredictionJob.score(
    deployment_id=deployment.id,
    intake_settings={"type": "localFile", "file": "prediction_template.csv"},
    output_settings={"type": "localFile", "path": "predictions_output.csv"},
)

# Monitor job
job_status = dr.BatchPredictionJob.get(job.id)
print(f"Job status: {job_status.status}")

# Download results when complete
if job_status.status == "completed":
    results = dr.BatchPredictionJob.download(job.id)
```

## Error handling

Common errors and solutions:

- **Missing required features**: Use `get_deployment_features` to get complete list
- **Wrong data types**: Check feature types and convert accordingly
- **Invalid categorical values**: Use training data sample to see valid values
- **Time series format errors**: Ensure datetime format matches training data

## SDK Setup

### Install DataRobot SDK

```bash
pip install datarobot datarobot-predict
```

### Initialize Client

```python
import datarobot as dr

# Initialize client with API credentials
dr.Client()
```

### Environment Variables

Set these environment variables or pass them directly:

- `DATAROBOT_API_TOKEN` - Your DataRobot API token
- `DATAROBOT_ENDPOINT` - Your DataRobot endpoint (default: https://app.datarobot.com)

## Resources

- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)
- [DataRobot Predictions Documentation](https://docs.datarobot.com/en/docs/predictions/index.html)
- [DataRobot API Reference](https://docs.datarobot.com/en/docs/api/reference/index.html)
- [Batch Predictions Guide](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html)

Referenced files: 4 — file references are metadata, not package contents.

View saved version →

---
name: datarobot-setup
description: Sets up DataRobot for local development including Python SDK, dr-cli, Agent Assist, and all required dependencies. Use when the user has not yet worked with DataRobot on this machine, OR when any DataRobot task fails due to missing or invalid credentials. Covers first-time setup, re-authentication, and credential recovery.
---

# DataRobot Local Development Setup

You are helping set up DataRobot for local development. Before installing anything, audit what is already present and only install what is missing. Follow these steps in order.

## Step 1: Pre-Flight Check (Detect Existing Installation)

Build a checklist of what is already installed before doing any installation work. Run each check and record the result. Skip any install step in Steps 3-7 whose check below already passes.

```bash
# Required tools — record version for each, or "missing"
command -v python3   && python3 --version
command -v git       && git --version
command -v uv        && uv --version
command -v dr        && dr --version
command -v pulumi    && pulumi version
command -v task      && task --version
command -v node      && node --version
command -v pip       && pip --version

# DataRobot CLI plugins (only meaningful if `dr` exists)
dr plugin list 2>/dev/null | grep -i assist

# Python SDK (in the user's active environment)
python3 -c "import datarobot; print(datarobot.__version__)" 2>/dev/null
python3 -c "import datarobot_predict; print('datarobot-predict installed')" 2>/dev/null

# Credential state
echo "DATAROBOT_API_TOKEN: ${DATAROBOT_API_TOKEN:+set (via env)}"
echo "DATAROBOT_ENDPOINT:  ${DATAROBOT_ENDPOINT:-not set}"
test -f ~/.config/datarobot/drconfig.yaml && echo "drconfig: found" || echo "drconfig: missing"

# If drconfig exists, verify credentials are actually valid
if test -f ~/.config/datarobot/drconfig.yaml; then
  dr auth check 2>/dev/null && echo "auth: valid" || echo "auth: INVALID (re-authentication required)"
fi
```

Compare each detected version to the minimums in the table in Step 3. Tell the user:
- What is already installed and at acceptable versions (skip)
- What is missing or below minimum version (install)
- Whether dr-cli is already authenticated (skip Step 6 if so, but confirm with user)

Only proceed past this step after presenting the diff to the user so they can confirm.

## Step 2: Detect Operating System

Detect the OS and tailor commands accordingly.

### Windows Users: WSL Required

**IMPORTANT**: DataRobot Agent Assist does NOT support native Windows. You MUST use WSL (Windows Subsystem for Linux).

#### Check if Running in WSL

```bash
uname -r | grep -i microsoft         # Returns output if in WSL
cat /proc/version | grep -i microsoft # Alternative
echo $WSL_DISTRO_NAME                 # Empty if not in WSL
```

#### If NOT in WSL

1. **Install WSL 2** (Windows 10/11). Open PowerShell as Administrator and run `wsl --install`, then restart when prompted. Default Ubuntu will be installed.
2. **Alternative manual install**: `wsl --install -d Ubuntu-22.04` from elevated PowerShell.
3. **Set up Ubuntu**: launch Ubuntu from the Start menu, create a username and password, then run `sudo apt update && sudo apt upgrade -y`.
4. **Return here**: once in WSL, re-run this skill from the Ubuntu terminal.

#### Supported Environments

- macOS — Homebrew install path
- Linux — distribution-specific installers
- WSL — Linux installers
- Native Windows — NOT supported; use WSL

## Step 3: Install Missing Core Dependencies

Only install the tools flagged as missing in Step 1.

| Tool | Minimum Version | Purpose |
|------|-----------------|---------|
| Python | 3.10+ | DataRobot SDK and Agent Assist |
| git | 2.30.0+ | Version control |
| uv | 0.9.0+ | Python package manager |
| dr-cli | 0.2.50+ | DataRobot CLI |
| Pulumi | 3.163.0+ | Infrastructure as Code |
| go-task | 3.43.3+ | Task runner |
| Node.js | 24+ | JavaScript runtime |

### macOS (Homebrew)

```bash
brew install datarobot-oss/taps/dr-cli uv pulumi/tap/pulumi go-task node git python
```

### Linux / WSL

Use the architecture-aware official installers below. These work on both x86_64 and ARM64. Some of them download and run an installer script, so show the user each command and get their confirmation before running it. Never substitute a different URL.

- **dr-cli** (universal installer — auto-detects architecture):
  ```bash
  curl -fsSL https://cli.datarobot.com/install | sh
  ```

- **Python 3.10+** (uses whatever Python the distro ships; on Ubuntu 22.04 that's 3.10, on 24.04 it's 3.12 — both satisfy the minimum):
  ```bash
  sudo apt update
  sudo apt install -y python3 python3-pip python3-venv
  ```

- **git**:
  ```bash
  sudo apt install -y git
  ```

- **uv** (Python package manager):
  ```bash
  curl -LsSf https://astral.sh/uv/install.sh | sh
  ```

- **Node.js 24** (via nvm):
  ```bash
  curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
  source ~/.bashrc   # or ~/.zshrc
  nvm install 24
  nvm use 24
  ```

- **Pulumi**:
  ```bash
  curl -fsSL https://get.pulumi.com | sh
  ```

- **go-task** (installed from PyPI with `uv`, so it needs no `sudo` and runs no downloaded script):
  ```bash
  uv tool install go-task-bin
  ```

Note: All Linux commands above also work in WSL.

## Step 4: Install Python SDK (in a virtual environment)

Modern Linux distributions enforce PEP 668 and reject `pip install` into the system Python. Always install the SDK into a `uv`-managed virtual environment so this works on Ubuntu 23.04+, 24.04, and macOS with Homebrew Python.

```bash
# Create and activate a project-scoped venv with uv
uv venv .venv
source .venv/bin/activate

# Install the SDK and prediction client into the venv
uv pip install datarobot datarobot-predict
```

If the user prefers `pip` directly, the same pattern applies — they must create and activate a venv first, then run `pip install datarobot datarobot-predict` inside it. Never instruct the user to run `pip install --break-system-packages` against the system Python.

## Step 5: Get API Key

If the user does not already have a DataRobot Personal API key:

1. Open: https://app.datarobot.com/account/developer-tools
2. Use the **Personal API keys** tab (NOT Application or Agent keys).
3. Wait for the user to provide the key.

## Step 6: Authenticate with dr-cli

If Step 1 showed `~/.config/datarobot/drconfig.yaml` already exists, ask the user if they want to re-authenticate or keep the existing credentials. Otherwise run:

```bash
dr auth login
```

This persists credentials in `~/.config/datarobot/drconfig.yaml`.

Optionally, offer to add these to the user's shell rc file (~/.zshrc, ~/.bashrc) for SDK use:

```bash
export DATAROBOT_ENDPOINT="<endpoint-url>"
export DATAROBOT_API_TOKEN="<api-token>"
```

## Step 7: Install Agent Assist Plugin

Skip if Step 1 showed the `assist` plugin already installed. Otherwise:

```bash
dr plugin install assist
```

## Step 8: Verify Installation

1. **CLI and plugins**:
   ```bash
   dr --version
   dr plugin list 2>/dev/null | grep -i assist
   ```

2. **Python SDK**:
   ```bash
   source .venv/bin/activate && python -c "import datarobot; datarobot.Client(connect_timeout=10); [print(p.project_name) for p in datarobot.Project.list(limit=3)]"
   ```
   if this fails due to the source command not working, try it without activating the venv

## Step 9: Print Summary

Summarize:
- Tools installed this session vs. tools already present
- Versions of every tool now on the path
- Config file locations:
  - `~/.config/datarobot/drconfig.yaml` (dr-cli)
  - Shell rc file (if env vars were added)
  - Project venv path (e.g. `./.venv`)
- Reminder: "Installation complete. Do not run `dr assist` yet unless in a dedicated empty directory."

## Important Notes

- **Do NOT run `dr assist`** during this setup. Only install and verify.
- Agent Assist must only be run from a dedicated empty directory to avoid overwriting existing files.
- Ensure all minimum version requirements are met before completing.
- If any verification step fails, troubleshoot before proceeding.

View saved version →

---
name: datarobot-workload-api
description: >-
  Use when the user wants to create, configure, scale, debug, observe, or roll
  out container workloads on DataRobot's Workload API. Triggers: deploy a
  container as a managed service; list/start/stop workloads; change replica
  counts or autoscaling; pick CPU/GPU compute bundles; inject DataRobot
  credentials as env vars; diagnose stuck/errored/crash-looping workloads
  (CrashLoopBackOff, ImagePullBackOff, OOMKilled, probe failures, exec format
  error); pull application logs/OpenTelemetry traces/metrics/request stats;
  create or iterate container artifacts; build images server-side; lock
  artifacts for production; zero-downtime rolling artifact replacement.
---

# DataRobot Workload API

Runs containers as managed, autoscalable DataRobot services. Four jobs — pick by intent:

1. **Create / configure / scale** — deploy a container; change replicas, resources, autoscaling, bundle; inject credentials
2. **Diagnose** — workload stuck, errored, or crash-looping
3. **Observe** — logs, traces, metrics, service stats for a running workload
4. **Artifact lifecycle** — iterate drafts, build images, lock for production, roll out new versions

## Prerequisites

Auth: like `gh`. `dr auth login` (or existing `.env`/`~/.config/datarobot/drconfig.yaml`) persists credentials — no per-run env vars needed. Verify with `dr auth check`. On failure, run `datarobot-setup`.

**Keep the CLI current** (like `datarobot-agent-assist`): `dr self update --force`. Never gate on a pinned version. If a subcommand still errors `unknown command` after update (seen with `config`/`up` — `references/declarative-cli-deploy.md`), use the raw-REST fallback below.

`DATAROBOT_ENDPOINT` (must end `/api/v2`) and `DATAROBOT_API_TOKEN`: required as env vars only for raw REST (`scripts/`, `httpx`/`curl`) or CI — these bypass CLI auth. Header: `Authorization: Bearer ${DATAROBOT_API_TOKEN}`. Not in the `datarobot` Python SDK — call REST directly.

**Transport:** examples use Python `httpx` (`pip install httpx`); `curl` or the `pulumi-datarobot` provider work too.

## Bundled scripts

Python in `scripts/`. Each uses `httpx`, reads `DATAROBOT_ENDPOINT` + `DATAROBOT_API_TOKEN`:

- `wait_for_running.py <workload_id>` — poll until `running`; exit 2 on terminal failure, 3 on timeout
- `diagnose_workload.py <workload_id>` — run the 5-step debug flow, print a diagnosis (`--json` for machine output)
- `wait_for_build.py <artifact_id> <build_id>` — poll a server-side image build; dumps last 2KB of logs on `FAILED`
- `wait_for_replacement.py <workload_id>` — poll a rolling replacement; handles the 404-when-cleared case
- `check_limits.py` — print effective org scaling limits via `/account/info/`

## Deeper docs in references/

In `references/`:

- `status-vocabulary.md` — status enums and transitions
- `common-error-patterns.md` — CrashLoopBackOff / ImagePullBackOff / OOMKilled / probe / exec-format
- `schema-reference.md` — schemas, credential-type→key maps, spec quirks
- `lifecycle-flows.md` — draft→lock→prod rules, replacement preconditions, redeploy matrix
- `code-to-workload.md` — deploy from source: `dr` CLI, `codeRef`, Execution Environments
- `declarative-cli-deploy.md` — `dr workload config`/`up`: one-command create/build/deploy
- `web-uis-behind-the-edge.md` — web UI through the endpoint: prefix, auth gate, CSRF, WebSockets

## OpenAPI spec is source of truth

At `${DATAROBOT_ENDPOINT}/openapi.yaml` (some instances omit Workload API paths — see `references/schema-reference.md`). **~5 MB — never dump whole.** Save once, slice with `yq` (or `print()` one key in Python):

```bash
curl -sS "${DATAROBOT_ENDPOINT}/openapi.yaml" -o /tmp/wapi-spec.yaml
yq '.components.schemas.CreateWorkloadRequest' /tmp/wapi-spec.yaml
```

Workload paths key with `/api/v2/` — see `references/schema-reference.md`.

---

# 1. Create / configure / scale

## Deploy a container (common case)

```yaml
# spec.yaml — JSON also accepted; spec is sent verbatim
name: my-api-service
importance: low
artifact:
  name: my-api-service-artifact
  spec:
    type: service
    containerGroups:
      - name: default
        containers:
          - name: main
            imageUri: ghcr.io/org/my-app:latest
            port: 8000
            primary: true
            readinessProbe: {path: /readyz, port: 8000, initialDelaySeconds: 10}
            livenessProbe: {path: /healthz, port: 8000, initialDelaySeconds: 30}
runtime:
  containerGroups:
    - name: default          # must match artifact.spec.containerGroups[].name (above)
      replicaCount: 1
      containers:
        - name: main
          resourceAllocation: {cpu: 1, memory: "512MB"}
```

```bash
dr workload create --spec-file spec.yaml         # 4xx: 400=schema/limit, 403=cap (run check_limits.py), 409=name conflict
dr workload get <workload_id>                    # or `dr workload status` — poll until status=running
```

Lifecycle one-liners: `dr workload {stop|start|delete|endpoint|list} <id>`.

Raw fallback when CLI unavailable: `httpx.post(f"{base}/workloads/", headers=headers, json=spec)` + `r.raise_for_status()` + `r.json()["id"]`; poll with `python scripts/wait_for_running.py <workload_id>`.

**Critical gotchas:**

- `importance`: `low`/`moderate`/`high`/`critical`. `type`: `service` (default) or `nim`. One container per group must be `primary: true`.
- `cpu`: cores (float OK). `memory`: decimal string (`"512MB"`, units B/KB/MB/GB) or byte integer. No Kubernetes binary suffixes (`Mi`/`Gi`).
- `port` must be `>= 1024`. Container must actually listen on it (image env var or entrypoint).
- Image needs a **linux/amd64** manifest. Apple Silicon defaults to ARM64: crash-loops with `exec format error`. Build: `docker buildx build --platform linux/amd64,linux/arm64 -t <ref> --push .`.
- Status: `submitted` → `provisioning` → `launching` → `running` (happy path); `updating` = rolling redeploy; `errored` recoverable; `failed`/`terminated` unrecoverable. Full table: `references/status-vocabulary.md`.

## Web UI through the endpoint

Browser web app (UI + backend/API/WebSocket) via `dr workload endpoint <id>`, not headless: edge gateway serves it under a path prefix. **Strips prefix inbound** (no outbound rewrite; app must be sub-path aware). **Is the auth gate** (DataRobot login), **hijacks `Authorization`** (→ `401 {"message":"Invalid API key"}`, container never sees it). **Passes WebSockets through**. Pattern: base-path = prefix (from `WORKLOAD_ID`) + inbound shim, CSRF off, unauthenticated probe path. Disable app auth only with user sign-off. Detail: `references/web-uis-behind-the-edge.md`.

## Update paths

| User intent | Endpoint | Effect |
|---|---|---|
| Rename / redescribe / change importance | `PATCH /workloads/{id}/` | Metadata only — no restart |
| Change replicas / resources / autoscaling on the same artifact | `PATCH /workloads/{id}/settings/` | Triggers rolling redeploy |
| Deploy a different artifact (new image / version) | `POST /workloads/{id}/replacement/` | Rolling swap — see section 4 |

## Replicas, resources, autoscaling

`PATCH /workloads/{wid}/settings/`, full body. Use `replicaCount` OR `autoscaling`, not both. Read via `GET /workloads/{wid}/settings/` first, then PATCH back:

```python
httpx.patch(
    f"{base}/workloads/{wid}/settings/",
    headers=headers,
    json={
        "runtime": {
            "containerGroups": [
                {
                    "name": "default",
                    "replicaCount": 3,
                    "containers": [
                        {
                            "name": "main",
                            "resourceAllocation": {"cpu": 2, "memory": "1GB"},
                        }
                    ],
                    # OR: "autoscaling": {"enabled": True, "policies": [{
                    #       "scalingMetric": "cpuAverageUtilization",
                    #       "target": 70, "minCount": 1, "maxCount": 10}]}
                }
            ]
        }
    },
)
```

`scalingMetric` values: `cpuAverageUtilization`, `httpRequestsConcurrency`, `gpuCacheUtilization`, `gpuRequestQueueDepth`, or a custom NIM metric. Settings updates are **rolling** — zero-downtime needs `replicaCount >= 2` (or autoscaling `minCount >= 2`).

## Org-set scaling limits

Admin-set caps: `maxConcurrentWorkloads`, `maxWorkloadReplicas`. `0` = unlimited; users can't change them. Read via **`GET /account/info/`** → `{"limits": {"maxConcurrentWorkloads": N, "maxWorkloadReplicas": M}}` (or `python scripts/check_limits.py`). `/users/{uid}/` and `/organizations/{id}/` need Admin API access. Over limit: **HTTP 403** `{"detail": "Requested replicas (N) exceeds the maximum allowed (M)."}` — check limits first; propose the max, or flag admin help needed.

## GPU/VRAM via compute bundle

`resourceAllocation` accepts only `cpu`, `memory`, `gpu` (count) — no `gpuType`/`gpuMemory` field. For a GPU model/VRAM size: `GET /mlops/compute/bundles/` lists bundles (`cpu.small`, `gpu.l4.small`, `gpu.a10g.medium`); pass `"resourceBundles": ["gpu.l4.small"]` (a list, but only ONE bundle allowed) under the container group. A set bundle overrides `resourceAllocation` cpu/memory.

## Credential injection — no hardcoded secrets

DataRobot stores credentials centrally, injects into `environmentVars` by reference:

```python
"environmentVars": [
    {"name": "PLAIN_VAR", "value": "literal-value"},
    {"source": "dr-credential", "name": "AWS_ACCESS_KEY_ID",
     "drCredentialId": "<credential-id>", "key": "awsAccessKeyId"},
]
```

Flow: `GET /credentials/?limit=50` → note `credentialType` → look up valid `key` names in `references/schema-reference.md`.

## Create from artifact

Provide `artifactId` instead of an inline `artifact` block. `runtime`'s `containerGroups[].name`/`containers[].name` must match the artifact's.

---

# 2. Diagnose

## Full diagnosis, one command

```bash
python scripts/diagnose_workload.py <workload_id>
```

Runs the 5 steps below, prints a report (status, logTail signals, flagged events, proton K8s detail, evidence, recommended fix, console URL). `--json` for machine output. Empty `Evidence`: pull application logs via section 3 — don't guess from status alone.

## The 5-step flow

Script does this; steps below for ambiguous output or one-off calls.

1. **`GET /workloads/{id}/`** — `status`, `statusDetails.logTail` (~30 lines; scan for `error`/`exception`/`traceback`/`killed`/`permission denied`/`connection refused`), `statusDetails.conditions`. `statusDetails` is `null` during `submitted`/`provisioning` — guard for it.
2. **`GET /workloads/{id}/events/`** — flag `type: Warning` or `reason` with `Failed`/`Error`/`Kill`/`OOM`; the last Warning before `errored` is usually the trigger.
3. **`GET /workloads/{id}/protons/`** — pick `role: "active"` (or the `candidate` during a rolling replacement; else newest `createdAt`).
4. **`GET /workloads/{id}/protons/{pid}/statusDetails/`** — `204` while initializing (not an error). Read `replicas[*].containers[*].status`+`restartCount` → `replicas[*].conditions[*]` (any `value:false`) → `overallStatus.summary`.
5. **Application logs** — section 3.

Common patterns (`CrashLoopBackOff`, `ImagePullBackOff`, `OOMKilled`, probe/pending, `exec format error`) and fixes: `references/common-error-patterns.md`.

## Diagnosis output

```
Workload {id} — Diagnosis
- Status: {current}
- Root cause: {one sentence}
- Evidence: {the specific logTail line, condition, container reason, or event}
- Recommended fix: {next step — section 1 (settings), section 4 (artifact), or app code}
- Console: https://app.datarobot.com/console-nextgen/workloads/{id}/overview
```

---

# 3. Observe

| Stream | Endpoint | Needs app instrumentation? |
|---|---|---|
| Logs | `/otel/workload/{id}/logs/` | No — auto from stdout/stderr |
| Traces | `/otel/workload/{id}/traces/` | **Yes** (OTEL spans) |
| Metrics | `/otel/workload/{id}/metrics/autocollectedValues/` | Partially |
| Service stats | `/workloads/{id}/stats/` | No — DataRobot edge proxy |
| Replacement history | `/workloads/{id}/history/` | No — platform |
| Lifecycle events | `/workloads/{id}/events/` | No — platform |

Check `r.status_code` before `.json()`: 401 bad token, 404 not found, 429 rate-limited (exponential backoff). List endpoints accept `limit` + `offset`.

## Logs

```bash
dr workload logs <wid> --level error --limit 100   # --follow streams; --output-format json
```

`--level` is a MINIMUM severity threshold, not an exact match. For substring filtering on the message body, or proton-scoped logs (proton IDs: section 2), drop to REST — `dr workload logs` lacks these filters:

```python
r = httpx.get(
    f"{base}/otel/workload/{wid}/logs/",
    headers=headers,
    params=[
        ("searchKeys", "proton_id"),
        ("searchValues", pid),
        ("searchKeys", "level"),
        ("searchValues", "error"),
    ],
)
```

`searchKeys`/`searchValues` are positional parallel lists — pass a **list of tuples** to httpx (dict can't repeat keys). `includes=<substring>`: case-sensitive substring filter on the message body.

## Traces

```python
traces = httpx.get(f"{base}/otel/workload/{wid}/traces/", headers=headers).json()[
    "data"
]
# summary: traceId, rootSpanName, rootServiceName, duration (NANOSECONDS), spansCount, errorSpansCount
trace_id = next(
    (t["traceId"] for t in traces if t.get("errorSpansCount", 0) > 0),
    traces[0]["traceId"],
)
trace = httpx.get(
    f"{base}/otel/workload/{wid}/traces/{trace_id}/", headers=headers
).json()
```

> **`duration` is NANOSECONDS**, summaries and spans. Divide by 1,000,000 for ms before display. Empty `data`: app not instrumented — tell the user to wire up OTEL.

## Metrics + service stats

Convert before display: `bytes`→MB (`/1024**2`), `nanocores`→cores (`/1_000_000`), `percentage` already %.

```python
stats = httpx.get(f"{base}/workloads/{wid}/stats/", headers=headers).json()
# {"period": {...}, "metrics": {totalRequests, serverErrors, userErrors, slowRequests,
#   responseTime, requestsPerMinute, concurrentRequests, *ErrorRate}}. /workloads/stats/ = aggregate.
```

> **Destructive:** `DELETE /workloads/{id}/stats/?metricName=<name>` zeroes a metric's history — only on explicit request.

## Output format

Logs: `timestamp | level | message`, ERROR/CRITICAL first. Traces: table sorted by errors desc then recency. Metrics: apply unit conversion before display. Service stats one-liner: *"`{totalRequests}` requests, `{totalErrorRate*100:.2f}%` errors, `{responseTime:.1f}` ms avg, `{requestsPerMinute}` req/min."* Empty data: state *why* (not running, not instrumented, empty window) — not just "no data".

---

# 4. Artifact lifecycle

**Artifact**: immutable-after-lock definition of what a workload runs (image, port, env vars, probes). **Workload**: the running instance + its runtime (replicas, resources, autoscaling). Resources do NOT belong on the artifact.

## Choose a redeploy path

Find the running artifact (`workload["artifactId"]`), check `artifact["status"]`. A running workload does **not** auto-adopt a rebuild — redeploy is required.

- **Same draft (C2W loop) — in-place edit or rebuild.** PATCH/rebuild the draft, roll with `PATCH /workloads/{id}/settings/`: re-send the runtime body (even unchanged — triggers rolling `202`, re-reads current spec + latest `COMPLETED` build). Zero-downtime at ≥2 replicas. (`POST /replacement/` onto the same draft also works.)
- **Different / locked artifact.** `POST /replacement/` onto the other artifact ID. Locked in-place edit: clone → PATCH clone → lock → replace onto the clone.

**Lock:** `dr artifact lock <id>` (= `PATCH /artifacts/{id}/ {"status":"locked"}`). **Promote** (`POST /workloads/{wid}/promote/`, 200) locks the running draft in place, no restart. Runtime-only changes (replicas/resources/autoscaling) → `PATCH /settings/`; a PATCH to the artifact affects live workloads only on redeploy.

Preconditions (status-match, same-artifact rule) and the full redeploy matrix: `references/lifecycle-flows.md`.

## Image source

The artifact's `imageUri` must point at a registry DataRobot can pull from (image-pull creds not yet accepted at workload creation). Two paths:

1. **Bring your own image** — public registry or one the admin pre-configured. `docker buildx ... --platform linux/amd64`, push, set `imageUri`. Default flow.
2. **Code-to-Workload (C2W)** — no local Docker / no public registry: `dr artifact code init` + `sync`, then `dr artifact build create` builds server-side, pushes to DataRobot's internal registry, and populates `imageUri`. Full flow: `references/code-to-workload.md`.

Poll: `python scripts/wait_for_build.py <artifact_id> <build_id>`; only drafts build. **`imageUri` is build-managed** — never hand-PATCH it (`422` "not permitted on this cluster"), never PATCH the spec *mid-build* (whole-spec write clobbers the pending image → redeploys the old one). Edit spec before `build create` or after `COMPLETED`.

> **C2W may still need `ENABLE_WORKLOAD_API_CONTAINERS=true`** (org-set; not re-verified this session).

## Rolling artifact replacement

```python
httpx.post(
    f"{base}/workloads/{wid}/replacement/",
    headers=headers,
    json={
        "artifactId": new_artifact_id,
        "strategy": "rolling",  # only "rolling" supported
        "config": {"warmupDurationMinutes": 2, "keepOldVersionMinutes": 5},  # optional
        # "runtime": {...}  # optional; same shape as PATCH /settings/
    },
)
```

Monitor: `python scripts/wait_for_replacement.py <workload_id>`. Preconditions: status must match (draft↔draft/locked↔locked, else `400`); same-artifact replacement 422s for locked, works for drafts — for a draft roll without replacement, use `PATCH /settings/`. **Not idempotent** — a second `POST` queues another swap. `GET .../replacement/` `404` = none in progress. `DELETE` to cancel. Detail: `references/lifecycle-flows.md`.

---

## Related skills

- `datarobot-setup` — install SDK, configure auth, set env vars
- `datarobot-app-framework-cicd` — declarative artifact + workload management via Pulumi and CI/CD
- `datarobot-external-agent-monitoring` — instrument arbitrary agent code with OTEL → DataRobot

Referenced files: 12 — file references are metadata, not package contents.