Update to DataRobot Skills

{
  "description": "Guidance for setting up CI/CD pipelines for DataRobot application templates using GitLab, GitHub Actions, and Pulumi for infrastructure as code. Use when setting up CI/CD pipelines, configuring deployments, or managing infrastructure for DataRobot application templates.",
  "included_files": [
    {
      "relative_path": "QUICK_REFERENCE.md",
      "size_in_bytes": 4080
    },
    {
      "relative_path": "README.md",
      "size_in_bytes": 6710
    },
    {
      "relative_path": "examples/github-cd-pulumi-cloud.yml",
      "size_in_bytes": 1156
    },
    {
      "relative_path": "examples/github-deploy-pulumi-cloud.yml",
      "size_in_bytes": 1318
    },
    {
      "relative_path": "examples/github-destroy-pulumi-cloud.yml",
      "size_in_bytes": 1229
    },
    {
      "relative_path": "examples/workflows-README.md",
      "size_in_bytes": 3049
    },
    {
      "relative_path": "references/example-taskfile.yaml",
      "size_in_bytes": 895
    },
    {
      "relative_path": "references/workflow-examples.md",
      "size_in_bytes": 2586
    },
    {
      "relative_path": "scripts/decrypt-secrets.sh",
      "size_in_bytes": 1257
    },
    {
      "relative_path": "scripts/encrypt-secrets.sh",
      "size_in_bytes": 1933
    },
    {
      "relative_path": "scripts/github-cd.yml",
      "size_in_bytes": 2060
    },
    {
      "relative_path": "scripts/github-deploy.yml",
      "size_in_bytes": 5124
    },
    {
      "relative_path": "scripts/github-destroy.yml",
      "size_in_bytes": 1965
    },
    {
      "relative_path": "scripts/gitlab-ci.yml",
      "size_in_bytes": 4863
    },
    {
      "relative_path": "scripts/infra-README.md",
      "size_in_bytes": 6567
    },
    {
      "relative_path": "scripts/pulumi-setup.sh",
      "size_in_bytes": 11011
    },
    {
      "relative_path": "scripts/setup-github-secrets.sh",
      "size_in_bytes": 3678
    },
    {
      "relative_path": "scripts/setup-gitlab-variables.sh",
      "size_in_bytes": 4512
    },
    {
      "relative_path": "scripts/taskfile-snippets.yaml",
      "size_in_bytes": 5747
    }
  ],
  "name": "datarobot-app-framework-cicd",
  "skill_md_contents": "---\nname: datarobot-app-framework-cicd\ndescription: Guidance for setting up CI/CD pipelines for DataRobot application templates using GitLab, GitHub Actions, and Pulumi for infrastructure as code. Use when setting up CI/CD pipelines, configuring deployments, or managing infrastructure for DataRobot application templates.\ncontext-tokens: \"~6 000 (SKILL.md) + ~2 000 (scripts/*) + ~400 per examples/* file\"\n---\n\n# DataRobot Application Templates CI/CD Skill\n\nThis skill provides comprehensive guidance for setting up production-grade CI/CD pipelines for DataRobot application templates, including automated testing, review deployments, and continuous delivery.\n\n## Quick Start\n\n**Default behavior:** When a user asks to \"set up CI/CD\" without specifying a platform or backend, always use the [Simple Path](#simple-path-pulumi-cloud--github-secrets) below — three workflow files, two GitHub Secrets, done. Do not create `infra/scripts/`, do not add CI/CD tasks to `infra/Taskfile.yaml`, do not involve GPG encryption unless the user explicitly asks for it.\n\nOnly deviate from the simple path when the user specifies:\n- A specific Pulumi state backend (Azure Blob, S3, GCS) → use `scripts/` and see [Implementation Pattern](#implementation-pattern)\n- GitLab CI/CD → see [GitLab CI/CD Configuration](#gitlab-cicd-configuration)\n- Many secrets to manage → consider GPG approach in `scripts/`\n\n## Simple Path: Pulumi Cloud + GitHub Secrets\n\nFor most data scientists and AI engineers, this is all you need. No GPG encryption, no cloud storage account, no extra scripts.\n\n**What to create in the user's repository:**\n\n1. Copy the three workflow files to `.github/workflows/`:\n\n   | Source | Destination | Trigger |\n   |--------|-------------|---------|\n   | `examples/github-cd-pulumi-cloud.yml` | `.github/workflows/cd.yml` | Automatic — every merge to `main` |\n   | `examples/github-deploy-pulumi-cloud.yml` | `.github/workflows/deploy-pr.yml` | Manual — user picks PR branch + enters stack name (e.g. `pr-42`) |\n   | `examples/github-destroy-pulumi-cloud.yml` | `.github/workflows/destroy.yml` | Manual — user enters stack name to tear down |\n\n2. Create `.github/workflows/README.md` from `examples/workflows-README.md`. This is the setup guide that tells the user exactly what secrets and variables to add and how.\n\n3. Tell the user to follow the setup guide in `.github/workflows/README.md`.\n\nThat's it. Do **not** add anything to `infra/Taskfile.yaml` or create `infra/scripts/` for this path.\n\n**Required GitHub Secrets** (both required — no defaults):\n\n| Name | Kind |\n|------|------|\n| `DATAROBOT_API_TOKEN` | Secret |\n| `PULUMI_ACCESS_TOKEN` | Secret |\n\n**Optional GitHub Variable** (defaults to `ci` if not set):\n\n| Name | Kind | Default |\n|------|------|---------|\n| `PULUMI_STACK_CI_NAME` | Variable | `ci` |\n\n**When to use the advanced approach (GPG + DIY backends) instead:**\n- You have many secrets (GPG encrypts all of `.env` behind a single passphrase — only one GitHub Secret needed)\n- Your organization prohibits Pulumi Cloud and requires a self-managed backend (Azure Blob / S3 / GCS)\n- You need GitLab CI/CD\n\nThe templates and scripts for all of these are in `scripts/` in this skill directory. If the skill has already been propagated to the project's `infra/` directory (common in downstream templates), look in `infra/scripts/` instead. See the [Implementation Pattern](#implementation-pattern) section below for full setup guidance.\n\n| Scenario | Key files in `scripts/` |\n|----------|------------------------|\n| Azure Blob / S3 / GCS Pulumi backend | `pulumi-setup.sh`, `taskfile-snippets.yaml` |\n| GitHub Actions + GPG secrets | `github-deploy.yml`, `github-cd.yml`, `encrypt-secrets.sh`, `setup-github-secrets.sh` |\n| GitLab CI/CD | `gitlab-ci.yml`, `setup-gitlab-variables.sh` |\n\n### Adapting the deploy command\n\nThe example workflows use `uv run pulumi up --yes` directly. Before copying them, check `infra/Taskfile.yaml` — the project may already wrap the deploy command in a task:\n\n```bash\ncat infra/Taskfile.yaml   # look for 'up-yes', 'deploy', or similar tasks\n```\n\n| What you find | What to use in CI |\n|---------------|-------------------|\n| `up-yes` task | `task up-yes` — non-interactive, purpose-built for CI; prefer this over raw Pulumi |\n| `deploy` task (alias for `up`) | Avoid — typically runs `pulumi up` interactively; only safe in CI if you confirm it passes `-y` internally |\n| No Taskfile or no relevant task | Keep `uv run pulumi up --yes` as-is |\n\nTo use `task` in a workflow, add an install step and swap the run command:\n\n```yaml\n- name: Install Task\n  run: pip install go-task-bin\n\n- name: Deploy\n  working-directory: infra\n  env:\n    DATAROBOT_API_TOKEN: ${{ secrets.DATAROBOT_API_TOKEN }}\n    PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n  run: |\n    uv sync --all-extras\n    task up-yes\n```\n\n### DataRobot API token (service account)\n\n`DATAROBOT_API_TOKEN` should come from a **DataRobot service account** — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team.\n\nTo set one up: ask your DataRobot admin to create a dedicated user (e.g. `ci-bot@your-org.com`). Under that account, go to **Developer Tools → API Key** and generate a token. Store it as the `DATAROBOT_API_TOKEN` secret in GitHub.\n\n> **Note:** This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. \"Service account\" here just means a non-personal DataRobot user.\n\n## Implementation Pattern\n\nWhen implementing CI/CD for an application template, follow this structure:\n\n**Project Structure:**\n```\napplication-template-root/\n├── infra/\n│   ├── README.md                   # ⚠️ GENERATE THIS — tailored to the chosen CI/CD platform and Pulumi backend\n│   ├── Taskfile.yaml               # ⚠️ CI/CD tasks go HERE — copy from infra/scripts/taskfile-snippets.yaml\n│   └── scripts/                    # Copy entire scripts/ directory here\n│       ├── README.md               # Copy from scripts/infra-README.md\n│       ├── setup-github-secrets.sh\n│       ├── setup-gitlab-variables.sh\n│       ├── encrypt-secrets.sh\n│       ├── decrypt-secrets.sh\n│       ├── pulumi-setup.sh\n│       ├── gitlab-ci.yml\n│       ├── github-deploy.yml\n│       ├── github-cd.yml\n│       ├── github-destroy.yml\n│       └── taskfile-snippets.yaml\n├── .env                            # User's secrets (never commit!)\n├── .env.gpg                        # Encrypted secrets (commit for GitHub)\n├── .gitlab-ci.yml                  # Copy from infra/scripts/gitlab-ci.yml\n├── .github/\n│   └── workflows/\n│       ├── deploy.yml              # Copy from infra/scripts/github-deploy.yml (PR review deploys)\n│       ├── cd.yml                  # Copy from infra/scripts/github-cd.yml (push-to-main CD)\n│       └── destroy.yml             # Copy from infra/scripts/github-destroy.yml\n└── Taskfile.yml                    # Root Taskfile — ADD ONLY one `includes` entry (see below). DO NOT add tasks here.\n```\n\n**Key Points:**\n- **⚠️ ALWAYS generate `infra/README.md`** tailored to the chosen platform and backend — see \"Generating infra/README.md\" below\n- All CI/CD scripts go in `infra/scripts/` directory\n- **⚠️ CRITICAL: All CI/CD tasks go in `infra/Taskfile.yaml` — NEVER add CI/CD tasks directly to the root `Taskfile.yml`**\n- `.env` and `.env.gpg` stay in project root\n- Scripts in `infra/scripts/` reference `../../.env` (two levels up)\n- Root `Taskfile.yml` gets exactly ONE addition: an `includes` entry pointing to `./infra/Taskfile.yaml`\n- CI/CD configs (`.gitlab-ci.yml`, `.github/workflows/`) are copied to standard locations\n\n**Root Taskfile.yml — the only change needed:**\n```yaml\n# Add this includes block to the existing root Taskfile.yml:\nincludes:\n  infra:\n    taskfile: ./infra/Taskfile.yaml\n    dir: infra\n# Tasks are then run as: task infra:encrypt-secrets, task infra:setup-github-secrets, etc.\n```\n\n### Generating infra/README.md\n\nAfter determining the user's CI/CD platform (GitHub/GitLab) and Pulumi backend, **always create `infra/README.md`** with content tailored to their choices. It should cover:\n\n1. **Architecture overview** — which platform was chosen and why, and which Pulumi backend\n2. **First-time setup** — the exact sequence of `task infra:*` commands needed to bootstrap\n3. **Day-to-day tasks** — a table or list of the `task infra:*` commands relevant to their platform\n4. **How deployments work** — short description of each trigger:\n   - GitHub: `deploy.yml` fires on PR open/sync (review stack), `cd.yml` fires on push to main (CI stack), `destroy.yml` is manual\n   - GitLab: `review_app` is manual on MR, `deploy_ci` fires on push to default branch, `destroy_review_app` is manual\n5. **Secrets / credentials** — what variables/secrets are needed and where they live (GitHub Secrets, GitLab CI/CD variables, `.env.gpg`)\n6. **Stack migration note** — if backend was migrated from a local stack, document what was done so future contributors understand the history\n\nAdjust section titles, task names, and stack-naming strategy to match what was actually configured. The README should be accurate enough that a new contributor can set up CI/CD without referring to any other document.\n\n## Workflow examples\n\nSee [`references/workflow-examples.md`](references/workflow-examples.md) for step-by-step examples covering GitLab CI/CD, GitHub Actions with GPG secrets, and continuous delivery setup.\n\n## Using Task for workflow management\n\nApplication templates use [Task](https://taskfile.dev) to simplify local development and CI/CD workflows. Task provides a unified interface for Python and TypeScript/React components.\n\n### Example Taskfile.yaml\n\nSee [`references/example-taskfile.yaml`](references/example-taskfile.yaml) for a complete example.\n\n### Using Task in CI/CD\n\n```bash\n# Install Task\npip install go-task-bin\n\n# Install dependencies\ntask install\n\n# Run linters (with fixes)\ntask lint\n\n# Run linters (check only)\ntask lint-check\n\n# Run tests\ntask test\n```\n\n## GitLab CI/CD Configuration\n\nThe complete pipeline configuration lives in `scripts/gitlab-ci.yml`. Copy it to your repository root:\n\n```bash\ncp infra/scripts/gitlab-ci.yml .gitlab-ci.yml\n```\n\nKey pipeline jobs:\n- `lint` / `test` — run on every same-project MR\n- `review_app` — manual deploy per MR; stack name driven by the `PULUMI_STACK_REVIEW_NAME` CI/CD variable\n- `deploy_ci` — automatic deploy on merge to default branch; stack name driven by `PULUMI_STACK_CI_NAME`\n- `destroy_review_app` — manual cleanup of review stacks\n\n`PULUMI_STACK_REVIEW_NAME` and `PULUMI_STACK_CI_NAME` must be set as plain CI/CD variables in GitLab (Settings → CI/CD → Variables). The pipeline file includes sensible defaults that project-level variables override.\n\n## GitHub Actions Configuration\n\nThe complete workflow files live in `scripts/`:\n\n- `scripts/github-deploy.yml` → copy to `.github/workflows/deploy.yml`\n- `scripts/github-destroy.yml` → copy to `.github/workflows/destroy.yml`\n\n```bash\nmkdir -p .github/workflows\ncp infra/scripts/github-deploy.yml .github/workflows/deploy.yml\ncp infra/scripts/github-destroy.yml .github/workflows/destroy.yml\n```\n\nThe deploy workflow triggers on pull requests and derives `PULUMI_STACK_NAME` from the `PULUMI_STACK_REVIEW_NAME` Actions variable and the PR number. Set `PULUMI_STACK_REVIEW_NAME` and `PULUMI_STACK_CI_NAME` as repository **variables** (Settings → Secrets and variables → Actions → **Variables** tab), not secrets.\n\n## Pulumi State Management\n\n### Pulumi Cloud Backend (Recommended)\n\nThe simplest approach for managing Pulumi state:\n\n```bash\n# Install Pulumi (macOS; other platforms: https://www.pulumi.com/docs/install/)\nbrew install pulumi/tap/pulumi\n\n# Login to Pulumi Cloud\npulumi login\n\n# Create/select stack\npulumi stack select --create dev\n\n# Deploy\npulumi up\n```\n\n**CI/CD Setup**: Add `PULUMI_ACCESS_TOKEN` to your CI/CD secrets. Get token from [Pulumi Console](https://app.pulumi.com/account/tokens).\n\n### DIY Backend Options\n\nFor organizations that cannot use Pulumi Cloud:\n\n#### Azure Blob Storage\n\n```bash\n# Login to Azure backend\npulumi login azblob://container-name\n\n# Set Azure credentials\nexport AZURE_STORAGE_ACCOUNT=myaccount\nexport AZURE_STORAGE_KEY=mykey\n```\n\n#### AWS S3\n\n```bash\n# Login to S3 backend\npulumi login s3://bucket-name\n\n# AWS credentials from environment\nexport AWS_ACCESS_KEY_ID=...\nexport AWS_SECRET_ACCESS_KEY=...\n```\n\n#### Google Cloud Storage\n\n```bash\n# Login to GCS backend\npulumi login gs://bucket-name\n\n# GCP credentials from environment\nexport GOOGLE_CREDENTIALS=...\n```\n\n### Migrating Stacks to a Different Backend\n\nWhen a developer has an existing local stack (a `Pulumi.<stackname>.yaml` file) that was\ncreated against a different backend than the CI/CD destination, the stack state must be\nexported and re-imported before switching.\n\n`pulumi-setup.sh` handles this automatically: it checks `pulumi whoami --verbose` for the\n**Backend URL** and compares it with the target URL. If they differ and local stack files\nexist, it offers to migrate them.\n\n**Manual migration steps** (if not using the script):\n\n```bash\n# 1. Confirm current backend and stacks\npulumi whoami --verbose        # note \"Backend URL:\"\npulumi stack ls -a             # list stacks on current backend\n\n# 2. Export each stack that exists locally (Pulumi.<name>.yaml)\npulumi stack export --stack <stackname> --file <stackname>-backup.json\n\n# 3. Login to the new backend (set any required credentials first)\n#    Examples:\npulumi login                           # Pulumi Cloud\npulumi login azblob://my-container     # Azure Blob\npulumi login s3://my-bucket            # AWS S3\n\n# 4. Create the stack in the new backend and import state\npulumi stack select --create <stackname>\npulumi stack import --file <stackname>-backup.json\n\n# 5. Clean up the backup\nrm <stackname>-backup.json\n```\n\n**Key signals that migration is needed:**\n- `pulumi whoami --verbose` shows `Backend URL: file://` (local) but CI/CD uses cloud storage\n- Backend URL domain/scheme differs between developer machine and CI target\n\n### Managing Stacks Across Environments\n\n```bash\n# List all stacks\npulumi stack ls -a\n\n# Output:\n# NAME                                 LAST UPDATE   RESOURCE COUNT\n# organization/project/prod            1 day ago     15\n# organization/project/staging         2 days ago    12\n# organization/project/dev             1 hour ago    10\n# github-pr-repo-42                    3 hours ago   13\n\n# Select and update a stack\npulumi stack select dev\npulumi up\n\n# View stack outputs\npulumi stack output --json\n\n# Delete a stack\npulumi stack rm review-app-123 --yes\n```\n\n## Secrets Management\n\nAll credentials (DataRobot API token, Pulumi access token, LLM keys, cloud storage keys) are stored in `.env` and committed to the repository encrypted as `.env.gpg`. The only secret that needs to be configured in the CI/CD system directly is `CICD_SECRET_PASSPHRASE` (the GPG passphrase). Non-sensitive stack name variables (`PULUMI_STACK_CI_NAME`, `PULUMI_STACK_REVIEW_NAME`) are set as plain variables, not secrets.\n\n### DataRobot API token (service account)\n\n`DATAROBOT_API_TOKEN` should come from a **DataRobot service account** — a DataRobot user created for automation, not tied to anyone's personal login. This prevents CI/CD from breaking when the engineer who originally set it up leaves the team.\n\nTo set one up: ask your DataRobot admin to create a dedicated user (e.g. `ci-bot@your-org.com`). Under that account, go to **Developer Tools → API Key** and generate a token. Store it as the `DATAROBOT_API_TOKEN` secret in your CI/CD system.\n\n> **Note:** This is purely a DataRobot concept — it has no relation to Pulumi state management or backend configuration. \"Service account\" here just means a non-personal DataRobot user.\n\n### GitHub\n\nRun `scripts/setup-github-secrets.sh` for interactive setup — it sets `CICD_SECRET_PASSPHRASE` as a repository secret and `PULUMI_STACK_CI_NAME` / `PULUMI_STACK_REVIEW_NAME` as repository variables.\n\nTo encrypt `.env` for CI:\n\n```bash\ntask infra:encrypt-secrets\n# or: ./infra/scripts/encrypt-secrets.sh\n```\n\nAdd the resulting `.env.gpg` to git. For local decryption:\n\n```bash\ntask infra:decrypt-secrets\n# or: ./infra/scripts/decrypt-secrets.sh\n```\n\n### GitLab\n\nRun `scripts/setup-gitlab-variables.sh` for interactive setup — it sets:\n- `CICD_SECRET_PASSPHRASE` — masked, for decrypting `.env.gpg`\n- `GITLAB_API_TOKEN` — masked, for posting MR comments\n- `PULUMI_STACK_CI_NAME` / `PULUMI_STACK_REVIEW_NAME` — plain variables\n\nAlternatively configure in the UI: Project Settings → CI/CD → Variables. Mark `CICD_SECRET_PASSPHRASE` and `GITLAB_API_TOKEN` as **Masked** and **Protected**.\n\n## Best practices\n\n### CI/CD Pipeline Design\n\n1. **Fast feedback**: Run linting and testing in parallel\n2. **Manual gates**: Make review apps manual to save resources\n3. **Automatic cleanup**: Provide easy ways to destroy test environments\n4. **Stack isolation**: Use unique stack names per PR/MR\n5. **Idempotent operations**: Design deployments to be safely re-runnable\n\n### Pulumi State\n\n1. **Use centralized backends**: Enable collaboration and CI/CD\n2. **Stack naming conventions**: Use consistent patterns (e.g., `github-pr-{repo}-{number}`)\n3. **Clean up stacks**: Remove unused stacks to reduce clutter\n4. **State locking**: Backends handle this automatically\n5. **Backup state**: Cloud backends provide automatic backups\n\n### Security\n\n1. **Never commit secrets**: Use .gitignore for .env files\n2. **Encrypt sensitive data**: Use GPG for GitHub, CI/CD variables for GitLab\n3. **Rotate credentials**: Regularly update API tokens and keys\n4. **Scope permissions**: Use least-privilege access for service accounts\n5. **Audit access**: Monitor who has access to secrets\n\n### Resource Management\n\n1. **Tag resources**: Use consistent tagging for tracking\n2. **Set TTLs**: Consider time-to-live for review environments\n3. **Monitor costs**: Track resource usage per environment\n4. **Auto-cleanup**: Implement automatic deletion of old review apps\n5. **Resource limits**: Set quotas to prevent runaway costs\n\n## Troubleshooting\n\n### Common Issues\n\n**Pulumi state conflicts:**\n- Ensure only one deployment runs at a time per stack\n- Use unique stack names for concurrent deployments\n- Check backend connection and credentials\n\n**Secret decryption failures:**\n- Verify GPG passphrase is correct\n- Check .env.gpg file is in repository\n- Ensure GPG is installed in CI environment\n\n**Deployment timeouts:**\n- Increase timeout values in workflow\n- Check DataRobot API connectivity\n- Verify resource provisioning isn't blocked\n\n**Stack not found:**\n- List stacks: `pulumi stack ls -a`\n- Verify backend connection\n- Check stack name matches pattern\n\n**Resource conflicts:**\n- Use unique names per stack\n- Check for orphaned resources\n- Review Pulumi state for inconsistencies\n\n## Example Repositories\n\nReference implementations:\n\n- **GitLab**: [demo-data-agent](https://gitlab.com/datarobot-oss/demo-data-agent) - Complete GitLab CI/CD setup\n- **GitHub**: [demo-talk-to-my-data-agent](https://github.com/datarobot-forks/demo-talk-to-my-data-agent) - Complete GitHub Actions setup\n\n## Resources\n\n- [Task Documentation](https://taskfile.dev)\n- [Pulumi Documentation](https://www.pulumi.com/docs/)\n- [Pulumi State and Backends](https://www.pulumi.com/docs/iac/concepts/state-and-backends/)\n- [GitLab CI/CD](https://docs.gitlab.com/ci/)\n- [GitHub Actions](https://docs.github.com/actions)\n- [DataRobot Application Templates](https://docs.datarobot.com/en/docs/wb-apps/app-templates/index.html)\n- [DataRobot Codespaces](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html)\n"
}