Domino Data Lab

View saved version →

---
name: domino-ai-gateway
description: Access external LLM providers through Domino AI Gateway - a secure proxy with centralized API key management, usage monitoring, and compliance. Supports OpenAI, AWS Bedrock, Azure OpenAI, Anthropic, and more. Use when calling LLMs from Domino, configuring AI Gateway endpoints, or monitoring LLM usage and costs.
---

# Domino AI Gateway Skill

## Description
This skill helps users work with Domino AI Gateway - a secure proxy for accessing external Large Language Model (LLM) providers with centralized management, monitoring, and compliance.

## Activation
Activate this skill when users want to:
- Access LLM providers (OpenAI, AWS Bedrock, etc.) in Domino
- Configure AI Gateway endpoints
- Monitor LLM usage and costs
- Understand secure API key management
- Use LLMs in workspaces and jobs

## What is AI Gateway?

Domino AI Gateway provides:
- **Secure LLM Access**: Proxy to external LLM providers
- **Centralized API Keys**: Keys stored securely, never exposed to users
- **Usage Monitoring**: Track all LLM interactions
- **Access Control**: Granular permissions per endpoint
- **Audit Logging**: Compliance-ready logs

## Supported LLM Providers

| Provider | Models |
|----------|--------|
| OpenAI | GPT-4, GPT-4 Turbo, GPT-3.5 |
| AWS Bedrock | Claude, Titan, Llama 2 |
| Azure OpenAI | GPT-4, GPT-3.5 |
| Anthropic | Claude 3, Claude 2 |
| Google Vertex AI | PaLM, Gemini |
| Cohere | Command, Embed |

## Creating an AI Gateway Endpoint

### Via Domino UI
1. Go to **Endpoints** > **Gateway LLMs**
2. Click **Create Endpoint**
3. Configure:
   - **Name**: Endpoint name (e.g., `openai-gpt4`)
   - **Provider**: Select LLM provider
   - **Model**: Specific model to use
   - **API Key**: Provider API key (stored securely)
   - **Access**: Who can use this endpoint
4. Click **Create**

### Via API
```python
# Create endpoint via Domino API
import requests, os

TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]

response = requests.post(
    f"{BASE}/api/aigateway/v1/endpoints",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json={
        "name": "openai-gpt4",
        "provider": "openai",
        "model": "gpt-4",
        "providerApiKey": "sk-..."
    }
)
```

## Using AI Gateway in Code

### OpenAI-Compatible Interface
AI Gateway provides an OpenAI-compatible interface:

```python
from openai import OpenAI

# Configure client to use AI Gateway
client = OpenAI(
    api_key="not-needed",  # Handled by AI Gateway
    base_url="https://your-domino.com/api/aigateway/v1/openai"
)

# Use like standard OpenAI
response = client.chat.completions.create(
    model="openai-gpt4",  # Your endpoint name
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ]
)

print(response.choices[0].message.content)
```

### With LangChain
```python
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="openai-gpt4",  # Endpoint name
    openai_api_key="not-needed",
    openai_api_base="https://your-domino.com/api/aigateway/v1/openai"
)

response = llm.invoke("What is machine learning?")
print(response.content)
```

### Direct API Call
```python
import requests, os

TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]

response = requests.post(
    f"{BASE}/api/aigateway/v1/chat/completions",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {TOKEN}",
    },
    json={
        "model": "openai-gpt4",
        "messages": [{"role": "user", "content": "Hello!"}]
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

## Access Control

### Endpoint Permissions
Configure who can use each endpoint:
- **Everyone**: All Domino users
- **Specific Users**: Named individuals
- **Organizations**: Specific Domino organizations

### Setting Permissions
1. Go to endpoint settings
2. Click **Access Control*

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-api-intro
description: Introduction to calling Domino platform APIs. Covers authentication (API proxy, PAT, service account), which host to use for which API, pagination limits, and common errors. Use before any other Domino API, SDK, app, extension, or governance automation skill.
compatibility: Domino 6.3 and Domino Cloud. Requires DOMINO_API_HOST; in a run use DOMINO_API_PROXY or the localhost:8899 access-token endpoint, outside a run a Personal Access Token.
---

# Domino platform API intro

Applies to Domino 6.3 and Domino Cloud.

Read this skill first whenever you automate Domino over HTTP or the Python SDK.

## Authentication

Follow https://docs.domino.ai/cloud/reference/api/domino-api-authentication for every call (API proxy in-run, access token + platform host, PAT, and service account).

Legacy user API keys (`X-Domino-Api-Key`, `DOMINO_USER_API_KEY`) are described as deprecated, never recommended. Product migration docs: https://docs.domino.ai/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy

| Context | Summary (details on the auth page) |
|---------|-------------------------------------|
| In-run | Prefer `DOMINO_API_PROXY` with **no** `Authorization` header when set; else Bearer from `http://localhost:8899/access-token` on `DOMINO_USER_HOST` (or `DOMINO_API_HOST`) |
| Outside a run | Public deployment HTTPS URL + Bearer **PAT** (you) or **service account** (automation) |

Host selection (gateway, governance 404, WebVFS, inference) is separate from auth mode: [HOSTS.md](./HOSTS.md).

## Reference files (this skill)

| File | Use when |
|------|----------|
| [HOSTS.md](./HOSTS.md) | Pick base URL (gateway, WebVFS, inference, governance external URL) |
| [LIMITS.md](./LIMITS.md) | Pagination, caps, deprecated env names |
| [ERRORS.md](./ERRORS.md) | Retries, 502/timeouts, job status fields, datasource 500 workaround |
| [SDK-MAP.md](./SDK-MAP.md) | Which plugin skill or doc covers each surface |
| [API-SPECS.md](./API-SPECS.md) | OpenAPI files and route discovery |

## Next steps

Use domain skills under `skills/` (python-sdk, apps, governance, datasets, and others). They assume the auth and host rules above.

Referenced files: 5 — file references are metadata, not package contents.

View saved version →

---
name: domino-app-init
description: Scaffold a new Domino-ready web application (Vite + React, Streamlit, Dash, Flask, or Gradio) with proxy-compatible settings, port 8888 binding, and an app.sh entry point. Use when the user wants to start, initialize, or scaffold a new app to publish on Domino, or asks for a starter app.sh for a web framework.
---

# Initialize a Domino App

Create a web application that works behind Domino's reverse proxy on the first publish.

## Inputs

Collect these before writing files. Skip any the user has already given.

1. **Framework**: `vite-react` (default), `streamlit`, `dash`, `flask`, or `gradio`.
2. **Project name**: name for the app folder and `package.json` (React only).
3. **Model API integration**: whether the app calls a Domino model endpoint, and if so its URL.

If the target directory already contains an app, stop and ask before overwriting anything.

## Steps

1. Copy the matching template from this skill's `assets/` folder into the project:

   | Framework | Files to copy |
   |---|---|
   | `vite-react` | `assets/vite-react/vite.config.js`, `assets/vite-react/app.sh`, `assets/vite-react/.env.example` |
   | `streamlit` | `assets/streamlit/app.sh` |
   | `dash` | `assets/dash/app.sh`, and `assets/dash/app.py.template` saved as `app.py` |
   | `flask`, `gradio` | No template; write the files from the snippets below |

2. Make `app.sh` executable (`chmod +x app.sh`).
3. Keep these invariants in every generated file:
   - Bind to `0.0.0.0`, never `localhost` or `127.0.0.1`.
   - Listen on port `8888` unless the user asks for another port.
   - For Vite, set `base: './'` so assets resolve under the proxy prefix.
4. If the user wants Model API integration, add the endpoint URL and token as environment
   variables (see `.env.example`) and never hardcode credentials.
5. Finish with the post-initialization checklist below.

## Reference snippets

### React / Vite: `vite.config.js`

```javascript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [react()],
  base: './',
  server: { host: '0.0.0.0', port: 8888, strictPort: true },
  preview: { host: '0.0.0.0', port: 8888, strictPort: true },
  build: { outDir: 'dist', assetsDir: 'assets' }
})
```

### React / Vite: `app.sh`

```bash
#!/bin/bash
set -e
cd /mnt/code
npm ci
npm run build
npx serve -s dist -l 8888 --no-clipboard
```

### Streamlit: `app.sh`

```bash
#!/bin/bash
set -e
streamlit run app.py \
    --server.port 8888 \
    --server.address 0.0.0.0 \
    --server.headless true
```

### Dash: `app.py`

```python
import os
from dash import Dash, html

app = Dash(__name__)
app.layout = html.Div([html.H1("Domino Dash App")])

if __name__ == '__main__':
    app.run_server(host='0.0.0.0', port=8888, debug=False)
```

### Flask: `app.py` and `app.sh`

```python
import os
from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Domino Flask App"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8888, debug=False)
```

```bash
#!/bin/bash
set -e
python app.py
```

### Gradio: `app.py` (use the Flask `app.sh` above)

```python
import gradio as gr

demo = gr.Interface(fn=lambda text: text, inputs="text", outputs="text")

if __name__ == '__main__':
    demo.launch(server_name='0.0.0.0', server_port=8888)
```

## Post-initialization checklist

Tell the user to:

1. Configure environment variables in the Domino project settings.
2. Set `app.sh` as the app entry point when publishing the app.
3. Test locally with `./app.sh` before deploying.
4. Confirm the app listens on port 8888.

For deployment details, CI/CD, and proxy troubleshooting, use the `domino-apps` skill. If the app
shows 404s or blank pages after publishing, use the `domino-debug-proxy` skill.

Referenced files: 6 — file references are metadata, not package contents.

View saved version →

---
name: domino-apps
description: Deploy web applications to Domino Data Lab with expertise in React apps (Vite) behind Domino's reverse proxy. Covers app.sh configuration, port configuration, base path handling for SPAs, CI/CD with GitHub Actions, and proxy troubleshooting. Use when deploying apps to Domino, setting up CI/CD pipelines, fixing broken routing, or configuring JavaScript frameworks for Domino's proxy.
---

# Domino Apps Skill

This skill provides comprehensive knowledge for deploying web applications to Domino Data Lab, with special focus on React applications using Vite.

## Key Concepts

### Domino App Architecture

Domino apps run in containers behind a reverse proxy that:
1. Authenticates users via Domino's auth system
2. Strips the URL prefix before forwarding to your app
3. Routes traffic to your app container
4. Handles infrastructure provisioning, routing, and resource management

**Note:** Port selection is flexible; port 8888 is no longer required. You can use any port your application prefers.

### Critical Configuration Points

1. **Host Binding**: Bind to `0.0.0.0` (not localhost) so Domino can reach your app
2. **Relative Base Path**: Use `base: './'` in Vite config for React apps
3. **app.sh**: Entry point script (launch file) that Domino executes

## Related Documentation

- [API-APPS.md](./API-APPS.md) - REST automation: v1 vs beta, publish chain, management vs app URL, inference split
- [REACT-VITE-GUIDE.md](./REACT-VITE-GUIDE.md) - Deep dive into Vite + React configuration
- [REACT-CICD.md](./REACT-CICD.md) - CI/CD setup with GitHub Actions
- [FRAMEWORKS.md](./FRAMEWORKS.md) - Streamlit, Dash, Flask configurations
- [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) - Common issues and solutions

## Quick Start

### React with Vite

```javascript
// vite.config.js
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [react()],
  base: './',  // CRITICAL for Domino proxy
  server: { host: '0.0.0.0', port: 8888, strictPort: true },
  preview: { host: '0.0.0.0', port: 8888, strictPort: true },
})
```

```bash
# app.sh
#!/bin/bash
set -e
cd /mnt/code
npm ci
npm run build
npx serve -s dist -l 8888 --no-clipboard
```

### Streamlit

```bash
# app.sh
#!/bin/bash
streamlit run app.py --server.port 8888 --server.address 0.0.0.0
```

### Dash/Flask

```bash
# app.sh
#!/bin/bash
python app.py  # Must bind to 0.0.0.0:8888
```

## Environment Variables

Domino provides these environment variables to your app:

| Variable | Description |
|----------|-------------|
| `DOMINO_PROJECT_NAME` | Current project name |
| `DOMINO_PROJECT_OWNER` | Project owner username |
| `DOMINO_RUN_ID` | Current run identifier |
| `DOMINO_STARTING_USERNAME` | User who started the app |

## Inter-App Communication

Domino apps can communicate with each other using bearer token authentication. This is useful for:
- Calling Model APIs from an app
- App-to-app service calls
- Accessing internal Domino services

### Getting the Access Token

Domino provides an access token service at `localhost:8899`:

```python
import requests

# Get bearer token for inter-app communication
API_TOKEN = requests.get("http://localhost:8899/access-token").text
```

### Making Authenticated Requests

```python
import requests

# 1. Get access token from Domino's token service
API_TOKEN = requests.get("http://localhost:8899/access-token").text

# 2. Set up headers with bearer token
headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

# 3. Make request to another Domino app or service
payload = {
    "query": "Your request data here"
}

try:
    resp = requests.post(
        "https://your-domino-instance/apps-internal/APP_ID/endpoint",
        json=payload,
        headers=headers,
        timeout=100
    )
    resp.raise_for_status()
    data = resp.json()
    print(data)
except requests.exceptions.RequestException as err:
    print("API call failed:", 

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 5 — file references are metadata, not package contents.

View saved version →

---
name: domino-data-connectivity
description: Connect Domino workloads to external data sources including AWS S3 (via Mountpoint CSI driver), credential propagation with AWS IRSA and Azure Entra ID, and External Data Volumes. Use when configuring S3 access, setting up credential propagation, or connecting to cloud data sources from Domino.
---

# Domino Data Connectivity Skill

This skill provides comprehensive knowledge for connecting Domino workloads to external data sources, including AWS S3, Azure storage, and credential propagation.

## Safety Rules

The setup guides in this skill are platform-administrator procedures that change cloud IAM and
Kubernetes cluster state. When using them:

- **Explain, don't execute.** Show `aws iam`, `eksctl`, `helm`, `kubectl`, `az` commands and
  IAM/RBAC policies for the user to review and run. Do not run them yourself unless the user
  explicitly asks you to run that specific command.
- **Least privilege.** Scope every policy to named buckets, endpoints, or resources. Never
  propose `"Resource": "*"` or wildcard actions such as `s3:*`.
- **No secrets in output.** Never print, log, or echo access keys, tokens, or client secrets, and
  never ask the user to paste one into the chat.
- **Read-only troubleshooting.** Prefer `get`/`describe` commands over `exec` into running pods.

## Key Concepts

### Data Access Options in Domino

| Option | Use Case |
|--------|----------|
| **Datasets** | Project-level data storage |
| **Data Sources** | External database connections |
| **External Data Volumes (EDV)** | Mount external storage as volumes |
| **S3 Mountpoint** | Direct S3 access as file system |
| **Credential Propagation** | Pass user identity to cloud services |

### Credential Propagation Methods

| Method | Cloud | Description |
|--------|-------|-------------|
| **IRSA** | AWS | IAM Role for Service Accounts via OIDC |
| **Azure Entra ID** | Azure | User-based credential propagation |

## Related Documentation

- [S3-MOUNTPOINT.md](./S3-MOUNTPOINT.md) - AWS S3 as local file system
- [AWS-IRSA.md](./AWS-IRSA.md) - AWS IAM Role for Service Accounts
- [AZURE-CREDENTIALS.md](./AZURE-CREDENTIALS.md) - Azure Entra ID integration

## Quick Start

### Accessing S3 Data

With Mountpoint S3 configured, access S3 as a local file system:

```python
import pandas as pd

# S3 data appears as local files
df = pd.read_parquet("/mnt/s3-data/datasets/sales.parquet")
```

### Using IRSA for AWS Services

With IRSA configured, AWS SDK uses automatic credentials:

```python
import boto3

# No explicit credentials needed - IRSA provides them
s3 = boto3.client('s3')
response = s3.list_objects_v2(Bucket='my-bucket')
```

### Accessing External Data Volumes

EDVs are mounted at configured paths:

```python
# Read from external volume
with open("/mnt/external-data/config.json") as f:
    config = json.load(f)
```

## When to Use Each Option

### Use S3 Mountpoint When:
- Working with large datasets stored in S3
- Need file system interface to S3
- Want to avoid EFS costs for large data
- Require multi-region data access

### Use IRSA When:
- Need AWS service access from Domino workloads
- Policy prohibits long-lived credentials
- Require user-level audit trails
- Need cross-account role assumption

### Use Azure Entra When:
- Working with Azure data services
- Need OAuth-based authentication
- Require user-level access control
- Need centralized RBAC

## Blueprint References

- [S3 Mountpoint Blueprint](https://domino.ai/resources/blueprints/transforming-s3-into-local-storage)
- [AWS IRSA Blueprint](https://domino.ai/resources/blueprints/aws-irsa)
- [Azure Credential Propagation Blueprint](https://domino.ai/resources/blueprints/domino-azure-credential-propagation)

## GitHub Repository

Implementation templates available at:
https://github.com/dominodatalab/domino-blueprints

Referenced files: 3 — file references are metadata, not package contents.

View saved version →

---
name: domino-data-sdk
description: Use the domino-data Python SDK (dominodatalab-data) for programmatic data access in Domino. Covers DataSourceClient for SQL queries and object storage, DatasetClient for dataset files, TrainingSets for ML data versioning, Feature Store, and VectorDB (Pinecone) integration. Use when querying data sources, downloading datasets, managing training sets, or working with vector databases in Domino.
---

# Domino Data SDK Skill

This skill provides comprehensive knowledge for working with the `domino-data` Python SDK (`dominodatalab-data`) - the official library for Domino's Access Data features.

## Installation

```bash
# Via pip
pip install -U dominodatalab-data

# Via Poetry
poetry add dominodatalab-data

# In Domino environment (requirements.txt)
dominodatalab-data>=6.0.0
```

## Key Components

| Module | Purpose |
|--------|---------|
| `DataSourceClient` | Query SQL databases and access object stores |
| `DatasetClient` | Read files from Domino Datasets |
| `TrainingSets` | Version and manage ML training data |
| `Feature Store` | Manage ML features with Git integration |
| `VectorDB` | Pinecone vector database integration |

## Related Documentation

- [DATA-SOURCES.md](./DATA-SOURCES.md) - SQL queries and object storage
- [DATASETS.md](./DATASETS.md) - Dataset file operations
- [TRAINING-SETS.md](./TRAINING-SETS.md) - Training data versioning
- [VECTORDB.md](./VECTORDB.md) - Pinecone integration

## Quick Start

### Query a Data Source

```python
from domino_data.data_sources import DataSourceClient

# Initialize client (auto-configured in Domino)
client = DataSourceClient()

# Get a data source by name
ds = client.get_datasource("my-redshift-db")

# Execute SQL query
result = ds.query("SELECT * FROM customers WHERE region = 'US'")

# Convert to pandas DataFrame
df = result.to_pandas()

# Or save to parquet
result.to_parquet("output.parquet")
```

### Access Object Storage

```python
from domino_data.data_sources import DataSourceClient

client = DataSourceClient()
ds = client.get_datasource("my-s3-bucket")

# List objects
objects = ds.list_objects(prefix="data/", page_size=100)

# Download a file
ds.download_file("data/input.csv", "local_input.csv")

# Upload a file
ds.put("data/output.csv", open("results.csv", "rb").read())

# Get signed URL
url = ds.get_key_url("data/file.csv", is_read_write=False)
```

### Read from Datasets

```python
from domino_data.datasets import DatasetClient

client = DatasetClient()

# Get dataset by name
dataset = client.get_dataset("training-data")

# List files
files = dataset.list_files(prefix="images/")

# Download file
dataset.download("model.pkl", "local_model.pkl", max_workers=4)

# Get file content as bytes
content = dataset.get("config.json")
```

### Training Sets

```python
from domino_data.training_sets import (
    create_training_set_version,
    get_training_set,
    list_training_sets
)
import pandas as pd

# Create training set version from DataFrame
df = pd.DataFrame({
    "id": [1, 2, 3],
    "feature_a": [0.1, 0.2, 0.3],
    "label": [1, 0, 1]
})

version = create_training_set_version(
    training_set_name="customer-churn",
    df=df,
    key_columns=["id"],
    description="Initial training data"
)

# Get training set
ts = get_training_set("customer-churn")

# List all training sets
all_sets = list_training_sets()
```

### Vector Database (Pinecone)

```python
from domino_data.vectordb import (
    domino_pinecone3x_init_params,
    domino_pinecone3x_index_params
)
from pinecone import Pinecone

# Initialize Pinecone client with Domino credentials
init_params = domino_pinecone3x_init_params("my-pinecone-ds")
pc = Pinecone(**init_params)

# Get index parameters
index_params = domino_pinecone3x_index_params("my-pinecone-ds", "embeddings")
index = pc.Index(**index_params)

# Query vectors
results = index.query(
    vector=[0.1, 0.2, 0.3, ...],
    top_k=10,
    include_metadata=True
)
```

## Authentication

The library auto-configures a

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 4 — file references are metadata, not package contents.

View saved version →

---
name: domino-datasets
description: Work with Domino Datasets - high-performance, versioned filesystem storage. Covers dataset creation, snapshots for versioning, sharing across projects, mounting paths (/domino/datasets/), and performance optimization. Use when managing data storage, creating reproducible data versions, or sharing data between projects.
---

# Domino Datasets Skill

## Description
This skill helps users work with Domino Datasets - high-performance, versioned filesystem storage for data science projects.

## Activation
Activate this skill when users want to:
- Create or manage Domino Datasets
- Work with dataset snapshots and versioning
- Share data between projects
- Access large datasets efficiently
- Understand dataset paths and mounting

## What is a Domino Dataset?

A Domino Dataset is:
- **High-performance storage**: Network filesystem optimized for data science
- **Versioned**: Create snapshots for reproducibility
- **Shareable**: Access across projects
- **Scalable**: No file size or count limits
- **Persistent**: Data persists across executions

## Creating a Dataset

### Via Domino UI
1. Navigate to your project
2. Go to **Data** > **Domino Datasets**
3. Click **Create New Dataset**
4. Enter:
   - **Name**: Dataset name (e.g., `training-data`)
   - **Description**: What the dataset contains
5. Click **Create**

### Via Python SDK
```python
from domino import Domino

domino = Domino("project-owner/project-name")

# Create a new dataset
dataset = domino.datasets_create(
    name="training-data",
    description="Training data for classification model"
)
```

## Dataset Paths

Dataset paths differ based on your **project type**. Domino has two project types with different mount structures.

### DFS (Domino File System) Projects

DFS projects use `/domino` as the root:

```
/domino
   |--/datasets
      |--/local               <== Local datasets and snapshots
         |--/clapton          <== Read-write dataset for owner and editor, read-only for reader
         |--/mingus           <== Read-write dataset for owner and editor, read-only for reader
         |--/snapshots        <== Snapshot folder organized by dataset
            |--/clapton       <== Read-write for owner and editor, read-only for reader
               |--/tag1          <== Mounted under latest tag
               |--/1             <== Always mounted under the snapshot number
               |--/2
            |--/mingus
               |--/tag2
               |--/1
               |--/2
      |--/ella                <== Read-write shared dataset for owner and editor, Read-only for reader
      |--/davis               <== Read-write shared dataset for owner and editor, Read-only for reader
      |--/snapshots           <== Shared datasets snapshots organized by dataset
         |--/ella             <== Read-write for owner and editor, read-only for reader
            |--/tag3          <== Mounted under latest tag
            |--/1             <== Always mounted under the snapshot number
            |--/2
         |--/davis
            |--/tag4
            |--/1
            |--/2
```

| Dataset Type | Path |
|-------------|------|
| Local datasets | `/domino/datasets/local/{dataset-name}/` |
| Local snapshots | `/domino/datasets/local/snapshots/{dataset-name}/{tag-or-number}/` |
| Shared datasets | `/domino/datasets/{dataset-name}/` |
| Shared snapshots | `/domino/datasets/snapshots/{dataset-name}/{tag-or-number}/` |

### Git-Based Projects

Git-based projects use `/mnt` as the root:

```
/mnt
   |--/data                  <== Local datasets and snapshots
     |--/clapton             <== Read-write dataset for owner and editor, read-only for reader
     |--/mingus              <== Read-write dataset for owner and editor, read-only for reader
     |--/snapshots           <== Snapshot folder organized by dataset
        |--/clapton          <== Read-write for owner and editor, read-only for reader
           |--/tag1          <== Mounted under 

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-debug-proxy
description: Diagnose and fix Domino reverse-proxy and routing problems in published web apps by checking vite.config.js, package.json, and app.sh for wrong ports, localhost binding, absolute base paths, missing SPA fallback, or missing build steps. Use when a Domino app shows connection refused, 404s on CSS/JS assets, a blank page, or broken deep links.
---

# Debug Domino App Proxy Issues

Find the configuration mistakes that break apps behind Domino's reverse proxy, report them, and
fix them with the user's agreement.

## Steps

1. Locate the app's config files: `vite.config.js`/`vite.config.ts`, `package.json`, `app.sh`,
   and any framework entry point (`app.py`, `server.js`).
2. Run every check below and record each result as pass or fail.
3. Print a report in the format shown under "Report format".
4. List the recommended fixes and ask whether to apply them. Apply only what the user approves.
5. For anything you can't fix from config, give the manual investigation steps.

## Checks

| # | Check | Pass condition |
|---|---|---|
| 1 | Port | The server listens on `8888`; Vite also sets `strictPort: true` |
| 2 | Base path | Vite sets `base: './'`; other frameworks use relative asset paths |
| 3 | Host binding | The server binds to `0.0.0.0`, not `localhost` or `127.0.0.1` |
| 4 | `app.sh` | It `cd`s to the code directory, runs the build, and then serves the output |
| 5 | SPA fallback | Static serving uses `npx serve -s dist` (the `-s` flag) |
| 6 | Asset references | No hardcoded root-absolute paths such as `/assets/...` or `fetch('/api/...')` |

## Common issues

| Symptom | Cause | Fix |
|---|---|---|
| Connection refused | Wrong port, for example 3000 | `server: { port: 8888, strictPort: true }` |
| 404 on CSS/JS files | `base` is `/` or missing | `base: './'` in `defineConfig` |
| App unreachable from the browser | Bound to localhost | `server: { host: '0.0.0.0' }` |
| Deep links return 404 | `serve` without `-s` | `npx serve -s dist -l 8888` |
| Blank page or stale content | `app.sh` has no build step | Add `npm run build` before `serve` |

A corrected `app.sh` for a Vite app:

```bash
#!/bin/bash
set -e
cd /mnt/code
npm ci
npm run build
npx serve -s dist -l 8888
```

## Report format

```
Domino Proxy Debug Report
=========================

Checking: vite.config.js
✅ Port: 8888
✅ Base path: './'
✅ Host: 0.0.0.0

Checking: app.sh
✅ Working directory: /mnt/code
✅ Build command: npm run build
❌ Serve command missing -s flag

Summary: 1 issue found

Recommended fixes:
1. Update the app.sh serve command:
   - Current: npx serve dist -l 8888
   + Fixed:   npx serve -s dist -l 8888
```

## Manual investigation

If the config is correct but the app still fails:

1. Open the browser developer tools console and look for failed module or asset loads.
2. In the Network tab, check whether failing requests drop the app's proxy URL prefix (the part
   of the path before the app's own routes).
3. Check the app logs on the app's page in Domino.
4. Confirm the app was republished after the last code change.

For general deployment guidance, use the `domino-apps` skill.

View saved version →

---
name: domino-distributed-computing
description: Work with distributed computing frameworks in Domino including Apache Spark, Ray, and Dask clusters. Covers cluster configuration, on-demand clusters, choosing between frameworks, PySpark usage, and scaling workloads. Use when processing large datasets, parallel ML training, or running distributed compute jobs.
---

# Domino Distributed Computing Skill

## Description
This skill helps users work with distributed computing frameworks in Domino - Spark, Ray, and Dask clusters for scaling compute-intensive workloads.

## Activation
Activate this skill when users want to:
- Run Spark, Ray, or Dask clusters in Domino
- Scale data processing or ML training
- Configure distributed cluster settings
- Understand when to use each framework

## Supported Frameworks

| Framework | Best For |
|-----------|----------|
| **Apache Spark** | Large-scale data processing, SQL, ETL |
| **Ray** | Distributed ML, hyperparameter tuning, RL |
| **Dask** | Parallel pandas, NumPy at scale |
| **MPI** | Scientific computing, HPC workloads |

## When to Use Each Framework

### Spark
- Processing terabyte-scale data
- SQL analytics on big data
- ETL pipelines
- Structured data processing

### Ray
- Distributed model training
- Hyperparameter optimization
- Reinforcement learning
- Generic Python parallelization

### Dask
- Scaling pandas workflows
- Parallel NumPy operations
- Lazy evaluation needed
- Familiar pandas/NumPy API preferred

## Launching On-Demand Clusters

### Via Domino UI
1. Start a workspace or job
2. Check **Attach compute cluster**
3. Select:
   - **Cluster Type**: Spark, Ray, or Dask
   - **Worker Count**: Number of workers
   - **Hardware Tier**: Resources per worker
   - **Auto-scaling**: Enable/disable
4. Launch

### Via Python SDK
```python
from domino import Domino

domino = Domino("project-owner/project-name")

# Start workspace with Spark cluster
workspace = domino.workspace_start(
    hardware_tier_name="medium",
    cluster_config={
        "clusterType": "Spark",
        "workerCount": 4,
        "workerHardwareTier": "medium",
        "masterHardwareTier": "medium"
    }
)
```

## Apache Spark

### Connecting to Spark
```python
from pyspark.sql import SparkSession

# Domino auto-configures Spark
spark = SparkSession.builder.getOrCreate()

# Check configuration
print(f"Spark version: {spark.version}")
print(f"Executors: {spark.sparkContext.defaultParallelism}")
```

### Reading Data
```python
# Read CSV
df = spark.read.csv("/mnt/data/dataset/data.csv", header=True, inferSchema=True)

# Read Parquet
df = spark.read.parquet("/mnt/data/dataset/")

# Read from database
df = spark.read.jdbc(
    url="jdbc:postgresql://host:5432/db",
    table="schema.table",
    properties={"user": "user", "password": "pass"}
)
```

### Processing Data
```python
from pyspark.sql import functions as F

# Transformations
result = df.filter(F.col("value") > 100) \
    .groupBy("category") \
    .agg(F.mean("value").alias("avg_value")) \
    .orderBy("avg_value", ascending=False)

# Show results
result.show()
```

### Machine Learning with Spark MLlib
```python
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml import Pipeline

# Prepare features
assembler = VectorAssembler(
    inputCols=["feature1", "feature2", "feature3"],
    outputCol="features"
)

# Create model
rf = RandomForestClassifier(
    featuresCol="features",
    labelCol="label",
    numTrees=100
)

# Build pipeline
pipeline = Pipeline(stages=[assembler, rf])
model = pipeline.fit(train_df)
predictions = model.transform(test_df)
```

### Writing Results
```python
# Write Parquet (recommended)
result.write.parquet("/mnt/artifacts/output/", mode="overwrite")

# Write CSV
result.write.csv("/mnt/artifacts/output.csv", header=True)
```

## Ray

### Connecting to Ray
```python
import ray

# Domino auto-initializes Ray
# Or manually connect
ray.init(address="auto")

prin

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-environments
description: Create and customize Domino Compute Environments - Docker containers defining tools, packages, and configurations. Covers Dockerfile customization, package installation, IDE configuration, DSE (Domino Standard Environments), and troubleshooting build failures. Use when installing dependencies, customizing environments, or fixing environment issues.
---

# Domino Compute Environments Skill

## Description
This skill helps users create, customize, and manage Domino Compute Environments - Docker-based containers that define the tools, packages, and configurations for workspaces, jobs, and other executions.

## Activation
Activate this skill when users want to:
- Create or customize a compute environment
- Install packages or dependencies
- Configure Dockerfile instructions
- Understand environment best practices
- Troubleshoot environment build issues

## What is a Compute Environment?

A Domino Compute Environment is a Docker container image that contains:
- Operating system and base tools
- Programming languages (Python, R)
- IDEs (Jupyter, VS Code, RStudio)
- Libraries and packages
- Custom configurations

## Domino Standard Environments (DSEs)

Domino provides pre-built environments with common tools:

| Environment | Includes |
|-------------|----------|
| Domino Standard Environment | Python 3.9, R 4.1, Jupyter, VS Code |
| Domino Spark Environment | Spark 3.x, PySpark |
| Domino Ray Environment | Ray for distributed computing |
| Domino GPU Environment | CUDA, cuDNN, GPU libraries |

## Creating a Custom Environment

### Via Domino UI
1. Go to **Environments** in Domino
2. Click **Create Environment**
3. Configure:
   - **Name**: Descriptive name
   - **Base Image**: Start from DSE or custom image
   - **Dockerfile Instructions**: Add customizations
4. Click **Build**

### Dockerfile Instructions

Add instructions to customize the environment. Do NOT include `FROM` statement.

```dockerfile
# Install system packages
RUN apt-get update && apt-get install -y \
    libpq-dev \
    graphviz \
    && rm -rf /var/lib/apt/lists/*

# Install Python packages
RUN pip install --no-cache-dir \
    pandas==2.0.0 \
    scikit-learn==1.3.0 \
    tensorflow==2.13.0

# Install R packages
RUN R -e "install.packages(c('tidyverse', 'caret'), repos='https://cloud.r-project.org')"

# Set environment variables
ENV MODEL_PATH=/mnt/artifacts/model.pkl
```

## Package Installation Methods

### Method 1: Dockerfile Instructions (Recommended)
Best for packages that should always be available.

```dockerfile
RUN pip install pandas numpy scikit-learn
```

**Pros**: Fast startup, consistent environment
**Cons**: Requires environment rebuild for changes

### Method 2: requirements.txt
Packages installed at execution startup.

```text
# requirements.txt in project root
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
```

**Pros**: No rebuild needed, per-project customization
**Cons**: Slower startup

### Method 3: Pre/Post-run Scripts
Custom scripts that run at execution start.

```bash
#!/bin/bash
# pre-run.sh
pip install -q custom-package
```

### Method 4: Runtime Installation
Install during execution (temporary).

```python
!pip install package-name
```

## Dockerfile Best Practices

### Combine RUN Commands
```dockerfile
# Good: Single layer
RUN pip install pandas numpy scikit-learn matplotlib

# Bad: Multiple layers
RUN pip install pandas
RUN pip install numpy
RUN pip install scikit-learn
```

### Clean Up
```dockerfile
RUN apt-get update && apt-get install -y \
    package1 \
    package2 \
    && rm -rf /var/lib/apt/lists/*
```

### Pin Versions
```dockerfile
# Good: Reproducible
RUN pip install pandas==2.0.0 scikit-learn==1.3.0

# Bad: May break
RUN pip install pandas scikit-learn
```

### Use --no-cache-dir
```dockerfile
RUN pip install --no-cache-dir package-name
```

## Environment Variables

### Set in Dockerfile
```dockerfile
ENV MY_VAR=value
ENV DATA_PATH=/mnt/data
```

### Set in Domino UI
1. Go t

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-experiment-setup
description: Set up MLflow experiment tracking in a Domino project by generating an experiment_setup.py helper with deployment-unique experiment names, Domino context tags, and framework auto-logging, plus an example training script. Use when the user asks to add or set up experiment tracking, MLflow logging, or autologging for a traditional ML project in Domino.
compatibility: Domino 6.3 and Domino Cloud. Runs in a Domino workspace or job, where MLflow tracking is preconfigured.
---

# Set Up Domino Experiment Tracking

Applies to Domino 6.3 and Domino Cloud.

Add MLflow experiment tracking to a traditional ML project. For LLM or agent tracing, use the
`domino-trace-setup` skill instead.

## Inputs

1. **Experiment base name**: use the one the user gave; otherwise use the project directory name.
2. **ML frameworks**: detect them from `requirements.txt`, `pyproject.toml`, `environment.yml`,
   or imports in the code.

Do not stop to ask for either. Make the changes with these defaults, then state the base name
and frameworks you used in the reply so the user can change them.

## Steps

1. Copy `assets/experiment_setup.py` from this skill's folder into the project. It provides:
   - `setup_experiment(base_name)`: sets an experiment named `<base>-<project>-<username>`.
   - `log_domino_context()`: tags the run with the Domino user, project, run ID, and hardware tier.
   - `setup_autolog()`: enables MLflow autologging for whichever supported frameworks are installed.
2. Report which frameworks were detected and will be autologged, for example:

   ```
   Detected frameworks:
   ✅ scikit-learn (requirements.txt)
   ✅ xgboost (requirements.txt)
   ❌ tensorflow (not found)
   ```

3. If the project trains large models, add the multipart-upload settings below.
4. Create an example training script, or wire the helpers into the user's existing script.

## Example training script

```python
# train.py
from experiment_setup import setup_experiment, log_domino_context, setup_autolog
import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

experiment_name = setup_experiment("iris-classifier")
setup_autolog()

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

with mlflow.start_run(run_name="random-forest-v1"):
    log_domino_context()
    mlflow.log_param("custom_param", "value")

    model = RandomForestClassifier(n_estimators=100, max_depth=5)
    model.fit(X_train, y_train)

    test_accuracy = model.score(X_test, y_test)
    mlflow.log_metric("test_accuracy", test_accuracy)
    print(f"Test accuracy: {test_accuracy:.4f}")
    print(f"Run ID: {mlflow.active_run().info.run_id}")
```

## Important rules

- **Experiment names must be unique across the entire Domino deployment**, not just the project.
  Always keep the username and project suffix that `setup_experiment` adds.
- For large artifacts (LLMs, deep learning checkpoints), enable multipart upload:

  ```python
  os.environ['MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD'] = "true"
  os.environ['MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE'] = "104857600"  # 100MB
  ```

For run comparison, the model registry, and deeper MLflow usage, use the
`domino-experiment-tracking` skill.

Referenced files: 1 — file references are metadata, not package contents.

View saved version →

---
name: domino-experiment-tracking
description: Track traditional ML experiments in Domino using the MLflow-based Experiment Manager. Covers experiment setup, auto-logging for sklearn/TensorFlow/PyTorch, manual logging, artifact storage, run comparison, and model registration. Use when training ML models, logging metrics and parameters, comparing model runs, or registering models.
---

# Domino Experiment Tracking Skill

This skill provides comprehensive knowledge for tracking ML experiments in Domino Data Lab using the built-in MLflow-based Experiment Manager.

## Key Concepts

### Experiment Manager Overview

Domino's Experiment Manager is built on MLflow and provides:
- Automatic and manual logging of parameters, metrics, and artifacts
- Run comparison and visualization
- Model versioning and registry
- Integration with Domino projects and jobs

### Critical Configuration

**Experiment names must be unique across the entire Domino deployment.** Always append username or project name to ensure uniqueness.

## Related Documentation

- [MLFLOW-BASICS.md](./MLFLOW-BASICS.md) - Auto-logging, manual logging
- [COMPARING-RUNS.md](./COMPARING-RUNS.md) - Run comparison, export
- [MODEL-REGISTRY.md](./MODEL-REGISTRY.md) - Model registration & stages

## Quick Start

```python
import mlflow
import os

# CRITICAL: Experiment names must be unique across Domino deployment
username = os.environ.get('DOMINO_STARTING_USERNAME', 'unknown')
experiment_name = f"my-experiment-{username}"

# Set the experiment
mlflow.set_experiment(experiment_name)

# Enable auto-logging (easiest approach)
mlflow.autolog()

# Run training
with mlflow.start_run(run_name="my-first-run"):
    model.fit(X_train, y_train)

    # Optional: manually log additional items
    mlflow.log_param("custom_param", "value")
    mlflow.log_metric("custom_metric", 0.95)
```

## Supported Frameworks

| Framework | Auto-log Command |
|-----------|------------------|
| Scikit-learn | `mlflow.sklearn.autolog()` |
| TensorFlow/Keras | `mlflow.tensorflow.autolog()` |
| PyTorch | `mlflow.pytorch.autolog()` |
| XGBoost | `mlflow.xgboost.autolog()` |
| LightGBM | `mlflow.lightgbm.autolog()` |
| All at once | `mlflow.autolog()` |

## Environment Variables

Domino automatically configures MLflow to use the built-in tracking server. These variables are pre-set:

| Variable | Description |
|----------|-------------|
| `MLFLOW_TRACKING_URI` | Domino's MLflow server URL |
| `DOMINO_STARTING_USERNAME` | User running the experiment |
| `DOMINO_PROJECT_NAME` | Current project name |
| `DOMINO_RUN_ID` | Domino job run ID |

## Documentation Links

- Domino Experiment Tracking: https://docs.dominodatalab.com/en/latest/user_guide/da707d/track-and-monitor-experiments/
- Domino Model Registry: https://docs.dominodatalab.com/en/latest/user_guide/3b6ae5/manage-models-with-model-registry/

Referenced files: 3 — file references are metadata, not package contents.

View saved version →

---
name: domino-extensions
description: Build and operate Domino UI Extensions (mount Domino Apps in the shell with page context). Covers extension_manifest.json, official installer vs manual Extension API, publish and identity requirements, auth and host splits, and beta REST routes. Use when creating an Extension, wiring mount points, publishing an Extension App from a project, or automating Extension install lifecycle. Not for generic App deployment alone (see domino-apps) or SPA scaffolding (see domino-ui-bootstrap).
---

# Domino Extensions

An **Extension** surfaces a published **App** at a Domino UI **mount point** (project sidebar, dataset page, model details, file context menus, admin panel). Domino opens the App in a new tab and passes page context as query parameters. The App must support **extended identity propagation** so it can act as the viewing user with consent.

## When to use other skills

| Goal | Skill |
|------|--------|
| Vite/React design system, `@dominodatalab/extensions-tools`, proxy-safe URLs | [domino-ui-bootstrap](../domino-ui-bootstrap/SKILL.md) |
| `app.sh`, ports, SPA base path, generic App CI | [domino-apps](../apps/SKILL.md) |
| REST auth, jobs, projects, platform `/api/*` and `/v4/*` | [python-sdk](../python-sdk/SKILL.md) |
| Governance from an extension backend | [domino-governance](../domino-governance/SKILL.md) plus external URL rules below |

Product overview: https://docs.domino.ai/cloud/platform-capabilities/features/extensions

Official catalog install (SysAdmin/CloudAdmin): https://docs.domino.ai/cloud/platform-capabilities/features/extensions/install-domino-official-extensions

Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . For HTTP client setup in Extension backends and install scripts, use [python-sdk/SKILL.md](../python-sdk/SKILL.md#authentication).

## Extension manifest (`extension_manifest.json`)

Domino Official Extensions and custom Extension repos ship a manifest at the repo root. Schema version is **`manifestSchemaVersion`: 1** (integer). The official installer reads this file from a GitHub release; manual flows still mirror the same shape when you create entities yourself.

Top-level sections:

| Section | Purpose |
|---------|---------|
| `project` | Hosting project (`name`, `description`, `visibility`, `isRestricted`, ...) |
| `app` | Compute environment, entry point, vanity URL, visibility, mounts, hardware tier |
| `extension` | Extension record: `name`, `description`, `enabled`, `uiMountPointTypeConfigs` |

For full manifest examples and patterns (compute environment, mount configs, deep linking, autoscaling), refer to Domino's official Extension repos:

- https://github.com/dominodatalab/AutoML_Extension
- https://github.com/dominodatalab/clinical-data-explorer
- https://github.com/dominodatalab/AutoDocumentation_Extension

Each ships an `extension_manifest.json` at the repo root; read the file matching your Domino version rather than copying from this skill.

### Mount points (`uiMountPointTypeConfigs`)

Keys match the Extensions API enum: `projectSidebar`, `dataset`, `datasetFileContext`, `modelDetails`, `adminPanel`, `netAppVolume`, `netAppVolumeFileContext`. Each mount defines `enabled`, scope (for example `allProjects` or project lists), and `urlConfig` with `contextualQueryParams` Domino passes to the App.

Domino documents five primary user-facing mount types on the product page; the API includes NetApp-related mount types for volume integrations.

### Deep linking vs iframe (`renderIFrame`)

Apps can run full-page (deep linking) or inside an iframe. Platform convention: **`renderIFrame = !deepLinkingEnabled`**. Manifest field **`enableDeepLinking: true`** means the App expects full-page navigation; set `false` when the UI should load in an iframe. Align manifest, App publish settings, and start overrides so you do not mix modes silently.

## Publish from a project (custom Extension)

Typical human workflow:

1. **Bui

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-flows
description: Orchestrate multi-step ML workflows using Domino Flows (built on Flyte). Define DAGs with typed inputs/outputs, heterogeneous environments, automatic lineage, and reproducibility. Use when building data pipelines, multi-stage training workflows, or processes requiring orchestration and monitoring.
---

# Domino Flows Skill

This skill provides comprehensive knowledge for orchestrating ML workflows using Domino Flows, built on the Flyte platform.

## Key Concepts

### What are Domino Flows?

Domino Flows enable:
- **DAG-based orchestration**: Define workflows as directed acyclic graphs
- **Typed interfaces**: Strong typing for inputs and outputs
- **Heterogeneous environments**: Different environments per task
- **Automatic lineage**: Track data and model provenance
- **Reproducibility**: Version-controlled workflows
- **Scalability**: Distributed execution across compute resources

### Core Components

| Component | Description |
|-----------|-------------|
| **Task** | Single unit of work (runs as a Domino Job) |
| **Workflow** | DAG connecting tasks |
| **Artifact** | Typed input/output passed between tasks |
| **Launch Plan** | Configured workflow execution |

## Related Documentation

- [FLOW-BASICS.md](./FLOW-BASICS.md) - DAG concepts, task definitions
- [EXAMPLES.md](./EXAMPLES.md) - Common flow patterns

## Quick Start

> ⚠️ **Critical**: Domino Flows does NOT support native Flyte `@task` decorators.
> Tasks must use `DominoJobTask` + `DominoJobConfig`. Only `@workflow` is unchanged.

### Basic Flow

Each task runs as a Domino Job. Stage scripts read from `/workflow/inputs/<name>`
and write to `/workflow/outputs/o0`. Pass `PYTHONPATH=/mnt/code` in the command.

```python
from flytekit import workflow
from flytekitplugins.domino.task import DominoJobConfig, DominoJobTask

preprocess_task = DominoJobTask(
    name="Preprocess Data",
    domino_job_config=DominoJobConfig(
        Command="bash -c 'PYTHONPATH=/mnt/code python /mnt/code/stages/preprocess.py'",
    ),
    inputs={"input_path": str},
    outputs={"o0": str},
    use_latest=True,
)

train_task = DominoJobTask(
    name="Train Model",
    domino_job_config=DominoJobConfig(
        Command="bash -c 'PYTHONPATH=/mnt/code python /mnt/code/stages/train.py'",
    ),
    inputs={"preprocess_output": str},
    outputs={"o0": str},
    use_latest=True,
)

@workflow
def training_pipeline(input_path: str = "/mnt/data/raw.csv") -> str:
    preprocess_output = preprocess_task(input_path=input_path)
    result = train_task(preprocess_output=preprocess_output)
    return result
```

### Stage Script Pattern

```python
# stages/preprocess.py
import json, os

INPUTS, OUTPUTS = "/workflow/inputs", "/workflow/outputs"

def main():
    input_path = open(f"{INPUTS}/input_path").read().strip()
    # ... do work ...
    os.makedirs(OUTPUTS, exist_ok=True)
    with open(f"{OUTPUTS}/o0", "w") as f:
        f.write(json.dumps({"output_path": "/mnt/artifacts/processed.parquet"}))

if __name__ == "__main__":
    main()
```

### Running the Flow

```bash
# Always commit and push first — jobs run against remote repo state
git add -A && git commit -m "..." && git push

# Trigger remotely
PYTHONPATH=/mnt/code pyflyte run --remote \
    my_flow.py training_pipeline \
    --input_path "/mnt/data/raw.csv"
```

## When to Use Flows

### Good Use Cases

- Data processing → Model training pipelines
- ETL with ML steps
- Multi-stage training with different environments
- Processes requiring reproducibility and lineage
- Scheduled/triggered workflows

### Not Ideal For

- Single dataset with many small computations
- Tasks that write to mutable shared state
- Simple single-step processes
- Real-time inference (use Model APIs instead)

## Documentation Links

- Domino Flows: https://docs.dominodatalab.com/en/latest/user_guide/78acf5/orchestrate-with-flows/
- Flyte Documentation: https://docs.flyte.org/

Referenced files: 2 — file references are metadata, not package contents.

View saved version →

---
name: domino-genai-tracing
description: Trace and evaluate GenAI applications including LLM calls, agents, RAG pipelines, and multi-step AI systems in Domino. Uses the Domino SDK (@add_tracing decorator, DominoRun context) with MLflow 3.2.0. Captures token usage, latency, cost, tool calls, and errors. Supports LLM-as-judge evaluators and custom metrics. Use when building agents, debugging LLM applications, or needing audit trails for GenAI systems.
---

# Domino GenAI Tracing Skill

This skill provides comprehensive knowledge for tracing and evaluating GenAI applications in Domino Data Lab, including LLM calls, agents, RAG pipelines, and multi-step AI systems.

## Two Deployment Modes

GenAI tracing works differently depending on where your code runs:

| Mode | Where traces appear | When to use |
|------|-------------------|-------------|
| **Deployed App** (Production) | App **Performance tab** | FastAPI/Flask apps deployed as Domino Apps |
| **Development / Evaluation** | **Experiments** UI | Batch scripts, Domino Jobs, Workspaces |

**Critical difference:** In a deployed Domino App, Domino auto-creates an experiment named `agent_experiment_{app_id}` and the Performance tab reads from it. If you call `mlflow.set_experiment()` or wrap calls in `DominoRun()`, traces go to your custom experiment instead — and the Performance tab won't see them.

## Key Concepts

### What GenAI Tracing Captures

The Domino SDK automatically captures:
- **Token usage** - Input and output tokens per call
- **Latency** - Time for each operation
- **Cost** - Estimated cost per call
- **Tool calls** - Function/tool invocations
- **Errors** - Exceptions and failure modes
- **Model parameters** - Temperature, max_tokens, etc.

### Core Components

1. **`@add_tracing` decorator** - Wraps agent functions to capture traces (works standalone — no `DominoRun` required)
2. **`mlflow.start_span()`** - Creates child spans for LLM calls and tool executions inside the agent loop
3. **`DominoRun` context manager** - Groups traces into runs for development/evaluation (Experiments UI only)
4. **Evaluators** - Custom functions to score outputs
5. **MLflow integration** - View traces in Experiment Manager or App Performance tab

## Related Documentation

- [TRACING-SETUP.md](./TRACING-SETUP.md) - Environment & SDK setup
- [ADD-TRACING-DECORATOR.md](./ADD-TRACING-DECORATOR.md) - @add_tracing usage, span_type, autolog_frameworks
- [DOMINO-RUN.md](./DOMINO-RUN.md) - DominoRun context manager (development/evaluation only)
- [EVALUATORS.md](./EVALUATORS.md) - LLM-as-judge, custom evaluators
- [MULTI-AGENT-EXAMPLE.md](./MULTI-AGENT-EXAMPLE.md) - Complete multi-agent example

## Quick Start — Deployed App (Production)

For a FastAPI app deployed as a Domino App. Traces appear in the **App Performance tab**.

```python
# main.py — lifespan: autolog only, NO set_experiment, NO DominoRun
import mlflow
from fastapi import FastAPI
from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Each autolog in its own try/except so one failure doesn't block the other
    try:
        mlflow.openai.autolog()
    except Exception:
        pass
    try:
        mlflow.anthropic.autolog()
    except Exception:
        pass
    yield

app = FastAPI(lifespan=lifespan)
```

```python
# orchestrator.py — @add_tracing on the core agent function
import mlflow
from domino.agents.tracing import add_tracing

@add_tracing(
    name="agent_turn",
    span_type="AGENT",
    autolog_frameworks=["openai", "anthropic"],
)
async def run_agent(messages: list[dict]) -> dict:
    # LLM calls and tool calls go here (see ADD-TRACING-DECORATOR.md)
    ...

# router.py — call the traced function directly, NO DominoRun wrapper
@app.post("/chat")
async def chat(request: ChatRequest):
    return await run_agent(request.messages)
```

## Quick Start — Development / Evaluation

For batch scripts, Domino Jobs, or Workspaces. Traces appear in the **Experiments UI**.

`

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 5 — file references are metadata, not package contents.

View saved version →

---
name: domino-governance
description: Manage model risk governance in Domino using policies, bundles, and evidence. Covers creating governance bundles, attaching model artifacts and MLflow results as evidence, progressing through policy stages, and documenting findings. Use when the user mentions governance, compliance, bundles, policies, model risk management, SR 11-7, NIST AI RMF, or audit trails.
---

# Domino Governance Skill

This skill provides knowledge for managing model risk governance in Domino Data Lab using the Governance API.

## Configuration

Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . **Do not use API keys** (`X-Domino-Api-Key`, `DOMINO_USER_API_KEY`).

Governance REST lives at `/api/governance/v1/*`. Pick the base where `GET $BASE/policy-overviews` returns **HTTP 200** (try the table rows in order):

| Context | Base | Authorization |
|---------|------|----------------|
| In-run, `DOMINO_API_PROXY` set (5.4.0+ JWT credential propagation) | `{DOMINO_API_PROXY}/api/governance/v1` | None (proxy adds starting-user JWT) |
| In-run, no proxy, governance on platform API gateway | `{DOMINO_USER_HOST or DOMINO_API_HOST}/api/governance/v1` | Bearer from `GET http://localhost:8899/access-token` |
| In-run but proxy or sidecar returns **404** on `/api/governance/v1/*` (gateway off or governance not registered on 8763) | `{public deployment URL}/api/governance/v1` | Bearer PAT or SA token |
| Outside any run | `{public deployment URL}/api/governance/v1` | Bearer PAT or SA token |

```bash
GOV="/api/governance/v1"
PREFIX="/policy-overviews"

# 1) In-run, proxy (try first when DOMINO_API_PROXY is set)
BASE="$DOMINO_API_PROXY$GOV"
curl -s "$BASE$PREFIX"

# 2) In-run, platform gateway (no proxy)
HOST="${DOMINO_USER_HOST:-$DOMINO_API_HOST}"
TOKEN="$(curl -s http://localhost:8899/access-token)"
BASE="$HOST$GOV"
curl -s "$BASE$PREFIX" -H "Authorization: Bearer $TOKEN"

# 3) Public deployment (outside run, or 404 on 1 and 2)
BASE="https://your-deployment.domino.tech$GOV"
TOKEN="$PAT_OR_SA_TOKEN"
curl -s "$BASE$PREFIX" -H "Authorization: Bearer $TOKEN"
```

Do **not** scrape the JWT `iss` claim to derive the cluster URL. Prefer `DOMINO_API_PROXY` in-run when step 1 succeeds; otherwise step 2 or 3.

**Takeaways for agents**

1. **Proxy + no header is correct only when routing works.** `DOMINO_API_PROXY` injects JWT ([product auth doc](https://docs.domino.ai/cloud/reference/api/domino-api-authentication)); it does not add `/api/governance/v1` if guardrails is not reachable on that path.
2. **Try steps 1 → 2 → 3** until `GET $BASE/policy-overviews` returns HTTP 200. Do not assume step 1 works on every deployment.
3. **In-run without proxy is not “use public URL.”** Use `DOMINO_USER_HOST` (or `DOMINO_API_HOST`) plus Bearer from `http://localhost:8899/access-token` when governance is on the platform API gateway.
4. **Public deployment URL + PAT/SA** when you are outside a run, or when step 1 or 2 returns **404** (common when the API gateway is off or governance is not registered on port 8763).
5. **Suffix is always** `/api/governance/v1` on whichever host/base you chose ([Governance API](https://docs.domino.ai/cloud/reference/api/governance-api)).

**Examples below:** Omit `Authorization` only when `TOKEN` is unset (proxy path). For gateway or public URL, set `TOKEN` and use `${TOKEN:+-H "Authorization: Bearer $TOKEN"}` in curl blocks.

## Key Concepts

### Policy (Template)
A **policy** is a reusable governance template that defines the stages, evidence requirements, and approval gates a model must pass through. Examples: SR 11-7, NIST AI RMF, internal model risk frameworks. Policies are created by administrators in the Domino UI.

### Bundle (Living Document)
A **bundle** is the compliance document for a *specific model* in a *specific project*. It follows a policy and accumulates evidence as the model progresses through development, validation, and approval. One project can have multiple 

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 12 — file references are metadata, not package contents.

View saved version →

---
name: domino-jobs
description: Create, run, and manage Domino Jobs - batch executions for scripts, training, and data processing. Covers job configuration, hardware tiers, scheduled jobs (cron), monitoring status, viewing logs, and API-driven execution. Use when running batch workloads, scheduling recurring tasks, or automating training pipelines.
---

# Domino Jobs Skill

## Description
This skill helps users create, run, and manage Domino Jobs - batch executions for running scripts, training models, and processing data.

## Activation
Activate this skill when users want to:
- Run a script or notebook as a batch job
- Schedule recurring jobs
- Configure job settings (hardware, environment)
- Monitor job status and results
- Run jobs via API or CLI

## What is a Domino Job?

A Job is a batch execution that runs a script or command in Domino. Unlike workspaces, jobs:
- Run to completion without user interaction
- Are fully reproducible with tracked inputs/outputs
- Can be scheduled to run automatically
- Scale to any hardware tier

## Creating and Running Jobs

### Via Domino UI
1. Navigate to your project
2. Click **Jobs** in the navigation
3. Click **Run**
4. Configure:
   - **File to Run**: Script path (e.g., `train.py`)
   - **Arguments**: Command-line arguments (optional)
   - **Hardware Tier**: Select resources
   - **Compute Environment**: Select environment
5. Click **Start**

### Via Python (requests)
```python
import requests, os

TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
PROJECT_ID = os.environ["DOMINO_PROJECT_ID"]
headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}

# Start a job
response = requests.post(
    f"{BASE}/api/jobs/v1/jobs",
    headers=headers,
    json={
        "projectId": PROJECT_ID,
        "runCommand": "python train.py --epochs 100",
        "title": "Training run",
    }
)
job = response.json()
print(f"Job ID: {job['id']}")
```

### Via Domino CLI
```bash
# Start a job with script
domino run train.py

# Run with arguments
domino run train.py arg1 arg2 arg3

# Wait for job to complete
domino run --wait train.py arg1 arg2

# Run direct command (not a script)
domino run --direct "pip freeze | grep pandas"
```

### Via REST API
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
curl -X POST "$DOMINO_API_HOST/api/jobs/v1/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"projectId\": \"$DOMINO_PROJECT_ID\",
    \"runCommand\": \"python train.py\",
    \"hardwareTierId\": \"tier-id\",
    \"environmentId\": \"env-id\"
  }"
```

## Job Commands

### Run Python Script
```bash
python train.py
```

### Run with Arguments
```bash
python train.py --data /mnt/data/train.csv --output /mnt/artifacts/model.pkl
```

### Run Jupyter Notebook
```bash
jupyter nbconvert --to notebook --execute notebook.ipynb
```

### Run R Script
```bash
Rscript analysis.R
```

### Run Shell Script
```bash
bash pipeline.sh
```

## Scheduled Jobs

### Creating a Scheduled Job

1. Go to **Scheduled Jobs** in your project
2. Click **New Scheduled Job**
3. Configure:
   - **Name**: Descriptive name
   - **Command**: Script to execute
   - **Schedule**: Cron expression or preset
   - **Hardware Tier**: Resources to use
   - **Environment**: Compute environment

### Schedule Examples

| Schedule | Cron Expression |
|----------|-----------------|
| Every hour | `0 0 * * * ?` |
| Daily at midnight | `0 0 0 * * ?` |
| Every Monday 9 AM | `0 0 9 ? * MON` |
| First of month | `0 0 0 1 * ?` |

### Cron Expression Format

Domino scheduled jobs use **Quartz** cron (six required fields, optional seventh year field; seconds first). This is not Unix five-field crontab. Use `?` in day-of-month or day-of-week when the other field is set. Field rules and special characters (`*`, `-`, `/`, `L`, `W`, `#`): [Quartz CronTrigger tutorial](https://www.quartz-scheduler.org/documentation/quartz-2.5.x/tu

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-launchers
description: Create Domino Launchers - parameterized web forms for self-service job execution. Enable business users to run analyses, generate reports, and trigger batch predictions without coding. Covers parameter types, email notifications, result delivery, and access control. Use when building self-service data products or enabling non-technical users.
---

# Domino Launchers Skill

## Description
This skill helps users create and use Domino Launchers - web forms that allow non-technical users to run parameterized jobs and receive results.

## Activation
Activate this skill when users want to:
- Create self-service data products
- Build parameterized job interfaces
- Enable business users to run analyses
- Generate reports on demand
- Share reproducible workflows

## What is a Launcher?

A Launcher is:
- **Web Form**: UI for entering parameters
- **Job Trigger**: Runs a script with user inputs
- **Results Delivery**: Sends output via email/dashboard
- **Self-Service**: Business users can run without coding

## Use Cases

- **Report Generation**: Parameterized business reports
- **Batch Predictions**: Score data on demand
- **Data Exports**: Custom data extracts
- **Analysis Requests**: Ad-hoc analytics
- **Model Testing**: Test models with different inputs

## Creating a Launcher

### Via Domino UI
1. Go to your project
2. Navigate to **Deployments** > **Launchers**
3. Click **New Launcher**
4. Configure:
   - **Name**: Descriptive name
   - **Description**: What the launcher does
   - **Command**: Script to run
   - **Parameters**: Input fields
   - **Hardware Tier**: Resources
   - **Environment**: Compute environment

### Launcher Script (Python)
```python
# generate_report.py
import argparse
import pandas as pd

# Parse launcher parameters
parser = argparse.ArgumentParser()
parser.add_argument('--start-date', required=True)
parser.add_argument('--end-date', required=True)
parser.add_argument('--region', default='all')
args = parser.parse_args()

# Generate report
df = generate_report(args.start_date, args.end_date, args.region)

# Save results (will be available to launcher user)
df.to_csv('/mnt/results/report.csv', index=False)
df.to_html('/mnt/results/report.html', index=False)
```

### Launcher Script (R)
```r
# launcher.R
args <- commandArgs(trailingOnly = TRUE)
a <- as.integer(args[1])
b <- as.integer(args[2])

if (is.na(a)) {
  print("A is not a number")
} else if (is.na(b)) {
  print("B is not a number")
} else {
  paste("The sum of", a, "and", b, "is:", a + b)
}
```

Command: `launcher.R ${A} ${B}`

### Launcher Command
```bash
# Python script with named arguments
python generate_report.py --start-date ${start_date} --end-date ${end_date} --region ${region}

# Python script with positional arguments
my_script.py -x=1 ${file} ${start_date}

# R script with positional arguments
launcher.R ${A} ${B}
```

**Note:** Parameter values are enclosed in single quotes, preserving special characters. File parameters pass the file path. Multi-select parameters pass comma-separated values.

## Parameter Types

### Text Input
```yaml
name: customer_name
type: text
label: Customer Name
required: true
default: ""
```

### Dropdown/Select
```yaml
name: region
type: select
label: Region
options:
  - North America
  - Europe
  - Asia Pacific
default: North America
```

### Date Picker
```yaml
name: start_date
type: date
label: Start Date
required: true
```

### Number
```yaml
name: quantity
type: number
label: Quantity
min: 1
max: 1000
default: 100
```

### File Upload
```yaml
name: input_file
type: file
label: Input File
accept: .csv,.xlsx
```

## Generating Results

### File-Based Results
Any files created in `/mnt/results/` are available as results:

```python
# Save multiple output formats
df.to_csv('/mnt/results/data.csv')
df.to_excel('/mnt/results/data.xlsx')
fig.savefig('/mnt/results/chart.png')
```

### HTML Email Content
Create `email.html` for custom email body:
```python
# Generate HTML for ema

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-model-endpoints
description: Deploy and monitor model API endpoints in Domino. Covers creating prediction endpoints, version management, Grafana dashboards for latency/errors/resources, alerting, and GPU inference with NVIDIA Triton. Use when deploying models as APIs, monitoring production endpoints, or debugging endpoint issues.
---

# Domino Model Endpoints Skill

This skill provides comprehensive knowledge for deploying and monitoring model API endpoints in Domino Data Lab.

## Key Concepts

### Model Endpoints Overview

Domino Model Endpoints provide:
- REST API for model predictions
- Automatic scaling and load balancing
- Version management
- Built-in monitoring with Grafana
- Authentication via API tokens

### Endpoint Lifecycle

```
Train Model → Register → Deploy Endpoint → Monitor → Update Version
```

## Related Documentation

- [domino-model-serving/SKILL.md](../domino-model-serving/SKILL.md) - REST deploy, registry v1/v2, lifecycle pitfalls
- [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) - modelServing routes and invoke split
- [DEPLOY-ENDPOINT.md](./DEPLOY-ENDPOINT.md) - Creating model APIs
- [MONITORING.md](./MONITORING.md) - Grafana, metrics, alerts
- [SCALING.md](./SCALING.md) - GPU inference, Triton, scaling

## Environment Requirements

**Important:** Model APIs use the **default environment** set for your project. The environment must have the `uwsgi` Python package installed for model endpoints to work.

### Required Package
```dockerfile
# Add to your environment's Dockerfile instructions
RUN pip install uwsgi
```

Or in requirements.txt:
```
uwsgi
```

### Setting Default Environment
1. Go to **Project Settings** → **Execution Preferences**
2. Set the **Default Environment** that includes `uwsgi`
3. This environment will be used for all Model API deployments

## Quick Start

### 1. Create Endpoint Function

```python
# model.py
def predict(features):
    """
    Domino calls this function for predictions.

    Args:
        features: Input data (dict, list, or primitive)

    Returns:
        JSON-serializable prediction result
    """
    import pickle

    # Load model (cached after first call)
    with open('model.pkl', 'rb') as f:
        model = pickle.load(f)

    prediction = model.predict([features])
    return {"prediction": prediction.tolist()}
```

### 2. Deploy via Domino UI

1. Go to **Publish** → **Model APIs**
2. Click **New Model**
3. Configure:
   - Name: `my-classifier`
   - File: `model.py`
   - Function: `predict`
   - Environment: Select compute environment
4. Click **Publish**

### 3. Call the Endpoint

```bash
curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -d '{"data": {"features": [1.0, 2.0, 3.0]}}' \
  https://your-domino.com/models/abc123/latest/model
```

## Environment Variables

When calling endpoints from apps:

| Variable | Description |
|----------|-------------|
| `MODEL_API_URL` | Full endpoint URL |
| `MODEL_API_TOKEN` | Bearer token for authentication |

## Key Metrics to Monitor

| Metric | Target |
|--------|--------|
| Latency P50 | < 100ms |
| Latency P99 | < 500ms |
| Error Rate | < 1% |
| CPU Usage | < 80% |
| Memory | Stable (no growth) |

## Documentation Links

OpenAPI and route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md) (public routes section).

Referenced files: 3 — file references are metadata, not package contents.

View saved version →

---
name: domino-modeling-assistant
description: Enable AI-assisted model development within Domino by writing needed model and training code and using MCP (Model Context Protocol) servers to execute domino jobs. AI coding assistants like Cursor and GitHub Copilot can execute commands as Domino jobs, maintaining security, governance, and reproducibility. Use when setting up AI code assistants to work with Domino, configuring MCP servers, or enabling vibe modeling workflows.
---

# Introduction

This skill provides comprehensive knowledge for enabling modeling assistant within Domino Data Lab, allowing AI coding assistants to interact with the platform.

Vibe modeling refers to using AI code assistants to go beyond pure code generation and assist with:
- Experiment setup and configuration
- Data analysis and exploration
- Model training and evaluation
- Results interpretation

## Key Components

### MCP (Model Context Protocol)

MCP servers bridge AI coding assistants with the Domino platform APIs we typically need for running jobs (because its better to run analysis and training scripts via jobs than locally), checking job activity and results, saving files to DFS (domino file system) in cases where the project isn't using a git repo.

The Domino MCP Server is **bundled with this plugin** and starts automatically when the plugin is installed from its source repository. Skills-only distributions don't include it.

If the `domino_server` tools (such as `run_domino_job`) aren't available in this session, don't pretend to call them. Tell the user, point them at [SETUP.md](./SETUP.md#register-the-server-manually), and meanwhile use the `domino-jobs` and `domino-python-sdk` skills to run work through the Domino REST API.

- **Inside a Domino workspace:** Authentication and project detection are fully automatic.
- **Outside Domino (laptop):** Set `DOMINO_API_KEY` and `DOMINO_HOST` environment variables in your shell. See [SETUP.md](./SETUP.md) for details.

## Related Documentation

- [SETUP.md](./SETUP.md) - Complete setup guide for the modeling assistant


## Important Considerations

You are a Domino Data Lab powered agentic coding tool that helps write code in addition to running tasks on the Domino Data Lab platform on behalf of the user using available tool functions provided by the domino_server MCP server. Whenever possible run commands as domino jobs rather than on the local terminal.

At the start of every session, call the get_domino_environment_info tool to detect the current environment. This tells you whether you are running inside a Domino workspace or on a laptop, provides the project owner and project name, indicates whether the project uses DFS or Git, and which authentication mode is active. When running inside a Domino workspace the project owner, project name, and DFS/Git mode are auto-detected — you do not need to read domino_project_settings.md. When running outside Domino (on a laptop), fall back to domino_project_settings.md for the project name, user name, and dfs setting.

When running a job, always check its status and results if completed and briefly explain any conclusions from the result of the job run. If a job result ever includes an mlflow or experiment run URL, always share that with the user using the open_web_browser tool.

Any requests related to understanding or manipulating project data should assume a dataset file is already part of the domino project and accessible via job runs. Always create scripts to understand and transform data via job runs. The script can assume all project data is accessible under the '/mnt/data/' directory or the '/mnt/imported/data/' directory, be sure to understand the full path to a dataset file before using it by running a job to list all folder contents recursively. Analytical outputs should be in plain text tabular format sent to stdout, this makes it easier to check results from the job run.

If the project is DFS instead of Git based (auto-detected inside Domi

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 1 — file references are metadata, not package contents.

View saved version →

---
name: domino-model-monitoring
description: Monitor deployed models in Domino including drift detection, model quality tracking, and alerting. Covers data drift analysis, prediction capture, baseline comparison, alert configuration, and remediation workflows. Use when monitoring production models, detecting drift, or setting up model health alerts.
---

# Domino Model Monitoring Skill

## Description
This skill helps users monitor deployed models in Domino, including drift detection, model quality tracking, and alerting.

## Activation
Activate this skill when users want to:
- Monitor deployed model performance
- Set up drift detection
- Configure monitoring alerts
- Analyze prediction data
- Understand model degradation

## What is Model Monitoring?

Domino Model Monitoring provides:
- **Data Drift Detection**: Detect changes in input data distributions
- **Model Quality Tracking**: Monitor prediction accuracy over time
- **Alerting**: Get notified when metrics exceed thresholds
- **Prediction Capture**: Log predictions for analysis
- **Reproducibility**: Diagnose issues with captured data

## Setting Up Monitoring

### Prerequisites
1. Deployed Model API in Domino
2. Training dataset (for baseline)
3. Ground truth data (optional, for quality metrics)

### Enable Monitoring
1. Go to your Model API page
2. Click **Monitoring** tab
3. Click **Set Up Monitoring**
4. Upload training dataset
5. Configure drift detection settings

### Register Training Data
```python
# Training data provides baseline for drift detection
# Upload via UI or programmatically

import pandas as pd

# Your training data
train_df = pd.read_csv("training_data.csv")

# Save for monitoring setup
train_df.to_csv("/mnt/artifacts/training_data.csv", index=False)
```

## Drift Detection

### Types of Drift

| Drift Type | Description |
|------------|-------------|
| **Data Drift** | Input feature distributions change |
| **Concept Drift** | Relationship between inputs and outputs changes |
| **Prediction Drift** | Output distribution changes |

### Statistical Tests

Domino supports multiple drift detection tests:

| Test | Best For |
|------|----------|
| **Kullback-Leibler Divergence** | General-purpose, most common |
| **Population Stability Index (PSI)** | Finance industry standard |
| **Wasserstein Distance** | Comparing distributions |
| **Energy Distance** | Multivariate distributions |

### Configure Drift Detection

1. Go to Model API > **Monitoring**
2. Click **Configure Drift Detection**
3. For each feature:
   - Select test type
   - Set threshold
   - Enable/disable alerts

### Example Thresholds

| Test | Low Drift | Medium Drift | High Drift |
|------|-----------|--------------|------------|
| KL Divergence | < 0.1 | 0.1 - 0.2 | > 0.2 |
| PSI | < 0.1 | 0.1 - 0.25 | > 0.25 |

## Prediction Capture

### How It Works
Domino automatically captures predictions:
1. Model receives request
2. Prediction is made
3. Input/output logged to dataset
4. Data available for drift analysis

### Access Captured Data
```python
import pandas as pd

# Predictions captured in Domino Dataset
predictions_df = pd.read_parquet(
    "/mnt/data/model-predictions/predictions.parquet"
)

print(predictions_df.head())
```

### Capture Frequency
- Predictions batched hourly
- Full data available in monitoring dataset
- Retention configurable by admin

## Model Quality Monitoring

### With Ground Truth
If you provide ground truth labels:

```python
# Upload ground truth
ground_truth = pd.DataFrame({
    "prediction_id": [...],
    "actual_label": [...]
})

# Upload to monitoring
ground_truth.to_csv("/mnt/artifacts/ground_truth.csv", index=False)
```

### Quality Metrics
- Accuracy
- Precision/Recall
- F1 Score
- AUC-ROC
- Mean Squared Error (regression)

### Schedule Quality Checks
1. Go to Monitoring > **Quality**
2. Upload ground truth dataset
3. Configure metric thresholds
4. Set check frequency

## Alerting

### Configure Alerts
1. Go to Model API > **Monitoring**
2. Click 

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-model-serving
description: Deploy, invoke, and retire Domino model APIs and registered models via REST. Covers modelServing lifecycle, registered-models v1 vs v2 paths, MLflow tracking vs registry API, inference URLs vs management API, and GenAI endpoint vanity URLs. Use when automating model deployment, predictions, registry updates, or debugging stop/archive/delete behavior.
---

# Domino model serving (REST)

Programmatic model deployment and inference on Domino. UI-focused endpoint monitoring stays in [model-endpoints](../model-endpoints/SKILL.md). Shared REST field catalogs live in [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) and [API-MODELS.md](../python-sdk/API-MODELS.md).

Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . **Do not use API keys.**

## Configuration

| Context | Base | Authorization |
|---------|------|----------------|
| In-run, `DOMINO_API_PROXY` set | `{DOMINO_API_PROXY}` | None |
| In-run, no proxy | `{DOMINO_USER_HOST or DOMINO_API_HOST}` | Bearer from `http://localhost:8899/access-token` |
| Outside run | Public deployment URL | Bearer PAT or SA |

Management paths include `/api/modelServing/v1/...`, `/api/registeredmodels/v1|v2/...`. Confirm with `GET {base}/api/modelServing/v1/modelApis?limit=1` returning HTTP 200.

```bash
curl ${TOKEN:+-H "Authorization: Bearer $TOKEN"} "$BASE/api/modelServing/v1/modelApis?limit=1"
```

## Three surfaces (do not merge)

| Surface | Purpose |
|---------|---------|
| **Model Serving REST** | `/api/modelServing/v1/modelApis` deploy, versions, lifecycle |
| **Registered models REST** | `/api/registeredmodels/v2` list/register; `/api/registeredmodels/v1/{modelName}` get/update/versions |
| **MLflow tracking** | `MLFLOW_TRACKING_URI` / runs UI; not the same as registered-models REST |

Logging a run to MLflow does not replace registry or modelServing calls for deployment automation.

## Management vs inference URL

- Create/list/update: platform base + `/api/modelServing/v1/...` (or registered-models paths).
- **Predict:** use the **`url`** field on the model API or version (often `.../models/.../latest/model`), on the deployment ingress. That path is not routed through `DOMINO_API_HOST` sidecar the same way as `/api/`.

GenAI: management API vs `https://.../endpoints/{vanity}` split. See [model-endpoints](../model-endpoints/SKILL.md) and the GenAI section in [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md).

## Lifecycle (model APIs)

Typical order:

1. Train / register (MLflow and/or `POST /api/registeredmodels/v2/...`).
2. `POST /api/modelServing/v1/modelApis` (or new version on existing API).
3. Poll `GET /api/modelServing/v1/modelApis/{modelApiId}` until `status` is `Running` (or `Failed`); then read **`url`** for invoke.
4. **Stop** vs **archive** vs **delete** are different product operations; names and reliability differ by route.
5. Treat **DELETE** on deployments/model APIs as best-effort; verify resource gone before assuming cleanup.
6. **Stop** may be long-running and not idempotent; poll status instead of fire-and-forget retry loops.

Details and route list: [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md).

## Registered model path key

`modelName` in `/api/registeredmodels/v1/{modelName}` is the **registered model name string**, not an opaque UUID. List/register often use **v2**; get/update/versions use **v1** with that name in the path.

## Invocation auth

Use the model API token documented on https://docs.domino.ai for the model endpoint invoke path. Older examples may show a basic-auth pattern with the token as both user and password; treat that as legacy and confirm in [API-SPECS.md](../domino-api-intro/API-SPECS.md) before hardcoding.

## Related documentation

- [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) - REST lifecycle and routes
- [API-MODELS.md](../python-sdk/API-MODELS.md) - broader models API catalog
- [model-endpoints](../model-endpoint

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-projects
description: Work with Domino Projects including Git integration, DFS vs Git-based projects, collaboration, and version control. Covers project creation, Git provider setup (GitHub, GitLab, Bitbucket), branch management, collaborator permissions, and project settings. Use when creating projects, setting up Git repos, or managing team collaboration.
---

# Domino Projects and Git Skill

## Description
This skill helps users work with Domino Projects - including project creation, Git integration, version control, and collaboration features.

## Activation
Activate this skill when users want to:
- Create or configure a Domino project
- Set up Git integration
- Understand DFS vs Git-based projects
- Collaborate with team members
- Manage project settings and permissions

## Project Types

### Git-Based Projects
- Code stored in external Git provider (GitHub, GitLab, Bitbucket)
- Full Git workflow support
- Better for teams familiar with Git
- Access to Git provider features (PRs, issues)

### Domino File System (DFS) Projects
- Code stored in Domino's internal file system
- Automatic versioning (Git under the hood)
- Simpler for users unfamiliar with Git
- Good for research and experimentation

## Creating a Project

### Via Domino UI
1. Click **New Project**
2. Enter project name
3. Select project type:
   - **Git-based**: Connect to Git repository
   - **DFS**: Use Domino File System
4. Configure settings
5. Click **Create**

### Via Python SDK
```python
from domino import Domino

domino = Domino()

# Create DFS project
project = domino.project_create(
    project_name="my-project",
    owner_name="your-username"
)

print(f"Project ID: {project['id']}")
```

## Git-Based Projects

### Connecting to Git Repository
1. Create project as Git-based
2. Enter repository URL
3. Configure authentication:
   - **SSH Key**: Add SSH public key to Git provider
   - **HTTPS**: Use personal access token

### Git Credentials
```bash
# Add SSH key to Domino account
# Go to Account Settings > Git Credentials

# Or use HTTPS with token
https://username:token@github.com/org/repo.git
```

### Working with Git in Workspaces

#### Sync Changes
Click **Sync** in workspace to push changes:
1. Domino commits changes
2. Pushes to configured branch
3. Updates project state

#### Manual Git Commands
```bash
# Check status
git status

# Add and commit
git add .
git commit -m "Update model training"

# Push to remote
git push origin main

# Pull latest
git pull origin main
```

#### IDE Git Integration
- VS Code: Use Source Control panel
- JupyterLab: Use Git extension
- RStudio: Use Git pane

### Branch Management
```bash
# Create feature branch
git checkout -b feature/new-model

# Work on branch
git add .
git commit -m "Add new model"
git push origin feature/new-model

# Switch branches
git checkout main
```

### Multi-Repository Support
Domino supports multiple Git repositories per project:

1. Go to Project Settings > **Repositories**
2. Add additional repositories
3. Configure mount paths
4. Enable/disable per execution

```python
# Access code from multiple repos
# Repo 1: /mnt/code/main-repo/
# Repo 2: /mnt/code/shared-lib/
from shared_lib import utils
```

## DFS Projects

### How DFS Works
- Files stored in Domino-managed Git
- Automatic commits on sync
- Version history tracked
- Simplified workflow

### Sync Files
1. Work in workspace
2. Click **Sync** when ready
3. Enter commit message
4. Files pushed to Domino

### View History
1. Go to **Files** in project
2. Click **History**
3. View all commits
4. Restore previous versions

## Project Structure

### Recommended Structure
```
project/
├── README.md           # Project documentation
├── requirements.txt    # Python dependencies
├── data/               # Local data (small files)
├── src/                # Source code
│   ├── train.py
│   ├── evaluate.py
│   └── utils.py
├── notebooks/          # Jupyter notebooks
│   └── exploration.ipynb
└── tests/              # Unit te

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-python-sdk
description: Programmatically interact with Domino using python-domino SDK and REST APIs. Covers authentication, running jobs, managing projects, file operations, model deployment, and automation. Use when automating Domino workflows, integrating with CI/CD, or building custom tooling around Domino.
---

# Domino Python SDK Skill

## Description
This skill helps users work with the Domino Python SDK (python-domino) and REST APIs to programmatically interact with Domino.

## Activation
Activate this skill when users want to:
- Use the Domino Python SDK
- Make API calls to Domino
- Automate Domino workflows
- Integrate Domino with external systems
- Query Domino programmatically

## Overview

Domino provides two main programmatic interfaces:
- **python-domino**: Python SDK for common operations
- **REST API**: Full HTTP API for all Domino features

## Installation

### python-domino
```bash
# Install from PyPI
pip install dominodatalab

# Or install with extras
pip install "dominodatalab[data]"
```

### In Domino Environment
Add to requirements.txt:
```
dominodatalab>=1.4.0
```

Or Dockerfile:
```dockerfile
RUN pip install dominodatalab
```

## Authentication

Canonical guide: https://docs.domino.ai/cloud/reference/api/domino-api-authentication

**Do not use API keys** (`X-Domino-Api-Key`, `DOMINO_USER_API_KEY`, `api_key=`). Use PAT or service account tokens only when calling from **outside** a run.

### In-run (workspace, job, app backend)

Domino injects `DOMINO_USER_HOST` / `DOMINO_API_HOST` (same base; prefer `DOMINO_USER_HOST`). When JWT credential propagation is enabled, `DOMINO_API_PROXY` is the forwarded platform API base (not the same as `http://localhost:8899/access-token`).

| Pattern | When | Code |
|---------|------|------|
| **API proxy (preferred)** | `DOMINO_API_PROXY` is set (Domino **5.4.0+** in runs with JWT credential propagation configured) | Call `{DOMINO_API_PROXY}{path}` with **no** `Authorization` header. The JWT sidecar adds the **starting user** access JWT on the forwarded request. |
| **Access token + platform host** | Older deployments, or you intentionally call `DOMINO_USER_HOST` / `DOMINO_API_HOST` instead of the proxy URL | Fetch a short-lived JWT, then Bearer on the platform base. |

**API proxy (preferred on 5.4.0+):**

```python
import os
import requests

base_url = os.environ["DOMINO_API_PROXY"].rstrip("/")
headers = {}

response = requests.get(f"{base_url}/v4/users/self")
```

```bash
curl "$DOMINO_API_PROXY/v4/users/self"
```

The proxy is not guaranteed on every deployment or run type. If `DOMINO_API_PROXY` is missing, use the access-token pattern or PAT from outside the cluster.

**Access token + `DOMINO_USER_HOST` (legacy-friendly in-run):** use the [In-run setup block](#in-run-setup-block-for-examples-in-this-skill) below; it covers both proxy and access-token paths.

**Before Domino 5.4.0:** JWT file propagation (`DOMINO_TOKEN_FILE`) was used before the API proxy; legacy behavior may still exist if admins enable `EnableLegacyJwtTooling`. Prefer migrating to the proxy pattern on supported versions.

### Outside a run (laptop, CI, cron outside Domino)

Your code is **not** executing inside a Domino workspace, job, or app container. Domino does **not** inject `DOMINO_API_PROXY`, `DOMINO_USER_HOST`, or an access-token sidecar. You must supply both:

1. **Base URL** — the deployment URL users open in the browser (HTTPS), for example `https://yourcompany.engineering.domino.tech`. Not `http://127.0.0.1:8763` and not values copied from an in-run environment.
2. **Credential** — `Authorization: Bearer` with a token you store securely (secret manager, CI variable, not committed to git):
   - **Personal Access Token (PAT)** when the automation acts as you. Create under Account settings or `POST /api/pat/v1/tokens` while already authenticated.
   - **Service account token** when a pipeline or integration runs without a human user. An admin provisions the service account 

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 9 — file references are metadata, not package contents.

View saved version →

---
name: domino-trace-setup
description: Set up GenAI tracing for an LLM or agent application in Domino by checking MLflow 3.2.0 and Domino SDK requirements, then adding a tracing_setup.py helper, evaluators, a config.yaml, and @add_tracing / DominoRun wiring. Use when the user asks to add tracing, observability, or evaluation to an agent or LLM app running on Domino.
---

# Set Up Domino GenAI Tracing

Instrument an agent or LLM application so its calls, spans, and evaluator scores show up in
Domino experiments. For traditional ML tracking, use the `domino-experiment-setup` skill instead.

## Inputs

1. **LLM framework**: `openai` (default), `anthropic`, or `langchain`. Detect from the code if possible.
2. **Agent name**: name for the traced entry-point function.
3. **Evaluator**: whether to add quality scoring, and whether to use an LLM-as-judge.
4. **Aggregated metrics**: which evaluator metrics to summarize per run.

## Steps

1. **Check requirements.** Confirm `mlflow==3.2.0` and the Domino SDK with agents support are
   installed. If not, tell the user to add them to the environment's Dockerfile:

   ```dockerfile
   RUN pip install mlflow==3.2.0
   RUN pip install "dominodatalab[data,agents]==2.2.0"
   ```

2. **Copy helpers** from this skill's `assets/` folder into the project:
   - `assets/tracing_setup.py`: `setup_tracing(framework)`, `create_evaluator()`,
     `create_llm_judge_evaluator()`, and `get_aggregation_metrics()`.
   - `assets/config.yaml`: agent configuration logged with `DominoRun(agent_config_path=...)`.
     Replace the example model names with the ones the user actually calls.
3. **Instrument the agent.** Decorate the entry point with `@add_tracing` and run it inside
   `DominoRun`, following the example below. Keep the user's existing logic unchanged.
4. **Explain how to view traces** (see "Viewing traces").

## Example traced agent

```python
import mlflow
from domino.agents.tracing import add_tracing
from domino.agents.logging import DominoRun
from openai import OpenAI

mlflow.openai.autolog()
client = OpenAI()

def quality_evaluator(inputs, output):
    return {
        "response_length": len(output.get("response", "")),
        "confidence": output.get("confidence", 0),
    }

@add_tracing(name="my_agent", evaluator=quality_evaluator)
def my_agent(query: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    )
    return {
        "response": response.choices[0].message.content,
        "confidence": 0.9,
        "model": "gpt-4o-mini",
    }

if __name__ == "__main__":
    aggregated_metrics = [("response_length", "mean"), ("confidence", "mean")]

    with DominoRun(
        run_name="example-run",
        agent_config_path="config.yaml",
        custom_summary_metrics=aggregated_metrics,
    ) as run:
        for query in ["What is machine learning?", "Explain neural networks"]:
            result = my_agent(query)
            print(f"Q: {query}\nA: {result['response'][:100]}...\n")
        print(f"Run ID: {run.run_id}")
```

## Framework autologging

| Framework | Setup |
|---|---|
| OpenAI | `mlflow.openai.autolog()` then `OpenAI()` |
| Anthropic | `mlflow.anthropic.autolog()` then `Anthropic()` |
| LangChain | `mlflow.langchain.autolog()` then e.g. `ChatOpenAI(model="gpt-4o-mini")` |

## Viewing traces

1. Go to **Experiments** in Domino.
2. Open the experiment the run logged to (by default `tracing-{username}`).
3. Select the run and open the **Traces** tab.

The trace view shows the span tree (agents, tools, messages), token usage, latency, and evaluator
scores.

For evaluators, multi-agent patterns, and `DominoRun` details, use the `domino-genai-tracing` skill.

Referenced files: 2 — file references are metadata, not package contents.

View saved version →

---
name: domino-ui-bootstrap
description: Bootstrap or retrofit a Vite + React 18 + TypeScript project so it uses the Domino design system (`@dominodatalab/extensions-tools`). Scoped to a **fully standalone SPA frontend for a Domino application or extension**, not snippets, library additions, or work inside an existing Domino monorepo package. Use whenever the user wants to create, scaffold, refactor, retrofit, or set up such an SPA (a React app, web app, frontend, UI, or extension) that should "look like Domino", use Domino components, follow the Domino design system, or integrate with the Domino platform, including when they mention DominoThemeProviderDecorator, extensions-tools, or base-components without saying "Domino". Also use to make an existing standalone React/Vite project "Domino-styled" or add Domino theming. Handles React 18 / react-router 5 pinning, Storybook MCP registration, theme provider wiring, and build verification.
---

# Domino UI Bootstrap

This skill makes a project use the Domino design system correctly. It works in three contexts:

- **Empty or non-existent directory** — scaffold a Vite + React 18 + TS project from scratch, then apply the Domino setup.
- **Existing Vite + React project** — leave the project alone, retrofit just the Domino bits (deps, theme provider, MCP, agent instructions file).
- **Existing React project on a different bundler/setup** — flag the mismatch to the user, then proceed only with their direction.

In all three cases, the skill executes the work end-to-end (runs commands, edits files), but it adapts to what's already there instead of overwriting blindly.

## Why this exists

A handful of choices look arbitrary but aren't — they match the rest of the Domino ecosystem and are easy to get wrong:

- **React 18, not 19.** `@dominodatalab/extensions-tools` peer-depends on React 18. React 19 fails at install.
- **react-router 5, not 6.** The library uses v5 APIs (`HashRouter` from `react-router-dom@5`).
- **HashRouter, not BrowserRouter.** This is what the library expects internally. It also gives cleaner isolation between the app's frontend and backend — client-side routes live entirely after the `#`, so they never collide with backend paths or the Domino proxy prefix, and inner navigation/linking works without server-side rewrite rules.
- **The npm package name is `@dominodatalab/extensions-tools`.** Storybook code snippets show imports from `@domino/base-components` — that's a Storybook-internal alias. Rewrite every such import to `@dominodatalab/extensions-tools` before pasting into a Domino project. This is the single most common mistake.
- **`DominoThemeProviderDecorator` wraps the whole React tree.** Without it, Domino components render unstyled or crash.
- **URLs must survive the proxy path.** Domino serves the app under a prefix (e.g. `/preview/<appId>/`), so asset and API URLs must be document-relative — not root-absolute. Vite's default `base: '/'` emits `/assets/…` and bypasses the proxy; user-written `fetch('/api/…')` does the same. Set `base: './'` and build API URLs against `document.baseURI`. See Step 8. Invisible in local dev (`pathname` is `/`), surfaces only after deployment.

Treat these as invariants the project must satisfy by the end. How you get there depends on what's already in the target directory.

## Host-specific files

Steps 2, 6, 10, 11, and 13 write files that depend on which coding assistant is running this skill. Use the column for the current host. If you can't tell which host you are, or it isn't listed, ask the user which files their assistant reads.

| Purpose | Claude Code | Codex |
|---|---|---|
| Agent instructions file | `CLAUDE.md` | `AGENTS.md` |
| MCP server registration | `.mcp.json` | `.codex/config.toml` |
| Per-user MCP enablement | `.claude/settings.local.json` | None needed; Codex loads `.codex/config.toml` once the user trusts the project |
| Reload after MCP changes | Restart Claude Code | Restart Codex |

The rest of this sk

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: domino-ui-design
description: Build Domino-styled web applications matching the Domino Design System. Use when creating or styling Domino apps (Dash, Streamlit, Flask, FastAPI+HTML), generating UI components, reviewing UX, or when the user mentions "Domino app", "Domino UI", "Domino dashboard", or "Domino design system". Covers the full stack — FastAPI backend, Ant Design 5.x frontend via CDN, Domino theme tokens (colors, typography, spacing), chart styling with Highcharts, Domino API authentication, proxy-safe routing, UX writing, error handling, empty states, form design, table patterns, and layout best practices. Also use for UX reviews of existing Domino app screenshots.
---

# Domino UI Design System Skill

## Architecture

Domino apps use **FastAPI** (Python backend) + **Ant Design 5.x** (frontend via CDN, no build step). Avoid inline styles and JS — use statically referenced CSS/JS files. Output errors to stdout.

### CDN Setup

Include in HTML `<head>`:

```html
<script src="https://unpkg.com/react@18/umd/react.production.min.js"></script>
<script src="https://unpkg.com/react-dom@18/umd/react-dom.production.min.js"></script>
<script src="https://unpkg.com/antd@5.11.2/dist/antd.min.js"></script>
<link href="https://unpkg.com/antd@5.11.2/dist/reset.css" rel="stylesheet">
<script src="https://unpkg.com/dayjs@1.11.10/dayjs.min.js"></script>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
<script src="https://code.highcharts.com/highcharts.js"></script>
```

Use `React.createElement()` (aliased as `h`) — no JSX, no build step:

```javascript
const { ConfigProvider, Button, Table, Modal, Select, Input, Form, Space, Card, Tag, Alert } = antd;
const { createElement: h, useState, useEffect } = React;
```

## Domino Theme

Wrap the app in `ConfigProvider` with this theme:

```javascript
const dominoTheme = {
  token: {
    colorPrimary: '#543FDE',          // Purple
    colorPrimaryHover: '#3B23D1',
    colorPrimaryActive: '#311EAE',
    colorText: '#2E2E38',
    colorTextSecondary: '#65657B',
    colorTextTertiary: '#8F8FA3',
    colorSuccess: '#28A464',
    colorWarning: '#CCB718',
    colorError: '#C20A29',
    colorInfo: '#0070CC',
    colorBgContainer: '#FFFFFF',
    colorBgLayout: '#FAFAFA',
    colorBorder: '#E0E0E0',
    fontFamily: 'Inter, Lato, Helvetica Neue, Helvetica, Arial, sans-serif',
    fontSize: 14,
    borderRadius: 4,
    borderRadiusLG: 8,
  },
  components: {
    Button: { primaryShadow: 'none', defaultShadow: 'none' },
    Table: { headerBg: '#FAFAFA', rowHoverBg: '#F5F5F5' },
  },
};
```

### Top Navigation Bar

Height: 44px, Background: `#2E2E38`, Text: `#FFFFFF`. Use the Domino logo (height 32px). Primary actions on right. Only add nav items with real destinations.

**Domino Logo:** Reference `assets/domino-logo.svg`, resolved relative to this skill's directory. The SVG is white text on transparent background, designed for the dark nav bar.

### Charts (Highcharts)

```javascript
Highcharts.setOptions({
  colors: ['#543FDE', '#0070CC', '#28A464', '#CCB718', '#FF6543', '#E835A7', '#2EDCC4', '#A9734C'],
  chart: { style: { fontFamily: 'Inter, Lato, Helvetica Neue, Arial, sans-serif' } },
});
```

## Domino API Integration

### Authentication

Re-acquire the access token on **every** API call (it expires quickly):

```python
import requests, os

def get_auth_headers():
    token = requests.get('http://localhost:8899/access-token').text.strip()
    return {'Authorization': f'Bearer {token}'}
```

### API Base Paths

Two separate APIs with **different** base paths:

| API | Base Path | Example |
|-----|-----------|---------|
| Platform APIs (public-api.json) | `/api/` | `{DOMINO_API_HOST}/api/jobs/beta/jobs?projectId=...` |
| Legacy Platform APIs | `/v4/` | `{DOMINO_API_HOST}/v4/jobs?projectId=...` |
| Governance APIs | `/api/governance/v1/` | `{CLUSTER_URL}/api/governance/v1/bundles?project_id=...` |

**Do NOT** add `/api/` prefix to

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 2 — file references are metadata, not package contents.

View saved version →

---
name: domino-workspaces
description: Work with Domino Workspaces - interactive development environments including Jupyter, JupyterLab, VS Code, and RStudio. Covers launching workspaces, configuring hardware tiers, environment selection, volume mounting, SSH access, and package installation. Use when setting up development environments, configuring workspace settings, or troubleshooting IDE issues.
---

# Domino Workspaces Skill

## Description
This skill helps users work with Domino Workspaces - interactive development environments for data science work including Jupyter notebooks, JupyterLab, VS Code, and RStudio.

## Activation
Activate this skill when users want to:
- Launch or configure a workspace in Domino
- Work with Jupyter notebooks, VS Code, or RStudio
- Configure workspace settings (hardware tier, environment, volumes)
- Understand workspace persistence and file management
- Use SSH to connect to remote workspaces
- Install packages in workspaces

## Workspace Types

### Available IDEs
Domino provides these default workspace types:
- **Jupyter Notebook**: Classic notebook interface
- **JupyterLab**: Modern Jupyter interface with file browser
- **VS Code**: Full-featured code editor with extensions
- **RStudio**: IDE for R development
- **Custom IDEs**: Can configure additional workspace types

## Launching a Workspace

### Via Domino UI
1. Navigate to your project
2. Click **Workspaces** in the navigation
3. Click **Launch Workspace**
4. Select:
   - **IDE**: Choose Jupyter, VS Code, RStudio, etc.
   - **Hardware Tier**: CPU/memory/GPU resources
   - **Compute Environment**: Docker environment with tools
   - **Volume Size**: Persistent storage (if needed)
5. Click **Launch**

### Via Python SDK
```python
from domino import Domino

domino = Domino("project-owner/project-name")

# Start a workspace
workspace = domino.workspace_start(
    environment_id="env-123",
    hardware_tier_name="small",
    workspace_type="JupyterLab"
)

print(f"Workspace ID: {workspace['workspaceId']}")
```

## File Persistence

### Persistent Directories
Work saved to these directories persists across workspace sessions:
- `/mnt/` - Project files (synced with Domino)
- `/mnt/data/` - Domino Datasets
- `/mnt/artifacts/` - Project artifacts
- `/mnt/imported/` - Imported data

### Home Directory Persistence
Domino 6.1+ supports home directory persistence:
- Installed packages persist across sessions
- User configurations (.bashrc, .vimrc) are retained
- Reduces workspace startup time

## Package Installation

### Temporary (Current Session)
```bash
# Python packages
pip install pandas numpy scikit-learn

# R packages
install.packages("tidyverse")
```

### Persistent (Via Environment)
Add to environment's Dockerfile instructions:
```dockerfile
RUN pip install pandas numpy scikit-learn
```

Or use requirements.txt in your project:
```text
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
```

## SSH Access to Workspaces

Domino 6.1+ allows SSH connections from local machines:

1. **Enable SSH** in workspace settings
2. **Get connection details** from running workspace
3. **Connect via SSH**:
```bash
ssh -i ~/.ssh/domino_key user@workspace-hostname
```

Benefits:
- Use local IDE (VS Code Remote, PyCharm)
- Access remote compute from local machine
- Browse files locally while computing remotely

## Git Integration in Workspaces

### Using Git
```bash
# Clone a repo in workspace
git clone https://github.com/user/repo.git

# Standard git workflow
git add .
git commit -m "Update model"
git push origin main
```

### Sync with Domino
Click **Sync** in the workspace UI to push changes back to Domino project.

## Workspace Operations

### Stop a Workspace
```python
domino.workspace_stop(workspace_id)
```

### Resume a Workspace
Stopped workspaces can be resumed to continue work.

### View Workspace Logs
Access logs for debugging through the Domino UI or API.

## Best Practices

1. **Save frequently**: Sync work to Domino regularly
2. **Use appropriate 

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 2 — file references are metadata, not package contents.

View saved version →

---
name: netapp-volumes
description: Work with Domino Volumes for NetApp ONTAP - enterprise-grade, multi-terabyte storage with near-instant snapshots. Covers volume creation, snapshot versioning with commit messages, cross-project sharing, mount paths (/mnt/netapp-volumes/ or /domino/netapp-volumes/), and the NetApp Volumes REST API. Use when managing large-scale data storage, needing fast no-copy snapshots, or integrating existing NetApp ONTAP infrastructure with Domino.
---

# Domino NetApp Volumes Skill

## Description
This skill helps users work with Domino Volumes for NetApp ONTAP — enterprise-grade storage that mounts NetApp ONTAP filesystems directly into Domino workloads, enabling multi-terabyte scale with near-instant snapshotting.

## Activation
Activate this skill when users want to:
- Create or manage NetApp Volumes in Domino
- Work with NetApp volume snapshots and versioning
- Share large-scale data across projects or teams
- Access existing NetApp ONTAP storage from Domino workspaces, jobs, or apps
- Use the NetApp Volumes REST API programmatically
- Understand mount paths for NetApp volumes
- Choose between NetApp Volumes and Domino Datasets

## What is a Domino NetApp Volume?

A Domino NetApp Volume is:
- **Enterprise-grade storage**: Backed by NetApp ONTAP via Kubernetes PVCs with the `netapp-storage` storage class
- **Multi-terabyte scale**: No practical upper limit — suitable for very large datasets
- **Near-instant snapshots**: ~3 seconds even for 100+ GB volumes using redirect-on-write (no extra storage consumed at snapshot time)
- **Versioned with commit messages**: Snapshots support human-readable tags and commit messages, unlike Datasets
- **Shareable**: Attach a single volume to multiple projects and teams
- **Persistent**: Data persists across executions

### NetApp Volumes vs. Domino Datasets

| Feature | NetApp Volumes | Domino Datasets |
|---------|---------------|-----------------|
| Storage backend | External NetApp ONTAP | Domino-managed NFS/EFS |
| Data scale | Multi-terabyte and beyond | Up to ~1 TB |
| Snapshot speed | ~3 seconds (any size) | Scales with data size |
| Snapshot storage cost | No extra space (redirect-on-write) | Duplicates physical data |
| Commit messages on snapshots | Supported | Not supported |
| Create new volume from snapshot | Read-only clones | Can create editable dataset |
| Cross-project sharing | Yes | Yes |
| Admin prerequisite | Admin must register filesystems | None |
| REST API | Full dedicated API | Via Domino Python SDK |

**Use NetApp Volumes when:**
- Data exceeds ~1 TB
- Near-instant snapshotting is required
- Your organization already has NetApp ONTAP infrastructure
- You need snapshot commit messages for audit trails
- High-performance shared storage across many teams

**Use Domino Datasets when:**
- Data is Domino-managed with no external infrastructure dependency
- You need to create editable new versions from snapshots
- Data is under ~1 TB

---

## Prerequisites

An admin must register at least one NetApp filesystem (Kubernetes PVC with the `netapp-storage` label) before users can create volumes. Contact your Domino administrator if no filesystems are available.

---

## Creating a NetApp Volume

### From the Domino Home Page
1. Navigate to **Data > NetApp Volumes** in the toolbar
2. Click **Add NetApp Volume > Create Volume**
3. Fill in the form:
   - **Name**: Letters, numbers, underscores, hyphens only
   - **Description**: Brief overview of the data
   - **Data Plane**: Select from available options
   - **NetApp Filesystem**: Choose a registered filesystem
   - **Capacity**: Set maximum storage allocation
4. Click **Next** to configure permissions
5. Assign users with **Reader / Editor / Owner** roles
6. Click **Finish**
7. Manually add the volume to projects afterward

### From within a Project (recommended — auto-associates)
1. Open a project → **Data > NetApp Volumes** (left panel)
2. Click **Add NetApp Volume > Create Volume**
3. Fill in the

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: tags-and-properties
description: Manage Domino tags and properties via the Taxonomy API. Covers tags/namespaces (create, list, update, delete; tag entities — project, model, dataset, app, project_template, netapp_volume; query by tag; autocomplete; merge; CSV import/export) AND properties (typed metadata fields — text, number, date, boolean, url, user, organization, select, multi_select — with per-entity values, groups, and versioned entities like model_version/app_version). Use when organizing entities with tags, building hierarchical namespaces, defining typed metadata fields, setting property values on projects/models/apps, finding entities by tag, bulk-tagging during onboarding, or migrating taxonomy across environments.
---

# Domino Tags and Properties Skill

## Description

This skill covers Domino's Taxonomy API, which manages two complementary
metadata systems for organizing entities (projects, models, datasets, apps,
project templates, NetApp volumes):

- **Tags** — hierarchical labels grouped into namespaces. An entity either has
  a tag or it doesn't.
- **Properties** — typed metadata *fields* (`text`, `number`, `date`,
  `select`, …), each holding a value per entity.

It documents every public endpoint with curl examples that work today against
a Domino cluster where the taxonomy service is enabled. Properties are covered
in depth in [PROPERTIES.md](./PROPERTIES.md) (definitions) and
[PROPERTY-VALUES.md](./PROPERTY-VALUES.md) (per-entity values).

## Activation

Activate this skill when the user wants to:

- Tag a project, model, dataset, app, project template, or NetApp volume
- Find all entities that share a tag, or query by multiple tags
- Build a hierarchical taxonomy (namespaces with nested tags)
- Bulk-tag entities during onboarding
- Migrate a taxonomy tree across Domino environments (CSV export/import)
- Merge duplicate tags
- Tune autocomplete results in a tagging UI
- Define typed metadata fields (properties) — e.g. a `Budget` number, a
  `Review Date`, an `Owner` user, or a `Status` single-select
- Set, read, or clear property values on an entity (incl. versioned entities
  like `model_version` / `app_version`)
- Group properties or manage property definitions across a deployment

## Configuration

Auth via the local access-token endpoint. Never use `DOMINO_USER_API_KEY`.

```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
# Taxonomy is served through the Domino API host gateway.
BASE="$DOMINO_API_HOST/api/taxonomy/v1"
H="Authorization: Bearer $TOKEN"
```

## Key Concepts

| Concept | Description |
|---------|-------------|
| **Namespace** | Top-level group for **tags** (e.g. `Indication`, `Analysis`). Has `label`, optional `description`, and `allowMultipleAssignments` flag. |
| **Tag** | A label inside a namespace. Can be hierarchical via `parentId` (e.g. `Clinical_Data / SDTM`). Has `label`, `namespaceId`, optional `description` and `parentId`, and `status` (`active` / `inactive`). |
| **Property** | A typed metadata *field* — has a `label`, a `type` (`text`, `number`, `date`, `boolean`, `url`, `user`, `organization`, `user_or_org`, `select`, `multi_select`), a set of `allowedEntities`, an optional `groupName`, and (for select types) `allowedValues`. See [PROPERTIES.md](./PROPERTIES.md). |
| **Property value** | The typed value a property holds for one specific entity, set via `PATCH /property-values`. See [PROPERTY-VALUES.md](./PROPERTY-VALUES.md). |
| **Property group** | Free-form `groupName` string that buckets properties in the values view; ungrouped properties fall under `Miscellaneous`. |
| **EntityType** | Enum over entities. Tags apply to `dataset`, `project`, `project_template`, `model`, `app`, `netapp_volume`. Properties additionally support the *versioned* types `model_version` and `app_version`. |
| **`allowMultipleAssignments`** | (Tags) When `true`, an entity can hold multiple tags from the same namespace. When `false`, applying a new tag from the namespace replaces a

Preview of the first 4,000 characters. Read full instructions →

Referenced files: 4 — file references are metadata, not package contents.