Domino Data Lab v2026.603.4
Build and ship ML on Domino
Save this plugin to Following and return to review observed changes. Follow up to 10 plugins free.
From the marketplace listing
Work with the Domino Data Lab platform from ChatGPT and Codex. Skills cover workspaces, jobs, compute environments, datasets, NetApp volumes, projects, apps, model endpoints and monitoring, MLflow experiment tracking, GenAI tracing, Flows, Spark/Ray/Dask, AI Gateway, governance, tags and properties, and the Domino Python and data SDKs. Intended for data scientists and ML engineers who already have access to a Domino deployment. The skills provide guidance and code; they don't connect to Domino on their own, so running jobs or calling APIs still requires your own Domino credentials.
Language: English · Automatically detected from descriptions.
Oct 27, 2026 · 1 saved observations
Metadata · Oct 27, 2026 · 00:55 UTC · Compared with Oct 24, 2026 · 13:54 UTC
Package or technical metadata updated
Observed in package metadata. These changes alone do not establish a new customer-facing feature.
Oct 24, 2026 · 13:54 UTC
Not present
Oct 27, 2026 · 00:55 UTC
false
From the developer’s listing and package; capabilities are not independently tested. Package observed Nov 3, 2026.
Build and ship ML on Domino
Specific limitations have not been established. Missing information does not mean unrestricted capabilities.
Pricing and account requirements have not been established from the available package declarations. Unknown pricing does not mean free access.
Help me deploy a Streamlit dashboard to Domino.
---
name: domino-ai-gateway
description: Access external LLM providers through Domino AI Gateway - a secure proxy with centralized API key management, usage monitoring, and compliance. Supports OpenAI, AWS Bedrock, Azure OpenAI, Anthropic, and more. Use when calling LLMs from Domino, configuring AI Gateway endpoints, or monitoring LLM usage and costs.
---
# Domino AI Gateway Skill
## Description
This skill helps users work with Domino AI Gateway - a secure proxy for accessing external Large Language Model (LLM) providers with centralized management, monitoring, and compliance.
## Activation
Activate this skill when users want to:
- Access LLM providers (OpenAI, AWS Bedrock, etc.) in Domino
- Configure AI Gateway endpoints
- Monitor LLM usage and costs
- Understand secure API key management
- Use LLMs in workspaces and jobs
## What is AI Gateway?
Domino AI Gateway provides:
- **Secure LLM Access**: Proxy to external LLM providers
- **Centralized API Keys**: Keys stored securely, never exposed to users
- **Usage Monitoring**: Track all LLM interactions
- **Access Control**: Granular permissions per endpoint
- **Audit Logging**: Compliance-ready logs
## Supported LLM Providers
| Provider | Models |
|----------|--------|
| OpenAI | GPT-4, GPT-4 Turbo, GPT-3.5 |
| AWS Bedrock | Claude, Titan, Llama 2 |
| Azure OpenAI | GPT-4, GPT-3.5 |
| Anthropic | Claude 3, Claude 2 |
| Google Vertex AI | PaLM, Gemini |
| Cohere | Command, Embed |
## Creating an AI Gateway Endpoint
### Via Domino UI
1. Go to **Endpoints** > **Gateway LLMs**
2. Click **Create Endpoint**
3. Configure:
- **Name**: Endpoint name (e.g., `openai-gpt4`)
- **Provider**: Select LLM provider
- **Model**: Specific model to use
- **API Key**: Provider API key (stored securely)
- **Access**: Who can use this endpoint
4. Click **Create**
### Via API
```python
# Create endpoint via Domino API
import requests, os
TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
response = requests.post(
f"{BASE}/api/aigateway/v1/endpoints",
headers={"Authorization": f"Bearer {TOKEN}"},
json={
"name": "openai-gpt4",
"provider": "openai",
"model": "gpt-4",
"providerApiKey": "sk-..."
}
)
```
## Using AI Gateway in Code
### OpenAI-Compatible Interface
AI Gateway provides an OpenAI-compatible interface:
```python
from openai import OpenAI
# Configure client to use AI Gateway
client = OpenAI(
api_key="not-needed", # Handled by AI Gateway
base_url="https://your-domino.com/api/aigateway/v1/openai"
)
# Use like standard OpenAI
response = client.chat.completions.create(
model="openai-gpt4", # Your endpoint name
messages=[
{"role": "user", "content": "Hello, how are you?"}
]
)
print(response.choices[0].message.content)
```
### With LangChain
```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="openai-gpt4", # Endpoint name
openai_api_key="not-needed",
openai_api_base="https://your-domino.com/api/aigateway/v1/openai"
)
response = llm.invoke("What is machine learning?")
print(response.content)
```
### Direct API Call
```python
import requests, os
TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
response = requests.post(
f"{BASE}/api/aigateway/v1/chat/completions",
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {TOKEN}",
},
json={
"model": "openai-gpt4",
"messages": [{"role": "user", "content": "Hello!"}]
}
)
result = response.json()
print(result["choices"][0]["message"]["content"])
```
## Access Control
### Endpoint Permissions
Configure who can use each endpoint:
- **Everyone**: All Domino users
- **Specific Users**: Named individuals
- **Organizations**: Specific Domino organizations
### Setting Permissions
1. Go to endpoint settings
2. Click **Access Control**
3. Add users or organizations
4. Save changes
## Monitoring and Logging
### View Usage
1. Go to **Endpoints** > **Gateway LLMs**
2. Click on endpoint name
3. View metrics:
- Request count
- Token usage
- Response times
- Error rates
### Download Logs
```bash
# Via UI: Endpoints > Gateway LLMs > Download logs
# Logs include:
# - Timestamp
# - User
# - Model
# - Input/Output tokens
# - Response time
# - Status
```
### Log Format
```json
{
"timestamp": "2024-01-15T10:30:00Z",
"user": "user@company.com",
"endpoint": "openai-gpt4",
"model": "gpt-4",
"inputTokens": 150,
"outputTokens": 200,
"durationMs": 1500,
"status": "success"
}
```
## Cost Management
### Track Costs
AI Gateway tracks token usage per:
- User
- Project
- Endpoint
- Time period
### Set Limits (Admin)
Admins can configure:
- Token limits per user/project
- Request rate limits
- Cost alerts
## Best Practices
### 1. Use Endpoint Names Consistently
```python
# Define endpoint once
LLM_ENDPOINT = "production-gpt4"
# Use throughout code
response = client.chat.completions.create(
model=LLM_ENDPOINT,
messages=[...]
)
```
### 2. Handle Rate Limits
```python
import time
from openai import RateLimitError
def call_llm_with_retry(messages, max_retries=3):
for attempt in range(max_retries):
try:
return client.chat.completions.create(
model="openai-gpt4",
messages=messages
)
except RateLimitError:
if attempt < max_retries - 1:
time.sleep(2 ** attempt)
else:
raise
```
### 3. Log Important Calls
```python
import logging
logger = logging.getLogger(__name__)
def query_llm(prompt):
logger.info(f"Querying LLM with prompt length: {len(prompt)}")
response = client.chat.completions.create(
model="openai-gpt4",
messages=[{"role": "user", "content": prompt}]
)
logger.info(f"Response tokens: {response.usage.total_tokens}")
return response.choices[0].message.content
```
### 4. Use Streaming for Long Responses
```python
# Streaming response
stream = client.chat.completions.create(
model="openai-gpt4",
messages=[{"role": "user", "content": "Write a long story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
## Security
### API Key Management
- Keys stored in Domino's secure vault
- Never exposed to end users
- Rotatable without code changes
### Data Privacy
- Requests logged for audit
- PII can be masked (admin config)
- Data retention policies configurable
## Troubleshooting
### Authentication Error
```
Error: 401 Unauthorized
```
- Verify Domino API key is valid
- Check endpoint access permissions
- Ensure correct base URL
### Rate Limit Exceeded
```
Error: 429 Too Many Requests
```
- Implement retry logic
- Contact admin for limit increase
- Use multiple endpoints for load distribution
### Model Not Found
```
Error: Model 'model-name' not found
```
- Verify endpoint name is correct
- Check endpoint exists and is active
- Confirm you have access to the endpoint
## API Reference
Before writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.
**Get the cluster base URL:** `$DOMINO_API_HOST` (injected by Domino into every workspace, job, and app).
Fetch the swagger spec:
```bash
# No authentication required for the public API spec
curl "$DOMINO_API_HOST/assets/public-api.json"
# Browser UI: $DOMINO_API_HOST/assets/lib/swagger-ui/index.html?url=/assets/public-api.json#/
```
**Public docs (workflow context and field explanations):**
- [API Guide](https://docs.dominodatalab.com/en/latest/api_guide/f35c19/api-guide/)
- [AI Gateway](https://docs.dominodatalab.com/en/latest/user_guide/c9ac47/ai-gateway/)
- [Monitor AI Gateway LLM logs](https://docs.dominodatalab.com/en/cloud/admin_guide/984c09/monitor-ai-gateway-large-language-model-llm-logs/)
--- name: domino-api-intro description: Introduction to calling Domino platform APIs. Covers authentication (API proxy, PAT, service account), which host to use for which API, pagination limits, and common errors. Use before any other Domino API, SDK, app, extension, or governance automation skill. compatibility: Domino 6.3 and Domino Cloud. Requires DOMINO_API_HOST; in a run use DOMINO_API_PROXY or the localhost:8899 access-token endpoint, outside a run a Personal Access Token. --- # Domino platform API intro Applies to Domino 6.3 and Domino Cloud. Read this skill first whenever you automate Domino over HTTP or the Python SDK. ## Authentication Follow https://docs.domino.ai/cloud/reference/api/domino-api-authentication for every call (API proxy in-run, access token + platform host, PAT, and service account). Legacy user API keys (`X-Domino-Api-Key`, `DOMINO_USER_API_KEY`) are described as deprecated, never recommended. Product migration docs: https://docs.domino.ai/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy | Context | Summary (details on the auth page) | |---------|-------------------------------------| | In-run | Prefer `DOMINO_API_PROXY` with **no** `Authorization` header when set; else Bearer from `http://localhost:8899/access-token` on `DOMINO_USER_HOST` (or `DOMINO_API_HOST`) | | Outside a run | Public deployment HTTPS URL + Bearer **PAT** (you) or **service account** (automation) | Host selection (gateway, governance 404, WebVFS, inference) is separate from auth mode: [HOSTS.md](./HOSTS.md). ## Reference files (this skill) | File | Use when | |------|----------| | [HOSTS.md](./HOSTS.md) | Pick base URL (gateway, WebVFS, inference, governance external URL) | | [LIMITS.md](./LIMITS.md) | Pagination, caps, deprecated env names | | [ERRORS.md](./ERRORS.md) | Retries, 502/timeouts, job status fields, datasource 500 workaround | | [SDK-MAP.md](./SDK-MAP.md) | Which plugin skill or doc covers each surface | | [API-SPECS.md](./API-SPECS.md) | OpenAPI files and route discovery | ## Next steps Use domain skills under `skills/` (python-sdk, apps, governance, datasets, and others). They assume the auth and host rules above.
Referenced files: 5 — file references are metadata, not package contents.
---
name: domino-app-init
description: Scaffold a new Domino-ready web application (Vite + React, Streamlit, Dash, Flask, or Gradio) with proxy-compatible settings, port 8888 binding, and an app.sh entry point. Use when the user wants to start, initialize, or scaffold a new app to publish on Domino, or asks for a starter app.sh for a web framework.
---
# Initialize a Domino App
Create a web application that works behind Domino's reverse proxy on the first publish.
## Inputs
Collect these before writing files. Skip any the user has already given.
1. **Framework**: `vite-react` (default), `streamlit`, `dash`, `flask`, or `gradio`.
2. **Project name**: name for the app folder and `package.json` (React only).
3. **Model API integration**: whether the app calls a Domino model endpoint, and if so its URL.
If the target directory already contains an app, stop and ask before overwriting anything.
## Steps
1. Copy the matching template from this skill's `assets/` folder into the project:
| Framework | Files to copy |
|---|---|
| `vite-react` | `assets/vite-react/vite.config.js`, `assets/vite-react/app.sh`, `assets/vite-react/.env.example` |
| `streamlit` | `assets/streamlit/app.sh` |
| `dash` | `assets/dash/app.sh`, and `assets/dash/app.py.template` saved as `app.py` |
| `flask`, `gradio` | No template; write the files from the snippets below |
2. Make `app.sh` executable (`chmod +x app.sh`).
3. Keep these invariants in every generated file:
- Bind to `0.0.0.0`, never `localhost` or `127.0.0.1`.
- Listen on port `8888` unless the user asks for another port.
- For Vite, set `base: './'` so assets resolve under the proxy prefix.
4. If the user wants Model API integration, add the endpoint URL and token as environment
variables (see `.env.example`) and never hardcode credentials.
5. Finish with the post-initialization checklist below.
## Reference snippets
### React / Vite: `vite.config.js`
```javascript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
export default defineConfig({
plugins: [react()],
base: './',
server: { host: '0.0.0.0', port: 8888, strictPort: true },
preview: { host: '0.0.0.0', port: 8888, strictPort: true },
build: { outDir: 'dist', assetsDir: 'assets' }
})
```
### React / Vite: `app.sh`
```bash
#!/bin/bash
set -e
cd /mnt/code
npm ci
npm run build
npx serve -s dist -l 8888 --no-clipboard
```
### Streamlit: `app.sh`
```bash
#!/bin/bash
set -e
streamlit run app.py \
--server.port 8888 \
--server.address 0.0.0.0 \
--server.headless true
```
### Dash: `app.py`
```python
import os
from dash import Dash, html
app = Dash(__name__)
app.layout = html.Div([html.H1("Domino Dash App")])
if __name__ == '__main__':
app.run_server(host='0.0.0.0', port=8888, debug=False)
```
### Flask: `app.py` and `app.sh`
```python
import os
from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
return "Domino Flask App"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8888, debug=False)
```
```bash
#!/bin/bash
set -e
python app.py
```
### Gradio: `app.py` (use the Flask `app.sh` above)
```python
import gradio as gr
demo = gr.Interface(fn=lambda text: text, inputs="text", outputs="text")
if __name__ == '__main__':
demo.launch(server_name='0.0.0.0', server_port=8888)
```
## Post-initialization checklist
Tell the user to:
1. Configure environment variables in the Domino project settings.
2. Set `app.sh` as the app entry point when publishing the app.
3. Test locally with `./app.sh` before deploying.
4. Confirm the app listens on port 8888.
For deployment details, CI/CD, and proxy troubleshooting, use the `domino-apps` skill. If the app
shows 404s or blank pages after publishing, use the `domino-debug-proxy` skill.
Referenced files: 6 — file references are metadata, not package contents.
---
name: domino-apps
description: Deploy web applications to Domino Data Lab with expertise in React apps (Vite) behind Domino's reverse proxy. Covers app.sh configuration, port configuration, base path handling for SPAs, CI/CD with GitHub Actions, and proxy troubleshooting. Use when deploying apps to Domino, setting up CI/CD pipelines, fixing broken routing, or configuring JavaScript frameworks for Domino's proxy.
---
# Domino Apps Skill
This skill provides comprehensive knowledge for deploying web applications to Domino Data Lab, with special focus on React applications using Vite.
## Key Concepts
### Domino App Architecture
Domino apps run in containers behind a reverse proxy that:
1. Authenticates users via Domino's auth system
2. Strips the URL prefix before forwarding to your app
3. Routes traffic to your app container
4. Handles infrastructure provisioning, routing, and resource management
**Note:** Port selection is flexible; port 8888 is no longer required. You can use any port your application prefers.
### Critical Configuration Points
1. **Host Binding**: Bind to `0.0.0.0` (not localhost) so Domino can reach your app
2. **Relative Base Path**: Use `base: './'` in Vite config for React apps
3. **app.sh**: Entry point script (launch file) that Domino executes
## Related Documentation
- [API-APPS.md](./API-APPS.md) - REST automation: v1 vs beta, publish chain, management vs app URL, inference split
- [REACT-VITE-GUIDE.md](./REACT-VITE-GUIDE.md) - Deep dive into Vite + React configuration
- [REACT-CICD.md](./REACT-CICD.md) - CI/CD setup with GitHub Actions
- [FRAMEWORKS.md](./FRAMEWORKS.md) - Streamlit, Dash, Flask configurations
- [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) - Common issues and solutions
## Quick Start
### React with Vite
```javascript
// vite.config.js
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
export default defineConfig({
plugins: [react()],
base: './', // CRITICAL for Domino proxy
server: { host: '0.0.0.0', port: 8888, strictPort: true },
preview: { host: '0.0.0.0', port: 8888, strictPort: true },
})
```
```bash
# app.sh
#!/bin/bash
set -e
cd /mnt/code
npm ci
npm run build
npx serve -s dist -l 8888 --no-clipboard
```
### Streamlit
```bash
# app.sh
#!/bin/bash
streamlit run app.py --server.port 8888 --server.address 0.0.0.0
```
### Dash/Flask
```bash
# app.sh
#!/bin/bash
python app.py # Must bind to 0.0.0.0:8888
```
## Environment Variables
Domino provides these environment variables to your app:
| Variable | Description |
|----------|-------------|
| `DOMINO_PROJECT_NAME` | Current project name |
| `DOMINO_PROJECT_OWNER` | Project owner username |
| `DOMINO_RUN_ID` | Current run identifier |
| `DOMINO_STARTING_USERNAME` | User who started the app |
## Inter-App Communication
Domino apps can communicate with each other using bearer token authentication. This is useful for:
- Calling Model APIs from an app
- App-to-app service calls
- Accessing internal Domino services
### Getting the Access Token
Domino provides an access token service at `localhost:8899`:
```python
import requests
# Get bearer token for inter-app communication
API_TOKEN = requests.get("http://localhost:8899/access-token").text
```
### Making Authenticated Requests
```python
import requests
# 1. Get access token from Domino's token service
API_TOKEN = requests.get("http://localhost:8899/access-token").text
# 2. Set up headers with bearer token
headers = {
"Authorization": f"Bearer {API_TOKEN}",
"Content-Type": "application/json"
}
# 3. Make request to another Domino app or service
payload = {
"query": "Your request data here"
}
try:
resp = requests.post(
"https://your-domino-instance/apps-internal/APP_ID/endpoint",
json=payload,
headers=headers,
timeout=100
)
resp.raise_for_status()
data = resp.json()
print(data)
except requests.exceptions.RequestException as err:
print("API call failed:", err)
```
### Key Points
- **Token endpoint**: `http://localhost:8899/access-token` (only accessible from within Domino)
- **Token type**: Bearer token for Authorization header
- **Use cases**: Model API calls, app-to-app communication, internal services
- **Timeout**: Set appropriate timeouts for long-running requests
## REST automation (apps API)
For create, publish, start, and stop from code, read [API-APPS.md](./API-APPS.md). Use `/api/apps/v1` for writes; use `/api/apps/beta` only for instance logs, views, and runtime telemetry.
## API Reference
OpenAPI and route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md) (public routes section).
### API Version Prioritization Rules
- **Prioritize the Apps v1 API (`/api/apps/v1`)** for all apps creation, update, preview, publication, start and stop workflows.
- **The Beta API (`/api/apps/beta`) is deprecated** for new automation and maintained for backward compatibility and runtime reads. Do not build new functionality against `beta` endpoints if a `v1` equivalent exists.
- **Instance Read Operations Exception**: Instance read operations (such as fetching active container logs, `realTimeLogs`, `views`, listing active running instances, or issuing an instance-level `DELETE`) were not migrated to the `v1` spec. They remain under `/api/apps/beta`.
*Rule of Thumb:* Use `v1` for automating write or publish actions. Fall back to `beta` for streaming runtime telemetry like logs or views.
**Authentication:** https://docs.domino.ai/cloud/reference/api/domino-api-authentication . In-run: `DOMINO_API_PROXY` (no Authorization header) or `DOMINO_USER_HOST` + `http://localhost:8899/access-token`. Outside run: public deployment URL + PAT/SA. Do not use API keys.
**Get the cluster base URL:** `$DOMINO_USER_HOST` or `$DOMINO_API_HOST` in-run; confirm apps routes with `GET .../api/apps/v1/apps?limit=1`.
**Public docs:**
- https://docs.domino.ai/cloud/platform-capabilities/features/apps
**Blueprint Reference:**
- [React CI/CD Blueprint](https://github.com/dominodatalab/domino-blueprints/tree/main/React-app-deployment-with-CICD)
Referenced files: 5 — file references are metadata, not package contents.
---
name: domino-data-connectivity
description: Connect Domino workloads to external data sources including AWS S3 (via Mountpoint CSI driver), credential propagation with AWS IRSA and Azure Entra ID, and External Data Volumes. Use when configuring S3 access, setting up credential propagation, or connecting to cloud data sources from Domino.
---
# Domino Data Connectivity Skill
This skill provides comprehensive knowledge for connecting Domino workloads to external data sources, including AWS S3, Azure storage, and credential propagation.
## Safety Rules
The setup guides in this skill are platform-administrator procedures that change cloud IAM and
Kubernetes cluster state. When using them:
- **Explain, don't execute.** Show `aws iam`, `eksctl`, `helm`, `kubectl`, `az` commands and
IAM/RBAC policies for the user to review and run. Do not run them yourself unless the user
explicitly asks you to run that specific command.
- **Least privilege.** Scope every policy to named buckets, endpoints, or resources. Never
propose `"Resource": "*"` or wildcard actions such as `s3:*`.
- **No secrets in output.** Never print, log, or echo access keys, tokens, or client secrets, and
never ask the user to paste one into the chat.
- **Read-only troubleshooting.** Prefer `get`/`describe` commands over `exec` into running pods.
## Key Concepts
### Data Access Options in Domino
| Option | Use Case |
|--------|----------|
| **Datasets** | Project-level data storage |
| **Data Sources** | External database connections |
| **External Data Volumes (EDV)** | Mount external storage as volumes |
| **S3 Mountpoint** | Direct S3 access as file system |
| **Credential Propagation** | Pass user identity to cloud services |
### Credential Propagation Methods
| Method | Cloud | Description |
|--------|-------|-------------|
| **IRSA** | AWS | IAM Role for Service Accounts via OIDC |
| **Azure Entra ID** | Azure | User-based credential propagation |
## Related Documentation
- [S3-MOUNTPOINT.md](./S3-MOUNTPOINT.md) - AWS S3 as local file system
- [AWS-IRSA.md](./AWS-IRSA.md) - AWS IAM Role for Service Accounts
- [AZURE-CREDENTIALS.md](./AZURE-CREDENTIALS.md) - Azure Entra ID integration
## Quick Start
### Accessing S3 Data
With Mountpoint S3 configured, access S3 as a local file system:
```python
import pandas as pd
# S3 data appears as local files
df = pd.read_parquet("/mnt/s3-data/datasets/sales.parquet")
```
### Using IRSA for AWS Services
With IRSA configured, AWS SDK uses automatic credentials:
```python
import boto3
# No explicit credentials needed - IRSA provides them
s3 = boto3.client('s3')
response = s3.list_objects_v2(Bucket='my-bucket')
```
### Accessing External Data Volumes
EDVs are mounted at configured paths:
```python
# Read from external volume
with open("/mnt/external-data/config.json") as f:
config = json.load(f)
```
## When to Use Each Option
### Use S3 Mountpoint When:
- Working with large datasets stored in S3
- Need file system interface to S3
- Want to avoid EFS costs for large data
- Require multi-region data access
### Use IRSA When:
- Need AWS service access from Domino workloads
- Policy prohibits long-lived credentials
- Require user-level audit trails
- Need cross-account role assumption
### Use Azure Entra When:
- Working with Azure data services
- Need OAuth-based authentication
- Require user-level access control
- Need centralized RBAC
## Blueprint References
- [S3 Mountpoint Blueprint](https://domino.ai/resources/blueprints/transforming-s3-into-local-storage)
- [AWS IRSA Blueprint](https://domino.ai/resources/blueprints/aws-irsa)
- [Azure Credential Propagation Blueprint](https://domino.ai/resources/blueprints/domino-azure-credential-propagation)
## GitHub Repository
Implementation templates available at:
https://github.com/dominodatalab/domino-blueprints
Referenced files: 3 — file references are metadata, not package contents.
---
name: domino-data-sdk
description: Use the domino-data Python SDK (dominodatalab-data) for programmatic data access in Domino. Covers DataSourceClient for SQL queries and object storage, DatasetClient for dataset files, TrainingSets for ML data versioning, Feature Store, and VectorDB (Pinecone) integration. Use when querying data sources, downloading datasets, managing training sets, or working with vector databases in Domino.
---
# Domino Data SDK Skill
This skill provides comprehensive knowledge for working with the `domino-data` Python SDK (`dominodatalab-data`) - the official library for Domino's Access Data features.
## Installation
```bash
# Via pip
pip install -U dominodatalab-data
# Via Poetry
poetry add dominodatalab-data
# In Domino environment (requirements.txt)
dominodatalab-data>=6.0.0
```
## Key Components
| Module | Purpose |
|--------|---------|
| `DataSourceClient` | Query SQL databases and access object stores |
| `DatasetClient` | Read files from Domino Datasets |
| `TrainingSets` | Version and manage ML training data |
| `Feature Store` | Manage ML features with Git integration |
| `VectorDB` | Pinecone vector database integration |
## Related Documentation
- [DATA-SOURCES.md](./DATA-SOURCES.md) - SQL queries and object storage
- [DATASETS.md](./DATASETS.md) - Dataset file operations
- [TRAINING-SETS.md](./TRAINING-SETS.md) - Training data versioning
- [VECTORDB.md](./VECTORDB.md) - Pinecone integration
## Quick Start
### Query a Data Source
```python
from domino_data.data_sources import DataSourceClient
# Initialize client (auto-configured in Domino)
client = DataSourceClient()
# Get a data source by name
ds = client.get_datasource("my-redshift-db")
# Execute SQL query
result = ds.query("SELECT * FROM customers WHERE region = 'US'")
# Convert to pandas DataFrame
df = result.to_pandas()
# Or save to parquet
result.to_parquet("output.parquet")
```
### Access Object Storage
```python
from domino_data.data_sources import DataSourceClient
client = DataSourceClient()
ds = client.get_datasource("my-s3-bucket")
# List objects
objects = ds.list_objects(prefix="data/", page_size=100)
# Download a file
ds.download_file("data/input.csv", "local_input.csv")
# Upload a file
ds.put("data/output.csv", open("results.csv", "rb").read())
# Get signed URL
url = ds.get_key_url("data/file.csv", is_read_write=False)
```
### Read from Datasets
```python
from domino_data.datasets import DatasetClient
client = DatasetClient()
# Get dataset by name
dataset = client.get_dataset("training-data")
# List files
files = dataset.list_files(prefix="images/")
# Download file
dataset.download("model.pkl", "local_model.pkl", max_workers=4)
# Get file content as bytes
content = dataset.get("config.json")
```
### Training Sets
```python
from domino_data.training_sets import (
create_training_set_version,
get_training_set,
list_training_sets
)
import pandas as pd
# Create training set version from DataFrame
df = pd.DataFrame({
"id": [1, 2, 3],
"feature_a": [0.1, 0.2, 0.3],
"label": [1, 0, 1]
})
version = create_training_set_version(
training_set_name="customer-churn",
df=df,
key_columns=["id"],
description="Initial training data"
)
# Get training set
ts = get_training_set("customer-churn")
# List all training sets
all_sets = list_training_sets()
```
### Vector Database (Pinecone)
```python
from domino_data.vectordb import (
domino_pinecone3x_init_params,
domino_pinecone3x_index_params
)
from pinecone import Pinecone
# Initialize Pinecone client with Domino credentials
init_params = domino_pinecone3x_init_params("my-pinecone-ds")
pc = Pinecone(**init_params)
# Get index parameters
index_params = domino_pinecone3x_index_params("my-pinecone-ds", "embeddings")
index = pc.Index(**index_params)
# Query vectors
results = index.query(
vector=[0.1, 0.2, 0.3, ...],
top_k=10,
include_metadata=True
)
```
## Authentication
The library auto-configures authentication inside Domino workspaces and jobs using injected environment variables:
```python
# Environment variables used automatically inside Domino:
# DOMINO_TOKEN_FILE - Token file location (preferred, short-lived)
# DOMINO_API_PROXY - API proxy URL
# DOMINO_DATA_API_GATEWAY - Data API gateway (default: http://127.0.0.1:8766)
#
# DOMINO_USER_API_KEY - Legacy API key (deprecated, will be removed)
```
For external use (e.g., CI/CD outside a Domino execution):
> **Note:** `DOMINO_USER_API_KEY` is deprecated and will be removed in a future Domino release. Prefer running data-SDK code from inside a Domino workspace or job where token-based auth is injected automatically.
```python
import os
os.environ["DOMINO_USER_API_KEY"] = "your-api-key" # deprecated
os.environ["DOMINO_API_HOST"] = "https://your-domino.com"
from domino_data.data_sources import DataSourceClient
client = DataSourceClient()
```
## Error Handling
```python
from domino_data.data_sources import DominoError, UnauthenticatedError
try:
result = ds.query("SELECT * FROM table")
except UnauthenticatedError:
print("Authentication failed - check API key")
except DominoError as e:
print(f"Domino error: {e}")
```
## Best Practices
1. **Use within Domino**: Auth is automatic in workspaces/jobs
2. **Parallel downloads**: Use `max_workers` for large files
3. **Pagination**: Use `page_size` when listing many objects
4. **Training Sets**: Version your training data for reproducibility
5. **Connection reuse**: Reuse client instances when possible
## Package Info
- **PyPI**: `dominodatalab-data`
- **GitHub**: https://github.com/dominodatalab/domino-data
- **License**: Apache 2.0
- **Python**: 3.8+
Referenced files: 4 — file references are metadata, not package contents.
---
name: domino-datasets
description: Work with Domino Datasets - high-performance, versioned filesystem storage. Covers dataset creation, snapshots for versioning, sharing across projects, mounting paths (/domino/datasets/), and performance optimization. Use when managing data storage, creating reproducible data versions, or sharing data between projects.
---
# Domino Datasets Skill
## Description
This skill helps users work with Domino Datasets - high-performance, versioned filesystem storage for data science projects.
## Activation
Activate this skill when users want to:
- Create or manage Domino Datasets
- Work with dataset snapshots and versioning
- Share data between projects
- Access large datasets efficiently
- Understand dataset paths and mounting
## What is a Domino Dataset?
A Domino Dataset is:
- **High-performance storage**: Network filesystem optimized for data science
- **Versioned**: Create snapshots for reproducibility
- **Shareable**: Access across projects
- **Scalable**: No file size or count limits
- **Persistent**: Data persists across executions
## Creating a Dataset
### Via Domino UI
1. Navigate to your project
2. Go to **Data** > **Domino Datasets**
3. Click **Create New Dataset**
4. Enter:
- **Name**: Dataset name (e.g., `training-data`)
- **Description**: What the dataset contains
5. Click **Create**
### Via Python SDK
```python
from domino import Domino
domino = Domino("project-owner/project-name")
# Create a new dataset
dataset = domino.datasets_create(
name="training-data",
description="Training data for classification model"
)
```
## Dataset Paths
Dataset paths differ based on your **project type**. Domino has two project types with different mount structures.
### DFS (Domino File System) Projects
DFS projects use `/domino` as the root:
```
/domino
|--/datasets
|--/local <== Local datasets and snapshots
|--/clapton <== Read-write dataset for owner and editor, read-only for reader
|--/mingus <== Read-write dataset for owner and editor, read-only for reader
|--/snapshots <== Snapshot folder organized by dataset
|--/clapton <== Read-write for owner and editor, read-only for reader
|--/tag1 <== Mounted under latest tag
|--/1 <== Always mounted under the snapshot number
|--/2
|--/mingus
|--/tag2
|--/1
|--/2
|--/ella <== Read-write shared dataset for owner and editor, Read-only for reader
|--/davis <== Read-write shared dataset for owner and editor, Read-only for reader
|--/snapshots <== Shared datasets snapshots organized by dataset
|--/ella <== Read-write for owner and editor, read-only for reader
|--/tag3 <== Mounted under latest tag
|--/1 <== Always mounted under the snapshot number
|--/2
|--/davis
|--/tag4
|--/1
|--/2
```
| Dataset Type | Path |
|-------------|------|
| Local datasets | `/domino/datasets/local/{dataset-name}/` |
| Local snapshots | `/domino/datasets/local/snapshots/{dataset-name}/{tag-or-number}/` |
| Shared datasets | `/domino/datasets/{dataset-name}/` |
| Shared snapshots | `/domino/datasets/snapshots/{dataset-name}/{tag-or-number}/` |
### Git-Based Projects
Git-based projects use `/mnt` as the root:
```
/mnt
|--/data <== Local datasets and snapshots
|--/clapton <== Read-write dataset for owner and editor, read-only for reader
|--/mingus <== Read-write dataset for owner and editor, read-only for reader
|--/snapshots <== Snapshot folder organized by dataset
|--/clapton <== Read-write for owner and editor, read-only for reader
|--/tag1 <== Mounted under latest tag
|--/1 <== Always mounted under the snapshot number
|--/2
|--/mingus
|--/tag2
|--/1
|--/2
|--/imported
|--/data
|--/ella <== Read-write shared dataset for owner and editor, read-only for reader
|--/davis <== Read-write shared dataset for owner and editor, read-only for reader
|--/snapshots <== Shared dataset snapshots organized by dataset
|--/ella <== Read-write for owner and editor, read-only for reader
|--/tag3 <== Mounted under latest tag
|--/1 <== Always mounted under the snapshot number
|--/2
|--/davis
|--/tag4
|--/1
|--/2
```
| Dataset Type | Path |
|-------------|------|
| Local datasets | `/mnt/data/{dataset-name}/` |
| Local snapshots | `/mnt/data/snapshots/{dataset-name}/{tag-or-number}/` |
| Shared datasets | `/mnt/imported/data/{dataset-name}/` |
| Shared snapshots | `/mnt/imported/data/snapshots/{dataset-name}/{tag-or-number}/` |
### How to Identify Your Project Type
Check which paths exist in your execution:
```python
import os
if os.path.exists("/domino/datasets"):
print("DFS Project")
dataset_root = "/domino/datasets/local"
elif os.path.exists("/mnt/data"):
print("Git-Based Project")
dataset_root = "/mnt/data"
```
### Permissions
Both project types follow the same permission model:
- **Owners/Editors**: Read-write access to datasets
- **Readers**: Read-only access
### Example: Reading Data
```python
import pandas as pd
# Git-Based Project
df = pd.read_csv("/mnt/data/training-data/customers.csv")
# DFS Project
df = pd.read_csv("/domino/datasets/local/training-data/customers.csv")
# List files
import os
files = os.listdir("/mnt/data/training-data/") # Git-Based
files = os.listdir("/domino/datasets/local/training-data/") # DFS
```
## Uploading Data
### Via Domino UI
1. Go to dataset page
2. Click **Upload**
3. Select files (up to 50GB or 50,000 files via UI)
4. Click **Upload**
### Via Domino CLI (Large Uploads)
```bash
# For large uploads, use CLI
domino upload /local/path/to/data /mnt/data/training-data/
```
### Via Code in Workspace
```python
import shutil
# Copy from local to dataset
shutil.copy("local_file.csv", "/mnt/data/training-data/")
# Write directly
df.to_csv("/mnt/data/training-data/processed.csv", index=False)
```
## Snapshots
### What is a Snapshot?
A snapshot is a read-only, immutable version of your dataset at a point in time. Use snapshots for:
- Reproducibility
- Versioning training data
- Rolling back to previous states
### Create a Snapshot
```python
# Via Python SDK
snapshot = domino.datasets_snapshot(
dataset_name="training-data",
tag="v1.0"
)
```
Or via UI:
1. Go to dataset page
2. Click **Create Snapshot**
3. Add optional tag (e.g., `v1.0`, `production`)
### Access Snapshots
```python
# Latest snapshot
df = pd.read_csv("/mnt/data/training-data/data.csv")
# Specific tagged snapshot
df = pd.read_csv("/mnt/data/training-data@v1.0/data.csv")
```
### Snapshot Limits
- Default limit: 20 snapshots per dataset
- Configurable by admins
- Oldest snapshots auto-deleted when limit reached
## Tags
### What are Tags?
Tags provide friendly names for snapshots:
- `production`: Current production data
- `v1.0`, `v2.0`: Version numbers
- `2024-01-15`: Date-based tags
### Move Tags
Tags can be moved to different snapshots:
```python
# Move 'production' tag to latest snapshot
domino.datasets_tag(
dataset_name="training-data",
snapshot_id="snapshot-123",
tag="production"
)
```
## Sharing Datasets
### Within Organization
1. Go to dataset settings
2. Set visibility to **Organization**
3. Other projects can mount the dataset
### Cross-Project Access
```python
# Import dataset from another project
# Configured in project settings
df = pd.read_csv("/mnt/data/shared-dataset/data.csv")
```
## Best Practices
### 1. Use Appropriate Storage
| Data Type | Storage |
|-----------|---------|
| Large training data | Domino Dataset |
| Model artifacts | `/mnt/artifacts/` |
| Code | Git/Project files |
| Temporary files | `/tmp/` |
### 2. Organize Data
```
/mnt/data/my-dataset/
├── raw/
│ ├── customers.csv
│ └── transactions.csv
├── processed/
│ ├── features.parquet
│ └── labels.parquet
└── metadata/
└── schema.json
```
### 3. Use Efficient Formats
```python
# Parquet for tabular data (faster, smaller)
df.to_parquet("/mnt/data/dataset/data.parquet")
# Feather for pandas DataFrames
df.to_feather("/mnt/data/dataset/data.feather")
# HDF5 for numerical arrays
import h5py
with h5py.File("/mnt/data/dataset/data.h5", "w") as f:
f.create_dataset("features", data=features)
```
### 4. Document Data
Include README and schema:
```python
# Write metadata
metadata = {
"created": "2024-01-15",
"source": "Customer database",
"columns": {"id": "int", "name": "string", "value": "float"}
}
with open("/mnt/data/dataset/metadata.json", "w") as f:
json.dump(metadata, f)
```
### 5. Snapshot Before Changes
```python
# Create snapshot before processing
domino.datasets_snapshot(
dataset_name="training-data",
tag="pre-processing"
)
# Then modify data
process_data()
```
## Reading Large Datasets
### Chunked Reading
```python
# Read in chunks
chunks = pd.read_csv(
"/mnt/data/dataset/large_file.csv",
chunksize=100000
)
for chunk in chunks:
process(chunk)
```
### Lazy Loading with Dask
```python
import dask.dataframe as dd
# Read without loading into memory
df = dd.read_parquet("/mnt/data/dataset/large_data.parquet")
# Process lazily
result = df.groupby("category").mean().compute()
```
### Memory Mapping
```python
import numpy as np
# Memory-map large arrays
data = np.memmap(
"/mnt/data/dataset/features.dat",
dtype='float32',
mode='r',
shape=(1000000, 100)
)
```
## Troubleshooting
### Dataset Not Found
- Verify dataset name is correct
- Check dataset is mounted to project
- Confirm you have access permissions
### Permission Denied
- Check project role (need Contributor+)
- Verify dataset sharing settings
- Contact dataset owner
### Slow Performance
- Use efficient file formats (Parquet > CSV)
- Read only needed columns
- Use chunked/lazy loading for large files
### Snapshot Failed
- Check disk quota
- Verify no files are open/locked
- Check snapshot limit not reached
## Documentation Reference
- [Create and manage Datasets](https://docs.dominodatalab.com/en/latest/user_guide/0a8d11/create-and-manage-datasets/)
- [Version data with Snapshots](https://docs.dominodatalab.com/en/cloud/user_guide/dbdbff/version-data-with-snapshots/)
- [Datasets best practices](https://archive.docs.dominodatalab.com/en/4.4/user_guide/a222c9/datasets-best-practices/)
---
name: domino-debug-proxy
description: Diagnose and fix Domino reverse-proxy and routing problems in published web apps by checking vite.config.js, package.json, and app.sh for wrong ports, localhost binding, absolute base paths, missing SPA fallback, or missing build steps. Use when a Domino app shows connection refused, 404s on CSS/JS assets, a blank page, or broken deep links.
---
# Debug Domino App Proxy Issues
Find the configuration mistakes that break apps behind Domino's reverse proxy, report them, and
fix them with the user's agreement.
## Steps
1. Locate the app's config files: `vite.config.js`/`vite.config.ts`, `package.json`, `app.sh`,
and any framework entry point (`app.py`, `server.js`).
2. Run every check below and record each result as pass or fail.
3. Print a report in the format shown under "Report format".
4. List the recommended fixes and ask whether to apply them. Apply only what the user approves.
5. For anything you can't fix from config, give the manual investigation steps.
## Checks
| # | Check | Pass condition |
|---|---|---|
| 1 | Port | The server listens on `8888`; Vite also sets `strictPort: true` |
| 2 | Base path | Vite sets `base: './'`; other frameworks use relative asset paths |
| 3 | Host binding | The server binds to `0.0.0.0`, not `localhost` or `127.0.0.1` |
| 4 | `app.sh` | It `cd`s to the code directory, runs the build, and then serves the output |
| 5 | SPA fallback | Static serving uses `npx serve -s dist` (the `-s` flag) |
| 6 | Asset references | No hardcoded root-absolute paths such as `/assets/...` or `fetch('/api/...')` |
## Common issues
| Symptom | Cause | Fix |
|---|---|---|
| Connection refused | Wrong port, for example 3000 | `server: { port: 8888, strictPort: true }` |
| 404 on CSS/JS files | `base` is `/` or missing | `base: './'` in `defineConfig` |
| App unreachable from the browser | Bound to localhost | `server: { host: '0.0.0.0' }` |
| Deep links return 404 | `serve` without `-s` | `npx serve -s dist -l 8888` |
| Blank page or stale content | `app.sh` has no build step | Add `npm run build` before `serve` |
A corrected `app.sh` for a Vite app:
```bash
#!/bin/bash
set -e
cd /mnt/code
npm ci
npm run build
npx serve -s dist -l 8888
```
## Report format
```
Domino Proxy Debug Report
=========================
Checking: vite.config.js
✅ Port: 8888
✅ Base path: './'
✅ Host: 0.0.0.0
Checking: app.sh
✅ Working directory: /mnt/code
✅ Build command: npm run build
❌ Serve command missing -s flag
Summary: 1 issue found
Recommended fixes:
1. Update the app.sh serve command:
- Current: npx serve dist -l 8888
+ Fixed: npx serve -s dist -l 8888
```
## Manual investigation
If the config is correct but the app still fails:
1. Open the browser developer tools console and look for failed module or asset loads.
2. In the Network tab, check whether failing requests drop the app's proxy URL prefix (the part
of the path before the app's own routes).
3. Check the app logs on the app's page in Domino.
4. Confirm the app was republished after the last code change.
For general deployment guidance, use the `domino-apps` skill.
---
name: domino-distributed-computing
description: Work with distributed computing frameworks in Domino including Apache Spark, Ray, and Dask clusters. Covers cluster configuration, on-demand clusters, choosing between frameworks, PySpark usage, and scaling workloads. Use when processing large datasets, parallel ML training, or running distributed compute jobs.
---
# Domino Distributed Computing Skill
## Description
This skill helps users work with distributed computing frameworks in Domino - Spark, Ray, and Dask clusters for scaling compute-intensive workloads.
## Activation
Activate this skill when users want to:
- Run Spark, Ray, or Dask clusters in Domino
- Scale data processing or ML training
- Configure distributed cluster settings
- Understand when to use each framework
## Supported Frameworks
| Framework | Best For |
|-----------|----------|
| **Apache Spark** | Large-scale data processing, SQL, ETL |
| **Ray** | Distributed ML, hyperparameter tuning, RL |
| **Dask** | Parallel pandas, NumPy at scale |
| **MPI** | Scientific computing, HPC workloads |
## When to Use Each Framework
### Spark
- Processing terabyte-scale data
- SQL analytics on big data
- ETL pipelines
- Structured data processing
### Ray
- Distributed model training
- Hyperparameter optimization
- Reinforcement learning
- Generic Python parallelization
### Dask
- Scaling pandas workflows
- Parallel NumPy operations
- Lazy evaluation needed
- Familiar pandas/NumPy API preferred
## Launching On-Demand Clusters
### Via Domino UI
1. Start a workspace or job
2. Check **Attach compute cluster**
3. Select:
- **Cluster Type**: Spark, Ray, or Dask
- **Worker Count**: Number of workers
- **Hardware Tier**: Resources per worker
- **Auto-scaling**: Enable/disable
4. Launch
### Via Python SDK
```python
from domino import Domino
domino = Domino("project-owner/project-name")
# Start workspace with Spark cluster
workspace = domino.workspace_start(
hardware_tier_name="medium",
cluster_config={
"clusterType": "Spark",
"workerCount": 4,
"workerHardwareTier": "medium",
"masterHardwareTier": "medium"
}
)
```
## Apache Spark
### Connecting to Spark
```python
from pyspark.sql import SparkSession
# Domino auto-configures Spark
spark = SparkSession.builder.getOrCreate()
# Check configuration
print(f"Spark version: {spark.version}")
print(f"Executors: {spark.sparkContext.defaultParallelism}")
```
### Reading Data
```python
# Read CSV
df = spark.read.csv("/mnt/data/dataset/data.csv", header=True, inferSchema=True)
# Read Parquet
df = spark.read.parquet("/mnt/data/dataset/")
# Read from database
df = spark.read.jdbc(
url="jdbc:postgresql://host:5432/db",
table="schema.table",
properties={"user": "user", "password": "pass"}
)
```
### Processing Data
```python
from pyspark.sql import functions as F
# Transformations
result = df.filter(F.col("value") > 100) \
.groupBy("category") \
.agg(F.mean("value").alias("avg_value")) \
.orderBy("avg_value", ascending=False)
# Show results
result.show()
```
### Machine Learning with Spark MLlib
```python
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml import Pipeline
# Prepare features
assembler = VectorAssembler(
inputCols=["feature1", "feature2", "feature3"],
outputCol="features"
)
# Create model
rf = RandomForestClassifier(
featuresCol="features",
labelCol="label",
numTrees=100
)
# Build pipeline
pipeline = Pipeline(stages=[assembler, rf])
model = pipeline.fit(train_df)
predictions = model.transform(test_df)
```
### Writing Results
```python
# Write Parquet (recommended)
result.write.parquet("/mnt/artifacts/output/", mode="overwrite")
# Write CSV
result.write.csv("/mnt/artifacts/output.csv", header=True)
```
## Ray
### Connecting to Ray
```python
import ray
# Domino auto-initializes Ray
# Or manually connect
ray.init(address="auto")
print(f"Cluster resources: {ray.cluster_resources()}")
```
### Parallel Tasks
```python
import ray
@ray.remote
def process_item(item):
# Your processing logic
return item * 2
# Run in parallel
items = [1, 2, 3, 4, 5]
futures = [process_item.remote(item) for item in items]
results = ray.get(futures)
print(results) # [2, 4, 6, 8, 10]
```
### Distributed Training with Ray Train
```python
from ray import train
from ray.train import ScalingConfig
from ray.train.torch import TorchTrainer
def train_func():
# Training logic
model = create_model()
for epoch in range(10):
train_epoch(model)
train.report({"loss": loss})
trainer = TorchTrainer(
train_func,
scaling_config=ScalingConfig(num_workers=4, use_gpu=True)
)
result = trainer.fit()
```
### Hyperparameter Tuning with Ray Tune
```python
from ray import tune
from ray.tune import CLIReporter
def objective(config):
# Training with hyperparameters
model = train_model(
learning_rate=config["lr"],
batch_size=config["batch_size"]
)
return {"accuracy": accuracy}
analysis = tune.run(
objective,
config={
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([32, 64, 128])
},
num_samples=20,
progress_reporter=CLIReporter()
)
print(f"Best config: {analysis.best_config}")
```
## Dask
### Connecting to Dask
```python
from dask.distributed import Client
# Domino auto-configures Dask
client = Client()
print(f"Dashboard: {client.dashboard_link}")
print(f"Workers: {len(client.scheduler_info()['workers'])}")
```
### Dask DataFrames (Parallel pandas)
```python
import dask.dataframe as dd
# Read large CSV files
df = dd.read_csv("/mnt/data/dataset/*.csv")
# Parallel operations (lazy)
result = df.groupby("category")["value"].mean()
# Execute
computed_result = result.compute()
```
### Dask Arrays (Parallel NumPy)
```python
import dask.array as da
# Create large array
x = da.random.random((100000, 100000), chunks=(1000, 1000))
# Operations (lazy)
result = x.mean()
# Compute
value = result.compute()
```
### Dask ML
```python
from dask_ml.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
# Distributed hyperparameter search
param_grid = {
"n_estimators": [100, 200, 300],
"max_depth": [10, 20, 30]
}
grid_search = GridSearchCV(
RandomForestClassifier(),
param_grid,
cv=3
)
grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")
```
## GPU Clusters
### Spark RAPIDS
```python
# Use GPU-accelerated Spark
spark = SparkSession.builder \
.config("spark.rapids.sql.enabled", "true") \
.getOrCreate()
# Operations automatically use GPU
df = spark.read.parquet("/mnt/data/large_dataset/")
result = df.groupBy("category").agg({"value": "mean"})
```
### Ray with GPUs
```python
@ray.remote(num_gpus=1)
def train_on_gpu():
import torch
device = torch.device("cuda")
# GPU training logic
return model
# Run on GPU workers
futures = [train_on_gpu.remote() for _ in range(4)]
```
## Autoscaling
### Enable Autoscaling
Configure clusters to scale based on workload:
```python
cluster_config = {
"clusterType": "Spark",
"workerCount": 2,
"maxWorkerCount": 10, # Scale up to 10
"autoScaling": True
}
```
### Monitor Scaling
View cluster status in Domino UI or via dashboard URLs.
## Best Practices
### 1. Choose Right Framework
- SQL/ETL: Spark
- ML/Parallel Python: Ray
- Pandas at scale: Dask
### 2. Right-size Clusters
- Start small, scale up
- Monitor resource usage
- Use autoscaling when unsure
### 3. Data Locality
```python
# Keep data close to compute
# Use Domino Datasets or cloud storage in same region
df = spark.read.parquet("/mnt/data/dataset/")
```
### 4. Persist Intermediate Results
```python
# Cache frequently used DataFrames
df.cache()
df.persist()
```
## Troubleshooting
### Cluster Won't Start
- Check hardware tier availability
- Verify cluster environment builds
- Review cluster logs
### Out of Memory
- Increase worker memory
- Add more workers
- Optimize code (reduce shuffles)
### Slow Performance
- Check data locality
- Review partition sizes
- Monitor cluster dashboard
## Documentation Reference
- [On-demand distributed computing](https://docs.dominodatalab.com/en/latest/user_guide/8b4418/on-demand-distributed-computing/)
- [Spark, Dask, Ray comparison](https://domino.ai/blog/spark-dask-ray-choosing-the-right-framework)
- [Access data with Dask](https://docs.dominodatalab.com/en/latest/user_guide/0919a6/access-data-with-dask/)
---
name: domino-environments
description: Create and customize Domino Compute Environments - Docker containers defining tools, packages, and configurations. Covers Dockerfile customization, package installation, IDE configuration, DSE (Domino Standard Environments), and troubleshooting build failures. Use when installing dependencies, customizing environments, or fixing environment issues.
---
# Domino Compute Environments Skill
## Description
This skill helps users create, customize, and manage Domino Compute Environments - Docker-based containers that define the tools, packages, and configurations for workspaces, jobs, and other executions.
## Activation
Activate this skill when users want to:
- Create or customize a compute environment
- Install packages or dependencies
- Configure Dockerfile instructions
- Understand environment best practices
- Troubleshoot environment build issues
## What is a Compute Environment?
A Domino Compute Environment is a Docker container image that contains:
- Operating system and base tools
- Programming languages (Python, R)
- IDEs (Jupyter, VS Code, RStudio)
- Libraries and packages
- Custom configurations
## Domino Standard Environments (DSEs)
Domino provides pre-built environments with common tools:
| Environment | Includes |
|-------------|----------|
| Domino Standard Environment | Python 3.9, R 4.1, Jupyter, VS Code |
| Domino Spark Environment | Spark 3.x, PySpark |
| Domino Ray Environment | Ray for distributed computing |
| Domino GPU Environment | CUDA, cuDNN, GPU libraries |
## Creating a Custom Environment
### Via Domino UI
1. Go to **Environments** in Domino
2. Click **Create Environment**
3. Configure:
- **Name**: Descriptive name
- **Base Image**: Start from DSE or custom image
- **Dockerfile Instructions**: Add customizations
4. Click **Build**
### Dockerfile Instructions
Add instructions to customize the environment. Do NOT include `FROM` statement.
```dockerfile
# Install system packages
RUN apt-get update && apt-get install -y \
libpq-dev \
graphviz \
&& rm -rf /var/lib/apt/lists/*
# Install Python packages
RUN pip install --no-cache-dir \
pandas==2.0.0 \
scikit-learn==1.3.0 \
tensorflow==2.13.0
# Install R packages
RUN R -e "install.packages(c('tidyverse', 'caret'), repos='https://cloud.r-project.org')"
# Set environment variables
ENV MODEL_PATH=/mnt/artifacts/model.pkl
```
## Package Installation Methods
### Method 1: Dockerfile Instructions (Recommended)
Best for packages that should always be available.
```dockerfile
RUN pip install pandas numpy scikit-learn
```
**Pros**: Fast startup, consistent environment
**Cons**: Requires environment rebuild for changes
### Method 2: requirements.txt
Packages installed at execution startup.
```text
# requirements.txt in project root
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
```
**Pros**: No rebuild needed, per-project customization
**Cons**: Slower startup
### Method 3: Pre/Post-run Scripts
Custom scripts that run at execution start.
```bash
#!/bin/bash
# pre-run.sh
pip install -q custom-package
```
### Method 4: Runtime Installation
Install during execution (temporary).
```python
!pip install package-name
```
## Dockerfile Best Practices
### Combine RUN Commands
```dockerfile
# Good: Single layer
RUN pip install pandas numpy scikit-learn matplotlib
# Bad: Multiple layers
RUN pip install pandas
RUN pip install numpy
RUN pip install scikit-learn
```
### Clean Up
```dockerfile
RUN apt-get update && apt-get install -y \
package1 \
package2 \
&& rm -rf /var/lib/apt/lists/*
```
### Pin Versions
```dockerfile
# Good: Reproducible
RUN pip install pandas==2.0.0 scikit-learn==1.3.0
# Bad: May break
RUN pip install pandas scikit-learn
```
### Use --no-cache-dir
```dockerfile
RUN pip install --no-cache-dir package-name
```
## Environment Variables
### Set in Dockerfile
```dockerfile
ENV MY_VAR=value
ENV DATA_PATH=/mnt/data
```
### Set in Domino UI
1. Go to Environment settings
2. Add environment variables
3. Variables available in all executions
### Access in Code
```python
import os
value = os.environ.get('MY_VAR', 'default')
```
## GPU Environments
### CUDA Setup
```dockerfile
# Ensure base image has CUDA
# Add GPU-specific packages
RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```
### TensorFlow GPU
```dockerfile
RUN pip install tensorflow[and-cuda]
```
### Verify GPU Access
```python
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU count: {torch.cuda.device_count()}")
```
## IDE Configuration
### Custom Jupyter Config
```dockerfile
RUN mkdir -p /home/domino/.jupyter && \
echo "c.NotebookApp.token = ''" >> /home/domino/.jupyter/jupyter_notebook_config.py
```
### VS Code Extensions
Pre-install extensions:
```dockerfile
RUN code-server --install-extension ms-python.python
```
## Environment Revisions
Domino tracks environment versions:
- Each build creates a new revision
- Can roll back to previous revisions
- Executions can specify which revision to use
### View Revisions
Go to Environment > **Revisions** tab
### Use Specific Revision
Select revision when launching workspace or job.
## Troubleshooting Builds
### Build Fails
1. Check Dockerfile syntax
2. Verify base image exists
3. Check network access for package downloads
4. Review build logs for specific errors
### Common Errors
**Package not found**:
```dockerfile
# Wrong
RUN pip install sklearn
# Right
RUN pip install scikit-learn
```
**Permission denied**:
```dockerfile
# Run as root if needed
USER root
RUN apt-get update && apt-get install -y package
USER domino
```
**Timeout during build**:
- Reduce number of packages
- Use pre-built wheels
- Check network connectivity
### Test Locally
```bash
# Build and test locally before adding to Domino
docker build -t test-env -f Dockerfile .
docker run -it test-env python -c "import pandas; print(pandas.__version__)"
```
## Best Practices Summary
1. **Start from DSE**: Use Domino Standard Environment as base
2. **Minimize layers**: Combine RUN commands
3. **Pin versions**: Ensure reproducibility
4. **Document changes**: Comment Dockerfile instructions
5. **Test before building**: Verify packages work together
6. **Clean up**: Remove cache and temporary files
7. **Regular updates**: Keep packages current for security
## Documentation Reference
- [Customize your Environment](https://docs.dominodatalab.com/en/latest/user_guide/5dd2c1/customize-your-environment/)
- [Add packages to Environments](https://docs.dominodatalab.com/en/latest/user_guide/bfa148/add-packages-to-environments/)
- [Domino Standard Environments](https://docs.dominodatalab.com/en/latest/user_guide/0d73c6/domino-standard-environments/)
- [Best Practices for Environments](https://docs.dominodatalab.com/en/cloud/user_guide/0e4da9/best-practices-for-domino-environments/)
---
name: domino-experiment-setup
description: Set up MLflow experiment tracking in a Domino project by generating an experiment_setup.py helper with deployment-unique experiment names, Domino context tags, and framework auto-logging, plus an example training script. Use when the user asks to add or set up experiment tracking, MLflow logging, or autologging for a traditional ML project in Domino.
compatibility: Domino 6.3 and Domino Cloud. Runs in a Domino workspace or job, where MLflow tracking is preconfigured.
---
# Set Up Domino Experiment Tracking
Applies to Domino 6.3 and Domino Cloud.
Add MLflow experiment tracking to a traditional ML project. For LLM or agent tracing, use the
`domino-trace-setup` skill instead.
## Inputs
1. **Experiment base name**: use the one the user gave; otherwise use the project directory name.
2. **ML frameworks**: detect them from `requirements.txt`, `pyproject.toml`, `environment.yml`,
or imports in the code.
Do not stop to ask for either. Make the changes with these defaults, then state the base name
and frameworks you used in the reply so the user can change them.
## Steps
1. Copy `assets/experiment_setup.py` from this skill's folder into the project. It provides:
- `setup_experiment(base_name)`: sets an experiment named `<base>-<project>-<username>`.
- `log_domino_context()`: tags the run with the Domino user, project, run ID, and hardware tier.
- `setup_autolog()`: enables MLflow autologging for whichever supported frameworks are installed.
2. Report which frameworks were detected and will be autologged, for example:
```
Detected frameworks:
✅ scikit-learn (requirements.txt)
✅ xgboost (requirements.txt)
❌ tensorflow (not found)
```
3. If the project trains large models, add the multipart-upload settings below.
4. Create an example training script, or wire the helpers into the user's existing script.
## Example training script
```python
# train.py
from experiment_setup import setup_experiment, log_domino_context, setup_autolog
import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
experiment_name = setup_experiment("iris-classifier")
setup_autolog()
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
iris.data, iris.target, test_size=0.2, random_state=42
)
with mlflow.start_run(run_name="random-forest-v1"):
log_domino_context()
mlflow.log_param("custom_param", "value")
model = RandomForestClassifier(n_estimators=100, max_depth=5)
model.fit(X_train, y_train)
test_accuracy = model.score(X_test, y_test)
mlflow.log_metric("test_accuracy", test_accuracy)
print(f"Test accuracy: {test_accuracy:.4f}")
print(f"Run ID: {mlflow.active_run().info.run_id}")
```
## Important rules
- **Experiment names must be unique across the entire Domino deployment**, not just the project.
Always keep the username and project suffix that `setup_experiment` adds.
- For large artifacts (LLMs, deep learning checkpoints), enable multipart upload:
```python
os.environ['MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD'] = "true"
os.environ['MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE'] = "104857600" # 100MB
```
For run comparison, the model registry, and deeper MLflow usage, use the
`domino-experiment-tracking` skill.
Referenced files: 1 — file references are metadata, not package contents.
---
name: domino-experiment-tracking
description: Track traditional ML experiments in Domino using the MLflow-based Experiment Manager. Covers experiment setup, auto-logging for sklearn/TensorFlow/PyTorch, manual logging, artifact storage, run comparison, and model registration. Use when training ML models, logging metrics and parameters, comparing model runs, or registering models.
---
# Domino Experiment Tracking Skill
This skill provides comprehensive knowledge for tracking ML experiments in Domino Data Lab using the built-in MLflow-based Experiment Manager.
## Key Concepts
### Experiment Manager Overview
Domino's Experiment Manager is built on MLflow and provides:
- Automatic and manual logging of parameters, metrics, and artifacts
- Run comparison and visualization
- Model versioning and registry
- Integration with Domino projects and jobs
### Critical Configuration
**Experiment names must be unique across the entire Domino deployment.** Always append username or project name to ensure uniqueness.
## Related Documentation
- [MLFLOW-BASICS.md](./MLFLOW-BASICS.md) - Auto-logging, manual logging
- [COMPARING-RUNS.md](./COMPARING-RUNS.md) - Run comparison, export
- [MODEL-REGISTRY.md](./MODEL-REGISTRY.md) - Model registration & stages
## Quick Start
```python
import mlflow
import os
# CRITICAL: Experiment names must be unique across Domino deployment
username = os.environ.get('DOMINO_STARTING_USERNAME', 'unknown')
experiment_name = f"my-experiment-{username}"
# Set the experiment
mlflow.set_experiment(experiment_name)
# Enable auto-logging (easiest approach)
mlflow.autolog()
# Run training
with mlflow.start_run(run_name="my-first-run"):
model.fit(X_train, y_train)
# Optional: manually log additional items
mlflow.log_param("custom_param", "value")
mlflow.log_metric("custom_metric", 0.95)
```
## Supported Frameworks
| Framework | Auto-log Command |
|-----------|------------------|
| Scikit-learn | `mlflow.sklearn.autolog()` |
| TensorFlow/Keras | `mlflow.tensorflow.autolog()` |
| PyTorch | `mlflow.pytorch.autolog()` |
| XGBoost | `mlflow.xgboost.autolog()` |
| LightGBM | `mlflow.lightgbm.autolog()` |
| All at once | `mlflow.autolog()` |
## Environment Variables
Domino automatically configures MLflow to use the built-in tracking server. These variables are pre-set:
| Variable | Description |
|----------|-------------|
| `MLFLOW_TRACKING_URI` | Domino's MLflow server URL |
| `DOMINO_STARTING_USERNAME` | User running the experiment |
| `DOMINO_PROJECT_NAME` | Current project name |
| `DOMINO_RUN_ID` | Domino job run ID |
## Documentation Links
- Domino Experiment Tracking: https://docs.dominodatalab.com/en/latest/user_guide/da707d/track-and-monitor-experiments/
- Domino Model Registry: https://docs.dominodatalab.com/en/latest/user_guide/3b6ae5/manage-models-with-model-registry/
Referenced files: 3 — file references are metadata, not package contents.
---
name: domino-extensions
description: Build and operate Domino UI Extensions (mount Domino Apps in the shell with page context). Covers extension_manifest.json, official installer vs manual Extension API, publish and identity requirements, auth and host splits, and beta REST routes. Use when creating an Extension, wiring mount points, publishing an Extension App from a project, or automating Extension install lifecycle. Not for generic App deployment alone (see domino-apps) or SPA scaffolding (see domino-ui-bootstrap).
---
# Domino Extensions
An **Extension** surfaces a published **App** at a Domino UI **mount point** (project sidebar, dataset page, model details, file context menus, admin panel). Domino opens the App in a new tab and passes page context as query parameters. The App must support **extended identity propagation** so it can act as the viewing user with consent.
## When to use other skills
| Goal | Skill |
|------|--------|
| Vite/React design system, `@dominodatalab/extensions-tools`, proxy-safe URLs | [domino-ui-bootstrap](../domino-ui-bootstrap/SKILL.md) |
| `app.sh`, ports, SPA base path, generic App CI | [domino-apps](../apps/SKILL.md) |
| REST auth, jobs, projects, platform `/api/*` and `/v4/*` | [python-sdk](../python-sdk/SKILL.md) |
| Governance from an extension backend | [domino-governance](../domino-governance/SKILL.md) plus external URL rules below |
Product overview: https://docs.domino.ai/cloud/platform-capabilities/features/extensions
Official catalog install (SysAdmin/CloudAdmin): https://docs.domino.ai/cloud/platform-capabilities/features/extensions/install-domino-official-extensions
Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . For HTTP client setup in Extension backends and install scripts, use [python-sdk/SKILL.md](../python-sdk/SKILL.md#authentication).
## Extension manifest (`extension_manifest.json`)
Domino Official Extensions and custom Extension repos ship a manifest at the repo root. Schema version is **`manifestSchemaVersion`: 1** (integer). The official installer reads this file from a GitHub release; manual flows still mirror the same shape when you create entities yourself.
Top-level sections:
| Section | Purpose |
|---------|---------|
| `project` | Hosting project (`name`, `description`, `visibility`, `isRestricted`, ...) |
| `app` | Compute environment, entry point, vanity URL, visibility, mounts, hardware tier |
| `extension` | Extension record: `name`, `description`, `enabled`, `uiMountPointTypeConfigs` |
For full manifest examples and patterns (compute environment, mount configs, deep linking, autoscaling), refer to Domino's official Extension repos:
- https://github.com/dominodatalab/AutoML_Extension
- https://github.com/dominodatalab/clinical-data-explorer
- https://github.com/dominodatalab/AutoDocumentation_Extension
Each ships an `extension_manifest.json` at the repo root; read the file matching your Domino version rather than copying from this skill.
### Mount points (`uiMountPointTypeConfigs`)
Keys match the Extensions API enum: `projectSidebar`, `dataset`, `datasetFileContext`, `modelDetails`, `adminPanel`, `netAppVolume`, `netAppVolumeFileContext`. Each mount defines `enabled`, scope (for example `allProjects` or project lists), and `urlConfig` with `contextualQueryParams` Domino passes to the App.
Domino documents five primary user-facing mount types on the product page; the API includes NetApp-related mount types for volume integrations.
### Deep linking vs iframe (`renderIFrame`)
Apps can run full-page (deep linking) or inside an iframe. Platform convention: **`renderIFrame = !deepLinkingEnabled`**. Manifest field **`enableDeepLinking: true`** means the App expects full-page navigation; set `false` when the UI should load in an iframe. Align manifest, App publish settings, and start overrides so you do not mix modes silently.
## Publish from a project (custom Extension)
Typical human workflow:
1. **Build the App** in a project (frontend plus optional backend). Enable extended identity propagation; Flask/Dash read proxied headers by default.
2. **Publish the App** as a SysAdmin or CloudAdmin (App must be published before it backs an Extension). Use the Apps API publish chain (`/api/apps/v1/...` and related routes). See [python-sdk/API-APPS.md](../python-sdk/API-APPS.md); confirm paths in [API-SPECS.md](../domino-api-intro/API-SPECS.md) (**Public routes**).
3. **Create the Extension** via Admin UI or **`POST /api/extensions/beta/extensions`** with `appId`, optional `appVersionId`, `name`, `enabled`, and `uiMountPointTypeConfigs`.
REST surface (beta): prefix **`/api/extensions/beta/`** (`extensions`, `extensions-ui`, `official-installs`, ...). Confirm operation IDs and bodies in [API-SPECS.md](../domino-api-intro/API-SPECS.md) (public routes section).
```python
import os
import requests
if os.environ.get("DOMINO_API_PROXY"):
base_url = os.environ["DOMINO_API_PROXY"].rstrip("/")
headers = {}
else:
base_url = (os.environ.get("DOMINO_USER_HOST") or os.environ.get("DOMINO_API_HOST") or "").rstrip("/")
token = requests.get("http://localhost:8899/access-token").text.strip()
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
body = {
"name": "My Extension",
"enabled": True,
"appId": "app-id-from-publish",
"uiMountPointTypeConfigs": {
"projectSidebar": {
"enabled": True,
"allProjects": True,
"urlConfig": {"contextualQueryParams": ["projectId"]},
}
},
}
response = requests.post(f"{base_url}/api/extensions/beta/extensions", headers=headers, json=body)
```
Only Admins create Extensions; viewers grant consent when the App acts as them.
### Official Extension install (manifest-driven)
For Domino-built Extensions, an admin installs from **Manage Domino-official Extensions** in the Admin panel. The installer creates project, environment, App, and Extension from the release manifest (background job with retry/cancel). API analogs: `official-install-menu`, `POST .../official-installs`, snapshot status endpoints under `/api/extensions/beta/official-installs/`.
Do not edit installer-managed project, environment, App, or Extension by hand; use the official install UI for version changes.
## Auth, identity, and hosts in Extension Apps
| Context | Guidance |
|---------|----------|
| **App backend** calling platform APIs as the **starting user** | `DOMINO_API_PROXY` if set (no Authorization header); else access-token plus platform host. See [python-sdk](../python-sdk/SKILL.md). |
| **Visitor identity** in the browser | Visitor JWT from the App ingress; validate with JWKS. **`GET /v4/users/self` with a visitor JWT often fails** for privileged data; do not assume it replaces admin APIs for org/role lists. |
| **Governance** `/api/governance/v1/*` | Base URL and auth: [domino-governance — Configuration](../domino-governance/SKILL.md#configuration). Bearer PAT or SA only; never API keys. |
| **Inference** `/endpoints/{vanity}` | Use the **`url`** from the GenAI/management API response, not `DOMINO_USER_HOST`. |
| **Automation outside any run** | Public deployment URL plus PAT or SA only. |
Domino does not inject a single `DOMINO_EXTERNAL_URL`; derive public URL from deployment config, forwarded headers, or app conventions (`DOMINO_PUBLIC_HOST` / `DOMINO_EXTERNAL_HOST` are app patterns, not guaranteed core run injection).
## Platform caveats (Apps API + Extensions)
These affect Extension Apps the same as standalone Apps:
| Topic | Behavior |
|-------|----------|
| **`netAppVolumeIds` on App version create** | Accepted in the API but NetApp volumes may **not mount** (silent no-op vs workspace parity). Prefer explicit volume workflows; manifest `mountNetAppVolumes` does not fix API no-op alone. |
| **App delete and vanity URL** | Deleting an App may **not release** its vanity URL for immediate reuse; recreate failures may need admin cleanup. |
| **Apps beta vs v1** | Extension backing Apps may be created or published through beta or v1 routes; confirm routes in [API-SPECS.md](../domino-api-intro/API-SPECS.md) (public routes section). Prefer documented v1 publish flows for new automation where available. |
## Prerequisites on the deployment
- Extensions feature enabled.
- **`SecureIdentityPropagationToAppsEnabled`** and extended identity propagation for Apps (default on Domino Cloud).
- For official installs: platform egress to GitHub for catalog, manifests, and release artifacts.
## Related API reference
OpenAPI and route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md).
- Apps publish chain: [python-sdk/API-APPS.md](../python-sdk/API-APPS.md)
---
name: domino-flows
description: Orchestrate multi-step ML workflows using Domino Flows (built on Flyte). Define DAGs with typed inputs/outputs, heterogeneous environments, automatic lineage, and reproducibility. Use when building data pipelines, multi-stage training workflows, or processes requiring orchestration and monitoring.
---
# Domino Flows Skill
This skill provides comprehensive knowledge for orchestrating ML workflows using Domino Flows, built on the Flyte platform.
## Key Concepts
### What are Domino Flows?
Domino Flows enable:
- **DAG-based orchestration**: Define workflows as directed acyclic graphs
- **Typed interfaces**: Strong typing for inputs and outputs
- **Heterogeneous environments**: Different environments per task
- **Automatic lineage**: Track data and model provenance
- **Reproducibility**: Version-controlled workflows
- **Scalability**: Distributed execution across compute resources
### Core Components
| Component | Description |
|-----------|-------------|
| **Task** | Single unit of work (runs as a Domino Job) |
| **Workflow** | DAG connecting tasks |
| **Artifact** | Typed input/output passed between tasks |
| **Launch Plan** | Configured workflow execution |
## Related Documentation
- [FLOW-BASICS.md](./FLOW-BASICS.md) - DAG concepts, task definitions
- [EXAMPLES.md](./EXAMPLES.md) - Common flow patterns
## Quick Start
> ⚠️ **Critical**: Domino Flows does NOT support native Flyte `@task` decorators.
> Tasks must use `DominoJobTask` + `DominoJobConfig`. Only `@workflow` is unchanged.
### Basic Flow
Each task runs as a Domino Job. Stage scripts read from `/workflow/inputs/<name>`
and write to `/workflow/outputs/o0`. Pass `PYTHONPATH=/mnt/code` in the command.
```python
from flytekit import workflow
from flytekitplugins.domino.task import DominoJobConfig, DominoJobTask
preprocess_task = DominoJobTask(
name="Preprocess Data",
domino_job_config=DominoJobConfig(
Command="bash -c 'PYTHONPATH=/mnt/code python /mnt/code/stages/preprocess.py'",
),
inputs={"input_path": str},
outputs={"o0": str},
use_latest=True,
)
train_task = DominoJobTask(
name="Train Model",
domino_job_config=DominoJobConfig(
Command="bash -c 'PYTHONPATH=/mnt/code python /mnt/code/stages/train.py'",
),
inputs={"preprocess_output": str},
outputs={"o0": str},
use_latest=True,
)
@workflow
def training_pipeline(input_path: str = "/mnt/data/raw.csv") -> str:
preprocess_output = preprocess_task(input_path=input_path)
result = train_task(preprocess_output=preprocess_output)
return result
```
### Stage Script Pattern
```python
# stages/preprocess.py
import json, os
INPUTS, OUTPUTS = "/workflow/inputs", "/workflow/outputs"
def main():
input_path = open(f"{INPUTS}/input_path").read().strip()
# ... do work ...
os.makedirs(OUTPUTS, exist_ok=True)
with open(f"{OUTPUTS}/o0", "w") as f:
f.write(json.dumps({"output_path": "/mnt/artifacts/processed.parquet"}))
if __name__ == "__main__":
main()
```
### Running the Flow
```bash
# Always commit and push first — jobs run against remote repo state
git add -A && git commit -m "..." && git push
# Trigger remotely
PYTHONPATH=/mnt/code pyflyte run --remote \
my_flow.py training_pipeline \
--input_path "/mnt/data/raw.csv"
```
## When to Use Flows
### Good Use Cases
- Data processing → Model training pipelines
- ETL with ML steps
- Multi-stage training with different environments
- Processes requiring reproducibility and lineage
- Scheduled/triggered workflows
### Not Ideal For
- Single dataset with many small computations
- Tasks that write to mutable shared state
- Simple single-step processes
- Real-time inference (use Model APIs instead)
## Documentation Links
- Domino Flows: https://docs.dominodatalab.com/en/latest/user_guide/78acf5/orchestrate-with-flows/
- Flyte Documentation: https://docs.flyte.org/
Referenced files: 2 — file references are metadata, not package contents.
---
name: domino-genai-tracing
description: Trace and evaluate GenAI applications including LLM calls, agents, RAG pipelines, and multi-step AI systems in Domino. Uses the Domino SDK (@add_tracing decorator, DominoRun context) with MLflow 3.2.0. Captures token usage, latency, cost, tool calls, and errors. Supports LLM-as-judge evaluators and custom metrics. Use when building agents, debugging LLM applications, or needing audit trails for GenAI systems.
---
# Domino GenAI Tracing Skill
This skill provides comprehensive knowledge for tracing and evaluating GenAI applications in Domino Data Lab, including LLM calls, agents, RAG pipelines, and multi-step AI systems.
## Two Deployment Modes
GenAI tracing works differently depending on where your code runs:
| Mode | Where traces appear | When to use |
|------|-------------------|-------------|
| **Deployed App** (Production) | App **Performance tab** | FastAPI/Flask apps deployed as Domino Apps |
| **Development / Evaluation** | **Experiments** UI | Batch scripts, Domino Jobs, Workspaces |
**Critical difference:** In a deployed Domino App, Domino auto-creates an experiment named `agent_experiment_{app_id}` and the Performance tab reads from it. If you call `mlflow.set_experiment()` or wrap calls in `DominoRun()`, traces go to your custom experiment instead — and the Performance tab won't see them.
## Key Concepts
### What GenAI Tracing Captures
The Domino SDK automatically captures:
- **Token usage** - Input and output tokens per call
- **Latency** - Time for each operation
- **Cost** - Estimated cost per call
- **Tool calls** - Function/tool invocations
- **Errors** - Exceptions and failure modes
- **Model parameters** - Temperature, max_tokens, etc.
### Core Components
1. **`@add_tracing` decorator** - Wraps agent functions to capture traces (works standalone — no `DominoRun` required)
2. **`mlflow.start_span()`** - Creates child spans for LLM calls and tool executions inside the agent loop
3. **`DominoRun` context manager** - Groups traces into runs for development/evaluation (Experiments UI only)
4. **Evaluators** - Custom functions to score outputs
5. **MLflow integration** - View traces in Experiment Manager or App Performance tab
## Related Documentation
- [TRACING-SETUP.md](./TRACING-SETUP.md) - Environment & SDK setup
- [ADD-TRACING-DECORATOR.md](./ADD-TRACING-DECORATOR.md) - @add_tracing usage, span_type, autolog_frameworks
- [DOMINO-RUN.md](./DOMINO-RUN.md) - DominoRun context manager (development/evaluation only)
- [EVALUATORS.md](./EVALUATORS.md) - LLM-as-judge, custom evaluators
- [MULTI-AGENT-EXAMPLE.md](./MULTI-AGENT-EXAMPLE.md) - Complete multi-agent example
## Quick Start — Deployed App (Production)
For a FastAPI app deployed as a Domino App. Traces appear in the **App Performance tab**.
```python
# main.py — lifespan: autolog only, NO set_experiment, NO DominoRun
import mlflow
from fastapi import FastAPI
from contextlib import asynccontextmanager
@asynccontextmanager
async def lifespan(app: FastAPI):
# Each autolog in its own try/except so one failure doesn't block the other
try:
mlflow.openai.autolog()
except Exception:
pass
try:
mlflow.anthropic.autolog()
except Exception:
pass
yield
app = FastAPI(lifespan=lifespan)
```
```python
# orchestrator.py — @add_tracing on the core agent function
import mlflow
from domino.agents.tracing import add_tracing
@add_tracing(
name="agent_turn",
span_type="AGENT",
autolog_frameworks=["openai", "anthropic"],
)
async def run_agent(messages: list[dict]) -> dict:
# LLM calls and tool calls go here (see ADD-TRACING-DECORATOR.md)
...
# router.py — call the traced function directly, NO DominoRun wrapper
@app.post("/chat")
async def chat(request: ChatRequest):
return await run_agent(request.messages)
```
## Quick Start — Development / Evaluation
For batch scripts, Domino Jobs, or Workspaces. Traces appear in the **Experiments UI**.
```python
import mlflow
from domino.agents.tracing import add_tracing
from domino.agents.logging import DominoRun
mlflow.openai.autolog()
@add_tracing(name="my_agent", autolog_frameworks=["openai"])
def my_agent(query: str) -> str:
response = llm.invoke(query)
return response
# DominoRun groups traces into a run visible in Experiments
with DominoRun() as run:
result = my_agent("What is machine learning?")
```
## Framework Support
| Framework | Auto-log Command |
|-----------|------------------|
| OpenAI | `mlflow.openai.autolog()` |
| Anthropic | `mlflow.anthropic.autolog()` |
| LangChain | `mlflow.langchain.autolog()` |
## Viewing Traces
### Deployed Apps (Production)
1. Navigate to your Domino App
2. Click the **Performance** tab
3. Traces appear automatically (routed to `agent_experiment_{app_id}`)
### Development / Evaluation
1. Navigate to **Experiments** in your Domino project
2. Select the experiment
3. Select a run
4. View the **Traces** tab for span tree visualization
## Blueprint Reference
Official GenAI Tracing Tutorial:
https://github.com/dominodatalab/GenAI-Tracing-Tutorial
## Documentation Links
- Domino GenAI Tracing: https://docs.dominodatalab.com/en/cloud/user_guide/fc1922/set-up-and-run-genai-traces/
Referenced files: 5 — file references are metadata, not package contents.
---
name: domino-governance
description: Manage model risk governance in Domino using policies, bundles, and evidence. Covers creating governance bundles, attaching model artifacts and MLflow results as evidence, progressing through policy stages, and documenting findings. Use when the user mentions governance, compliance, bundles, policies, model risk management, SR 11-7, NIST AI RMF, or audit trails.
---
# Domino Governance Skill
This skill provides knowledge for managing model risk governance in Domino Data Lab using the Governance API.
## Configuration
Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . **Do not use API keys** (`X-Domino-Api-Key`, `DOMINO_USER_API_KEY`).
Governance REST lives at `/api/governance/v1/*`. Pick the base where `GET $BASE/policy-overviews` returns **HTTP 200** (try the table rows in order):
| Context | Base | Authorization |
|---------|------|----------------|
| In-run, `DOMINO_API_PROXY` set (5.4.0+ JWT credential propagation) | `{DOMINO_API_PROXY}/api/governance/v1` | None (proxy adds starting-user JWT) |
| In-run, no proxy, governance on platform API gateway | `{DOMINO_USER_HOST or DOMINO_API_HOST}/api/governance/v1` | Bearer from `GET http://localhost:8899/access-token` |
| In-run but proxy or sidecar returns **404** on `/api/governance/v1/*` (gateway off or governance not registered on 8763) | `{public deployment URL}/api/governance/v1` | Bearer PAT or SA token |
| Outside any run | `{public deployment URL}/api/governance/v1` | Bearer PAT or SA token |
```bash
GOV="/api/governance/v1"
PREFIX="/policy-overviews"
# 1) In-run, proxy (try first when DOMINO_API_PROXY is set)
BASE="$DOMINO_API_PROXY$GOV"
curl -s "$BASE$PREFIX"
# 2) In-run, platform gateway (no proxy)
HOST="${DOMINO_USER_HOST:-$DOMINO_API_HOST}"
TOKEN="$(curl -s http://localhost:8899/access-token)"
BASE="$HOST$GOV"
curl -s "$BASE$PREFIX" -H "Authorization: Bearer $TOKEN"
# 3) Public deployment (outside run, or 404 on 1 and 2)
BASE="https://your-deployment.domino.tech$GOV"
TOKEN="$PAT_OR_SA_TOKEN"
curl -s "$BASE$PREFIX" -H "Authorization: Bearer $TOKEN"
```
Do **not** scrape the JWT `iss` claim to derive the cluster URL. Prefer `DOMINO_API_PROXY` in-run when step 1 succeeds; otherwise step 2 or 3.
**Takeaways for agents**
1. **Proxy + no header is correct only when routing works.** `DOMINO_API_PROXY` injects JWT ([product auth doc](https://docs.domino.ai/cloud/reference/api/domino-api-authentication)); it does not add `/api/governance/v1` if guardrails is not reachable on that path.
2. **Try steps 1 → 2 → 3** until `GET $BASE/policy-overviews` returns HTTP 200. Do not assume step 1 works on every deployment.
3. **In-run without proxy is not “use public URL.”** Use `DOMINO_USER_HOST` (or `DOMINO_API_HOST`) plus Bearer from `http://localhost:8899/access-token` when governance is on the platform API gateway.
4. **Public deployment URL + PAT/SA** when you are outside a run, or when step 1 or 2 returns **404** (common when the API gateway is off or governance is not registered on port 8763).
5. **Suffix is always** `/api/governance/v1` on whichever host/base you chose ([Governance API](https://docs.domino.ai/cloud/reference/api/governance-api)).
**Examples below:** Omit `Authorization` only when `TOKEN` is unset (proxy path). For gateway or public URL, set `TOKEN` and use `${TOKEN:+-H "Authorization: Bearer $TOKEN"}` in curl blocks.
## Key Concepts
### Policy (Template)
A **policy** is a reusable governance template that defines the stages, evidence requirements, and approval gates a model must pass through. Examples: SR 11-7, NIST AI RMF, internal model risk frameworks. Policies are created by administrators in the Domino UI.
### Bundle (Living Document)
A **bundle** is the compliance document for a *specific model* in a *specific project*. It follows a policy and accumulates evidence as the model progresses through development, validation, and approval. One project can have multiple bundles (e.g., one per model version).
### Evidence (Proof)
Evidence comes in two forms:
1. **Attachments** — Files, model versions, and reports attached to the bundle (visible in the "Attachments" tab)
2. **EvidenceSet answers** — Responses to policy-defined form questions (visible in the "Evidence" tab for each stage)
### Finding (Issue)
A **finding** documents a problem, risk, or concern discovered during review. Findings have severity levels and are tracked as part of the audit trail.
## Related Documentation
When automating a specific flow, open the matching workflow file first instead of re-deriving from the 8-step section below.
- [BUNDLE-LIFECYCLE.md](./BUNDLE-LIFECYCLE.md) - Stage sequences, creating bundles, progressing stages
- [EVIDENCE-WORKFLOW.md](./EVIDENCE-WORKFLOW.md) - Attachment types, evidence submission, findings
- [ONBOARD-MODEL.md](./ONBOARD-MODEL.md) - Onboard a model into governance
- [EVIDENCE-ROUND-TRIP.md](./EVIDENCE-ROUND-TRIP.md) - Evidence round-trip with findings
- [APPROVAL-GATE.md](./APPROVAL-GATE.md) - Approval gate automation
- [MONITORING-REREVIEW.md](./MONITORING-REREVIEW.md) - Monitoring-driven re-review
- [RECERTIFICATION-BULK.md](./RECERTIFICATION-BULK.md) - Periodic recertification (bulk)
- [AUDIT-EXPORT.md](./AUDIT-EXPORT.md) - Regulatory audit export
- [POLICY-LIFECYCLE.md](./POLICY-LIFECYCLE.md) - Policy lifecycle and rollout
- [MODEL-RETIREMENT.md](./MODEL-RETIREMENT.md) - Model retirement
- [WAIVER-EXCEPTION.md](./WAIVER-EXCEPTION.md) - Exception or waiver
- [READ-CONTEXT.md](./READ-CONTEXT.md) - Read governance context for downstream use
## Governance API Reference
All endpoints are under `$BASE` (`/api/governance/v1`). See [Configuration](#configuration) for when to send `Authorization`.
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/policy-overviews` | GET | List available policy templates |
| `/bundles` | POST | Create a new governance bundle (optional inline `attachments`) |
| `/bundles/{bundleId}` | GET | Inspect bundle: stages, attachments, status |
| `/bundles/{bundleId}` | PATCH | Update bundle `state`, primary `stage` name, or `policyId` |
| `/bundles` | GET | List all bundles (filter by `projectId`) |
| `/bundles/{bundleId}/attachments` | POST | Attach evidence after create (model versions, reports) |
| `/bundles/{bundleId}/stages/{stageId}` | PATCH | Update stage **assignee** only (not stage completion) |
| `/rpc/submit-result-to-policy` | POST | Answer policy evidence questions |
| `/rpc/compute-policy` | POST | Recompute policy for a bundle (returns `ComputedPolicy`) |
| `/results/latest` | GET | Latest artifact results (`bundleID` query param) |
| `/rpc/publish-approval-event` | POST | Publish approval workflow events (approve, reject, etc.) |
| `/policies/{policyId}` | GET | Get full policy with evidenceSet IDs |
| `/findings` | POST | Create a finding (issue) during review |
| `/findings/{id}` | PUT | Update a finding (including `status`) |
**Bundle fields:** `stage` (string) is the primary policy stage name; set it with `PATCH /bundles/{id}` to move the workflow. `state` (`Active`, `Archived`, `Complete`) is bundle lifecycle; set when archiving or completing the bundle. `currentStageInfo.status` (`NotStarted`, `InProgress`, `Done`) is stage progress on GET only, not writable on `/stages/{stageId}`.
## Standard 8-Step Governance Workflow
Follow these steps when setting up governance for a model:
### Step 1: Discover Policies
```bash
curl -s "$BASE/policy-overviews" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"}
```
Review available templates. Note the `id` of the policy you want to use.
### Step 2: Get the Project ID
The project ID is needed to create a bundle. Use the `DOMINO_PROJECT_ID` environment variable (available inside Domino) or look it up via the gateway API.
### Step 3: Create a Bundle
Attachments may be inline on `POST /bundles` or added later via `POST /bundles/{id}/attachments`. Prefer inline when creating and attaching a model in one step (see [ONBOARD-MODEL.md](./ONBOARD-MODEL.md)).
```bash
curl -X POST "$BASE/bundles" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} \
-H "Content-Type: application/json" \
-d '{
"projectId": "your-project-id",
"name": "My Model v1.0",
"policyId": "policy-uuid"
}'
```
Save the returned `id` as your `BUNDLE_ID`.
### Step 4: Inspect the Bundle
```bash
curl -s "$BASE/bundles/$BUNDLE_ID" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"}
```
This reveals the policy's stage structure, attachments, and approval status. **Note**: This does NOT return evidenceSet IDs — see Step 6 for how to discover those.
### Step 5: Attach Evidence
See [EVIDENCE-WORKFLOW.md](./EVIDENCE-WORKFLOW.md) for full details. Two attachment types are supported:
```bash
# Attach a registered model version
curl -X POST "$BASE/bundles/$BUNDLE_ID/attachments" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} -H "Content-Type: application/json" \
-d '{"type":"ModelVersion","identifier":{"name":"model-name","version":5},"name":"Display Name"}'
# Attach a project file (notebook, report, etc.)
curl -X POST "$BASE/bundles/$BUNDLE_ID/attachments" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} -H "Content-Type: application/json" \
-d '{"type":"Report","identifier":{"branch":"main","commit":"abc...","source":"git","filename":"path/to/file"},"name":"Display Name"}'
```
### Step 6: Answer Evidence Questions (EvidenceSet)
Evidence questions are the interactive forms shown in the Domino UI under each stage's "Evidence" tab. They are defined in the policy YAML as `evidenceSet` items.
#### 6a. Discover EvidenceSet IDs
EvidenceSet IDs are NOT in the bundle response. Fetch them from the **policy** endpoint:
```bash
curl -s "$BASE/policies/$POLICY_ID" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"}
```
The response contains `stages[]` → `evidenceSet[]` → `artifacts[]` with full UUIDs for each evidence item and artifact.
#### 6b. Submit Answers
```bash
curl -X POST "$BASE/rpc/submit-result-to-policy" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} \
-H "Content-Type: application/json" \
-d '{
"bundleId": "bundle-uuid",
"policyId": "policy-uuid",
"evidenceId": "evidence-uuid",
"content": {
"artifact-uuid": "value"
}
}'
```
**Key details**:
- Use `policyId` (NOT `policyVersionId`)
- `evidenceId` is the evidence set item UUID (from policy response)
- `content` is a map of `{artifactId: value}`
- For **radio/textinput/textarea/select**: value is a string
- For **checkbox/multiSelect**: value is an array of strings
- Submit one artifact at a time per call, or multiple artifacts in the same evidence item together
### Step 7: Progress Stages
Stage completion is driven by evidence, approvals, and workflow events, not by `status: Complete` on the stage subresource. `PATCH /bundles/{bundleId}/stages/{stageId}` only updates **assignee**. After evidence and approvals:
1. Publish approval events (see [APPROVAL-GATE.md](./APPROVAL-GATE.md)):
```bash
curl -X POST "$BASE/rpc/publish-approval-event" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} \
-H "Content-Type: application/json" \
-d '{
"bundleId": "bundle-uuid",
"policyId": "policy-uuid",
"projectId": "project-uuid",
"eventType": "RequestApproved",
"approvalId": "approval-uuid"
}'
```
2. When the policy requires explicitly setting the primary stage name, use `PATCH /bundles/{bundleId}`:
```bash
curl -X PATCH "$BASE/bundles/$BUNDLE_ID" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} \
-H "Content-Type: application/json" \
-d '{"stage": "Next stage name from policy"}'
```
Re-fetch the bundle and verify `stage` (primary stage name) or `currentStageInfo.status` (`NotStarted`, `InProgress`, `Done`) changed before continuing. Details: [BUNDLE-LIFECYCLE.md](./BUNDLE-LIFECYCLE.md).
### Step 8: Document Findings (if any)
```bash
curl -X POST "$BASE/findings" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} -H "Content-Type: application/json" \
-d '{
"bundleId": "bundle-uuid",
"policyVersionId": "policy-version-uuid",
"name": "Finding title",
"description": "Detailed description...",
"severity": "High",
"approver": {"id": "org-uuid", "name": "model-gov-org"},
"assignee": {"id": "user-uuid", "name": "username"}
}'
```
Close or resolve with **PUT** `/findings/{id}` and `"status": "Done"` (not `PATCH` or `state: Closed`). See [EVIDENCE-ROUND-TRIP.md](./EVIDENCE-ROUND-TRIP.md).
Get the `policyVersionId` and user/org IDs from the bundle response (Step 4).
## Viewing in Domino UI
After creating a bundle and attaching evidence, the bundle is visible in the Domino UI:
**Project** > **Govern** > **Bundles** > click the bundle name
The UI shows:
- **Overview** — Stage progression and classification
- **Evidence** tab (per stage) — Interactive forms for evidenceSet questions
- **Attachments** tab — Files, model versions, and reports
- **Findings** tab — Documented issues with severity
## After compute-policy (polling)
`POST /rpc/compute-policy` returns **200** with a `ComputedPolicy` body. Freshness lives on artifact results, not a single top-level flag.
1. Call `POST /rpc/compute-policy` with `bundleId` and `policyId`.
2. Poll `GET /results/latest?bundleID={bundleId}` (and optional `policyID`, `artifactID`) until the artifacts you care about have `isLatest: true`, or inspect `results[].isLatest` on the compute response when present.
3. Only then treat evidence as current for submit, stage advance, or read-context output.
Reuse this loop in every workflow that calls `compute-policy` (see [ONBOARD-MODEL.md](./ONBOARD-MODEL.md), [EVIDENCE-ROUND-TRIP.md](./EVIDENCE-ROUND-TRIP.md), [READ-CONTEXT.md](./READ-CONTEXT.md)).
## Known platform behaviors
Short pointers; full gotchas live in the workflow files linked under [Related Documentation](#related-documentation).
| Behavior | Where |
|----------|--------|
| Governance base URL (proxy vs gateway vs public on 404) | [Configuration](#configuration) |
| Verify stage actually advanced after approvals | [APPROVAL-GATE.md](./APPROVAL-GATE.md), [BUNDLE-LIFECYCLE.md](./BUNDLE-LIFECYCLE.md) |
| Poll after `compute-policy` | [After compute-policy (polling)](#after-compute-policy-polling) |
| `ModelVersion` attachment `identifier.name` | [EVIDENCE-WORKFLOW.md](./EVIDENCE-WORKFLOW.md), [ONBOARD-MODEL.md](./ONBOARD-MODEL.md) |
| Draft vs submitted evidence (`isLatest`) | [EVIDENCE-WORKFLOW.md](./EVIDENCE-WORKFLOW.md) |
| Close findings (PUT + `status`, new evidence first) | [EVIDENCE-ROUND-TRIP.md](./EVIDENCE-ROUND-TRIP.md) |
| Policy version retire / upgrade on bundles | [POLICY-LIFECYCLE.md](./POLICY-LIFECYCLE.md) |
| Multi-bundle per model | [READ-CONTEXT.md](./READ-CONTEXT.md) |
| Policy attach RPC vs nucleus guardrails proxy | [POLICY-LIFECYCLE.md](./POLICY-LIFECYCLE.md) |
| Open findings before stage advance | [EVIDENCE-ROUND-TRIP.md](./EVIDENCE-ROUND-TRIP.md) |
## Documentation Reference
OpenAPI and route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md) (public routes section).
**Governance API base:** See [Configuration](#configuration) (proxy, in-run gateway + access-token, or public URL + PAT/SA when governance 404s on the sidecar). Do not scrape the JWT `iss` claim.
**Product docs:**
- Governance overview: https://docs.domino.ai/cloud/platform-capabilities/features/governance
- Governance API: https://docs.domino.ai/cloud/reference/api/governance-api
For governance workflows and pitfalls, see [Related Documentation](#related-documentation).
Referenced files: 12 — file references are metadata, not package contents.
---
name: domino-jobs
description: Create, run, and manage Domino Jobs - batch executions for scripts, training, and data processing. Covers job configuration, hardware tiers, scheduled jobs (cron), monitoring status, viewing logs, and API-driven execution. Use when running batch workloads, scheduling recurring tasks, or automating training pipelines.
---
# Domino Jobs Skill
## Description
This skill helps users create, run, and manage Domino Jobs - batch executions for running scripts, training models, and processing data.
## Activation
Activate this skill when users want to:
- Run a script or notebook as a batch job
- Schedule recurring jobs
- Configure job settings (hardware, environment)
- Monitor job status and results
- Run jobs via API or CLI
## What is a Domino Job?
A Job is a batch execution that runs a script or command in Domino. Unlike workspaces, jobs:
- Run to completion without user interaction
- Are fully reproducible with tracked inputs/outputs
- Can be scheduled to run automatically
- Scale to any hardware tier
## Creating and Running Jobs
### Via Domino UI
1. Navigate to your project
2. Click **Jobs** in the navigation
3. Click **Run**
4. Configure:
- **File to Run**: Script path (e.g., `train.py`)
- **Arguments**: Command-line arguments (optional)
- **Hardware Tier**: Select resources
- **Compute Environment**: Select environment
5. Click **Start**
### Via Python (requests)
```python
import requests, os
TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
PROJECT_ID = os.environ["DOMINO_PROJECT_ID"]
headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
# Start a job
response = requests.post(
f"{BASE}/api/jobs/v1/jobs",
headers=headers,
json={
"projectId": PROJECT_ID,
"runCommand": "python train.py --epochs 100",
"title": "Training run",
}
)
job = response.json()
print(f"Job ID: {job['id']}")
```
### Via Domino CLI
```bash
# Start a job with script
domino run train.py
# Run with arguments
domino run train.py arg1 arg2 arg3
# Wait for job to complete
domino run --wait train.py arg1 arg2
# Run direct command (not a script)
domino run --direct "pip freeze | grep pandas"
```
### Via REST API
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
curl -X POST "$DOMINO_API_HOST/api/jobs/v1/jobs" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"projectId\": \"$DOMINO_PROJECT_ID\",
\"runCommand\": \"python train.py\",
\"hardwareTierId\": \"tier-id\",
\"environmentId\": \"env-id\"
}"
```
## Job Commands
### Run Python Script
```bash
python train.py
```
### Run with Arguments
```bash
python train.py --data /mnt/data/train.csv --output /mnt/artifacts/model.pkl
```
### Run Jupyter Notebook
```bash
jupyter nbconvert --to notebook --execute notebook.ipynb
```
### Run R Script
```bash
Rscript analysis.R
```
### Run Shell Script
```bash
bash pipeline.sh
```
## Scheduled Jobs
### Creating a Scheduled Job
1. Go to **Scheduled Jobs** in your project
2. Click **New Scheduled Job**
3. Configure:
- **Name**: Descriptive name
- **Command**: Script to execute
- **Schedule**: Cron expression or preset
- **Hardware Tier**: Resources to use
- **Environment**: Compute environment
### Schedule Examples
| Schedule | Cron Expression |
|----------|-----------------|
| Every hour | `0 0 * * * ?` |
| Daily at midnight | `0 0 0 * * ?` |
| Every Monday 9 AM | `0 0 9 ? * MON` |
| First of month | `0 0 0 1 * ?` |
### Cron Expression Format
Domino scheduled jobs use **Quartz** cron (six required fields, optional seventh year field; seconds first). This is not Unix five-field crontab. Use `?` in day-of-month or day-of-week when the other field is set. Field rules and special characters (`*`, `-`, `/`, `L`, `W`, `#`): [Quartz CronTrigger tutorial](https://www.quartz-scheduler.org/documentation/quartz-2.5.x/tutorials/crontrigger.html).
```
┌───────────── second (0-59)
│ ┌───────────── minute (0-59)
│ │ ┌───────────── hour (0-23)
│ │ │ ┌───────────── day of month (1-31)
│ │ │ │ ┌───────────── month (1-12)
│ │ │ │ │ ┌───────────── day of week (0-7, SUN-SAT)
│ │ │ │ │ │
* * * * * *
```
### Run Modes
**Sequential**: Wait for previous job to complete before starting next
```python
# Good for jobs that depend on previous output
# Example: Daily model retrain that uses previous day's data
```
**Concurrent**: Allow multiple jobs to run simultaneously
```python
# Good for independent jobs
# Example: Hourly data refresh that doesn't depend on previous runs
```
## Job Notifications
### Email Notifications
Configure in job settings:
- Success notifications
- Failure notifications
- Specific email addresses
### Model API Updates
Trigger Model API republish after job completes:
1. In scheduled job settings
2. Select **Update Model API** option
3. Choose the Model API to update
## Accessing Job Results
### Output Files
Files written to `/mnt/` directories are available after job completion:
- `/mnt/results/` - Custom outputs
- `/mnt/artifacts/` - Model artifacts
### Job Logs
View logs in Domino UI or via API:
```python
# Get job logs
logs = domino.runs_get_logs(run_id)
print(logs)
```
### Stdout/Stderr
All print statements and errors are captured in job logs.
## Environment Variables in Jobs
```python
import os
# Domino-provided
run_id = os.environ.get('DOMINO_RUN_ID')
project_name = os.environ.get('DOMINO_PROJECT_NAME')
username = os.environ.get('DOMINO_USER_NAME')
# Custom (set in project or job settings)
api_key = os.environ.get('MY_API_KEY')
```
## Job Best Practices
### 1. Parameterize Scripts
```python
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--data-path', required=True)
parser.add_argument('--model-output', required=True)
parser.add_argument('--epochs', type=int, default=100)
args = parser.parse_args()
```
### 2. Log Progress
```python
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.info("Starting training...")
logger.info(f"Epoch {epoch}/{total_epochs}")
logger.info("Training complete!")
```
### 3. Handle Failures Gracefully
```python
try:
train_model(data)
except Exception as e:
logger.error(f"Training failed: {e}")
# Save checkpoint
save_checkpoint(model, "checkpoint.pt")
raise
```
### 4. Save Artifacts
```python
import joblib
# Save model
joblib.dump(model, "/mnt/artifacts/model.joblib")
# Save metrics
with open("/mnt/artifacts/metrics.json", "w") as f:
json.dump(metrics, f)
```
## Monitoring Jobs
### Job Status
- **Pending**: Waiting for resources
- **Running**: Currently executing
- **Succeeded**: Completed successfully
- **Failed**: Exited with error
- **Stopped**: Manually cancelled
### Check Status via API
```python
status = domino.runs_status(run_id)
print(f"Status: {status['status']}")
print(f"Started: {status['startedAt']}")
```
### Stop a Running Job
```python
domino.runs_stop(run_id)
```
## Troubleshooting
### Job Fails Immediately
- Check script syntax
- Verify file paths exist
- Check environment has required packages
### Job Times Out
- Increase hardware tier resources
- Optimize code performance
- Check for infinite loops
### Out of Memory
- Use larger hardware tier
- Optimize data loading (chunking, generators)
- Clear variables when no longer needed
## API Reference
Before writing or verifying any API call, confirm endpoint paths and field names in [API-SPECS.md](../domino-api-intro/API-SPECS.md) (use the **Public routes** section for `/api/jobs/…`; use **Internal routes** for legacy `/v4/jobs/…` and `/v4/runs/…`). Use public docs for workflow context and field explanations.
**Get the cluster base URL:** `$DOMINO_API_HOST` (injected by Domino into every workspace, job, and app).
**Public docs (workflow context and field explanations):**
- [API Guide](https://docs.dominodatalab.com/en/latest/api_guide/f35c19/api-guide/)
- [Create and run Jobs](https://docs.dominodatalab.com/en/latest/user_guide/af97b7/create-and-run-jobs/)
- [Scheduled Jobs](https://docs.dominodatalab.com/en/latest/user_guide/673577/scheduled-jobs/)
- [Work with Jobs](https://docs.dominodatalab.com/en/latest/user_guide/942549/work-with-jobs/)
---
name: domino-launchers
description: Create Domino Launchers - parameterized web forms for self-service job execution. Enable business users to run analyses, generate reports, and trigger batch predictions without coding. Covers parameter types, email notifications, result delivery, and access control. Use when building self-service data products or enabling non-technical users.
---
# Domino Launchers Skill
## Description
This skill helps users create and use Domino Launchers - web forms that allow non-technical users to run parameterized jobs and receive results.
## Activation
Activate this skill when users want to:
- Create self-service data products
- Build parameterized job interfaces
- Enable business users to run analyses
- Generate reports on demand
- Share reproducible workflows
## What is a Launcher?
A Launcher is:
- **Web Form**: UI for entering parameters
- **Job Trigger**: Runs a script with user inputs
- **Results Delivery**: Sends output via email/dashboard
- **Self-Service**: Business users can run without coding
## Use Cases
- **Report Generation**: Parameterized business reports
- **Batch Predictions**: Score data on demand
- **Data Exports**: Custom data extracts
- **Analysis Requests**: Ad-hoc analytics
- **Model Testing**: Test models with different inputs
## Creating a Launcher
### Via Domino UI
1. Go to your project
2. Navigate to **Deployments** > **Launchers**
3. Click **New Launcher**
4. Configure:
- **Name**: Descriptive name
- **Description**: What the launcher does
- **Command**: Script to run
- **Parameters**: Input fields
- **Hardware Tier**: Resources
- **Environment**: Compute environment
### Launcher Script (Python)
```python
# generate_report.py
import argparse
import pandas as pd
# Parse launcher parameters
parser = argparse.ArgumentParser()
parser.add_argument('--start-date', required=True)
parser.add_argument('--end-date', required=True)
parser.add_argument('--region', default='all')
args = parser.parse_args()
# Generate report
df = generate_report(args.start_date, args.end_date, args.region)
# Save results (will be available to launcher user)
df.to_csv('/mnt/results/report.csv', index=False)
df.to_html('/mnt/results/report.html', index=False)
```
### Launcher Script (R)
```r
# launcher.R
args <- commandArgs(trailingOnly = TRUE)
a <- as.integer(args[1])
b <- as.integer(args[2])
if (is.na(a)) {
print("A is not a number")
} else if (is.na(b)) {
print("B is not a number")
} else {
paste("The sum of", a, "and", b, "is:", a + b)
}
```
Command: `launcher.R ${A} ${B}`
### Launcher Command
```bash
# Python script with named arguments
python generate_report.py --start-date ${start_date} --end-date ${end_date} --region ${region}
# Python script with positional arguments
my_script.py -x=1 ${file} ${start_date}
# R script with positional arguments
launcher.R ${A} ${B}
```
**Note:** Parameter values are enclosed in single quotes, preserving special characters. File parameters pass the file path. Multi-select parameters pass comma-separated values.
## Parameter Types
### Text Input
```yaml
name: customer_name
type: text
label: Customer Name
required: true
default: ""
```
### Dropdown/Select
```yaml
name: region
type: select
label: Region
options:
- North America
- Europe
- Asia Pacific
default: North America
```
### Date Picker
```yaml
name: start_date
type: date
label: Start Date
required: true
```
### Number
```yaml
name: quantity
type: number
label: Quantity
min: 1
max: 1000
default: 100
```
### File Upload
```yaml
name: input_file
type: file
label: Input File
accept: .csv,.xlsx
```
## Generating Results
### File-Based Results
Any files created in `/mnt/results/` are available as results:
```python
# Save multiple output formats
df.to_csv('/mnt/results/data.csv')
df.to_excel('/mnt/results/data.xlsx')
fig.savefig('/mnt/results/chart.png')
```
### HTML Email Content
Create `email.html` for custom email body:
```python
# Generate HTML for email
html_content = f"""
<html>
<body>
<h1>Report for {args.start_date} to {args.end_date}</h1>
<p>Summary: {summary}</p>
{df.to_html()}
</body>
</html>
"""
with open('/mnt/results/email.html', 'w') as f:
f.write(html_content)
```
### Rich Reports
Use notebooks for rich reports:
```python
# Use papermill to execute parameterized notebook
import papermill as pm
pm.execute_notebook(
'report_template.ipynb',
'/mnt/results/report.ipynb',
parameters={
'start_date': args.start_date,
'end_date': args.end_date
}
)
```
## Notifications
### Email Configuration
Configure notification recipients:
1. In launcher settings
2. Add email addresses
3. Results sent automatically on completion
### Email Contents
- Link to results in Domino
- Attached files (if enabled)
- Custom HTML body (if email.html created)
## Access Control
### Who Can Run Launchers
- **Contributors**: Can create and run launchers
- **Launcher Users**: Can run launchers only
- **Results Consumers**: Can view results only
### Setting Permissions
1. Go to Project Settings
2. Add users with appropriate roles
3. Share launcher URL with users
## Running a Launcher
### Via UI
1. Go to launcher page
2. Fill in parameters
3. Click **Run**
4. Wait for results (email notification)
### Via API
```python
import requests, os
TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
response = requests.post(
f"{BASE}/v4/launchers/{{launcher_id}}/run",
headers={"Authorization": f"Bearer {TOKEN}"},
json={
"parameters": {
"start_date": "2024-01-01",
"end_date": "2024-01-31",
"region": "North America"
}
}
)
run_id = response.json()["runId"]
```
## Viewing Results
### Via Email
Results link sent to configured recipients.
### Via Jobs Dashboard
Each launcher run creates a job:
1. Go to **Jobs** in project
2. Find the launcher job
3. View results in job details
### Programmatic Access
```python
# Get launcher run results
results = domino.runs_get_results(run_id)
```
## Example: Scoring Launcher
### Script (score_data.py)
```python
import argparse
import pandas as pd
import joblib
parser = argparse.ArgumentParser()
parser.add_argument('--input-file', required=True)
parser.add_argument('--output-format', default='csv')
args = parser.parse_args()
# Load model
model = joblib.load('/mnt/artifacts/model.joblib')
# Load and score data
df = pd.read_csv(args.input_file)
predictions = model.predict(df)
df['prediction'] = predictions
# Save results
if args.output_format == 'csv':
df.to_csv('/mnt/results/predictions.csv', index=False)
else:
df.to_excel('/mnt/results/predictions.xlsx', index=False)
```
### Launcher Configuration
```yaml
name: Score Customer Data
command: python score_data.py --input-file ${input_file} --output-format ${output_format}
parameters:
- name: input_file
type: file
label: Customer Data (CSV)
required: true
- name: output_format
type: select
label: Output Format
options: [csv, xlsx]
default: csv
```
## Best Practices
### 1. Clear Parameter Names
Use descriptive labels users understand.
### 2. Input Validation
```python
# Validate inputs in script
if args.end_date < args.start_date:
raise ValueError("End date must be after start date")
```
### 3. Progress Logging
```python
print("Loading data...")
print(f"Processing {len(df)} records...")
print("Generating report...")
print("Complete!")
```
### 4. Error Handling
```python
try:
process_data()
except Exception as e:
# Save error message as result
with open('/mnt/results/error.txt', 'w') as f:
f.write(f"Error: {str(e)}")
raise
```
### 5. Documentation
Include help text in launcher description.
## Troubleshooting
### Launcher Fails
- Check script runs manually first
- Verify file paths are correct
- Review job logs for errors
### No Email Received
- Check email addresses configured
- Verify email server settings (admin)
- Check spam folder
### Wrong Results
- Verify parameter passing
- Check variable substitution syntax
- Test with known inputs
## Documentation Reference
- [Use Launchers](https://docs.dominodatalab.com/en/latest/user_guide/f4e1e3/use-launchers/)
---
name: domino-model-endpoints
description: Deploy and monitor model API endpoints in Domino. Covers creating prediction endpoints, version management, Grafana dashboards for latency/errors/resources, alerting, and GPU inference with NVIDIA Triton. Use when deploying models as APIs, monitoring production endpoints, or debugging endpoint issues.
---
# Domino Model Endpoints Skill
This skill provides comprehensive knowledge for deploying and monitoring model API endpoints in Domino Data Lab.
## Key Concepts
### Model Endpoints Overview
Domino Model Endpoints provide:
- REST API for model predictions
- Automatic scaling and load balancing
- Version management
- Built-in monitoring with Grafana
- Authentication via API tokens
### Endpoint Lifecycle
```
Train Model → Register → Deploy Endpoint → Monitor → Update Version
```
## Related Documentation
- [domino-model-serving/SKILL.md](../domino-model-serving/SKILL.md) - REST deploy, registry v1/v2, lifecycle pitfalls
- [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) - modelServing routes and invoke split
- [DEPLOY-ENDPOINT.md](./DEPLOY-ENDPOINT.md) - Creating model APIs
- [MONITORING.md](./MONITORING.md) - Grafana, metrics, alerts
- [SCALING.md](./SCALING.md) - GPU inference, Triton, scaling
## Environment Requirements
**Important:** Model APIs use the **default environment** set for your project. The environment must have the `uwsgi` Python package installed for model endpoints to work.
### Required Package
```dockerfile
# Add to your environment's Dockerfile instructions
RUN pip install uwsgi
```
Or in requirements.txt:
```
uwsgi
```
### Setting Default Environment
1. Go to **Project Settings** → **Execution Preferences**
2. Set the **Default Environment** that includes `uwsgi`
3. This environment will be used for all Model API deployments
## Quick Start
### 1. Create Endpoint Function
```python
# model.py
def predict(features):
"""
Domino calls this function for predictions.
Args:
features: Input data (dict, list, or primitive)
Returns:
JSON-serializable prediction result
"""
import pickle
# Load model (cached after first call)
with open('model.pkl', 'rb') as f:
model = pickle.load(f)
prediction = model.predict([features])
return {"prediction": prediction.tolist()}
```
### 2. Deploy via Domino UI
1. Go to **Publish** → **Model APIs**
2. Click **New Model**
3. Configure:
- Name: `my-classifier`
- File: `model.py`
- Function: `predict`
- Environment: Select compute environment
4. Click **Publish**
### 3. Call the Endpoint
```bash
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-d '{"data": {"features": [1.0, 2.0, 3.0]}}' \
https://your-domino.com/models/abc123/latest/model
```
## Environment Variables
When calling endpoints from apps:
| Variable | Description |
|----------|-------------|
| `MODEL_API_URL` | Full endpoint URL |
| `MODEL_API_TOKEN` | Bearer token for authentication |
## Key Metrics to Monitor
| Metric | Target |
|--------|--------|
| Latency P50 | < 100ms |
| Latency P99 | < 500ms |
| Error Rate | < 1% |
| CPU Usage | < 80% |
| Memory | Stable (no growth) |
## Documentation Links
OpenAPI and route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md) (public routes section).
Referenced files: 3 — file references are metadata, not package contents.
--- name: domino-modeling-assistant description: Enable AI-assisted model development within Domino by writing needed model and training code and using MCP (Model Context Protocol) servers to execute domino jobs. AI coding assistants like Cursor and GitHub Copilot can execute commands as Domino jobs, maintaining security, governance, and reproducibility. Use when setting up AI code assistants to work with Domino, configuring MCP servers, or enabling vibe modeling workflows. --- # Introduction This skill provides comprehensive knowledge for enabling modeling assistant within Domino Data Lab, allowing AI coding assistants to interact with the platform. Vibe modeling refers to using AI code assistants to go beyond pure code generation and assist with: - Experiment setup and configuration - Data analysis and exploration - Model training and evaluation - Results interpretation ## Key Components ### MCP (Model Context Protocol) MCP servers bridge AI coding assistants with the Domino platform APIs we typically need for running jobs (because its better to run analysis and training scripts via jobs than locally), checking job activity and results, saving files to DFS (domino file system) in cases where the project isn't using a git repo. The Domino MCP Server is **bundled with this plugin** and starts automatically when the plugin is installed from its source repository. Skills-only distributions don't include it. If the `domino_server` tools (such as `run_domino_job`) aren't available in this session, don't pretend to call them. Tell the user, point them at [SETUP.md](./SETUP.md#register-the-server-manually), and meanwhile use the `domino-jobs` and `domino-python-sdk` skills to run work through the Domino REST API. - **Inside a Domino workspace:** Authentication and project detection are fully automatic. - **Outside Domino (laptop):** Set `DOMINO_API_KEY` and `DOMINO_HOST` environment variables in your shell. See [SETUP.md](./SETUP.md) for details. ## Related Documentation - [SETUP.md](./SETUP.md) - Complete setup guide for the modeling assistant ## Important Considerations You are a Domino Data Lab powered agentic coding tool that helps write code in addition to running tasks on the Domino Data Lab platform on behalf of the user using available tool functions provided by the domino_server MCP server. Whenever possible run commands as domino jobs rather than on the local terminal. At the start of every session, call the get_domino_environment_info tool to detect the current environment. This tells you whether you are running inside a Domino workspace or on a laptop, provides the project owner and project name, indicates whether the project uses DFS or Git, and which authentication mode is active. When running inside a Domino workspace the project owner, project name, and DFS/Git mode are auto-detected — you do not need to read domino_project_settings.md. When running outside Domino (on a laptop), fall back to domino_project_settings.md for the project name, user name, and dfs setting. When running a job, always check its status and results if completed and briefly explain any conclusions from the result of the job run. If a job result ever includes an mlflow or experiment run URL, always share that with the user using the open_web_browser tool. Any requests related to understanding or manipulating project data should assume a dataset file is already part of the domino project and accessible via job runs. Always create scripts to understand and transform data via job runs. The script can assume all project data is accessible under the '/mnt/data/' directory or the '/mnt/imported/data/' directory, be sure to understand the full path to a dataset file before using it by running a job to list all folder contents recursively. Analytical outputs should be in plain text tabular format sent to stdout, this makes it easier to check results from the job run. If the project is DFS instead of Git based (auto-detected inside Domino, or dfs=true in domino_project_settings.md when outside Domino), the datasets path is under /domino/datasets/* Any scripts used to analyze or transform data within a Domino project should not be deleted. When performing analysis, generate useful summary charts in an image format and save to the project files. Always check if our local project has uncommitted changes. For Git-based projects, you must commit and push changes before attempting to run any domino jobs otherwise domino can't see the new file changes. For DFS-based projects (auto-detected or dfs=true in domino_project_settings.md), use the MCP Server file sync functions (upload_file_to_domino_project, smart_sync_file, etc.) instead of git before running any jobs. When training a model use mlflow instrumentation assuming a server is running, no need to set the url or anything, it should just work. If domino_project_settings.md is not present, the owner and project_name can potentially be determined by environment variables if running inside a Domino workspace: DOMINO_PROJECT_NAME DOMINO_PROJECT_OWNER ## MCP Server Tools The Domino MCP Server (bundled at `mcp-servers/domino_mcp_server/`) provides tools for interacting with jobs and the DFS file system (if not using git). | Tool | Description | |------|-------------| | `get_domino_environment_info` | Detect workspace vs laptop, project info, auth mode | | `run_domino_job` | Execute commands as Domino jobs | | `check_domino_job_run_status` | Check if a job is finished, in-progress, or errored | | `check_domino_job_run_results` | Retrieve stdout results from a completed job | | `open_web_browser` | Open a URL (e.g. MLflow experiment link) in the browser | | `list_domino_project_files` | List files in a DFS project | | `upload_file_to_domino_project` | Upload a file to a DFS project | | `download_file_from_domino_project` | Download a file from a DFS project | | `sync_local_file_to_domino` | Read a local file and upload it to a DFS project | | `smart_sync_file` | Upload with conflict detection for DFS projects | Upstream source: https://github.com/dominodatalab/domino_mcp_server
Referenced files: 1 — file references are metadata, not package contents.
---
name: domino-model-monitoring
description: Monitor deployed models in Domino including drift detection, model quality tracking, and alerting. Covers data drift analysis, prediction capture, baseline comparison, alert configuration, and remediation workflows. Use when monitoring production models, detecting drift, or setting up model health alerts.
---
# Domino Model Monitoring Skill
## Description
This skill helps users monitor deployed models in Domino, including drift detection, model quality tracking, and alerting.
## Activation
Activate this skill when users want to:
- Monitor deployed model performance
- Set up drift detection
- Configure monitoring alerts
- Analyze prediction data
- Understand model degradation
## What is Model Monitoring?
Domino Model Monitoring provides:
- **Data Drift Detection**: Detect changes in input data distributions
- **Model Quality Tracking**: Monitor prediction accuracy over time
- **Alerting**: Get notified when metrics exceed thresholds
- **Prediction Capture**: Log predictions for analysis
- **Reproducibility**: Diagnose issues with captured data
## Setting Up Monitoring
### Prerequisites
1. Deployed Model API in Domino
2. Training dataset (for baseline)
3. Ground truth data (optional, for quality metrics)
### Enable Monitoring
1. Go to your Model API page
2. Click **Monitoring** tab
3. Click **Set Up Monitoring**
4. Upload training dataset
5. Configure drift detection settings
### Register Training Data
```python
# Training data provides baseline for drift detection
# Upload via UI or programmatically
import pandas as pd
# Your training data
train_df = pd.read_csv("training_data.csv")
# Save for monitoring setup
train_df.to_csv("/mnt/artifacts/training_data.csv", index=False)
```
## Drift Detection
### Types of Drift
| Drift Type | Description |
|------------|-------------|
| **Data Drift** | Input feature distributions change |
| **Concept Drift** | Relationship between inputs and outputs changes |
| **Prediction Drift** | Output distribution changes |
### Statistical Tests
Domino supports multiple drift detection tests:
| Test | Best For |
|------|----------|
| **Kullback-Leibler Divergence** | General-purpose, most common |
| **Population Stability Index (PSI)** | Finance industry standard |
| **Wasserstein Distance** | Comparing distributions |
| **Energy Distance** | Multivariate distributions |
### Configure Drift Detection
1. Go to Model API > **Monitoring**
2. Click **Configure Drift Detection**
3. For each feature:
- Select test type
- Set threshold
- Enable/disable alerts
### Example Thresholds
| Test | Low Drift | Medium Drift | High Drift |
|------|-----------|--------------|------------|
| KL Divergence | < 0.1 | 0.1 - 0.2 | > 0.2 |
| PSI | < 0.1 | 0.1 - 0.25 | > 0.25 |
## Prediction Capture
### How It Works
Domino automatically captures predictions:
1. Model receives request
2. Prediction is made
3. Input/output logged to dataset
4. Data available for drift analysis
### Access Captured Data
```python
import pandas as pd
# Predictions captured in Domino Dataset
predictions_df = pd.read_parquet(
"/mnt/data/model-predictions/predictions.parquet"
)
print(predictions_df.head())
```
### Capture Frequency
- Predictions batched hourly
- Full data available in monitoring dataset
- Retention configurable by admin
## Model Quality Monitoring
### With Ground Truth
If you provide ground truth labels:
```python
# Upload ground truth
ground_truth = pd.DataFrame({
"prediction_id": [...],
"actual_label": [...]
})
# Upload to monitoring
ground_truth.to_csv("/mnt/artifacts/ground_truth.csv", index=False)
```
### Quality Metrics
- Accuracy
- Precision/Recall
- F1 Score
- AUC-ROC
- Mean Squared Error (regression)
### Schedule Quality Checks
1. Go to Monitoring > **Quality**
2. Upload ground truth dataset
3. Configure metric thresholds
4. Set check frequency
## Alerting
### Configure Alerts
1. Go to Model API > **Monitoring**
2. Click **Alerts**
3. Configure:
- Metric to monitor
- Threshold
- Alert recipients (email)
### Alert Types
- Drift threshold exceeded
- Quality metric below threshold
- Model API health issues
- Prediction volume anomalies
### Disable Noisy Alerts
Click the bell icon next to features to exclude from alerts.
## Viewing Monitoring Data
### Monitoring Dashboard
Go to Model API > **Monitoring** to see:
- Drift trends over time
- Feature distributions
- Quality metrics
- Alert history
### Export Data
```python
# Export monitoring data for custom analysis
import pandas as pd
drift_report = pd.read_csv("/mnt/data/monitoring/drift_report.csv")
print(drift_report)
```
## Responding to Drift
### Investigation Workflow
1. **Alert received**: Drift detected on feature X
2. **Investigate**: View feature distribution changes
3. **Diagnose**: Compare current vs training data
4. **Action**: Retrain or update model
### Retrain Model
```python
# When drift is detected, retrain with recent data
from sklearn.ensemble import RandomForestClassifier
# Load recent data
recent_data = pd.read_csv("/mnt/data/recent_predictions.csv")
# Combine with ground truth
training_data = merge_with_ground_truth(recent_data)
# Retrain
model = RandomForestClassifier()
model.fit(training_data[features], training_data[label])
# Deploy new version
joblib.dump(model, "/mnt/artifacts/model_v2.joblib")
```
### Automated Retraining
Set up scheduled job to retrain when drift detected:
```python
# scheduled_retrain.py
from domino import Domino
domino = Domino("project/model-project")
# Check drift status
drift_status = check_drift_metrics()
if drift_status["max_drift"] > 0.2:
# Trigger retrain job
domino.runs_start(
command="python retrain.py",
hardware_tier_name="medium"
)
```
## Best Practices
### 1. Baseline with Quality Data
Use clean, representative training data for baseline.
### 2. Monitor Key Features
Focus on features with highest importance:
```python
# Identify important features
importances = model.feature_importances_
top_features = sorted(
zip(feature_names, importances),
key=lambda x: x[1],
reverse=True
)[:10]
```
### 3. Set Appropriate Thresholds
- Start with conservative thresholds
- Adjust based on business impact
- Different thresholds for different features
### 4. Include Business Context
Not all drift requires action:
- Seasonal variations may be expected
- New customer segments may cause drift
- Consider business impact before reacting
### 5. Regular Reviews
Schedule periodic monitoring reviews:
- Weekly: Check drift trends
- Monthly: Review alert configurations
- Quarterly: Assess model performance
## Troubleshooting
### No Data in Monitoring
- Verify Model API is receiving traffic
- Check prediction capture is enabled
- Wait for hourly batch processing
### Drift Always High
- Review training data quality
- Check for data preprocessing differences
- Verify feature encoding consistency
### Alerts Not Sending
- Check email configuration
- Verify alert thresholds
- Review spam folders
## Documentation Reference
- [Drift detection for monitored models](https://docs.dominodatalab.com/en/latest/user_guide/86bc1f/drift-detection-for-monitored-models/)
- [Domino endpoint drift detection](https://docs.dominodatalab.com/en/latest/user_guide/c97091/domino-endpoint-drift-detection/)
- [Monitor workflows](https://docs.dominodatalab.com/en/latest/user_guide/85f76d/monitor-workflows/)
---
name: domino-model-serving
description: Deploy, invoke, and retire Domino model APIs and registered models via REST. Covers modelServing lifecycle, registered-models v1 vs v2 paths, MLflow tracking vs registry API, inference URLs vs management API, and GenAI endpoint vanity URLs. Use when automating model deployment, predictions, registry updates, or debugging stop/archive/delete behavior.
---
# Domino model serving (REST)
Programmatic model deployment and inference on Domino. UI-focused endpoint monitoring stays in [model-endpoints](../model-endpoints/SKILL.md). Shared REST field catalogs live in [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) and [API-MODELS.md](../python-sdk/API-MODELS.md).
Authentication: https://docs.domino.ai/cloud/reference/api/domino-api-authentication . **Do not use API keys.**
## Configuration
| Context | Base | Authorization |
|---------|------|----------------|
| In-run, `DOMINO_API_PROXY` set | `{DOMINO_API_PROXY}` | None |
| In-run, no proxy | `{DOMINO_USER_HOST or DOMINO_API_HOST}` | Bearer from `http://localhost:8899/access-token` |
| Outside run | Public deployment URL | Bearer PAT or SA |
Management paths include `/api/modelServing/v1/...`, `/api/registeredmodels/v1|v2/...`. Confirm with `GET {base}/api/modelServing/v1/modelApis?limit=1` returning HTTP 200.
```bash
curl ${TOKEN:+-H "Authorization: Bearer $TOKEN"} "$BASE/api/modelServing/v1/modelApis?limit=1"
```
## Three surfaces (do not merge)
| Surface | Purpose |
|---------|---------|
| **Model Serving REST** | `/api/modelServing/v1/modelApis` deploy, versions, lifecycle |
| **Registered models REST** | `/api/registeredmodels/v2` list/register; `/api/registeredmodels/v1/{modelName}` get/update/versions |
| **MLflow tracking** | `MLFLOW_TRACKING_URI` / runs UI; not the same as registered-models REST |
Logging a run to MLflow does not replace registry or modelServing calls for deployment automation.
## Management vs inference URL
- Create/list/update: platform base + `/api/modelServing/v1/...` (or registered-models paths).
- **Predict:** use the **`url`** field on the model API or version (often `.../models/.../latest/model`), on the deployment ingress. That path is not routed through `DOMINO_API_HOST` sidecar the same way as `/api/`.
GenAI: management API vs `https://.../endpoints/{vanity}` split. See [model-endpoints](../model-endpoints/SKILL.md) and the GenAI section in [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md).
## Lifecycle (model APIs)
Typical order:
1. Train / register (MLflow and/or `POST /api/registeredmodels/v2/...`).
2. `POST /api/modelServing/v1/modelApis` (or new version on existing API).
3. Poll `GET /api/modelServing/v1/modelApis/{modelApiId}` until `status` is `Running` (or `Failed`); then read **`url`** for invoke.
4. **Stop** vs **archive** vs **delete** are different product operations; names and reliability differ by route.
5. Treat **DELETE** on deployments/model APIs as best-effort; verify resource gone before assuming cleanup.
6. **Stop** may be long-running and not idempotent; poll status instead of fire-and-forget retry loops.
Details and route list: [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md).
## Registered model path key
`modelName` in `/api/registeredmodels/v1/{modelName}` is the **registered model name string**, not an opaque UUID. List/register often use **v2**; get/update/versions use **v1** with that name in the path.
## Invocation auth
Use the model API token documented on https://docs.domino.ai for the model endpoint invoke path. Older examples may show a basic-auth pattern with the token as both user and password; treat that as legacy and confirm in [API-SPECS.md](../domino-api-intro/API-SPECS.md) before hardcoding.
## Related documentation
- [API-MODEL-SERVING.md](../python-sdk/API-MODEL-SERVING.md) - REST lifecycle and routes
- [API-MODELS.md](../python-sdk/API-MODELS.md) - broader models API catalog
- [model-endpoints](../model-endpoints/SKILL.md) - Grafana, Triton, UI deploy
- Model registry (product): https://docs.domino.ai/cloud/platform-capabilities/features/model-registry
---
name: domino-projects
description: Work with Domino Projects including Git integration, DFS vs Git-based projects, collaboration, and version control. Covers project creation, Git provider setup (GitHub, GitLab, Bitbucket), branch management, collaborator permissions, and project settings. Use when creating projects, setting up Git repos, or managing team collaboration.
---
# Domino Projects and Git Skill
## Description
This skill helps users work with Domino Projects - including project creation, Git integration, version control, and collaboration features.
## Activation
Activate this skill when users want to:
- Create or configure a Domino project
- Set up Git integration
- Understand DFS vs Git-based projects
- Collaborate with team members
- Manage project settings and permissions
## Project Types
### Git-Based Projects
- Code stored in external Git provider (GitHub, GitLab, Bitbucket)
- Full Git workflow support
- Better for teams familiar with Git
- Access to Git provider features (PRs, issues)
### Domino File System (DFS) Projects
- Code stored in Domino's internal file system
- Automatic versioning (Git under the hood)
- Simpler for users unfamiliar with Git
- Good for research and experimentation
## Creating a Project
### Via Domino UI
1. Click **New Project**
2. Enter project name
3. Select project type:
- **Git-based**: Connect to Git repository
- **DFS**: Use Domino File System
4. Configure settings
5. Click **Create**
### Via Python SDK
```python
from domino import Domino
domino = Domino()
# Create DFS project
project = domino.project_create(
project_name="my-project",
owner_name="your-username"
)
print(f"Project ID: {project['id']}")
```
## Git-Based Projects
### Connecting to Git Repository
1. Create project as Git-based
2. Enter repository URL
3. Configure authentication:
- **SSH Key**: Add SSH public key to Git provider
- **HTTPS**: Use personal access token
### Git Credentials
```bash
# Add SSH key to Domino account
# Go to Account Settings > Git Credentials
# Or use HTTPS with token
https://username:token@github.com/org/repo.git
```
### Working with Git in Workspaces
#### Sync Changes
Click **Sync** in workspace to push changes:
1. Domino commits changes
2. Pushes to configured branch
3. Updates project state
#### Manual Git Commands
```bash
# Check status
git status
# Add and commit
git add .
git commit -m "Update model training"
# Push to remote
git push origin main
# Pull latest
git pull origin main
```
#### IDE Git Integration
- VS Code: Use Source Control panel
- JupyterLab: Use Git extension
- RStudio: Use Git pane
### Branch Management
```bash
# Create feature branch
git checkout -b feature/new-model
# Work on branch
git add .
git commit -m "Add new model"
git push origin feature/new-model
# Switch branches
git checkout main
```
### Multi-Repository Support
Domino supports multiple Git repositories per project:
1. Go to Project Settings > **Repositories**
2. Add additional repositories
3. Configure mount paths
4. Enable/disable per execution
```python
# Access code from multiple repos
# Repo 1: /mnt/code/main-repo/
# Repo 2: /mnt/code/shared-lib/
from shared_lib import utils
```
## DFS Projects
### How DFS Works
- Files stored in Domino-managed Git
- Automatic commits on sync
- Version history tracked
- Simplified workflow
### Sync Files
1. Work in workspace
2. Click **Sync** when ready
3. Enter commit message
4. Files pushed to Domino
### View History
1. Go to **Files** in project
2. Click **History**
3. View all commits
4. Restore previous versions
## Project Structure
### Recommended Structure
```
project/
├── README.md # Project documentation
├── requirements.txt # Python dependencies
├── data/ # Local data (small files)
├── src/ # Source code
│ ├── train.py
│ ├── evaluate.py
│ └── utils.py
├── notebooks/ # Jupyter notebooks
│ └── exploration.ipynb
└── tests/ # Unit tests
└── test_model.py
```
### Special Paths
- `/mnt/code/`: Project files
- `/mnt/data/`: Domino Datasets
- `/mnt/artifacts/`: Output artifacts
- `/mnt/imported/`: Imported project files
## Collaboration
### Project Roles
| Role | Permissions |
|------|-------------|
| **Owner** | Full control, delete project |
| **Admin** | Manage members, settings |
| **Contributor** | Edit files, run executions |
| **Launcher User** | Run launchers only |
| **Results Consumer** | View results only |
### Adding Collaborators
1. Go to Project Settings > **Access**
2. Click **Add Collaborator**
3. Search for user
4. Assign role
5. Click **Add**
### Sharing Projects
- **Private**: Only invited users
- **Organization**: All org members can view
- **Public**: Anyone in Domino instance
## Project Settings
### Environment
Set default compute environment:
1. Go to Project Settings > **Environment**
2. Select default environment
3. Optionally allow overrides
### Hardware Tier
Set default resources:
1. Go to Project Settings > **Hardware Tier**
2. Select default tier
3. Set GPU preferences
### Environment Variables
Add project-level variables:
```
API_KEY=your-api-key
MODEL_VERSION=v2.0
DEBUG=false
```
Access in code:
```python
import os
api_key = os.environ.get('API_KEY')
```
## Importing/Exporting Projects
### Import Project
1. Click **New Project**
2. Select **Import**
3. Choose source:
- Git repository
- ZIP file
- Another Domino project
### Export Project
```bash
# Download project files
domino download project-owner/project-name
```
### Fork Project
1. Go to project page
2. Click **Fork**
3. Creates copy in your account
## Reproducibility
### Execution Records
Every execution tracked:
- Exact code version
- Environment used
- Hardware tier
- Inputs/outputs
- Start/end times
### Reproduce Past Execution
1. Go to Jobs or Workspaces
2. Find execution to reproduce
3. Click **Reproduce**
4. Creates workspace with same settings
### Environment Locking
Lock environment to specific revision:
```python
# In project settings or via SDK
domino.project_update(
project_name="my-project",
environment_id="env-specific-revision"
)
```
## Best Practices
### 1. Use Git for Code
Keep code in version control for:
- Change tracking
- Code reviews
- Collaboration
- Rollback capability
### 2. Separate Code and Data
- Code: Git repository
- Large data: Domino Datasets
- Artifacts: `/mnt/artifacts/`
### 3. Document Projects
Include README with:
- Project purpose
- Setup instructions
- How to run
- Dependencies
### 4. Use Environments
Don't install packages ad-hoc; use:
- Dockerfile instructions
- requirements.txt
- Environment configuration
### 5. Regular Commits
```bash
# Commit frequently with meaningful messages
git commit -m "Add data preprocessing step"
git commit -m "Fix model evaluation bug"
```
## Troubleshooting
### Git Authentication Failed
- Verify credentials in Domino account settings
- Check token hasn't expired
- Ensure repository access permissions
### Sync Conflicts
```bash
# Pull latest first
git pull origin main
# Resolve conflicts
git status
# Edit conflicting files
git add .
git commit -m "Resolve conflicts"
```
### Files Not Appearing
- Check file is in correct path
- Verify sync completed
- Refresh browser
## Documentation Reference
- [Version control and Git](https://docs.dominodatalab.com/en/latest/user_guide/bd26e2/version-control-and-git/)
- [Git-based Projects](https://docs.dominodatalab.com/en/latest/user_guide/910370/git-based-projects/)
- [Use Git in your Workspace](https://docs.dominodatalab.com/en/cloud/user_guide/0d2247/use-git-in-your-workspace/)
- [Projects overview](https://docs.dominodatalab.com/en/cloud/user_guide/5b332c/projects-overview/)
---
name: domino-python-sdk
description: Programmatically interact with Domino using python-domino SDK and REST APIs. Covers authentication, running jobs, managing projects, file operations, model deployment, and automation. Use when automating Domino workflows, integrating with CI/CD, or building custom tooling around Domino.
---
# Domino Python SDK Skill
## Description
This skill helps users work with the Domino Python SDK (python-domino) and REST APIs to programmatically interact with Domino.
## Activation
Activate this skill when users want to:
- Use the Domino Python SDK
- Make API calls to Domino
- Automate Domino workflows
- Integrate Domino with external systems
- Query Domino programmatically
## Overview
Domino provides two main programmatic interfaces:
- **python-domino**: Python SDK for common operations
- **REST API**: Full HTTP API for all Domino features
## Installation
### python-domino
```bash
# Install from PyPI
pip install dominodatalab
# Or install with extras
pip install "dominodatalab[data]"
```
### In Domino Environment
Add to requirements.txt:
```
dominodatalab>=1.4.0
```
Or Dockerfile:
```dockerfile
RUN pip install dominodatalab
```
## Authentication
Canonical guide: https://docs.domino.ai/cloud/reference/api/domino-api-authentication
**Do not use API keys** (`X-Domino-Api-Key`, `DOMINO_USER_API_KEY`, `api_key=`). Use PAT or service account tokens only when calling from **outside** a run.
### In-run (workspace, job, app backend)
Domino injects `DOMINO_USER_HOST` / `DOMINO_API_HOST` (same base; prefer `DOMINO_USER_HOST`). When JWT credential propagation is enabled, `DOMINO_API_PROXY` is the forwarded platform API base (not the same as `http://localhost:8899/access-token`).
| Pattern | When | Code |
|---------|------|------|
| **API proxy (preferred)** | `DOMINO_API_PROXY` is set (Domino **5.4.0+** in runs with JWT credential propagation configured) | Call `{DOMINO_API_PROXY}{path}` with **no** `Authorization` header. The JWT sidecar adds the **starting user** access JWT on the forwarded request. |
| **Access token + platform host** | Older deployments, or you intentionally call `DOMINO_USER_HOST` / `DOMINO_API_HOST` instead of the proxy URL | Fetch a short-lived JWT, then Bearer on the platform base. |
**API proxy (preferred on 5.4.0+):**
```python
import os
import requests
base_url = os.environ["DOMINO_API_PROXY"].rstrip("/")
headers = {}
response = requests.get(f"{base_url}/v4/users/self")
```
```bash
curl "$DOMINO_API_PROXY/v4/users/self"
```
The proxy is not guaranteed on every deployment or run type. If `DOMINO_API_PROXY` is missing, use the access-token pattern or PAT from outside the cluster.
**Access token + `DOMINO_USER_HOST` (legacy-friendly in-run):** use the [In-run setup block](#in-run-setup-block-for-examples-in-this-skill) below; it covers both proxy and access-token paths.
**Before Domino 5.4.0:** JWT file propagation (`DOMINO_TOKEN_FILE`) was used before the API proxy; legacy behavior may still exist if admins enable `EnableLegacyJwtTooling`. Prefer migrating to the proxy pattern on supported versions.
### Outside a run (laptop, CI, cron outside Domino)
Your code is **not** executing inside a Domino workspace, job, or app container. Domino does **not** inject `DOMINO_API_PROXY`, `DOMINO_USER_HOST`, or an access-token sidecar. You must supply both:
1. **Base URL** — the deployment URL users open in the browser (HTTPS), for example `https://yourcompany.engineering.domino.tech`. Not `http://127.0.0.1:8763` and not values copied from an in-run environment.
2. **Credential** — `Authorization: Bearer` with a token you store securely (secret manager, CI variable, not committed to git):
- **Personal Access Token (PAT)** when the automation acts as you. Create under Account settings or `POST /api/pat/v1/tokens` while already authenticated.
- **Service account token** when a pipeline or integration runs without a human user. An admin provisions the service account and token.
```python
import requests
deployment_url = "https://yourcompany.engineering.domino.tech"
pat = "..." # from your secret store; never hardcode in shared repos
response = requests.get(
f"{deployment_url.rstrip('/')}/v4/users/self",
headers={"Authorization": f"Bearer {pat}"},
)
```
See https://docs.domino.ai/cloud/reference/api/domino-api-authentication .
### In-run setup block (for examples in this skill)
Use inside a workspace, job, or app only:
```python
import os
import requests
if os.environ.get("DOMINO_API_PROXY"):
base_url = os.environ["DOMINO_API_PROXY"].rstrip("/")
headers = {}
else:
base_url = (os.environ.get("DOMINO_USER_HOST") or os.environ.get("DOMINO_API_HOST") or "").rstrip("/")
token = requests.get("http://localhost:8899/access-token").text.strip()
headers = {"Authorization": f"Bearer {token}"}
```
### python-domino inside Domino
```python
from domino import Domino
domino = Domino("owner/project-name")
```
Configure the SDK with host + Bearer token per the product auth page. Never pass `api_key=`.
## Common Operations
### Projects
```python
from domino import Domino
domino = Domino()
# Create project
project = domino.project_create(
project_name="my-new-project",
owner_name="username"
)
# Get project info
info = domino.project_info()
print(f"Project: {info['name']}")
print(f"ID: {info['id']}")
```
### Jobs (Runs)
```python
# Start a job
run = domino.runs_start(
command="python train.py --epochs 100",
hardware_tier_name="medium",
environment_id="env-id"
)
print(f"Run ID: {run['runId']}")
# Start job with different commit
run = domino.runs_start(
command="python train.py",
commit_id="abc123"
)
# Check status
status = domino.runs_status(run['runId'])
print(f"Status: {status['status']}")
# Wait for completion
domino.runs_wait(run['runId'])
# Get logs
logs = domino.runs_get_logs(run['runId'])
print(logs)
# Stop a run
domino.runs_stop(run['runId'])
```
### Workspaces
```python
# Start workspace
workspace = domino.workspace_start(
hardware_tier_name="medium",
environment_id="env-id",
workspace_type="JupyterLab"
)
print(f"Workspace ID: {workspace['workspaceId']}")
# Stop workspace
domino.workspace_stop(workspace['workspaceId'])
```
### Files
```python
# Upload file
domino.files_upload(
path="local/file.csv",
dest_path="/mnt/code/data/"
)
# Download file
domino.files_download(
path="/mnt/code/results/output.csv",
dest_path="local/output.csv"
)
# List files
files = domino.files_list("/mnt/code/")
for f in files:
print(f['path'])
```
### Datasets
```python
# Create dataset
dataset = domino.datasets_create(
name="training-data",
description="Training dataset"
)
# List datasets
datasets = domino.datasets_list()
# Create snapshot
snapshot = domino.datasets_snapshot(
dataset_name="training-data",
tag="v1.0"
)
```
### Environments
```python
# List environments
environments = domino.environments_list()
for env in environments:
print(f"{env['name']}: {env['id']}")
# Get environment details
env = domino.environment_get("env-id")
```
### Model APIs
```python
# Publish model
model = domino.model_publish(
file="model.py",
function="predict",
environment_id="env-id",
name="my-classifier",
description="Classification model"
)
print(f"Model ID: {model['id']}")
# List models
models = domino.models_list()
# Get model info
model_info = domino.model_get("model-id")
```
## REST API
Use the [Authentication](#authentication) setup (`base_url`, `headers`) for examples below.
```python
response = requests.get(f"{base_url}/api/projects/beta/projects", headers=headers or None)
projects = response.json()
```
### Common Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v4/projects` | GET | List projects |
| `/v4/projects/{id}/runs` | POST | Start a run |
| `/v4/projects/{id}/runs/{runId}` | GET | Get run status |
| `/v4/projects/{id}/files` | GET | List files |
| `/v4/gateway/runs/{runId}/logs` | GET | Get run logs |
| `/v4/models` | GET | List models |
| `/v4/models/{id}/latest/model` | POST | Call model |
## Domino Data API
Separate SDK for data access:
```python
from domino_data.data_sources import DataSourceClient
# Initialize client
client = DataSourceClient()
# List data sources
sources = client.list_data_sources()
# Query data source
df = client.get_datasource("my-datasource").query(
"SELECT * FROM customers WHERE region = 'US'"
)
```
## Automation Examples
### CI/CD Integration
```python
# trigger_training.py - Call from CI/CD pipeline
from domino import Domino
import sys
domino = Domino("team/ml-project")
# Start training job
run = domino.runs_start(
command="python train.py",
hardware_tier_name="gpu-large"
)
# Wait for completion
result = domino.runs_wait(run['runId'])
if result['status'] != 'Succeeded':
print(f"Training failed: {result['status']}")
sys.exit(1)
print("Training completed successfully!")
```
### Batch Job Scheduler
```python
# Run multiple experiments
from domino import Domino
import itertools
domino = Domino("team/experiments")
# Parameter grid
params = {
"learning_rate": [0.01, 0.001, 0.0001],
"batch_size": [32, 64, 128]
}
# Generate combinations
combinations = list(itertools.product(*params.values()))
param_names = list(params.keys())
# Submit all experiments
runs = []
for combo in combinations:
param_str = " ".join(
f"--{name}={value}"
for name, value in zip(param_names, combo)
)
run = domino.runs_start(
command=f"python experiment.py {param_str}",
hardware_tier_name="gpu-small"
)
runs.append(run['runId'])
print(f"Started run {run['runId']} with {param_str}")
# Wait for all to complete
for run_id in runs:
result = domino.runs_wait(run_id)
print(f"Run {run_id}: {result['status']}")
```
### Model Deployment Pipeline
```python
from domino import Domino
domino = Domino("team/model-deployment")
# 1. Train model
train_run = domino.runs_start(command="python train.py")
domino.runs_wait(train_run['runId'])
# 2. Evaluate model
eval_run = domino.runs_start(command="python evaluate.py")
domino.runs_wait(eval_run['runId'])
# 3. Deploy if evaluation passes
# (Check evaluation results first)
model = domino.model_publish(
file="serve.py",
function="predict",
name="production-model"
)
print(f"Model deployed: {model['id']}")
```
## Error Handling
```python
from domino import Domino
from domino.exceptions import DominoException
try:
domino = Domino("team/project")
run = domino.runs_start(command="python train.py")
except DominoException as e:
print(f"Domino error: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
```
## Best Practices
### 1. Follow [Authentication](#authentication)
Proxy without a header when `DOMINO_API_PROXY` is set; otherwise access-token + platform host; PAT/SA only outside a run.
### 2. Handle Rate Limits
```python
import time
from domino.exceptions import DominoException
def api_call_with_retry(func, max_retries=3):
for attempt in range(max_retries):
try:
return func()
except DominoException as e:
if "rate limit" in str(e).lower():
time.sleep(2 ** attempt)
else:
raise
raise Exception("Max retries exceeded")
```
### 3. Log API Calls
```python
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def start_run(command):
logger.info(f"Starting run: {command}")
run = domino.runs_start(command=command)
logger.info(f"Run ID: {run['runId']}")
return run
```
## Detailed API Reference
For comprehensive REST API documentation, see these specialized guides:
| Guide | Description |
|-------|-------------|
| [API-PROJECTS.md](API-PROJECTS.md) | Projects, collaborators, Git repos, goals |
| [API-JOBS.md](API-JOBS.md) | Jobs, scheduled jobs, logs, tags |
| [API-DATASETS.md](API-DATASETS.md) | Datasets, snapshots, tags, grants |
| [API-MODELS.md](API-MODELS.md) | Model APIs, deployments, registry |
| [API-MODEL-SERVING.md](API-MODEL-SERVING.md) | Lifecycle, v1/v2 registry split, invoke vs management |
| [API-ENVIRONMENTS.md](API-ENVIRONMENTS.md) | Environments, revisions, Dockerfile |
| [API-APPS.md](API-APPS.md) | Apps endpoint catalog (see [apps/API-APPS.md](../apps/API-APPS.md) for v1 automation) |
| [API-ADMIN.md](API-ADMIN.md) | Users, orgs, hardware tiers, data sources |
| [API-REFERENCE.md](API-REFERENCE.md) | Complete endpoint reference |
## Documentation Reference
OpenAPI and route discovery: [API-SPECS.md](../domino-api-intro/API-SPECS.md).
**Product docs:**
- [Domino API authentication](https://docs.domino.ai/cloud/reference/api/domino-api-authentication)
- [python-domino Library](https://docs.domino.ai/cloud/reference/python-sdk/python-wrapper-for-domino-api)
- [GitHub Repository](https://github.com/dominodatalab/python-domino)
Referenced files: 9 — file references are metadata, not package contents.
---
name: domino-trace-setup
description: Set up GenAI tracing for an LLM or agent application in Domino by checking MLflow 3.2.0 and Domino SDK requirements, then adding a tracing_setup.py helper, evaluators, a config.yaml, and @add_tracing / DominoRun wiring. Use when the user asks to add tracing, observability, or evaluation to an agent or LLM app running on Domino.
---
# Set Up Domino GenAI Tracing
Instrument an agent or LLM application so its calls, spans, and evaluator scores show up in
Domino experiments. For traditional ML tracking, use the `domino-experiment-setup` skill instead.
## Inputs
1. **LLM framework**: `openai` (default), `anthropic`, or `langchain`. Detect from the code if possible.
2. **Agent name**: name for the traced entry-point function.
3. **Evaluator**: whether to add quality scoring, and whether to use an LLM-as-judge.
4. **Aggregated metrics**: which evaluator metrics to summarize per run.
## Steps
1. **Check requirements.** Confirm `mlflow==3.2.0` and the Domino SDK with agents support are
installed. If not, tell the user to add them to the environment's Dockerfile:
```dockerfile
RUN pip install mlflow==3.2.0
RUN pip install "dominodatalab[data,agents]==2.2.0"
```
2. **Copy helpers** from this skill's `assets/` folder into the project:
- `assets/tracing_setup.py`: `setup_tracing(framework)`, `create_evaluator()`,
`create_llm_judge_evaluator()`, and `get_aggregation_metrics()`.
- `assets/config.yaml`: agent configuration logged with `DominoRun(agent_config_path=...)`.
Replace the example model names with the ones the user actually calls.
3. **Instrument the agent.** Decorate the entry point with `@add_tracing` and run it inside
`DominoRun`, following the example below. Keep the user's existing logic unchanged.
4. **Explain how to view traces** (see "Viewing traces").
## Example traced agent
```python
import mlflow
from domino.agents.tracing import add_tracing
from domino.agents.logging import DominoRun
from openai import OpenAI
mlflow.openai.autolog()
client = OpenAI()
def quality_evaluator(inputs, output):
return {
"response_length": len(output.get("response", "")),
"confidence": output.get("confidence", 0),
}
@add_tracing(name="my_agent", evaluator=quality_evaluator)
def my_agent(query: str) -> dict:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": query}],
)
return {
"response": response.choices[0].message.content,
"confidence": 0.9,
"model": "gpt-4o-mini",
}
if __name__ == "__main__":
aggregated_metrics = [("response_length", "mean"), ("confidence", "mean")]
with DominoRun(
run_name="example-run",
agent_config_path="config.yaml",
custom_summary_metrics=aggregated_metrics,
) as run:
for query in ["What is machine learning?", "Explain neural networks"]:
result = my_agent(query)
print(f"Q: {query}\nA: {result['response'][:100]}...\n")
print(f"Run ID: {run.run_id}")
```
## Framework autologging
| Framework | Setup |
|---|---|
| OpenAI | `mlflow.openai.autolog()` then `OpenAI()` |
| Anthropic | `mlflow.anthropic.autolog()` then `Anthropic()` |
| LangChain | `mlflow.langchain.autolog()` then e.g. `ChatOpenAI(model="gpt-4o-mini")` |
## Viewing traces
1. Go to **Experiments** in Domino.
2. Open the experiment the run logged to (by default `tracing-{username}`).
3. Select the run and open the **Traces** tab.
The trace view shows the span tree (agents, tools, messages), token usage, latency, and evaluator
scores.
For evaluators, multi-agent patterns, and `DominoRun` details, use the `domino-genai-tracing` skill.
Referenced files: 2 — file references are metadata, not package contents.
---
name: domino-ui-bootstrap
description: Bootstrap or retrofit a Vite + React 18 + TypeScript project so it uses the Domino design system (`@dominodatalab/extensions-tools`). Scoped to a **fully standalone SPA frontend for a Domino application or extension**, not snippets, library additions, or work inside an existing Domino monorepo package. Use whenever the user wants to create, scaffold, refactor, retrofit, or set up such an SPA (a React app, web app, frontend, UI, or extension) that should "look like Domino", use Domino components, follow the Domino design system, or integrate with the Domino platform, including when they mention DominoThemeProviderDecorator, extensions-tools, or base-components without saying "Domino". Also use to make an existing standalone React/Vite project "Domino-styled" or add Domino theming. Handles React 18 / react-router 5 pinning, Storybook MCP registration, theme provider wiring, and build verification.
---
# Domino UI Bootstrap
This skill makes a project use the Domino design system correctly. It works in three contexts:
- **Empty or non-existent directory** — scaffold a Vite + React 18 + TS project from scratch, then apply the Domino setup.
- **Existing Vite + React project** — leave the project alone, retrofit just the Domino bits (deps, theme provider, MCP, agent instructions file).
- **Existing React project on a different bundler/setup** — flag the mismatch to the user, then proceed only with their direction.
In all three cases, the skill executes the work end-to-end (runs commands, edits files), but it adapts to what's already there instead of overwriting blindly.
## Why this exists
A handful of choices look arbitrary but aren't — they match the rest of the Domino ecosystem and are easy to get wrong:
- **React 18, not 19.** `@dominodatalab/extensions-tools` peer-depends on React 18. React 19 fails at install.
- **react-router 5, not 6.** The library uses v5 APIs (`HashRouter` from `react-router-dom@5`).
- **HashRouter, not BrowserRouter.** This is what the library expects internally. It also gives cleaner isolation between the app's frontend and backend — client-side routes live entirely after the `#`, so they never collide with backend paths or the Domino proxy prefix, and inner navigation/linking works without server-side rewrite rules.
- **The npm package name is `@dominodatalab/extensions-tools`.** Storybook code snippets show imports from `@domino/base-components` — that's a Storybook-internal alias. Rewrite every such import to `@dominodatalab/extensions-tools` before pasting into a Domino project. This is the single most common mistake.
- **`DominoThemeProviderDecorator` wraps the whole React tree.** Without it, Domino components render unstyled or crash.
- **URLs must survive the proxy path.** Domino serves the app under a prefix (e.g. `/preview/<appId>/`), so asset and API URLs must be document-relative — not root-absolute. Vite's default `base: '/'` emits `/assets/…` and bypasses the proxy; user-written `fetch('/api/…')` does the same. Set `base: './'` and build API URLs against `document.baseURI`. See Step 8. Invisible in local dev (`pathname` is `/`), surfaces only after deployment.
Treat these as invariants the project must satisfy by the end. How you get there depends on what's already in the target directory.
## Host-specific files
Steps 2, 6, 10, 11, and 13 write files that depend on which coding assistant is running this skill. Use the column for the current host. If you can't tell which host you are, or it isn't listed, ask the user which files their assistant reads.
| Purpose | Claude Code | Codex |
|---|---|---|
| Agent instructions file | `CLAUDE.md` | `AGENTS.md` |
| MCP server registration | `.mcp.json` | `.codex/config.toml` |
| Per-user MCP enablement | `.claude/settings.local.json` | None needed; Codex loads `.codex/config.toml` once the user trusts the project |
| Reload after MCP changes | Restart Claude Code | Restart Codex |
The rest of this skill calls these the **instructions file**, the **MCP config**, and the **MCP enablement file**.
---
## Step 1 — Elicit project info
Ask the user, in a single turn, for:
1. **Target path** — the absolute or workspace-relative path where the Vite + React app will live ("the app folder"). Ask this as a plain chat question, not a multiple-choice picker, because paths are free-form.
2. **Project root path** — where shared project config (the instructions file, MCP config, and MCP enablement file) should live. Default to the target path; set this to a parent directory when the target is a subfolder of a larger repo (monorepos, an existing project with apps under `apps/`, etc.) — those files belong at the repo root, not nested inside the app folder. Ask explicitly; don't assume. If the user doesn't volunteer one, propose the target path and confirm.
3. **Display name** — free-form, any string. Used in the instructions file, UI titles, and the hand-off message. Example: `Domino Frontend`.
4. **Package name** — the value that goes in `package.json`'s `name` field. npm requires lowercase, no spaces, no leading dot or underscore, URL-safe. Default to the kebab-case-lowercase of the display name (e.g. `Domino Frontend` → `domino-frontend`) and confirm. If retrofitting, default to whatever the existing `package.json` says and confirm.
5. **What to do if the path has unexpected content** — present this as a short list of choices once you've inspected the directory (Step 2). The options depend on what you find; see Step 2.
If the user has already given any of these in the conversation, skip the corresponding question. If they give only one name, treat it as the display name and derive the package name from it — don't ask twice for the same information, just confirm the normalized package name. Throughout the rest of this skill, "project root" means the path from question 2 (equal to the target path unless the user said otherwise) and "app folder" means the target path from question 1.
---
## Step 2 — Survey the target directory
Before changing anything, find out what's there. This determines which branch of the skill you take.
Look for:
- Does the path exist? Is it empty?
- Is there a `package.json`? If so:
- Is `react` already a dependency? At what major version?
- Is `vite` a devDependency? Some other bundler (`webpack`, `next`, `parcel`, `cra`)?
- Does `@dominodatalab/extensions-tools` already appear (any version)?
- Is there a `src/` directory with `main.tsx` / `main.jsx` / `index.tsx` / `index.jsx`?
- At the **project root** (from Step 1 — may equal the app folder, may not): is there an existing instructions file, MCP config, or MCP enablement file (see "Host-specific files")? These live at the project root, not the app folder, so check there even when the app folder is a fresh empty subdirectory.
Classify the directory into one of these states, then ask the user how to proceed if needed:
| State | What you found | What to do |
|---|---|---|
| **Empty / nonexistent** | No files, or directory doesn't exist | Go to Step 3 (scaffold from scratch). |
| **Vite + React already** | `package.json` lists `vite` and `react` | Skip scaffolding. Go to Step 4 (align deps), then continue. |
| **React but not Vite** | React present, bundler is webpack/CRA/Next/etc. | Stop and ask the user: do they want to migrate to Vite, or keep their bundler and just add the Domino library? The skill is Vite-shaped; if they keep their bundler, the version pinning and theme-provider wiring still apply, but the scaffolding steps don't. |
| **Unrelated content** | Files exist but no `package.json`, or a non-React `package.json` | Ask the user: overwrite (delete and scaffold fresh), bootstrap in place (let Vite prompt), or pick a different path. |
| **Already partially Domino-wired** | `@dominodatalab/extensions-tools` already in deps | Treat as a verification/repair job: check each invariant below and only touch what's actually wrong. |
Run `node -v` and `npm -v`. Stop if Node is older than 20.
---
## Step 3 — Scaffold (only if empty / nonexistent / overwrite chosen)
From the target directory:
```bash
npm create vite@latest . -- --template react-ts
```
Do not run `npm install` immediately afterward. The next step rewrites `package.json` so the install picks up the right versions in one pass.
If the directory had existing content and the user chose "overwrite", clear it first (`rm -rf` the contents, not the directory itself).
---
## Step 4 — Align `package.json` to the Domino invariants
You're enforcing constraints, not writing a fixed file. View the current `package.json` and make sure it satisfies these:
**Required `dependencies`** (add or pin to these exact versions):
- `@dominodatalab/extensions-tools` — `latest` is fine for fresh projects; use a specific version if the user gave one or if one is already pinned in the existing `package.json`. (See the post-install pin step at the end of this section.)
- `react` — `18.2.0`
- `react-dom` — `18.2.0`
- `react-router` — `5.3.4`
- `react-router-dom` — `5.3.4`
**Required `devDependencies`** (the React typings must match React 18):
- `@types/react` — `18.2.0`
- `@types/react-dom` — `18.2.0`
- `@types/react-router` — `5.1.20`
- `@types/react-router-dom` — `^5.3.3`
**Node-version branch — check this BEFORE leaving the scaffold's toolchain alone.**
Run `node -v`. The current `npm create vite@latest` scaffold writes Vite 9 / TypeScript 6 / ESLint 10 / `@types/node` 24, all of which require Node ≥ 20.19. The Domino default workspace image ships Node 20.18.3, so the scaffold's defaults will fail `npm run build` (rolldown native binding error) and TS will error on `erasableSyntaxOnly` (a 5.6+ flag) and on missing `composite: true` for `tsc -b`.
- **If Node ≥ 20.19:** the scaffold's defaults are fine. Skip to "Leave alone" below.
- **If Node < 20.19:** pin the toolchain to a Node-20.18-compatible set before installing:
- `vite` — `^5.4.0`
- `@vitejs/plugin-react` — `^4.3.4`
- `typescript` — `~5.5.4` (or `~5.6` if you keep `erasableSyntaxOnly` — but the simpler path is to drop the flag)
- `@types/node` — `^20.12.0`
- Remove the `eslint`, `@eslint/js`, `eslint-plugin-react-hooks`, `eslint-plugin-react-refresh`, `typescript-eslint`, and `globals` entries from `devDependencies`. The Vite-9 scaffold's ESLint pins all require Node ≥ 20.19 — easier to drop them than to find a compatible set.
- Strip the `erasableSyntaxOnly` line from `tsconfig.app.json` and `tsconfig.node.json` (TS 5.6+ only).
- Add `"composite": true` to both `tsconfig.app.json` and `tsconfig.node.json` (required when `build` runs `tsc -b`).
- Delete `eslint.config.js` since ESLint was removed.
**Leave alone:**
- Vite, ESLint, TypeScript, `@vitejs/plugin-react`, and any other tooling already in the file — assuming the Node-version branch above didn't tell you to touch them. Whatever versions are there are fine if they work on the user's Node.
- Any scripts, fields, or sections you don't have a specific reason to change. In particular, don't rewrite `scripts` if the existing ones work; only add `dev`/`build`/`preview` if they're missing.
- Any project-specific dependencies the user already has (state libraries, icon packs, data-fetching libs, etc.). The skill is additive.
**Remove or downgrade:**
- If `react` or `react-dom` is at 19+, downgrade to `18.2.0`. Same for the `@types/*`.
- If `react-router-dom` is at 6+, downgrade to `5.3.4`. Warn the user — their existing routes use v6 syntax (`<Routes>` / `element={}`) and will need to be rewritten to v5 (`<Switch>` / `component={}` or `render={}`). Don't silently rewrite their routes; tell them what needs to change.
**After install — pin the resolved `@dominodatalab/extensions-tools` version.**
Once Step 5 has run cleanly and `latest` was used, resolve the floating tag to a concrete version and rewrite `package.json`:
```bash
npm view @dominodatalab/extensions-tools version
```
Replace `"latest"` in `dependencies` with the exact version string this returns. Skip if a specific version was already pinned (the user explicitly chose one, or you retrofitted). This keeps future installs reproducible — `latest` drifts.
---
## Step 5 — Install
```bash
npm install
```
Likely failures:
- **Peer-dep error about React 19** → Step 4 didn't take. Re-check `package.json`. Don't use `--legacy-peer-deps`.
- **404 / E401 / EAUTH on `@dominodatalab/extensions-tools`** → the user needs to authenticate to a private npm registry. Stop and ask how they normally authenticate. Don't switch to a tarball or alternative source on your own.
---
## Step 6 — Register the Storybook MCP
The Storybook MCP is what lets future assistant sessions look up real component props instead of guessing. Register a server named `storybook` at the **project root** (from Step 1 — not the app folder if those are different paths) that points at the Domino library's live Storybook:
- URL: `https://main--60c0de3f60dd96003bdcb1a1.chromatic.com/mcp`
- Transport: HTTP (streamable HTTP)
These rules apply on every host:
- If the MCP config already exists at the project root (with other MCP servers, or from a prior bootstrap), merge — don't overwrite. Add the `storybook` entry alongside whatever's there.
- If a `storybook` server is already registered at a different URL, ask the user before changing it.
- If no MCP config exists at the project root, create one there — never inside the app folder.
### Claude Code
**`.mcp.json`** registers the server with transport `http` and the URL above.
**`.claude/settings.local.json`** must:
- Pre-allow tools under the `mcp__storybook` namespace.
- Enable project-level MCP servers (`enableAllProjectMcpServers: true`).
- List `storybook` in the enabled servers.
Both files must sit in the same directory: if they get split across folders, Claude Code won't apply the allow-list to the registered MCP and the server stays disabled. If `.claude/settings.local.json` already exists, merge the relevant fields and preserve any other permissions or settings.
### Codex
**`.codex/config.toml`** registers the server:
```toml
[mcp_servers.storybook]
url = "https://main--60c0de3f60dd96003bdcb1a1.chromatic.com/mcp"
```
Codex loads project `.codex/config.toml` only for trusted projects. Tell the user to trust the project if Codex hasn't asked yet. If the file already exists, add the table without touching other settings.
---
## Step 7 — Wire up the React entry point
The project's React entry (`src/main.tsx` for a fresh scaffold, but it might be `src/index.tsx` or similar in an existing project) needs to satisfy these invariants:
- `DominoThemeProviderDecorator` from `@dominodatalab/extensions-tools` wraps the entire app tree. It must be an ancestor of every Domino component that gets rendered.
- A `HashRouter` from `react-router-dom` (v5) wraps the app inside the theme provider. Not `BrowserRouter`, not v6.
- The app component is rendered via `createRoot` from `react-dom/client` (React 18 idiom).
- `StrictMode` is fine to keep if Vite added it; not required.
For a fresh scaffold, write a clean `main.tsx` from scratch. For an existing project:
- If the entry already wraps the app in some other provider (Redux store, React Query client, custom theme), keep those wrappers and insert `DominoThemeProviderDecorator` so it's an ancestor of any Domino components. The usual place is outermost or just inside `StrictMode`, but the user's existing structure may dictate otherwise — don't blindly reorder providers that have ordering requirements (e.g., Redux usually goes outermost; auth providers often need to be high up).
- If the project already has a router, check which one. If it's `BrowserRouter` from v5, swap to `HashRouter`. If it's any router from v6, you'll need to convert routes too (see Step 4's note) — flag this to the user before doing it.
- Don't remove existing CSS imports. The theme provider injects its own styles, so a separate Domino CSS import isn't needed, but the user's app CSS should stay.
A heads-up worth passing to the user: in a standalone (non-Domino-backend) environment, `DominoThemeProviderDecorator` will try to fetch user / white-label data and fail silently. The UI still renders with defaults. If they want to suppress those requests, point them at `node_modules/@dominodatalab/extensions-tools/README.md` for the static `useStoreHook` prop.
---
## Step 8 — Make URLs proxy-aware (assets AND API calls)
When this app runs inside Domino it's served under a proxy prefix (e.g. `https://<host>/preview/<appId>/`). Root-absolute URLs bypass the proxy and 404 — both asset tags emitted as `/assets/index-*.js` and user-written `fetch('/api/…')` calls. This is invisible in local dev (where `pathname` is `/` and root-absolute happens to work) and only surfaces after deployment.
Two things to get right:
### 8a. `base: './'` in `vite.config.ts` (the scaffold does NOT set this)
The `npm create vite` scaffold leaves `base` at its default `/`, which emits root-absolute asset URLs (`/assets/…`) that bypass the proxy. Set it explicitly so emitted assets are document-relative:
```ts
export default defineConfig({
base: './',
plugins: [react()],
})
```
### 8b. `apiBase` resolved against `document.baseURI`
Build API URLs off `document.baseURI` so they resolve against the proxied path:
```ts
const apiBase = new URL('api', document.baseURI).href
fetch(`${apiBase}/projects`)
```
Don't construct API URLs from `window.location.pathname` (`pathname.replace(/[^/]*$/, '') + 'api'` etc.) — those either bypass the proxy or break under deeper routes. Use `document.baseURI`.
**When to apply this:**
- **Fresh scaffolds / starter screens you're generating:** write 8a and 8b in from the start. If the starter screen calls a backend, route it through `apiBase`.
- **Retrofits:** add 8a if `base` is missing/`/`. For API calls, scan for `fetch('/`, `axios.get('/`, `new WebSocket('ws`, and any `pathname`-based URL construction. Surface root-absolute or pathname-based URLs and recommend the `document.baseURI` form — **don't silently rewrite their fetches** (some may intentionally target another host).
This overlaps with the `domino-apps` skill, which covers the full deploy shape (launch script, port binding, build output location). Point the user there for an actual app publish.
---
## Step 9 — Starter screen (only for fresh scaffolds)
Skip this step for retrofits — don't overwrite the user's existing `App.tsx` and don't touch their existing CSS.
For a fresh scaffold, replace Vite's default `App.tsx` with something that exercises a few real Domino components, to prove the install works. The minimum it should demonstrate:
- An import from `@dominodatalab/extensions-tools` (so the install path is verified).
- At least one component that depends on the theme provider (e.g., `Button`, `Card`, `Typography`) — this proves Step 7 is wired correctly.
- Optionally, `IconResolver` to demonstrate the icon system.
### Replace `src/index.css` (fresh scaffolds only)
Vite's default `src/index.css` ships ~80 lines of template chrome: a fixed-width centered `#root`, custom color tokens, `h1`/`h2`/`p`/`code` overrides, dark-mode rules, and decorative styles for the Vite hero. All of it overrides or conflicts with `DominoThemeProviderDecorator`'s tokens. On a fresh scaffold this isn't "the user's app CSS" — it's template noise.
- Overwrite `src/index.css` with a minimal reset:
```css
body { margin: 0; }
#root { min-height: 100svh; }
```
- Delete `src/App.css` if `App.tsx` no longer imports it.
This overrides the "don't remove existing CSS imports" line in Step 7 for the fresh-scaffold case. Step 7's guidance is about preserving real user CSS in a real codebase; it doesn't apply to Vite-template chrome on an empty scaffold. On retrofits, the Step 7 rule stands — leave existing CSS alone.
### Components safe to use without an MCP query
These are the components and prop shapes confirmed to work against the published `@dominodatalab/extensions-tools`. Use them for the starter screen without needing to consult the MCP. For anything beyond this set, query the Storybook MCP first (see the instructions-file workflow in Step 10) — don't guess.
| Component | Safe usage |
|---|---|
| `Button` | `type='primary' \| 'secondary' \| 'tertiary'`, `onClick`, children. |
| `Card` | `title`, `extra`, `helpMessage`, `noPadding`, children. **No `size` prop.** |
| `Row` / `Col` / `Space` | Ant-style. `Space` takes `direction`, `size`. |
| `Tag` | `type` (**not `color`**). Values: `user-generated`, `success`, `danger`, `warning`. |
| `Typography` | **Namespace, not a wrapper.** Render `Typography.H1` / `.H2` / `.H3` / `.Text`. Never `<Typography>…</Typography>` — it'll throw React error #130 at runtime. |
| `Typography.Text` | Optional `type='BodyDefault' \| 'BodyDefaultStrong' \| 'BodySmall' \| 'BodySmallStrong' \| 'BodyCode'`. |
| `SpinnerWrapper` | Loading wrapper. |
### Watch out: `node_modules` README uses the Storybook alias
`node_modules/@dominodatalab/extensions-tools/README.md` (and any code snippets it embeds) imports from `@domino/base-components` — that's the Storybook-internal alias, not the published package name. Even though `node_modules` reads as authoritative, **every import in this project must use `@dominodatalab/extensions-tools`**. The instructions file written in Step 10 reminds future sessions of this, but the starter screen is the first place it can go wrong.
---
## Step 10 — Write or update the instructions file
The **project root** (from Step 1 — not the app folder if those are different paths) should have an instructions file (`CLAUDE.md` or `AGENTS.md`; see "Host-specific files") that tells future assistant sessions four things:
1. The npm package is `@dominodatalab/extensions-tools`. All imports in this project use that name.
2. Storybook code snippets (and the `node_modules/@dominodatalab/extensions-tools/README.md`) import from `@domino/base-components` — that's a Storybook alias. Rewrite to `@dominodatalab/extensions-tools` before pasting.
3. Component APIs come from the Storybook MCP. Don't invent props.
4. **URLs must be proxy-aware (assets and API).** When deployed, this app is served under a proxy prefix (e.g. `/preview/<appId>/`). Two things keep URLs working (see Step 8): `base: './'` in `vite.config.ts`, and an `apiBase` resolved against `document.baseURI`. Root-absolute URLs (`fetch('/api/…')`) bypass the proxy and 404.
```ts
const apiBase = new URL('api', document.baseURI).href
fetch(`${apiBase}/projects`)
```
It should also describe the MCP lookup workflow (`list-all-documentation` → `get-documentation` → `get-documentation-for-story`), note that React 18 / react-router 5 versions are pinned for peer-dep reasons, and remind that all backend URLs route through `apiBase` (not root-absolute paths).
If the instructions file already exists at the project root, **merge** rather than overwrite — preserve whatever project-specific guidance is there, and add a Domino section. The user's existing file may have important info about their codebase that you'd erase by replacing it. If the project already has the other host's instructions file (for example `CLAUDE.md` when you are writing `AGENTS.md`), mention it so the user can keep the two in sync. If no instructions file exists at the project root, create one there — never inside the app folder, even when the app folder is a subdirectory of the project root.
---
## Step 11 — Update `.gitignore`
The **app folder** (the target path from Step 1) needs a `.gitignore` that excludes the things this skill (and a normal Vite + Domino workflow) generates but that shouldn't be committed. Make sure the following entries are present:
- `node_modules/` — npm install output.
- `dist/` and `build/` — Vite production builds.
- `.vite/` — Vite's dev cache.
- `*.log`, `npm-debug.log*`, `yarn-debug.log*`, `yarn-error.log*` — package manager logs.
- `.DS_Store`, `Thumbs.db` — OS junk.
- `.env`, `.env.local`, `.env.*.local` — local environment files. Keep `.env.example` if the project has one.
- **Claude Code only:** `.claude/settings.local.json` — this is the per-user MCP/permissions file. It can leak machine-specific paths and personal preferences. `.mcp.json` **should** be committed (it's shared project config); `.claude/settings.local.json` should not. Codex's `.codex/config.toml` from Step 6 is shared project config and needs no entry. Only add this entry when the project root equals the app folder — when they differ, `.claude/settings.local.json` lives at the project root (Step 6), so it belongs in *that* directory's `.gitignore`. Don't create or modify a project-root `.gitignore` for this; surface the missing entry to the user and let them decide.
**How to handle the file itself:**
- If `.gitignore` doesn't exist, create it with the entries above.
- If it already exists (Vite's scaffold writes one), read it first and **append only the entries that aren't already covered**. Don't duplicate lines and don't reorder what's there. A grep-and-append per missing entry is fine.
- Preserve any project-specific patterns the user already has (their own ignored directories, secrets paths, build artifacts from other tools, etc.).
- If the user has committed something this skill is now telling git to ignore (e.g., they checked in `node_modules` once by accident, or the MCP enablement file is already tracked), don't run `git rm` on their behalf — surface it and let them decide.
---
## Step 12 — Verify
The most reliable verification is a clean production build:
```bash
npm run build
```
**`npm install` is not a sufficient check.** It can succeed while the toolchain is broken in ways that only surface at build time — most commonly the rolldown native-binding error when Vite 9 runs on Node < 20.19, which `install` happily writes to disk and only fails when `build` tries to load. Always run `npm run build` before reporting success.
This catches version mismatches deterministically. If it passes, the pinning and the entry-point wiring are correct.
`npm run dev` is a softer check — it'll start even with some misconfigurations. Run it if the user wants to eyeball the result, but `npm run build` is the one that gates "done".
**`npm run build` does NOT catch the proxy-URL bug from Step 8** — that's a runtime/deploy failure, not a compile one. After a successful build, also confirm the URL wiring by inspecting the built `dist/index.html`:
- Asset tags are **relative** (`src="./assets/…"`, `href="./assets/…"`) — not `/assets/…`. If they're root-absolute, `base: './'` is missing (Step 8a).
- API calls use `new URL('api', document.baseURI)` (Step 8b). `grep -rn "pathname.replace" src/` should return nothing — any hit is a pathname-based URL builder to replace.
Expected, not a failure: Vite will emit a warning that some chunks are larger than 500 KB. The Domino library bundles a lot — this is normal for Domino apps and doesn't need chasing.
If `build` fails:
- **Rolldown native binding error** (`Cannot find module @rolldown/binding-*` or similar) → the Vite toolchain in `package.json` requires Node ≥ 20.19 but the workspace is on an older Node. Go back to Step 4's Node-version branch and apply the downgrade set.
- JSX runtime or React typing errors → React or `@types/react` versions don't match. Recheck Step 4.
- TypeScript errors about `erasableSyntaxOnly` or missing `composite` → leftover from a TS-6 scaffold on a downgraded TS. Recheck the Node-version branch in Step 4 (strip `erasableSyntaxOnly`, add `composite: true`).
- Missing modules from `@dominodatalab/extensions-tools` → install didn't complete. Check `node_modules/@dominodatalab/extensions-tools/dist`.
- TypeScript errors in the user's existing code (only relevant on retrofits) → don't try to fix them as part of this skill. Surface them to the user and let them decide.
If the build is green but **assets or API calls 404 after deploy**:
- **Cause:** root-absolute URLs (asset tags pointing to `/assets/…`, or `fetch('/api/…')`) bypass the proxy.
- **Fix:** apply both parts of Step 8 — `base: './'` (8a) and `apiBase` via `document.baseURI` (8b).
- Verify the running app is on the latest commit and was re-published — a stale publish serves the old build regardless of code changes.
---
## Step 13 — Hand off
Tell the user:
- Where the project lives.
- How to start the dev server.
- That future component additions should go through the Storybook MCP (which the instructions file you wrote will remind the next session about).
- Anything that changed in their existing code (downgraded versions, swapped router, edited entry point) so they're not surprised.
- **Bundle-size warning is expected.** Vite emits a `>500 KB chunk` warning on build because the Domino component library bundles a lot. It's not a failure — don't chase it.
- **Port 8888 is reserved inside Domino workspaces.** If the user runs both a Vite dev server and a backend dev server inside a Domino workspace, port 8888 is occupied by code-server. Pick a different port for one of them.
- **Next step toward a deployable app:** suggest that the user create an `app.sh` launch script that runs `npm run build` to produce the frontend bundle. Remind them that `npm run build` alone is not enough to serve the app — they'll need a backend or a static file server to actually serve the built `dist/` output. Point them at the `domino-apps` skill for the full deploy shape.
---
## Guiding principles when in doubt
- **Describe the goal, change the minimum.** The invariants in this skill are about Domino integration. Everything else in the user's project is their business — don't touch what isn't broken.
- **Read before you write.** On any retrofit, view a file before modifying it. The user's structure matters; preserve it where the Domino constraints don't override it.
- **Surface, don't silently fix.** If the user's existing code conflicts with a Domino invariant (router v6 routes, React 19 hooks), tell them what needs to change. Don't rewrite their app under the hood.
- **Test cases over conviction.** When unsure whether a component or prop exists, query the Storybook MCP. Don't ship invented props.
---
## What this skill is not for
- Editing `@dominodatalab/extensions-tools` source. Only the published npm package is consumed; upstream changes need a release from the library repo.
- Setting up test runners, CI, Tailwind, or other tooling on top. If the user wants those, finish the Domino bootstrap first (Step 12 green), then handle them separately — too many things can fail at once otherwise.
- Authenticating to private npm registries. If install fails on auth, ask the user; don't guess at credentials or workarounds.
- Building anything beyond a minimal app *before* the Storybook MCP is running. If the user asks for more than a minimal application while the MCP is not yet available, only scaffold the boilerplate plus a minimal app, then instruct the user to restart their assistant after finishing the app configuration so the MCP loads and the rest can be built against real component APIs.
---
name: domino-ui-design
description: Build Domino-styled web applications matching the Domino Design System. Use when creating or styling Domino apps (Dash, Streamlit, Flask, FastAPI+HTML), generating UI components, reviewing UX, or when the user mentions "Domino app", "Domino UI", "Domino dashboard", or "Domino design system". Covers the full stack — FastAPI backend, Ant Design 5.x frontend via CDN, Domino theme tokens (colors, typography, spacing), chart styling with Highcharts, Domino API authentication, proxy-safe routing, UX writing, error handling, empty states, form design, table patterns, and layout best practices. Also use for UX reviews of existing Domino app screenshots.
---
# Domino UI Design System Skill
## Architecture
Domino apps use **FastAPI** (Python backend) + **Ant Design 5.x** (frontend via CDN, no build step). Avoid inline styles and JS — use statically referenced CSS/JS files. Output errors to stdout.
### CDN Setup
Include in HTML `<head>`:
```html
<script src="https://unpkg.com/react@18/umd/react.production.min.js"></script>
<script src="https://unpkg.com/react-dom@18/umd/react-dom.production.min.js"></script>
<script src="https://unpkg.com/antd@5.11.2/dist/antd.min.js"></script>
<link href="https://unpkg.com/antd@5.11.2/dist/reset.css" rel="stylesheet">
<script src="https://unpkg.com/dayjs@1.11.10/dayjs.min.js"></script>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
<script src="https://code.highcharts.com/highcharts.js"></script>
```
Use `React.createElement()` (aliased as `h`) — no JSX, no build step:
```javascript
const { ConfigProvider, Button, Table, Modal, Select, Input, Form, Space, Card, Tag, Alert } = antd;
const { createElement: h, useState, useEffect } = React;
```
## Domino Theme
Wrap the app in `ConfigProvider` with this theme:
```javascript
const dominoTheme = {
token: {
colorPrimary: '#543FDE', // Purple
colorPrimaryHover: '#3B23D1',
colorPrimaryActive: '#311EAE',
colorText: '#2E2E38',
colorTextSecondary: '#65657B',
colorTextTertiary: '#8F8FA3',
colorSuccess: '#28A464',
colorWarning: '#CCB718',
colorError: '#C20A29',
colorInfo: '#0070CC',
colorBgContainer: '#FFFFFF',
colorBgLayout: '#FAFAFA',
colorBorder: '#E0E0E0',
fontFamily: 'Inter, Lato, Helvetica Neue, Helvetica, Arial, sans-serif',
fontSize: 14,
borderRadius: 4,
borderRadiusLG: 8,
},
components: {
Button: { primaryShadow: 'none', defaultShadow: 'none' },
Table: { headerBg: '#FAFAFA', rowHoverBg: '#F5F5F5' },
},
};
```
### Top Navigation Bar
Height: 44px, Background: `#2E2E38`, Text: `#FFFFFF`. Use the Domino logo (height 32px). Primary actions on right. Only add nav items with real destinations.
**Domino Logo:** Reference `assets/domino-logo.svg`, resolved relative to this skill's directory. The SVG is white text on transparent background, designed for the dark nav bar.
### Charts (Highcharts)
```javascript
Highcharts.setOptions({
colors: ['#543FDE', '#0070CC', '#28A464', '#CCB718', '#FF6543', '#E835A7', '#2EDCC4', '#A9734C'],
chart: { style: { fontFamily: 'Inter, Lato, Helvetica Neue, Arial, sans-serif' } },
});
```
## Domino API Integration
### Authentication
Re-acquire the access token on **every** API call (it expires quickly):
```python
import requests, os
def get_auth_headers():
token = requests.get('http://localhost:8899/access-token').text.strip()
return {'Authorization': f'Bearer {token}'}
```
### API Base Paths
Two separate APIs with **different** base paths:
| API | Base Path | Example |
|-----|-----------|---------|
| Platform APIs (public-api.json) | `/api/` | `{DOMINO_API_HOST}/api/jobs/beta/jobs?projectId=...` |
| Legacy Platform APIs | `/v4/` | `{DOMINO_API_HOST}/v4/jobs?projectId=...` |
| Governance APIs | `/api/governance/v1/` | `{CLUSTER_URL}/api/governance/v1/bundles?project_id=...` |
**Do NOT** add `/api/` prefix to v4 platform endpoints.
### Environment Variables (available in Domino)
`DOMINO_API_HOST`, `DOMINO_PROJECT_ID`, `DOMINO_PROJECT_NAME`, `DOMINO_PROJECT_OWNER`
Do not add mock/fake data if the API isn't working. Prompt user to set env vars.
## Running the App
Use relative URLs (not absolute) — Domino's nginx proxy modifies the root URL.
```bash
#!/bin/bash
uvicorn app:app --host 0.0.0.0 --port 8888
```
## UX Design Rules
For detailed UX principles, writing guidelines, component patterns, and review checklists, read [references/ux-design-rules.md](references/ux-design-rules.md).
Key rules to always follow:
- **Empty states must be actionable** — answer: What is this? Why is it empty? What can I do?
- **Error messages must be human-readable** with guidance on what to do next
- **Disabled elements must explain why** they're disabled (tooltip)
- **Truncated table text must have tooltips**
- **Icon-only buttons must have tooltips**
- **Use sentence case** for UI text (not Title Case)
- **No exclamation points** in UI copy
- **Side panels should overlay**, not push/crush main content
- **Checkboxes should use positive framing** — checked = ON/enabled, never checked = OFF/disabled
## UX Review
When reviewing a Domino app UI, read [references/ux-design-rules.md](references/ux-design-rules.md) and check against the review checklists. Classify issues as High/Medium/Low severity.
## Documentation Reference
Before writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.
**Get the cluster base URL:** `$DOMINO_API_HOST` (injected by Domino into every workspace, job, and app).
This skill uses two swagger docs:
**Public API** (covers `/api/` endpoints — v1, v2, beta — no auth required):
```bash
curl "$DOMINO_API_HOST/assets/public-api.json"
# Browser UI: $DOMINO_API_HOST/assets/lib/swagger-ui/index.html?url=/assets/public-api.json#/
```
**Governance API** (covers `/api/governance/v1/` endpoints, requires bearer token):
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
# Governance is not routed through $DOMINO_API_HOST. Derive the external cluster URL
# from the JWT iss claim — works in any workspace type.
CLUSTER_URL=$(echo $TOKEN | cut -d'.' -f2 | python3 -c "
import sys, base64, json, re
p = sys.stdin.read().strip()
p += '=' * (-len(p) % 4)
print(re.sub(r'/auth/realms/.*', '', json.loads(base64.b64decode(p))['iss']))
")
curl -H "Authorization: Bearer $TOKEN" "$CLUSTER_URL/api/governance/swagger/doc.json"
# Browser UI (must be logged in): $CLUSTER_URL/api/governance/swagger/index.html
```
**Public docs (workflow context and field explanations):**
- [API Guide](https://docs.dominodatalab.com/en/latest/api_guide/f35c19/api-guide/)
Referenced files: 2 — file references are metadata, not package contents.
---
name: domino-workspaces
description: Work with Domino Workspaces - interactive development environments including Jupyter, JupyterLab, VS Code, and RStudio. Covers launching workspaces, configuring hardware tiers, environment selection, volume mounting, SSH access, and package installation. Use when setting up development environments, configuring workspace settings, or troubleshooting IDE issues.
---
# Domino Workspaces Skill
## Description
This skill helps users work with Domino Workspaces - interactive development environments for data science work including Jupyter notebooks, JupyterLab, VS Code, and RStudio.
## Activation
Activate this skill when users want to:
- Launch or configure a workspace in Domino
- Work with Jupyter notebooks, VS Code, or RStudio
- Configure workspace settings (hardware tier, environment, volumes)
- Understand workspace persistence and file management
- Use SSH to connect to remote workspaces
- Install packages in workspaces
## Workspace Types
### Available IDEs
Domino provides these default workspace types:
- **Jupyter Notebook**: Classic notebook interface
- **JupyterLab**: Modern Jupyter interface with file browser
- **VS Code**: Full-featured code editor with extensions
- **RStudio**: IDE for R development
- **Custom IDEs**: Can configure additional workspace types
## Launching a Workspace
### Via Domino UI
1. Navigate to your project
2. Click **Workspaces** in the navigation
3. Click **Launch Workspace**
4. Select:
- **IDE**: Choose Jupyter, VS Code, RStudio, etc.
- **Hardware Tier**: CPU/memory/GPU resources
- **Compute Environment**: Docker environment with tools
- **Volume Size**: Persistent storage (if needed)
5. Click **Launch**
### Via Python SDK
```python
from domino import Domino
domino = Domino("project-owner/project-name")
# Start a workspace
workspace = domino.workspace_start(
environment_id="env-123",
hardware_tier_name="small",
workspace_type="JupyterLab"
)
print(f"Workspace ID: {workspace['workspaceId']}")
```
## File Persistence
### Persistent Directories
Work saved to these directories persists across workspace sessions:
- `/mnt/` - Project files (synced with Domino)
- `/mnt/data/` - Domino Datasets
- `/mnt/artifacts/` - Project artifacts
- `/mnt/imported/` - Imported data
### Home Directory Persistence
Domino 6.1+ supports home directory persistence:
- Installed packages persist across sessions
- User configurations (.bashrc, .vimrc) are retained
- Reduces workspace startup time
## Package Installation
### Temporary (Current Session)
```bash
# Python packages
pip install pandas numpy scikit-learn
# R packages
install.packages("tidyverse")
```
### Persistent (Via Environment)
Add to environment's Dockerfile instructions:
```dockerfile
RUN pip install pandas numpy scikit-learn
```
Or use requirements.txt in your project:
```text
pandas>=2.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
```
## SSH Access to Workspaces
Domino 6.1+ allows SSH connections from local machines:
1. **Enable SSH** in workspace settings
2. **Get connection details** from running workspace
3. **Connect via SSH**:
```bash
ssh -i ~/.ssh/domino_key user@workspace-hostname
```
Benefits:
- Use local IDE (VS Code Remote, PyCharm)
- Access remote compute from local machine
- Browse files locally while computing remotely
## Git Integration in Workspaces
### Using Git
```bash
# Clone a repo in workspace
git clone https://github.com/user/repo.git
# Standard git workflow
git add .
git commit -m "Update model"
git push origin main
```
### Sync with Domino
Click **Sync** in the workspace UI to push changes back to Domino project.
## Workspace Operations
### Stop a Workspace
```python
domino.workspace_stop(workspace_id)
```
### Resume a Workspace
Stopped workspaces can be resumed to continue work.
### View Workspace Logs
Access logs for debugging through the Domino UI or API.
## Best Practices
1. **Save frequently**: Sync work to Domino regularly
2. **Use appropriate hardware**: Match resources to workload
3. **Clean up**: Stop workspaces when not in use to save resources
4. **Version control**: Use Git for code versioning
5. **Environment management**: Use Domino Environments for reproducibility
## Troubleshooting
### Workspace Won't Start
- Check hardware tier availability
- Verify environment builds successfully
- Check resource quotas
### Lost Work
- Check `/mnt/` directories for persisted files
- Review Domino project files
- Check Git history if using version control
### Slow Startup
- Use smaller base environments
- Enable package persistence
- Pre-install packages in environment Dockerfile
## Documentation Reference
- [Use Workspaces](https://docs.dominodatalab.com/en/latest/user_guide/867b72/use-workspaces/)
- [Launch a Workspace](https://docs.dominodatalab.com/en/latest/user_guide/e6e601/workspaces)
- [Start a Jupyter Workspace](https://docs.dominodatalab.com/en/latest/user_guide/93aef2/start-a-jupyter-workspace/)
Referenced files: 2 — file references are metadata, not package contents.
---
name: netapp-volumes
description: Work with Domino Volumes for NetApp ONTAP - enterprise-grade, multi-terabyte storage with near-instant snapshots. Covers volume creation, snapshot versioning with commit messages, cross-project sharing, mount paths (/mnt/netapp-volumes/ or /domino/netapp-volumes/), and the NetApp Volumes REST API. Use when managing large-scale data storage, needing fast no-copy snapshots, or integrating existing NetApp ONTAP infrastructure with Domino.
---
# Domino NetApp Volumes Skill
## Description
This skill helps users work with Domino Volumes for NetApp ONTAP — enterprise-grade storage that mounts NetApp ONTAP filesystems directly into Domino workloads, enabling multi-terabyte scale with near-instant snapshotting.
## Activation
Activate this skill when users want to:
- Create or manage NetApp Volumes in Domino
- Work with NetApp volume snapshots and versioning
- Share large-scale data across projects or teams
- Access existing NetApp ONTAP storage from Domino workspaces, jobs, or apps
- Use the NetApp Volumes REST API programmatically
- Understand mount paths for NetApp volumes
- Choose between NetApp Volumes and Domino Datasets
## What is a Domino NetApp Volume?
A Domino NetApp Volume is:
- **Enterprise-grade storage**: Backed by NetApp ONTAP via Kubernetes PVCs with the `netapp-storage` storage class
- **Multi-terabyte scale**: No practical upper limit — suitable for very large datasets
- **Near-instant snapshots**: ~3 seconds even for 100+ GB volumes using redirect-on-write (no extra storage consumed at snapshot time)
- **Versioned with commit messages**: Snapshots support human-readable tags and commit messages, unlike Datasets
- **Shareable**: Attach a single volume to multiple projects and teams
- **Persistent**: Data persists across executions
### NetApp Volumes vs. Domino Datasets
| Feature | NetApp Volumes | Domino Datasets |
|---------|---------------|-----------------|
| Storage backend | External NetApp ONTAP | Domino-managed NFS/EFS |
| Data scale | Multi-terabyte and beyond | Up to ~1 TB |
| Snapshot speed | ~3 seconds (any size) | Scales with data size |
| Snapshot storage cost | No extra space (redirect-on-write) | Duplicates physical data |
| Commit messages on snapshots | Supported | Not supported |
| Create new volume from snapshot | Read-only clones | Can create editable dataset |
| Cross-project sharing | Yes | Yes |
| Admin prerequisite | Admin must register filesystems | None |
| REST API | Full dedicated API | Via Domino Python SDK |
**Use NetApp Volumes when:**
- Data exceeds ~1 TB
- Near-instant snapshotting is required
- Your organization already has NetApp ONTAP infrastructure
- You need snapshot commit messages for audit trails
- High-performance shared storage across many teams
**Use Domino Datasets when:**
- Data is Domino-managed with no external infrastructure dependency
- You need to create editable new versions from snapshots
- Data is under ~1 TB
---
## Prerequisites
An admin must register at least one NetApp filesystem (Kubernetes PVC with the `netapp-storage` label) before users can create volumes. Contact your Domino administrator if no filesystems are available.
---
## Creating a NetApp Volume
### From the Domino Home Page
1. Navigate to **Data > NetApp Volumes** in the toolbar
2. Click **Add NetApp Volume > Create Volume**
3. Fill in the form:
- **Name**: Letters, numbers, underscores, hyphens only
- **Description**: Brief overview of the data
- **Data Plane**: Select from available options
- **NetApp Filesystem**: Choose a registered filesystem
- **Capacity**: Set maximum storage allocation
4. Click **Next** to configure permissions
5. Assign users with **Reader / Editor / Owner** roles
6. Click **Finish**
7. Manually add the volume to projects afterward
### From within a Project (recommended — auto-associates)
1. Open a project → **Data > NetApp Volumes** (left panel)
2. Click **Add NetApp Volume > Create Volume**
3. Fill in the same form fields as above
4. The volume is automatically associated with the current project on creation
### Via REST API
```python
import os
import requests
# Auth token from in-cluster token service
token = requests.get("http://localhost:8899/access-token").text.strip()
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
api_url = os.environ["DOMINO_API_HOST"]
remotefs_url = os.environ["DOMINO_REMOTE_FILE_SYSTEM_HOSTPORT"]
# Look up your user ID (username is available in the DOMINO_USER_NAME env var)
user_resp = requests.get(
f"{api_url}/v4/users?userName={os.environ['DOMINO_USER_NAME']}",
headers=headers
).json()
user_id = user_resp[0]["id"]
# Create a volume (capacity is in bytes; grants is required)
response = requests.post(
f"{remotefs_url}/remotefs/v1/volumes",
headers=headers,
json={
"name": "large-training-data",
"description": "Multi-TB training dataset for vision models",
"filesystemId": "<filesystem-id>",
"capacity": 5_000_000_000_000, # 5 TB in bytes
"grants": [
{"targetId": user_id, "targetRole": "VolumeOwner"}
]
}
)
volume = response.json()
print(f"Created volume ID: {volume['id']}")
```
---
## Adding a Volume to a Project
### Via Domino UI
1. Go to project → **Data > NetApp Volumes**
2. Click **Add NetApp Volume > Add Existing Volume**
3. Select the volume from the list
4. Configure access level for the project
### Via REST API
```python
# Attach a volume to a project
requests.post(
f"{remotefs_url}/remotefs/v1/rpc/attach-volume-to-project",
headers=headers,
json={
"volumeId": "<volume-id>",
"projectId": "<project-id>"
}
)
```
---
## Mount Paths
Mount paths depend on your **project type**. Check which exists in your execution to determine your project type.
### Git-Based Projects
| Volume Type | Mount Path |
|-------------|-----------|
| Live volume | `/mnt/netapp-volumes/<volume-name>/` |
| Snapshot by number | `/mnt/netapp-volumes/snapshots/<volume-name>/<snapshot-number>/` |
| Snapshot by tag | `/mnt/netapp-volumes/snapshot-tags/<volume-name>/<tag-name>/` |
### DFS (Domino File System) Projects
| Volume Type | Mount Path |
|-------------|-----------|
| Live volume | `/domino/netapp-volumes/<volume-name>/` |
| Snapshot by number | `/domino/netapp-volumes/snapshots/<volume-name>/<snapshot-number>/` |
| Snapshot by tag | `/domino/netapp-volumes/snapshot-tags/<volume-name>/<tag-name>/` |
### Snapshot Access Behavior
There are two ways to access snapshots, with an important behavioral difference:
- **`/snapshots/<volume-name>/<number>/`** — Accessed by snapshot number. When a new snapshot is taken while a workspace is running, the new numbered directory appears **immediately** in that workspace without a restart.
- **`/snapshot-tags/<volume-name>/<tag-name>/`** — Accessed by tag name. Each snapshot has at most one active tag path — a symlink to its numbered snapshot directory. If you apply multiple tags to the same snapshot, only the most recently applied tag creates a path; earlier tags for that snapshot are not accessible by path. Tag paths for new snapshots are also **not** visible in a running workspace — you must **restart the workspace** for them to appear.
Use the numbered path when you need to access a fresh snapshot from within a live workspace. Use the tagged path for stable, named references in reproducible runs.
> **Important:** Each snapshot exposes only one tag path at a time — the most recently applied tag. Older tags on the same snapshot do not have accessible paths.
> **Important:** Renaming a volume changes its mount path. Update any hardcoded paths in your code after renaming.
### Identify Your Project Type
```python
import os
if os.path.exists("/domino/netapp-volumes"):
print("DFS Project")
netapp_root = "/domino/netapp-volumes"
elif os.path.exists("/mnt/netapp-volumes"):
print("Git-Based Project")
netapp_root = "/mnt/netapp-volumes"
```
### Permissions
- **Owners/Editors**: Read-write access to the live volume
- **Readers**: Read-only access
### Example: Reading Data
```python
import pandas as pd
# Git-Based Project
df = pd.read_parquet("/mnt/netapp-volumes/large-training-data/features.parquet")
# DFS Project
df = pd.read_parquet("/domino/netapp-volumes/large-training-data/features.parquet")
# Read from a specific snapshot tag
df = pd.read_parquet("/mnt/netapp-volumes/snapshot-tags/large-training-data/v2.0/features.parquet")
```
### Example: Writing Data
```python
# Write directly to the live volume
df.to_parquet("/mnt/netapp-volumes/large-training-data/processed/output.parquet", index=False)
# List files
import os
files = os.listdir("/mnt/netapp-volumes/large-training-data/")
```
---
## Snapshots and Versioning
### What is a Snapshot?
A snapshot is a read-only, immutable record of the volume's data at a specific point in time. NetApp snapshots use redirect-on-write — no additional storage is consumed at creation time.
**Advantages over Domino Dataset snapshots:**
- Near-instant for any volume size (~3 seconds)
- Support commit messages for documentation and audit trails
- No storage overhead at creation time
### Create a Snapshot via UI
1. Go to project → **Data > NetApp Volumes**
2. Select the volume → click **Take Snapshot**
3. Add an optional **commit message** (e.g., "Added Q4 customer records")
4. Add an optional **tag name** (e.g., `v2.0`, `production`, `2024-Q4`)
5. Click **Confirm**
### Create a Snapshot via REST API
```python
# Create snapshot with description and one or more tags (tagNames is an array)
response = requests.post(
f"{remotefs_url}/remotefs/v1/snapshots",
headers=headers,
json={
"volumeId": "<volume-id>",
"description": "Added Q4 customer records — 50M new rows",
"tagNames": ["v2.0"]
}
)
snapshot = response.json()
print(f"Snapshot ID: {snapshot['id']}")
```
### Add a Tag to an Existing Snapshot
```python
requests.post(
f"{remotefs_url}/remotefs/v1/snapshots/{snapshot_id}/tags",
headers=headers,
json={"name": "production"}
)
```
### Create Snapshot After a Job Run
```python
# Snapshot tied to a specific Domino run for reproducibility
requests.post(
f"{remotefs_url}/remotefs/v1/rpc/create-snapshot-from-run",
headers=headers,
json={
"volumeId": "<volume-id>",
"runId": "<domino-run-id>",
"userId": "<user-id>",
"description": "post-training-run-42"
}
)
```
### Restore a Snapshot
```python
requests.post(
f"{remotefs_url}/remotefs/v1/rpc/restore-snapshot",
headers=headers,
json={"snapshotId": "<snapshot-id>"}
)
```
---
## Roles and Permissions
| Role | API Value | Capabilities |
|------|-----------|-------------|
| **Reader** | `VolumeReader` | View files and snapshots; mount as read-only |
| **Editor** | `VolumeEditor` | All Reader capabilities + modify description, create/delete snapshots, manage shared access, manage users |
| **Owner** | `VolumeOwner` | All Editor capabilities + update volume grants, request deletion |
### Update Permissions via UI
1. Go to **Data > NetApp Volumes** → select volume
2. Click the three-dot menu → **Edit permissions**
3. Add/remove users and assign roles
### Update Permissions via REST API
```python
# targetRole values: "VolumeOwner", "VolumeEditor", "VolumeReader"
requests.put(
f"{remotefs_url}/remotefs/v1/volumes/{volume_id}/grants",
headers=headers,
json=[
{"targetId": "<user-id>", "targetRole": "VolumeEditor"},
{"targetId": "<other-user-id>", "targetRole": "VolumeReader"}
]
)
```
---
## Using NetApp Volumes in Jobs
### Via Domino REST API
```python
requests.post(
f"{api_url}/api/jobs/v1/jobs",
headers=headers,
json={
"projectId": "<project-id>",
"runCommand": "python train.py",
"hardwareTierId": "<hardware-tier-id>",
"environmentId": "<environment-id>", # required — use DOMINO_ENVIRONMENT_ID env var
"netAppVolumeIds": ["<volume-id>"],
"snapshotNetAppVolumesOnCompletion": True # auto-snapshot mounted volumes when job finishes
}
)
```
Set `snapshotNetAppVolumesOnCompletion: true` to automatically take a snapshot of all mounted NetApp volumes when the job completes. This is the recommended approach for training jobs — it captures the exact state of the volume at the end of the run without requiring a separate API call.
---
## Listing Volumes and Snapshots
```python
# List all volumes accessible to you
volumes = requests.get(
f"{remotefs_url}/remotefs/v1/volumes",
headers=headers
).json()
for v in volumes["data"]:
capacity_tb = v["capacity"] / 1_000_000_000_000
print(f"{v['name']} — {capacity_tb:.1f} TB — ID: {v['id']}")
# List snapshots for a volume
snapshots = requests.get(
f"{remotefs_url}/remotefs/v1/snapshots",
headers=headers,
params={"volumeId": "<volume-id>"}
).json()
for s in snapshots["data"]:
print(f"Snapshot {s['id']} v{s['version']} — {s.get('description', '')} — tags: {[t['name'] for t in s.get('tags', [])]}")
```
---
## Best Practices
### 1. Use Appropriate Storage
| Data Type | Storage |
|-----------|---------|
| Large data of any file type where the entire filesystem should be versioned as a unit, with intentional snapshots | NetApp Volume |
| Small output files — charts, reports, model binaries | Artifacts / DFS (auto-versioned per file, not suitable for large files) |
| Code | Git / Project files |
### 2. Snapshot Before Changes
```python
# Always snapshot before modifying large volumes
requests.post(
f"{remotefs_url}/remotefs/v1/snapshots",
headers=headers,
json={
"volumeId": "<volume-id>",
"description": "Pre-processing baseline snapshot",
"tagNames": ["pre-processing-2024-01"]
}
)
# Then run your data transformation
process_data()
```
### 3. Use Efficient File Formats
```python
# Parquet for tabular data (faster reads, smaller storage)
df.to_parquet("/mnt/netapp-volumes/dataset/data.parquet")
# Feather for fast pandas I/O
df.to_feather("/mnt/netapp-volumes/dataset/data.feather")
# HDF5 for large numerical arrays
import h5py
with h5py.File("/mnt/netapp-volumes/dataset/arrays.h5", "w") as f:
f.create_dataset("features", data=features_array)
```
### 4. Organize Data
```
/mnt/netapp-volumes/my-volume/
├── raw/
│ ├── 2024-Q1/
│ └── 2024-Q2/
├── processed/
│ ├── features.parquet
│ └── labels.parquet
└── metadata/
└── schema.json
```
### 5. Use Snapshot Tags for Reproducibility
Reference snapshot tags (not snapshot IDs) in your training scripts so that tagged paths remain stable across runs:
```python
# Reproducible reference using a tag
TRAINING_DATA = "/mnt/netapp-volumes/snapshot-tags/dataset/v2.0/"
df = pd.read_parquet(f"{TRAINING_DATA}/features.parquet")
```
### 6. Reading Large Volumes Efficiently
```python
import dask.dataframe as dd
# Lazy read — no data loaded until .compute()
df = dd.read_parquet("/mnt/netapp-volumes/dataset/large_data.parquet")
result = df.groupby("category").mean().compute()
```
---
## Troubleshooting
### Volume Not Found at Mount Path
- Verify the volume is added to the project (**Data > NetApp Volumes**)
- Confirm the volume name matches the path exactly (case-sensitive)
- Check that the workspace/job was started after the volume was added to the project
- Restart the workspace to pick up newly added volumes
### Permission Denied
- Check your role on the volume (need Editor or Owner to write)
- Verify you have been granted access by the volume owner
- For read-only snapshots, use the snapshot tag path instead of the live volume path
### Snapshot Tag Not Mounted
- Each snapshot has at most **one** tag path — the most recently applied tag. Earlier tags on the same snapshot are never surfaced as directories and cannot be accessed by path.
- Tag paths for new snapshots do **not** appear in a running workspace. The numbered snapshot directory (`/snapshots/<name>/<number>/`) appears immediately, but the tag symlink path only becomes visible after restarting the workspace.
- Verify the tag was created successfully via the UI or API
### No NetApp Filesystems Available
- Contact your Domino administrator — they must register NetApp ONTAP filesystems before volumes can be created
- Admins can register filesystems via **Admin > NetApp Volumes**
### Slow Read Performance
- Use columnar formats (Parquet, Feather) instead of CSV for tabular data
- Read only needed columns: `pd.read_parquet(path, columns=["col1", "col2"])`
- Use Dask or chunked reading for files larger than available RAM
---
## Documentation Reference
Before writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.
**Get the cluster base URL:** `$DOMINO_API_HOST` (injected by Domino into every workspace, job, and app).
Fetch the NetApp Volumes swagger spec (requires bearer token):
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
# The swagger UI is only accessible via the external cluster URL (not $DOMINO_API_HOST).
# Derive it from the JWT iss claim — works in any workspace type.
CLUSTER_URL=$(echo $TOKEN | cut -d'.' -f2 | python3 -c "
import sys, base64, json, re
p = sys.stdin.read().strip()
p += '=' * (-len(p) % 4)
print(re.sub(r'/auth/realms/.*', '', json.loads(base64.b64decode(p))['iss']))
")
curl -H "Authorization: Bearer $TOKEN" "$CLUSTER_URL/domino-netapp-volumes/swagger/doc.json"
# Browser UI (must be logged in): $CLUSTER_URL/domino-netapp-volumes/swagger/index.html
```
**Public docs (workflow context and field explanations):**
- [NetApp Volumes REST API Reference](https://docs.dominodatalab.com/en/cloud/api_guide/b3b2a1/domino-netapp-volumes-api/)
- [Work with NetApp Volumes](https://docs.dominodatalab.com/en/cloud/user_guide/06da1b/work-with-netapp-volumes/)
- [Create NetApp Volumes](https://docs.dominodatalab.com/en/cloud/user_guide/e6887f/create-netapp-volumes-from-domino-or-a-project/)
- [Add or Remove NetApp Volumes on Projects](https://docs.dominodatalab.com/en/cloud/user_guide/306570/add-or-remove-netapp-volumes-on-projects/)
- [View and Edit NetApp Volumes](https://docs.dominodatalab.com/en/cloud/user_guide/93fa12/view-and-edit-netapp-volumes/)
- [Version Data with Snapshots](https://docs.dominodatalab.com/en/cloud/user_guide/dbdbff/version-data-with-snapshots/)
- [Access Data in Domino (comparison guide)](https://docs.dominodatalab.com/en/cloud/user_guide/16d9c1/access-data-in-domino/)
---
name: tags-and-properties
description: Manage Domino tags and properties via the Taxonomy API. Covers tags/namespaces (create, list, update, delete; tag entities — project, model, dataset, app, project_template, netapp_volume; query by tag; autocomplete; merge; CSV import/export) AND properties (typed metadata fields — text, number, date, boolean, url, user, organization, select, multi_select — with per-entity values, groups, and versioned entities like model_version/app_version). Use when organizing entities with tags, building hierarchical namespaces, defining typed metadata fields, setting property values on projects/models/apps, finding entities by tag, bulk-tagging during onboarding, or migrating taxonomy across environments.
---
# Domino Tags and Properties Skill
## Description
This skill covers Domino's Taxonomy API, which manages two complementary
metadata systems for organizing entities (projects, models, datasets, apps,
project templates, NetApp volumes):
- **Tags** — hierarchical labels grouped into namespaces. An entity either has
a tag or it doesn't.
- **Properties** — typed metadata *fields* (`text`, `number`, `date`,
`select`, …), each holding a value per entity.
It documents every public endpoint with curl examples that work today against
a Domino cluster where the taxonomy service is enabled. Properties are covered
in depth in [PROPERTIES.md](./PROPERTIES.md) (definitions) and
[PROPERTY-VALUES.md](./PROPERTY-VALUES.md) (per-entity values).
## Activation
Activate this skill when the user wants to:
- Tag a project, model, dataset, app, project template, or NetApp volume
- Find all entities that share a tag, or query by multiple tags
- Build a hierarchical taxonomy (namespaces with nested tags)
- Bulk-tag entities during onboarding
- Migrate a taxonomy tree across Domino environments (CSV export/import)
- Merge duplicate tags
- Tune autocomplete results in a tagging UI
- Define typed metadata fields (properties) — e.g. a `Budget` number, a
`Review Date`, an `Owner` user, or a `Status` single-select
- Set, read, or clear property values on an entity (incl. versioned entities
like `model_version` / `app_version`)
- Group properties or manage property definitions across a deployment
## Configuration
Auth via the local access-token endpoint. Never use `DOMINO_USER_API_KEY`.
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
# Taxonomy is served through the Domino API host gateway.
BASE="$DOMINO_API_HOST/api/taxonomy/v1"
H="Authorization: Bearer $TOKEN"
```
## Key Concepts
| Concept | Description |
|---------|-------------|
| **Namespace** | Top-level group for **tags** (e.g. `Indication`, `Analysis`). Has `label`, optional `description`, and `allowMultipleAssignments` flag. |
| **Tag** | A label inside a namespace. Can be hierarchical via `parentId` (e.g. `Clinical_Data / SDTM`). Has `label`, `namespaceId`, optional `description` and `parentId`, and `status` (`active` / `inactive`). |
| **Property** | A typed metadata *field* — has a `label`, a `type` (`text`, `number`, `date`, `boolean`, `url`, `user`, `organization`, `user_or_org`, `select`, `multi_select`), a set of `allowedEntities`, an optional `groupName`, and (for select types) `allowedValues`. See [PROPERTIES.md](./PROPERTIES.md). |
| **Property value** | The typed value a property holds for one specific entity, set via `PATCH /property-values`. See [PROPERTY-VALUES.md](./PROPERTY-VALUES.md). |
| **Property group** | Free-form `groupName` string that buckets properties in the values view; ungrouped properties fall under `Miscellaneous`. |
| **EntityType** | Enum over entities. Tags apply to `dataset`, `project`, `project_template`, `model`, `app`, `netapp_volume`. Properties additionally support the *versioned* types `model_version` and `app_version`. |
| **`allowMultipleAssignments`** | (Tags) When `true`, an entity can hold multiple tags from the same namespace. When `false`, applying a new tag from the namespace replaces any existing one. |
| **Taxonomy tree** | The full nested view: namespaces → root tags → child tags. Returned by `GET /taxonomy`. |
| **Limits** | `GET /config` returns `maxDepth` (max tag nesting), `maxLabelLength` (max characters per label), `maxSelectAllowedValuesCount` (max options on a select property), and `maxSelectAllowedValueLength` (max length of one option). |
## Taxonomy API Reference
All endpoints are under `$BASE` (`/api/taxonomy/v1`). Authenticate every
request with `Authorization: Bearer $TOKEN`.
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/config` | GET | Get configuration limits |
| `/taxonomy` | GET | Get full taxonomy tree (nested namespaces + tags) |
| `/namespaces` | GET / POST | List / create namespaces |
| `/namespaces/{namespaceId}` | GET / PUT / DELETE | Get / update / delete a namespace |
| `/namespaces/bulk-delete` | POST | Bulk delete namespaces — see [BULK-OPS.md](./BULK-OPS.md) |
| `/tags` | GET / POST | List / create tags |
| `/tags/{tagId}` | GET / PUT / DELETE | Get / update / delete a tag |
| `/tags/{tagId}/entities` | GET | List entities tagged with a tag |
| `/tags/autocomplete` | GET | Autocomplete tag suggestions for a query |
| `/tags/bulk-delete` | POST | Bulk delete tags — see [BULK-OPS.md](./BULK-OPS.md) |
| `/rpc/merge-tags` | POST | Merge tags — see [BULK-OPS.md](./BULK-OPS.md) |
| `/entities` | GET | Get entities by one or more tag IDs |
| `/entity-tags` | GET / DELETE | Get tags for entities / delete all tags for an entity |
| `/rpc/tag-entity` | POST | Tag an entity |
| `/rpc/untag-entity` | POST | Remove specific tags from an entity |
| `/rpc/export-to-file` | POST | Export taxonomy as CSV — see [IMPORT-EXPORT.md](./IMPORT-EXPORT.md) |
| `/rpc/import-from-file` | POST | Import taxonomy from CSV — see [IMPORT-EXPORT.md](./IMPORT-EXPORT.md) |
| `/rpc/validate-file` | POST | Validate a CSV before import — see [IMPORT-EXPORT.md](./IMPORT-EXPORT.md) |
| `/properties` | GET / POST | List / create property definitions — see [PROPERTIES.md](./PROPERTIES.md) |
| `/properties/{propertyId}` | GET / PUT / DELETE | Get / update / soft-delete a property — see [PROPERTIES.md](./PROPERTIES.md) |
| `/property-groups` | GET | List distinct property group names — see [PROPERTIES.md](./PROPERTIES.md) |
| `/property-values/{entityType}/{entityId}` | GET / PATCH / DELETE | Get / set-clear / delete-all property values for an entity — see [PROPERTY-VALUES.md](./PROPERTY-VALUES.md) |
## Common Workflows
### Workflow 1 — Tag a project (data scientist)
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
BASE="$DOMINO_API_HOST/api/taxonomy/v1"
H="Authorization: Bearer $TOKEN"
# 1. Discover the tag you want to apply
curl -s -H "$H" "$BASE/tags/autocomplete?q=clinical" | python3 -m json.tool
# 2. Apply it to your project
curl -X POST -H "$H" -H "Content-Type: application/json" \
-d "{\"entityType\":\"project\",\"entityId\":\"$DOMINO_PROJECT_ID\",\"tagIds\":[\"<tag-id>\"]}" \
"$BASE/rpc/tag-entity"
# 3. Verify
curl -s -H "$H" "$BASE/entity-tags?entityType=project&entityIds=$DOMINO_PROJECT_ID"
```
### Workflow 2 — Build a hierarchical taxonomy (governance admin)
```bash
# Create a namespace
NS=$(curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d '{"label":"Analysis","description":"Type of analysis","allowMultipleAssignments":false}' \
"$BASE/namespaces" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
# Create a parent tag
PARENT=$(curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d "{\"label\":\"Interim\",\"namespaceId\":\"$NS\"}" \
"$BASE/tags" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
# Create a child tag under it
curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d "{\"label\":\"Milestone_01\",\"namespaceId\":\"$NS\",\"parentId\":\"$PARENT\"}" \
"$BASE/tags"
```
Tag depth is capped at `maxDepth` from `GET /config` (5 on most deployments).
### Workflow 3 — Find all entities with a tag (discovery)
```bash
# Single tag
curl -s -H "$H" "$BASE/tags/<tag-id>/entities" | python3 -m json.tool
# Multiple tags (intersection) — tagIds is a REPEATED query parameter, not comma-separated
curl -s -H "$H" "$BASE/entities?tagIds=<tag-id-1>&tagIds=<tag-id-2>&entityType=project"
```
Both endpoints return paginated results with `meta.pagination.{total,limit,offset}`.
Paginate with `?limit=50&offset=50`.
> **Heads-up:** `tagIds` on `/entities` must be repeated per value
> (`?tagIds=A&tagIds=B`). Comma-separating returns
> `400 {"message":"invalid tag ID: A,B"}`. By contrast, `entityIds` on
> `/entity-tags` *is* comma-separated. Yes, the convention is inconsistent.
### Workflow 4 — Autocomplete in a tagging UI (discovery)
```bash
curl -s -H "$H" "$BASE/tags/autocomplete?q=clin"
```
Response:
```json
{"items":[
{"id":"23ca6d78-...","path":"Clinical_Data"},
{"id":"0b55bf67-...","path":"Clinical_Data / SDTM"},
{"id":"57454f72-...","path":"Clinical_Data / ADaM"}
]}
```
`path` shows the full hierarchy with ` / ` between levels — surface it
verbatim in the UI to disambiguate same-named tags in different branches.
## Namespaces
```bash
# List
curl -s -H "$H" "$BASE/namespaces?limit=50&offset=0"
# Create (label required)
curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d '{"label":"Indication","description":"Therapeutic area","allowMultipleAssignments":true}' \
"$BASE/namespaces"
# 201 → returns full Namespace including id, status, timestamps
# Get one
curl -s -H "$H" "$BASE/namespaces/<id>"
# Update (label and status both required)
curl -s -X PUT -H "$H" -H "Content-Type: application/json" \
-d '{"label":"Indication","description":"updated","status":"active","allowMultipleAssignments":true}' \
"$BASE/namespaces/<id>"
# Delete — cascades through the namespace's tags and entity-tag bindings.
# Returns 204 even on a non-empty namespace. There is no "are you sure" prompt.
# Reach for `namespaces/bulk-delete` when removing several at once.
curl -s -X DELETE -H "$H" "$BASE/namespaces/<id>"
# 204 No Content
```
## Tags
```bash
# List with optional filters
curl -s -H "$H" "$BASE/tags?namespaceId=<ns>&limit=50"
# Create (label and namespaceId required, parentId optional for hierarchy)
curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d '{"label":"SDTM","namespaceId":"<ns>","description":"Study Data Tabulation Model","parentId":"<parent-tag>"}' \
"$BASE/tags"
# Get one — includes entityCount broken down by EntityType
curl -s -H "$H" "$BASE/tags/<id>"
# Update (label required; you can change parentId to re-parent within the same namespace)
curl -s -X PUT -H "$H" -H "Content-Type: application/json" \
-d '{"label":"SDTMv2","description":"new desc","status":"active","parentId":"<new-parent>"}' \
"$BASE/tags/<id>"
# Delete (single)
curl -s -X DELETE -H "$H" "$BASE/tags/<id>"
# 204 No Content
```
## Entity Tags
```bash
# Tag an entity (entityType, entityId, tagIds[] all required)
curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d '{"entityType":"project","entityId":"<project-id>","tagIds":["<tag-1>","<tag-2>"]}' \
"$BASE/rpc/tag-entity"
# 201 → {"entityId":"...","entityType":"project","entityName":"...","tags":[...]}
#
# Constraint: if `tagIds` contains more than one tag from the SAME namespace,
# that namespace must have `allowMultipleAssignments=true`. Otherwise:
# 400 → {"message":"namespace does not allow multiple tag assignments per entity"}
# To replace a tag in a single-assign namespace, send a separate request — the
# new tag overwrites the prior one for that namespace.
# Remove specific tags from an entity
curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d '{"entityType":"project","entityId":"<project-id>","tagIds":["<tag-1>"]}' \
"$BASE/rpc/untag-entity"
# 200 → {"entityId":"...","entityType":"project","entityName":"...","removedTagIds":[...]}
# Get tags for one or more entities (entityIds is comma-separated)
curl -s -H "$H" "$BASE/entity-tags?entityType=project&entityIds=<id-1>,<id-2>"
# 200 → {"data":{"<id-1>":[Tag,...], "<id-2>":[Tag,...]}}
# Remove ALL tags from an entity
curl -s -X DELETE -H "$H" "$BASE/entity-tags?entityType=project&entityId=<id>"
# 200 → {"removedCount":N}
```
`entityType` must be one of: `dataset`, `project`, `project_template`,
`model`, `app`, `netapp_volume`. Other values return 400.
## Taxonomy Tree
```bash
curl -s -H "$H" "$BASE/taxonomy" | python3 -m json.tool
```
Returns an array of `TreeNamespace` objects, each with nested `tags` of type
`TreeTag`:
```json
[{
"id":"821e1455-...",
"label":"Indication",
"description":"Therapeutic area",
"status":"active",
"allowMultipleAssignments":false,
"tags":[
{"id":"...", "label":"Oncology", "status":"active", "children":[
{"id":"...", "label":"Breast_Cancer", "status":"active", "children":[]}
]}
]
}]
```
Use this to render the full taxonomy in a UI in one call.
## Config
```bash
curl -s -H "$H" "$BASE/config"
# {"maxDepth":5,"maxLabelLength":128,"maxSelectAllowedValuesCount":100,"maxSelectAllowedValueLength":2048}
```
| Field | Applies to |
|-------|-----------|
| `maxDepth` | Max tag nesting depth |
| `maxLabelLength` | Max characters for a namespace/tag/property label (and property group name) |
| `maxSelectAllowedValuesCount` | Max options on a `select`/`multi_select` property |
| `maxSelectAllowedValueLength` | Max length of a single select option |
Surface these limits in any UI that lets users create tags, namespaces, or
properties so the user gets immediate validation feedback.
## Properties
Properties are typed metadata *fields* — a complement to tags. Where a tag is
present-or-absent, a property holds a typed value per entity (`Budget = 50000`,
`Review Date = 2026-01-31`, `Status = "Approved"`).
Two layers:
1. **Definitions** — the field schema (`label`, `type`, `allowedEntities`,
optional `groupName` and `allowedValues`). Managed under `/properties`;
create/update/delete requires the Librarian or Admin role. Full reference:
[PROPERTIES.md](./PROPERTIES.md).
2. **Values** — the value a property holds for a specific entity. Managed under
`/property-values/{entityType}/{entityId}`; permissions are per-entity, not
role-based. Full reference: [PROPERTY-VALUES.md](./PROPERTY-VALUES.md).
### Workflow 5 — Define a property (governance admin)
```bash
curl -s -X POST -H "$H" -H "Content-Type: application/json" \
-d '{
"label":"Review Status",
"groupName":"Governance",
"type":"select",
"allowedEntities":["project","model"],
"allowedValues":[{"value":"Draft"},{"value":"In Review"},{"value":"Approved"}]
}' \
"$BASE/properties"
# 201 → full Property with id, status, timestamps
```
`type` is immutable after creation. `select`/`multi_select` require
`allowedValues`; other types reject it. `project_template` is not allowed in
`allowedEntities` — use `project` and it applies to templates too. See
[PROPERTIES.md](./PROPERTIES.md).
### Workflow 6 — Set property values on an entity (data scientist)
```bash
# See which properties apply and their current values
curl -s -H "$H" "$BASE/property-values/project/$DOMINO_PROJECT_ID" | python3 -m json.tool
# Set several at once (best-effort batch: per-item failures don't abort)
curl -s -X PATCH -H "$H" -H "Content-Type: application/json" \
-d '{"items":[
{"propertyId":"<budget-id>","value":"50000"},
{"propertyId":"<review-status-id>","value":"Approved"}
]}' \
"$BASE/property-values/project/$DOMINO_PROJECT_ID"
```
`multi_select` properties take a `values[]` array; all other types use `value`.
An empty `value` clears the property. Versioned entities (`model_version`,
`app_version`) require a `?version=` query param. See
[PROPERTY-VALUES.md](./PROPERTY-VALUES.md).
## Bulk Operations and Tag Merging
See [BULK-OPS.md](./BULK-OPS.md) for `tags/bulk-delete`,
`namespaces/bulk-delete`, and `rpc/merge-tags`.
## Import / Export
See [IMPORT-EXPORT.md](./IMPORT-EXPORT.md) for `rpc/export-to-file`,
`rpc/import-from-file`, and `rpc/validate-file` — useful for migrating a
taxonomy across Domino environments.
## Best Practices
- **One namespace per dimension.** Don't pile orthogonal concepts (e.g.
Indication and Analysis Type) into one namespace; separate so users can
filter cleanly.
- **Use `allowMultipleAssignments=false`** for mutually exclusive concepts
(e.g. "Phase" — a project is in exactly one phase). Use `true` when an
entity can plausibly carry several values from the same namespace
(e.g. multiple "Indication" tags).
- **Hierarchy via `parentId`** rather than encoding hierarchy into labels
(e.g. `Clinical_Data/SDTM`). This lets autocomplete and tree views render
the structure for you.
- **Query `GET /config` once** at app startup and cache `maxLabelLength` /
`maxDepth` for client-side validation.
- **Pin tag IDs, not labels.** Tag labels can be renamed via `PUT /tags/{id}`;
IDs are stable. Workflows that automate tagging should reference tag IDs.
- **Discoverability**: use `GET /taxonomy` for full-tree views,
`GET /tags/autocomplete?q=...` for typeaheads, and
`GET /entities?tagIds=A&tagIds=B&entityType=...` (one `tagIds` per value,
not comma-separated) for filtered lists.
## Troubleshooting
### `Public api endpoint not found` (404)
The taxonomy service is not registered on the gateway you are calling. Two
common causes:
1. **Wrong base URL.** Taxonomy is served through the Domino API host gateway.
Ensure `BASE` is set to `$DOMINO_API_HOST/api/taxonomy/v1` and that
`$DOMINO_API_HOST` is populated (it is set automatically in Domino
workspaces, jobs, and apps).
2. **Taxonomy not enabled on this deployment.** Older or stripped-down
deployments may not include the taxonomy microservice. Confirm with
your Domino administrator before working around this.
### `400 Bad Request` on `POST /rpc/tag-entity`
Ensure `entityType` is exactly one of:
`dataset | project | project_template | model | app | netapp_volume`.
Casing matters — `Project` and `PROJECT` will be rejected.
### `400` on `POST /tags`
`namespaceId` must reference an existing namespace, and the `label` must be
non-empty and ≤ `maxLabelLength` (from `/config`). If you intend to nest the
tag, `parentId` must reference a tag in the same namespace.
### Deleting a namespace removes its tags too
`DELETE /namespaces/{id}` is a cascading delete: it removes the namespace,
every tag inside it, and every entity-tag binding pointing at those tags.
There is no "namespace must be empty" check — confirm with the user before
calling it against a shared cluster. `namespaces/bulk-delete` has the same
cascade behavior across multiple IDs.
## Documentation Reference
Before writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.
**Taxonomy API base:** `$DOMINO_API_HOST/api/taxonomy/v1` (served through the
Domino API host gateway; `$DOMINO_API_HOST` is populated automatically in
workspaces, jobs, and apps).
Fetch the taxonomy swagger spec (requires bearer token):
```bash
TOKEN=$(curl -s http://localhost:8899/access-token)
curl -H "Authorization: Bearer $TOKEN" "$DOMINO_API_HOST/api/taxonomy/swagger/doc.json"
# Browser UI — use the external cluster URL (must be logged in):
# https://<your-cluster>/api/taxonomy/swagger/index.html
```
**Public docs (workflow context and field explanations):**
- [Taxonomy API guide](https://docs.dominodatalab.com/en/cloud/api_guide/fc6b7c/taxonomy-api/)
- [PROPERTIES.md](./PROPERTIES.md) — property definitions (typed metadata fields)
- [PROPERTY-VALUES.md](./PROPERTY-VALUES.md) — setting property values on entities
- [BULK-OPS.md](./BULK-OPS.md) — bulk-delete + merge-tags + migration patterns
- [IMPORT-EXPORT.md](./IMPORT-EXPORT.md) — CSV export/import for taxonomy migration
Referenced files: 4 — file references are metadata, not package contents.
Publisher declarations from the archived package. These are separate from our research and the live service's terms.
Package observed Nov 3, 2026.
plugins_6ac012b82680819195f8a26cbe32c246
Download plugin data (JSON)