Valohai Skills

View saved version →

---
name: valohai-design-pipelines
description: Analyze ML project structure and design Valohai pipelines to orchestrate multi-step workflows. Use this skill when a user wants to create a Valohai pipeline, connect multiple ML steps into an automated workflow, identify pipeline opportunities in their codebase, design data flow between preprocessing/training/evaluation/inference steps, or add conditional logic and parallel execution to pipelines. Triggers on mentions of pipelines, workflows, DAGs, orchestration, multi-step ML, or connecting Valohai steps.
license: Apache-2.0
metadata:
  author: valohai
  version: "1.0"
  compatibility: Requires Python ML projects. Designed for Claude Code or similar AI coding agents.
---

# Design Valohai Pipelines

Analyze ML project structure to identify pipeline opportunities and design multi-step Valohai pipelines that orchestrate preprocessing, training, evaluation, and deployment workflows.

## What Pipelines Do

A Valohai pipeline connects individual steps into an automated workflow:
- **Nodes** = Individual steps (preprocessing, training, evaluation, inference, deployment)
- **Edges** = Data flow between steps (outputs become inputs, metadata becomes parameters)

Valohai automatically executes nodes in the right order, passes data between steps, runs independent nodes in parallel, and tracks complete data lineage.

## Step-by-Step: Analyze the Project

### 1. Identify Pipeline Stages

Examine the ML project to find distinct stages. Common patterns:

**Standard ML Pipeline:**
```
Raw Data → Preprocessing → Training → Evaluation → (Deployment)
```

**With Feature Engineering:**
```
Raw Data → Feature Engineering → Train/Val/Test Split → Training → Evaluation
```

**With Hyperparameter Optimization:**
```
Preprocessing → HPO Sweep (Task) → Select Best → Final Training → Evaluation
```

**Multi-Model Pipeline:**
```
                    ┌→ Train Model A ─┐
Preprocessing ──────┤                  ├→ Ensemble → Evaluation
                    └→ Train Model B ─┘
```

**Full Production Pipeline:**
```
Data Ingestion → Preprocessing → Training → Evaluation → Model Validation → Deployment → Smoke Test
```

### 2. Map Data Dependencies

For each stage, identify:
- **What files does it need?** (input data, models, configs)
- **What files does it produce?** (processed data, trained models, metrics, reports)
- **What metrics does it produce?** (accuracy, loss, F1 that downstream steps might need)
- **What parameters should flow between steps?** (best learning rate, selected features)

### 3. Identify Parallelization Opportunities

Steps that don't depend on each other can run in parallel:
- Training multiple model variants
- Processing different data splits
- Running different evaluation metrics
- A/B training with different hyperparameters

## Step-by-Step: Create the Pipeline YAML

### Design Principle: Every Step Should Run Independently

**IMPORTANT**: Whenever possible, give every input a `default` value so each step can be tested and debugged on its own with `vh execution run step-name --adhoc`. Pipeline edges override these defaults at runtime.

Look for default values in the existing code, README, documentation, config files, or data paths already referenced in the project. Only add a default if you can find a real, meaningful value. Do not invent placeholder URLs or dummy paths - leave the input without a default if no real value is available.

### 1. Ensure Steps Are Defined First

Each pipeline node references an existing step. Define all steps before the pipeline:

```yaml
- step:
    name: preprocess-data
    image: python:3.11
    command:
      - pip install -r requirements.txt
      - python preprocess.py {parameters}
    parameters:
      - name: test_split
        type: float
        default: 0.2
    inputs:
      - name: raw-data
        default: s3://bucket/raw-data/

- step:
    name: train-model
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - pip install -r requirements.txt
      - python train.py {parameters}
    parameters:
      - name: epochs
        type: integer
        default: 50
      - name: learning_rate
        type: float
        default: 0.001
    inputs:
      - name: train-data
        default: s3://bucket/processed/train/
      - name: val-data
        default: s3://bucket/processed/val/

- step:
    name: evaluate-model
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - pip install -r requirements.txt
      - python evaluate.py
    inputs:
      - name: model
        default: datum://latest-trained-model
      - name: test-data
        default: s3://bucket/processed/test/
```

### 2. Define the Pipeline

```yaml
- pipeline:
    name: training-pipeline
    nodes:
      - name: preprocess
        type: execution
        step: preprocess-data

      - name: train
        type: execution
        step: train-model
        override:
          inputs:
            - name: train-data
            - name: val-data

      - name: evaluate
        type: execution
        step: evaluate-model
        override:
          inputs:
            - name: model
            - name: test-data

    edges:
      # Preprocessing outputs → training inputs
      - [preprocess.output.*train*, train.input.train-data]
      - [preprocess.output.*val*, evaluate.input.test-data]
      - [preprocess.output.*val*, train.input.val-data]

      # Training output → evaluation input
      - [train.output.model*, evaluate.input.model]
```

### 3. Edge Syntax Reference

Edges define data flow: `[source, target]`

#### Source Formats

| Format | Description | Example |
|--------|-------------|---------|
| `node.output.*` | All outputs from a node | `preprocess.output.*` |
| `node.output.*.csv` | Outputs matching extension | `preprocess.output.*.csv` |
| `node.output.name*` | Outputs starting with name | `train.output.model*` |
| `node.output.*name*` | Outputs containing name | `preprocess.output.*train*` |
| `node.metadata.metric` | A logged metric value | `train.metadata.best_lr` |
| `node.parameter.name` | A parameter value | `train.parameter.learning_rate` |

#### Target Formats

| Format | Description | Example |
|--------|-------------|---------|
| `node.input.name` | An input on the target node | `train.input.train-data` |
| `node.parameter.name` | A parameter on the target node | `finetune.parameter.learning_rate` |
| `deploy.file.endpoint.name` | A file for a deployment endpoint | `deploy.file.predict.model` |

### 4. Override Node Defaults

When a node receives data from an edge, clear its default input so the edge value takes priority:

```yaml
nodes:
  - name: train
    type: execution
    step: train-model
    override:
      inputs:
        - name: train-data    # No default - will come from edge
        - name: val-data       # No default - will come from edge
      parameters:
        - name: epochs
          default: 200         # Override the step's default
```

## Advanced Pipeline Patterns

### Parallel Training

```yaml
- pipeline:
    name: parallel-training
    nodes:
      - name: preprocess
        type: execution
        step: preprocess-data

      - name: train-resnet
        type: execution
        step: train-model
        override:
          parameters:
            - name: model_type
              default: resnet50
          inputs:
            - name: train-data

      - name: train-efficientnet
        type: execution
        step: train-model
        override:
          parameters:
            - name: model_type
              default: efficientnet_b0
          inputs:
            - name: train-data

      - name: select-best
        type: execution
        step: compare-models
        override:
          inputs:
            - name: model-a
            - name: model-b

    edges:
      - [preprocess.output.*, train-resnet.input.train-data]
      - [preprocess.output.*, train-efficientnet.input.train-data]
      - [train-resnet.output.model*, select-best.input.model-a]
      - [train-efficientnet.output.model*, select-best.input.model-b]
```

### Hyperparameter Sweep with Task Node

```yaml
- pipeline:
    name: hpo-pipeline
    nodes:
      - name: preprocess
        type: execution
        step: preprocess-data

      - name: hpo-sweep
        type: task
        step: train-model
        override:
          inputs:
            - name: train-data

      - name: deploy-best
        type: execution
        step: deploy-model

    edges:
      - [preprocess.output.*, hpo-sweep.input.train-data]
      - [hpo-sweep.output.model*, deploy-best.input.model]
```

### Conditional Execution (Actions)

```yaml
- pipeline:
    name: conditional-pipeline
    nodes:
      - name: train
        type: execution
        step: train-model
        actions:
          # Stop pipeline if accuracy is too low
          - when: node-complete
            if: metadata.accuracy < 0.5
            then: stop-pipeline

      - name: evaluate
        type: execution
        step: evaluate-model
        actions:
          # Require human approval before proceeding
          - when: node-starting
            then: require-approval

      - name: deploy
        type: execution
        step: deploy-model

    edges:
      - [train.output.model*, evaluate.input.model]
      - [evaluate.output.*, deploy.input.model]
```

#### Action Reference

**When (triggers):**
- `node-starting` - Before the node begins
- `node-complete` - After the node succeeds
- `node-error` - When the node fails

**If (conditions):**
- `metadata.metric_name >= value` - Based on logged metric
- `metadata.metric_name < value`
- Can reference any metadata key logged via JSON printing

**Then (actions):**
- `stop-pipeline` - Halt the entire pipeline
- `require-approval` - Pause until human approves

### Passing Metrics as Parameters

```yaml
edges:
  # Pass a logged metric from training as a parameter to fine-tuning
  - [train.metadata.best_learning_rate, finetune.parameter.learning_rate]

  # Pass a parameter value between nodes
  - [train.parameter.user_id, evaluate.parameter.user_id]
```

### Pipeline-Level Parameters

```yaml
- pipeline:
    name: configurable-pipeline
    parameters:
      - name: learning_rate
        default: 0.001
        targets:
          - train.parameter.learning_rate
          - finetune.parameter.learning_rate
      - name: dataset_version
        default: v2
        target: preprocess.parameter.version
    nodes:
      # ... nodes ...
    edges:
      # ... edges ...
```

### Deployment Node

```yaml
- pipeline:
    name: train-and-deploy
    nodes:
      - name: train
        type: execution
        step: train-model

      - name: deploy
        type: deployment
        deployment: my-deployment
        endpoints:
          - predict
        actions:
          - when: node-starting
            then: require-approval

    edges:
      - [train.output.model*, deploy.file.predict.model]
```

### Error Handling

```yaml
nodes:
  - name: train
    type: execution
    step: train-model
    on-error: stop-all    # stop-all | stop-next | continue | retry
```

| Strategy | Description |
|----------|-------------|
| `stop-all` | Stop the entire pipeline |
| `stop-next` | Don't start downstream nodes, but let parallel nodes finish |
| `continue` | Ignore the error and continue |
| `retry` | Retry the failed node |

### Node Reuse (Caching)

Skip nodes that already ran successfully with identical configuration:

```yaml
- pipeline:
    name: cached-pipeline
    reuse-executions: true
    nodes:
      # ... nodes ...
    edges:
      # ... edges ...
```

A node is considered unchanged when: same inputs, same command, same commit hash, same parameters, same step configuration.

## How to Identify Pipeline Opportunities

When analyzing a codebase, look for:

1. **Sequential scripts**: `run_preprocess.sh && run_train.sh && run_eval.sh` → pipeline
2. **Shared data between scripts**: Output of one script is input to another → edges
3. **Multiple model training runs**: Same training script with different configs → parallel nodes or task node
4. **Manual model selection**: Comparing results and picking the best → conditional actions + metadata edges
5. **Deployment after training**: Manual model promotion → deployment node with approval gate
6. **Scheduled retraining**: Cron jobs running training → pipeline triggers
7. **Data validation before training**: Data quality checks → conditional node with stop-pipeline action
8. **A/B model evaluation**: Testing multiple approaches → parallel training + comparison node

## Validate the Pipeline YAML

**IMPORTANT**: After defining a pipeline in `valohai.yaml`, always run `vh lint` to validate before running:

```shell
vh lint
```

This catches issues like:
- References to undefined steps in nodes
- Invalid edge syntax or missing node names
- Duplicate node names
- Missing required fields (`name`, `nodes`, `edges`)
- Invalid action conditions

Fix all lint errors before attempting to run the pipeline.

## Running Pipelines

```shell
# Run with local code
vh pipeline run training-pipeline --adhoc

# Run from Git
vh pipeline run training-pipeline

# Override node parameters
vh pipeline run training-pipeline --adhoc --train.epochs=200

# Override pipeline-level parameters
vh pipeline run training-pipeline --adhoc --learning_rate=0.0001
```

## Edge Cases

- Every node must reference an existing step defined in the same `valohai.yaml`
- Edges require both `nodes` and `edges` sections in the pipeline
- When using `override` on a node to clear an input, list the input name without a default
- Output wildcards (`*`) match against the filename, not the full path
- A pipeline must have at least one node and one edge
- Nodes without incoming edges start immediately when the pipeline runs
- Multiple edges can target the same input (files are merged into the input directory)
- `reuse-executions` only works when the pipeline is run from a Git commit (not `--adhoc`)

View saved version →

---
name: valohai-migrate-data
description: Migrate data loading and model saving in ML code to use Valohai's input/output system. Use this skill when a user wants to configure data inputs from cloud storage (S3, Azure Blob, GCS), save model outputs to Valohai, replace hardcoded file paths, remove boto3/cloud SDK code, or set up the Valohai file I/O system. Triggers on mentions of inputs, outputs, data loading, model saving, S3, cloud storage, file paths, or Valohai data migration.
license: Apache-2.0
metadata:
  author: valohai
  version: "1.0"
  compatibility: Requires Python. Designed for Claude Code or similar AI coding agents.
---

# Valohai Data I/O Migration

Migrate data loading and model saving to use Valohai's managed input/output system. This eliminates cloud SDK boilerplate, handles authentication automatically, and enables full data lineage tracking.

## Philosophy

Valohai separates data access from code. Inputs are declared in `valohai.yaml` with cloud storage URLs. Valohai downloads them to `/valohai/inputs/{name}/` before your code runs. Outputs are saved to `/valohai/outputs/` and automatically uploaded. No boto3, no credentials in code, no download logic.

## Step-by-Step: Inputs

### 1. Identify Data Loading Code

Look for patterns like:
- `boto3.client('s3')` / `s3.download_file()`
- `from google.cloud import storage`
- `azure.storage.blob`
- `requests.get()` for downloading datasets
- `wget` or `curl` in shell commands
- Hardcoded local paths like `/data/train/`, `./datasets/`
- `pd.read_csv("path/to/data.csv")`
- `torch.load("model.pth")` for pretrained models

### Design Principle: Always Set Default Values for Inputs

**IMPORTANT**: Whenever possible, give every input a `default` value so the step can run independently without a pipeline. This lets users test individual steps with `vh execution run step-name --adhoc`. Pipeline edges override defaults at runtime.

Look for default values in the existing code, README, documentation, config files, or data paths already referenced in the project. Only add a default if you can find a real, meaningful value. Do not invent placeholder URLs or dummy paths.

### 2. Define Inputs in valohai.yaml

```yaml
- step:
    name: train-model
    image: tensorflow/tensorflow:2.6.0
    command:
      - pip install -r requirements.txt
      - python train.py
    inputs:
      - name: training-data
        default: s3://my-bucket/datasets/train.csv
      - name: pretrained-model
        default: datum://production-model-latest
        filename: model.h5
```

#### Input Options

```yaml
inputs:
  # Single file
  - name: config
    default: s3://bucket/config.json

  # Multiple files with wildcard
  - name: images
    default:
      - s3://bucket/images/*.jpg
      - s3://bucket/images/*.png

  # Multiple cloud sources
  - name: data
    default:
      - s3://aws-bucket/data.csv
      - azure://container/data.csv
      - gs://gcs-bucket/data.csv

  # Rename on download
  - name: model
    default: s3://bucket/models/v42-best.h5
    filename: model.h5

  # Preserve directory structure
  - name: dataset
    default: s3://bucket/dataset/**/*.json
    keep-directories: suffix

  # Optional input (can be empty)
  - name: checkpoint
    optional: true

  # Reference a Valohai datum (output from another execution)
  - name: pretrained
    default: datum://my-model-alias
```

#### keep-directories Options

- `none` (default): All files flat in `/valohai/inputs/{name}/`
- `suffix`: Keeps path after the wildcard root
- `full`: Keeps the full storage path

### 3. Update Python Code

Replace cloud SDK / download code with simple local file reads:

#### Before (with boto3)

```python
import boto3

s3 = boto3.client('s3', aws_access_key_id=KEY, aws_secret_access_key=SECRET)
s3.download_file('my-bucket', 'datasets/train.csv', '/tmp/train.csv')
df = pd.read_csv('/tmp/train.csv')
```

#### After (Valohai)

```python
import pandas as pd

# Files are already downloaded to /valohai/inputs/
df = pd.read_csv("/valohai/inputs/training-data/train.csv")
```

#### Reading Multiple Input Files

```python
import os
import glob

# All files in the input directory
image_dir = "/valohai/inputs/images/"
image_files = glob.glob(os.path.join(image_dir, "*.jpg"))

for image_path in image_files:
    process_image(image_path)
```

#### Loading a Model

```python
import tensorflow as tf

# With filename option, the file is always named model.h5
model = tf.keras.models.load_model("/valohai/inputs/pretrained-model/model.h5")
```

### 4. Override Inputs at Runtime

```shell
# Override via CLI
vh execution run train-model --training-data=s3://different-bucket/new-data.csv --adhoc

# Or use the web UI to browse and select files
```

## Step-by-Step: Outputs

### 1. Identify Output/Save Code

Look for:
- `model.save("model.h5")` / `torch.save(model, "model.pth")`
- `df.to_csv("results.csv")`
- `plt.savefig("plot.png")`
- `joblib.dump(model, "model.pkl")`
- Any code saving files locally

### 2. Update Save Paths

Simply change the output path to `/valohai/outputs/`:

#### Before

```python
model.save("model.h5")
df.to_csv("results.csv")
plt.savefig("loss_curve.png")
```

#### After

```python
model.save("/valohai/outputs/model.h5")
df.to_csv("/valohai/outputs/results.csv")
plt.savefig("/valohai/outputs/loss_curve.png")
```

That's the only change needed. Valohai automatically uploads everything in `/valohai/outputs/` to your configured cloud storage when the execution completes.

### 3. Directory Structure in Outputs

Subdirectories are preserved:

```python
import os

os.makedirs("/valohai/outputs/models", exist_ok=True)
os.makedirs("/valohai/outputs/plots", exist_ok=True)

model.save("/valohai/outputs/models/best_model.h5")
plt.savefig("/valohai/outputs/plots/training_curve.png")
```

### 4. Live Upload During Training (Checkpoints)

Files marked read-only are uploaded immediately, without waiting for execution to end:

```python
import os
from stat import S_IREAD, S_IRGRP, S_IROTH

for epoch in range(epochs):
    # Training...
    if epoch % 10 == 0:
        path = f"/valohai/outputs/checkpoint_epoch_{epoch}.pt"
        torch.save(model.state_dict(), path)
        os.chmod(path, S_IREAD | S_IRGRP | S_IROTH)  # Triggers immediate upload
```

### 5. Package Large Numbers of Output Files

**IMPORTANT**: If your code produces thousands of individual files (e.g., preprocessed images, tokenized text shards, tile crops), **package them into a single archive** before writing to `/valohai/outputs/`. Do NOT write thousands of small files individually.

Valohai makes a separate HTTPS request with a presigned URL for every file it uploads and downloads. With tens of thousands of small files, you'll spend more time on HTTP overhead than actual data transfer.

Use `tar` (without compression) to bundle files — this keeps packaging fast while eliminating the per-file HTTP overhead:

```python
import tarfile
import os

# WRONG - 20,000 individual files = 20,000 HTTPS requests on download
for i, image in enumerate(processed_images):
    save_image(image, f"/valohai/outputs/image_{i}.png")

# CORRECT - 1 tar archive = 1 HTTPS request on download
output_dir = "/tmp/processed_images"
os.makedirs(output_dir, exist_ok=True)
for i, image in enumerate(processed_images):
    save_image(image, f"{output_dir}/image_{i}.png")

# Package without compression (fast, no CPU overhead)
with tarfile.open("/valohai/outputs/processed_images.tar", "w") as tar:
    tar.add(output_dir, arcname="processed_images")
```

**Rule of thumb**: If you're writing more than ~1,000 files to outputs, tar them up. A few large files are always faster than many small ones on Valohai.

### 6. No Need to Declare Outputs in YAML

You do **not** need to add an `outputs` section to `valohai.yaml`. Any file your code writes to `/valohai/outputs/` is automatically captured and uploaded. Just save files there and Valohai handles the rest.

## Validate the YAML

**IMPORTANT**: After adding inputs to `valohai.yaml`, always run `vh lint` to validate:

```shell
vh lint
```

This catches issues like invalid input names, bad URL formats in defaults, and YAML syntax errors. Fix any errors before running.

## Common Migration Patterns

### Full Training Script Migration

```python
import argparse
import json
import os
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import joblib

parser = argparse.ArgumentParser()
parser.add_argument("--n_estimators", type=int, default=100)
parser.add_argument("--max_depth", type=int, default=10)
args = parser.parse_args()

# Load data from Valohai inputs
train_df = pd.read_csv("/valohai/inputs/training-data/train.csv")
test_df = pd.read_csv("/valohai/inputs/test-data/test.csv")

X_train, y_train = train_df.drop("target", axis=1), train_df["target"]
X_test, y_test = test_df.drop("target", axis=1), test_df["target"]

# Train model
model = RandomForestClassifier(n_estimators=args.n_estimators, max_depth=args.max_depth)
model.fit(X_train, y_train)

# Evaluate and log metrics
accuracy = accuracy_score(y_test, model.predict(X_test))
print(json.dumps({"accuracy": round(accuracy, 4)}))

# Save model to Valohai outputs
joblib.dump(model, "/valohai/outputs/model.pkl")
```

### Using datum:// References in Pipelines

Outputs from one execution can be referenced by another using `datum://` aliases:

```yaml
inputs:
  - name: model
    default: datum://production-model-latest
```

This references a specific versioned file by its alias, enabling reproducible pipeline connections.

## Creating Datasets with JSONL Metadata

Valohai uses a special metadata file (`valohai.metadata.jsonl`) to create and version datasets. Write this file to `/valohai/outputs/` alongside your output files.

### Before Creating a Dataset Version: Check Existing Versions

**IMPORTANT**: When creating a dataset version that builds on a previous version (using `from`), **always confirm with the user** which dataset version or alias to base it on. Do not guess or hardcode version numbers.

Use the CLI to fetch available data and aliases in the project:

```shell
# List data files in the project (shows datum URLs)
vh data list

# List aliases (shows named references like "production", "latest")
vh alias list
```

Ask the user:
1. Which dataset and version to base the new version on (e.g., `dataset://my-dataset/v2`)
2. Or which alias to use (e.g., `dataset://my-dataset/production`)
3. Or whether to skip basing it on a previous version entirely and create a fresh dataset

### Basic Dataset Creation

```python
import json

# Save your output files to /valohai/outputs/ as usual
df.to_csv("/valohai/outputs/processed_data.csv")

# Create the metadata file to register it as a dataset version
metadata = {
    "processed_data.csv": {
        "valohai.dataset-versions": ["dataset://my-dataset/v1"],
    },
}

with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f:
    for file_name, file_metadata in metadata.items():
        json.dump({"file": file_name, "metadata": file_metadata}, f)
        f.write("\n")
```

### Dataset with Multiple Files

```python
import json

metadata = {}

# Save multiple files and register them all in the same dataset version
for split in ["train", "val", "test"]:
    filename = f"{split}.csv"
    df.to_csv(f"/valohai/outputs/{filename}")
    metadata[filename] = {
        "valohai.dataset-versions": ["dataset://my-dataset/v1"],
    }

with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f:
    for file_name, file_metadata in metadata.items():
        json.dump({"file": file_name, "metadata": file_metadata}, f)
        f.write("\n")
```

### Incremental Dataset Updates

Create a new version based on a previous one, adding or excluding files:

```python
import json

metadata = {
    "new_data.csv": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://my-dataset/v3",
                "from": "dataset://my-dataset/v2",
                "start_fresh": False,
                "exclude": ["bad_file.csv", "old_file.csv"],
            },
        ],
    },
}

with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f:
    for file_name, file_metadata in metadata.items():
        json.dump({"file": file_name, "metadata": file_metadata}, f)
        f.write("\n")
```

### Dataset with Aliases

Aliases let you reference datasets by name (e.g., `production`, `latest`) instead of version number:

```python
import json

metadata = {
    "model.pkl": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://my-models/v5",
                "from": "dataset://my-models/v4",
                "targeting_aliases": ["production", "stable"],
            },
        ],
    },
}

with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f:
    for file_name, file_metadata in metadata.items():
        json.dump({"file": file_name, "metadata": file_metadata}, f)
        f.write("\n")
```

### Packaged Datasets (Large File Collections)

For datasets with 10,000+ small files, use packaging for 10-100x faster downloads:

```python
import json

metadata = {
    "images.tar": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://my-images/train-v1",
                "packaging": True,
            },
        ],
    },
}

with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f:
    for file_name, file_metadata in metadata.items():
        json.dump({"file": file_name, "metadata": file_metadata}, f)
        f.write("\n")
```

### Consuming Datasets as Inputs

Reference datasets in `valohai.yaml` inputs using the `dataset://` URI:

```yaml
inputs:
  - name: training-data
    default: dataset://my-dataset/latest
  - name: model
    default: dataset://my-models/production
```

## Execution Environment Reference

Every Valohai execution has:
- `/valohai/repository/` - Your code (working directory)
- `/valohai/inputs/{input-name}/` - Downloaded input files (read-only)
- `/valohai/outputs/` - Write output files here
- `/valohai/config/parameters.json` - Auto-generated parameter config

## Edge Cases

- Input directories are read-only - never write to `/valohai/inputs/`
- Input names in YAML map to directory names under `/valohai/inputs/`
- If multiple files have the same name across different default URLs, they overwrite each other
- When running locally (not on Valohai), `/valohai/inputs/` won't exist - consider adding a fallback path for local development
- No file size limits on outputs
- Any file type is supported for both inputs and outputs
- Wildcard inputs: all matching files are downloaded to the same directory

View saved version →

---
name: valohai-migrate-metrics
description: Add metrics and metadata tracking to ML code for Valohai. Use this skill when a user wants to track training metrics (loss, accuracy, F1, etc.), log experiment metadata, enable real-time metric visualization, or compare experiments in Valohai. Triggers on mentions of metrics, metadata, experiment tracking, logging accuracy/loss, or Valohai metric migration.
license: Apache-2.0
metadata:
  author: valohai
  version: "1.0"
  compatibility: Requires Python. Designed for Claude Code or similar AI coding agents.
---

# Valohai Metrics/Metadata Migration

Add metrics tracking to ML code so Valohai automatically captures, visualizes, and enables comparison across experiments. No special libraries required - just print JSON to stdout.

## Philosophy

Valohai captures metrics by detecting JSON printed to stdout during execution. This is deliberately simple and framework-agnostic. No SDK imports, no decorators, no special API calls. Just `print(json.dumps({...}))`.

## Step-by-Step Instructions

### 1. Identify Metrics to Track

Scan the user's ML code for values worth tracking:

- **Training metrics**: loss, accuracy, precision, recall, F1 score, AUC-ROC
- **Training dynamics**: learning rate (if scheduled), gradient norm, batch processing time
- **Validation metrics**: val_loss, val_accuracy, val_f1 (per epoch or interval)
- **Resource metrics**: GPU utilization, memory usage, throughput (samples/sec)
- **Final results**: best model score, total training time, convergence epoch
- **Custom KPIs**: any domain-specific metric the user cares about

### 2. Add JSON Printing to Code

The core pattern is simple - print a JSON dictionary to stdout:

```python
import json

# Log metrics at any point in your code
print(json.dumps({"accuracy": 0.92, "loss": 0.08}))
```

**CRITICAL: Group all metrics from the same moment into a single `json.dumps()` call.** Each `print(json.dumps(...))` creates one metadata event with one timestamp. If you print metrics separately, Valohai treats them as disconnected events and they cannot be correlated.

```python
# WRONG - 4 disconnected events, can't be correlated or plotted together
print(json.dumps({"inference_time_s": 0.45}))
print(json.dumps({"num_detections": 6}))
print(json.dumps({"confidence_threshold": 0.25}))
print(json.dumps({"iou_threshold": 0.7}))

# CORRECT - 1 event, all metrics linked together
print(json.dumps({
    "inference_time_s": 0.45,
    "num_detections": 6,
    "confidence_threshold": 0.25,
    "iou_threshold": 0.7,
}))
```

Same rule applies to training loops - one epoch = one `json.dumps()`:

```python
# WRONG
print(json.dumps({"epoch": epoch}))
print(json.dumps({"train_loss": train_loss}))
print(json.dumps({"val_accuracy": val_acc}))

# CORRECT
print(json.dumps({
    "epoch": epoch,
    "train_loss": train_loss,
    "val_accuracy": val_acc,
}))
```

Valohai automatically:
- Captures every JSON line printed to stdout
- Adds a UTC timestamp
- Makes values searchable, sortable, and plottable
- Enables real-time visualization during execution

### 3. Common Integration Patterns

#### Training Loop (Most Common)

```python
import json

for epoch in range(epochs):
    train_loss = train_one_epoch(model, train_loader, optimizer)
    val_loss, val_acc = validate(model, val_loader)

    print(json.dumps({
        "epoch": epoch,
        "train_loss": train_loss,
        "val_loss": val_loss,
        "val_accuracy": val_acc,
    }))
```

#### Batch-Level Logging

```python
import json

for epoch in range(epochs):
    for batch_idx, (data, target) in enumerate(train_loader):
        loss = train_step(model, data, target, optimizer)

        if batch_idx % 100 == 0:  # Log every N batches to stay under 50 events/sec
            print(json.dumps({
                "epoch": epoch,
                "batch": batch_idx,
                "loss": loss.item(),
            }))
```

#### Multiple Phases with Context

```python
import json

# Training phase
for epoch in range(epochs):
    train_metrics = train_epoch(model, train_loader)
    print(json.dumps({
        "epoch": epoch,
        "phase": "training",
        "loss": train_metrics["loss"],
        "accuracy": train_metrics["accuracy"],
    }))

    # Validation phase
    val_metrics = validate(model, val_loader)
    print(json.dumps({
        "epoch": epoch,
        "phase": "validation",
        "loss": val_metrics["loss"],
        "accuracy": val_metrics["accuracy"],
    }))
```

#### Final Summary Metrics

```python
import json

# After training completes
print(json.dumps({
    "best_val_accuracy": best_accuracy,
    "best_epoch": best_epoch,
    "total_training_time_seconds": elapsed,
    "final_train_loss": final_loss,
}))
```

### 4. Framework-Specific Examples

#### PyTorch

```python
import json
import time

for epoch in range(args.epochs):
    model.train()
    running_loss = 0.0
    correct = 0
    total = 0

    for batch_idx, (inputs, targets) in enumerate(train_loader):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()

    train_loss = running_loss / len(train_loader)
    train_acc = correct / total

    # Validation
    model.eval()
    val_loss, val_acc = evaluate(model, val_loader, criterion, device)

    print(json.dumps({
        "epoch": epoch,
        "train_loss": round(train_loss, 4),
        "train_accuracy": round(train_acc, 4),
        "val_loss": round(val_loss, 4),
        "val_accuracy": round(val_acc, 4),
        "learning_rate": optimizer.param_groups[0]["lr"],
    }))
```

#### TensorFlow/Keras (Custom Callback)

```python
import json
import tensorflow as tf

class ValohaiMetricsCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        if logs:
            metrics = {"epoch": epoch}
            metrics.update({k: round(float(v), 4) for k, v in logs.items()})
            print(json.dumps(metrics))

model.fit(
    x_train, y_train,
    epochs=args.epochs,
    batch_size=args.batch_size,
    validation_data=(x_val, y_val),
    callbacks=[ValohaiMetricsCallback()],
)
```

#### scikit-learn

```python
import json
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(json.dumps({
    "accuracy": round(accuracy_score(y_test, y_pred), 4),
    "precision": round(precision_score(y_test, y_pred, average="weighted"), 4),
    "recall": round(recall_score(y_test, y_pred, average="weighted"), 4),
    "f1_score": round(f1_score(y_test, y_pred, average="weighted"), 4),
}))
```

#### XGBoost / LightGBM

```python
import json

def valohai_callback(env):
    """Custom callback to log metrics to Valohai."""
    # Collect all metrics into one event per iteration
    metrics = {"iteration": env.iteration}
    for item in env.evaluation_result_list:
        metrics[item[0]] = round(item[1], 4)
    print(json.dumps(metrics))

model = xgb.train(
    params, dtrain,
    num_boost_round=100,
    evals=[(dtrain, "train"), (dval, "val")],
    callbacks=[valohai_callback],
)
```

### 5. What Valohai Does With Metrics

- **Execution table**: Sort and filter executions by any metric value
- **Time-series charts**: Plot metrics over epochs/steps, updated in real-time during training
- **Multi-execution comparison**: Overlay metrics from multiple runs on the same chart
- **CSV/JSON export**: Download metric data for external analysis
- **Pipeline conditions**: Use metrics to control pipeline flow (e.g., stop if accuracy > threshold)

## Best Practices

1. **One event = one `json.dumps()`** - all metrics from the same moment MUST be in a single print. Separate prints create disconnected events that can't be correlated
2. **Log progressively** throughout training, not just final results - enables real-time monitoring
3. **Use consistent metric names** across experiments for meaningful comparison
4. **Include a step/epoch counter** as a metric for proper time-series alignment
5. **Round floating-point values** to 4-6 decimal places to keep logs readable
6. **Print to stdout** (not stderr) - Valohai only captures JSON from stdout
7. **Ensure valid JSON** - use `json.dumps()` rather than manual string formatting
8. **Add context fields** like `phase: "training"` or `phase: "validation"` to distinguish metrics
9. **Log at reasonable intervals** - every epoch is good; every batch may be too noisy unless filtered
10. **Stay under 50 events/second** - Valohai enforces a rate limit of 500 JSON events per 10 seconds (50/s). Exceeding this triggers a warning and **events will be dropped silently**. If logging per-batch metrics, add a frequency filter (e.g., every N batches) to stay well under this limit

## Edge Cases

- Non-JSON stdout lines are ignored by Valohai (treated as regular log output)
- Multiple JSON prints per line: only the first valid JSON object is captured
- Nested JSON objects are flattened for display in the UI
- String values in metrics are supported (e.g., `"best_model": "epoch_42"`)
- Boolean values are supported
- Metrics can be used in pipeline edge conditions: `metadata.accuracy >= 0.9`
- If using `print()` with frameworks that also print to stdout, the JSON lines are still correctly identified
- **Rate limit**: More than 500 JSON events per 10 seconds triggers `More than 50.0 events per second are being written to stdout; some are ignored.` — dropped events are lost permanently. Use batch-level filtering (`if batch_idx % N == 0`) to control output rate

View saved version →

---
name: valohai-migrate-parameters
description: Migrate hardcoded values, hyperparameters, and configuration values in ML code to Valohai parameters. Use this skill when a user wants to make their ML scripts configurable through Valohai, expose hyperparameters for tuning, externalize configuration values, convert hardcoded values to command-line arguments, or add parameter definitions to valohai.yaml. Triggers on mentions of parameters, hyperparameters, configuration, argparse, configurable training, or Valohai parameter migration.
license: Apache-2.0
metadata:
  author: valohai
  version: "1.0"
  compatibility: Requires Python. Designed for Claude Code or similar AI coding agents.
---

# Valohai Parameter Migration

Migrate hardcoded values in ML code to Valohai-managed parameters. Parameters cover both **hyperparameters** (learning rate, epochs, batch size) and **configuration values** (model name, output format, feature flags, thresholds, file paths). This makes jobs configurable without code changes, enables experiment tracking, hyperparameter sweeps, and reproducibility.

## Philosophy

Valohai follows a **YAML-over-SDK** approach. Your ML code should not be entangled with the platform. Parameters are declared in `valohai.yaml` and injected as command-line arguments. Your code uses standard Python `argparse` to receive them.

## Step-by-Step Instructions

### 1. Identify Hardcoded Values

Scan the user's ML code for hardcoded values that should be configurable. Parameters are not just for hyperparameters — any value that a user might want to change between runs belongs here:

- **Training hyperparameters**: learning rate, batch size, epochs, dropout rate, weight decay
- **Model architecture**: hidden layers, units per layer, activation functions, model type/name
- **Data processing**: train/test split ratio, image size, sequence length, number of workers
- **Optimization**: optimizer type, scheduler type, warmup steps, gradient clipping
- **Configuration values**: output format, confidence thresholds, feature flags, model name/path, random seed, number of threads/workers, log level
- **Any magic numbers** in training loops, model definitions, or scripts

### 2. Add argparse to the Python Code

Convert hardcoded values to command-line arguments using Python's standard `argparse`:

```python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--learning_rate", type=float, default=0.001)
parser.add_argument("--epochs", type=int, default=10)
parser.add_argument("--batch_size", type=int, default=32)
parser.add_argument("--optimizer", type=str, default="adam")
parser.add_argument("--use_augmentation", action="store_true")
args = parser.parse_args()
```

Then replace every hardcoded reference with the parsed argument:

```python
# Before
learning_rate = 0.001

# After
learning_rate = args.learning_rate
```

### 3. Declare Parameters in valohai.yaml

Add a `parameters` section to the step definition:

```yaml
- step:
    name: train-model
    image: tensorflow/tensorflow:2.6.0
    command:
      - pip install -r requirements.txt
      - python train.py {parameters}
    parameters:
      - name: learning_rate
        type: float
        default: 0.001
      - name: epochs
        type: integer
        default: 10
      - name: batch_size
        type: integer
        default: 32
      - name: optimizer
        type: string
        default: adam
      - name: use_augmentation
        type: flag
        default: false
```

### Command Interpolation: How Parameters Get Passed

Valohai supports several placeholders in the `command` field. Understanding the difference is critical to avoid argument parsing errors.

**SYNTAX WARNING**: The delimiter is always a **colon** (`:`), never brackets or parentheses. `{parameter-value:name}` is correct. `{parameter-value[name]}` and `{parameter(name)}` are WRONG and will silently fail.

#### `{parameters}` - Preferred approach

Expands to ALL parameters with their `pass-as` format (default: `--name=value`). This is the safest and most common approach:

```yaml
command:
  - python train.py {parameters}
# Expands to: python train.py --learning_rate=0.001 --epochs=10 --optimizer=adam
```

#### `{parameter:name}` - Single parameter with pass-as format

Expands to the full `--name=value` formatted string for one parameter. Useful when parameters need to go in specific positions:

```yaml
command:
  - python train.py --model /valohai/inputs/model/*.onnx {parameter:conf} {parameter:iou}
# Expands to: python train.py --model /valohai/inputs/model/*.onnx --conf=0.5 --iou=0.7
```

#### `{parameter-value:name}` - Raw value only

Expands to just the raw value, without the `--name=` prefix. Use this when you need to embed the value into a custom argument format:

```yaml
command:
  - python train.py --threshold {parameter-value:threshold}
# Expands to: python train.py --threshold 0.5
```

#### Common mistake: using `{parameter:name}` where `{parameter-value:name}` is needed

```yaml
# WRONG - double-passes the flag name, causes "expected one argument" errors
command:
  - python main.py --conf={parameter:conf}
# Expands to: python main.py --conf=--conf=0.5   <-- BROKEN

# CORRECT - use {parameter-value:name} for raw values
command:
  - python main.py --conf={parameter-value:conf}
# Expands to: python main.py --conf=0.5

# BEST - just use {parameter:name} on its own or {parameters} for all
command:
  - python main.py {parameter:conf} {parameter:iou}
# Expands to: python main.py --conf=0.5 --iou=0.7
```

**Rule of thumb**: Use `{parameters}` to pass all parameters. Use `{parameter:name}` when you need specific parameters in specific positions. Only use `{parameter-value:name}` when embedding a value into a custom format. Never combine `--flag={parameter:name}` - that double-wraps the argument. **Always use colons** — `{parameter-value:name}` not `{parameter-value[name]}`.

### 4. Parameter Types Reference

| Type      | YAML              | Python argparse           | CLI Example          | Notes |
|-----------|--------------------|---------------------------|----------------------|-------|
| Integer   | `type: integer`    | `type=int`                | `--epochs=100`       | Whole numbers |
| Float     | `type: float`      | `type=float`              | `--lr=0.001`         | Decimal numbers |
| String    | `type: string`     | `type=str`                | `--model=resnet`     | Text values |
| Flag      | `type: flag`       | `action='store_true'`     | `--augment`          | When false, nothing is passed |

### 5. Advanced Parameter Options

#### Constrained Ranges

```yaml
parameters:
  - name: learning_rate
    type: float
    default: 0.001
    min: 0.0001
    max: 0.1
  - name: batch_size
    type: integer
    default: 32
    min: 8
    max: 512
```

#### Choices (Dropdown in UI)

```yaml
parameters:
  - name: optimizer
    type: string
    default: adam
    choices:
      - adam
      - sgd
      - rmsprop
      - adamw
```

#### Optional Parameters

```yaml
parameters:
  - name: pretrained_model
    type: string
    optional: true
```

#### Flag with Boolean Pass-Through

By default, flags only pass the flag name when true and pass nothing when false. For explicit true/false:

```yaml
parameters:
  - name: verbose
    type: flag
    default: false
    pass-true-as: --verbose=True
    pass-false-as: --verbose=False
```

#### Custom Pass Format

```yaml
parameters:
  - name: seed
    type: integer
    default: 42
    pass-as: --random-seed={v}
```

#### Multiple Values

```yaml
parameters:
  - name: layers
    type: integer
    default: 128
    multiple: separate
    multiple-separator: ","
```

### 6. Alternative: Config File Approach

If the code cannot use argparse (e.g., it reads a config file), Valohai also creates config files automatically:

```python
import json

# Valohai creates this file automatically from the parameter values
with open("/valohai/config/parameters.json") as f:
    params = json.load(f)
    learning_rate = params["learning_rate"]
    epochs = params["epochs"]
```

Or with YAML:

```python
import yaml

with open("/valohai/config/parameters.yaml") as f:
    params = yaml.safe_load(f)
```

## Common Patterns

### PyTorch Training Script

```python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--epochs", type=int, default=10)
parser.add_argument("--learning_rate", type=float, default=0.001)
parser.add_argument("--batch_size", type=int, default=32)
parser.add_argument("--weight_decay", type=float, default=1e-5)
parser.add_argument("--optimizer", type=str, default="adam", choices=["adam", "sgd", "adamw"])
args = parser.parse_args()

# Use args throughout your training code
optimizer = torch.optim.Adam(model.parameters(), lr=args.learning_rate, weight_decay=args.weight_decay)
train_loader = DataLoader(dataset, batch_size=args.batch_size)

for epoch in range(args.epochs):
    train(model, train_loader, optimizer)
```

### TensorFlow/Keras Training Script

```python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--epochs", type=int, default=50)
parser.add_argument("--learning_rate", type=float, default=0.001)
parser.add_argument("--batch_size", type=int, default=64)
parser.add_argument("--dropout_rate", type=float, default=0.5)
args = parser.parse_args()

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=args.learning_rate), ...)
model.fit(x_train, y_train, epochs=args.epochs, batch_size=args.batch_size)
```

## Validate the YAML

**IMPORTANT**: After adding parameters to `valohai.yaml`, always run `vh lint` to validate the configuration before attempting to run:

```shell
vh lint
```

This catches common issues like invalid parameter types, missing fields, bad indentation, and duplicate names. Fix any errors before proceeding.

## Overriding Parameters at Runtime

Once parameters are defined, users can override them without changing code:

```shell
# Via CLI
vh execution run train-model --epochs=100 --learning_rate=0.0001 --adhoc

# Parameters also appear in the Valohai web UI for easy changes
```

## Edge Cases

- If code already uses argparse, just add the matching `parameters` section to `valohai.yaml` - no code changes needed
- Parameter names in YAML must match the argparse argument names (without the `--` prefix)
- Use underscores in parameter names for consistency (both `--learning_rate` and `--learning-rate` work)
- Flag parameters: when false, nothing is passed to the command line (the argparse default handles it)
- The `{parameters}` placeholder is optional - if omitted, Valohai appends parameters to the last command

View saved version →

---
name: valohai-project-run
description: Create a Valohai project, link it to a local directory, and run executions or pipelines using the Valohai CLI. Use this skill when a user wants to set up a new Valohai project, link an existing project, run a step execution, run a pipeline, watch execution logs, or manage their Valohai project via the CLI. Triggers on mentions of vh project create, vh execution run, vh pipeline run, running on Valohai, or Valohai CLI commands.
license: Apache-2.0
metadata:
  author: valohai
  version: "1.0"
  compatibility: Requires valohai-cli installed (pip install valohai-cli). Designed for Claude Code or similar AI coding agents.
---

# Valohai Project Setup and Execution

Set up a Valohai project and run ML steps using the Valohai CLI (`vh`). This covers project creation, linking, running executions, running pipelines, and monitoring jobs.

## Prerequisites

```shell
# Install the CLI
pip install valohai-cli

# Login (interactive)
vh login

# Login with token (for SSO or CI/CD)
vh login --token YOUR_TOKEN

# Login to self-hosted instance
vh login --host https://your-company.valohai.io --token YOUR_TOKEN
```

## Step-by-Step: Project Setup

### 1. Create a New Project

```shell
# Create and link to current directory
vh project create --name my-ml-project

# Create with description
vh project create --name my-ml-project --description "Image classification pipeline"

# Create under an organization (CASE SENSITIVE - must match exactly)
vh project create --name my-ml-project --owner my-org

# Create under an organization team (CASE SENSITIVE)
vh project create --name my-ml-project --owner my-org:ml-team

# Create without linking
vh project create --name my-ml-project --no-link
```

This creates the project on Valohai and links the current directory to it.

### 2. Link to an Existing Project

```shell
# Interactive selection from your projects
vh project link

# Or specify directly
vh project link --project my-org/my-ml-project
```

### 3. Check Project Status

```shell
# See current project link
vh project status

# List all your projects
vh project list

# Open project in browser
vh project open
```

## Step-by-Step: Running Executions

### 1. Run a Step (Ad-hoc)

During development, use `--adhoc` to run local code without Git commits:

```shell
# Basic run with local code
vh execution run train-model --adhoc

# Run and watch logs in real-time
vh execution run train-model --adhoc --watch

# Run with parameter overrides
vh execution run train-model --adhoc --epochs=100 --learning_rate=0.0001

# Run with input overrides
vh execution run train-model --adhoc --training-data=s3://bucket/new-data.csv

# Run with specific Docker image override
vh execution run train-model --adhoc --image pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime

# Run with environment override
vh execution run train-model --adhoc --environment aws-eu-west-1-p3-2xlarge

# Run with tags
vh execution run train-model --adhoc --tag experiment-v2 --tag baseline

# Run with title
vh execution run train-model --adhoc --title "Baseline experiment with new data"

# Run with environment variables
vh execution run train-model --adhoc --var WANDB_MODE=disabled

# Combine multiple options
vh execution run train-model --adhoc --watch \
  --epochs=100 \
  --learning_rate=0.0001 \
  --training-data=s3://bucket/v2/train.csv \
  --tag baseline-v2
```

### 2. Run from a Git Commit

When code is committed and pushed:

```shell
# Run from latest fetched commit
vh execution run train-model

# Run from specific commit
vh execution run train-model --commit abc123

# Run from a branch/tag
vh execution run train-model --commit main
```

### 3. Monitor Executions

```shell
# List recent executions
vh execution list
vh execution list --count 10

# Stream logs from an execution
vh execution logs <counter-or-id> --stream

# Watch a running execution
vh execution watch <counter-or-id>

# Get execution details
vh execution info <counter-or-id>

# Open execution in browser
vh execution open <counter-or-id>

# List execution outputs
vh execution outputs <counter-or-id>

# Download outputs
vh execution outputs <counter-or-id> --download ./local-dir/

# Stop a running execution
vh execution stop <counter-or-id>
```

### 4. See Available Steps

```shell
# List steps in current commit
vh execution run
# (Without a step name, it shows available steps)
```

## Step-by-Step: Running Pipelines

### 1. Run a Pipeline

```shell
# Run pipeline with local code
vh pipeline run my-pipeline --adhoc

# Run from Git
vh pipeline run my-pipeline

# Run with title and tags
vh pipeline run my-pipeline --adhoc --title "Full training run" --tag weekly

# Override node parameters
vh pipeline run my-pipeline --adhoc --train-node.epochs=200

# Override node inputs
vh pipeline run my-pipeline --adhoc --train-node.dataset=s3://bucket/v2/

# Override pipeline-level parameters
vh pipeline run my-pipeline --adhoc --learning_rate=0.0001

# Override environment for all nodes
vh pipeline run my-pipeline --adhoc --environment aws-eu-west-1-p3-2xlarge
```

### 2. Monitor Pipelines

```shell
# List pipelines
vh pipeline list

# Stop a pipeline
vh pipeline stop <counter-or-id>
```

### 3. Debug Failed Pipeline Executions

**CRITICAL: Pipeline numbers and execution numbers are different things.** A pipeline contains multiple executions (one per node). When a pipeline fails, you must find and inspect the **specific failed execution**, not the pipeline itself.

```shell
# WRONG - this is the pipeline number, not an execution number
vh execution logs 42 --stream

# CORRECT workflow:
# 1. Open the pipeline in the browser to see which node failed
vh pipeline list

# 2. List executions to find the ones belonging to your pipeline
vh execution list

# 3. Stream logs from the specific EXECUTION that failed (not the pipeline number)
vh execution logs <execution-counter> --stream

# 4. Get details about the failed execution
vh execution info <execution-counter>
```

Each node in a pipeline creates its own execution with its own counter number. Always use the execution counter (visible in the execution list or the web UI) when fetching logs, not the pipeline counter.

## Step-by-Step: Complete Workflow

Here's the full workflow from zero to running:

```shell
# 1. Create project directory
mkdir my-ml-project && cd my-ml-project

# 2. Create Valohai project
vh project create --name my-ml-project

# 3. Create your training script (train.py)
# 4. Create valohai.yaml (see valohai-yaml-step skill)

# 5. Validate the YAML
vh lint

# 6. Run your first execution
vh execution run train-model --adhoc --watch

# 7. View results in browser
vh execution open

# 8. Run with different parameters
vh execution run train-model --adhoc --watch --epochs=100 --learning_rate=0.0001

# 9. Once satisfied, commit and push code
git add . && git commit -m "Add Valohai configuration"
git push origin main

# 10. Fetch latest commit on Valohai
vh project fetch

# 11. Run from Git (reproducible)
vh execution run train-model --watch
```

## YAML Management

**IMPORTANT**: Always run `vh lint` after any change to `valohai.yaml`. This is the first thing to try when executions fail due to configuration issues. It validates syntax, checks references, and catches common mistakes.

```shell
# Validate valohai.yaml - run this after every YAML change
vh lint

# Generate YAML step from a Python file (if using valohai-utils)
vh yaml step train.py

# Generate pipeline YAML from Python pipeline definition
vh yaml pipeline pipeline.py
```

## Other Useful Commands

```shell
# List available environments
vh environments

# List project commits
vh project commits

# List data/files in the project
vh data list

# List project aliases
vh alias list

# Manage environment variables
vh project env-var list
vh project env-var create MY_VAR=value
vh project env-var delete MY_VAR

# List tasks (hyperparameter sweeps)
vh task list
vh task stop <id>
```

## Common Workflows

### Quick Iteration Loop

```shell
# Edit code → run → check results → repeat
vh execution run train-model --adhoc --watch
# (Make changes to code)
vh execution run train-model --adhoc --watch --epochs=200
```

### Hyperparameter Comparison

```shell
# Run multiple experiments
vh execution run train-model --adhoc --tag lr-001 --learning_rate=0.001
vh execution run train-model --adhoc --tag lr-0001 --learning_rate=0.0001
vh execution run train-model --adhoc --tag lr-00001 --learning_rate=0.00001

# Compare in the web UI
vh project open
```

### Pipeline Development

```shell
# Test individual steps first
vh execution run preprocess-data --adhoc --watch
vh execution run train-model --adhoc --watch

# Then run the full pipeline
vh pipeline run training-pipeline --adhoc
```

## Edge Cases

- `--adhoc` packages and uploads your local working directory - ensure no large files or secrets are present (use `.valohaiignore` to exclude files)
- Parameters and inputs are passed as `--name=value` after the step name
- The CLI auto-detects parameters vs inputs: URLs (containing `://`) are treated as input overrides
- Without `--watch`, the command returns immediately after queuing the execution
- `vh lint` validates YAML syntax but doesn't check if Docker images or environments exist
- For remote-only projects (no local Git), omit `--adhoc` and use `--commit`

View saved version →

---
name: valohai-yaml-step
description: Create valohai.yaml step definitions for ML projects. Use this skill when a user wants to define a new step in valohai.yaml, configure Docker images, set up commands, define parameters/inputs for a step, or create a complete valohai.yaml from scratch for their ML project. Triggers on mentions of valohai.yaml, step definition, YAML configuration, Docker image selection, or creating Valohai steps.
license: Apache-2.0
metadata:
  author: valohai
  version: "1.0"
  compatibility: Requires Python. Designed for Claude Code or similar AI coding agents.
---

# Create Valohai YAML Steps

Create and configure `valohai.yaml` step definitions that describe how ML code should run on Valohai. A step defines the Docker image, commands, parameters, and inputs for an execution.

## Philosophy

The `valohai.yaml` file is the single source of truth for your ML workflows. It lives in the repository root and is version-controlled alongside your code. Your code stays framework-agnostic and portable - all platform configuration is in this file.

## Step-by-Step Instructions

### Design Principle: Steps Should Run Independently

**IMPORTANT**: Every step should be runnable on its own, without relying on a pipeline to provide its inputs. Whenever possible, set a `default` value for inputs so users can test and debug individual steps via `vh execution run step-name --adhoc` without needing to run the full pipeline first. Pipeline edges override these defaults at runtime.

Look for default values in the existing code, README, documentation, config files, or data paths already referenced in the project. Only add a default if you can find a real, meaningful value. Do not invent placeholder URLs or dummy paths.

### 1. Analyze the Project

Before creating the YAML, understand the project:

- **What scripts exist?** (train.py, preprocess.py, evaluate.py, infer.py)
- **What framework?** (PyTorch, TensorFlow, scikit-learn, XGBoost, HuggingFace, etc.)
- **What Python version?** (Check pyproject.toml, setup.py, .python-version)
- **What dependencies?** (requirements.txt, pyproject.toml, conda environment)
- **Does it need GPU?** (CUDA imports, torch.cuda, tf.config)
- **What are the inputs?** (datasets, pretrained models, config files)
- **What are the outputs?** (trained models, metrics, plots, predictions)
- **What are the configurable values?** (hyperparameters, paths, flags)

### 2. Choose the Docker Image

Select an appropriate base image:

| Framework | CPU Image | GPU Image |
|-----------|-----------|-----------|
| Python only | `python:3.11` | N/A |
| TensorFlow | `tensorflow/tensorflow:2.15.0` | `tensorflow/tensorflow:2.15.0-gpu` |
| PyTorch | `pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime` | Same (includes CUDA) |
| HuggingFace | `huggingface/transformers-pytorch-gpu:latest` | Same |
| scikit-learn | `python:3.11` | N/A |
| R | `r-base:4.3.0` | N/A |
| Custom | Your own Docker image from a registry | Your own Docker image |

**Guidelines:**
- The Docker image is the execution environment, not your code container
- Don't include your code or data in the image
- Pin version tags for reproducibility (avoid `latest`)
- If using GPU, choose an image with the right CUDA version

### 3. Install Dependencies in Commands

**IMPORTANT**: The Docker image is a clean base environment. Your project's dependencies (from `requirements.txt`, `pyproject.toml`, `environment.yml`, etc.) are NOT pre-installed. You must install them as part of the step commands BEFORE running your script, otherwise you'll get `ModuleNotFoundError`.

Look for dependency files in the project:

| File Found | Install Command |
|------------|----------------|
| `requirements.txt` | `pip install -r requirements.txt` |
| `pyproject.toml` (with pip) | `pip install .` or `pip install -e .` |
| `pyproject.toml` (with uv) | `pip install uv && uv pip install --system .` |
| `setup.py` | `pip install .` |
| `environment.yml` (conda) | `conda env update -n base -f environment.yml` |
| `Pipfile` | `pip install pipenv && pipenv install` |
| Inline dependencies | `pip install torch numpy pandas ...` |

Always place install commands before any Python script execution in the `command` list.

### 4. Keep Commands Simple — Use Helper Scripts

**IMPORTANT**: Step commands in `valohai.yaml` must be simple and human-readable. Never put bash loops, inline Python one-liners, complex piped commands, or multi-line logic directly in the `command` list.

If a step needs non-trivial setup or orchestration logic (e.g., downloading extra resources, converting formats, running multiple sub-steps), **create a helper Python script** in a `valohai/` folder and call it from the command:

```yaml
# WRONG - complex logic crammed into commands
command:
  - pip install -r requirements.txt
  - for f in /valohai/inputs/data/*.tar; do tar xf "$f" -C /tmp/data; done
  - python -c "import os; [os.rename(f, f.lower()) for f in os.listdir('/tmp/data')]"
  - python train.py {parameters}

# CORRECT - clean commands, logic lives in a helper script
command:
  - pip install -r requirements.txt
  - python valohai/prepare_data.py
  - python train.py {parameters}
```

The helper script `valohai/prepare_data.py` contains the extraction and renaming logic. This keeps the YAML readable, the logic testable, and avoids shell quoting nightmares.

**Rule of thumb**: If a command line is longer than ~80 characters or uses shell control flow (loops, conditionals, subshells), move it to a helper script.

### 5. Create the Step Definition

#### Minimal Step

```yaml
- step:
    name: train-model
    image: python:3.11
    command:
      - pip install -r requirements.txt
      - python train.py
```

#### Complete Step with All Options

```yaml
- step:
    name: train-model
    description: Train a classification model on the prepared dataset
    category: Training
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    environment: aws-eu-west-1-p3-2xlarge
    command:
      - pip install -r requirements.txt
      - python train.py {parameters}
    parameters:
      - name: epochs
        type: integer
        default: 50
        description: Number of training epochs
      - name: learning_rate
        type: float
        default: 0.001
        min: 0.00001
        max: 0.1
        description: Initial learning rate
      - name: batch_size
        type: integer
        default: 32
        choices: [8, 16, 32, 64, 128, 256]
      - name: model_type
        type: string
        default: resnet50
        choices: [resnet18, resnet50, resnet101, efficientnet_b0]
      - name: use_augmentation
        type: flag
        default: true
        pass-true-as: --use_augmentation=True
        pass-false-as: --use_augmentation=False
    inputs:
      - name: training-data
        default: s3://my-bucket/datasets/train/
      - name: validation-data
        default: s3://my-bucket/datasets/val/
      - name: pretrained-model
        optional: true
    time-limit: 2h
    no-output-timeout: 30m
```

### 5. Step Properties Reference

#### Required Properties

| Property | Description |
|----------|-------------|
| `name` | Unique step identifier. Used in CLI: `vh exec run <name>` |
| `image` | Docker image for the execution environment |
| `command` | Command(s) to run. String or list of strings |

#### Optional Properties

| Property | Description |
|----------|-------------|
| `description` | Help text shown in UI |
| `category` | Group steps in UI (e.g., "Training", "Preprocessing") |
| `environment` | Default compute environment (ID or name substring) |
| `parameters` | Configurable values (see parameter migration skill) |
| `inputs` | Data files to download before execution |
| `time-limit` | Max execution time (`"2h"`, `"1h 30m"`, or seconds) |
| `no-output-timeout` | Kill if no stdout for this duration |
| `environment-variables` | Environment variables with defaults |
| `environment-variable-groups` | Groups of env vars to include |
| `stop-condition` | Auto-stop condition based on metrics (e.g., `accuracy >= 0.99`) |
| `upload-store` | Override default output data store |
| `resources` | Kubernetes resource requests/limits |

### 6. Multi-Step YAML

Most ML projects have multiple steps:

```yaml
# Preprocessing step
- step:
    name: preprocess-data
    image: python:3.11
    command:
      - pip install pandas numpy scikit-learn
      - python preprocess.py {parameters}
    parameters:
      - name: test_split
        type: float
        default: 0.2
      - name: random_seed
        type: integer
        default: 42
    inputs:
      - name: raw-data
        default: s3://bucket/raw-data.csv

# Training step
- step:
    name: train-model
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - pip install -r requirements.txt
      - python train.py {parameters}
    parameters:
      - name: epochs
        type: integer
        default: 50
      - name: learning_rate
        type: float
        default: 0.001
      - name: batch_size
        type: integer
        default: 32
    inputs:
      - name: dataset
        default: s3://bucket/processed/
      - name: pretrained-model
        optional: true

# Evaluation step
- step:
    name: evaluate-model
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - pip install -r requirements.txt
      - python evaluate.py
    inputs:
      - name: model
        default: datum://latest-trained-model
      - name: test-data
        default: s3://bucket/test/

# Batch inference step
- step:
    name: batch-inference
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - pip install -r requirements.txt
      - python inference.py
    inputs:
      - name: model
        default: datum://production-model
      - name: data
        default: s3://bucket/inference-data/
```

### 7. Command Patterns

```yaml
# Single command
command: python train.py

# Multiple commands (run sequentially)
command:
  - pip install -r requirements.txt
  - python preprocess.py
  - python train.py {parameters}

# PARAMETER SYNTAX - uses COLONS, not brackets or parentheses
#
# {parameters}              → all params as --name=value
# {parameter:name}          → one param as --name=value
# {parameter-value:name}    → one param as raw value only
#
# WRONG: {parameter-value[name]} {parameter[name]} {parameter(name)}
# RIGHT: {parameter-value:name}  {parameter:name}  (always use colons!)

# {parameters} - expands ALL parameters with --name=value format (preferred)
command:
  - python train.py {parameters}
# Expands to: python train.py --epochs=50 --learning_rate=0.001

# {parameter:name} - ONE parameter as --name=value (standalone, never after --)
command:
  - python train.py --model /valohai/inputs/model/*.onnx {parameter:conf} {parameter:iou}
# Expands to: python train.py --model /valohai/inputs/model/*.onnx --conf=0.5 --iou=0.7

# {parameter-value:name} - ONE parameter as RAW VALUE ONLY (no --name= prefix)
command:
  - python train.py --threshold {parameter-value:threshold}
# Expands to: python train.py --threshold 0.5

# WRONG - never combine --flag={parameter:name}, it double-wraps the argument
# --conf={parameter:conf} expands to --conf=--conf=0.5 which BREAKS argparse

# Install from different sources
command:
  - DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y libgl1-mesa-glx
  - pip install -r requirements.txt
  - python train.py

# Conditional commands
command:
  - pip install -r requirements.txt
  - python preprocess.py && python train.py {parameters}
```

**IMPORTANT: Non-interactive commands and shell isolation.**

Each YAML list item runs in a **separate shell**. This means `export` on one line does NOT carry over to the next. Chain commands that share environment on the same line:

```yaml
# WRONG - export doesn't carry to the next line (separate shells)
command:
  - export DEBIAN_FRONTEND=noninteractive
  - apt-get update && apt-get install -y tzdata   # WILL STILL PROMPT

# CORRECT - chain export on the same line so it applies to both commands
command:
  - DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata
  - pip install -r requirements.txt
  - python train.py {parameters}
```

Always use `-y` (or `--yes`) and `DEBIAN_FRONTEND=noninteractive` with `apt-get install`. Executions run unattended - any interactive prompt will hang the execution indefinitely.

### 8. Environment Variables

```yaml
- step:
    name: train-model
    image: python:3.11
    command:
      - python train.py
    environment-variables:
      - name: WANDB_MODE
        default: disabled
        description: Disable W&B logging on Valohai
      - name: HF_HOME
        default: /valohai/inputs/hf-cache/
    environment-variable-groups:
      - secret-api-keys-group-uuid
```

### 9. Resource Configuration (Kubernetes)

```yaml
- step:
    name: train-large-model
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - python train.py
    resources:
      cpu:
        min: 4
        max: 8
      memory:
        min: 16384  # MB
        max: 32768
      devices:
        nvidia.com/gpu: 1
```

## Common Patterns by Project Type

### Computer Vision

```yaml
- step:
    name: train-detector
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    command:
      - pip install ultralytics
      - python train.py {parameters}
    parameters:
      - name: epochs
        type: integer
        default: 100
      - name: img_size
        type: integer
        default: 640
      - name: batch_size
        type: integer
        default: 16
    inputs:
      - name: dataset
        keep-directories: suffix
```

### NLP/LLM Fine-tuning

```yaml
- step:
    name: finetune-llm
    image: huggingface/transformers-pytorch-gpu:latest
    command:
      - pip install peft bitsandbytes
      - python finetune.py {parameters}
    parameters:
      - name: model_name
        type: string
        default: meta-llama/Llama-2-7b-hf
      - name: lora_r
        type: integer
        default: 16
      - name: learning_rate
        type: float
        default: 0.0002
      - name: num_train_epochs
        type: integer
        default: 3
    inputs:
      - name: training-data
    environment-variables:
      - name: HF_TOKEN
        optional: false
```

### Data Pipeline

```yaml
- step:
    name: etl-pipeline
    image: python:3.11
    command:
      - pip install pandas pyarrow
      - python etl.py {parameters}
    parameters:
      - name: date_from
        type: string
        default: "2024-01-01"
      - name: date_to
        type: string
        default: "2024-12-31"
    inputs:
      - name: raw-data
        default: s3://data-lake/raw/
        keep-directories: suffix
```

## Validation

**IMPORTANT**: Always run `vh lint` after creating or modifying `valohai.yaml`. This validates the YAML syntax, checks parameter types, verifies input/output definitions, and catches common mistakes before you attempt to run anything.

```shell
vh lint
```

If `vh lint` reports errors, fix them before proceeding. Common issues:
- Indentation errors (YAML is whitespace-sensitive)
- Missing required fields (`name`, `image`, `command`)
- Invalid parameter types (must be `integer`, `float`, `string`, or `flag`)
- Duplicate step or parameter names
- Invalid `pass-as` format strings

Once lint passes, test with a run:

```shell
vh execution run step-name --adhoc --watch
```

## Edge Cases

- The `valohai.yaml` file must be in the repository root
- Step names must be unique within the YAML file
- Step names can contain letters, numbers, spaces, hyphens, and parentheses
- If `{parameters}` is not in the command, parameters are appended to the last command
- Multiple YAML files can be used via imports (see Valohai docs on multi-file YAML)
- Commands run sequentially; if one fails, subsequent commands don't execute
- The working directory inside the container is `/valohai/repository/`
- Each YAML command list item runs in a separate shell - `export` on one line won't carry to the next
- Always use non-interactive flags (`-y`, `DEBIAN_FRONTEND=noninteractive`) for package installs - executions have no TTY