Update to Valohai Skills

{
  "description": "Analyze ML project structure and design Valohai pipelines to orchestrate multi-step workflows. Use this skill when a user wants to create a Valohai pipeline, connect multiple ML steps into an automated workflow, identify pipeline opportunities in their codebase, design data flow between preprocessing/training/evaluation/inference steps, or add conditional logic and parallel execution to pipelines. Triggers on mentions of pipelines, workflows, DAGs, orchestration, multi-step ML, or connecting Valohai steps.",
  "included_files": [],
  "name": "valohai-design-pipelines",
  "skill_md_contents": "---\nname: valohai-design-pipelines\ndescription: Analyze ML project structure and design Valohai pipelines to orchestrate multi-step workflows. Use this skill when a user wants to create a Valohai pipeline, connect multiple ML steps into an automated workflow, identify pipeline opportunities in their codebase, design data flow between preprocessing/training/evaluation/inference steps, or add conditional logic and parallel execution to pipelines. Triggers on mentions of pipelines, workflows, DAGs, orchestration, multi-step ML, or connecting Valohai steps.\nlicense: Apache-2.0\nmetadata:\n  author: valohai\n  version: \"1.0\"\n  compatibility: Requires Python ML projects. Designed for Claude Code or similar AI coding agents.\n---\n\n# Design Valohai Pipelines\n\nAnalyze ML project structure to identify pipeline opportunities and design multi-step Valohai pipelines that orchestrate preprocessing, training, evaluation, and deployment workflows.\n\n## What Pipelines Do\n\nA Valohai pipeline connects individual steps into an automated workflow:\n- **Nodes** = Individual steps (preprocessing, training, evaluation, inference, deployment)\n- **Edges** = Data flow between steps (outputs become inputs, metadata becomes parameters)\n\nValohai automatically executes nodes in the right order, passes data between steps, runs independent nodes in parallel, and tracks complete data lineage.\n\n## Step-by-Step: Analyze the Project\n\n### 1. Identify Pipeline Stages\n\nExamine the ML project to find distinct stages. Common patterns:\n\n**Standard ML Pipeline:**\n```\nRaw Data → Preprocessing → Training → Evaluation → (Deployment)\n```\n\n**With Feature Engineering:**\n```\nRaw Data → Feature Engineering → Train/Val/Test Split → Training → Evaluation\n```\n\n**With Hyperparameter Optimization:**\n```\nPreprocessing → HPO Sweep (Task) → Select Best → Final Training → Evaluation\n```\n\n**Multi-Model Pipeline:**\n```\n                    ┌→ Train Model A ─┐\nPreprocessing ──────┤                  ├→ Ensemble → Evaluation\n                    └→ Train Model B ─┘\n```\n\n**Full Production Pipeline:**\n```\nData Ingestion → Preprocessing → Training → Evaluation → Model Validation → Deployment → Smoke Test\n```\n\n### 2. Map Data Dependencies\n\nFor each stage, identify:\n- **What files does it need?** (input data, models, configs)\n- **What files does it produce?** (processed data, trained models, metrics, reports)\n- **What metrics does it produce?** (accuracy, loss, F1 that downstream steps might need)\n- **What parameters should flow between steps?** (best learning rate, selected features)\n\n### 3. Identify Parallelization Opportunities\n\nSteps that don't depend on each other can run in parallel:\n- Training multiple model variants\n- Processing different data splits\n- Running different evaluation metrics\n- A/B training with different hyperparameters\n\n## Step-by-Step: Create the Pipeline YAML\n\n### Design Principle: Every Step Should Run Independently\n\n**IMPORTANT**: Whenever possible, give every input a `default` value so each step can be tested and debugged on its own with `vh execution run step-name --adhoc`. Pipeline edges override these defaults at runtime.\n\nLook for default values in the existing code, README, documentation, config files, or data paths already referenced in the project. Only add a default if you can find a real, meaningful value. Do not invent placeholder URLs or dummy paths - leave the input without a default if no real value is available.\n\n### 1. Ensure Steps Are Defined First\n\nEach pipeline node references an existing step. Define all steps before the pipeline:\n\n```yaml\n- step:\n    name: preprocess-data\n    image: python:3.11\n    command:\n      - pip install -r requirements.txt\n      - python preprocess.py {parameters}\n    parameters:\n      - name: test_split\n        type: float\n        default: 0.2\n    inputs:\n      - name: raw-data\n        default: s3://bucket/raw-data/\n\n- step:\n    name: train-model\n    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime\n    command:\n      - pip install -r requirements.txt\n      - python train.py {parameters}\n    parameters:\n      - name: epochs\n        type: integer\n        default: 50\n      - name: learning_rate\n        type: float\n        default: 0.001\n    inputs:\n      - name: train-data\n        default: s3://bucket/processed/train/\n      - name: val-data\n        default: s3://bucket/processed/val/\n\n- step:\n    name: evaluate-model\n    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime\n    command:\n      - pip install -r requirements.txt\n      - python evaluate.py\n    inputs:\n      - name: model\n        default: datum://latest-trained-model\n      - name: test-data\n        default: s3://bucket/processed/test/\n```\n\n### 2. Define the Pipeline\n\n```yaml\n- pipeline:\n    name: training-pipeline\n    nodes:\n      - name: preprocess\n        type: execution\n        step: preprocess-data\n\n      - name: train\n        type: execution\n        step: train-model\n        override:\n          inputs:\n            - name: train-data\n            - name: val-data\n\n      - name: evaluate\n        type: execution\n        step: evaluate-model\n        override:\n          inputs:\n            - name: model\n            - name: test-data\n\n    edges:\n      # Preprocessing outputs → training inputs\n      - [preprocess.output.*train*, train.input.train-data]\n      - [preprocess.output.*val*, evaluate.input.test-data]\n      - [preprocess.output.*val*, train.input.val-data]\n\n      # Training output → evaluation input\n      - [train.output.model*, evaluate.input.model]\n```\n\n### 3. Edge Syntax Reference\n\nEdges define data flow: `[source, target]`\n\n#### Source Formats\n\n| Format | Description | Example |\n|--------|-------------|---------|\n| `node.output.*` | All outputs from a node | `preprocess.output.*` |\n| `node.output.*.csv` | Outputs matching extension | `preprocess.output.*.csv` |\n| `node.output.name*` | Outputs starting with name | `train.output.model*` |\n| `node.output.*name*` | Outputs containing name | `preprocess.output.*train*` |\n| `node.metadata.metric` | A logged metric value | `train.metadata.best_lr` |\n| `node.parameter.name` | A parameter value | `train.parameter.learning_rate` |\n\n#### Target Formats\n\n| Format | Description | Example |\n|--------|-------------|---------|\n| `node.input.name` | An input on the target node | `train.input.train-data` |\n| `node.parameter.name` | A parameter on the target node | `finetune.parameter.learning_rate` |\n| `deploy.file.endpoint.name` | A file for a deployment endpoint | `deploy.file.predict.model` |\n\n### 4. Override Node Defaults\n\nWhen a node receives data from an edge, clear its default input so the edge value takes priority:\n\n```yaml\nnodes:\n  - name: train\n    type: execution\n    step: train-model\n    override:\n      inputs:\n        - name: train-data    # No default - will come from edge\n        - name: val-data       # No default - will come from edge\n      parameters:\n        - name: epochs\n          default: 200         # Override the step's default\n```\n\n## Advanced Pipeline Patterns\n\n### Parallel Training\n\n```yaml\n- pipeline:\n    name: parallel-training\n    nodes:\n      - name: preprocess\n        type: execution\n        step: preprocess-data\n\n      - name: train-resnet\n        type: execution\n        step: train-model\n        override:\n          parameters:\n            - name: model_type\n              default: resnet50\n          inputs:\n            - name: train-data\n\n      - name: train-efficientnet\n        type: execution\n        step: train-model\n        override:\n          parameters:\n            - name: model_type\n              default: efficientnet_b0\n          inputs:\n            - name: train-data\n\n      - name: select-best\n        type: execution\n        step: compare-models\n        override:\n          inputs:\n            - name: model-a\n            - name: model-b\n\n    edges:\n      - [preprocess.output.*, train-resnet.input.train-data]\n      - [preprocess.output.*, train-efficientnet.input.train-data]\n      - [train-resnet.output.model*, select-best.input.model-a]\n      - [train-efficientnet.output.model*, select-best.input.model-b]\n```\n\n### Hyperparameter Sweep with Task Node\n\n```yaml\n- pipeline:\n    name: hpo-pipeline\n    nodes:\n      - name: preprocess\n        type: execution\n        step: preprocess-data\n\n      - name: hpo-sweep\n        type: task\n        step: train-model\n        override:\n          inputs:\n            - name: train-data\n\n      - name: deploy-best\n        type: execution\n        step: deploy-model\n\n    edges:\n      - [preprocess.output.*, hpo-sweep.input.train-data]\n      - [hpo-sweep.output.model*, deploy-best.input.model]\n```\n\n### Conditional Execution (Actions)\n\n```yaml\n- pipeline:\n    name: conditional-pipeline\n    nodes:\n      - name: train\n        type: execution\n        step: train-model\n        actions:\n          # Stop pipeline if accuracy is too low\n          - when: node-complete\n            if: metadata.accuracy < 0.5\n            then: stop-pipeline\n\n      - name: evaluate\n        type: execution\n        step: evaluate-model\n        actions:\n          # Require human approval before proceeding\n          - when: node-starting\n            then: require-approval\n\n      - name: deploy\n        type: execution\n        step: deploy-model\n\n    edges:\n      - [train.output.model*, evaluate.input.model]\n      - [evaluate.output.*, deploy.input.model]\n```\n\n#### Action Reference\n\n**When (triggers):**\n- `node-starting` - Before the node begins\n- `node-complete` - After the node succeeds\n- `node-error` - When the node fails\n\n**If (conditions):**\n- `metadata.metric_name >= value` - Based on logged metric\n- `metadata.metric_name < value`\n- Can reference any metadata key logged via JSON printing\n\n**Then (actions):**\n- `stop-pipeline` - Halt the entire pipeline\n- `require-approval` - Pause until human approves\n\n### Passing Metrics as Parameters\n\n```yaml\nedges:\n  # Pass a logged metric from training as a parameter to fine-tuning\n  - [train.metadata.best_learning_rate, finetune.parameter.learning_rate]\n\n  # Pass a parameter value between nodes\n  - [train.parameter.user_id, evaluate.parameter.user_id]\n```\n\n### Pipeline-Level Parameters\n\n```yaml\n- pipeline:\n    name: configurable-pipeline\n    parameters:\n      - name: learning_rate\n        default: 0.001\n        targets:\n          - train.parameter.learning_rate\n          - finetune.parameter.learning_rate\n      - name: dataset_version\n        default: v2\n        target: preprocess.parameter.version\n    nodes:\n      # ... nodes ...\n    edges:\n      # ... edges ...\n```\n\n### Deployment Node\n\n```yaml\n- pipeline:\n    name: train-and-deploy\n    nodes:\n      - name: train\n        type: execution\n        step: train-model\n\n      - name: deploy\n        type: deployment\n        deployment: my-deployment\n        endpoints:\n          - predict\n        actions:\n          - when: node-starting\n            then: require-approval\n\n    edges:\n      - [train.output.model*, deploy.file.predict.model]\n```\n\n### Error Handling\n\n```yaml\nnodes:\n  - name: train\n    type: execution\n    step: train-model\n    on-error: stop-all    # stop-all | stop-next | continue | retry\n```\n\n| Strategy | Description |\n|----------|-------------|\n| `stop-all` | Stop the entire pipeline |\n| `stop-next` | Don't start downstream nodes, but let parallel nodes finish |\n| `continue` | Ignore the error and continue |\n| `retry` | Retry the failed node |\n\n### Node Reuse (Caching)\n\nSkip nodes that already ran successfully with identical configuration:\n\n```yaml\n- pipeline:\n    name: cached-pipeline\n    reuse-executions: true\n    nodes:\n      # ... nodes ...\n    edges:\n      # ... edges ...\n```\n\nA node is considered unchanged when: same inputs, same command, same commit hash, same parameters, same step configuration.\n\n## How to Identify Pipeline Opportunities\n\nWhen analyzing a codebase, look for:\n\n1. **Sequential scripts**: `run_preprocess.sh && run_train.sh && run_eval.sh` → pipeline\n2. **Shared data between scripts**: Output of one script is input to another → edges\n3. **Multiple model training runs**: Same training script with different configs → parallel nodes or task node\n4. **Manual model selection**: Comparing results and picking the best → conditional actions + metadata edges\n5. **Deployment after training**: Manual model promotion → deployment node with approval gate\n6. **Scheduled retraining**: Cron jobs running training → pipeline triggers\n7. **Data validation before training**: Data quality checks → conditional node with stop-pipeline action\n8. **A/B model evaluation**: Testing multiple approaches → parallel training + comparison node\n\n## Validate the Pipeline YAML\n\n**IMPORTANT**: After defining a pipeline in `valohai.yaml`, always run `vh lint` to validate before running:\n\n```shell\nvh lint\n```\n\nThis catches issues like:\n- References to undefined steps in nodes\n- Invalid edge syntax or missing node names\n- Duplicate node names\n- Missing required fields (`name`, `nodes`, `edges`)\n- Invalid action conditions\n\nFix all lint errors before attempting to run the pipeline.\n\n## Running Pipelines\n\n```shell\n# Run with local code\nvh pipeline run training-pipeline --adhoc\n\n# Run from Git\nvh pipeline run training-pipeline\n\n# Override node parameters\nvh pipeline run training-pipeline --adhoc --train.epochs=200\n\n# Override pipeline-level parameters\nvh pipeline run training-pipeline --adhoc --learning_rate=0.0001\n```\n\n## Edge Cases\n\n- Every node must reference an existing step defined in the same `valohai.yaml`\n- Edges require both `nodes` and `edges` sections in the pipeline\n- When using `override` on a node to clear an input, list the input name without a default\n- Output wildcards (`*`) match against the filename, not the full path\n- A pipeline must have at least one node and one edge\n- Nodes without incoming edges start immediately when the pipeline runs\n- Multiple edges can target the same input (files are merged into the input directory)\n- `reuse-executions` only works when the pipeline is run from a Git commit (not `--adhoc`)\n"
}