Update to Valohai Skills

{
  "description": "Migrate data loading and model saving in ML code to use Valohai's input/output system. Use this skill when a user wants to configure data inputs from cloud storage (S3, Azure Blob, GCS), save model outputs to Valohai, replace hardcoded file paths, remove boto3/cloud SDK code, or set up the Valohai file I/O system. Triggers on mentions of inputs, outputs, data loading, model saving, S3, cloud storage, file paths, or Valohai data migration.",
  "included_files": [],
  "name": "valohai-migrate-data",
  "skill_md_contents": "---\nname: valohai-migrate-data\ndescription: Migrate data loading and model saving in ML code to use Valohai's input/output system. Use this skill when a user wants to configure data inputs from cloud storage (S3, Azure Blob, GCS), save model outputs to Valohai, replace hardcoded file paths, remove boto3/cloud SDK code, or set up the Valohai file I/O system. Triggers on mentions of inputs, outputs, data loading, model saving, S3, cloud storage, file paths, or Valohai data migration.\nlicense: Apache-2.0\nmetadata:\n  author: valohai\n  version: \"1.0\"\n  compatibility: Requires Python. Designed for Claude Code or similar AI coding agents.\n---\n\n# Valohai Data I/O Migration\n\nMigrate data loading and model saving to use Valohai's managed input/output system. This eliminates cloud SDK boilerplate, handles authentication automatically, and enables full data lineage tracking.\n\n## Philosophy\n\nValohai separates data access from code. Inputs are declared in `valohai.yaml` with cloud storage URLs. Valohai downloads them to `/valohai/inputs/{name}/` before your code runs. Outputs are saved to `/valohai/outputs/` and automatically uploaded. No boto3, no credentials in code, no download logic.\n\n## Step-by-Step: Inputs\n\n### 1. Identify Data Loading Code\n\nLook for patterns like:\n- `boto3.client('s3')` / `s3.download_file()`\n- `from google.cloud import storage`\n- `azure.storage.blob`\n- `requests.get()` for downloading datasets\n- `wget` or `curl` in shell commands\n- Hardcoded local paths like `/data/train/`, `./datasets/`\n- `pd.read_csv(\"path/to/data.csv\")`\n- `torch.load(\"model.pth\")` for pretrained models\n\n### Design Principle: Always Set Default Values for Inputs\n\n**IMPORTANT**: Whenever possible, give every input a `default` value so the step can run independently without a pipeline. This lets users test individual steps with `vh execution run step-name --adhoc`. Pipeline edges override defaults at runtime.\n\nLook for default values in the existing code, README, documentation, config files, or data paths already referenced in the project. Only add a default if you can find a real, meaningful value. Do not invent placeholder URLs or dummy paths.\n\n### 2. Define Inputs in valohai.yaml\n\n```yaml\n- step:\n    name: train-model\n    image: tensorflow/tensorflow:2.6.0\n    command:\n      - pip install -r requirements.txt\n      - python train.py\n    inputs:\n      - name: training-data\n        default: s3://my-bucket/datasets/train.csv\n      - name: pretrained-model\n        default: datum://production-model-latest\n        filename: model.h5\n```\n\n#### Input Options\n\n```yaml\ninputs:\n  # Single file\n  - name: config\n    default: s3://bucket/config.json\n\n  # Multiple files with wildcard\n  - name: images\n    default:\n      - s3://bucket/images/*.jpg\n      - s3://bucket/images/*.png\n\n  # Multiple cloud sources\n  - name: data\n    default:\n      - s3://aws-bucket/data.csv\n      - azure://container/data.csv\n      - gs://gcs-bucket/data.csv\n\n  # Rename on download\n  - name: model\n    default: s3://bucket/models/v42-best.h5\n    filename: model.h5\n\n  # Preserve directory structure\n  - name: dataset\n    default: s3://bucket/dataset/**/*.json\n    keep-directories: suffix\n\n  # Optional input (can be empty)\n  - name: checkpoint\n    optional: true\n\n  # Reference a Valohai datum (output from another execution)\n  - name: pretrained\n    default: datum://my-model-alias\n```\n\n#### keep-directories Options\n\n- `none` (default): All files flat in `/valohai/inputs/{name}/`\n- `suffix`: Keeps path after the wildcard root\n- `full`: Keeps the full storage path\n\n### 3. Update Python Code\n\nReplace cloud SDK / download code with simple local file reads:\n\n#### Before (with boto3)\n\n```python\nimport boto3\n\ns3 = boto3.client('s3', aws_access_key_id=KEY, aws_secret_access_key=SECRET)\ns3.download_file('my-bucket', 'datasets/train.csv', '/tmp/train.csv')\ndf = pd.read_csv('/tmp/train.csv')\n```\n\n#### After (Valohai)\n\n```python\nimport pandas as pd\n\n# Files are already downloaded to /valohai/inputs/\ndf = pd.read_csv(\"/valohai/inputs/training-data/train.csv\")\n```\n\n#### Reading Multiple Input Files\n\n```python\nimport os\nimport glob\n\n# All files in the input directory\nimage_dir = \"/valohai/inputs/images/\"\nimage_files = glob.glob(os.path.join(image_dir, \"*.jpg\"))\n\nfor image_path in image_files:\n    process_image(image_path)\n```\n\n#### Loading a Model\n\n```python\nimport tensorflow as tf\n\n# With filename option, the file is always named model.h5\nmodel = tf.keras.models.load_model(\"/valohai/inputs/pretrained-model/model.h5\")\n```\n\n### 4. Override Inputs at Runtime\n\n```shell\n# Override via CLI\nvh execution run train-model --training-data=s3://different-bucket/new-data.csv --adhoc\n\n# Or use the web UI to browse and select files\n```\n\n## Step-by-Step: Outputs\n\n### 1. Identify Output/Save Code\n\nLook for:\n- `model.save(\"model.h5\")` / `torch.save(model, \"model.pth\")`\n- `df.to_csv(\"results.csv\")`\n- `plt.savefig(\"plot.png\")`\n- `joblib.dump(model, \"model.pkl\")`\n- Any code saving files locally\n\n### 2. Update Save Paths\n\nSimply change the output path to `/valohai/outputs/`:\n\n#### Before\n\n```python\nmodel.save(\"model.h5\")\ndf.to_csv(\"results.csv\")\nplt.savefig(\"loss_curve.png\")\n```\n\n#### After\n\n```python\nmodel.save(\"/valohai/outputs/model.h5\")\ndf.to_csv(\"/valohai/outputs/results.csv\")\nplt.savefig(\"/valohai/outputs/loss_curve.png\")\n```\n\nThat's the only change needed. Valohai automatically uploads everything in `/valohai/outputs/` to your configured cloud storage when the execution completes.\n\n### 3. Directory Structure in Outputs\n\nSubdirectories are preserved:\n\n```python\nimport os\n\nos.makedirs(\"/valohai/outputs/models\", exist_ok=True)\nos.makedirs(\"/valohai/outputs/plots\", exist_ok=True)\n\nmodel.save(\"/valohai/outputs/models/best_model.h5\")\nplt.savefig(\"/valohai/outputs/plots/training_curve.png\")\n```\n\n### 4. Live Upload During Training (Checkpoints)\n\nFiles marked read-only are uploaded immediately, without waiting for execution to end:\n\n```python\nimport os\nfrom stat import S_IREAD, S_IRGRP, S_IROTH\n\nfor epoch in range(epochs):\n    # Training...\n    if epoch % 10 == 0:\n        path = f\"/valohai/outputs/checkpoint_epoch_{epoch}.pt\"\n        torch.save(model.state_dict(), path)\n        os.chmod(path, S_IREAD | S_IRGRP | S_IROTH)  # Triggers immediate upload\n```\n\n### 5. Package Large Numbers of Output Files\n\n**IMPORTANT**: If your code produces thousands of individual files (e.g., preprocessed images, tokenized text shards, tile crops), **package them into a single archive** before writing to `/valohai/outputs/`. Do NOT write thousands of small files individually.\n\nValohai makes a separate HTTPS request with a presigned URL for every file it uploads and downloads. With tens of thousands of small files, you'll spend more time on HTTP overhead than actual data transfer.\n\nUse `tar` (without compression) to bundle files — this keeps packaging fast while eliminating the per-file HTTP overhead:\n\n```python\nimport tarfile\nimport os\n\n# WRONG - 20,000 individual files = 20,000 HTTPS requests on download\nfor i, image in enumerate(processed_images):\n    save_image(image, f\"/valohai/outputs/image_{i}.png\")\n\n# CORRECT - 1 tar archive = 1 HTTPS request on download\noutput_dir = \"/tmp/processed_images\"\nos.makedirs(output_dir, exist_ok=True)\nfor i, image in enumerate(processed_images):\n    save_image(image, f\"{output_dir}/image_{i}.png\")\n\n# Package without compression (fast, no CPU overhead)\nwith tarfile.open(\"/valohai/outputs/processed_images.tar\", \"w\") as tar:\n    tar.add(output_dir, arcname=\"processed_images\")\n```\n\n**Rule of thumb**: If you're writing more than ~1,000 files to outputs, tar them up. A few large files are always faster than many small ones on Valohai.\n\n### 6. No Need to Declare Outputs in YAML\n\nYou do **not** need to add an `outputs` section to `valohai.yaml`. Any file your code writes to `/valohai/outputs/` is automatically captured and uploaded. Just save files there and Valohai handles the rest.\n\n## Validate the YAML\n\n**IMPORTANT**: After adding inputs to `valohai.yaml`, always run `vh lint` to validate:\n\n```shell\nvh lint\n```\n\nThis catches issues like invalid input names, bad URL formats in defaults, and YAML syntax errors. Fix any errors before running.\n\n## Common Migration Patterns\n\n### Full Training Script Migration\n\n```python\nimport argparse\nimport json\nimport os\nimport pandas as pd\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\nimport joblib\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"--n_estimators\", type=int, default=100)\nparser.add_argument(\"--max_depth\", type=int, default=10)\nargs = parser.parse_args()\n\n# Load data from Valohai inputs\ntrain_df = pd.read_csv(\"/valohai/inputs/training-data/train.csv\")\ntest_df = pd.read_csv(\"/valohai/inputs/test-data/test.csv\")\n\nX_train, y_train = train_df.drop(\"target\", axis=1), train_df[\"target\"]\nX_test, y_test = test_df.drop(\"target\", axis=1), test_df[\"target\"]\n\n# Train model\nmodel = RandomForestClassifier(n_estimators=args.n_estimators, max_depth=args.max_depth)\nmodel.fit(X_train, y_train)\n\n# Evaluate and log metrics\naccuracy = accuracy_score(y_test, model.predict(X_test))\nprint(json.dumps({\"accuracy\": round(accuracy, 4)}))\n\n# Save model to Valohai outputs\njoblib.dump(model, \"/valohai/outputs/model.pkl\")\n```\n\n### Using datum:// References in Pipelines\n\nOutputs from one execution can be referenced by another using `datum://` aliases:\n\n```yaml\ninputs:\n  - name: model\n    default: datum://production-model-latest\n```\n\nThis references a specific versioned file by its alias, enabling reproducible pipeline connections.\n\n## Creating Datasets with JSONL Metadata\n\nValohai uses a special metadata file (`valohai.metadata.jsonl`) to create and version datasets. Write this file to `/valohai/outputs/` alongside your output files.\n\n### Before Creating a Dataset Version: Check Existing Versions\n\n**IMPORTANT**: When creating a dataset version that builds on a previous version (using `from`), **always confirm with the user** which dataset version or alias to base it on. Do not guess or hardcode version numbers.\n\nUse the CLI to fetch available data and aliases in the project:\n\n```shell\n# List data files in the project (shows datum URLs)\nvh data list\n\n# List aliases (shows named references like \"production\", \"latest\")\nvh alias list\n```\n\nAsk the user:\n1. Which dataset and version to base the new version on (e.g., `dataset://my-dataset/v2`)\n2. Or which alias to use (e.g., `dataset://my-dataset/production`)\n3. Or whether to skip basing it on a previous version entirely and create a fresh dataset\n\n### Basic Dataset Creation\n\n```python\nimport json\n\n# Save your output files to /valohai/outputs/ as usual\ndf.to_csv(\"/valohai/outputs/processed_data.csv\")\n\n# Create the metadata file to register it as a dataset version\nmetadata = {\n    \"processed_data.csv\": {\n        \"valohai.dataset-versions\": [\"dataset://my-dataset/v1\"],\n    },\n}\n\nwith open(\"/valohai/outputs/valohai.metadata.jsonl\", \"w\") as f:\n    for file_name, file_metadata in metadata.items():\n        json.dump({\"file\": file_name, \"metadata\": file_metadata}, f)\n        f.write(\"\\n\")\n```\n\n### Dataset with Multiple Files\n\n```python\nimport json\n\nmetadata = {}\n\n# Save multiple files and register them all in the same dataset version\nfor split in [\"train\", \"val\", \"test\"]:\n    filename = f\"{split}.csv\"\n    df.to_csv(f\"/valohai/outputs/{filename}\")\n    metadata[filename] = {\n        \"valohai.dataset-versions\": [\"dataset://my-dataset/v1\"],\n    }\n\nwith open(\"/valohai/outputs/valohai.metadata.jsonl\", \"w\") as f:\n    for file_name, file_metadata in metadata.items():\n        json.dump({\"file\": file_name, \"metadata\": file_metadata}, f)\n        f.write(\"\\n\")\n```\n\n### Incremental Dataset Updates\n\nCreate a new version based on a previous one, adding or excluding files:\n\n```python\nimport json\n\nmetadata = {\n    \"new_data.csv\": {\n        \"valohai.dataset-versions\": [\n            {\n                \"uri\": \"dataset://my-dataset/v3\",\n                \"from\": \"dataset://my-dataset/v2\",\n                \"start_fresh\": False,\n                \"exclude\": [\"bad_file.csv\", \"old_file.csv\"],\n            },\n        ],\n    },\n}\n\nwith open(\"/valohai/outputs/valohai.metadata.jsonl\", \"w\") as f:\n    for file_name, file_metadata in metadata.items():\n        json.dump({\"file\": file_name, \"metadata\": file_metadata}, f)\n        f.write(\"\\n\")\n```\n\n### Dataset with Aliases\n\nAliases let you reference datasets by name (e.g., `production`, `latest`) instead of version number:\n\n```python\nimport json\n\nmetadata = {\n    \"model.pkl\": {\n        \"valohai.dataset-versions\": [\n            {\n                \"uri\": \"dataset://my-models/v5\",\n                \"from\": \"dataset://my-models/v4\",\n                \"targeting_aliases\": [\"production\", \"stable\"],\n            },\n        ],\n    },\n}\n\nwith open(\"/valohai/outputs/valohai.metadata.jsonl\", \"w\") as f:\n    for file_name, file_metadata in metadata.items():\n        json.dump({\"file\": file_name, \"metadata\": file_metadata}, f)\n        f.write(\"\\n\")\n```\n\n### Packaged Datasets (Large File Collections)\n\nFor datasets with 10,000+ small files, use packaging for 10-100x faster downloads:\n\n```python\nimport json\n\nmetadata = {\n    \"images.tar\": {\n        \"valohai.dataset-versions\": [\n            {\n                \"uri\": \"dataset://my-images/train-v1\",\n                \"packaging\": True,\n            },\n        ],\n    },\n}\n\nwith open(\"/valohai/outputs/valohai.metadata.jsonl\", \"w\") as f:\n    for file_name, file_metadata in metadata.items():\n        json.dump({\"file\": file_name, \"metadata\": file_metadata}, f)\n        f.write(\"\\n\")\n```\n\n### Consuming Datasets as Inputs\n\nReference datasets in `valohai.yaml` inputs using the `dataset://` URI:\n\n```yaml\ninputs:\n  - name: training-data\n    default: dataset://my-dataset/latest\n  - name: model\n    default: dataset://my-models/production\n```\n\n## Execution Environment Reference\n\nEvery Valohai execution has:\n- `/valohai/repository/` - Your code (working directory)\n- `/valohai/inputs/{input-name}/` - Downloaded input files (read-only)\n- `/valohai/outputs/` - Write output files here\n- `/valohai/config/parameters.json` - Auto-generated parameter config\n\n## Edge Cases\n\n- Input directories are read-only - never write to `/valohai/inputs/`\n- Input names in YAML map to directory names under `/valohai/inputs/`\n- If multiple files have the same name across different default URLs, they overwrite each other\n- When running locally (not on Valohai), `/valohai/inputs/` won't exist - consider adding a fallback path for local development\n- No file size limits on outputs\n- Any file type is supported for both inputs and outputs\n- Wildcard inputs: all matching files are downloaded to the same directory\n"
}