Update to Valohai Skills

{
  "description": "Add metrics and metadata tracking to ML code for Valohai. Use this skill when a user wants to track training metrics (loss, accuracy, F1, etc.), log experiment metadata, enable real-time metric visualization, or compare experiments in Valohai. Triggers on mentions of metrics, metadata, experiment tracking, logging accuracy/loss, or Valohai metric migration.",
  "included_files": [],
  "name": "valohai-migrate-metrics",
  "skill_md_contents": "---\nname: valohai-migrate-metrics\ndescription: Add metrics and metadata tracking to ML code for Valohai. Use this skill when a user wants to track training metrics (loss, accuracy, F1, etc.), log experiment metadata, enable real-time metric visualization, or compare experiments in Valohai. Triggers on mentions of metrics, metadata, experiment tracking, logging accuracy/loss, or Valohai metric migration.\nlicense: Apache-2.0\nmetadata:\n  author: valohai\n  version: \"1.0\"\n  compatibility: Requires Python. Designed for Claude Code or similar AI coding agents.\n---\n\n# Valohai Metrics/Metadata Migration\n\nAdd metrics tracking to ML code so Valohai automatically captures, visualizes, and enables comparison across experiments. No special libraries required - just print JSON to stdout.\n\n## Philosophy\n\nValohai captures metrics by detecting JSON printed to stdout during execution. This is deliberately simple and framework-agnostic. No SDK imports, no decorators, no special API calls. Just `print(json.dumps({...}))`.\n\n## Step-by-Step Instructions\n\n### 1. Identify Metrics to Track\n\nScan the user's ML code for values worth tracking:\n\n- **Training metrics**: loss, accuracy, precision, recall, F1 score, AUC-ROC\n- **Training dynamics**: learning rate (if scheduled), gradient norm, batch processing time\n- **Validation metrics**: val_loss, val_accuracy, val_f1 (per epoch or interval)\n- **Resource metrics**: GPU utilization, memory usage, throughput (samples/sec)\n- **Final results**: best model score, total training time, convergence epoch\n- **Custom KPIs**: any domain-specific metric the user cares about\n\n### 2. Add JSON Printing to Code\n\nThe core pattern is simple - print a JSON dictionary to stdout:\n\n```python\nimport json\n\n# Log metrics at any point in your code\nprint(json.dumps({\"accuracy\": 0.92, \"loss\": 0.08}))\n```\n\n**CRITICAL: Group all metrics from the same moment into a single `json.dumps()` call.** Each `print(json.dumps(...))` creates one metadata event with one timestamp. If you print metrics separately, Valohai treats them as disconnected events and they cannot be correlated.\n\n```python\n# WRONG - 4 disconnected events, can't be correlated or plotted together\nprint(json.dumps({\"inference_time_s\": 0.45}))\nprint(json.dumps({\"num_detections\": 6}))\nprint(json.dumps({\"confidence_threshold\": 0.25}))\nprint(json.dumps({\"iou_threshold\": 0.7}))\n\n# CORRECT - 1 event, all metrics linked together\nprint(json.dumps({\n    \"inference_time_s\": 0.45,\n    \"num_detections\": 6,\n    \"confidence_threshold\": 0.25,\n    \"iou_threshold\": 0.7,\n}))\n```\n\nSame rule applies to training loops - one epoch = one `json.dumps()`:\n\n```python\n# WRONG\nprint(json.dumps({\"epoch\": epoch}))\nprint(json.dumps({\"train_loss\": train_loss}))\nprint(json.dumps({\"val_accuracy\": val_acc}))\n\n# CORRECT\nprint(json.dumps({\n    \"epoch\": epoch,\n    \"train_loss\": train_loss,\n    \"val_accuracy\": val_acc,\n}))\n```\n\nValohai automatically:\n- Captures every JSON line printed to stdout\n- Adds a UTC timestamp\n- Makes values searchable, sortable, and plottable\n- Enables real-time visualization during execution\n\n### 3. Common Integration Patterns\n\n#### Training Loop (Most Common)\n\n```python\nimport json\n\nfor epoch in range(epochs):\n    train_loss = train_one_epoch(model, train_loader, optimizer)\n    val_loss, val_acc = validate(model, val_loader)\n\n    print(json.dumps({\n        \"epoch\": epoch,\n        \"train_loss\": train_loss,\n        \"val_loss\": val_loss,\n        \"val_accuracy\": val_acc,\n    }))\n```\n\n#### Batch-Level Logging\n\n```python\nimport json\n\nfor epoch in range(epochs):\n    for batch_idx, (data, target) in enumerate(train_loader):\n        loss = train_step(model, data, target, optimizer)\n\n        if batch_idx % 100 == 0:  # Log every N batches to stay under 50 events/sec\n            print(json.dumps({\n                \"epoch\": epoch,\n                \"batch\": batch_idx,\n                \"loss\": loss.item(),\n            }))\n```\n\n#### Multiple Phases with Context\n\n```python\nimport json\n\n# Training phase\nfor epoch in range(epochs):\n    train_metrics = train_epoch(model, train_loader)\n    print(json.dumps({\n        \"epoch\": epoch,\n        \"phase\": \"training\",\n        \"loss\": train_metrics[\"loss\"],\n        \"accuracy\": train_metrics[\"accuracy\"],\n    }))\n\n    # Validation phase\n    val_metrics = validate(model, val_loader)\n    print(json.dumps({\n        \"epoch\": epoch,\n        \"phase\": \"validation\",\n        \"loss\": val_metrics[\"loss\"],\n        \"accuracy\": val_metrics[\"accuracy\"],\n    }))\n```\n\n#### Final Summary Metrics\n\n```python\nimport json\n\n# After training completes\nprint(json.dumps({\n    \"best_val_accuracy\": best_accuracy,\n    \"best_epoch\": best_epoch,\n    \"total_training_time_seconds\": elapsed,\n    \"final_train_loss\": final_loss,\n}))\n```\n\n### 4. Framework-Specific Examples\n\n#### PyTorch\n\n```python\nimport json\nimport time\n\nfor epoch in range(args.epochs):\n    model.train()\n    running_loss = 0.0\n    correct = 0\n    total = 0\n\n    for batch_idx, (inputs, targets) in enumerate(train_loader):\n        inputs, targets = inputs.to(device), targets.to(device)\n        optimizer.zero_grad()\n        outputs = model(inputs)\n        loss = criterion(outputs, targets)\n        loss.backward()\n        optimizer.step()\n\n        running_loss += loss.item()\n        _, predicted = outputs.max(1)\n        total += targets.size(0)\n        correct += predicted.eq(targets).sum().item()\n\n    train_loss = running_loss / len(train_loader)\n    train_acc = correct / total\n\n    # Validation\n    model.eval()\n    val_loss, val_acc = evaluate(model, val_loader, criterion, device)\n\n    print(json.dumps({\n        \"epoch\": epoch,\n        \"train_loss\": round(train_loss, 4),\n        \"train_accuracy\": round(train_acc, 4),\n        \"val_loss\": round(val_loss, 4),\n        \"val_accuracy\": round(val_acc, 4),\n        \"learning_rate\": optimizer.param_groups[0][\"lr\"],\n    }))\n```\n\n#### TensorFlow/Keras (Custom Callback)\n\n```python\nimport json\nimport tensorflow as tf\n\nclass ValohaiMetricsCallback(tf.keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        if logs:\n            metrics = {\"epoch\": epoch}\n            metrics.update({k: round(float(v), 4) for k, v in logs.items()})\n            print(json.dumps(metrics))\n\nmodel.fit(\n    x_train, y_train,\n    epochs=args.epochs,\n    batch_size=args.batch_size,\n    validation_data=(x_val, y_val),\n    callbacks=[ValohaiMetricsCallback()],\n)\n```\n\n#### scikit-learn\n\n```python\nimport json\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n\nmodel.fit(X_train, y_train)\ny_pred = model.predict(X_test)\n\nprint(json.dumps({\n    \"accuracy\": round(accuracy_score(y_test, y_pred), 4),\n    \"precision\": round(precision_score(y_test, y_pred, average=\"weighted\"), 4),\n    \"recall\": round(recall_score(y_test, y_pred, average=\"weighted\"), 4),\n    \"f1_score\": round(f1_score(y_test, y_pred, average=\"weighted\"), 4),\n}))\n```\n\n#### XGBoost / LightGBM\n\n```python\nimport json\n\ndef valohai_callback(env):\n    \"\"\"Custom callback to log metrics to Valohai.\"\"\"\n    # Collect all metrics into one event per iteration\n    metrics = {\"iteration\": env.iteration}\n    for item in env.evaluation_result_list:\n        metrics[item[0]] = round(item[1], 4)\n    print(json.dumps(metrics))\n\nmodel = xgb.train(\n    params, dtrain,\n    num_boost_round=100,\n    evals=[(dtrain, \"train\"), (dval, \"val\")],\n    callbacks=[valohai_callback],\n)\n```\n\n### 5. What Valohai Does With Metrics\n\n- **Execution table**: Sort and filter executions by any metric value\n- **Time-series charts**: Plot metrics over epochs/steps, updated in real-time during training\n- **Multi-execution comparison**: Overlay metrics from multiple runs on the same chart\n- **CSV/JSON export**: Download metric data for external analysis\n- **Pipeline conditions**: Use metrics to control pipeline flow (e.g., stop if accuracy > threshold)\n\n## Best Practices\n\n1. **One event = one `json.dumps()`** - all metrics from the same moment MUST be in a single print. Separate prints create disconnected events that can't be correlated\n2. **Log progressively** throughout training, not just final results - enables real-time monitoring\n3. **Use consistent metric names** across experiments for meaningful comparison\n4. **Include a step/epoch counter** as a metric for proper time-series alignment\n5. **Round floating-point values** to 4-6 decimal places to keep logs readable\n6. **Print to stdout** (not stderr) - Valohai only captures JSON from stdout\n7. **Ensure valid JSON** - use `json.dumps()` rather than manual string formatting\n8. **Add context fields** like `phase: \"training\"` or `phase: \"validation\"` to distinguish metrics\n9. **Log at reasonable intervals** - every epoch is good; every batch may be too noisy unless filtered\n10. **Stay under 50 events/second** - Valohai enforces a rate limit of 500 JSON events per 10 seconds (50/s). Exceeding this triggers a warning and **events will be dropped silently**. If logging per-batch metrics, add a frequency filter (e.g., every N batches) to stay well under this limit\n\n## Edge Cases\n\n- Non-JSON stdout lines are ignored by Valohai (treated as regular log output)\n- Multiple JSON prints per line: only the first valid JSON object is captured\n- Nested JSON objects are flattened for display in the UI\n- String values in metrics are supported (e.g., `\"best_model\": \"epoch_42\"`)\n- Boolean values are supported\n- Metrics can be used in pipeline edge conditions: `metadata.accuracy >= 0.9`\n- If using `print()` with frameworks that also print to stdout, the JSON lines are still correctly identified\n- **Rate limit**: More than 500 JSON events per 10 seconds triggers `More than 50.0 events per second are being written to stdout; some are ignored.` — dropped events are lost permanently. Use batch-level filtering (`if batch_idx % N == 0`) to control output rate\n"
}