{"id":53769,"plugin_id":"plugins_6ac012b82680819195f8a26cbe32c246","kind":"skill","collection_source":"skill_api","comparison_source":null,"observed_at":"2026-10-24T13:55:01.474Z","digest":"b8cb23b745c74fe7309c29912bf1810e9700815b00ceb48f73b1de5a682ee084","against":null,"payload":{"description":"Monitor deployed models in Domino including drift detection, model quality tracking, and alerting. Covers data drift analysis, prediction capture, baseline comparison, alert configuration, and remediation workflows. Use when monitoring production models, detecting drift, or setting up model health alerts.","included_files":[],"name":"domino-model-monitoring","skill_md_contents":"---\nname: domino-model-monitoring\ndescription: Monitor deployed models in Domino including drift detection, model quality tracking, and alerting. Covers data drift analysis, prediction capture, baseline comparison, alert configuration, and remediation workflows. Use when monitoring production models, detecting drift, or setting up model health alerts.\n---\n\n# Domino Model Monitoring Skill\n\n## Description\nThis skill helps users monitor deployed models in Domino, including drift detection, model quality tracking, and alerting.\n\n## Activation\nActivate this skill when users want to:\n- Monitor deployed model performance\n- Set up drift detection\n- Configure monitoring alerts\n- Analyze prediction data\n- Understand model degradation\n\n## What is Model Monitoring?\n\nDomino Model Monitoring provides:\n- **Data Drift Detection**: Detect changes in input data distributions\n- **Model Quality Tracking**: Monitor prediction accuracy over time\n- **Alerting**: Get notified when metrics exceed thresholds\n- **Prediction Capture**: Log predictions for analysis\n- **Reproducibility**: Diagnose issues with captured data\n\n## Setting Up Monitoring\n\n### Prerequisites\n1. Deployed Model API in Domino\n2. Training dataset (for baseline)\n3. Ground truth data (optional, for quality metrics)\n\n### Enable Monitoring\n1. Go to your Model API page\n2. Click **Monitoring** tab\n3. Click **Set Up Monitoring**\n4. Upload training dataset\n5. Configure drift detection settings\n\n### Register Training Data\n```python\n# Training data provides baseline for drift detection\n# Upload via UI or programmatically\n\nimport pandas as pd\n\n# Your training data\ntrain_df = pd.read_csv(\"training_data.csv\")\n\n# Save for monitoring setup\ntrain_df.to_csv(\"/mnt/artifacts/training_data.csv\", index=False)\n```\n\n## Drift Detection\n\n### Types of Drift\n\n| Drift Type | Description |\n|------------|-------------|\n| **Data Drift** | Input feature distributions change |\n| **Concept Drift** | Relationship between inputs and outputs changes |\n| **Prediction Drift** | Output distribution changes |\n\n### Statistical Tests\n\nDomino supports multiple drift detection tests:\n\n| Test | Best For |\n|------|----------|\n| **Kullback-Leibler Divergence** | General-purpose, most common |\n| **Population Stability Index (PSI)** | Finance industry standard |\n| **Wasserstein Distance** | Comparing distributions |\n| **Energy Distance** | Multivariate distributions |\n\n### Configure Drift Detection\n\n1. Go to Model API > **Monitoring**\n2. Click **Configure Drift Detection**\n3. For each feature:\n   - Select test type\n   - Set threshold\n   - Enable/disable alerts\n\n### Example Thresholds\n\n| Test | Low Drift | Medium Drift | High Drift |\n|------|-----------|--------------|------------|\n| KL Divergence | < 0.1 | 0.1 - 0.2 | > 0.2 |\n| PSI | < 0.1 | 0.1 - 0.25 | > 0.25 |\n\n## Prediction Capture\n\n### How It Works\nDomino automatically captures predictions:\n1. Model receives request\n2. Prediction is made\n3. Input/output logged to dataset\n4. Data available for drift analysis\n\n### Access Captured Data\n```python\nimport pandas as pd\n\n# Predictions captured in Domino Dataset\npredictions_df = pd.read_parquet(\n    \"/mnt/data/model-predictions/predictions.parquet\"\n)\n\nprint(predictions_df.head())\n```\n\n### Capture Frequency\n- Predictions batched hourly\n- Full data available in monitoring dataset\n- Retention configurable by admin\n\n## Model Quality Monitoring\n\n### With Ground Truth\nIf you provide ground truth labels:\n\n```python\n# Upload ground truth\nground_truth = pd.DataFrame({\n    \"prediction_id\": [...],\n    \"actual_label\": [...]\n})\n\n# Upload to monitoring\nground_truth.to_csv(\"/mnt/artifacts/ground_truth.csv\", index=False)\n```\n\n### Quality Metrics\n- Accuracy\n- Precision/Recall\n- F1 Score\n- AUC-ROC\n- Mean Squared Error (regression)\n\n### Schedule Quality Checks\n1. Go to Monitoring > **Quality**\n2. Upload ground truth dataset\n3. Configure metric thresholds\n4. Set check frequency\n\n## Alerting\n\n### Configure Alerts\n1. Go to Model API > **Monitoring**\n2. Click **Alerts**\n3. Configure:\n   - Metric to monitor\n   - Threshold\n   - Alert recipients (email)\n\n### Alert Types\n- Drift threshold exceeded\n- Quality metric below threshold\n- Model API health issues\n- Prediction volume anomalies\n\n### Disable Noisy Alerts\nClick the bell icon next to features to exclude from alerts.\n\n## Viewing Monitoring Data\n\n### Monitoring Dashboard\nGo to Model API > **Monitoring** to see:\n- Drift trends over time\n- Feature distributions\n- Quality metrics\n- Alert history\n\n### Export Data\n```python\n# Export monitoring data for custom analysis\nimport pandas as pd\n\ndrift_report = pd.read_csv(\"/mnt/data/monitoring/drift_report.csv\")\nprint(drift_report)\n```\n\n## Responding to Drift\n\n### Investigation Workflow\n1. **Alert received**: Drift detected on feature X\n2. **Investigate**: View feature distribution changes\n3. **Diagnose**: Compare current vs training data\n4. **Action**: Retrain or update model\n\n### Retrain Model\n```python\n# When drift is detected, retrain with recent data\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Load recent data\nrecent_data = pd.read_csv(\"/mnt/data/recent_predictions.csv\")\n\n# Combine with ground truth\ntraining_data = merge_with_ground_truth(recent_data)\n\n# Retrain\nmodel = RandomForestClassifier()\nmodel.fit(training_data[features], training_data[label])\n\n# Deploy new version\njoblib.dump(model, \"/mnt/artifacts/model_v2.joblib\")\n```\n\n### Automated Retraining\nSet up scheduled job to retrain when drift detected:\n```python\n# scheduled_retrain.py\nfrom domino import Domino\n\ndomino = Domino(\"project/model-project\")\n\n# Check drift status\ndrift_status = check_drift_metrics()\n\nif drift_status[\"max_drift\"] > 0.2:\n    # Trigger retrain job\n    domino.runs_start(\n        command=\"python retrain.py\",\n        hardware_tier_name=\"medium\"\n    )\n```\n\n## Best Practices\n\n### 1. Baseline with Quality Data\nUse clean, representative training data for baseline.\n\n### 2. Monitor Key Features\nFocus on features with highest importance:\n```python\n# Identify important features\nimportances = model.feature_importances_\ntop_features = sorted(\n    zip(feature_names, importances),\n    key=lambda x: x[1],\n    reverse=True\n)[:10]\n```\n\n### 3. Set Appropriate Thresholds\n- Start with conservative thresholds\n- Adjust based on business impact\n- Different thresholds for different features\n\n### 4. Include Business Context\nNot all drift requires action:\n- Seasonal variations may be expected\n- New customer segments may cause drift\n- Consider business impact before reacting\n\n### 5. Regular Reviews\nSchedule periodic monitoring reviews:\n- Weekly: Check drift trends\n- Monthly: Review alert configurations\n- Quarterly: Assess model performance\n\n## Troubleshooting\n\n### No Data in Monitoring\n- Verify Model API is receiving traffic\n- Check prediction capture is enabled\n- Wait for hourly batch processing\n\n### Drift Always High\n- Review training data quality\n- Check for data preprocessing differences\n- Verify feature encoding consistency\n\n### Alerts Not Sending\n- Check email configuration\n- Verify alert thresholds\n- Review spam folders\n\n## Documentation Reference\n- [Drift detection for monitored models](https://docs.dominodatalab.com/en/latest/user_guide/86bc1f/drift-detection-for-monitored-models/)\n- [Domino endpoint drift detection](https://docs.dominodatalab.com/en/latest/user_guide/c97091/domino-endpoint-drift-detection/)\n- [Monitor workflows](https://docs.dominodatalab.com/en/latest/user_guide/85f76d/monitor-workflows/)\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}