Update to DataRobot Skills

{
  "description": "Tools and guidance for monitoring model performance, tracking data drift, managing model health, and detecting prediction anomalies. Use when monitoring deployed models, tracking drift, or investigating prediction anomalies.",
  "included_files": [],
  "name": "datarobot-model-monitoring",
  "skill_md_contents": "---\nname: datarobot-model-monitoring\ndescription: Tools and guidance for monitoring model performance, tracking data drift, managing model health, and detecting prediction anomalies. Use when monitoring deployed models, tracking drift, or investigating prediction anomalies.\n---\n\n# DataRobot Model Monitoring Skill\n\nThis skill provides comprehensive guidance for monitoring deployed models, tracking performance metrics, detecting data drift, and managing model health.\n\n## Quick Start\n\n**Most common use case**: Check deployment health and data drift\n\n1. **Check service stats**: `deployment.get_service_stats(...)` to review prediction volume/latency\n2. **Check drift**: `deployment.get_feature_drift(...)` / `deployment.get_target_drift(...)`\n3. **Compare over time**: Use `get_service_stats_over_time(...)` and drift periods to assess trends\n\n**Example**: \"Check the health of deployment abc123 and report any data drift issues\"\n\n## When to use this skill\n\nUse this skill when you need to:\n- Monitor model performance in production\n- Track data drift and feature drift\n- Detect prediction anomalies\n- Monitor prediction accuracy over time\n- Set up alerts for model degradation\n- Analyze model health metrics\n- Compare production performance to training performance\n\n## Key capabilities\n\n### 1. Performance Monitoring\n\n- Track prediction accuracy and metrics over time\n- Compare production metrics to training metrics\n- Monitor prediction volume and latency\n- Identify performance degradation trends\n\n### 2. Data Drift Detection\n\n- Detect changes in feature distributions\n- Identify feature drift (statistical changes)\n- Monitor target drift (if actuals available)\n- Alert on significant drift events\n\n### 3. Prediction Monitoring\n\n- Monitor prediction distributions\n- Detect prediction anomalies\n- Track prediction confidence scores\n- Identify unusual prediction patterns\n\n### 4. Health Management\n\n- Assess overall model health\n- Generate monitoring reports\n- Set up automated alerts\n- Manage model retraining triggers\n\n## Workflow examples\n\n### Example 1: Check model health and drift\n\n**User request**: \"Check the health of deployment abc123 and report any data drift issues.\"\n\n**Agent workflow**:\n1. Get deployment monitoring status\n2. Retrieve recent performance metrics\n3. Check for data drift in key features\n4. Compare current metrics to baseline (training)\n5. Identify any significant drift or degradation\n6. Report findings with recommendations\n\n### Example 2: Set up drift monitoring alerts\n\n**User request**: \"Set up alerts for deployment xyz789 to notify when feature drift exceeds 0.2.\"\n\n**Agent workflow**:\n1. Get deployment configuration\n2. Configure drift threshold (0.2)\n3. Set up alert notifications\n4. Specify which features to monitor\n5. Test alert configuration\n6. Confirm monitoring is active\n\n## Using DataRobot SDK\n\nThis skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:\n\n```bash\npip install datarobot\n```\n\n### Key SDK Operations\n\nUse these DataRobot SDK and MLOps API methods for monitoring:\n\n**Deployment Monitoring**:\n- `deployment.get_service_stats(...)` - Get service statistics (latency, volume, etc.)\n- `deployment.get_feature_drift(...)` - Get feature drift metrics (returns `FeatureDrift` objects)\n- `deployment.get_target_drift(...)` - Get target drift metrics (returns `TargetDrift`)\n- `deployment.get_prediction_results(...)` - Retrieve recorded prediction results (if enabled)\n\n**Model Performance**:\n- `model.metrics` - Model performance metrics (dict of `{metric: {partition: score}}`)\n- `model.get_roc_curve()` - Get ROC curve for comparison\n\n**Note**: Some monitoring features may require DataRobot MLOps API. See the [Common Patterns](#common-patterns) section below for examples.\n\n## Best practices\n\n1. **Regular monitoring**: Check model health regularly, not just when issues arise\n2. **Baseline comparison**: Always compare production metrics to training baseline\n3. **Drift thresholds**: Set appropriate drift thresholds based on your domain\n4. **Key features**: Focus monitoring on high-importance features\n5. **Automated alerts**: Set up alerts for critical issues\n6. **Historical analysis**: Track trends over time, not just point-in-time metrics\n\n## Common patterns\n\n### Pattern 1: Health check\n```python\nimport datarobot as dr\n\n# Initialize client\ndr.Client()\n\n# Get deployment\ndeployment = dr.Deployment.get(\"abc123\")\n\n# Get service stats (requires MLOps monitoring to be enabled)\n# ServiceStats exposes values via the .metrics dict, not as attributes.\nstats = deployment.get_service_stats()\nprint(f\"Prediction count: {stats.metrics['totalPredictions']}\")\nprint(f\"Mean response time (ms): {stats.metrics['responseTime']}\")\n\n# Get recorded prediction results (if available / enabled)\ntry:\n    recent = deployment.get_prediction_results(limit=10)\n    print(f\"Recent prediction results: {len(recent)}\")\nexcept Exception as e:\n    print(f\"Prediction results not available: {e}\")\n```\n\n### Pattern 2: Drift detection\n```python\nimport datarobot as dr\n\n# Get deployment\ndeployment = dr.Deployment.get(\"abc123\")\n\n# Get feature drift (requires MLOps monitoring)\ntry:\n    drifts = deployment.get_feature_drift()\n    high = [d for d in drifts if (d.drift_score or 0) > 0.2]\n    print(f\"Features with drift_score > 0.2: {len(high)}\")\n    for d in high[:10]:\n        print(f\"{d.name}: {d.drift_score}\")\nexcept Exception as e:\n    print(f\"Feature drift requires MLOps monitoring: {e}\")\n```\n\n## Monitoring metrics\n\n### Performance Metrics\n- **Accuracy**: Prediction accuracy (classification)\n- **RMSE/MAE**: Prediction error (regression)\n- **AUC**: Model discrimination (classification)\n- **Prediction volume**: Number of predictions made\n\n### Drift Metrics\n- **Feature drift**: Statistical changes in feature distributions\n- **Target drift**: Changes in target distribution (if available)\n- **Prediction drift**: Changes in prediction distributions\n- **Drift score**: Overall drift severity (0-1 scale)\n\n## Alert thresholds\n\nRecommended thresholds:\n\n- **High drift**: > 0.3 (significant changes, investigate immediately)\n- **Medium drift**: 0.15-0.3 (moderate changes, monitor closely)\n- **Low drift**: < 0.15 (minor changes, normal variation)\n\nAdjust thresholds based on your domain and use case sensitivity.\n\n## Model health status\n\n- **Healthy**: Performance within expected range, minimal drift\n- **Degrading**: Performance declining, some drift detected\n- **Unhealthy**: Significant performance issues or high drift\n- **Unknown**: Insufficient data for assessment\n\n## Error handling\n\nCommon errors and solutions:\n\n- **Insufficient data**: Need minimum prediction volume for monitoring\n- **Baseline unavailable**: Ensure training baseline is available\n- **Access issues**: Verify deployment permissions and access\n\n## SDK Setup\n\n### Install DataRobot SDK\n\n```bash\npip install datarobot\n```\n\n### Initialize Client\n\n```python\nimport datarobot as dr\n\ndr.Client()\n```\n\n**Note**: Some monitoring features require DataRobot MLOps API access. Check your DataRobot plan for MLOps availability.\n\n## Resources\n\n- [DataRobot Python SDK Documentation](https://datarobot-public-api-client.readthedocs-hosted.com/)\n- [DataRobot Model Monitoring Documentation](https://docs.datarobot.com/en/docs/mlops/monitor/index.html)\n\n"
}