Update to LaunchDarkly

{
  "name": "custom-metrics",
  "description": "Create, track, retrieve, update, and delete custom business metrics for configs. Covers full lifecycle: define metric kinds via API, emit events via SDK, and query results.",
  "included_files": [],
  "skill_md_contents": "---\nname: custom-metrics\ndescription: \"Create, track, retrieve, update, and delete custom business metrics for configs. Covers full lifecycle: define metric kinds via API, emit events via SDK, and query results.\"\nlicense: Apache-2.0\ncompatibility: Requires the LaunchDarkly server SDK and a LaunchDarkly API token with the `writer` role for metric management.\nmetadata:\n  author: launchdarkly\n  version: \"1.0.0-experimental\"\n---\n\n# Custom Metrics for Configs\n\nFull lifecycle management of custom business metrics: create metric definitions via API, track events via SDK, retrieve metric data, and manage metrics programmatically.\n\n## Prerequisites\n\n- LaunchDarkly SDK initialized (see `sdk`)\n- LaunchDarkly API token with `writer` role for metric management\n- Understanding of built-in agent metrics (see `built-in-metrics`)\n\n## API Key Detection\n\nBefore prompting the user for an API key, try to detect it automatically:\n\n1. **Check Claude MCP config** - Read `~/.claude/config.json` and look for `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY`\n2. **Check environment variables** - Look for `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, or `LD_API_KEY`\n3. **Prompt user** - Only if detection fails, ask the user for their API key\n\n```python\nimport os\nimport json\nfrom pathlib import Path\n\ndef get_launchdarkly_api_key():\n    \"\"\"Auto-detect LaunchDarkly API key from Claude config or environment.\"\"\"\n    # 1. Check Claude MCP config\n    claude_config = Path.home() / \".claude\" / \"config.json\"\n    if claude_config.exists():\n        try:\n            config = json.load(open(claude_config))\n            api_key = config.get(\"mcpServers\", {}).get(\"launchdarkly\", {}).get(\"env\", {}).get(\"LAUNCHDARKLY_API_KEY\")\n            if api_key:\n                return api_key\n        except (json.JSONDecodeError, IOError):\n            pass\n\n    # 2. Check environment variables\n    for var in [\"LAUNCHDARKLY_API_KEY\", \"LAUNCHDARKLY_API_TOKEN\", \"LD_API_KEY\"]:\n        if os.environ.get(var):\n            return os.environ[var]\n\n    return None\n```\n\n## Metrics Lifecycle Overview\n\n| Step | Method | Purpose |\n|------|--------|---------|\n| 1. Create | API | Define metric in LaunchDarkly |\n| 2. Track | SDK | Send events to the metric |\n| 3. Get | API | Retrieve metric definition/data |\n| 4. Update | API | Modify metric properties |\n| 5. Delete | API | Remove metric |\n\n## 1. Create Metric (API)\n\n**Required fields for numeric custom metrics:**\n- `successCriteria` - Must be one of: `\"HigherThanBaseline\"`, `\"LowerThanBaseline\"`\n- `unit` - e.g., `\"count\"`, `\"percent\"`, `\"milliseconds\"`\n\nThe API will return `400 Bad Request` if these are missing for numeric metrics.\n\n```python\nimport requests\nimport os\n\ndef create_metric(\n    project_key: str,\n    metric_key: str,\n    name: str,\n    kind: str = \"custom\",\n    is_numeric: bool = True,\n    unit: str = \"count\",\n    success_criteria: str = \"HigherThanBaseline\",\n    event_key: str = None,\n    description: str = None\n):\n    \"\"\"Create a new metric definition in LaunchDarkly.\"\"\"\n    API_TOKEN = os.environ.get(\"LAUNCHDARKLY_API_TOKEN\")\n\n    url = f\"https://app.launchdarkly.com/api/v2/metrics/{project_key}\"\n\n    payload = {\n        \"key\": metric_key,\n        \"name\": name,\n        \"kind\": kind,\n        \"isNumeric\": is_numeric,\n        \"eventKey\": event_key or metric_key\n    }\n\n    # Unit and successCriteria are required for numeric custom metrics\n    if is_numeric and kind == \"custom\":\n        payload[\"unit\"] = unit\n        payload[\"successCriteria\"] = success_criteria\n\n    if description:\n        payload[\"description\"] = description\n\n    headers = {\n        \"Authorization\": API_TOKEN,\n        \"Content-Type\": \"application/json\"\n    }\n\n    response = requests.post(url, json=payload, headers=headers)\n\n    if response.status_code == 201:\n        print(f\"[OK] Created metric: {metric_key}\")\n        return response.json()\n    elif response.status_code == 409:\n        print(f\"[INFO] Metric already exists: {metric_key}\")\n        return None\n    else:\n        print(f\"[ERROR] Failed to create metric: {response.status_code}\")\n        print(f\"        {response.text}\")\n        return None\n```\n\n**Metric Kinds:**\n- `custom` - Track any event (most common for agent metrics)\n- `pageview` - Track page views\n- `click` - Track click events\n\n**Success Criteria** (for numeric metrics):\n- `HigherThanBaseline` - Higher values are better (e.g., revenue, satisfaction)\n- `LowerThanBaseline` - Lower values are better (e.g., errors, latency)\n\n**Common Units:**\n- `count` - Generic count\n- `milliseconds` - Time duration\n- `percent` - Percentage values\n- `dollars` - Currency\n\n## 2. Track Events (SDK)\n\nOnce the metric is created, track events using the SDK:\n\n```python\nfrom ldclient import Context\nfrom ldclient.config import Config\nimport ldclient\n\n# Initialize (see sdk for details)\nldclient.set_config(Config(\"your-sdk-key\"))\nld_client = ldclient.get()\n\ndef track_metric(ld_client, user_id: str, metric_key: str, value: float, data: dict = None):\n    \"\"\"Track an event to a metric.\"\"\"\n    context = Context.builder(user_id).build()\n\n    ld_client.track(\n        metric_key,\n        context,\n        data=data,\n        metric_value=value\n    )\n```\n\n### Common Tracking Patterns\n\n```python\ndef track_conversion(ld_client, user_id: str, amount: float, config_key: str):\n    \"\"\"Track a conversion event with revenue.\"\"\"\n    context = Context.builder(user_id).build()\n\n    ld_client.track(\n        \"business.conversion\",\n        context,\n        data={\"configKey\": config_key, \"category\": \"electronics\"},\n        metric_value=amount\n    )\n\ndef track_task_success(ld_client, user_id: str, task_type: str, success: bool):\n    \"\"\"Track task completion success/failure.\"\"\"\n    context = Context.builder(user_id).build()\n\n    ld_client.track(\n        \"task.success_rate\",\n        context,\n        data={\"taskType\": task_type},\n        metric_value=1.0 if success else 0.0\n    )\n\ndef track_satisfaction(ld_client, user_id: str, score: float, feedback_type: str):\n    \"\"\"Track user satisfaction (0-100 scale).\"\"\"\n    context = Context.builder(user_id).build()\n\n    ld_client.track(\n        \"user.satisfaction\",\n        context,\n        data={\"feedbackType\": feedback_type},\n        metric_value=score\n    )\n\n    # Track negative feedback separately for alerts\n    if score < 50:\n        ld_client.track(\n            \"user.negative_feedback\",\n            context,\n            metric_value=1.0\n        )\n\ndef track_revenue(ld_client, user_id: str, revenue: float, source: str):\n    \"\"\"Track revenue generated after agent interaction.\"\"\"\n    context = Context.builder(user_id).set(\"tier\", \"premium\").build()\n\n    if revenue > 0:\n        ld_client.track(\n            \"revenue.impact\",\n            context,\n            data={\"source\": source},\n            metric_value=revenue\n        )\n```\n\n## 3. Get Metrics (API)\n\n### Get Single Metric\n\n```python\ndef get_metric(project_key: str, metric_key: str):\n    \"\"\"Get a single metric definition.\"\"\"\n    API_TOKEN = os.environ.get(\"LAUNCHDARKLY_API_TOKEN\")\n\n    url = f\"https://app.launchdarkly.com/api/v2/metrics/{project_key}/{metric_key}\"\n\n    headers = {\"Authorization\": API_TOKEN}\n\n    response = requests.get(url, headers=headers)\n\n    if response.status_code == 200:\n        metric = response.json()\n        print(f\"[OK] Metric: {metric['key']}\")\n        print(f\"     Name: {metric.get('name', 'N/A')}\")\n        print(f\"     Kind: {metric.get('kind', 'N/A')}\")\n        print(f\"     Numeric: {metric.get('isNumeric', False)}\")\n        print(f\"     Event Key: {metric.get('eventKey', 'N/A')}\")\n        return metric\n    elif response.status_code == 404:\n        print(f\"[INFO] Metric not found: {metric_key}\")\n        return None\n    else:\n        print(f\"[ERROR] Failed to get metric: {response.status_code}\")\n        return None\n```\n\n### List All Metrics\n\n```python\ndef list_metrics(project_key: str, limit: int = 20):\n    \"\"\"List all metrics in a project.\"\"\"\n    API_TOKEN = os.environ.get(\"LAUNCHDARKLY_API_TOKEN\")\n\n    url = f\"https://app.launchdarkly.com/api/v2/metrics/{project_key}\"\n\n    headers = {\"Authorization\": API_TOKEN}\n    params = {\"limit\": limit}\n\n    response = requests.get(url, headers=headers, params=params)\n\n    if response.status_code == 200:\n        data = response.json()\n        metrics = data.get(\"items\", [])\n        print(f\"[OK] Found {len(metrics)} metrics:\")\n        for metric in metrics:\n            numeric = \"numeric\" if metric.get(\"isNumeric\") else \"non-numeric\"\n            print(f\"     - {metric['key']} ({metric.get('kind', 'custom')}, {numeric})\")\n        return metrics\n    else:\n        print(f\"[ERROR] Failed to list metrics: {response.status_code}\")\n        return None\n```\n\n## 4. Update Metric (API)\n\n```python\ndef update_metric(project_key: str, metric_key: str, updates: list):\n    \"\"\"\n    Update a metric using JSON Patch operations.\n\n    Args:\n        updates: List of patch operations, e.g.:\n            [{\"op\": \"replace\", \"path\": \"/name\", \"value\": \"New Name\"}]\n    \"\"\"\n    API_TOKEN = os.environ.get(\"LAUNCHDARKLY_API_TOKEN\")\n\n    url = f\"https://app.launchdarkly.com/api/v2/metrics/{project_key}/{metric_key}\"\n\n    headers = {\n        \"Authorization\": API_TOKEN,\n        \"Content-Type\": \"application/json\"\n    }\n\n    response = requests.patch(url, json=updates, headers=headers)\n\n    if response.status_code == 200:\n        print(f\"[OK] Updated metric: {metric_key}\")\n        return response.json()\n    elif response.status_code == 404:\n        print(f\"[ERROR] Metric not found: {metric_key}\")\n        return None\n    else:\n        print(f\"[ERROR] Failed to update metric: {response.status_code}\")\n        print(f\"        {response.text}\")\n        return None\n\n# Example: Update metric name and description\ndef rename_metric(project_key: str, metric_key: str, new_name: str, new_description: str = None):\n    \"\"\"Rename a metric and optionally update description.\"\"\"\n    updates = [\n        {\"op\": \"replace\", \"path\": \"/name\", \"value\": new_name}\n    ]\n    if new_description:\n        updates.append({\"op\": \"replace\", \"path\": \"/description\", \"value\": new_description})\n\n    return update_metric(project_key, metric_key, updates)\n```\n\n## 5. Delete Metric (API)\n\n```python\ndef delete_metric(project_key: str, metric_key: str):\n    \"\"\"Delete a metric from the project.\"\"\"\n    API_TOKEN = os.environ.get(\"LAUNCHDARKLY_API_TOKEN\")\n\n    url = f\"https://app.launchdarkly.com/api/v2/metrics/{project_key}/{metric_key}\"\n\n    headers = {\"Authorization\": API_TOKEN}\n\n    response = requests.delete(url, headers=headers)\n\n    if response.status_code == 204:\n        print(f\"[OK] Deleted metric: {metric_key}\")\n        return True\n    elif response.status_code == 404:\n        print(f\"[INFO] Metric not found: {metric_key}\")\n        return False\n    else:\n        print(f\"[ERROR] Failed to delete metric: {response.status_code}\")\n        return False\n```\n\n## Complete Workflow Example\n\n```python\nimport os\nimport requests\nfrom ldclient import Context\nfrom ldclient.config import Config\nimport ldclient\n\n# Setup\nAPI_TOKEN = os.environ.get(\"LAUNCHDARKLY_API_TOKEN\")\nSDK_KEY = os.environ.get(\"LAUNCHDARKLY_SDK_KEY\")\nPROJECT_KEY = \"support-ai\"\n\nldclient.set_config(Config(SDK_KEY))\nld_client = ldclient.get()\n\n# 1. Create metric\ncreate_metric(\n    PROJECT_KEY,\n    \"ai.task.completion\",\n    name=\"Agent Task Completion Rate\",\n    kind=\"custom\",\n    is_numeric=True,\n    description=\"Tracks successful agent task completions\"\n)\n\n# 2. Track events\ncontext = Context.builder(\"user-123\").build()\nld_client.track(\"ai.task.completion\", context, metric_value=1.0)\nld_client.track(\"ai.task.completion\", context, metric_value=1.0)\nld_client.track(\"ai.task.completion\", context, metric_value=0.0)  # failure\nld_client.flush()\n\n# 3. Get metric definition\nmetric = get_metric(PROJECT_KEY, \"ai.task.completion\")\n\n# 4. Update metric name\nrename_metric(PROJECT_KEY, \"ai.task.completion\", \"Agent Task Success Rate\")\n\n# 5. List all metrics\nlist_metrics(PROJECT_KEY)\n\n# 6. Delete metric (when no longer needed)\n# delete_metric(PROJECT_KEY, \"ai.task.completion\")\n```\n\n## Session Metrics Tracker\n\n```python\nimport time\nfrom ldclient import Context\n\nclass SessionMetricsTracker:\n    \"\"\"Track metrics across an entire user session.\"\"\"\n\n    def __init__(self, ld_client):\n        self.ld_client = ld_client\n        self.session_data = {}\n\n    def start_session(self, user_id: str, session_id: str):\n        \"\"\"Initialize session tracking.\"\"\"\n        self.session_data[session_id] = {\n            \"user_id\": user_id,\n            \"start_time\": time.time(),\n            \"interactions\": 0,\n            \"successful_tasks\": 0\n        }\n\n    def track_interaction(self, session_id: str, success: bool):\n        \"\"\"Track individual interaction within session.\"\"\"\n        if session_id not in self.session_data:\n            return\n        session = self.session_data[session_id]\n        session[\"interactions\"] += 1\n        if success:\n            session[\"successful_tasks\"] += 1\n\n    def end_session(self, session_id: str):\n        \"\"\"Finalize and track session metrics.\"\"\"\n        if session_id not in self.session_data:\n            return None\n\n        session = self.session_data[session_id]\n        duration = time.time() - session[\"start_time\"]\n\n        context = Context.builder(session[\"user_id\"]).build()\n\n        # Track session duration\n        self.ld_client.track(\n            \"session.duration\",\n            context,\n            data={\"interactions\": session[\"interactions\"]},\n            metric_value=duration\n        )\n\n        # Track session success rate\n        if session[\"interactions\"] > 0:\n            success_rate = session[\"successful_tasks\"] / session[\"interactions\"]\n            self.ld_client.track(\n                \"session.success_rate\",\n                context,\n                metric_value=success_rate * 100\n            )\n\n        result = dict(session)\n        result[\"duration\"] = duration\n        del self.session_data[session_id]\n        return result\n```\n\n## Naming Conventions\n\n```python\n# Use dot notation for hierarchy\n\"quality.accuracy\"\n\"quality.relevance\"\n\"user.satisfaction\"\n\"user.engagement\"\n\"revenue.conversion\"\n\"task.success_rate\"\n\"session.duration\"\n\"ai.task.completion\"\n\"ai.recommendation.conversion\"\n```\n\n## Best Practices\n\n1. **Create Before Track** - Metric must exist before tracking events\n2. **Use Numeric Metrics** - Set `isNumeric=True` for aggregation\n3. **Consistent Keys** - Use same key in `create_metric()` and `ld_client.track()`\n4. **Always flush before close** - Call `ld_client.flush()` (await in Node) before `close()`. Trailing events are at risk of being lost otherwise, in short-lived scripts and long-running services alike. This is not a serverless-only rule; it applies to any process that exits.\n5. **Rate Limit** - Don't track on every keystroke\n\n## Viewing Metrics\n\nCustom metrics appear in:\n- **Metrics** page in LaunchDarkly UI\n- **Monitoring tab** of your config\n- Via API using `get_metric()` or `list_metrics()`\n\n## Related Skills\n\n- `sdk` - SDK setup\n- `built-in-metrics` - Built-in agent metrics (tokens, duration, cost)\n- `online-evals` - Quality metrics via judges\n\n## References\n\n- [Metrics API Documentation](https://apidocs.launchdarkly.com/tag/Metrics)\n- [Custom Events Documentation](https://docs.launchdarkly.com/sdk/features/events)\n- [Python SDK track() Reference](https://launchdarkly-python-sdk.readthedocs.io/)\n"
}