{"id":80439,"plugin_id":"plugin_asdk_app_6ae7c76148c081918210c44de9479cf3","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-11-04T06:14:59.159Z","digest":"9115ece961d82177518716c3251d09707e50bf8c34f48e93a65ecb20084a8367","against":null,"payload":{"description":"Check whether a running ABMeter experiment can answer its hypothesis yet. Use when the user asks about an experiment's results or significance.","included_files":[],"name":"analyze-experiment","skill_md_contents":"---\nname: analyze-experiment\ndescription: \"Check whether a running ABMeter experiment can answer its hypothesis yet. Use when the user asks about an experiment's results or significance.\"\n---\n\nReplace `<experiment-slug>` below with the slug of the experiment the user named. If they named none, find it with `search_experiments` and ask which one.\n\nCan the experiment '<experiment-slug>' answer its hypothesis?\n\n**Data Foundation:**\nAnalysis works by joining exposures (which users saw which variant) to events (what users did) via user_id.\nWhen you see \"n = 1500 users\", this means 1500 unique user_ids were exposed to that variant AND had matching events.\n\n**STEP 1: Understand the experiment**\n1. Use search_experiments tool to get '<experiment-slug>'\n2. Extract the hypothesis/question from its description\n3. Note its space_slug and its primary_metric_slug if set\n\n**STEP 2: Find an appropriate metric**\n1. Use search_metrics tool to find metrics related to the experiment's goal\n2. If primary_metric_slug is set on the experiment, use that\n3. Otherwise, identify the most relevant metric for the hypothesis\n\n**STEP 3: Analyze statistical significance**\n1. Use evaluate_experiment_effects tool with:\n   - experiment_slug: '<experiment-slug>'\n   - space_slug: [the experiment's space_slug]\n   - metric_slug: [the metric you identified]\n   - attribution_window: \"7d\" (or appropriate for the use case)\n\n2. Interpret the results from stats_by_audience:\n   - Check p_value: Is it < 0.05? (statistically significant)\n   - Check confidence_interval_95: Does it exclude 0? (significant)\n   - Check relative_lift: What is the effect size? (practical significance)\n   - Check n (sample size): Is it sufficient for reliable inference?\n\n**STEP 4: Answer these questions**\n\nCan we reliably answer the hypothesis?\n- YES: State the conclusion with confidence level\n  Example: \"Treatment increases conversions by 15% (p=0.02, 95% CI: [+5%, +25%])\"\n- NO: Explain what's preventing a reliable answer\n- PARTIALLY: Describe what we can and cannot conclude\n\nIf we cannot answer reliably, what's missing?\n- Insufficient sample size (report current n, suggest needed n for desired power)\n- High p-value with small effect (may never reach significance if true effect is small)\n- Wide confidence interval (need more data to narrow precision)\n- No events being tracked (implementation issue)\n\nWhat should we do next?\n- Continue: Estimate how much longer based on current traffic rate\n- Conclude: Use the conclude-experiment skill to analyze results, stop the experiment, and guide the user through making a decision\n- Fix implementation: If no data is being collected\n\nUse simple language focused on answering: \"Do we have our answer yet?\"\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}