ABMeter

View saved version →

---
name: analyze-experiment
description: "Check whether a running ABMeter experiment can answer its hypothesis yet. Use when the user asks about an experiment's results or significance."
---

Replace `<experiment-slug>` below with the slug of the experiment the user named. If they named none, find it with `search_experiments` and ask which one.

Can the experiment '<experiment-slug>' answer its hypothesis?

**Data Foundation:**
Analysis works by joining exposures (which users saw which variant) to events (what users did) via user_id.
When you see "n = 1500 users", this means 1500 unique user_ids were exposed to that variant AND had matching events.

**STEP 1: Understand the experiment**
1. Use search_experiments tool to get '<experiment-slug>'
2. Extract the hypothesis/question from its description
3. Note its space_slug and its primary_metric_slug if set

**STEP 2: Find an appropriate metric**
1. Use search_metrics tool to find metrics related to the experiment's goal
2. If primary_metric_slug is set on the experiment, use that
3. Otherwise, identify the most relevant metric for the hypothesis

**STEP 3: Analyze statistical significance**
1. Use evaluate_experiment_effects tool with:
   - experiment_slug: '<experiment-slug>'
   - space_slug: [the experiment's space_slug]
   - metric_slug: [the metric you identified]
   - attribution_window: "7d" (or appropriate for the use case)

2. Interpret the results from stats_by_audience:
   - Check p_value: Is it < 0.05? (statistically significant)
   - Check confidence_interval_95: Does it exclude 0? (significant)
   - Check relative_lift: What is the effect size? (practical significance)
   - Check n (sample size): Is it sufficient for reliable inference?

**STEP 4: Answer these questions**

Can we reliably answer the hypothesis?
- YES: State the conclusion with confidence level
  Example: "Treatment increases conversions by 15% (p=0.02, 95% CI: [+5%, +25%])"
- NO: Explain what's preventing a reliable answer
- PARTIALLY: Describe what we can and cannot conclude

If we cannot answer reliably, what's missing?
- Insufficient sample size (report current n, suggest needed n for desired power)
- High p-value with small effect (may never reach significance if true effect is small)
- Wide confidence interval (need more data to narrow precision)
- No events being tracked (implementation issue)

What should we do next?
- Continue: Estimate how much longer based on current traffic rate
- Conclude: Use the conclude-experiment skill to analyze results, stop the experiment, and guide the user through making a decision
- Fix implementation: If no data is being collected

Use simple language focused on answering: "Do we have our answer yet?"

View saved version →

---
name: conclude-experiment
description: "Analyze an ABMeter experiment's results, recommend a decision, and guide the user through concluding it on the website. Use when the user wants to stop an experiment or pick a winner."
---

Replace `<experiment-slug>` below with the slug of the experiment the user named. If they named none, find it with `search_experiments` and ask which one.

I want to conclude the experiment '<experiment-slug>'.

## Your Workflow (State-Aware)

First, use `search_experiments` to find '<experiment-slug>' and check its current state. Note its `space_slug`: pass it as `space_slug` to every tool call below.

### If RUNNING (most common path):

**1. Analyze effects FIRST -- before any destructive action**
- Use `search_experiments` to get experiment details (hypothesis, primary metric, audiences, variants)
- Use `evaluate_experiment_effects` with the primary metric (and guardrail metrics if set) to compute statistical results
- Interpret results: p-values, confidence intervals, effect sizes, sample sizes
- This analysis happens while the experiment is still live -- no user impact yet

**2. Formulate a recommendation**
Based on the statistical analysis, recommend one of:
- **Apply Winner** (specific audience): statistically significant positive effect (p < 0.05, CI excludes 0, meaningful effect size)
- **Keep Original Values**: control performs best or no significant difference -- results are conclusive but treatment didn't win
- **Reset to Draft**: data quality issues (too few users, implementation bugs, no events tracked)
- **Don't stop yet**: results are inconclusive and more data could help -- estimate how much longer based on current traffic rate
Always explain the reasoning clearly.

**3. Present recommendation to user BEFORE stopping**
- The recommended decision with rationale
- What will happen when that decision is applied (e.g., "Parameter `button_size` default will change from `medium` to `large` for all users")
- Suggested decision notes text
- **CRITICAL: Get user agreement before proceeding to destructive actions**
- Stopping is destructive: it removes the experiment from assignment_config, reverting all users to control

**4. Guide through UI actions (stop + decide)**
All state changes go through the Web UI for safety and auditability:
- Use `navigate_browser` with `resource_type: "experiment"` and `resource_slug: "<experiment-slug>"` to open the experiment page
- Instruct user to click **"Stop"** button and confirm -- wait for their confirmation
- After user confirms stop, use `search_experiments` to verify state is `stopped`
- Navigate to the decision page with the recommendation pre-populated:
  Use `navigate_browser` with `resource_type: "experiment_decision"`, `resource_slug: "<experiment-slug>"`, and `query_params` containing the recommendation as a JSON string.
  Example query_params:
  ```json
  {"decision": "apply_winner", "winning_audience_slug": "<slug from audiences array>", "decision_notes": "<your statistical summary>"}
  ```
- The decision modal opens automatically with the recommendation pre-selected, winning variant chosen, and decision notes filled in
- Tell the user: "The decision modal is open with my recommendation pre-selected. Review it and click **Confirm Decision**, or change the selection if you disagree."

**5. Verify the outcome**
- Use `search_experiments` to verify the experiment transitioned to `archived` state (or `init` if reset)

### If STOPPED (already stopped, need decision):

Skip the stopping steps. Go directly to:
1. Analyze effects (same as above -- `evaluate_experiment_effects` works on stopped experiments too)
2. Formulate recommendation
3. Present recommendation
4. Navigate to decision page with pre-populated recommendation using `navigate_browser` with `resource_type: "experiment_decision"` and `query_params` (use `winning_audience_slug` with a slug from the experiment's `audiences` array)
5. User confirms or overrides in the pre-populated modal

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: design-experiment
description: "Design an A/B experiment in ABMeter step by step and hand the launch to the user on the website. Use when the user wants to test a change."
---

You are helping design and launch an A/B experiment in ABMeter. Guide the user through a natural conversation that results in a well-designed, statistically sound experiment.

## CRITICAL: Manual Launch Requirement

**Experiments MUST be launched manually by the user through the Web UI.**
- You can CREATE experiments using tools
- You can DISPLAY experiments in the browser using `navigate_browser` tool
- You CANNOT and MUST NOT start/launch/transition experiments
- You MUST WAIT for the user to click "Start" in the browser
- After user confirms they clicked Start, use search_experiments to verify it's running
- DO NOT use curl, APIs, or any tools to change experiment state

## CRITICAL: Understanding Exposure Tracking & Experiment Analysis

**Event Data Model:**
Every event automatically includes:
- `user_id` - identifies who triggered the event (used to link events to experiment exposures)
- `occurred_at` - when the event happened

Custom fields are additional data defined per event type (e.g., amount, plan_slug).

**How Attribution Works:**
1. SDK calls `resolve_parameter()` → Exposure recorded (user_id + variant + timestamp)
2. App tracks event (e.g., purchase) → Event recorded (user_id + custom_fields + occurred_at)
3. Analysis joins events to exposures via user_id → "Users in variant A had X% higher conversion"

This is why user_id is fundamental - it's the link between "which variant did the user see" and "what did the user do".

**The Data Flow:**
1. **Events** are tracked by your application (e.g., purchase, click, page_view)
2. **Measures** are aggregations computed from events (e.g., count(purchase), sum(purchase.amount))
   - Discover available measures with search_event_types or search_metrics tools
3. **Metrics** are business KPIs with formulas over measures (e.g., "sum(purchase.amount)" for revenue per user)
4. **Exposures** are automatically tracked when SDK calls `resolve_parameter()` - no manual event needed

**How Experiment Analysis Works:**
- Results bucketed by (audience, time): exposure_count and metric_value per bucket
- Complex metrics apply formulas across measure buckets
- Statistical results (z-score, p-value, confidence intervals) are computed server-side

**Statistical Validity:**
- Results are valid when n ≥ 30 users per audience (Central Limit Theorem requirement)
- For smaller sample sizes, statistical conclusions may be unreliable
- If an experiment has insufficient data, advise the user to wait for more exposures before drawing conclusions


## CRITICAL: Understanding Spaces and Interference

- **Within a space**: Experiments are mutually exclusive - a user can ONLY be in ONE experiment at a time. This is automatic experiment isolation.
- **Example**: If space has 60% allocated to experiments A and B, a new 20% experiment C will NOT interfere - users get assigned to either A, B, or C, never multiple
- **Available capacity** (`allocation_available` from `search_spaces`) is the share of the space not held by a running or stopped experiment - these users are available for new experiments
- **Across spaces**: Experiments CAN interfere if they affect the same user behavior (e.g., both change pricing perception, both modify same funnel stage)
- Only warn about interference when experiments in DIFFERENT spaces might affect the same behavioral outcomes

## Required Information for Launch

To successfully launch an experiment, you need:

**Essential:**
- **Hypothesis/Question**: What change are we testing and why? (e.g., "Will a larger checkout button increase conversions?")
- **Primary Metric**: What measurement proves success? ALWAYS create a metric, even for simple cases
  - Simple case: "purchase_count" with formula "count(purchase)"
  - Revenue case: "revenue_per_user" with formula "sum(purchase.amount)"
- **Alloc

Preview of the first 4,000 characters. Read full instructions →

View saved version →

---
name: get-started
description: "Get started with ABMeter: its core concepts, what only the user can do on the ABMeter website, and the skills for designing, analyzing and concluding an experiment. Use when the user is new to ABMeter or asks what it can do."
---

Welcome to ABMeter! You now have access to a comprehensive experimentation platform for running A/B tests and managing feature rollouts.

**Core Concepts You Should Know:**
- **Space**: Your allocation pool of 100% of users. Experiments within a space share this pool without overlapping, ensuring clean test results.
- **Parameters**: Configuration values you can control (e.g., button_color, font_size) with default values. A parameter belongs to one space, and its slug is unique across all your spaces. A variant can set only parameters from its own space.
- **Variants**: Named collections of parameter overrides (e.g., a "high_contrast" variant might change multiple color parameters at once).
- **Experiments**: Controlled tests where you randomly assign user percentages to different variants to validate your hypotheses.
- **Feature Flags**: Targeted rollouts where you can assign specific variants to particular users (by email, user lists, or custom predicates).

**Skills You Can Use:**
- `design-experiment` - You can use this skill to set up a new A/B test. You'll be guided through discovery, setup, and launch phases with full control at each step.
- `analyze-experiment` - You can use this to check if your running experiments have achieved statistical significance.
- `conclude-experiment` - Analyze experiment results, get a recommendation, and conclude the experiment with a decision (apply winner, keep original values, or reset).
- `get-started` - You can return to this overview anytime.

**Running experiments are locked:**
Once an experiment runs, each user stays in the random audience they were first assigned to. A change that would move them needs the user's consent in the ABMeter web UI; here it is refused with an explanation. Example: running experiment `button-test` tests parameter `button-color`. A feature flag that overrides `button-color` would change what the experiment's users see. Only the user can enable a flag, in the web UI, and if it overrides `button-color` the web UI asks them to confirm first. Here, an edit that makes an enabled flag override `button-color` is refused: changing its audience or variant, or changing its variant's parameter values. New feature flags start off, so this check never blocks creating one.
Starting or stopping an experiment and enabling or disabling a feature flag also happen only in the web UI. Open the page for the user with `navigate_browser`.

**Direct Tools Available:**
You also have access to search and management tools:
- Search for existing experiments, feature flags, variants, or parameters
- Create or update parameters and variants
- View event tracking and metrics
- Set experiment audiences and allocation before an experiment starts
- Open any resource in the user's browser

**Getting Started:**
To launch your first experiment, you could say something like:
"Use the design-experiment skill to test if a blue button gets more clicks than the current one"

The system will then guide you through each phase, asking for your approval before making any changes.

What would you like to explore or test?