{"id":59854,"plugin_id":"plugin_asdk_app_6aa944d20a3081919d9f65692616ff4d","kind":"skill","collection_source":"plugin_package","comparison_source":"plugin_package","observed_at":"2026-10-27T00:28:52.455Z","digest":"98fbd2470c67614f02944540754b9bc2813e76da7627c4ad065cd2a3a21bae89","against":43809,"payload":{"description":"Create or measure an Omneky paid-media A/B test. Use for A/B test creation (one hypothesis, one variable, precommitted plan, launch live or paused as the user chooses) or A/B test measurement (z-test, CI, power, SRM, interim peek with no verdict, refuse a thin winner). Do not use for generic ROAS tables (omneky-analytics-and-catalogue), competitor analysis (omneky-competitor-analysis), first-run onboarding (omneky-onboarding), agency (omneky-agency), weekly marketing plans (omneky-marketing-plan), generating creatives (omneky-image-ads, omneky-product-video, omneky-avatar-ugc), or credit balance (omneky-billing). Keywords: A/B test, ab test create, ab test measurement, which variant won, significance, two-proportion z-test, MDE, SRM, interim peek, underpowered, commercially meaningful, budget split.","included_files":[{"relative_path":"references/playbook.md","size_in_bytes":7149}],"name":"omneky-ab-tests","skill_md_contents":"---\nname: omneky-ab-tests\ndescription: >-\n  Create or measure an Omneky paid-media A/B test. Use for A/B\n  test creation (one hypothesis, one variable, precommitted\n  plan, launch live or paused as the user chooses) or A/B test measurement (z-test, CI,\n  power, SRM, interim peek with no verdict, refuse a thin\n  winner). Do not use for generic ROAS tables\n  (omneky-analytics-and-catalogue), competitor analysis\n  (omneky-competitor-analysis), first-run onboarding\n  (omneky-onboarding), agency (omneky-agency), weekly marketing\n  plans (omneky-marketing-plan), generating creatives\n  (omneky-image-ads, omneky-product-video, omneky-avatar-ugc),\n  or credit balance (omneky-billing). Keywords: A/B test,\n  ab test create, ab test measurement, which variant won,\n  significance, two-proportion z-test, MDE, SRM, interim peek,\n  underpowered, commercially meaningful, budget split.\n---\n\n# A/B tests\n\nUse this when the user wants to set up an A/B test or read out which\nvariant won. One job: own the test plan and the readout.\n\nGate every tool on this session's `tools/list`. If a step's tool is\nmissing, say so and point to the Omneky app.\n\nPlan fields: [handoff-contract](../_shared/references/handoff-contract.md).\nFormulas: [ab-test-statistics](../_shared/references/ab-test-statistics.md).\nSourced ids: [preconditions](../_shared/references/preconditions.md).\nAcceptance gates: [creative-quality-gates](../_shared/references/creative-quality-gates.md).\nBrief: [creative-brief](../_shared/references/creative-brief.md).\nOther intents: [capability-registry](../_shared/references/capability-registry.md).\nWrite for: [marketer-persona](../_shared/references/marketer-persona.md).\nStep detail: [playbook](references/playbook.md) (Hard rules, Create, Measure).\n\n## Goal\n\nCreate a one-variable test with a precommitted plan (live or paused is\nthe user's choice in the host), or\nmeasure it with a two-proportion z-test and a marketer verdict.\n\n## When to use\n\n- Create an A/B test, measure one, \"which variant won\", significance,\n  stop rule, interim peek, SRM, MDE\n\n## When not to use\n\n- Generic ROAS table → `omneky-analytics-and-catalogue`\n- Launch without a test plan → `omneky-launch-and-connect`\n\n## Prerequisites\n\n| Need | Check | If missing |\n|---|---|---|\n| Brand | `list_brands` | Ask which |\n| Channel | `list_connections` (`connector`) | Connect or managed |\n| Budget floor | `get_launch_requirements` | Do not launch under the floor |\n| Variants | prior creative handoff | Owning creative skill |\n| Baseline | `get_performance_summary` with `date_range` | State an assumed `baseline_rate` |\n| Coverage (measure) | `get_performance_summary` (`include_coverage=true`) | Do not declare a winner |\n| Metrics (measure) | `get_performance_breakdown` / `get_performance_summary` | Narrow the window |\n\n## Flow\n\n**Create**\n1. Write one hypothesis with one changed variable.\n2. Record metric, MDE, duration (`horizon`) and stop rule before launch.\n3. Budget at or above the floor. One ad group with one ad per variant\n   does not guarantee an even spend split (delivery is auction-driven);\n   say so. For an even split, use one ad group (or campaign) per variant\n   with equal budgets, or the platform's A/B / split-test feature.\n4. Launch through `omneky-launch-and-connect`, live or paused as the\n   user chose (the host confirms).\n\n**Measure**\n1. Confirm coverage for the window; empty is missing data, not zero.\n2. Run the setup check on every look (broken setup; SRM only for a\n   randomized planned allocation, not default auction delivery). If it\n   fails, give the verdict Invalid at once and recommend a pause.\n   Never mutate here.\n3. Before `horizon`, report INTERIM with no statistical verdict.\n4. At `horizon`, run the z-test and give one verdict: Winner, Not\n   commercially meaningful, Inconclusive, or No detectable difference.\n5. Refuse a winner when the sample is thin.\n\n## Failure and recovery\n\nAuth / 401 → `omneky-getting-started-and-failures` (host OAuth).\nEmpty reporting → `get_performance_summary` (`include_coverage=true`); empty is not zero.\n`credit_insufficient` → nothing charged; state balance vs required\n(`omneky-billing`). No payment links.\n`needs_user_decision` → fill the missing detail and call again; ask in plain chat only if you cannot. Never dump JSON.\nMissing tool → this session may run an older published app snapshot: look\nthe name up in [tool-name-fallbacks](../_shared/references/tool-name-fallbacks.md)\nand call the older name if that one is on `tools/list`. If neither is\nlisted, say so; Omneky app. Do not invent it.\nNever ask for internal ids. Never silent-switch `brand_id`.\n\n## Gaps\n\nDecline social inbox, landing pages, subtitle burn-in, email/SMS,\norganic scheduling, Ad Library ads, try-on, and website builder\n([capability-gaps](../_shared/references/capability-gaps.md)).\nThere is no sequential-testing platform tool. Early stop is\nacknowledgement + correction, or not supported.\n"},"changes":[{"path":"/description","type":"changed","before":"Create or measure an Omneky paid-media A/B test. Use for A/B test creation (one hypothesis, one variable, precommitted plan, PAUSED launch) or A/B test measurement (z-test, CI, power, SRM, interim peek with no verdict, refuse a thin winner). Do not use for generic ROAS tables (omneky-analytics-and-catalogue), competitor analysis (omneky-competitor-analysis), first-run onboarding (omneky-onboarding), agency (omneky-agency), weekly marketing plans (omneky-marketing-plan), generating creatives (omneky-image-ads, omneky-product-video), or billing reads (omneky-billing). Keywords: A/B test, ab test create, ab test measurement, which variant won, significance, two-proportion z-test, MDE, SRM, interim peek, underpowered, commercially meaningful, budget split.","after":"Create or measure an Omneky paid-media A/B test. Use for A/B test creation (one hypothesis, one variable, precommitted plan, launch live or paused as the user chooses) or A/B test measurement (z-test, CI, power, SRM, interim peek with no verdict, refuse a thin winner). Do not use for generic ROAS tables (omneky-analytics-and-catalogue), competitor analysis (omneky-competitor-analysis), first-run onboarding (omneky-onboarding), agency (omneky-agency), weekly marketing plans (omneky-marketing-plan), generating creatives (omneky-image-ads, omneky-product-video, omneky-avatar-ugc), or credit balance (omneky-billing). Keywords: A/B test, ab test create, ab test measurement, which variant won, significance, two-proportion z-test, MDE, SRM, interim peek, underpowered, commercially meaningful, budget split."},{"path":"/included_files","type":"changed","before":[],"after":[{"relative_path":"references/playbook.md","size_in_bytes":7149}]},{"path":"/skill_md_contents","type":"changed","before":"---\nname: omneky-ab-tests\ndescription: >-\n  Create or measure an Omneky paid-media A/B test. Use for A/B\n  test creation (one hypothesis, one variable, precommitted\n  plan, PAUSED launch) or A/B test measurement (z-test, CI,\n  power, SRM, interim peek with no verdict, refuse a thin\n  winner). Do not use for generic ROAS tables\n  (omneky-analytics-and-catalogue), competitor analysis\n  (omneky-competitor-analysis), first-run onboarding\n  (omneky-onboarding), agency (omneky-agency), weekly marketing\n  plans (omneky-marketing-plan), generating creatives\n  (omneky-image-ads, omneky-product-video), or billing reads\n  (omneky-billing). Keywords: A/B test, ab test create, ab test\n  measurement, which variant won, significance, two-proportion\n  z-test, MDE, SRM, interim peek, underpowered, commercially\n  meaningful, budget split.\n---\n\n# A/B tests\n\nUse the connected Omneky MCP server. Gate every named tool on this\nsession's `tools/list`. If the tool for a requested step is missing,\nsay so and point to the Omneky app. Do not invent a tool.\n\nThis skill owns the **test plan** (create) and the **readout**\n(measure). Generic winner lists stay in\n`omneky-analytics-and-catalogue`. A one-off `PAUSED` launch without\na test plan stays in `omneky-launch-and-connect`.\n\nHandoff fields:\n[handoff-contract](../_shared/references/handoff-contract.md)\n(including the A/B test plan fields).\nFormulas and worked examples (example data only):\n[ab-test-statistics](../_shared/references/ab-test-statistics.md).\nSourced ids:\n[preconditions](../_shared/references/preconditions.md).\nAcceptance gates:\n[creative-quality-gates](../_shared/references/creative-quality-gates.md).\nStructured brief:\n[creative-brief](../_shared/references/creative-brief.md).\nOther intents:\n[capability-registry](../_shared/references/capability-registry.md).\nWrite for:\n[marketer-persona](../_shared/references/marketer-persona.md).\n\n## Goal\n\nCreate a one-variable `PAUSED` A/B test with a precommitted plan,\nor measure it with a two-proportion z-test and a marketer verdict.\n\n## When to use\n\n- A/B test create, A/B test measurement, which variant won,\n  significance, stop rule, interim peek, SRM, MDE\n\n## When not to use\n\n- Generic ROAS table → `omneky-analytics-and-catalogue`\n- Launch without a test plan → `omneky-launch-and-connect`\n\n## Prerequisites\n\n| Need | Check | If missing |\n|---|---|---|\n| Brand | `list_brands` | Ask which |\n| Channel | `get_channel_connection_status` | Connect first (`omneky-launch-and-connect`) |\n| Budget floor | `minimum_budget_for_objective` | Do not launch under the floor |\n| Variants | prior creative handoff | Owning creative skill |\n| Baseline | `get_daily_metrics` with `date_range` | State an assumed `baseline_rate` |\n| Coverage (measure) | Metric tools for the window; empty is missing data | Do not declare a winner |\n| Metrics (measure) | `get_performance_breakdown` / `get_daily_metrics` | Narrow the window |\n\n## Flow\n\nFollow Create or Measure. **Invalid** is exempt from the\nhorizon gate. The setup check (step 4) runs on every look,\nincluding pre-horizon peeks. If it fails, give the verdict\nInvalid immediately, Stop, and recommend pausing or fixing the\ntest (a pause is a later yes via `omneky-launch-and-connect`;\nnever mutate here). Only Winner, Not commercially\nmeaningful, Inconclusive, and No detectable difference wait\nfor `horizon`. A peek that passes the setup check is INTERIM\nwith no statistical verdict. Refuse a winner when the sample\nis thin.\n\n## Hard rules\n\n- Stick to the chosen `brand_id`. Never silent-switch brands.\n- One hypothesis, **one** variable. Extra changes are a new test.\n- Default new launches to `PAUSED`. Live spend needs an explicit\n  yes in the same summary.\n- Call the budget-floor tool (`minimum_budget_for_objective`)\n  before any budget. Do not set a daily budget under that floor.\n  Split the approved total across variants; each arm stays at or\n  above the floor or you do not launch.\n- Paid creatives that precede the test use the owning creative\n  skill: Never invent a price.\n- Precommit and record the test plan: `primary_metric` (one),\n  `baseline_rate`, `mde`, `alpha` (default 0.05), `power`\n  (default 0.8), `n_per_arm` from the two-proportion formula\n  (show it), `randomization_unit` plus the ad-level-split caveat,\n  `attribution_window`, and `horizon` (end date or sample).\n- Measure: **Invalid** is exempt from the horizon gate. The\n  setup check runs on every look, including pre-horizon\n  peeks. If it fails, give the verdict Invalid immediately,\n  Stop, and recommend pausing or fixing (pause is a later yes via\n  `omneky-launch-and-connect`; never mutate here). Only\n  Winner, Not commercially meaningful, Inconclusive, and\n  No detectable difference wait for `horizon`. A peek that\n  passes setup is INTERIM. Early stop needs an explicit\n  acknowledgement and a Bonferroni or O'Brien-Fleming\n  correction (or say it is not supported).\n- Report effect size + CI, p-value, and achieved power.\n  Significance is not commercial meaning — compare to `mde` and\n  a CPA / ROAS $ impact.\n- SRM → Invalid only on a randomized planned split (platform\n  A/B / split-test, or an explicit planned allocation). Default\n  one-ad-group, one-ad-per-variant delivery is auction-driven:\n  do not run SRM as a gate; report imbalance as a caveat and\n  compare rates, not totals. Broken setup (near-zero delivery,\n  mid-test change, more than one variable) → Invalid. Low\n  counts → Insufficient data. Many variants → Bonferroni.\n- Scale / kill / iterate needs a later yes\n  (`omneky-launch-and-connect`).\n\n## Create\n\n1. Write the test plan with the user (one turn if they already\n   gave it):\n   - Hypothesis (what should win, and why)\n   - Single variable (creative, or headline, or audience — pick one)\n   - `primary_metric` (CTR or CVR — one only)\n   - `baseline_rate` from `get_daily_metrics` with `date_range`,\n     or a stated assumption\n   - `mde` (absolute rate)\n   - `alpha` 0.05 and `power` 0.8 unless they set others\n   - `n_per_arm` from the two-proportion formula (show it;\n     [ab-test-statistics](../_shared/references/ab-test-statistics.md))\n   - `randomization_unit` and the caveat that platform ad-level\n     splits are not true user randomization unless platform A/B\n     is used\n   - `attribution_window`\n   - `horizon` (end date or sample). This is the stop rule.\n2. Variants come from the structured brief (or a prior creative\n   handoff). Same brief except the one variable.\n3. Structure: one campaign, one ad group, one ad per variant\n   (same targeting and budget split). Use an existing campaign\n   only when they named it (`campaign_spec.id`).\n4. Preflight the channel with `omneky-launch-and-connect` rules:\n   connect status, budget floor, targeting pre-call, copy / CTA /\n   landing URL, credits. Default new launches to `PAUSED`.\n5. One summary: channel, tool, variants, budget split, floor,\n   the full test plan, `PAUSED`. Wait for one yes.\n6. Call the matching `launch_*`. Then write the test plan back\n   (`primary_metric`, `baseline_rate`, `mde`, `alpha`, `power`,\n   `n_per_arm`, `randomization_unit`, `attribution_window`,\n   `horizon`, campaign / ad ids).\n\nDo not go live unless that yes said to go live.\n\n## Measure\n\n1. Reuse the test plan. Missing ids: `get_campaigns` /\n   `get_ad_groups` (pass `campaign_id`) / `get_ads`. Do not guess.\n2. Empty metrics for the brand + window + channel mean no imported\n   data → stop. Empty is not zero.\n3. `get_performance_breakdown` (dimension ad or creative) and\n   `get_daily_metrics` for the window. `get_performance_movers`\n   when they asked what changed vs the prior period.\n4. Setup check on every look, including pre-horizon peeks.\n   **Invalid** only for a broken setup: near-zero delivery\n   on a variant, a variant changed mid-test, or more than\n   one variable differs. Default setup (one ad group, one\n   ad per variant) is auction-driven: do not run SRM as a\n   gate. Report delivery imbalance as a caveat and compare\n   rates, not totals. Run SRM → Invalid only when traffic\n   was randomized with a planned split (platform A/B /\n   split-test, or an explicit planned allocation) and\n   χ² p < 0.001. If the setup check fails, give the verdict\n   Invalid immediately, Stop, and recommend pausing or fixing the\n   test (a pause is a later yes via\n   `omneky-launch-and-connect`; never mutate here).\n5. Sample size: state impressions, clicks, and conversions per\n   arm. Thin if either arm has fewer than about 1,000 impressions\n   or 30 clicks (or 10 conversions when the metric is CVR) →\n   Insufficient data. Do not declare a winner.\n6. Horizon not reached and setup check passed → **INTERIM**\n   status only. No statistical verdict (Winner / Not\n   commercially meaningful / Inconclusive / No detectable\n   difference). Do not relabel a failed setup check as\n   INTERIM. If they insist on early stopping, require an\n   explicit acknowledgement and apply Bonferroni across\n   looks or O'Brien-Fleming-style (see the statistics\n   reference). If they will not acknowledge, say early\n   stopping is not supported.\n7. Horizon reached (or corrected early stop): two-proportion\n   z-test on the precommitted metric. Two-sided. Show n,\n   successes, z, p-value, 95% CI, achieved power. This is a\n   simple frequentist check, not an uncorrected sequential test\n   — say that.\n8. Verdict:\n   - **Invalid** — exempt from the horizon gate. From the\n     step-4 setup check on every look, including peeks.\n     Immediate Stop. Recommend pausing or fixing (later yes\n     via `omneky-launch-and-connect`; never mutate here).\n   - Only these wait for `horizon` (or a corrected early\n     stop — lead with this):\n     - **Inconclusive (underpowered)** — power < 0.8 and not\n       significant\n     - **No detectable difference** — |z| < 1.96 and power ≥ 0.8\n     - Not commercially meaningful — significant but |lift| < `mde`\n       (or $ impact below the stated floor)\n     - **Winner** — significant, |lift| ≥ `mde`, $ impact stated\n9. Scale / kill / iterate is a later yes on\n   `omneky-launch-and-connect`. Do not mutate here.\n\n## Failure and recovery\n\nAuth / 401 → `omneky-getting-started-and-failures` (host OAuth).\nEmpty reporting → empty is missing data, not zero; narrow the window.\n`credit_insufficient` → nothing charged; the user can add credits in the Omneky app.\n`needs_user_decision` → `request_user_decision`; never dump JSON.\nMissing tool → say so; Omneky app. Do not invent it.\nNever ask for internal ids. Never silent-switch `brand_id`.\n\n## Gaps\n\nThis connector cannot do comment/social inbox, landing pages,\nsubtitle burn-in, email/SMS, organic scheduling, Meta Ad Library\ncompetitor ads, virtual try-on, or website builder. Decline those.\nSee [capability-gaps](../_shared/references/capability-gaps.md).\nThere is no sequential-testing platform tool. Early stop is\nacknowledgement + correction, or not supported.\n","after":"---\nname: omneky-ab-tests\ndescription: >-\n  Create or measure an Omneky paid-media A/B test. Use for A/B\n  test creation (one hypothesis, one variable, precommitted\n  plan, launch live or paused as the user chooses) or A/B test measurement (z-test, CI,\n  power, SRM, interim peek with no verdict, refuse a thin\n  winner). Do not use for generic ROAS tables\n  (omneky-analytics-and-catalogue), competitor analysis\n  (omneky-competitor-analysis), first-run onboarding\n  (omneky-onboarding), agency (omneky-agency), weekly marketing\n  plans (omneky-marketing-plan), generating creatives\n  (omneky-image-ads, omneky-product-video, omneky-avatar-ugc),\n  or credit balance (omneky-billing). Keywords: A/B test,\n  ab test create, ab test measurement, which variant won,\n  significance, two-proportion z-test, MDE, SRM, interim peek,\n  underpowered, commercially meaningful, budget split.\n---\n\n# A/B tests\n\nUse this when the user wants to set up an A/B test or read out which\nvariant won. One job: own the test plan and the readout.\n\nGate every tool on this session's `tools/list`. If a step's tool is\nmissing, say so and point to the Omneky app.\n\nPlan fields: [handoff-contract](../_shared/references/handoff-contract.md).\nFormulas: [ab-test-statistics](../_shared/references/ab-test-statistics.md).\nSourced ids: [preconditions](../_shared/references/preconditions.md).\nAcceptance gates: [creative-quality-gates](../_shared/references/creative-quality-gates.md).\nBrief: [creative-brief](../_shared/references/creative-brief.md).\nOther intents: [capability-registry](../_shared/references/capability-registry.md).\nWrite for: [marketer-persona](../_shared/references/marketer-persona.md).\nStep detail: [playbook](references/playbook.md) (Hard rules, Create, Measure).\n\n## Goal\n\nCreate a one-variable test with a precommitted plan (live or paused is\nthe user's choice in the host), or\nmeasure it with a two-proportion z-test and a marketer verdict.\n\n## When to use\n\n- Create an A/B test, measure one, \"which variant won\", significance,\n  stop rule, interim peek, SRM, MDE\n\n## When not to use\n\n- Generic ROAS table → `omneky-analytics-and-catalogue`\n- Launch without a test plan → `omneky-launch-and-connect`\n\n## Prerequisites\n\n| Need | Check | If missing |\n|---|---|---|\n| Brand | `list_brands` | Ask which |\n| Channel | `list_connections` (`connector`) | Connect or managed |\n| Budget floor | `get_launch_requirements` | Do not launch under the floor |\n| Variants | prior creative handoff | Owning creative skill |\n| Baseline | `get_performance_summary` with `date_range` | State an assumed `baseline_rate` |\n| Coverage (measure) | `get_performance_summary` (`include_coverage=true`) | Do not declare a winner |\n| Metrics (measure) | `get_performance_breakdown` / `get_performance_summary` | Narrow the window |\n\n## Flow\n\n**Create**\n1. Write one hypothesis with one changed variable.\n2. Record metric, MDE, duration (`horizon`) and stop rule before launch.\n3. Budget at or above the floor. One ad group with one ad per variant\n   does not guarantee an even spend split (delivery is auction-driven);\n   say so. For an even split, use one ad group (or campaign) per variant\n   with equal budgets, or the platform's A/B / split-test feature.\n4. Launch through `omneky-launch-and-connect`, live or paused as the\n   user chose (the host confirms).\n\n**Measure**\n1. Confirm coverage for the window; empty is missing data, not zero.\n2. Run the setup check on every look (broken setup; SRM only for a\n   randomized planned allocation, not default auction delivery). If it\n   fails, give the verdict Invalid at once and recommend a pause.\n   Never mutate here.\n3. Before `horizon`, report INTERIM with no statistical verdict.\n4. At `horizon`, run the z-test and give one verdict: Winner, Not\n   commercially meaningful, Inconclusive, or No detectable difference.\n5. Refuse a winner when the sample is thin.\n\n## Failure and recovery\n\nAuth / 401 → `omneky-getting-started-and-failures` (host OAuth).\nEmpty reporting → `get_performance_summary` (`include_coverage=true`); empty is not zero.\n`credit_insufficient` → nothing charged; state balance vs required\n(`omneky-billing`). No payment links.\n`needs_user_decision` → fill the missing detail and call again; ask in plain chat only if you cannot. Never dump JSON.\nMissing tool → this session may run an older published app snapshot: look\nthe name up in [tool-name-fallbacks](../_shared/references/tool-name-fallbacks.md)\nand call the older name if that one is on `tools/list`. If neither is\nlisted, say so; Omneky app. Do not invent it.\nNever ask for internal ids. Never silent-switch `brand_id`.\n\n## Gaps\n\nDecline social inbox, landing pages, subtitle burn-in, email/SMS,\norganic scheduling, Ad Library ads, try-on, and website builder\n([capability-gaps](../_shared/references/capability-gaps.md)).\nThere is no sequential-testing platform tool. Early stop is\nacknowledgement + correction, or not supported.\n"}],"summary":"Fields changed: 3. /description, /included_files, /skill_md_contents.","summary_kind":"deterministic","summary_metadata":{}}