Fiera Incrementality

View saved version →

---
name: geo-test-design
description: Design a geo experiment (geo lift, matched market, regional holdout, go-dark, holdback or heavy-up) from a CSV of KPI by region and day. Chooses treated regions, estimates the minimum detectable effect, test length and cost, and writes the test plan. Use when the user uploads or describes geo-level data and wants to set up a test.
---

# Geo test design

Designs a geo experiment from the user's data with the bundled script, then writes a plan the team can approve before launch.

The user's explicit instructions take priority over this guidance. Reply in the user's language.

## Inputs

- A CSV with one row per geo and day: date, geo, KPI, and ideally spend and value. Format in `references/data-format.md`.
- Test type: go-dark, holdback (with the share cut) or heavy-up (with the share added). Default go-dark if the question is "is it incremental"; heavy-up if the question is "should we spend more".
- Test length in days. If unknown, start with 28 and use the duration curve.
- Constraints: geos that must stay on, geos that must be treated, maximum share of KPI at risk.

If the file is missing, ask for it and say exactly what to export. If the file has the wrong shape (wide format, weekly totals only, no geo), reshape it with pandas before running the script. Tell the user what was changed.

## Steps

1. Run `python scripts/geolift.py profile --data <file> --out profile`. Read the warnings. Stop and ask when the panel cannot support a test: fewer than 6 geos, under 8 weeks of history, a geo with over a third of the KPI, broken tracking periods.
2. Run `python scripts/geolift.py design --data <file> --test-days <n> --type <type> [--intensity x] [--treated-share s] [--must-treat ...] [--never-treat ...] --out design`.
   - Default treated share 0.2. Raise it when the MDE is too high and the business accepts more risk.
   - If the user already chose the treated geos, pass `--treated` to evaluate their choice, and also run the search to show the alternative.
3. Read `recommended.mde_pct_validation`, not the selection MDE. Compare it with the effect the decision needs. Use `duration_curve` to propose a length.
4. If the MDE is too large, try in this order: longer test, larger treated share, finer geos, heavy-up instead of holdback. Rerun and report the change.
5. Fill `assets/test-plan-template.md` with the design output and the execution checklist from `references/geo-design-rules.md`, adapted to the platforms in scope.

## Output

- The treated geos, their share of KPI, and the MDE at the proposed length.
- The duration curve as a short table.
- Cost: spend withheld or added, and KPI at risk under two or three plausible effect sizes.
- Warnings from the script, translated into what they mean for this test.
- The filled test plan.
- Keep the seed and the full treated list in the plan: the readout needs them.

## Rules

- Never fabricate data or results. Every number comes from the user's file or the script output.
- Do not present the MDE as the expected effect. It is the smallest effect the test can see.
- The plan is written before launch and not changed after the data comes in.
- Platform settings that matter: Presence-only location targeting in Google Ads, exclusions in every automated campaign type, a freeze on other changes. Include them in the checklist.

## Scripts and references

- `scripts/geolift.py`, documented in `references/script-reference.md`.
- `references/geo-design-rules.md` for test types, units, duration, spillover and execution.
- `references/literature.md` for the methods (TBR, synthetic control, GeoLift, Meridian GeoX) and their published requirements.
- If code execution is not available, give the user the exact commands to run locally.

Referenced files: 12 — file references are metadata, not package contents.

View saved version →

---
name: geo-test-readout
description: Analyze a finished geo experiment or regional holdout from a CSV of KPI by region and day. Estimates the incremental effect with an interval, incremental CPA or ROAS, and runs robustness checks. Use when the user has run a geo test, regional go-dark, brand search blackout by region, or heavy-up and wants the result.
---

# Geo test readout

Estimates the effect of a geo experiment with the bundled script and reports it in a form a team can decide on.

The user's explicit instructions take priority over this guidance. Reply in the user's language.

## Inputs

- The CSV with the KPI by geo and day, covering the pre-period and the test (and cooldown). Spend by geo if available, value if available.
- Treated geos, test start, and the last day to measure (end of cooldown).
- Test type, and the test plan if one exists.

Ask for whatever is missing. If there is a test plan, use its treated list, dates and decision rule exactly.

## Steps

1. Run `python scripts/geolift.py profile --data <file>` and check that the treated geos exist and the dates cover the window.
2. Run `python scripts/geolift.py readout --data <file> --treated <geos> --start <date> --end <date> --type <type> --out readout`.
   - Add `--exclude-controls` for neighboring geos with spillover risk if the plan named a buffer.
3. Read `references/reading-results.md` and apply it:
   - Report effect, lift and interval from `primary`. Note `interval_kind`.
   - Compare with `secondary` (the other estimator). Disagreement means a fragile result.
   - Go through `checks` and `warnings`, and say what each failed check means.
   - Compute or report iCPA or iROAS from `money`, and set them next to the platform-reported CPA or ROAS if the user has them.
4. Apply the decision rule from the plan. If there was no plan, apply the decision mapping in `references/reading-results.md` and say the rule was set after the fact.
5. Show the chart (`readout.png`) and offer the daily series (`readout_daily.csv`).
6. Write the readout with `assets/readout-template.md`.

## Output

- First line: effect and interval, and what it means for the decision.
- Incremental cost or return against the reported one.
- Robustness table.
- Limits: what the result does not cover.
- Next step.

## Rules

- Never report a point estimate without its interval.
- An interval that includes zero is not proof of no effect. Give the MDE context.
- Do not rerun with different dates, geos or methods until the result looks better. If a change is justified (a tracking outage, a geo that leaked spend), make it once, say why, and report both versions.
- A go-dark with spend leakage in treated geos underestimates the effect; report the leakage.

## Scripts and references

- `scripts/geolift.py`, documented in `references/script-reference.md`.
- If code execution is not available, give the user the exact command to run locally.

Referenced files: 12 — file references are metadata, not package contents.

View saved version →

---
name: incrementality-planner
description: Explain incrementality and choose how to measure whether an ad channel, campaign type or budget change drives incremental results, then plan the test. Use when someone asks what incrementality, iROAS, lift or a holdout means, if a channel is incremental, whether to pause or scale ads (brand search, PMax, Meta, TikTok, YouTube), how to run a holdout, lift study, geo test or incrementality test, or how much of their attributed conversions are real.
---

# Incrementality planner

Turns a question about ad effectiveness into a measurement plan: which method, which data, how long, what it costs, and what decision it will support.

The user's explicit instructions take priority over this guidance. Reply in the user's language.

## When the user asks what incrementality is

Answer in plain terms first, then offer to plan a test:

- Incrementality is the share of results that would not have happened without the ads. Attribution assigns credit for conversions that happened; incrementality asks which of them the ads caused.
- It is measured by comparing a group exposed to the ads with a comparable group that was not: users held out by the platform, regions switched off, or periods with ads off.
- iROAS is incremental revenue divided by the spend that caused it. iCPA is spend divided by incremental conversions.
- Give one example from `references/literature.md` if useful (eBay brand search: 99.5% of lost clicks came back through organic results).

## Workflow

1. **Understand the decision.** Ask only what is missing, in one message, using the decision questions in `references/method-ladder.md`. If the user already gave the answers, do not ask again.
2. **Pick the method** with the ladder in `references/method-ladder.md`:
   - If the platform offers a native lift study and the account is likely eligible, recommend it first and explain how to request it. Details in `references/platform-lift.md`.
   - If the question spans channels, or the channel is search, or the platform has no lift tool, recommend a geo experiment.
   - If geos are not possible, recommend a switchback; if nothing can be changed, a directional pre/post.
   - Always state the rung: causal, quasi-experimental or directional.
3. **Size it.**
   - With a geo panel: hand over to the `geo-test-design` skill, which runs the design script.
   - Without data: run `python scripts/geolift.py quick` with the user's daily KPI and a noise assumption, and show the MDE by test length. Say it is a rough sizing.
4. **Deliver the plan** using `assets/test-plan-template.md`, filled with what is known and marked placeholders for what is not.

## Output

- One-line recommendation: method, scope, duration, and why.
- What it will and will not answer.
- Data needed, with the source for each item (`references/data-format.md`).
- Cost in KPI or spend, with the assumption it rests on.
- The plan, or the next step to get the plan (usually: export the geo panel).

## Rules

- Do not invent volumes, costs, eligibility thresholds or dates. When a platform rule matters and may have changed, say so and point to where the user can confirm it in their account.
- Attribution reports, spend-KPI correlation and "we paused it and nothing happened" are not incrementality. Say so politely when offered as evidence, and propose the test that would settle it.
- Recommend the KPI from the business backend over platform-reported conversions.
- If the decision needs a precision the test cannot reach (the MDE is larger than the effect that matters), say so before the user spends money on it.
- Keep it short. A plan is a page, not an essay.

## Evidence

When the user doubts that testing is needed, or asks what to expect, cite findings from `references/literature.md`. Examples: the eBay brand search experiment, or the Facebook comparison of observational methods with experiments. Quote only figures that appear there.

## Scripts

`scripts/geolift.py` is bundled with this skill. See `references/script-reference.md`. If code execution is not available, give the user the exact command to run locally (`python geolift.py ...`; it needs numpy, pandas and scipy).

Referenced files: 12 — file references are metadata, not package contents.

View saved version →

---
name: lift-result-review
description: Review an incrementality or lift result someone else produced, such as a Meta or Google lift study report, a vendor's geo test, an agency deck, or an internal claim that a channel is or is not incremental. Checks design, statistics and interpretation, and says how much weight the result can carry. Use when the user shares a lift result, a test report or a screenshot and asks whether to trust it.
---

# Lift result review

Gives an independent read on a lift or incrementality result: what it shows, what it does not, and whether it can support the decision being made.

The user's explicit instructions take priority over this guidance. Reply in the user's language.

## Steps

1. Identify what was run: platform lift study, geo experiment, switchback, pre/post, MMM output, or attribution presented as incrementality. Use `references/method-ladder.md` to place it on the ladder.
2. Extract the facts stated in the material: KPI, source, treated and control groups, dates, sample sizes, effect, interval or p-value, spend. Mark what is missing. Never fill gaps with assumptions presented as facts.
3. Check it against the list below.
4. If the user has the underlying data, offer to recompute it with the `geo-test-readout` or `quick-incrementality` skills.
5. Give the verdict.

## Checklist

Design:

- Was there a control group, and was assignment random (users or geos) or chosen?
- Was the plan, including the KPI and the decision rule, set before the data came in?
- Was the test long enough for the channel's carryover, with a cooldown for video and upper funnel?
- Platform lift: which conversions were counted? Pixel-only events miss offline and cross-device outcomes; was CAPI or offline upload used?

Statistics:

- Is there an interval, and does it include zero?
- What effect could the test detect (MDE or power)? A non-significant result from an underpowered test says little.
- Was the result picked from several metrics, segments or windows? If so, discount it.
- Geo tests: how many geos, how good was the pre-period fit, were placebos run?

Interpretation:

- Is "incremental conversions" being compared with attributed conversions to produce an incrementality factor? Is that factor applied to other campaigns or periods where it may not hold?
- Is a go-dark result being used to justify a budget increase (a different question)?
- Are lift results from different platforms being added together?
- Is "lift" relative to the treated group's baseline, or to all conversions? Percentages on different bases are not comparable.

## Verdict format

- One line: how much weight the result can carry (decision-grade, directional, or not usable), and why.
- What the result supports, and what it does not.
- Missing information that would change the verdict.
- The smallest next step that would make it decision-grade.

## References

- `references/platform-lift.md` for what Google and Meta lift studies measure and their thresholds.
- `references/literature.md` for methods and published evidence.
- `references/reading-results.md` for how results should be stated.

## Rules

- Be specific and fair: name the issue and its likely direction (overstates, understates, unknown).
- Do not dismiss platform lift studies as biased by default. A randomized user-level holdout is a strong design; its limits are scope (one platform, its own conversion data), not randomization.
- Do not invent numbers that are not in the material.

Referenced files: 12 — file references are metadata, not package contents.

View saved version →

---
name: quick-incrementality
description: Get a directional, low-effort read on incrementality without a formal experiment. Compares a period before and after a change (pause, budget cut, new campaign) against several baselines and their normal noise, or analyzes an on/off switchback in a single market. Use when a team paused or changed ads and wants to know if it mattered, cannot split by region, or wants a quick check before investing in a proper test.
---

# Quick incrementality

Directional methods for teams that cannot run a geo experiment or need a fast read. The output is always labeled directional and states what it cannot rule out.

The user's explicit instructions take priority over this guidance. Reply in the user's language.

## Choose

| Situation | Method |
|---|---|
| A change already happened, one date | `prepost` |
| The team can alternate on and off for 4+ blocks | `switchback` (plan it, then analyze it) |
| No data yet, need to know if a test is feasible | `quick` sizing |
| Brand search pause | See the brand section in `references/directional-methods.md`, prefer switchback |

## Steps for prepost

1. Ask for a daily CSV of the business KPI (not platform conversions), at least 8 weeks before the change, ideally over a year. Ask for spend, and for a control series if one exists: another country, organic-only, a product without ads.
2. Run `python scripts/geolift.py prepost --data <file> --start <change date> [--end <date>] [--control-col <col>] [--exclude-windows a:b] --out prepost`.
3. Report each comparator: change, its normal range, and how often past windows moved more. Then the verdict.
4. If the verdict is "directional signal", give the size as a range across comparators, never as one number.
5. Say what else changed in the window, if the user knows, and how it could explain the result.
6. Recommend the test that would confirm it (`references/method-ladder.md`), with a rough sizing via `quick` if useful.

## Steps for switchback

1. Plan: block length 1 to 2 weeks, at least 6 blocks, washout 2 to 3 days, schedule fixed before starting. Use only for channels without long carryover.
2. Analyze: `python scripts/geolift.py switchback --data <file> --off-periods a:b c:d ... --start <first block> --end <last block> --out switchback`.
3. Report the effect when off, the permutation p and the minimum attainable p.

## Rules

- Label every result "directional". Never call it causal.
- Never give one number without the noise band next to it.
- If the change is inside the normal band, say the data cannot tell it apart from a normal period. Do not say the channel has no effect.
- Do not mix platform-attributed conversions with the business KPI.

## Scripts and references

- `scripts/geolift.py`, documented in `references/script-reference.md`.
- `references/directional-methods.md` for the comparators and their failure modes.
- If code execution is not available, give the user the exact command to run locally.

Referenced files: 12 — file references are metadata, not package contents.