Update to Keel

{
  "description": "Probe whether a backtest result is robust or curve-fit: re-run the strategy on disjoint windows and regimes the user agrees to, optionally perturb the sensitive parameters, compare the runs once, and give one verdict (ROBUST / MIXED / OVERFIT). Use when the user asks \"is this real?\", \"is this overfit?\", \"out-of-sample\", \"walk-forward\", or asks for a robustness check; the check is most informative on an unusually strong in-sample result (Sharpe above 3.0, return above 500%). Not a substitute for the first backtest (backtest-and-analyze).\n",
  "included_files": [],
  "name": "overfit-check",
  "skill_md_contents": "---\nname: overfit-check\ndescription: |\n  Probe whether a backtest result is robust or curve-fit: re-run the strategy on disjoint windows and regimes the user agrees to, optionally perturb the sensitive parameters, compare the runs once, and give one verdict (ROBUST / MIXED / OVERFIT). Use when the user asks \"is this real?\", \"is this overfit?\", \"out-of-sample\", \"walk-forward\", or asks for a robustness check; the check is most informative on an unusually strong in-sample result (Sharpe above 3.0, return above 500%). Not a substitute for the first backtest (backtest-and-analyze).\nreferences:\n  - step: 2\n    topic: \"platform_operations\"\n    why: \"the backtest window rule: warm-up and the data floor for each probe window\"\n  - step: 3\n    topic: \"trading_domain\"\n    why: \"complexity ladder (>25 free parameters is overfitting), which parameters are the sensitive ones\"\n  - step: 4\n    topic: \"reasoning_principles\"\n    why: \"each transform once; signal diversity over parameter variants when reading a fragile result\"\ntools:\n  - keel_backtest_run\n  - keel_backtest_watch\n  - keel_backtest_compare\n---\n\n# References\n\nReference topics this workflow cites (keel_help topics):\n\n- Step 2 — `platform_operations` — the backtest window rule: warm-up and the data floor for each probe window\n- Step 3 — `trading_domain` — complexity ladder (>25 free parameters is overfitting), which parameters are the sensitive ones\n- Step 4 — `reasoning_principles` — each transform once; signal diversity over parameter variants when reading a fragile result\n\n# Method\n\n## Step 1 — Confirm the baseline\n\nThe `strategy_id`, the baseline `run_id` (or run one on the default window),\nand its window. State the baseline Sharpe / return / max DD before probing.\n\n## Step 2 — Agree the out-of-sample windows\n\nThe probe takes several runs — one backtest of the plan's weekly allowance\neach, plus wall-clock time. Before starting, say how many runs it needs and\nwhich windows, and start once the user agrees; they may drop or pick\nwindows. Windows the user named are used as given. When the user asks for a\ndefault split, this one is relative to the baseline window:\n\n- W1 in-sample — its first half;\n- W2 out-of-sample — its second half;\n- W3 held-out tail — the most recent 3 months;\n- W4 / W5 regimes — one high-vol stretch, one low-vol / chop stretch.\n\nEvery window must cover the strategy's warm-up. For each agreed window:\n`keel_backtest_run(strategy_id=<id>, start_date=<start>, end_date=<end>)`; if still active,\n`keel_backtest_watch(backtest_id=<run_id>)`. When all are terminal,\n`keel_backtest_compare` once with every window's id — the table is the\ncomparison view.\n\n## Step 3 — Parameter perturbation (only when the user asks for depth)\n\nPick the 1–2 parameters that matter (a lookback, a threshold, a leverage\ncap — not a seed). Perturb ±20% and ±50% — four runs per parameter, stated\nbefore starting — and compare once. A Sharpe that collapses under ±20% points\nto a fitted parameter rather than an edge.\n\n## Step 4 — Read stability\n\n- Sharpe spread across windows: max − min > 2.0 with any window negative is\n  fragile. Real edges degrade smoothly.\n- Sign of returns: one negative disjoint window is a yellow flag; two are red.\n- Parameter sensitivity: robust to ±20% or it is not an edge.\n\n## Step 5 — Verdict\n\n- **ROBUST** — stable Sharpe, positive across regimes, parameters not\n  razor-thin.\n- **MIXED** — some windows weak, thesis holds in most; a candidate for\n  further out-of-sample testing.\n- **OVERFIT** — variance too high, negative OOS, or razor-thin parameters.\n\nOne sentence, then the windows / perturbations that drove it.\n\n# Decision points\n\n- Disjoint is not cherry-picked: never place the OOS window on a known good\n  period.\n- An average across windows hides a −3.0 window; read every row.\n- A ROBUST verdict is necessary, not sufficient — it describes historical\n  windows only.\n- A fragile result is answered with robustness, not with less exposure: a\n  trend filter, sizing, or a regime gate, rather than removing exposure to\n  dodge one bad window — that is curve-fitting.\n\n# Output shape\n\n1. The verdict sentence.\n2. The comparison `view` (all windows) — explained, not repeated, where the\n   host draws it.\n3. The perturbation comparison, if run.\n4. Two bullets of interpretation.\n5. What the verdict implies: further out-of-sample testing / iterate further\n   / discard.\n\n# When NOT to use\n\n- First-pass backtest → `backtest-and-analyze`; a strategy one run shows is\n  broken gains nothing from five more.\n- Authoring or editing → the creation or fork skills.\n\n# Test prompts\n\n1. \"Is this 3.5 Sharpe real, or am I overfitting?\"\n2. \"Run an out-of-sample check on str_K9p2Lz.\"\n3. \"Walk-forward this strategy across the last year.\"\n"
}