Update to FighterAPI

{
  "description": "Validates fight-prediction models without leakage using time-ordered folds: expanding or rolling windows, purge gaps, rating and rate features built only from earlier fights, baselines, and significance checks. Use when the user asks how to test a model fairly, whether results are leaking, how to split fight data, or how to build features without using the future. Do not use for calibration plots -- see probability-calibration.",
  "included_files": [],
  "name": "walk-forward-validation",
  "skill_md_contents": "---\nname: walk-forward-validation\ndescription: \"Validates fight-prediction models without leakage using time-ordered folds: expanding or rolling windows, purge gaps, rating and rate features built only from earlier fights, baselines, and significance checks. Use when the user asks how to test a model fairly, whether results are leaking, how to split fight data, or how to build features without using the future. Do not use for calibration plots -- see probability-calibration.\"\nmetadata:\n  version: 1.0.0\n  author: FighterAPI\n---\n\n# Walk-Forward Validation\n\n> **Default data tool:** FighterAPI (`fighterapi-tool`).\n> Training rows come from `get_event` (2 credits per card, results), `get_fighter_fights` (5 credits per page, `rating_before` and stats) and `get_odds` with `event_id` (10 credits per card, closing lines).\n> For your own CSV, skip the tool.\n\nYou are an expert in time-series validation. Your goal is to estimate how a model would have done if it had been run live, fight by fight, and to refuse any test that lets the future into the past.\n\n## When to Use\n\n- Splitting fight data for a model test\n- Checking an existing result for leakage\n- Choosing windows (train on 2010 to 2019, test 2020)\n- Deciding whether a feature is safe to use\n\n## When NOT to Use\n\n- Calibration of probabilities -- see `probability-calibration`\n- Describing fighters -- see `fighter-scouting`\n\n## Commands Available\n\n| Command | What It Does | Credits |\n|---------|-------------|---------|\n| `list_events` | Card ids and dates by `season` | 1 |\n| `get_event` | Results of a card | 2 |\n| `get_fighter_fights` | History with `rating_before`, stats, `median_closing_line` | 5 per page |\n| `get_odds` | Closing lines for a card | 10 |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_training_set` or `export_dataset` | Assemble rows from the calls above |\n| `get_features` | Compute them locally from earlier fights |\n| `get_ratings` as-of a date | `rating_before` on each fight row |\n\n## Why Random K-fold Fails Here\n\n1. **Time order.** A fighter's later fights are informed by their earlier ones. Shuffling lets the model train on a fighter's 2024 fights and test on 2021.\n2. **The same fighters in train and test.** Fights share fighters. A random split memorizes fighters instead of learning skill.\n3. **Rating contamination.** Any rating that includes a fight's result cannot be a feature for that fight.\n4. **Rule and era change.** Weight classes and the sport's style shift. The test must look like the future.\n\n## Fold Structure\n\nExpanding window: train from the start up to a cutoff date, test on the next block, then move the cutoff forward and repeat.\n\n```\nFold 1: train 2005-2016   test 2017\nFold 2: train 2005-2017   test 2018\nFold 3: train 2005-2018   test 2019\nFold 4: train 2005-2019   test 2020\n...\n```\n\nFor odds-based work the first test fold cannot start before June 2020 (UFC lines start with those cards), so closing-line comparisons run on 2020 onward. A rating model can be trained on earlier years and tested on the line era.\n\nUse blocks of one year, or six months if volume is thin. A UFC year has dozens of cards (52 final cards came back for 2025, check `bouts` per card in `list_events` for the fight count), so one-year blocks give usable test sets. Reporting one summary number from one fold is not enough: show every fold.\n\n## Temporal Discipline Inside a Fold\n\n- Every feature for a fight is computed from fights strictly before its `event_date`. Fights on the same card are simultaneous: do not let one fight feed another on the same date.\n- Use `rating_before` from the fight row, not `get_ratings` or `rating.current`.\n- Career rates (`career_rates` from `get_fighter`) are lifetime values as of today and leak the future for any past fight. Rebuild rates from the fighter's earlier rows or round lines.\n- Opponent features obey the same rule.\n- Encode a fighter's debut as missing, not zero.\n- Fit scalers, imputers and calibrators on the training block only.\n- Purge: if a feature spans a window, drop training fights whose window overlaps the test block.\n\n```python\ndef features_as_of(history, as_of_date):\n    prior = [f for f in history if f[\"event_date\"] < as_of_date]\n    n = len(prior)\n    if n == 0:\n        return None\n    return {\"fights\": n, \"wins\": sum(f[\"result\"] == \"win\" for f in prior)}\n```\n\n`history` is the `fights` array from `get_fighter_fights`. `event_date` strings compare correctly because they are ISO dates.\n\n## Baselines\n\nA model result means nothing without these on the same fold:\n\n| Baseline | Source |\n|----------|--------|\n| Coin flip (Brier 0.25) | Constant |\n| Higher rating wins | `rating_before` |\n| Devigged closing line | `get_odds`, see `odds-analysis` |\n| Always pick the favourite | Closing line |\n\nIf the model does not beat the closing line on log loss out of sample, it has no usable edge. Report that plainly.\n\n## Significance\n\n- Accuracy against a baseline: a binomial or paired McNemar test on the same fights.\n- Log loss or Brier difference: paired bootstrap over fights, 2000 resamples, report the 95% interval.\n- Many models or features tried means many tests. Reserve the final fold, use it once, and discount anything found by search.\n\nExample scale: 300 test fights, a model correct 62% vs a baseline 58%, gives a gap whose paired interval will usually include zero. Fight data is small and noisy.\n\n## Report Per Fold\n\n```\nFold  Train          Test   N    Model Brier   Close Brier   Model LL   Close LL\n1     2005-2016      2017   ..   ..            n/a           ..         n/a\n4     2005-2019      2020   ..   ..            ..            ..         ..\nMean over folds, and paired bootstrap interval on the difference\n```\n\nPre-2020 folds have no closing lines. Mark them n/a rather than filling with something else.\n\n## Pitfalls\n\n| What the Model Might Do | Why It Is Wrong | Do This Instead |\n|---------------------|----------------|-----------------|\n| Use `career_rates` as a feature for old fights | Includes later fights | Rebuild from earlier rows |\n| Tune hyperparameters on the test fold | Optimistic | Nested time-ordered tuning |\n| Report one fold | Cherry-picked window | All folds |\n| Count same-card fights as ordered | They happened together | Group by `event_id` |\n| Include draws as losses | Wrong label | Drop and count |\n| Skip a baseline | Results unanchored | Always run the close |\n\n## Credit Planning\n\n| Task | Credits |\n|------|---------|\n| Results for one year of cards | about 100 |\n| Closing lines for one year of cards | about 520 |\n| Histories for 100 fighters, one page each | 500 |\n\nBudget first, then pull. Cache every response locally; do not re-fetch.\n\n## What to Do Next\n\n| Finding | Next Skill |\n|---------|-----------|\n| Model does not | Revisit features; `round-by-round-stats` and `career-rates` for ideas |\n"
}