{"id":71884,"plugin_id":"plugin_asdk_app_6ad146628be48191826f84406ff2c4c1","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-10-30T00:06:19.903Z","digest":"1d24cf42bf04c11232cea9fe8dd2fa3d8f798f5c56f26c774b6241ef0207c680","against":null,"payload":{"description":"Evaluates soccer prediction models without leaking the future: walk-forward and expanding-window splits by matchweek or season, point-in-time features, leakage checks specific to standings, season stats and ratings, and significance testing of a model against the closing line. Use when the user asks whether cross-validation is okay, how to split match data, whether their backtest leaked, or how to compare two models. Do not use for probability scaling, see probability-calibration.","included_files":[],"name":"walk-forward-validation","skill_md_contents":"---\nname: walk-forward-validation\ndescription: \"Evaluates soccer prediction models without leaking the future: walk-forward and expanding-window splits by matchweek or season, point-in-time features, leakage checks specific to standings, season stats and ratings, and significance testing of a model against the closing line. Use when the user asks whether cross-validation is okay, how to split match data, whether their backtest leaked, or how to compare two models. Do not use for probability scaling, see probability-calibration.\"\nmetadata:\n  version: 1.0.0\n  author: StrikerAPI\n---\n\n# Walk-Forward Validation\n\n> **Default data tool:** StrikerAPI (`strikerapi-tool`).\n> Results history comes cheap from `list_games` (2 credits for up to 200 matches). `get_ratings` with `date` gives point-in-time Elo (2). Most StrikerAPI calls return the state as of today, so rebuilding features as of an earlier date from match logs is usually the safe path.\n\nSoccer data is a time series with a regime change every August. A random k-fold split uses next year's results to predict last year's matches, which makes any model look better than it will be. Walk-forward validation trains on the past, predicts the next block, then steps forward.\n\n## When to Use\n\n- \"Can I use 5-fold cross-validation on matches?\" (No)\n- \"Is my model any good?\"\n- \"Did my backtest leak?\"\n- Choosing between two models, two feature sets or two hyperparameters\n- Tuning an Elo or a decay rate\n\n## When NOT to Use\n\n- Turning scores into calibrated probabilities: `probability-calibration`\n\n## Commands Available\n\n| Command | What It Does | Credits |\n|---------|-------------|---------|\n| `list_games` | Results by `season`, `from`/`to`, `status` for the training stream | 2 per call |\n| `get_ratings` | Elo as of a `date` | 2 |\n| `get_game` | Per-match xG and box score (5 each, use sparingly) | 5 |\n| `get_shot_map` | A club-season of shots with `game_id` and `date`, so you can rebuild xG per match | 10 |\n| `get_odds` | Closing prices for the benchmark, `season` mode | 2 per match |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_standings_as_of` | Not available. `get_standings` returns the table as it stands now (or at season end for a past season); rebuild an as-of table from `list_games` |\n| `get_team_stats_as_of` | Not available. `get_team_stats` averages the whole season; rebuild as-of averages from per-match data |\n| `get_form_history` | Rebuild from `list_games` |\n\n## The Split\n\n**Expanding window (default).** Train on everything before block `t`, predict block `t`, add block `t` to the training set, step.\n\n**Rolling window.** Same, but drop blocks older than N seasons. Use if the league's style or your model's parameters drift.\n\nBlock choice:\n\n| Block | When |\n|-------|------|\n| Matchweek (roughly 10 matches) | Models that update weekly: Elo, time-decayed Poisson |\n| Month | Cheaper retrain |\n| Season | Models with hard-to-refit parameters, and for the final report |\n\n```python\ndef walk_forward(matches, first_test_date, step_days=7, fit=None, predict=None):\n    matches = sorted(matches, key=lambda m: m[\"starts_at\"])\n    out, t = [], first_test_date\n    while t <= matches[-1][\"starts_at\"][:10]:\n        train = [m for m in matches if m[\"starts_at\"][:10] < t]\n        test = [m for m in matches if t <= m[\"starts_at\"][:10] < add_days(t, step_days)]\n        if train and test:\n            model = fit(train)\n            out += [(m, predict(model, m)) for m in test]\n        t = add_days(t, step_days)\n    return out\n```\n\n(`add_days` is a one-line date helper.) Keep the loop identical for every model you compare.\n\n## Leakage Checklist for Soccer\n\n| Leak | How it happens | Fix |\n|------|---------------|-----|\n| Table position | `get_standings` returns the table as of now, not as of the match date. Using it as a feature for a match in November uses May's table | Rebuild the as-of table from results |\n| Season xG and team averages | `get_team_stats` averages include the match you predict and everything after it | Compute rolling averages from per-match xG, shifted by one match |\n| Current Elo for past matches | `get_ratings` without `date` is today's rating | Use the rating before the match, or your own point-in-time Elo |\n| Player season totals | `get_player` rows are whole-season | Build from `get_player_games`, shifted |\n| Closing odds as a feature | If you predict before the close, the close is the future | Use opening or earlier snapshots only |\n| Same-match rows | Both clubs' rows from one match split across train and test | Keep a match intact on one side |\n| Promoted clubs | A club with no history gets a fabricated one | Give them a prior, flag them |\n| Tuning on the test blocks | Choosing hyperparameters from the final scores | Tune inside the training window, nested walk-forward |\n| Mid-season managerial or squad change | Treated as stationary | Use decay, check for breaks |\n\nShift rule: any feature for match `m` uses only matches dated strictly before `m`'s kickoff. If two matches share a day, treat same-day results as unavailable.\n\n## Metrics\n\nFor three-way outcomes (home, draw, away) with probabilities `p` and the one-hot outcome `o`:\n\n```\nLog loss = - log(p[actual])                                 mean over matches\nMulticlass Brier = sum_k (p_k - o_k) ** 2                   mean over matches\nRanked probability score (RPS) = sum over the 2 cumulative cut points of (cum_p - cum_o) ** 2 / 2\n```\n\nRPS respects the order home, draw, away, so predicting a draw when the home side wins is penalized less than predicting an away win. It is the standard metric for soccer. Compare each model with the market's devigged closing probabilities from `odds-analysis`; the gap in RPS is what you are after. Accuracy is a poor choice: a model that never picks the draw can still score 50%.\n\n## Significance\n\n1. Compute per-match loss for model A and the benchmark B on the same matches.\n2. Take the difference `d_i = loss_A_i - loss_B_i`.\n3. Mean and standard error: `se = std(d) / sqrt(n)`. A t-statistic above about 2 is suggestive, not proof.\n4. Or block-bootstrap `d` by matchweek (to keep within-week correlation) and read the 95% interval.\n\nHow much is enough? One league-season is 380 matches, which is not many for comparing two reasonable models whose per-match scores differ by small amounts. Measure the standard deviation of `d` on your own data and report the standard error rather than quoting a rule of thumb; expect that small differences will not be separable within a single season. If you tested several models or settings, apply a multiple-comparison correction or hold out a final untouched season.\n\n## Worked Example Plan\n\nEvaluate a Dixon-Coles model (`poisson-score-models`) against Pinnacle's closing prices for 2023-24 to 2025-26:\n\n1. Pull results: two `list_games` calls per season (12 credits for three seasons).\n2. Pull closing prices: `get_odds` with `season` and 380 matches is 760 credits per season. Budget this first; consider one season, or restrict to a team's matches with `team`.\n3. Run `walk_forward` with weekly steps starting at matchweek 11 each season (so ratings have data).\n4. Devig Pinnacle's close with `odds-analysis`.\n5. Report mean RPS for the model, the market, and the difference with its standard error.\n\nA common outcome: the market's RPS is lower than a goals-only model's. The honest conclusion is to find where the model differs from the market and test those differences on their own.\n\n## Pitfalls\n\n1. **Random splits.** Never. Even shuffled-within-season is leakage.\n2. **Few early-season matches.** Starting each season with an empty model gives poor early predictions. Carry ratings over with regression to the mean.\n3. **Changing the rules mid-experiment.** Define the metric, block size and start date before you look.\n4. **Selection by test score.** Picking the best of 20 settings by the test score inflates it. Hold out a final season.\n5. **Ignoring cup breaks and international windows.** They are not leakage but they change the nature of the next block.\n6. **Draw prediction.** Judge it separately in the calibration table, not just via accuracy.\n\n## Anti-patterns\n\n| Rationalization | Why It's Wrong | Do This Instead |\n|----------------|---------------|-----------------|\n| \"5-fold CV is standard\" | Future leaks into the past | Expanding window |\n| \"I used the standings as a feature\" | They were today's standings | Rebuild as-of |\n| \"Accuracy improved 2 points\" | Within noise on 380 matches | Compare RPS with a standard error |\n| \"Great test score, I tuned on it\" | Optimistic | Nested walk-forward and a final holdout |\n\n## Output Format\n\n```\nModel [name] vs [benchmark], [competition], [first test date] to [last], n = [matches]\nRPS: model [x.xxx], benchmark [x.xxx], difference [x.xxx] +/- [se]\nLog loss: model [x.xxx], benchmark [x.xxx]\nLeakage checks passed: [list]\nVerdict: [better / indistinguishable / worse], with the sample caveat\n```\n\n## What to Do Next\n\n| What You Found | Next Action | Skill |\n|----------------|-------------|-------|\n| Model competitive with market | Check calibration | `probability-calibration` |\n| Model clearly worse | Go back to features | `poisson-score-models`, `elo-ratings` |\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}