Update to FastballAPI

{
  "description": "Evaluates baseball prediction models honestly with time-ordered walk-forward splits: leakage-safe rolling features, expanding vs sliding windows, baselines (home field, Elo, closing line), log loss, Brier score and calibration checks. Use when the user asks how to validate a model, whether cross-validation is okay, how to engineer features without leakage, or whether a model's accuracy is real. Refuses random k-fold on games. Do not use for odds math: see odds-analysis. Do not use for prop pricing: see prop-modeling.",
  "included_files": [],
  "name": "walk-forward-validation",
  "skill_md_contents": "---\nname: walk-forward-validation\ndescription: \"Evaluates baseball prediction models honestly with time-ordered walk-forward splits: leakage-safe rolling features, expanding vs sliding windows, baselines (home field, Elo, closing line), log loss, Brier score and calibration checks. Use when the user asks how to validate a model, whether cross-validation is okay, how to engineer features without leakage, or whether a model's accuracy is real. Refuses random k-fold on games. Do not use for odds math: see odds-analysis. Do not use for prop pricing: see prop-modeling.\"\nmetadata:\n  version: 1.0.0\n  author: FastballAPI\n---\n\n# Walk-Forward Validation\n\n> **Default data tool:** FastballAPI (`fastballapi-tool`).\n> `get_games` (2 credits per call, up to 200 games) for results, `get_standings` (2) and `get_ratings` (2) for as-of-season context, `get_odds` (10 per date) for a market baseline. This skill is mostly method.\n\nYou make a model's score honest. Baseball is the hardest major sport to predict from team data: even the best clubs win about 60% of games, so the gap between a real model and noise is a couple of percentage points. Leakage or a bad split can create that much by itself.\n\n## When to Use\n\n- \"Is my win-probability model any good?\"\n- \"Can I use k-fold cross-validation on games?\" (no)\n- Choosing features and making sure they use only past information\n- Comparing a model with Elo, home field and the closing line\n- Checking whether predicted probabilities are calibrated\n- Setting up a train/test structure for props or totals\n\n## When NOT to Use\n\n- Computing fair probabilities from prices: `odds-analysis`\n- Pitcher or hitter prop math: `prop-modeling`\n- Team record luck: `pythagorean-record`\n\n## Commands Available\n\n| Command | What it does | Credits |\n|---------|--------------|---------|\n| `get_games` | Final scores by season, date range or team; `limit` up to 200, `offset` to page | 2 per call |\n| `get_standings` | Season standings (use only as-of-prior-season for features) | 2 |\n| `get_ratings` | Elo, \"as of the end of a season or today\" | 2 |\n| `get_odds` | Closing prices for a date (10) or a game (5), with `result` for finished games | 5 or 10 |\n| `get_team_stats` | Season team process stats | 5 |\n\n## Commands That Do NOT Exist\n\n| Not available | Use instead |\n|---------------|-------------|\n| `get_train_test_split`, `evaluate_model` | Implement the splits and metrics below |\n| `get_rolling_stats`, `get_team_form` | Build from `get_games` with a lag, as shown |\n| `get_elo_history` | `get_ratings` is end-of-season or today only. Build your own game-by-game Elo from `get_games` for as-of features |\n| `get_probable_pitchers` | Not available. Pitcher identity cannot be a pre-game feature from this API |\n| `get_lineups`, `get_weather`, `get_park_factors` | Not available |\n\nA hard limit to say up front: `get_ratings` returns one rating per team per call, as of a season's end or now. Using `get_ratings` for season N as a feature for games in season N is leakage. Use the previous season's end rating, or build a rolling Elo yourself.\n\n## Why Random K-Fold Fails\n\nRandom folds put games from the same week in train and test. A team's April and May games share its roster, its Elo and its streaks. The model sees the future of the team's season and the test score rises. Measure it: run both a random k-fold and a walk-forward split on the same data and report the gap. If the random score is higher, that gap is the leak.\n\nWalk-forward means every prediction is made with only information that existed before first pitch.\n\n## Splits\n\n**Expanding window (default).** Train seasons 2015 to N-1, test season N, for N from 2019 to the latest complete season. The train set grows each fold.\n\n**Sliding window.** Train on the last K seasons only, if the older data hurts (rule changes, the 2020 shortened season, pitch-clock effects). Test this with data, not by assumption.\n\n**Within a season (for in-season models).** Train through date D, predict D+1 to D+7, step forward weekly. Same rule: nothing after D enters the features.\n\nAlways report per-fold results, not just an average. Seasons differ.\n\n## Leakage-Safe Features\n\nEvery feature is built as of the start of the game.\n\n```python\nimport pandas as pd\n\ngames = games.sort_values([\"date\", \"game_id\"])\n# long format: one row per team per game\ng[\"runs_for_roll10\"] = (g.groupby(\"team\")[\"runs_for\"]\n                         .transform(lambda s: s.shift(1).rolling(10, min_periods=5).mean()))\n```\n\nThe `shift(1)` is the whole point: a rolling mean that includes the current game contains the answer. Rules:\n\n1. Shift before rolling or expanding.\n2. Season-level features (win pct, run differential) come from games before the date, not the final standings. Do not pull `get_standings` for the season being predicted.\n3. Reset or damp features at season boundaries (rosters change over the winter); carry over with shrinkage, not at full strength.\n4. Early-season rows with few prior games are noisy. Use `min_periods` and shrink toward the league mean: `(n * team_value + k * league_value) / (n + k)` with k picked by validation.\n5. Doubleheaders and same-day games: order by `starts_at`, and exclude a same-day game's result from the other game's features.\n6. Starting pitchers are usually the biggest input in baseball and this API has no probable-pitcher data. Team-only models are therefore structurally capped. Say that when reporting accuracy.\n\nCandidate features that are available and leakage-safe: rolling run differential, prior-season Elo, home/away flag, rest days (from schedule dates), rolling team batting OPS from `get_games`-derived runs and hits, bullpen usage from recent box scores.\n\n## Baselines the Model Must Beat\n\n| Baseline | Accuracy / log loss reference |\n|----------|-------------------------------|\n| Always pick home | Home win rate over your sample |\n| Prior-season Elo | Strong simple baseline |\n| Closing moneyline, devigged | The market. The hardest baseline |\n\nFor the market baseline, `get_odds` with `date` returns closing prices; devig per `odds-analysis`. Results are in `result.favourite_won` for finished games. If your model does not beat the closing line on log loss, it is not adding information the market lacks.\n\nReport accuracy alongside a bootstrap interval: with about 2,430 regular-season games, a standard error on accuracy is about 1 percentage point (sqrt(0.25 / 2430) = 0.0101). A 52% model against a 54% market is not close to settled.\n\n## Metrics\n\n```python\nimport numpy as np\ndef log_loss(y, p): p = np.clip(p, 1e-6, 1 - 1e-6); return -np.mean(y*np.log(p) + (1-y)*np.log(1-p))\ndef brier(y, p): return np.mean((p - y) ** 2)\n```\n\n- **Log loss** punishes confident misses. Primary metric.\n- **Brier** is less sensitive to extremes. Report both.\n- **Accuracy** only as a secondary number; it hides probability quality.\n- Compare log loss to the market's (devigged) log loss on the same games.\n\n## Calibration\n\nA probability of 60% should win about 60% of the time.\n\n1. Bin predictions (deciles, or fewer if the sample is small) in the test folds only.\n2. For each bin: mean predicted, observed win rate, count.\n3. Plot or tabulate. Bins with under 100 games are noise.\n4. If it is off, fit Platt scaling (logistic on the logit of the score) or isotonic regression on a separate calibration fold (the fold before the test fold), never on the test fold.\n5. Re-check on the test fold after calibrating.\n\nBaseball outcomes have low signal: most calibrated predictions will sit between 40% and 65%. A model that outputs 80% on a regular-season game is almost certainly overfit.\n\n## Worked Layout\n\n```\nFolds (expanding):  train 2015-2018 -> test 2019\n                    train 2015-2019 -> test 2021   (2020 treated separately: 60 games)\n                    train 2015-2021 -> test 2022\n                    ...\nPer fold report:    games, accuracy, log loss, Brier, market log loss, difference\nPooled:             mean difference and a bootstrap 95% interval over games\nCalibration:        decile table on pooled test predictions\n```\n\nOdds exist per book from July 2020 and as ESPN's final line earlier (see `fastballapi-tool`), so the market baseline has a shorter history than results. Say which seasons the comparison covers.\n\n## Credit Planning\n\n| Task | Credits |\n|------|---------|\n| One season of games (about 2,430 games, 13 calls at 200) | about 26 |\n| Ten seasons of games | about 260 |\n| Closing odds for 162 days of one season (`get_odds` by date) | about 1,620 |\n| Closing odds for ten seasons | about 16,000: plan the budget first |\n\nPull results once, store them, and add features locally.\n\n## Anti-patterns\n\n| Mistake | Why it is wrong | Do This |\n|---------|-----------------|---------|\n| Random k-fold on games | Leaks the future | Walk-forward |\n| Rolling mean without `shift(1)` | The current game is in the feature | Shift first |\n| `get_ratings` for season N as a feature in season N | End-of-season value includes future games | Previous season's end rating |\n| Final standings as features | Leakage | Games-before-date aggregates |\n| Tuning on the test fold | Optimistic | Tune on an inner earlier fold |\n| Reporting accuracy only | Hides probability quality | Log loss and calibration |\n| Calling a 1-point gain real | Inside the standard error | Interval over games |\n| Including 2020 as normal | 60-game season, different conditions | Handle separately |\n| Skipping the market baseline | Can't tell if you add information | Compare with devigged closes |\n\n## Output Format\n\n```\nWalk-forward, expanding window, test seasons 2019, 2021-2025 (2020 excluded)\nGames: [N]. Model log loss 0.6xx vs prior-season Elo 0.6xx vs devigged close 0.6xx\nAccuracy: x% (bootstrap 95% interval a to b); home baseline y%\nCalibration (deciles, pooled): within z points in every bin with 100+ games\nLeakage checks: shift(1) before rolling; no same-season ratings; no final standings\nRead: does or does not beat the closing line; sample covers [seasons].\nLimits: no starting pitcher, lineup or weather data in the API.\n```\n\n## What to Do Next\n\n| You found | Next | Skill |\n|-----------|------|-------|\n| Need market probabilities | Devig | `odds-analysis` |\n| A team-level luck question | Expected record | `pythagorean-record` |\n"
}