Update to BirdieAPI

{
  "description": "Validates golf models in time order: expanding-window splits by tournament date, tournament-grouped evaluation, point-in-time ratings, metrics for round scores, made cut, top 10 and winner probabilities, a market baseline, and significance checks with the small number of majors. Use when the user asks how to test a golf model, whether k-fold is okay, how to split by season, how to avoid leakage, or how to compare a model with devigged winner odds. Do not use for building features -- see feature-engineering. Do not use for probability reliability plots -- see probability-calibration.",
  "included_files": [],
  "name": "walk-forward-validation",
  "skill_md_contents": "---\nname: walk-forward-validation\ndescription: \"Validates golf models in time order: expanding-window splits by tournament date, tournament-grouped evaluation, point-in-time ratings, metrics for round scores, made cut, top 10 and winner probabilities, a market baseline, and significance checks with the small number of majors. Use when the user asks how to test a golf model, whether k-fold is okay, how to split by season, how to avoid leakage, or how to compare a model with devigged winner odds. Do not use for building features -- see feature-engineering. Do not use for probability reliability plots -- see probability-calibration.\"\nmetadata:\n  version: 1.0.0\n  author: BirdieAPI\n---\n\n# Walk-Forward Validation\n\n> **Default data tool:** BirdieAPI (`birdieapi-tool`).\n> Validation is computation on data you already pulled. The calls that matter are `get_ratings` with `as_of` (2 credits per date), `get_leaderboard` (2) and `get_winner_odds` (5) for the market baseline.\n\nYou are an expert in honest model evaluation. Golf makes it easy to fool yourself: one winner per event, a few dozen majors with prices, strong dependence within a tournament and ratings that quietly include the future if you let them. Your goal is a test whose number you would stand behind.\n\n## When to Use\n\n- \"Can I use k-fold cross-validation on golf results?\" (No.)\n- \"How do I split train and test?\"\n- \"Does my model beat the market?\"\n- \"How many majors do I need before this is meaningful?\"\n\n## When NOT to Use\n\n- Feature construction -- see `feature-engineering`\n- Reliability diagrams and scaling -- see `probability-calibration`\n\n## Commands Available\n\n| Command | What It Does | Credits |\n|---------|-------------|---------|\n| `get_ratings` | Point-in-time rating via `as_of` | 2 |\n| `get_leaderboard` | Outcomes: finish, cut status, rounds | 2 |\n| `get_winner_odds` | Market baseline for majors from 2021 | 5 |\n| `list_tournaments` | The calendar and `major` flag, for ordering and subsets | 1 |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_predictions` | Not served; the model is yours |\n| `get_historical_odds` for non-majors | Not available; only majors have prices |\n| `get_closing_line` | The latest pregame snapshot, which can be a week early for past majors |\n\n## Rules\n\n1. **Time order only.** Train on events before a date, test on events after. Never shuffle, never k-fold.\n2. **Group by tournament.** All rows of one tournament go in the same fold. Within an event, results are dependent (one winner, shared course and weather).\n3. **Point-in-time inputs.** Ratings with `as_of` the day before round 1. Season stats from the prior season. Course history from earlier editions.\n4. **Freeze choices before testing.** Features, hyperparameters, `sigma` and `tau` are set on the training window.\n5. **Compare against baselines.** At least the rating alone, and the devigged market where it exists.\n\n## How It Works\n\n**Step 1: Build the event list.**\n\n```\nlist_tournaments {\"season\": 2023, \"status\": \"final\"}\n```\n\nRepeat per season and sort by `start_date`. Add `\"major\": true` for the majors only.\n\n**Step 2: Choose a scheme.** An expanding window by season is the default:\n\n| Fold | Train | Test |\n|------|-------|------|\n| 1 | 2015-2020 | 2021 |\n| 2 | 2015-2021 | 2022 |\n| 3 | 2015-2022 | 2023 |\n| 4 | 2015-2023 | 2024 |\n| 5 | 2015-2024 | 2025 |\n\nRating-based models need little training data. The window mostly serves to fit `sigma`, `tau`, the shrinkage constant `k` from `course-fit` and any blend weight with the market.\n\n**Step 3: Pick metrics by target.**\n\n| Target | Metric | Notes |\n|--------|--------|-------|\n| Round `adj_vs_field` | RMSE, MAE | baseline: the golfer's rating |\n| Made cut | log loss, AUC | thousands of rows |\n| Top 10 | log loss, Brier | hundreds of positives |\n| Winner | log loss over the field | one positive per event |\n\nFor winners, the score of one event is `-log(p_winner)`. Average over events.\n\n```python\nimport numpy as np\n\ndef winner_log_loss(prob_by_event, winner_by_event, floor=1e-4):\n    s = []\n    for ev, probs in prob_by_event.items():        # probs: dict player_id -> prob, sums to 1\n        p = max(probs.get(winner_by_event[ev], 0.0), floor)\n        s.append(-np.log(p))\n    return float(np.mean(s))\n```\n\nThe floor stops one zero from producing infinity; state it. A winner at 0.2% costs 6.2 nats and a winner at 15% costs 1.9, so a handful of long-priced winners dominate the mean.\n\n**Step 4: The market baseline.** For each major from 2021, devig the latest pregame prices (`odds-analysis`), apply the same floor and score the market on the same events.\n\n```\nget_winner_odds {\"season\": 2025, \"limit\": 200}\n```\n\nA model that does not beat this baseline out of sample adds nothing over simply using the price. Score both on events where prices exist; do not score the model on more events than the market and compare.\n\n**Step 5: Significance.** About two dozen majors have prices, so about two dozen winners. Compute the per-event loss difference (model minus market) and its standard error:\n\n```python\nd = np.array(model_losses) - np.array(market_losses)\nse = d.std(ddof=1) / np.sqrt(len(d))\nprint(d.mean(), se)         # |mean| < 2 * se: no demonstrated difference\n```\n\nWith about 25 events, a small improvement will not clear two standard errors. Say \"no demonstrated difference\" rather than \"slightly better\". For more power, evaluate the intermediate targets (made cut, top 10, round scores) over many more events, which is a strength of this data.\n\n**Step 6: Report by subset.** Split majors from regular events only if each subset has enough rows. Put the count next to every number.\n\n**Step 7: Keep a final holdout.** If you iterate on the 2025 fold it becomes a training fold. Hold the latest full season untouched until the end.\n\n## Common failures\n\n| Failure | Symptom | Fix |\n|---------|---------|-----|\n| Random split | Test loss far better than live | split by date and tournament |\n| Current rating on past events | Suspiciously strong results for recent stars | `as_of` |\n| A zero-probability winner | Infinite loss | floor, and report it |\n| Tuning on the test fold | Gains vanish next year | frozen design, final holdout |\n| Different event sets for model and market | Apples to oranges | same events |\n| Counting golfers as independent | Fake sample size | the unit is the tournament |\n\n## Credit Usage\n\n| Operation | Credits | Notes |\n|-----------|---------|-------|\n| Ratings, one date per event, 100 events | 200 | |\n| Leaderboards, 100 events | 200 | |\n| Market for every major, 2021-2026 | 30 | one `season` call returns four majors, six seasons at 5 credits |\n| Whole study on 500 credits a month | spread over months | 500 credits a month |\n\n## Anti-patterns\n\n| What the Model Might Do | Why It's Wrong | Do This Instead |\n|---------------------|---------------|-----------------|\n| Report accuracy at picking the winner | One positive per 100 rows; accuracy is meaningless | Log loss over the field |\n| Call a 0.05 nat edge \"significant\" on 25 majors | Within noise | Show the standard error |\n| Validate once and ship | Regimes change | Re-run each season |\n| Score with finishing-position correlation | Field size varies | Use strokes versus the field |\n\n## Output Format\n\n```\nWalk-forward, expanding window by season, folds 2021-2025 (majors with prices: N = <count>)\nWinner log loss: model <x.xx>, devigged market <x.xx>, rating-only <x.xx>\nDifference (model - market): <mean> +/- <se> over N events. Demonstrated: <yes/no>.\nMade-cut log loss over <rows> golfer-events: <x.xxx>\n```\n\n## What to Do Next\n\n| What You Found | Next Action | Skill |\n|----------------|-------------|-------|\n| Model passes | Check probability reliability | `probability-calibration` |\n| Model is no better than the market | Treat the market as the estimate | `odds-analysis` |\n"
}