Update to DunkAPI

{
  "description": "Checks and fixes whether basketball model probabilities mean what they say: reliability tables, Brier score and log loss, Platt scaling and isotonic regression fitted out of sample, converting predicted margins to win and cover probabilities, and shrinking toward the market. Use when a model outputs win, cover or over probabilities, asks if 65% really wins 65% of the time, or needs probabilities from a margin model. Do not use for building the model, see feature-engineering. Do not use for choosing splits, see walk-forward-validation.",
  "included_files": [],
  "name": "probability-calibration",
  "skill_md_contents": "---\nname: probability-calibration\ndescription: \"Checks and fixes whether basketball model probabilities mean what they say: reliability tables, Brier score and log loss, Platt scaling and isotonic regression fitted out of sample, converting predicted margins to win and cover probabilities, and shrinking toward the market. Use when a model outputs win, cover or over probabilities, asks if 65% really wins 65% of the time, or needs probabilities from a margin model. Do not use for building the model, see feature-engineering. Do not use for choosing splits, see walk-forward-validation.\"\nmetadata:\n  version: 1.0.0\n  author: DunkAPI\n---\n\n# Probability Calibration\n\n> **Default data tool:** DunkAPI (`dunkapi-tool`) only for outcomes and the market benchmark: `list_games` (2 credits per page), `get_odds` (`result.home_covered`, `result.over`). Calibration itself is local.\n\nEvery number built on a probability inherits its error. An uncalibrated 70% that is really 60% turns a small edge into a negative one.\n\n## When to Use\n\n- \"My model says 68% for the Knicks. Can I trust that?\"\n- \"Is Platt scaling or isotonic better here?\"\n- \"How do I turn a predicted margin into a win probability?\"\n\n## When NOT to Use\n\n- Fitting the model itself, see `feature-engineering`\n- Time-ordered evaluation design, see `walk-forward-validation`\n\n## Commands Available\n\n| Command | Role | Credits |\n|---------|------|---------|\n| `list_games` | Final scores to label outcomes | 2 per page |\n| `get_odds` | Market probability to benchmark (and `home_covered`, `over` to label cover and total outcomes) | 5 per game, 10 per date |\n\n## Commands That Do NOT Exist\n\n| Not available | Use instead |\n|---------------|-------------|\n| `get_calibration`, `get_model_probabilities` | Compute locally from your own predictions |\n\n## What Calibrated Means\n\nAmong all games where the model said about 70%, the home side should win about 70% of them. Two separate qualities:\n\n- **Calibration:** do stated probabilities match frequencies?\n- **Discrimination (sharpness):** does the model separate likely from unlikely outcomes?\n\nA model that always says 58% for the home team in a league where home teams win 58% is calibrated and useless. You need both.\n\n## How It Works\n\n**Step 1: Use out-of-sample predictions only.** Generate predictions with `walk-forward-validation`. Calibrating on training predictions measures overfit, not reliability.\n\n**Step 2: Reliability table.** Bin predictions (deciles, or equal-count bins), and compare the mean prediction to the observed rate.\n\n```python\nimport numpy as np, pandas as pd\ndf[\"bin\"] = pd.qcut(df.p, 10, duplicates=\"drop\")\nrel = df.groupby(\"bin\").agg(pred=(\"p\",\"mean\"), obs=(\"y\",\"mean\"), n=(\"y\",\"size\"))\nrel[\"se\"] = np.sqrt(rel.obs * (1 - rel.obs) / rel.n)\n```\n\nRead it with the standard error: a bin of 120 games has a standard error near 4.5 points, so a 3-point miss is noise. Plot `obs` against `pred` with error bars; the diagonal is perfect.\n\n**Step 3: Scores.**\n\n| Metric | Formula | Notes |\n|--------|---------|-------|\n| Log loss | `-mean(y*log(p) + (1-y)*log(1-p))` | Coin flip 0.693; punishes overconfidence |\n| Brier | `mean((p - y)^2)` | Coin flip 0.25 |\n| Calibration slope | Logistic regression of `y` on `logit(p)` | 1.0 ideal; below 1 means overconfident |\n| Calibration intercept | Same regression | 0 ideal |\n\nBrier decomposes into reliability (bad) minus resolution (good) plus uncertainty (fixed). Improve reliability by calibrating; improve resolution only with better features.\n\n**Step 4: Choose a correction.**\n\n| Method | Use when | Risk |\n|--------|----------|------|\n| Platt (logistic on the score) | Under about 3,000 calibration games | Assumes a sigmoid shape |\n| Isotonic regression | 3,000+ calibration games | Steps; overfits small samples |\n| Shrink toward the market: `p' = w*p_model + (1-w)*p_market` | Model has some edge but is overconfident | Needs a devigged market probability (see `odds-analysis`) |\n\nNBA seasons are small. One season is about 1,230 games, so Platt or shrinkage is usually right; isotonic needs several seasons pooled.\n\n```python\nfrom sklearn.linear_model import LogisticRegression\nplatt = LogisticRegression(C=1e6).fit(logit(p_cal).reshape(-1, 1), y_cal)   # fit on a calibration block\np_adj = platt.predict_proba(logit(p_test).reshape(-1, 1))[:, 1]\n```\n\nCalibration data must be earlier than the data it is applied to. Fit on seasons `s-2..s-1`, apply to `s`.\n\n**Step 5: Margin models to probabilities.** If the model predicts a home margin `m`, the win probability is `Phi(m / sigma)`, where `sigma` is the residual standard deviation of margin around the prediction (about 12 points for NBA games; estimate it from your own out-of-sample residuals, because it changes with the era and the model). For covers against a spread `s` (home spread, negative when the home team is favoured): `P(cover) = Phi((m + s) / sigma)`. For totals use the same with the total prediction and its own sigma. Then calibrate those probabilities like any others.\n\n**Step 6: Re-check after correction.** Rebuild the reliability table on a later block. If the slope moved toward 1 and log loss dropped, keep it.\n\n### Worked Example (market benchmark)\n\nThe closing moneylines for game 401859967 from `get_odds` (SA home, NY away) devig to a home probability between 0.631 (DraftKings) and 0.651 (Bovada); Pinnacle's devigged value is 0.643. SA lost. One result says nothing about calibration: even a perfectly calibrated 64% favourite loses 36% of the time. That is why calibration is judged over hundreds of games, with bin standard errors, never from a single outcome.\n\n## Common Failures\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Slope well below 1 | Overfit model; extreme probabilities | Shrink toward the market or a prior |\n| Over-prediction of home wins | Home advantage double counted | Check intercept; refit |\n| Good overall, bad on underdogs | Regression to the mean in tails | Calibrate by bin; isotonic with enough data |\n| Calibrated in-sample only | Calibration fitted on training predictions | Refit on held-out data |\n| Probabilities above 0.9 for NBA games | Model too sure | Cap; NBA favourites rarely exceed 90% |\n| Different periods calibrate differently | Drift | Calibrate on a rolling recent window |\n\n## Sample Size Warning\n\nReliability needs data. With 1,000 games in 10 bins, each bin has 100 games and a standard error near 5 points. Do not slice by team, month and conference at once; every slice you add multiplies the false alarms.\n\n## Credit Usage\n\nNo calibration calls. Outcome labels come free from the `list_games` scores you already pulled. If you want cover and over labels, use `get_odds` by date (10 credits per day of games) and read `result.home_covered` and `result.over`.\n\n## Output Format\n\n```\nOut-of-sample games: [n], seasons [..]\nLog loss [x] (market [x]) | Brier [x] | slope [x], intercept [x]\nReliability (deciles): [pred -> obs +/- se] ...\nCorrection: [Platt/isotonic/shrink w=..], fitted on [window]\nAfter: log loss [x], slope [x]\nUsable range: [pred interval where bins are within 1.5 se]\n```\n\n## What to Do Next\n\n| Found | Next | Skill |\n|-------|------|-------|\n| Calibrated, still behind the market | Look for better features | `feature-engineering` |\n| Need a market probability | Remove the vig | `odds-analysis` |\n"
}