Update to EndzoneAPI

{
  "description": "Checks and repairs predicted probabilities for NFL outcomes: reliability tables, Brier score, log loss, expected calibration error, Platt scaling, isotonic regression, and sample-size limits. Use when user asks if a model's 65% means 65%, how to calibrate, Brier score, log loss, why a model is overconfident. Do not use for building the model, see feature-engineering. Do not use for time-ordered testing, see walk-forward-validation. Do not use for turning odds into probability, see odds-analysis.",
  "included_files": [],
  "name": "probability-calibration",
  "skill_md_contents": "---\nname: probability-calibration\ndescription: \"Checks and repairs predicted probabilities for NFL outcomes: reliability tables, Brier score, log loss, expected calibration error, Platt scaling, isotonic regression, and sample-size limits. Use when user asks if a model's 65% means 65%, how to calibrate, Brier score, log loss, why a model is overconfident. Do not use for building the model, see feature-engineering. Do not use for time-ordered testing, see walk-forward-validation. Do not use for turning odds into probability, see odds-analysis.\"\nmetadata:\n  version: 1.0.0\n  author: EndzoneAPI\n---\n\n# Probability Calibration\n\n> **Default data tool:** EndzoneAPI (`endzoneapi-tool`) for outcomes only: `get_games` (5 credits) returns `result`, `spread_line`, `total`, `total_line` for scoring predictions. Calibration itself is local code.\n> For your own predictions CSV: no credits used.\n\nYou are an expert in probabilistic forecast quality. Your goal is to check that a stated probability matches the observed frequency, and to fix it only when the evidence supports a fix.\n\n## When to Use\n\n- \"My model says 70% for the home team, is that right?\"\n- \"Is my model overconfident?\"\n- \"Compute Brier score and log loss\"\n- \"Should I use Platt scaling or isotonic?\"\n- \"Are my cover probabilities good enough to size bets?\"\n\n## When NOT to Use\n\n- Constructing inputs, see `feature-engineering`\n- Choosing training windows, see `walk-forward-validation`\n- Devigging market odds, see `odds-analysis`\n\n## Commands Available\n\n| Command | Use Here | Credits |\n|---------|---------|---------|\n| `get_games` | Realized outcomes and consensus lines for scoring | 5 per query |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_calibration` | Compute from your predictions and `get_games` outcomes |\n| `get_model_probabilities` | Not provided; bring your own |\n\n## Metrics\n\nFor predictions `p_i` in [0, 1] and outcomes `y_i` in {0, 1}:\n\n```\nBrier    = mean((p - y)^2)\nLogLoss  = -mean(y * ln(p) + (1 - y) * ln(1 - p))\n```\n\nReference points for a binary game outcome at 50/50 forecasts: Brier 0.25, log loss 0.693. A forecast has to beat those, then beat the market's own probabilities (see `odds-analysis`) on the same games.\n\n**Reliability table.** Bin predictions (10 equal-count bins are better than 10 equal-width bins with small samples), then compare mean prediction to observed frequency per bin.\n\n```python\ndf[\"bin\"] = pd.qcut(df.p, 10, duplicates=\"drop\")\nrel = df.groupby(\"bin\").agg(pred=(\"p\", \"mean\"), obs=(\"y\", \"mean\"), n=(\"y\", \"size\"))\n```\n\n**Expected calibration error:** `sum(n_bin / N * |obs - pred|)`. Report with a bootstrap interval; with a few hundred games the interval is wide.\n\n## Reading the Table\n\n| Pattern | Meaning | Fix |\n|---------|---------|-----|\n| Predictions spread wider than outcomes (80% bin hits 68%) | Overconfident | Shrink toward the base rate, Platt scaling |\n| Predictions narrower than outcomes | Underconfident | Rare; sharpen with Platt slope above 1 |\n| Good in the middle, bad at the tails | Few tail samples | Do not fix tails with a flexible method on tiny n |\n| Systematic offset (home always 3 points too high) | Bias | Intercept in Platt scaling |\n\n## The Sample-Size Problem in Football\n\nAn NFL season has 272 regular season games. In 10 bins, that is about 27 games a bin per season. The standard error of an observed frequency near 0.5 at n = 27 is about 0.10. A bin at 0.60 predicted and 0.50 observed is inside the noise. Pool several seasons, use fewer bins (5), and show intervals. Do not conclude miscalibration from one season of one bin.\n\n## Repairs\n\n**Platt scaling (default).** Fit a logistic regression of outcome on the logit of your prediction:\n\n```python\nfrom sklearn.linear_model import LogisticRegression\nz = np.log(p / (1 - p)).reshape(-1, 1)\ncal = LogisticRegression(C=1e6).fit(z, y)      # two parameters: slope and intercept\np_cal = cal.predict_proba(z)[:, 1]\n```\n\nTwo parameters, stable on a few hundred games. A slope below 1 confirms overconfidence.\n\n**Isotonic regression.** Flexible monotone fit. Needs thousands of rows; on a few hundred games it overfits steps. Use only with enough data.\n\n**Fit the calibrator on out-of-sample predictions.** The calibration map must be learned on predictions from a walk-forward run (see `walk-forward-validation`), then applied forward in time. Fitting it on the same data used to score it reports an inflated gain.\n\n## Regression Predictions\n\nIf the model predicts a margin, convert to a win probability with a normal approximation:\n\n```\nP(home wins) = Phi(predicted_margin / sigma)\n```\n\nwith `sigma` estimated from your own residuals (`result` minus predicted margin). NFL margins are not normal around the spread; key numbers (3, 7) put extra mass at specific margins. For cover probabilities near the spread, estimate the empirical distribution of `result - spread_line` from the seasons you have and use it instead of the normal curve. Compute both and compare on held-out games.\n\n## Using Calibration for Bets\n\n\n## Credit Usage\n\nFive credits per `get_games` query. Everything else is local.\n\n## Anti-patterns\n\n| Rationalization | Why It's Wrong | Do This Instead |\n|----------------|---------------|-----------------|\n| \"Accuracy is 64% so it's good\" | Says nothing about confidence | Report Brier and log loss next to the market's |\n| \"Calibrate on the test set\" | Leaks the answer | Calibrate on prior out-of-sample predictions |\n| \"Isotonic is more flexible so better\" | Overfits small samples | Platt first |\n| \"One season's reliability chart shows bias\" | 27 games a bin | Pool seasons, show intervals |\n| \"Clip probabilities at 0.99\" | Log loss punishes confident misses | Check the model, do not hide it |\n\n## Output Format\n\n```\nCalibration, out-of-sample, 2022-2025 (N = 1,088 games)\nBrier: 0.xxx (market 0.xxx) | Log loss: 0.xxx (market 0.xxx)\nPlatt slope 0.xx, intercept 0.xx -> [overconfident / fine]\nReliability (5 bins): pred x.xx obs x.xx n=NNN (95% CI x.xx to x.xx) ...\nVerdict: [calibrated / adjust / insufficient data]\n```\n\n## What to Do Next\n\n| What You Found | Next Action | Skill |\n|----------------|-------------|-------|\n| Poorly calibrated | Rebuild inputs or add data | `feature-engineering` |\n"
}