Update to StrikerAPI

{
  "description": "Checks and fixes whether soccer model probabilities mean what they say: per-class reliability for home, draw and away, ranked probability score, log loss, multiclass Brier, and Platt, isotonic or temperature scaling for three-way outputs. Use when the user asks if their probabilities are calibrated, why the model is overconfident, how to fix the draw, or whether to trust a 60% number. Do not use for building the model, see poisson-score-models or elo-ratings. Do not use for the walk-forward split, see walk-forward-validation. Do not use for devigging, see odds-analysis.",
  "included_files": [],
  "name": "probability-calibration",
  "skill_md_contents": "---\nname: probability-calibration\ndescription: \"Checks and fixes whether soccer model probabilities mean what they say: per-class reliability for home, draw and away, ranked probability score, log loss, multiclass Brier, and Platt, isotonic or temperature scaling for three-way outputs. Use when the user asks if their probabilities are calibrated, why the model is overconfident, how to fix the draw, or whether to trust a 60% number. Do not use for building the model, see poisson-score-models or elo-ratings. Do not use for the walk-forward split, see walk-forward-validation. Do not use for devigging, see odds-analysis.\"\nmetadata:\n  version: 1.0.0\n  author: StrikerAPI\n---\n\n# Probability Calibration\n\n> **Default data tool:** StrikerAPI (`strikerapi-tool`).\n> Calibration works on your model's out-of-sample predictions plus outcomes. Outcomes come from `list_games` (2 credits per call); the market benchmark from `get_odds` (5 per match, 2 per match by season).\n\n\n## When to Use\n\n- \"Are my win probabilities calibrated?\"\n- \"The model is overconfident on favorites\"\n- \"My draws are always too low\"\n- Picking between a raw model, a scaled model and the market\n- Before computing any edge\n\n## When NOT to Use\n\n- Fitting the underlying model: `poisson-score-models`, `elo-ratings`\n- Designing the out-of-sample split: `walk-forward-validation`\n- Converting prices to probabilities: `odds-analysis`\n\n## Commands Available\n\n| Command | What It Does | Credits |\n|---------|-------------|---------|\n| `list_games` | Outcomes (`result` H, D or A, scores) | 2 per call |\n| `get_odds` | Market close and `close_prob` as a benchmark | 5 or 2 per match |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_model_probabilities`, `get_predictions` | StrikerAPI provides none; you supply the model output |\n| `get_calibration` | Compute it from the code below |\n\n## Inputs\n\nA table with one row per match, with out-of-sample predictions (never in-sample):\n\n```\nmatch_id, date, p_home, p_draw, p_away, result   (result in {H, D, A})\n```\n\nProbabilities must sum to 1. Check it. Everything below assumes the predictions came from a walk-forward run.\n\n## Metrics\n\n```python\nimport numpy as np\n\ndef log_loss(P, y):            # P: (n, 3) probabilities in order H, D, A; y: indices 0, 1, 2\n    return -np.mean(np.log(np.clip(P[np.arange(len(y)), y], 1e-12, 1)))\n\ndef brier(P, y):\n    O = np.eye(3)[y]\n    return np.mean(((P - O) ** 2).sum(axis=1))\n\ndef rps(P, y):\n    O = np.eye(3)[y]\n    cum_p, cum_o = P.cumsum(axis=1)[:, :2], O.cumsum(axis=1)[:, :2]\n    return np.mean(((cum_p - cum_o) ** 2).sum(axis=1) / 2)\n```\n\n| Metric | Reads as | Notes |\n|--------|----------|-------|\n| Log loss | Penalizes confident misses | Sensitive to extremes; clip |\n| Multiclass Brier | Mean squared error on the triple | Robust |\n| RPS | Ordered Brier on cumulative outcomes | Standard for soccer; lower is better |\n\nAlways report the same metric for the market (devigged close) on the same matches. The absolute number matters less than the gap.\n\n## Reliability, Per Class\n\nCalibration is checked for each class separately. For home win:\n\n1. Bin matches by `p_home` (deciles, or fewer when n is small).\n2. In each bin, compare the mean predicted probability to the observed home-win frequency.\n3. Plot observed against predicted with a confidence interval (`sqrt(p (1 - p) / n)` per bin).\n\nRepeat for the draw and away win.\n\n```python\ndef reliability(p, hit, bins=10):\n    edges = np.quantile(p, np.linspace(0, 1, bins + 1))\n    idx = np.clip(np.digitize(p, edges[1:-1]), 0, bins - 1)\n    rows = []\n    for b in range(bins):\n        m = idx == b\n        if m.sum():\n            rows.append((p[m].mean(), hit[m].mean(), int(m.sum())))\n    return rows        # (mean predicted, observed rate, count)\n```\n\nExpected calibration error is the count-weighted mean of the absolute gap; report it with the bin counts, since 380 matches in ten bins is 38 each and each bin's rate has a standard error near 7 points.\n\n## The Draw Problem\n\nDraws are the hardest class. A model's draw probability tends to stay inside a narrow band, so the draw reliability curve is a short, noisy segment, and it is easy to get wrong in a systematic way. Typical failure modes:\n\n- A Poisson model with independent goals understates draws; Dixon-Coles helps\n- An Elo-based model treats a draw as half a win and has no draw model at all\n- A model fit across leagues applies one draw rate everywhere\n\nCheck the average predicted draw probability against the observed draw frequency first. If they differ by more than about 2 points over several hundred matches, fix the draw rate before anything else.\n\n## Fixing It\n\nFit the calibrator on one set of matches and test it on a later one. Never fit and test on the same predictions.\n\n| Method | When | Notes |\n|--------|------|-------|\n| **Temperature scaling** | Probabilities uniformly too confident or too timid | One parameter `T` on the logits of the triple. Cheap, hard to overfit. Start here |\n| **Multinomial logistic recalibration** | Class-specific bias (draw too low) | Logistic regression on `log(p_h / p_d)`, `log(p_a / p_d)` |\n| **Platt, one-vs-rest** | Binary question like \"home win\" | Logistic fit on the logit of one probability; renormalize the three |\n| **Isotonic** | Plenty of data (thousands of matches) | Flexible; overfits on one season of 380 matches |\n| **Blend with the market** | Always worth testing | `p = w * p_model + (1 - w) * p_market` with `w` fit on a prior period. A model that cannot beat this has little to add |\n\n```python\nfrom scipy.optimize import minimize_scalar\n\ndef fit_temperature(P, y):\n    logits = np.log(np.clip(P, 1e-12, 1))\n    def nll(T):\n        z = logits / T\n        z -= z.max(axis=1, keepdims=True)\n        Q = np.exp(z); Q /= Q.sum(axis=1, keepdims=True)\n        return log_loss(Q, y)\n    return minimize_scalar(nll, bounds=(0.5, 3.0), method=\"bounded\").x\n```\n\n`T` above 1 flattens (the model was overconfident); below 1 sharpens.\n\n## Worked Example Plan\n\n1. Produce walk-forward predictions for a season of Premier League matches with `poisson-score-models` (the 380 outcomes come from two `list_games` calls, 4 credits).\n2. Compute log loss, Brier and RPS for the model on all 380.\n3. Pull the devigged closes from `get_odds` (season mode, 760 credits; consider a single club's matches with `team` to cut the cost while testing) and compute the same metrics.\n4. Draw the three reliability curves. Report the mean predicted draw rate against the observed rate.\n5. Fit `T` on the first half, apply to the second half, compare.\n6. Report the scores with standard errors, using the per-match differences against the market.\n\nDo not state results before running this; the numbers belong to your model and sample.\n\n## Sample Size Guide\n\n| Matches | What you can conclude |\n|---------|----------------------|\n| Under 200 | Overall draw rate and rough bias only |\n| 380 (one league-season) | Coarse reliability in 4 or 5 bins; temperature scaling |\n| 1,500+ | Per-class curves, multinomial recalibration |\n| 5,000+ | Isotonic, league-specific effects |\n\n## Pitfalls\n\n1. **Calibrating on in-sample predictions.** In-sample probabilities are overconfident by construction.\n2. **Binning with small n.** Ten bins on 200 matches is mostly noise; use fewer.\n3. **Fixing the wrong thing.** Bad ranking cannot be fixed by calibration. Check discrimination (does the model separate H from A?) as well.\n4. **Calibrating the favorite only.** Look at all three classes.\n5. **Different leagues.** Calibrate per competition when draw rates differ.\n6. **Time drift.** Recalibrate on a rolling window and check the scaling stays stable.\n7. **Treating the market as perfectly calibrated.** It is the best public benchmark, not a law. Test it too.\n\n## Anti-patterns\n\n| Rationalization | Why It's Wrong | Do This Instead |\n|----------------|---------------|-----------------|\n| \"Accuracy is 55%, so it is fine\" | Says nothing about the probabilities | Reliability and RPS |\n| \"Isotonic on one season\" | Overfits | Temperature scaling |\n| \"Calibrated on the test set\" | Optimistic | Train the calibrator earlier, test later |\n| \"The draw is unpredictable, skip it\" | The draw is a third of the triple | Check the average draw rate at least |\n\n## Output Format\n\n```\n[Model] - [competition] [period], n = [matches], out-of-sample\n              Model      Scaled     Market (devigged close)\nLog loss      [x.xxx]    [x.xxx]    [x.xxx]\nBrier         [x.xxx]    [x.xxx]    [x.xxx]\nRPS           [x.xxx]    [x.xxx]    [x.xxx]\nMean p(draw) [x.xxx] vs observed [x.xxx]\nVerdict: [calibrated / overconfident / draw biased], T = [x.xx]\n```\n\n## What to Do Next\n\n| What You Found | Next Action | Skill |\n|----------------|-------------|-------|\n| Still behind the market | Improve features and model | `poisson-score-models` |\n"
}