Update to FighterAPI

{
  "description": "Checks and fixes the calibration of fight win probabilities from a model or a rating: reliability tables, Brier score, log loss, calibration slope and intercept, Platt scaling and isotonic regression, and how to compare with the devigged closing line. Use when the user asks whether a model's probabilities are trustworthy, wants a reliability diagram, or wants to tune the Elo scale. Do not use to decide whether the market is efficient -- see favourite-calibration.",
  "included_files": [],
  "name": "probability-calibration",
  "skill_md_contents": "---\nname: probability-calibration\ndescription: \"Checks and fixes the calibration of fight win probabilities from a model or a rating: reliability tables, Brier score, log loss, calibration slope and intercept, Platt scaling and isotonic regression, and how to compare with the devigged closing line. Use when the user asks whether a model's probabilities are trustworthy, wants a reliability diagram, or wants to tune the Elo scale. Do not use to decide whether the market is efficient -- see favourite-calibration.\"\nmetadata:\n  version: 1.0.0\n  author: FighterAPI\n---\n\n# Probability Calibration\n\n> **Default data tool:** FighterAPI (`fighterapi-tool`).\n> Outcomes and pre-fight ratings: `get_fighter_fights` (5 credits per page, `rating_before` and `result`). Market benchmark: `get_odds` with `event_id` (10 credits per card).\n> For your own predictions file, skip the tool.\n\nYou are an expert in probabilistic forecasting. Your goal is to answer one question: when the model says 70%, does the fighter win about 70% of the time, and is that better than the closing line says?\n\n## When to Use\n\n- \"Is my fight model calibrated?\"\n- \"Is the Elo scale of 400 right for MMA?\"\n- \"Build a reliability table\"\n- \"Compare my model to the closing line\"\n\n## When NOT to Use\n\n- Judging whether the betting market is calibrated -- see `favourite-calibration`\n- Splitting data safely -- see `walk-forward-validation`\n\n## Commands Available\n\n| Command | What It Does | Credits |\n|---------|-------------|---------|\n| `get_fighter_fights` | `rating_before`, `result`, `median_closing_line` per fight for one fighter | 5 per page |\n| `get_event` | Results and winners for a card | 2 |\n| `get_odds` | Closing lines and `winner` for a card, the benchmark | 10 |\n| `get_fight` | Pre-fight `elo.before` for both sides | 5 |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_predictions` | Not available: the API does not publish fight predictions. Bring your own model output |\n| `get_model_probabilities` | Same |\n| `get_calibration` | Compute with the code below |\n\n## Inputs\n\nEach row needs: the model probability for one named fighter, the outcome (1 if that fighter won, 0 if not), the fight date, and ideally the devigged closing probability for the same fighter. Orient every row by `fighter_id`, never by side `a` or `b`. Drop draws and no contests, and count them.\n\nPredictions must be made with information from before the fight. If the model used ratings or rates, those must be `rating_before` or rates computed from earlier fights only (`walk-forward-validation`).\n\n## Metrics\n\n| Metric | Formula | Reading |\n|--------|---------|---------|\n| Brier score | mean((p - y)^2) | Lower is better. Coin flip is 0.25 |\n| Log loss | -mean(y ln p + (1 - y) ln(1 - p)) | Punishes confident misses hard. Coin flip is 0.693 |\n| Calibration slope | coefficient on logit(p) in a logistic fit of y | 1 is ideal. Below 1 means overconfident |\n| Calibration intercept | with slope fixed at 1 | Positive means the model under-predicts the favourite |\n| Expected calibration error | weighted mean of abs(bin gap) | Summary of the reliability table |\n\n```python\nimport numpy as np\ndef brier(p, y): return float(np.mean((p - y) ** 2))\ndef logloss(p, y, eps=1e-6):\n    p = np.clip(p, eps, 1 - eps)\n    return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))\n```\n\n## Reliability Table\n\n1. Bin predictions. With a few hundred fights use 5 bins of equal count (quantile bins), not equal width. Equal width leaves empty bins at the extremes.\n2. In each bin: n, mean predicted, observed rate, and a Wilson interval (see `favourite-calibration`).\n3. Plot predicted on x, observed on y, with the diagonal. Bars for n below 30 should be drawn lightly or merged.\n\nFight data is small. Ten bins on 400 fights gives 40 per bin and an interval about plus or minus 15 points. Prefer fewer bins and the slope and intercept, which use all the data.\n\n## Fixing Miscalibration\n\n- **Platt scaling.** Fit `logit(p_cal) = a + b * logit(p)` by logistic regression on held-out predictions. Two parameters, stable on small samples.\n- **Isotonic regression.** Flexible, but needs a few hundred rows per side and can overfit. Check on a later period.\n- **Temperature.** For an Elo prior, tune the `scale` in `p = 1/(1+10^(-d/400))` by minimizing log loss on a calibration window.\n\nFit the correction on one period and evaluate it on the next. Fitting and scoring the same rows rewards overfitting.\n\n## Compare with the Market\n\nScore the devigged closing probability on exactly the same rows (`odds-analysis`). That is the benchmark. If the model's Brier or log loss is not lower than the market's on out-of-sample rows, the model has no information the market lacks, however good it looks in isolation.\n\nTest the difference with a paired bootstrap over fights (resample fight indices, recompute the score gap, repeat 2000 times) and report the interval. With a few hundred fights the interval will usually include zero. Say so.\n\n## Pitfalls\n\n| What the Model Might Do | Why It Is Wrong | Do This Instead |\n|---------------------|----------------|-----------------|\n| Evaluate on training rows | Optimistic | Hold out later fights |\n| Use current ratings for past fights | Leaks results | `rating_before` |\n| Bin at fixed width | Empty and tiny bins | Quantile bins, or slope and intercept |\n| Judge by accuracy alone | Hides overconfidence | Brier and log loss |\n| Compare to 50% | Weak baseline | Compare to the devigged close |\n| Pool PFL and UFC | Different books, markets | Separate runs |\n\n## Worked Example\n\nRequest: \"Is the Elo scale of 400 right?\"\n\n```\nget_ratings(weight_class=\"Lightweight\", limit=50)    # who is active (2 credits)\nget_fighter_fights(fighter_id=..., limit=200)         # per fighter: rating_before, rating_after, result\n```\n\nEach history row gives `rating_before` for the fighter. The opponent's pre-fight rating is on the opponent's own row for the same `bout_id`, so collect rows for both fighters and join on `bout_id`. For each fight compute `d = R_self - R_opp`, then `p_win` at several scales (300, 400, 500), score each with log loss on fights after a chosen date, and pick the scale that minimizes it on an earlier window and confirm on the later one. Expect a flat optimum: report the interval of scales within one standard error, not a single best number.\n\n## Output Format\n\n```\nFights scored: N (draws/NC dropped: k), window: dates\n              Brier   LogLoss   Slope   Intercept\nModel          x.xxx   x.xxx     x.xx    x.xx\nClosing line   x.xxx   x.xxx     x.xx    x.xx\nPaired bootstrap, model minus market Brier: x.xxx (95% CI a to b)\nReliability (5 quantile bins): table\nVerdict: calibrated within noise / overconfident / no information beyond market\n```\n\nThe numbers above are placeholders for layout. Replace them with computed values.\n\n## What to Do Next\n\n| Finding | Next Skill |\n|---------|-----------|\n| Not better than the close | Improve features, retest with `walk-forward-validation` |\n| Need price benchmarks over many seasons | `favourite-calibration` |\n"
}