{"id":74758,"plugin_id":"plugin_asdk_app_6ad1467bd8f88191bead2eb0a9b87e49","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-10-31T15:03:55.418Z","digest":"a66483b47fdb8d6982deb497e96fac23718e4c4ed0bda2065b678c1fa8edcf97","against":null,"payload":{"description":"Checking and fixing tennis win probabilities: reliability tables, calibration slope and intercept, Brier decomposition, Platt scaling, isotonic regression, temperature scaling, and calibrating against the market close. Use when the user asks whether a model's 70 percent means 70 percent, why a model is overconfident on favorites, or how to correct probabilities. Do not use for training the model: see feature-engineering. Do not use for choosing validation splits: see walk-forward-validation.","included_files":[],"name":"probability-calibration","skill_md_contents":"---\nname: probability-calibration\ndescription: \"Checking and fixing tennis win probabilities: reliability tables, calibration slope and intercept, Brier decomposition, Platt scaling, isotonic regression, temperature scaling, and calibrating against the market close. Use when the user asks whether a model's 70 percent means 70 percent, why a model is overconfident on favorites, or how to correct probabilities. Do not use for training the model: see feature-engineering. Do not use for choosing validation splits: see walk-forward-validation.\"\nmetadata:\n  version: 1.0.0\n  author: AcesAPI\n---\n\n# Probability Calibration\n\n> **Default data tool:** AcesAPI (`acesapi-tool`).\n> Out-of-sample predictions with outcomes are the input. Results and `winner_id` come from `get_matches` (2 credits per 100). Market closes for comparison come from `get_match` (2 each).\n\nYou test whether predicted probabilities match reality. In tennis this is where models usually fail: a rating gap turns into a probability through a formula with assumptions (a fixed 400-point scale, one format, constant surface effect) and those assumptions break at the extremes and in best-of-five.\n\n## When to Use\n\n- \"When my model says 75 percent, how often does the player win?\"\n- \"My Elo probabilities seem too extreme for heavy favorites\"\n- \"Should I calibrate?\"\n- \"Is the market better calibrated than my model?\"\n\n## When NOT to Use\n\n- Building features or fitting models: see `feature-engineering`\n- Deciding how to split data: see `walk-forward-validation`\n\n## Commands Available\n\n| Command | What It Does | Credits |\n|---------|-------------|---------|\n| `get_matches` | Outcomes (`winner_id`) and format context for scoring predictions | 2 |\n| `get_match` | Pre-match ratings and market closes for comparison | 2 |\n| `get_ratings` | Current ratings for forward predictions | 2 |\n| `get_player_matches` | `rating_before` for many matches of one player | 5 |\n\n## Commands That Do NOT Exist\n\n| Not Available | Use Instead |\n|--------------|-------------|\n| `get_calibration` | Compute from your prediction log |\n| `get_model_probability` | Your model; the API serves data only |\n| `get_historical_predictions` | Not available |\n\n## Step 1: Orient the Predictions\n\nCalibration needs one probability per match for one named side and a 0 or 1 label. Make it unambiguous: `p` = probability that the focal player wins, `y` = 1 if `winner_id` equals the focal player's id. If you keep both orientations of each match, score one orientation, or the table duplicates every row and halves your apparent uncertainty.\n\n## Step 2: Reliability Table\n\nBin predictions, then compare predicted and realized.\n\n```\nbins:    [0.0, 0.1) ... [0.9, 1.0]     # use equal-count bins when n is small\nfor each bin: n, mean(p), mean(y), mean(y) - mean(p)\n```\n\nFor tennis, put extra bins above 0.8, because heavy favorites are common (the best players are favored at 85 to 95 percent in early rounds). A model that says 92 percent and wins 88 percent costs real money when a price is -1000.\n\nReport `n` per bin. A bin with 12 matches has a standard error near 0.14 for a 50 percent rate. Show intervals (Wilson) for bins under 100.\n\n## Step 3: Summary Numbers\n\n| Measure | Definition | Reading |\n|---------|------------|---------|\n| Brier | `mean((p - y)^2)` | 0.25 for a coin flip |\n| Log loss | `-mean(y ln p + (1-y) ln(1-p))` | 0.693 for a coin flip |\n| Calibration intercept and slope | Fit `logit(P(y=1)) = a + b * logit(p)` | Perfect: a = 0, b = 1. b < 1 means overconfident; b > 1 means underconfident |\n| Expected calibration error | `sum(n_k / n * abs(mean y_k - mean p_k))` | Sensitive to binning; report with the bin scheme |\n\nSlope is the clearest single number. Elo-based tennis probabilities tend to show a slope below 1 in best-of-three matches among close players and above 1 in best-of-five, because best of five amplifies the better player's edge. Test it on your data; do not assume.\n\n## Step 4: Fix It\n\nFit the calibrator on a validation block earlier than the test block (see `walk-forward-validation`). Never fit and score on the same matches.\n\n| Method | Use When | Notes |\n|--------|----------|-------|\n| Platt (logistic on logit(p)) | Under about 2,000 matches; slope off but shape monotone | Two parameters, stable |\n| Temperature scaling | Same, simpler | One parameter `T` in `sigmoid(logit(p)/T)` |\n| Isotonic | Over 3,000 matches and non-sigmoid miscalibration | Flat steps; overfits small samples |\n| Per-format correction | Best-of-five vs best-of-three differ | Add a format term in the logistic |\n| Blend with the market | Covered events | `logit(p_final) = a + b logit(p_model) + c logit(p_market)`, fit walk-forward |\n\nFormat and surface often matter more than the method: fit separate intercepts or slopes by format first.\n\n## Step 5: Calibration Against the Market\n\n\n## Worked Example (method, with the credit cost)\n\nYou score 1,200 Masters and Slam matches from 2025 with an Elo model.\n\n1. `get_matches` pages for the season to collect outcomes: about 90 credits per tour.\n2. Predictions from your own Elo (see `feature-engineering`), pre-match only.\n3. Reliability table in ten bins, plus extra bins at 0.8, 0.85, 0.9, 0.95.\n4. Fit `logit(P(y)) = a + b logit(p)` on 2023 and 2024, apply to 2025.\n5. Report slope, Brier and log loss before and after, per format.\n6. For the 2025 matches that have odds, add the market column. This costs 2 credits per match.\n\nExpected deliverable: a table of bins, the fitted `a` and `b`, the before and after scores, and one sentence stating whether the correction survives the next season.\n\n## Pitfalls\n\n- Calibrating on the same data you evaluate on.\n- Pooling best-of-three and best-of-five.\n- Judging calibration with accuracy.\n- Over-binning small samples; use equal-count bins.\n- Treating a calibrated model as a profitable one. Calibration is necessary for betting, not sufficient. The market is also calibrated.\n- Using walkovers and retirements in the evaluation set inconsistently.\n- Rating drift: Elo ratings stretch over time as players cluster; recalibrate every season.\n\n## Credit Usage\n\n| Task | Credits |\n|------|---------|\n| Outcomes for a tour season | about 90 |\n| Market column for covered matches | 2 per match |\n| Calibration on user data | 0 |\n\n## Anti-patterns\n\n| What the Model Might Do | Why It's Wrong | Do This Instead |\n|---------------------|---------------|-----------------|\n| Report accuracy as calibration | Different property | Reliability table and slope |\n| Fit isotonic on 400 matches | Overfits | Platt or temperature |\n| Calibrate on the test season | Leak | Calibrate on earlier seasons |\n| Merge formats | Hides the best-of-five effect | Split or add a format term |\n| Say \"well calibrated\" from the Brier score | Brier mixes sharpness and calibration | Show the table |\n\n## Output Format\n\n```\nn = [matches], seasons [a-b], format [bo3 / bo5 / both]\nCalibration intercept [a], slope [b]\nBrier [x] (coin flip 0.250), log loss [y] (coin flip 0.693)\nBin   n    mean p   mean y   gap\n0.9+  ..   0.93     0.89    -0.04\n```\n\n## What to Do Next\n\n| What You Found | Next Action | Skill |\n|----------------|-------------|-------|\n| Persistent bias by surface | Revisit features | `feature-engineering` |\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}