ML Referee

View saved version →

---
name: referee
description: Evaluate incremental feature quality or bounded feature repairs for supervised machine learning and forecasting with ML Referee. Use for access checks, preparing evaluation data, preflight and quotes, authorized evaluations, result interpretation, and report-integrity checks—not financial recommendations or deployment certification.
---

# ML Referee

Guide the user from a declared prediction objective to a bounded, evidence-backed
feature-quality opinion. Use the connected ML Referee tools; discover current
capabilities and schemas rather than inventing fields or results. A plugin
installation, successful login, approved beta request, and available credits are
separate states. Do not imply one establishes the others.

## Connection and beta access

- Start with `describe_referee` and `get_referee_readiness`. If connection or
  authorization is required, use the host's connection flow. Never request an
  Auth0 application client secret, password, or access token in chat.
- Report actual access and credit status. A balance check must not upload data,
  request access, redeem a code, or submit an evaluation.
- If the user asks for beta access, call `request_beta_access` with a supported
  lane, fixed use-case category, and stable idempotency key. Explain that it
  creates a pending request, not credits or guaranteed approval. Give the request
  ID for private correspondence with the operator. Do not promise an email was
  sent. Check `get_beta_request_status` when asked, without repeated requests.
- If the user supplies an invitation for redemption, use `redeem_beta_invite`
  and then check readiness. Treat the code as a secret; do not echo it in reports
  or support messages. Never invent an invitation or choose a credit grant.
- Beta access requires human approval, is free and card-free, and expires as
  reported by the service. Do not request card/bank details or initiate payment.
  Support is available at https://mlreferee.com/support.

## Prepare the evidence before uploading

Ask for missing essentials: the prediction objective, labeled outcome, baseline,
candidate family, observation and label-availability times, and validation plan.
Do not fabricate labels, scores, row identity, permission to upload, or evidence.
Keep observation time distinct from when a label actually became available.

Only upload data the user has authorized for this evaluation. During beta,
evaluation datasets must be synthetic, non-personal, or genuinely anonymized.
Do not upload regulated data, identifiable or pseudonymized person-level records,
raw private prompts, credentials, or payment details. Account/support contact
information belongs in the designated channels, not evaluation datasets. A
successful preflight is not certification that a dataset satisfies these rules.

- Feature quality: use `list_verticals` and `get_input_schema`, then
  `upload_private_dataset` and free `preflight_dataset` for the chosen contract.
- For 33–1,000 candidates, use campaign shards with identical rows, timing, and
  baseline roles, at most 32 candidates per shard, and one common contract. Use
  `upload_campaign_shard` and `quote_campaign`; do not present a shard as evidence
  for the complete family. Use discovered campaign limits and schemas.
- Feature repair: use `get_repair_schema`, `upload_repair_matrix`, and
  `preflight_repair`. Establish a reference and later comparison period; bounded
  transforms must be fitted only to the reference data.
- For `provided_oof`, require immutable baseline and candidate scores,
  deterministic row IDs, and the evidence digest binding all score columns.
  A digest does not establish that out-of-fold generation was valid.

Resolve failed preflight conditions without weakening the declared evaluation
just to obtain a passing result. Do not silently change error targets after
looking at evidence. The public synthetic example and its contract are at
https://api.mlreferee.com/public/verticals/gene

Preview of the first 4,000 characters. Read full instructions →