Update to Inngest

{
  "description": "Use when building, migrating, or debugging Agent Evals on Inngest: scoring AI agent or workflow outcomes, deferred scorers, sessions, traces, step experiments, experiment variant attribution, Insights queries, or production eval loops for prompts, models, tools, providers, and agent behavior. Covers TypeScript SDK v4 scoring beta APIs, `scoreMiddleware`, `step.score`, `inngest.score`, `createScorer`, `defer`, `group.experiment`, `experimentRef`, `meta.sessions`, and when to use durable workflow primitives for outcome-based evaluation.",
  "included_files": [],
  "name": "inngest-agent-evals",
  "skill_md_contents": "---\nname: inngest-agent-evals\ndescription: \"Use when building, migrating, or debugging Agent Evals on Inngest: scoring AI agent or workflow outcomes, deferred scorers, sessions, traces, step experiments, experiment variant attribution, Insights queries, or production eval loops for prompts, models, tools, providers, and agent behavior. Covers TypeScript SDK v4 scoring beta APIs, `scoreMiddleware`, `step.score`, `inngest.score`, `createScorer`, `defer`, `group.experiment`, `experimentRef`, `meta.sessions`, and when to use durable workflow primitives for outcome-based evaluation.\"\n---\n\n# Inngest Agent Evals\n\nUse this skill when the user wants to evaluate AI agents or AI workflows in\nproduction, add scoring, compare prompts/models/tools, group related runs, or\ndebug why an agent outcome was good or bad.\n\nAgent Evals is not a separate package. It is the production evaluation workflow\nbuilt from Inngest functions, durable steps, scores, deferred scorers, sessions,\ntraces, experiments, and Insights.\n\n## Canonical References\n\nCheck the public docs first when exact API details matter:\n\n- Agent Evals overview: https://www.inngest.com/docs/learn/agent-evals\n- Scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/scoring\n- Deferred scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/deferred-scoring\n- Step experiments guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/step-experiments\n- Sessions: https://www.inngest.com/docs/features/events-triggers/sessions\n- Traces: https://www.inngest.com/docs/platform/monitor/traces\n- Insights: https://www.inngest.com/docs/platform/monitor/insights\n- Scoring reference: https://www.inngest.com/docs/reference/typescript/v4/functions/scoring\n- `group.experiment()` reference: https://www.inngest.com/docs/reference/typescript/v4/functions/group-experiment\n- Launch context: https://www.inngest.com/blog/introducing-agent-evals\n\n## Decision Flow\n\nStart from the outcome, not the mechanism:\n\n1. Identify the product or quality signal: helpful click, ticket resolution,\n   conversion, guardrail pass, retrieval quality, model confidence, latency,\n   cost, human review, or LLM-as-judge result.\n2. Decide when the signal appears:\n   - During the run: use direct scoring with `step.score()` or\n     `inngest.score()`.\n   - After the run: use deferred scoring with `createScorer()` and `defer()`,\n     or explicit `inngest.score({ runId })`.\n3. Decide how humans will inspect related work:\n   - Use `meta.sessions` for repeated high-cardinality identifiers such as\n     `conversation_id`, `ticket_id`, `agent_run_id`, or `import_id`.\n   - Use Insights for ad hoc historical analysis, low-cardinality filters, SQL\n     aggregates, and broad investigation.\n4. Decide whether to compare variants:\n   - Use `group.experiment()` for prompt, model, provider, tool, workflow, or\n     rollout comparisons.\n   - Pass `experimentRef` to deferred or later scores when the score should be\n     credited to the selected variant.\n5. Preserve traceability:\n   - Keep model calls, tool calls, waits, database writes, and scoring work in\n     durable steps so traces explain why the score happened.\n\n## Setup Requirements\n\n- Use TypeScript SDK v4 and install the latest SDK: `npm install inngest@latest`.\n- Scoring and deferred scoring are beta APIs; verify current imports against\n  the docs before shipping user-facing examples.\n- `step.score()` requires `scoreMiddleware()` from `inngest/experimental` on\n  the Inngest client.\n- `createScorer()` is imported from `inngest/experimental` and the scorer must\n  be registered in the serve handler with other functions.\n- Sessions sent through the TypeScript SDK require v4.7.0 or later.\n- Step experiments require v4.8.0 or later for `group.experiment()` and the\n  `experiment` helper from `inngest`.\n\nClient shape:\n\n```typescript\nimport { Inngest } from \"inngest\";\nimport { scoreMiddleware } from \"inngest/experimental\";\n\nexport const inngest = new Inngest({\n  id: \"support-agent\",\n  middleware: [scoreMiddleware()],\n});\n```\n\n## Sessions\n\nAdd sessions when events belong to a user flow, conversation, ticket, import,\nor agent task that someone will inspect repeatedly.\n\n```typescript\nawait inngest.send({\n  name: \"support/ticket.created\",\n  data: {\n    ticketId: \"tk_123\",\n    message: \"I can't sign in.\",\n  },\n  meta: {\n    sessions: {\n      ticket_id: \"tk_123\",\n    },\n  },\n});\n```\n\nRules:\n\n- Use stable, non-secret, high-cardinality IDs.\n- Keep the key generic, such as `conversation_id`; put the actual ID in the\n  value.\n- Pass sessions explicitly through `step.invoke()` and `step.sendEvent()` when\n  downstream runs should join the same session.\n- Do not use sessions for labels like `environment: prod`; use Insights for\n  that style of filtering.\n\n## Direct Scoring\n\nUse direct scoring when the score is known during the function run.\n\nGood direct scores:\n\n- guardrail pass/fail\n- JSON validity\n- retrieval confidence\n- tool success\n- model confidence\n- inline LLM-as-judge result\n\n```typescript\nexport const answerTicket = inngest.createFunction(\n  {\n    id: \"answer-ticket\",\n    triggers: [{ event: \"support/ticket.created\" }],\n  },\n  async ({ event, step }) => {\n    const answer = await step.run(\"generate-answer\", () =>\n      generateAnswer(event.data.ticketId)\n    );\n\n    const passed = await step.run(\"check-answer\", () => validateAnswer(answer));\n\n    await step.score(\"score-answer-quality\", {\n      name: \"answer-quality\",\n      value: passed,\n    });\n\n    return { answer, passed };\n  }\n);\n```\n\nUse stable score names. Changing a score name creates a separate metric.\nScore values must be finite numbers or booleans.\n\n## Deferred Scoring\n\nUse deferred scoring when the useful signal arrives after the parent workflow\nfinishes, such as user feedback, conversion, ticket reopen, retention, or a\nslow LLM-as-judge run.\n\n```typescript\nimport { createScorer } from \"inngest/experimental\";\nimport { z } from \"zod\";\n\nexport const feedbackScorer = createScorer(\n  inngest,\n  {\n    id: \"support-feedback-scorer\",\n    schema: z.object({ ticketId: z.string() }),\n  },\n  async ({ event, step }) => {\n    const feedback = await step.waitForEvent(\"wait-for-feedback\", {\n      event: \"support/feedback.received\",\n      timeout: \"7d\",\n      if: `async.data.ticketId == '${event.data.ticketId}'`,\n    });\n\n    return {\n      name: \"user-feedback\",\n      value: feedback?.data.helpful ? 1 : 0,\n    };\n  }\n);\n```\n\nTrigger it from the producing function:\n\n```typescript\nasync ({ event, step, defer }) => {\n  const answer = await step.run(\"generate-answer\", () =>\n    generateAnswer(event.data.ticketId)\n  );\n\n  defer(\"score-feedback\", {\n    function: feedbackScorer,\n    data: { ticketId: event.data.ticketId },\n  });\n\n  return { answer };\n}\n```\n\nGuardrails:\n\n- Register scorers with `serve({ functions: [...] })`.\n- `defer()` is fire-and-forget; do not `await` it.\n- The parent run is attributed automatically for deferred scorers.\n- Return a default score or `null` when a signal times out, based on the\n  product semantics.\n\n## Experiments\n\nUse `group.experiment()` when comparing prompts, models, providers, tools,\nworkflow rewrites, or operational settings against real traffic.\n\n```typescript\nimport { experiment } from \"inngest\";\n\nconst { result, variant, experimentRef } = await group.experiment(\n  \"answer-style\",\n  {\n    variants: {\n      concise: () => step.run(\"answer-concise\", () => answerConcise(event.data)),\n      detailed: () =>\n        step.run(\"answer-detailed\", () => answerDetailed(event.data)),\n    },\n    select: experiment.bucket(event.data.accountId, {\n      weights: { concise: 50, detailed: 50 },\n    }),\n  }\n);\n```\n\nSelection strategy:\n\n- `experiment.weighted()` for run-level traffic splits.\n- `experiment.bucket(stableId, { weights })` when a user, account, or tenant\n  should usually keep the same experience.\n- `experiment.custom()` when assignment comes from a database, flag service, or\n  rollout table.\n- `experiment.fixed()` to force one variant during testing or after choosing a\n  winner.\n\nRules:\n\n- Each variant callback must call at least one `step.*` tool.\n- Keep experiment IDs and variant names stable because they appear in traces.\n- The selected variant is memoized for retries and replays.\n- Persist `experimentRef` and the parent run ID if a later, separate process\n  will score the selected variant.\n\nWhen deferred scoring a variant, pass the ref:\n\n```typescript\ndefer(\"score-answer-feedback\", {\n  function: feedbackScorer,\n  data: { ticketId: event.data.ticketId },\n  experiment: experimentRef,\n});\n```\n\nWhen scoring from a later run explicitly:\n\n```typescript\nawait inngest.score.experiment({\n  name: \"clickthrough\",\n  value: 1,\n  experiment: experimentRef,\n  runId: originalRunId,\n});\n```\n\n## Brownfield Migration\n\nWhen adding Agent Evals to an existing app:\n\n1. Search for existing agent/workflow outputs and where users act on them.\n2. Find durable identifiers already in the domain: conversation ID, ticket ID,\n   account ID, import ID, run ID, or recommendation ID.\n3. Add `meta.sessions` at event producers before building dashboards around\n   correlation.\n4. Add one direct score where the outcome is already known.\n5. Add a deferred scorer only after the signal event exists or can be emitted\n   cleanly.\n6. Add `group.experiment()` only around one isolated decision at a time.\n7. Keep old prompt/model/tool behavior stable until scoring proves the new path.\n\nUse `inngest-brownfield-audit` first when the repo has many candidate\nworkflows. Use `inngest-agents` with this skill when the work is an AgentKit or\ndurable agent workflow.\n\n## Anti-Patterns\n\n- Starting with a prompt experiment before naming the outcome metric.\n- Scoring with unstable names, strings, objects, `NaN`, or `Infinity`.\n- Forgetting `scoreMiddleware()` and then assuming `step.score()` is missing.\n- Running scorers in an external queue when the scorer needs durable waits.\n- Keeping experiment assignment only in memory.\n- Scoring an experiment variant from a later run without the original `runId`.\n- Using sessions for low-cardinality labels or sensitive personal data.\n- Leaving model/tool work outside steps, making traces unable to explain the\n  score.\n\n## Verification\n\n- Typecheck the Inngest client, functions, scorer schemas, and serve handler.\n- Confirm scorers are registered with the serve endpoint.\n- Test the producer emits events with expected `meta.sessions`.\n- Test direct scores use stable names and finite number/boolean values.\n- Test deferred scorer timeout behavior.\n- For experiments, test that each variant callback contains step work and that\n  later scoring receives `experimentRef` plus the original run ID when needed.\n- If possible, run the Inngest dev server and inspect traces, sessions, scores,\n  and experiment variant selection in the dashboard or through available CLI/API\n  tooling.\n"
}