evaluations/agent-loop.json

{
  "name": "Build an Autonomous Agent Loop with Iteration Cap",
  "skills": ["conductor"],
  "query": "Build me an autonomous agent loop that uses an HTTP tool, tries up to 10 think/act iterations, accumulates chat history across iterations, and stops when the model emits an answer",
  "expected_behavior": [
    "Step 1: Consult examples/ai-agent-loop.md, references/workflow-definition.md (DO_WHILE + LLM_CHAT_COMPLETE), references/graaljs-gotchas.md, and references/template-resolution.md",
    "Step 2: Seed chat history with a SET_VARIABLE task that initializes workflow.variables.messages with a system prompt + user question, using Conductor's {role, message} schema (NOT {role, content})",
    "Step 3: Construct the DO_WHILE task with evaluatorType: \"graaljs\" at the top level, NOT just on inner tasks",
    "Step 4: loopCondition is an IIFE: (function(){ return $.<loopRef>['iteration'] < $.max_iterations && $.final_response === ''; })();  — NOT an if/else statement, NOT a named function",
    "Step 5: DO_WHILE's inputParameters wires (a) its own output back in (self-reference pattern), (b) max_iterations from workflow.input, (c) final_response from workflow.variables — because $.workflow.* is NOT in scope inside loopCondition",
    "Step 6: The think task uses LLM_CHAT_COMPLETE with jsonOutput: true and reads messages from ${workflow.variables.messages}",
    "Step 7: A SWITCH routes on the parsed action; defaultCase is empty ([]) so malformed LLM emissions do not poison state",
    "Step 8: Tool branch is HTTP (optional: true) followed by a single JSON_JQ_TRANSFORM that does BOTH (a) stringify the structured response via `tojson` AND (b) concatenate new messages onto the existing chat history via `.current + [...]`. This avoids INLINE entirely (`JSON.stringify` returns `\"{}\"` on Java-Map-backed proxies, `String()` returns `{k=v}`). Reference fields explicitly — `.current`, `.tool_result` — because Conductor's JQ input is the whole inputParameters map.",
    "Step 9: A SET_VARIABLE then writes the merged messages array back to workflow.variables.messages — `messages: ${<jqRef>.output.result}`. The SET_VARIABLE must NOT overwrite with just the new entries (that loses the system prompt and prior turns).",
    "Step 10: Answer branch sets workflow.variables.final_response via SET_VARIABLE",
    "Step 11: Workflow-level timeoutSeconds + timeoutPolicy: TIME_OUT_WF are set",
    "Step 12: outputParameters reads ${workflow.variables.final_response} and ${<loopRef>.output.iteration} (NOT ${<loopRef>.iteration})",
    "Step 13: Write the workflow JSON to a file before registering"
  ],
  "success_criteria": [
    "DO_WHILE task has `evaluatorType: \"graaljs\"` at the top level",
    "`loopCondition` is in IIFE form (e.g. `(function(){ return ...; })();`) — not the older `if (X) { true; } else { false; }` statement form",
    "`loopCondition` includes a hard iteration cap (e.g. `iteration < 10`)",
    "DO_WHILE's `inputParameters` includes the self-reference pattern (its own taskReferenceName mapped to `${<ref>.output}`)",
    "Workflow-level inputs/variables used inside the loopCondition are plumbed through `inputParameters` — the script does NOT use `$.workflow.input.*` or `$.workflow.variables.*`",
    "LLM_CHAT_COMPLETE uses Conductor's `{role, message}` schema, NOT `{role, content}`",
    "LLM_CHAT_COMPLETE sets `jsonOutput: true` (or otherwise explicitly handles result parsing) so the SWITCH can route on a parsed `action` field",
    "SWITCH that routes on the LLM action has an empty (`[]`) `defaultCase` so junk replies don't overwrite final state",
    "Stringifying the HTTP tool result for embedding in the next LLM message uses JSON_JQ_TRANSFORM with `tojson` — NOT INLINE with `JSON.stringify` or `String()`",
    "Chat history is **accumulated, not replaced**, across iterations: the JSON_JQ_TRANSFORM concatenates the existing messages with the new entries (e.g. `.current + [...]`) and SET_VARIABLE writes that merged array back. A SET_VARIABLE that writes only the new entries — losing the system prompt and earlier turns — is wrong.",
    "JQ `queryExpression` references fields by name (e.g. `.current`, `.tool_result`, `.data`) rather than bare `.` — Conductor's JQ task passes the full `inputParameters` map as the JQ input, so bare `.` is the `{field1: ..., field2: ...}` wrapper, not the value.",
    "Workflow-level `timeoutSeconds` AND `timeoutPolicy: TIME_OUT_WF` are set",
    "outputParameters reads the iteration counter as `${<loopRef>.output.iteration}` (with `.output`), NOT `${<loopRef>.iteration}`",
    "Agent does not invent a non-existent task type like AGENT_LOOP or REACT_LOOP"
  ]
}