{"id":53745,"plugin_id":"plugins_6ac012b82680819195f8a26cbe32c246","kind":"skill","collection_source":"skill_api","comparison_source":null,"observed_at":"2026-10-24T13:54:56.964Z","digest":"d90810b48c3eafad47422856443f717f24958e4e5264811256d5f5963a65a491","against":null,"payload":{"description":"Trace and evaluate GenAI applications including LLM calls, agents, RAG pipelines, and multi-step AI systems in Domino. Uses the Domino SDK (@add_tracing decorator, DominoRun context) with MLflow 3.2.0. Captures token usage, latency, cost, tool calls, and errors. Supports LLM-as-judge evaluators and custom metrics. Use when building agents, debugging LLM applications, or needing audit trails for GenAI systems.","included_files":[{"relative_path":"ADD-TRACING-DECORATOR.md","size_in_bytes":12508},{"relative_path":"DOMINO-RUN.md","size_in_bytes":8763},{"relative_path":"EVALUATORS.md","size_in_bytes":12676},{"relative_path":"MULTI-AGENT-EXAMPLE.md","size_in_bytes":22290},{"relative_path":"TRACING-SETUP.md","size_in_bytes":7258}],"name":"domino-genai-tracing","skill_md_contents":"---\nname: domino-genai-tracing\ndescription: Trace and evaluate GenAI applications including LLM calls, agents, RAG pipelines, and multi-step AI systems in Domino. Uses the Domino SDK (@add_tracing decorator, DominoRun context) with MLflow 3.2.0. Captures token usage, latency, cost, tool calls, and errors. Supports LLM-as-judge evaluators and custom metrics. Use when building agents, debugging LLM applications, or needing audit trails for GenAI systems.\n---\n\n# Domino GenAI Tracing Skill\n\nThis skill provides comprehensive knowledge for tracing and evaluating GenAI applications in Domino Data Lab, including LLM calls, agents, RAG pipelines, and multi-step AI systems.\n\n## Two Deployment Modes\n\nGenAI tracing works differently depending on where your code runs:\n\n| Mode | Where traces appear | When to use |\n|------|-------------------|-------------|\n| **Deployed App** (Production) | App **Performance tab** | FastAPI/Flask apps deployed as Domino Apps |\n| **Development / Evaluation** | **Experiments** UI | Batch scripts, Domino Jobs, Workspaces |\n\n**Critical difference:** In a deployed Domino App, Domino auto-creates an experiment named `agent_experiment_{app_id}` and the Performance tab reads from it. If you call `mlflow.set_experiment()` or wrap calls in `DominoRun()`, traces go to your custom experiment instead — and the Performance tab won't see them.\n\n## Key Concepts\n\n### What GenAI Tracing Captures\n\nThe Domino SDK automatically captures:\n- **Token usage** - Input and output tokens per call\n- **Latency** - Time for each operation\n- **Cost** - Estimated cost per call\n- **Tool calls** - Function/tool invocations\n- **Errors** - Exceptions and failure modes\n- **Model parameters** - Temperature, max_tokens, etc.\n\n### Core Components\n\n1. **`@add_tracing` decorator** - Wraps agent functions to capture traces (works standalone — no `DominoRun` required)\n2. **`mlflow.start_span()`** - Creates child spans for LLM calls and tool executions inside the agent loop\n3. **`DominoRun` context manager** - Groups traces into runs for development/evaluation (Experiments UI only)\n4. **Evaluators** - Custom functions to score outputs\n5. **MLflow integration** - View traces in Experiment Manager or App Performance tab\n\n## Related Documentation\n\n- [TRACING-SETUP.md](./TRACING-SETUP.md) - Environment & SDK setup\n- [ADD-TRACING-DECORATOR.md](./ADD-TRACING-DECORATOR.md) - @add_tracing usage, span_type, autolog_frameworks\n- [DOMINO-RUN.md](./DOMINO-RUN.md) - DominoRun context manager (development/evaluation only)\n- [EVALUATORS.md](./EVALUATORS.md) - LLM-as-judge, custom evaluators\n- [MULTI-AGENT-EXAMPLE.md](./MULTI-AGENT-EXAMPLE.md) - Complete multi-agent example\n\n## Quick Start — Deployed App (Production)\n\nFor a FastAPI app deployed as a Domino App. Traces appear in the **App Performance tab**.\n\n```python\n# main.py — lifespan: autolog only, NO set_experiment, NO DominoRun\nimport mlflow\nfrom fastapi import FastAPI\nfrom contextlib import asynccontextmanager\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    # Each autolog in its own try/except so one failure doesn't block the other\n    try:\n        mlflow.openai.autolog()\n    except Exception:\n        pass\n    try:\n        mlflow.anthropic.autolog()\n    except Exception:\n        pass\n    yield\n\napp = FastAPI(lifespan=lifespan)\n```\n\n```python\n# orchestrator.py — @add_tracing on the core agent function\nimport mlflow\nfrom domino.agents.tracing import add_tracing\n\n@add_tracing(\n    name=\"agent_turn\",\n    span_type=\"AGENT\",\n    autolog_frameworks=[\"openai\", \"anthropic\"],\n)\nasync def run_agent(messages: list[dict]) -> dict:\n    # LLM calls and tool calls go here (see ADD-TRACING-DECORATOR.md)\n    ...\n\n# router.py — call the traced function directly, NO DominoRun wrapper\n@app.post(\"/chat\")\nasync def chat(request: ChatRequest):\n    return await run_agent(request.messages)\n```\n\n## Quick Start — Development / Evaluation\n\nFor batch scripts, Domino Jobs, or Workspaces. Traces appear in the **Experiments UI**.\n\n```python\nimport mlflow\nfrom domino.agents.tracing import add_tracing\nfrom domino.agents.logging import DominoRun\n\nmlflow.openai.autolog()\n\n@add_tracing(name=\"my_agent\", autolog_frameworks=[\"openai\"])\ndef my_agent(query: str) -> str:\n    response = llm.invoke(query)\n    return response\n\n# DominoRun groups traces into a run visible in Experiments\nwith DominoRun() as run:\n    result = my_agent(\"What is machine learning?\")\n```\n\n## Framework Support\n\n| Framework | Auto-log Command |\n|-----------|------------------|\n| OpenAI | `mlflow.openai.autolog()` |\n| Anthropic | `mlflow.anthropic.autolog()` |\n| LangChain | `mlflow.langchain.autolog()` |\n\n## Viewing Traces\n\n### Deployed Apps (Production)\n\n1. Navigate to your Domino App\n2. Click the **Performance** tab\n3. Traces appear automatically (routed to `agent_experiment_{app_id}`)\n\n### Development / Evaluation\n\n1. Navigate to **Experiments** in your Domino project\n2. Select the experiment\n3. Select a run\n4. View the **Traces** tab for span tree visualization\n\n## Blueprint Reference\n\nOfficial GenAI Tracing Tutorial:\nhttps://github.com/dominodatalab/GenAI-Tracing-Tutorial\n\n## Documentation Links\n\n- Domino GenAI Tracing: https://docs.dominodatalab.com/en/cloud/user_guide/fc1922/set-up-and-run-genai-traces/\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}