{"id":53709,"plugin_id":"plugins_6ac012b82680819195f8a26cbe32c246","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-10-24T13:54:52.742Z","digest":"4ed105cddcbb8eb2f596d4b3d0f8336be4aa62508735650817d169bbcc6c7f90","against":null,"payload":{"description":"Set up GenAI tracing for an LLM or agent application in Domino by checking MLflow 3.2.0 and Domino SDK requirements, then adding a tracing_setup.py helper, evaluators, a config.yaml, and @add_tracing / DominoRun wiring. Use when the user asks to add tracing, observability, or evaluation to an agent or LLM app running on Domino.","included_files":[{"relative_path":"assets/config.yaml","size_in_bytes":1485},{"relative_path":"assets/tracing_setup.py","size_in_bytes":6298}],"name":"domino-trace-setup","skill_md_contents":"---\nname: domino-trace-setup\ndescription: Set up GenAI tracing for an LLM or agent application in Domino by checking MLflow 3.2.0 and Domino SDK requirements, then adding a tracing_setup.py helper, evaluators, a config.yaml, and @add_tracing / DominoRun wiring. Use when the user asks to add tracing, observability, or evaluation to an agent or LLM app running on Domino.\n---\n\n# Set Up Domino GenAI Tracing\n\nInstrument an agent or LLM application so its calls, spans, and evaluator scores show up in\nDomino experiments. For traditional ML tracking, use the `domino-experiment-setup` skill instead.\n\n## Inputs\n\n1. **LLM framework**: `openai` (default), `anthropic`, or `langchain`. Detect from the code if possible.\n2. **Agent name**: name for the traced entry-point function.\n3. **Evaluator**: whether to add quality scoring, and whether to use an LLM-as-judge.\n4. **Aggregated metrics**: which evaluator metrics to summarize per run.\n\n## Steps\n\n1. **Check requirements.** Confirm `mlflow==3.2.0` and the Domino SDK with agents support are\n   installed. If not, tell the user to add them to the environment's Dockerfile:\n\n   ```dockerfile\n   RUN pip install mlflow==3.2.0\n   RUN pip install \"dominodatalab[data,agents]==2.2.0\"\n   ```\n\n2. **Copy helpers** from this skill's `assets/` folder into the project:\n   - `assets/tracing_setup.py`: `setup_tracing(framework)`, `create_evaluator()`,\n     `create_llm_judge_evaluator()`, and `get_aggregation_metrics()`.\n   - `assets/config.yaml`: agent configuration logged with `DominoRun(agent_config_path=...)`.\n     Replace the example model names with the ones the user actually calls.\n3. **Instrument the agent.** Decorate the entry point with `@add_tracing` and run it inside\n   `DominoRun`, following the example below. Keep the user's existing logic unchanged.\n4. **Explain how to view traces** (see \"Viewing traces\").\n\n## Example traced agent\n\n```python\nimport mlflow\nfrom domino.agents.tracing import add_tracing\nfrom domino.agents.logging import DominoRun\nfrom openai import OpenAI\n\nmlflow.openai.autolog()\nclient = OpenAI()\n\ndef quality_evaluator(inputs, output):\n    return {\n        \"response_length\": len(output.get(\"response\", \"\")),\n        \"confidence\": output.get(\"confidence\", 0),\n    }\n\n@add_tracing(name=\"my_agent\", evaluator=quality_evaluator)\ndef my_agent(query: str) -> dict:\n    response = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=[{\"role\": \"user\", \"content\": query}],\n    )\n    return {\n        \"response\": response.choices[0].message.content,\n        \"confidence\": 0.9,\n        \"model\": \"gpt-4o-mini\",\n    }\n\nif __name__ == \"__main__\":\n    aggregated_metrics = [(\"response_length\", \"mean\"), (\"confidence\", \"mean\")]\n\n    with DominoRun(\n        run_name=\"example-run\",\n        agent_config_path=\"config.yaml\",\n        custom_summary_metrics=aggregated_metrics,\n    ) as run:\n        for query in [\"What is machine learning?\", \"Explain neural networks\"]:\n            result = my_agent(query)\n            print(f\"Q: {query}\\nA: {result['response'][:100]}...\\n\")\n        print(f\"Run ID: {run.run_id}\")\n```\n\n## Framework autologging\n\n| Framework | Setup |\n|---|---|\n| OpenAI | `mlflow.openai.autolog()` then `OpenAI()` |\n| Anthropic | `mlflow.anthropic.autolog()` then `Anthropic()` |\n| LangChain | `mlflow.langchain.autolog()` then e.g. `ChatOpenAI(model=\"gpt-4o-mini\")` |\n\n## Viewing traces\n\n1. Go to **Experiments** in Domino.\n2. Open the experiment the run logged to (by default `tracing-{username}`).\n3. Select the run and open the **Traces** tab.\n\nThe trace view shows the span tree (agents, tools, messages), token usage, latency, and evaluator\nscores.\n\nFor evaluators, multi-agent patterns, and `DominoRun` details, use the `domino-genai-tracing` skill.\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}