← ManufactCONTENT HISTORY
WHAT CHANGED
Update to Manufact
Snapshot Nov 10, 2026 · 06:18 UTC · version 2.2.0
Collection source: downloaded plugin package.
First saved snapshot
No earlier snapshot is available to establish a change.
Save Manufact and check future changes in Following.
Up to 3 plugins free · no card required.
Compare saved observations
Download comparison JSONFull technical diff · 0 changed fields
Full snapshot data
{
"description": "Test MCP servers and plugins across supported clients; create evals and investigate failures. Use when a user wants to check whether models pick the right tools and give useful answers in real clients, compare models or clients, rerun tests after a change, or understand why a test failed. Discover supported clients, models, and agent runtimes with get_eval_options.",
"included_files": [],
"name": "mcp-plugin-testing",
"skill_md_contents": "---\nname: mcp-plugin-testing\ndescription: Test MCP servers and plugins across supported clients; create evals and investigate failures. Use when a user wants to check whether models pick the right tools and give useful answers in real clients, compare models or clients, rerun tests after a change, or understand why a test failed. Discover supported clients, models, and agent runtimes with get_eval_options.\nmetadata:\n owner: manufact\n---\n\n# Test an MCP server or plugin in real clients\n\nEvals run a saved conversation against the server's production endpoint in one\nor more clients, then a judge scores the result against passing criteria. They\nshow whether models actually discover and call the right tools, with the right\narguments, and give useful answers. They complement the publishing checklist,\nwhich inspects protocol and metadata only (see the `mcp-plugin-publishing`\nskill).\n\n## 1. Pick the server\n\n- Reuse organization and server IDs already established in the conversation.\n- Otherwise call `list_servers` without `organizationId` to search every\n accessible organization, and confirm the server with `get_server`. Eval tools\n need both the server's `organizationId` and `serverId`.\n- The server needs a healthy production deployment, because evals always target\n the production endpoint. If it has none, use the `mcp-plugin-deploy` skill\n first. To test a branch, promote or merge it first; preview URLs are not\n eval targets.\n- If the server requires OAuth, complete `authenticate_deployment` first as\n described in `mcp-plugin-publishing`, so clients can connect.\n\n## 2. Discover clients and models\n\nCall `get_eval_options` before choosing clients. Use only what it returns:\n\n- **Inspector**: the Manufact chat client. Choose any number of OpenRouter\n models with `list_eval_models` (filter with `query` or `provider`). Each model\n runs as its own execution. Omitting clients uses Inspector with the default\n model.\n- **Web clients** (`webClients`): ChatGPT and Claude on the web. Each runs once\n per suite with no model choice and does not enforce `disallowedTools`.\n- **Runtimes**: agent harnesses such as Codex, Claude Code, or Cursor. Pick\n only runtimes with `available: true`, choose a model from that runtime's\n `models`, and tell the user about any `limitations` or the `reason` it is\n unavailable.\n\nOther MCP clients can still connect to the server's MCP URL, but they can't be\ntested here unless `get_eval_options` lists them. Say so instead of implying\ncoverage.\n\n## 3. Create or update suites\n\nCheck `list_eval_suites` first and reuse a matching suite with\n`update_eval_suite` rather than creating a duplicate. Read one with\n`get_eval_suite` when you need its full configuration.\n\nCreate a suite with `create_eval_suite` for each behavior worth checking:\n\n- **`flow`**: the conversation in order. Use realistic user wording, not the\n tool's name. Add a `tool_call` step after a user step when the model must\n call a specific tool; match only the arguments that matter (`\"any\"`,\n `contains`, or `pattern`).\n- **`rubric`**: concrete, observable passing criteria the judge can check.\n- **`threshold`**: the minimum judge score to pass; the default is 0.7.\n- **`clients`**: the clients and models chosen in step 2.\n- **`disallowedTools`**: tools the model must not call, such as destructive\n tools in a read-only scenario.\n\nCover the main jobs users bring to the server, plus at least one case where the\nmodel should decline or ask for clarification. If you can read the server's\nsource, derive the cases from its actual tools.\n\n## 4. Run the suites\n\n1. If the user asked to run tests, call `run_eval_suites` with up to ten\n `suiteIds` without asking again. If you are starting runs on your own\n initiative, say which suites and clients will run first: each execution\n uses plan resources.\n2. Each suite starts its own background run. Read each receipt's `outcome`; a\n suite that was `not_started` or has an `error` did not run.\n3. If the start response was lost or uncertain, check `list_eval_runs` before\n starting again. `cancel_eval_run` stops a run that is no longer needed.\n\n## 5. Read the results\n\n1. Call `get_eval_report` with the returned `runIds`. Runs take a few minutes.\n While a run is still in progress, tell the user and check again when they\n ask; don't poll in a tight loop.\n2. Report per client and model: pass or fail, score, and the tool calls made.\n3. Use `list_eval_runs` to compare with earlier runs of the same suite.\n\n## 6. Investigate failures\n\nFor each failed execution, walk through the judge findings and the tool calls:\n\n| Symptom | Likely cause and fix |\n| --- | --- |\n| The expected tool was never called | The tool's title or description does not match how users ask. Improve the description, or check that the tool is exposed to that client. |\n| The wrong tool was called | Two tools overlap. Sharpen both descriptions so each states when to use it. |\n| Right tool, wrong arguments | Parameter descriptions are vague, or the schema allows ambiguous values. Describe each parameter and its format. |\n| The tool errored | Read `get_deployment_runtime_logs` or the `mcp-plugin-analytics` skill's issue tools for the server-side error. |\n| Tools worked but the answer failed the rubric | The output lacks what the answer needs, or the rubric is stricter than intended. Fix the output or the rubric, not both at once. |\n| Only one client fails | Check that client's `limitations` and whether it supports the feature in use, such as widgets or authentication. |\n| The run failed before executing | Connection or authentication failed. Fix access, then rerun. |\n\nFix the cause in the server's code, push, and let the deployment finish. Rerun\nonly the affected suites, once something has changed. Report what passed, what\nfailed and why, and what remains untested, including clients that\n`get_eval_options` did not offer.\n"
}SHA-256 of public snapshot: 0c39ee8fc5cc99738dab7e0fdf3d1eda7287999a597cf57b7df78b51ae043fa5