Skill instructions
manufact-analytics4.67 KB
View saved version →
---
name: manufact-analytics
description: Read Manufact Cloud server analytics and investigate usage, errors, latency, clients, actors, sessions, or issues. Choose focused evidence tools or catalog-driven metrics, with scoped read-only SQL for questions the typed metrics cannot express. Use for existing server telemetry, not deployment changes or payload-capture configuration.
metadata:
owner: manufact
---
# Answer questions from server analytics
Find the server with `list_organizations`, `list_servers`, and `get_server` when
its ID is not already known. Use returned IDs. Keep the user's server, time
window, environment, and client or tool selections throughout the analysis.
## Choose the smallest useful read
- Use `get_server_analytics_summary` for a quick overview and comparison with
the preceding period. It returns current-period buckets and totals.
- Use `get_server_issues_trend` for aggregate issue occurrence trends across
all environments, independently of a paginated issue list.
- Use `list_server_issues` and `get_server_issue` for grouped errors, or
`list_server_analytics_sessions` and `get_server_analytics_session` for
individual interactions and their events. Read a list first; fetch details
only for the records relevant to the user's question. Actor-related groupings
are available through the catalog-driven metrics and SQL queries.
- Use `get_server_analytics_catalog` before composing a typed metrics or SQL
query. Discover dataset columns, metrics, dimensions, permitted filters,
units, time scopes, and query limits from the live catalog. Do not invent
fields or assume all metrics work on all datasets.
- Use `query_server_analytics_metrics` for grouped counts, trends, error
rates, and latency or token statistics that the catalog can express.
- Use `query_server_analytics_sql` when a useful read needs an expression or
grouping that the typed metrics cannot express. Use only catalog-listed
datasets and columns, with the same scoped time window and filters. The
backend constrains the query to the selected server and permits read-only
queries only; there is no unrestricted database connection.
- Use `open_analytics_view` when the user wants an interactive dashboard.
Opening the view does not provide analytical evidence in chat. Read the
relevant tools to substantiate an answer. The view's app-only
`get_server_analytics_dashboard_data` fetches a bounded dashboard batch
internally; section failures can be independent.
Read [references/queries.md](references/queries.md) for query examples,
pagination, scope distinctions, and recovery guidance.
## Explain what the evidence means
Use the response's units and counting definition. Interactions, transport
sessions, requests, and tool calls are different populations. The summary's
`sessions` field counts distinct interactions. Do not compare it directly with
a transport-session count. Summary errorRate is a percentage; use the catalog
for the units of typed metrics rather than assuming the same scale.
Report verified and approximate actor counts separately. Verified actors may
identify a conversation or token as well as an account; they do not necessarily
count people. Approximate actors are IP-derived. Distinct actor and session
counts use sketches by default; accuracy is separate from identity trust.
Estimated output tokens are not billed model tokens. Latency percentiles can
be approximate, and a null metric is unavailable rather than zero. Compute
an average only from its stated
numerator and denominator, excluding events without the required measurement.
Carry response metadata into conclusions: requested versus effective window,
retention limits, approximation, partial results, warnings, and truncation.
An empty result means no matching retained data was returned; it does not
prove the server has never received traffic. A truncated top-N result is not
the complete population. Do not sum distinct counts across groups or time
buc
Preview of the first 4,000 characters. Read full instructions →
Referenced files: 1 — file references are metadata, not package contents.
manufact-deploy6.93 KB
View saved version →
---
name: manufact-deploy
description: Use when a user wants to deploy an MCP server or MCP app to Manufact Cloud, redeploy or promote a deployment, set up preview branches, environment variables, or domains, or find out why a deployment failed. Covers repository requirements, the GitHub-to-live-URL flow, and the build/runtime log troubleshooting loop.
metadata:
owner: manufact
---
# Deploy an MCP server on Manufact Cloud
A server on Manufact Cloud is built from a GitHub repository. Every deployment
clones one branch, detects the runtime, builds it, and starts it in a
container. Before switching traffic, Manufact health-checks the new deployment
with an MCP `initialize` request. Production keeps serving the previous
deployment until the new one passes.
## 1. Check that the repository is deployable
If you can see the user's code, check these before the first deploy. Most
failed first deploys trace back to one of them. See
[references/build-runtimes.md](references/build-runtimes.md) for detection rules.
- **The MCP endpoint is served at `/mcp`.** The health check sends its request to
`<base>/mcp`.
- **The server listens on `0.0.0.0` and the `PORT` environment variable.**
Manufact injects both. Presets default to port 3000 for Node and 8000 for
Python. Servers that bind to `localhost` or a hard-coded port fail the health
check.
- **The runtime can be detected.**
- Node needs `package.json` plus a lockfile.
- Python needs `pyproject.toml` or `requirements.txt`.
- Anything else, Deno included, needs a committed `Dockerfile`.
- **Build and start scripts exist.** Node servers need a `start` script; a `build`
script runs only if it exists. Otherwise, set `buildCommand` and `startCommand`.
- **Monorepos set `rootDir`** to the server's subdirectory.
- **Secrets are not committed.** They become environment variables (step 4).
- **The repository is on GitHub and pushed.** Manufact deploys the remote branch,
not local changes.
mcp-use projects scaffolded with `npx create-mcp-use-app@latest` meet all of
these by default.
## 2. Find or create the server
1. **Pick the organization.** Use `list_organizations`, choosing as described in
the `manufact-onboarding` skill.
2. **Look for an existing server** with `list_servers`. Confirm a candidate by
its repository (`get_server`), not its name. Reuse a matching server unless
the user explicitly wants a second one, for example another root in a monorepo.
3. **Check GitHub access** with `list_github_installations`. You need an
installation where `isActive` is true and whose account owns the repository.
- If there is none, or the repository isn't covered, call
`get_github_connect_url`.
- Give the user the link. They finish installing or granting access on
GitHub; when they confirm, check again with `list_github_installations`.
- Repositories owned by a GitHub organization need the Hobby plan or higher.
4. **Create the server** with `create_server`.
- Pass `organizationId`, `githubConnectionId`, and `repoFullName` (`owner/repo`).
- Add `branch`, `rootDir`, commands, or `port` only when they differ from the
defaults.
- `create_server` **also starts the first deployment**, so don't call `deploy`
afterward.
5. **Existing server, new code or settings:** call `deploy` with `serverId`.
Omitting `branch` deploys the production branch. After pushing to the
production branch you usually don't need to call it at all: pushes deploy
automatically.
## 3. Follow the deployment to a result
The steps shown in the dashboard are Cloning → Analyzing → Building →
Deploying → Health Checks → Complete.
- Call `get_deployment` with the returned `deploymentId` to check status.
Builds usually take a few minutes.
- An MCP URL alone does not mean the deployment is healthy.
- **Success:** give the user the MCP URL (`https://<slug>.run.mcp-use.com/mcp`)
and offer next steps. Those can be connecting it to a client, running the
publishinPreview of the first 4,000 characters. Read full instructions →
Referenced files: 2 — file references are metadata, not package contents.
manufact-getting-started7.96 KB
View saved version →
---
name: manufact-getting-started
description: Get started with Manufact Cloud to build, deploy, test, and monitor MCP servers and plugins. Use right after a user connects Manufact, or when they ask what Manufact does or where to start. Covers the account model, a quick-start template deploy that needs no GitHub, and routing to deployment, testing, analytics, or publishing for the most popular AI Agents.
metadata:
owner: manufact
---
# Get started with Manufact Cloud
Manufact Cloud builds, hosts, tests, and monitors MCP servers and MCP apps,
which users install as plugins or connectors in their clients. It is built by
the team behind the open-source mcp-use SDKs, but it hosts servers built with
any framework. A deployed server gets a live MCP URL that any MCP client can
connect to, including ChatGPT, Claude, Cursor, Grok, and Muse.
Not every feature covers every client. Read support from the tools rather than
from this list: `get_eval_options` returns the clients and runtimes that evals
can run in, and the publishing checklist reports readiness only for the
platforms it returns.
The user is already signed in: connecting this MCP server completed the OAuth
flow. Never ask them for passwords, API keys, or tokens.
## How Manufact Cloud is organized
```text
User
└── Organizations (a user belongs to one or more)
└── Servers (each server is one MCP server)
└── Deployments (each server has many; one serves production)
```
- **User**: the signed-in account. It can belong to several organizations.
- **Organization**: owns servers, GitHub connections, and members.
New accounts get a Primary organization automatically.
- **Server**: one MCP server. It has a source repository, build settings,
environment variables, domains, analytics, checklist runs, eval suites,
and an optional public chat page.
- **Deployment**: one build of the server from a branch and commit.
- The production branch (usually `main`) deploys on every push and serves
`https://<slug>.run.mcp-use.com/mcp`.
- Other branches get preview URLs.
- A newer deployment can fail while an older one keeps serving traffic.
Tools follow the same hierarchy. Organization IDs scope servers, server IDs
scope deployments, and so on down. Use only IDs that tools return or that the
user gives you. Never guess an ID, and reuse IDs already established in the
conversation instead of looking them up again.
## First conversation
1. **Introduce Manufact briefly.** One or two sentences is enough, unless the user
asks for more.
2. **Look at the account before suggesting anything.** Call `list_servers`
without `organizationId`; it lists servers across every accessible
organization, grouped by organization. Call `list_organizations` only when
you need organization names or an organization with no servers.
3. **Summarize what you found**, in plain language: which organizations, how
many servers, and each server's name and latest deployment status. Call
`get_server` only for a server the user wants to look at more closely.
4. **Offer the next steps that fit their state**, as the table below shows. Offer
two or three concrete options, not the whole tool list.
| User's state | Suggested next step |
| --- | --- |
| No servers, wants to try Manufact quickly | Deploy a starter with `create_server_from_template`. It needs no GitHub connection and starts the first deployment. |
| No servers, has an MCP server repo on GitHub | Deploy it. Follow the `mcp-plugin-deploy` skill. |
| Server with a failed latest deployment | Debug it with the recovery steps in the `mcp-plugin-deploy` skill. |
| Healthy server, wants to know whether it works in real clients | Follow the `mcp-plugin-testing` skill to create and run evals. |
| Healthy server, not yet published | Check publishing readiness. Follow the `mcp-plugin-publishing` skill. |
| Healthy server, wants others to try it | Enable a public chat page (`configure_public_chat`). |
| Server, waPreview of the first 4,000 characters. Read full instructions →
manufact-onboarding6.18 KB
View saved version →
---
name: manufact-onboarding
description: Use right after a user connects the Manufact Cloud MCP server, or when they ask what Manufact is, what they can do with it, or where to start. Explains the platform model (users, organizations, servers, deployments), inspects the user's account, and routes them to deploying, managing, or publishing an MCP server.
metadata:
owner: manufact
---
# Onboard a new Manufact user
Manufact Cloud hosts MCP servers and MCP apps. It is built by the team behind the
open-source mcp-use SDKs, but it hosts servers built with any framework. Users
deploy from GitHub, get a live MCP URL, and then test, observe, and publish the
server to clients such as ChatGPT, Claude, and Cursor.
The user is already signed in: connecting this MCP server completed the OAuth
flow. Never ask them for passwords, API keys, or tokens.
## How Manufact Cloud is organized
```text
User
└── Organizations (a user belongs to one or more)
└── Servers (each server is one MCP server)
└── Deployments (each server has many; one serves production)
```
- **User**: the signed-in account. It can belong to several organizations.
- **Organization**: owns servers, GitHub connections, members, billing, and
credits. New accounts get a Primary organization automatically.
- **Server**: one MCP server. It has a GitHub repository, build settings,
environment variables, domains, analytics, checklist runs, eval suites,
submission packs, and an optional public chat page.
- **Deployment**: one build of the server from a branch and commit.
- The production branch (usually `main`) deploys on every push and serves
`https://<slug>.run.mcp-use.com/mcp`.
- Other branches get preview URLs.
- A newer deployment can fail while an older one keeps serving traffic.
Tools follow the same hierarchy. Organization IDs scope servers, server IDs
scope deployments, and so on down. Use only IDs that tools return or that the
user gives you. Never guess an ID.
## First conversation
1. **Introduce Manufact briefly.** One or two sentences is enough, unless the user
asks for more.
2. **Look at the account before suggesting anything.**
- Call `list_organizations`.
- One organization: use it without asking.
- Several organizations: ask which one to work in, unless the conversation
already makes it clear.
- Call `list_servers` for that organization.
3. **Summarize what you found**, in plain language: which organization, how many
servers, and each server's name and latest deployment status. Call
`get_server` only for a server the user wants to look at more closely.
4. **Offer the next steps that fit their state**, as the table below shows. Offer
two or three concrete options, not the whole tool list.
| User's state | Suggested next step |
| --- | --- |
| No servers, has an MCP server repo on GitHub | Deploy it. Follow the `manufact-deploy` skill. |
| No servers, no code yet | Scaffold one with `npx create-mcp-use-app@latest`, push it to GitHub, then deploy. |
| Server with a failed latest deployment | Debug it with the recovery steps in the `manufact-deploy` skill. |
| Healthy server, not yet published | Check publishing readiness. Follow the `manufact-publish-ready` skill. |
| Healthy server, wants others to try it | Enable a public chat page (`configure_public_chat`, Hobby plan or higher). |
| Server, wants to understand usage or errors | Follow the `manufact-analytics` skill to inspect scoped metrics and evidence. |
| Server hosted elsewhere | Connect it in the dashboard (see below), then run the checklist. |
## What each area covers
| Area | Main tools | Use for |
| --- | --- | --- |
| Organizations and team | `list_organizations`, `create_organization`, `list_team_members`, `send_team_invite` | Choose a workspace and invite teammates. |
| GitHub | `list_github_installations`, `get_github_connect_url`, `connect_github_repo_to_server` | Give Manufact access to repositories. |
| Servers Preview of the first 4,000 characters. Read full instructions →
manufact-publish-ready6.87 KB
View saved version →
---
name: manufact-publish-ready
description: Use when a user wants to know whether their MCP server or MCP app is ready to publish to ChatGPT, Claude, or another app directory, wants to run or fix the Manufact publishing checklist, test the server with evals, or prepare submission assets such as screenshots, a demo video, and submission metadata.
metadata:
owner: manufact
---
# Get an MCP server ready to publish
App directories such as the ChatGPT app directory and Claude's connector
directory reject servers for protocol errors, missing tool metadata, broken
widgets, and policy problems. Manufact finds these before a reviewer does.
"Ready" means:
- the server is deployed and healthy,
- the publishing checklist has no blocking failures for the target platform,
- the server behaves correctly in real clients, and
- the submission assets exist.
Work through the steps below in order. Checklist runs, eval runs, and
submission packs consume credits, so fix everything you can find for free
before spending them.
## 1. Pick the server and target
1. Find the server with `list_organizations`, `list_servers`, and `get_server`.
The server must have a live deployment. If it doesn't, use the
`manufact-deploy` skill first.
2. Ask which platforms the user is targeting: ChatGPT, Claude, or both. The
answer decides which failures matter.
3. Check whether a run already exists with `list_checklist_runs`. If a recent
run matches the current deployment, read it with `get_checklist_run` instead
of starting a new one.
## 2. Review the code first (free)
If you can read the server's source, check it against the common blocking
failures before running anything. See
[references/checklist-checks.md](references/checklist-checks.md) for the full
list.
- **Every tool has a top-level `title` and a description.**
- **Every tool sets boolean `readOnlyHint`, `destructiveHint`, and
`openWorldHint`.** The values must match what the tool actually does.
- **Tool names are unique, at most 64 characters, and use only letters, digits,
`_`, `-`, or `.`.**
- **Input schemas are objects.** Declared output schemas are valid.
- **For MCP apps with widgets:**
- Each widget resource uses a `ui://` URI with the MIME type
`text/html;profile=mcp-app`.
- It declares one exact HTTPS domain.
- Its CSP lists every external origin it loads or connects to.
- **Listings don't overpromise.** Descriptions and behavior avoid policy
problems: misleading claims, collecting sensitive data without need, and
unexpected destructive actions.
Fix what you find, commit, push, and let the deployment finish before
continuing.
## 3. Authenticate protected servers
If the server requires OAuth, the checklist needs credentials to connect.
1. Call `authenticate_deployment` with `serverId`. Pass `branch` only for a
preview. Give the user the returned `authenticationUrl` as a clickable link.
2. After the user says they're done, call it again with the returned
`sessionId`.
3. Only `authenticated` means success. `pending` means the user hasn't
finished. `expired` or `failed` means you need a new link.
4. Never ask the user for OAuth credentials, tokens, or passwords.
## 4. Run the publishing checklist
1. **Confirm with the user before starting.** Each call starts a new billed run.
On the Hobby plan and above, it also runs the paid end-to-end checks in real
ChatGPT and Claude sessions.
2. Call `run_publishing_checklist` with `serverId`. Omit `branch` to check
production.
3. **Read the run with `get_checklist_run`** and the returned `runId`. Runs take
several minutes. `completed` and `failed` are final; `pending` and `running`
mean the run is still in progress.
- If the start response was lost or uncertain, look at
`list_checklist_runs` before starting another run.
4. **Read the final result:**
- `isReadyForChatgpt` and `isReadyForClaudeai`: a platform is ready when no
check with severity `error` fails for it. Warnings anPreview of the first 4,000 characters. Read full instructions →
Referenced files: 2 — file references are metadata, not package contents.
mcp-plugin-analytics4.83 KB
View saved version →
---
name: mcp-plugin-analytics
description: Analyze MCP server and plugin usage, errors, latency, and client traffic on Manufact Cloud. Use to investigate a deployed server's usage, errors, latency, clients, actors, sessions, or issues. Choose focused evidence tools or catalog-driven metrics, with scoped read-only SQL for questions the typed metrics cannot express. Use for existing server telemetry, not deployment changes or payload-capture configuration.
metadata:
owner: manufact
---
# Answer questions from server analytics
Reuse a server ID already established in the conversation. Otherwise find the
server with `list_servers`, omitting `organizationId` to search every
accessible organization, and confirm it with `get_server`. Use returned IDs. Keep the user's server, time
window, environment, and client or tool selections throughout the analysis.
## Choose the smallest useful read
- Use `get_server_analytics_summary` for a quick overview and comparison with
the preceding period. It returns current-period buckets and totals.
- Use `get_server_issues_trend` for aggregate issue occurrence trends across
all environments, independently of a paginated issue list.
- Use `list_server_issues` and `get_server_issue` for grouped errors, or
`list_server_analytics_sessions` and `get_server_analytics_session` for
individual interactions and their events. Read a list first; fetch details
only for the records relevant to the user's question. Actor-related groupings
are available through the catalog-driven metrics and SQL queries.
- Use `get_server_analytics_catalog` before composing a typed metrics or SQL
query. Discover dataset columns, metrics, dimensions, permitted filters,
units, time scopes, and query limits from the live catalog. Do not invent
fields or assume all metrics work on all datasets.
- Use `query_server_analytics_metrics` for grouped counts, trends, error
rates, and latency or token statistics that the catalog can express.
- Use `query_server_analytics_sql` when a useful read needs an expression or
grouping that the typed metrics cannot express. Use only catalog-listed
datasets and columns, with the same scoped time window and filters. The
backend constrains the query to the selected server and permits read-only
queries only; there is no unrestricted database connection.
- Use `open_analytics_view` when the user wants an interactive dashboard.
Opening the view does not provide analytical evidence in chat. Read the
relevant tools to substantiate an answer. The view's app-only
`get_server_analytics_dashboard_data` fetches a bounded dashboard batch
internally; section failures can be independent.
Read [references/queries.md](references/queries.md) for query examples,
pagination, scope distinctions, and recovery guidance.
## Explain what the evidence means
Use the response's units and counting definition. Interactions, transport
sessions, requests, and tool calls are different populations. The summary's
`sessions` field counts distinct interactions. Do not compare it directly with
a transport-session count. Summary errorRate is a percentage; use the catalog
for the units of typed metrics rather than assuming the same scale.
Report verified and approximate actor counts separately. Verified actors may
identify a conversation or token as well as an account; they do not necessarily
count people. Approximate actors are IP-derived. Distinct actor and session
counts use sketches by default; accuracy is separate from identity trust.
Output token counts are estimates. Latency percentiles can
be approximate, and a null metric is unavailable rather than zero. Compute
an average only from its stated
numerator and denominator, excluding events without the required measurement.
Carry response metadata into conclusions: requested versus effective window,
retention limits, approximation, partial results, warnings, and truncation.
An empty result means no matching retained data was returned; it does
Preview of the first 4,000 characters. Read full instructions →
Referenced files: 1 — file references are metadata, not package contents.
mcp-plugin-deploy7.92 KB
View saved version →
---
name: mcp-plugin-deploy
description: Deploy MCP servers and plugins on Manufact Cloud; configure builds, domains, and environments. Use when a user wants to deploy an MCP server or MCP app from GitHub or a starter template, redeploy or promote a deployment, set up preview branches, environment variables, or domains, or find out why a deployment failed. The live MCP URL works with the most popular AI Agents.
metadata:
owner: manufact
---
# Deploy an MCP server or plugin on Manufact Cloud
A server on Manufact Cloud is built from a GitHub repository. Every deployment
clones one branch, detects the runtime, builds it, and starts it in a
container. Before switching traffic, Manufact health-checks the new deployment
with an MCP `initialize` request. Production keeps serving the previous
deployment until the new one passes.
## 1. Check that the repository is deployable
If you can see the user's code, check these before the first deploy. Most
failed first deploys trace back to one of them. See
[references/build-runtimes.md](references/build-runtimes.md) for detection rules.
- **The MCP endpoint is served at `/mcp`.** The health check sends its request to
`<base>/mcp`.
- **The server listens on `0.0.0.0` and the `PORT` environment variable.**
Manufact injects both. Presets default to port 3000 for Node and 8000 for
Python. Servers that bind to `localhost` or a hard-coded port fail the health
check.
- **The runtime can be detected.**
- Node needs `package.json` plus a lockfile.
- Python needs `pyproject.toml` or `requirements.txt`.
- Anything else, Deno included, needs a committed `Dockerfile`.
- **Build and start scripts exist.** Node servers need a `start` script; a `build`
script runs only if it exists. Otherwise, set `buildCommand` and `startCommand`.
- **Monorepos set `rootDir`** to the server's subdirectory.
- **Secrets are not committed.** They become environment variables (step 4).
- **The repository is on GitHub and pushed.** Manufact deploys the remote branch,
not local changes.
mcp-use projects scaffolded with `npx create-mcp-use-app@latest` meet all of
these by default.
## 2. Find or create the server
Reuse server and organization IDs already established in the conversation;
skip the lookups below when you have them.
1. **Look for an existing server** with `list_servers`. Omit `organizationId`
to search every accessible organization at once. Confirm a candidate by its
repository (`get_server`), not its name. Reuse a matching server unless the
user explicitly wants a second one, for example another root in a monorepo.
Ask only when more than one server still matches.
2. **Pick the organization for a new server.** Use the organization of a
server you just found or one the user named. Otherwise call
`list_organizations`: with one organization, use it without asking; with
several, ask which one unless the conversation already makes it clear.
3. **No repository to deploy?** Use `create_server_from_template` and skip
steps 4 and 5. It needs only `organizationId` and no GitHub connection,
deploys the `mcp-apps` starter unless the user picks `mcp-server` or
`blank`, keeps the source in a private Manufact-managed repository, and
starts the first deployment.
4. **Check GitHub access** with `list_github_installations`. You need an
installation where `isActive` is true and whose account owns the repository.
- If there is none, or the repository isn't covered, call `connect_github`.
Clients that support it open GitHub setup directly and get the new
connection back. Otherwise give the user the returned link, and when they
confirm, check `list_github_installations` once. Don't poll. Call
`connect_github` again for a fresh link if the old one expired.
- If the user hasn't named a repository, offer choices from
`list_github_repositories`.
5. **Create the server** with `create_server`.
- Pass `organizationId`, `githubConnectionId`, and `repoPreview of the first 4,000 characters. Read full instructions →
Referenced files: 2 — file references are metadata, not package contents.
mcp-plugin-publishing10.1 KB
View saved version →
---
name: mcp-plugin-publishing
description: Check publishing readiness for MCP servers and plugins; prepare supported directory submissions. Use when a user wants to know whether an MCP server or MCP app is ready for supported AI agent directories, run or fix the Manufact publishing checklist, or package an Agent Plugin with listing metadata, review cases, and a demo recording. For standalone behavior testing, use mcp-plugin-testing.
metadata:
owner: manufact
---
# Get an MCP server or plugin ready to publish
App directories such as the ChatGPT app directory and Claude's connector
directory reject servers for protocol errors, missing tool metadata, broken
widgets, and policy problems. Manufact finds these before a reviewer does.
"Ready" means:
- the server is deployed and healthy,
- the publishing checklist has no blocking failures for the target platform,
- the server behaves correctly in real clients, and
- the plugin package and required listing and review materials are complete.
For server readiness, review the code and work through the checks below in order.
The live checklist result is authoritative; the reference files explain checks
but can lag behind it. Don't quote plan names, prices, or directory rules from
memory.
For a packaging or ZIP-only request, start at step 8, reuse available readiness
evidence, and
report unverified behavior. Package preparation can proceed without a checklist
run or submission-portal access.
## 1. Pick the server and target
1. Reuse a server ID already established in the conversation. Otherwise find
the server with `list_servers`, omitting `organizationId` to search every
accessible organization, and confirm it with `get_server`. The server must
have a live deployment. If it doesn't, use the `mcp-plugin-deploy` skill
first.
2. Resolve which platforms the user is targeting. The checklist reports
readiness for the platforms it returns, currently ChatGPT and Claude; the
answer decides which failures matter. Other clients, such as Cursor, Grok,
or Muse, can connect to the server's MCP URL, but this checklist does not
assess a directory for them.
3. Check whether a run already exists with `list_checklist_runs`. If a recent
run matches the current deployment, read it with `get_checklist_run` instead
of starting a new one.
## 2. Review the code first
If you can read the server's source, check it against the common blocking
failures before running anything. See
[references/checklist-checks.md](references/checklist-checks.md) for the full
list.
- **Every tool has a top-level `title` and a description.**
- **Every tool sets boolean `readOnlyHint`, `destructiveHint`, and
`openWorldHint`.** The values must match what the tool actually does.
- **Tool names are unique, at most 64 characters, and use only letters, digits,
`_`, `-`, or `.`.**
- **Input schemas are objects.** Declared output schemas are valid.
- **For MCP apps with widgets:**
- Each widget resource uses a `ui://` URI with the MIME type
`text/html;profile=mcp-app`.
- It declares one exact HTTPS domain.
- Its CSP lists every external origin it loads or connects to.
- **Listings don't overpromise.** Descriptions and behavior avoid policy
problems: misleading claims, collecting sensitive data without need, and
unexpected destructive actions.
Fix what you find, commit, push, and let the deployment finish before
continuing.
## 3. Authenticate protected servers
If the server requires OAuth, the checklist needs credentials to connect.
1. Call `authenticate_deployment` with `serverId`. Pass `branch` only for a
preview. Clients that support it open the consent page directly and get
the session's status back. Otherwise give the user the returned
`authenticationUrl` as a clickable link.
2. If the result is still `pending`, wait until the user says they're done,
then call it once with the returned `sessionId`. Don't poll.
3. Only `authenticated` means success. `pending` means Preview of the first 4,000 characters. Read full instructions →
Referenced files: 2 — file references are metadata, not package contents.
mcp-plugin-testing5.77 KB
View saved version →
---
name: mcp-plugin-testing
description: Test MCP servers and plugins across supported clients; create evals and investigate failures. Use when a user wants to check whether models pick the right tools and give useful answers in real clients, compare models or clients, rerun tests after a change, or understand why a test failed. Discover supported clients, models, and agent runtimes with get_eval_options.
metadata:
owner: manufact
---
# Test an MCP server or plugin in real clients
Evals run a saved conversation against the server's production endpoint in one
or more clients, then a judge scores the result against passing criteria. They
show whether models actually discover and call the right tools, with the right
arguments, and give useful answers. They complement the publishing checklist,
which inspects protocol and metadata only (see the `mcp-plugin-publishing`
skill).
## 1. Pick the server
- Reuse organization and server IDs already established in the conversation.
- Otherwise call `list_servers` without `organizationId` to search every
accessible organization, and confirm the server with `get_server`. Eval tools
need both the server's `organizationId` and `serverId`.
- The server needs a healthy production deployment, because evals always target
the production endpoint. If it has none, use the `mcp-plugin-deploy` skill
first. To test a branch, promote or merge it first; preview URLs are not
eval targets.
- If the server requires OAuth, complete `authenticate_deployment` first as
described in `mcp-plugin-publishing`, so clients can connect.
## 2. Discover clients and models
Call `get_eval_options` before choosing clients. Use only what it returns:
- **Inspector**: the Manufact chat client. Choose any number of OpenRouter
models with `list_eval_models` (filter with `query` or `provider`). Each model
runs as its own execution. Omitting clients uses Inspector with the default
model.
- **Web clients** (`webClients`): ChatGPT and Claude on the web. Each runs once
per suite with no model choice and does not enforce `disallowedTools`.
- **Runtimes**: agent harnesses such as Codex, Claude Code, or Cursor. Pick
only runtimes with `available: true`, choose a model from that runtime's
`models`, and tell the user about any `limitations` or the `reason` it is
unavailable.
Other MCP clients can still connect to the server's MCP URL, but they can't be
tested here unless `get_eval_options` lists them. Say so instead of implying
coverage.
## 3. Create or update suites
Check `list_eval_suites` first and reuse a matching suite with
`update_eval_suite` rather than creating a duplicate. Read one with
`get_eval_suite` when you need its full configuration.
Create a suite with `create_eval_suite` for each behavior worth checking:
- **`flow`**: the conversation in order. Use realistic user wording, not the
tool's name. Add a `tool_call` step after a user step when the model must
call a specific tool; match only the arguments that matter (`"any"`,
`contains`, or `pattern`).
- **`rubric`**: concrete, observable passing criteria the judge can check.
- **`threshold`**: the minimum judge score to pass; the default is 0.7.
- **`clients`**: the clients and models chosen in step 2.
- **`disallowedTools`**: tools the model must not call, such as destructive
tools in a read-only scenario.
Cover the main jobs users bring to the server, plus at least one case where the
model should decline or ask for clarification. If you can read the server's
source, derive the cases from its actual tools.
## 4. Run the suites
1. If the user asked to run tests, call `run_eval_suites` with up to ten
`suiteIds` without asking again. If you are starting runs on your own
initiative, say which suites and clients will run first: each execution
uses plan resources.
2. Each suite starts its own background run. Read each receipt's `outcome`; a
suite that was `not_started` or has an `error` did not run.
3. If the start res
Preview of the first 4,000 characters. Read full instructions →