Update to Microsoft Dataverse

{
  "description": "One-step setup for a Dataverse environment — installs tools, authenticates, registers the MCP server, and writes `.env`. Use when starting a new project, switching environments, fixing authentication, or troubleshooting an MCP connection that won't come up.",
  "included_files": [
    {
      "relative_path": "references/erp-detection.md",
      "size_in_bytes": 1480
    },
    {
      "relative_path": "references/headless-hosts.md",
      "size_in_bytes": 17525
    },
    {
      "relative_path": "references/mcp-configuration.md",
      "size_in_bytes": 29925
    },
    {
      "relative_path": "references/tools-setup.md",
      "size_in_bytes": 12225
    }
  ],
  "name": "dv-connect",
  "skill_md_contents": "---\r\nname: dv-connect\r\ndescription: One-step setup for a Dataverse environment — installs tools, authenticates, registers the MCP server, and writes `.env`. Use when starting a new project, switching environments, fixing authentication, or troubleshooting an MCP connection that won't come up.\r\n---\r\n\r\n# Skill: Connect\r\n\r\nOne-step, idempotent Dataverse connection. Each step checks if it's already done and skips.\r\n\r\n> **Environment-First Rule** — All metadata and plugin registrations are created **in the environment** via API/scripts, then pulled into the repo. Never hand-write solution XML to create components.\r\n\r\n**Execute steps in order; do not skip ahead.** **Exception:** Step 0 can short-circuit the flow if the workspace is already set up.\r\n\r\n> **Host entry test (FIRST — before Step 0).** A **local Windows/macOS host is capable by default, whichever agent drives it** — it can run the CLIs, use persistent credentials, and host local MCP servers; an approval/sandbox gate isn't a constraint, and a **missing CLI = install it**. **Constrained only when** a runtime **can't start**, auth **can't persist**, or the host is explicitly **ChatGPT Work Mode / Codex cloud / CI / no-keyring Linux** (deterministic table: [headless-hosts.md](references/headless-hosts.md)). **Constrained →** install **only** Python + pip deps, `.env`, `scripts/auth.py`, verify `python scripts/auth.py --check`, **skip** CLI / PAC / MCP. **Capable →** normal flow below.\r\n\r\n---\r\n\r\n## Step 0: Detect existing setup (run this first)\r\n\r\nBefore touching anything, check whether this workspace is already connected to a Dataverse environment. Repeating setup on an already-configured workspace overwrites `.env`, re-registers MCP, and wastes time.\r\n\r\nRun these checks in order. If **all four pass**, skip straight to Step 7 (final verification) and stop there.\r\n\r\n1. **`.env` is present and complete** — file exists at the workspace root and contains non-empty values for `DATAVERSE_URL`, `TENANT_ID`, and `MCP_CLIENT_ID`\r\n2. **MCP is registered** — `.mcp.json` (Claude Code) or the equivalent Copilot / Cursor config file has a `dataverse-*` server entry pointing at the `DATAVERSE_URL` from `.env`\r\n3. **Both auth surfaces match `.env`** — `dataverse auth who` shows a profile whose `Environment Url` matches `DATAVERSE_URL`, AND `pac org who` against a PAC profile for the same URL succeeds. (DV CLI auth covers Connect / Data / Query / Metadata / MCP / Python; PAC auth covers `dv-solution` and `dv-admin`. Both are front-loaded at connect time so neither prompts later.)\r\n4. **Python SDK is importable and current** — `python -c \"from PowerPlatform.Dataverse.client import DataverseClient; import pandas; from importlib.metadata import version; v=version('PowerPlatform-Dataverse-Client'); assert int(v.split('.')[0])>=1, f'SDK {v} is outdated, need >=1.0.0'\"` exits 0\r\n\r\n**If all pass:** Refresh `DATAVERSE_PLUGIN_VERSION` in `.env` if it's stale, confirm the detected setup (URL, profile, MCP server), and jump to Step 7. Do not otherwise rewrite `.env`, re-register MCP, or re-run `pip install`.\r\n\r\n**If any check fails:** Proceed through the normal flow (Steps 1–7), but still use each step's own skip condition. A partially-configured workspace doesn't need a full redo — e.g., if only `.env` and MCP are missing but tools and auth are fine, start at Step 2 or Step 3.\r\n\r\n---\r\n\r\n## Step 1: Ensure tools are installed\r\n\r\nCheck each tool independently -- report all missing tools at once. See [tools-setup.md](references/tools-setup.md) for install commands.\r\n\r\n| Tool | Check |\r\n|---|---|\r\n| Python 3 | `python --version` |\r\n| Git | `git --version` |\r\n| Node.js | `node --version` |\r\n| PAC CLI | `pac` (prints version banner; `pac --version` is not valid) (see [tools-setup.md](references/tools-setup.md) if not in PATH) |\r\n| Dataverse CLI | `npm list -g @microsoft/dataverse` |\r\n| .NET SDK | `dotnet --version` |\r\n| Azure CLI | `az --version` |\r\n\r\n.NET SDK is needed for PAC CLI but NOT for the Dataverse CLI (the npm package bundles its own runtime). Node.js powers the Dataverse CLI npm package (`@microsoft/dataverse`), which is used as the MCP proxy and for scripted data plane actions. Azure CLI is used as a fallback for environment discovery when PAC CLI isn't available (see [mcp-configuration.md](references/mcp-configuration.md) Step 3b). GitHub CLI is not needed for connecting — it's used later for ALM/CI/CD scenarios (see `dv-solution`).\r\n\r\nIf any tool is missing, install it (see [tools-setup.md](references/tools-setup.md)), then verify. If `winget` installs a tool but it's not in PATH, ask the user to restart the terminal.\r\n\r\nAfter Python is confirmed, check if deps are already present before installing:\r\n```\r\npython -c \"from PowerPlatform.Dataverse.client import DataverseClient; import azure.identity, msal, msal_extensions, requests, pandas; print('OK')\"\r\n```\r\nIf it prints `OK`, skip pip. Otherwise:\r\n```\r\npip install --upgrade azure-identity requests PowerPlatform-Dataverse-Client pandas msal msal-extensions\r\n```\r\n\r\n`msal` + `msal-extensions` let `scripts/auth.py` reuse the `dataverse auth create` cache -- one sign-in for CLI, MCP, Python.\r\n\r\nAfter Node.js is confirmed, install the Dataverse CLI **only if missing** (do not re-run on every connect -- on managed devices each `@latest` fetch can trigger npm-registry security prompts; see [tools-setup.md](references/tools-setup.md)):\r\n```\r\nnpm install -g @microsoft/dataverse@latest\r\n```\r\n\r\n**Skip condition:** All tools present, Python SDK installed, and `pandas` importable (`python -c \"import pandas\"`).\r\n\r\n---\r\n\r\n## Step 2: Discover and select the environment\r\n\r\nBefore asking the user for a URL, check what's already available.\r\n\r\n> **Auth tool choice.** Two tools, two AAD apps, two caches — front-load both at connect:\r\n>\r\n> 1. **`dataverse auth create`** (app `0c412cc3-…`) covers DV CLI + MCP + Python.\r\n> 2. **`pac auth create`** (PAC's own app) covers `dv-solution` + `dv-admin`.\r\n\r\nCheck for an existing DV CLI profile first, then fall back to PAC for environment discovery if needed:\r\n\r\n```\r\ndataverse auth list\r\ndataverse auth who\r\npac auth list   # PAC profiles are still useful for env discovery / pac org list\r\n```\r\n\r\n**If `dataverse auth who` shows a profile and its environment matches the user's target:**\r\n- Reuse it. Set `DATAVERSE_URL` and `TENANT_ID` from the profile.\r\n\r\n**If no DV CLI profile exists (or it points at the wrong environment):**\r\n- Ask: \"Do you want to connect to an existing environment or create a new one?\"\r\n\r\n**Before selecting, check for tenant/region mismatch.** If the target URL uses a different region than the authenticated account's environments, create a new profile for the correct tenant rather than reuse the old one:\r\n\r\n```\r\ndataverse auth create --environment <url>          # interactive (WAM broker on Windows → no browser tab)\r\ndataverse auth create --environment <url> --deviceCode   # headless / remote / SSH\r\n```\r\n\r\nIf the user hits an admin-consent error, the CLI prints the correct scope-scoped consent URL to share with a tenant admin — do not synthesize one.\r\n\r\n**To switch between existing DV CLI profiles:**\r\n```\r\ndataverse auth select --name <profile-name>\r\n```\r\n\r\n**To create a new environment** (requires admin permissions):\r\n```\r\npac admin create --name \"<name>\" --type \"<type>\" --region \"<region>\"\r\n```\r\nIf this fails with permissions error, guide the user to [Power Platform Admin Center](https://admin.powerplatform.microsoft.com/) to create it, then connect.\r\n\r\n**Confirm connection:**\r\n```\r\ndataverse auth who\r\ndataverse org who      # or: pac org who\r\n```\r\nParse the output to extract `DATAVERSE_URL`, `TENANT_ID`, and — on ERP-linked envs — `ERP_URL` (see [`erp-detection.md`](references/erp-detection.md)).\r\n\r\nIf neither command shows a tenant ID, fall back to:\r\n```bash\r\ncurl -sI https://<org>.crm.dynamics.com/api/data/v9.2/ \\\r\n  | grep -i \"WWW-Authenticate\" \\\r\n  | sed -n 's|.*login\\.microsoftonline\\.com/\\([^/]*\\).*|\\1|p'\r\n```\r\n\r\n### Step 2b: Front-load PAC CLI auth for the same environment\r\n\r\nPAC uses its own AAD app, so a separate sign-in is required for `dv-solution` and `dv-admin` — do it now.\r\n\r\n```\r\npac auth list                                       # skip if a profile for $DATAVERSE_URL exists\r\npac auth create --name <orgid> --environment <DATAVERSE_URL>\r\n```\r\n\r\nUse the same account as Step 2. If PAC CLI is not installed, skip with a note that `dv-solution` / `dv-admin` will need it later.\r\n\r\n\r\n---\r\n\r\n## Step 3: Create .env\r\n\r\nPresent authentication options:\r\n\r\n> How would you like to authenticate with Dataverse?\r\n> 1. **Interactive login (recommended)** — Sign in via browser. No app registration needed. Token stays cached across sessions.\r\n> 2. **Service principal (for CI/CD)** — Uses CLIENT_ID and CLIENT_SECRET from an Azure app registration.\r\n\r\nWrite `.env` directly — do not instruct the user to create it:\r\n\r\nDetect the current tool (Claude or Copilot) from context and set `MCP_CLIENT_ID` automatically:\r\n- Claude (CLI or VSCode extension): `0c412cc3-0dd6-449b-987f-05b053db9457`\r\n- GitHub Copilot: `aebc6443-996d-45c2-90f0-388ff96faa56`\r\n\r\nAlso set plugin attribution variables for User-Agent tagging. **Fill in the two literals below from your own context** — you (the agent) loaded this plugin, so you already know both values:\r\n\r\n- `PLUGIN_VERSION` — the `version` field of your loaded plugin manifest (e.g. `\"1.5.0\"`). At runtime, `auth.py` re-reads this from the live manifest via host env vars; this `.env` entry is a fallback for offline cases.\r\n- `AGENT` — your host identity, one of: `claude-code`, `copilot`, `cursor`, `codex`, or `unknown`. Must match an entry in `_ALLOWED_AGENTS` in `auth.py` — if you don't recognize your host, use `unknown`.\r\n\r\n```python\r\n# Substitute these two literals from your loaded plugin context.\r\n# Do NOT leave the angle-bracket placeholders — replace with real values.\r\nplugin_version = \"<plugin manifest version, e.g. 1.5.0>\"\r\nagent_host = \"<your host name: claude-code | copilot | cursor | codex | unknown>\"\r\n\r\nwith open(\".env\", \"w\") as f:\r\n    f.write(f\"DATAVERSE_URL={dataverse_url}\\n\")\r\n    f.write(f\"TENANT_ID={tenant_id}\\n\")\r\n    f.write(f\"MCP_CLIENT_ID={mcp_client_id}\\n\")\r\n    f.write(f\"DATAVERSE_PLUGIN_VERSION={plugin_version}\\n\")\r\n    f.write(f\"DATAVERSE_PLUGIN_AGENT={agent_host}\\n\")\r\n    f.write(f\"SOLUTION_NAME={solution_name}\\n\")\r\n    f.write(f\"PUBLISHER_PREFIX=\\n\")  # filled in when solution is created\r\n    f.write(f\"PAC_AUTH_PROFILE=nonprod\\n\")\r\n    if client_id:\r\n        f.write(f\"CLIENT_ID={client_id}\\n\")\r\n    if client_secret:\r\n        f.write(f\"CLIENT_SECRET={client_secret}\\n\")\r\n```\r\n\r\nEnsure `.env` is in `.gitignore`:\r\n\r\n```python\r\nimport os\r\n\r\nGITIGNORE_ENTRIES = [\r\n    \".env\", \".vscode/settings.json\", \".claude/mcp_settings.json\",\r\n    \".token_cache.bin\", \".dataverse/\", \"*.snk\", \"__pycache__/\", \"*.pyc\",\r\n    \"solutions/*.zip\", \"plugins/**/bin/\", \"plugins/**/obj/\",\r\n]\r\ngitignore = open(\".gitignore\").read() if os.path.exists(\".gitignore\") else \"\"\r\nmissing = [e for e in GITIGNORE_ENTRIES if e not in gitignore]\r\nif missing:\r\n    with open(\".gitignore\", \"a\") as f:\r\n        f.write(\"\\n\" + \"\\n\".join(missing) + \"\\n\")\r\n```\r\n\r\n**Skip condition:** `.env` already exists with all required values.\r\n\r\n---\r\n\r\n## Step 4: Set up project structure (new projects only)\r\n\r\nIf this is a new project (no `scripts/` directory):\r\n\r\n```\r\nmkdir -p solutions plugins scripts\r\n```\r\n\r\nCopy plugin scripts:\r\n```\r\ncp .github/plugins/dataverse/scripts/auth.py scripts/\r\n```\r\n\r\nCopy `templates/CLAUDE.md` to the repo root if it doesn't exist. Replace placeholders (`{{DATAVERSE_URL}}`, `{{SOLUTION_NAME}}`, `{{PUBLISHER_PREFIX}}`) with values from `.env`.\r\n\r\n**Skip condition:** `scripts/auth.py` exists.\r\n\r\n---\r\n\r\n## Step 5: Verify the connection\r\n\r\n```\r\ndataverse auth who\r\npac org who\r\npython scripts/auth.py --check\r\n```\r\n\r\n`--check` makes a **real data-plane call** (not just a token) — the only proof the org is actually reachable; a token can be minted while the org domain is blocked. All must resolve the same user/environment, proving the DV CLI cache, the PAC profile (Step 2b), and Python's reuse of the shared cache are wired.\r\n\r\n**If any fail:**\r\n- `dataverse auth who` fails → re-run Step 2.\r\n- `pac org who` fails → re-run Step 2b.\r\n- `python scripts/auth.py --check` prints a device-code URL → browser/WAM cache has no Python-reusable token. **Auto-fix:** re-run `dataverse auth create --environment <url> --deviceCode`, then retry. If it *still* prompts, check `pip show msal msal-extensions`. **Headless hosts** (ChatGPT / Codex cloud / CI): `dataverse auth create` can't persist here — don't loop (see [headless-hosts.md](references/headless-hosts.md)).\r\n- `python scripts/auth.py --check` prints `NOT REACHABLE` with a connection/timeout error → the org domain is blocked by network egress, not auth. Do NOT report success or a count — see [headless-hosts.md](references/headless-hosts.md) remediation.\r\n- Other Python error → check SDK install and `.env`.\r\n\r\nBefore metadata work, also confirm the account has the `prvCreateEntity` customization privilege — see [tools-setup.md](references/tools-setup.md#privilege-preflight).\r\n\r\n---\r\n\r\n## Step 6: Configure MCP server\r\n\r\n**Skip this step** if MCP is already configured:\r\n- `.mcp.json` or `~/.copilot/mcp-config.json` or `~/.cursor/mcp.json` or `~/.codex/config.toml` contains a Dataverse server entry\r\n- `claude mcp list` shows a `dataverse-*` server registered\r\n\r\nIf MCP is not configured, follow [mcp-configuration.md](references/mcp-configuration.md):\r\n\r\n1. Detect which tool the user is running (Copilot, Claude, Cursor, or Codex) from context\r\n2. Set `MCP_CLIENT_ID` based on tool choice\r\n3. Get environment URL from `.env`\r\n4. Default to GA endpoint (`/api/mcp`)\r\n5. Register the MCP server per host (see the per-host blocks below)\r\n6. Handle admin consent and allowlist — prefer `dataverse mcp allow <MCP_CLIENT_ID>` over the portal (one-time per tenant/environment)\r\n\r\n**Plugin attribution for MCP:** This plugin uses the **stdio proxy** transport (`npx @microsoft/dataverse mcp <url>`). When registering it, include `DATAVERSE_OPERATION_CONTEXT` in the env block so the CLI appends it to its User-Agent on requests to `/api/mcp`. Build the value from `.env`:\r\n\r\n```\r\nDATAVERSE_OPERATION_CONTEXT=app=dataverse-skills/{DATAVERSE_PLUGIN_VERSION};skill=mcp-direct;agent={DATAVERSE_PLUGIN_AGENT}\r\n```\r\n\r\nFor Claude Code (`claude mcp add -t stdio`), pass it via `-e DATAVERSE_OPERATION_CONTEXT=...`. For Copilot/Cursor JSON configs, add it to the `\"env\"` object in the stdio server entry; for Codex, add it to its `[mcp_servers.<name>.env]` table.\r\n\r\n**Important:** MCP configuration requires an editor/CLI restart.\r\n\r\n**For Copilot:** Write the JSON config, then:\r\n> ✅ Dataverse MCP server configured. **Restart your editor** for changes to take effect.\r\n\r\n**For Claude:** Run the `claude mcp add` command, then warn the user about the auth popup that will appear on next launch:\r\n> ✅ Dataverse MCP server registered. Restart Claude Code to enable MCP tools.\r\n> Remember to **use `claude --continue` to resume the session** without losing context.\r\n>\r\n> On restart, a browser window may open to sign in to your Dataverse environment (the MCP proxy authenticating on your behalf). See [mcp-configuration.md](references/mcp-configuration.md) for details.\r\n\r\n**For Cursor:** Write the JSON config, then:\r\n> ✅ Dataverse MCP server `dataverse-{orgid}` configured in `~/.cursor/mcp.json`. **Reload the Cursor window** (Ctrl+Shift+P → \"Developer: Reload Window\") for the new MCP server to appear under Settings → Tools & MCPs.\r\n>\r\n> On first use the `npx @microsoft/dataverse` proxy signs in via browser device code, then reuses the shared cache silently. See [mcp-configuration.md](references/mcp-configuration.md).\r\n\r\n**For Codex:** Write the TOML config to `~/.codex/config.toml`. Codex loads MCP tools only at startup, so don't claim they're callable until the user restarts. Tell the user:\r\n> ✅ Dataverse MCP server `dataverse-{orgid}` configured in `~/.codex/config.toml`. **Restart Codex** (CLI) or reload the Codex IDE to load the MCP tools.\r\n\r\n---\r\n\r\n## Step 7: Final verification\r\n\r\nAfter the editor/CLI restarts, **both** of these must succeed before declaring the setup complete:\r\n\r\n**Check 1: `claude mcp list` (or Copilot equivalent) shows ✓ Connected**\r\n```\r\nclaude mcp list\r\n```\r\nThis proves the MCP server process starts and speaks the MCP protocol. It does NOT by itself prove that data operations work — authentication, environment allowlisting, and endpoint reachability are only exercised on the first real tool call.\r\n\r\n**Check 2: Agent successfully lists tables via `describe`/`search` and returns data**\r\n> \"List the tables in my Dataverse environment.\"\r\n\r\nThis proves end-to-end wiring: auth, tenant consent, environment allowlist, and endpoint reachability are all correct. If the agent falls back to PAC CLI or Web API, see [mcp-configuration.md](references/mcp-configuration.md) troubleshooting.\r\n\r\nOnly when **both** checks pass is the setup verified.\r\n\r\n**Interpreting failures:**\r\n\r\n- If Check 1 fails (server not ✓ Connected): the MCP server itself cannot start. Re-run Step 6 and check that `npx`/Node.js are installed and the MCP registration succeeded.\r\n- If Check 1 passes but Check 2 fails (server starts but `describe`/`search` errors): the server can speak MCP but cannot reach or read Dataverse. Run `--validate` below to diagnose.\r\n\r\n**Diagnostic — `--validate` (for failure investigation only):**\r\n```\r\nnpx @microsoft/dataverse mcp {DATAVERSE_URL} --validate\r\n```\r\nThis exercises two Dataverse MCP endpoints with a fresh authentication handshake and reports detailed errors (auth, allowlist, consent, endpoint reachability):\r\n\r\n- **GA / Production endpoint** — `{DATAVERSE_URL}/api/mcp`. This is the one the plugin actually uses at runtime.\r\n- **Preview endpoint** — `{DATAVERSE_URL}/api/mcp_preview`. Opt-in per environment; not used by the plugin.\r\n\r\n**Do not use `--validate` as a success gate on first-time setup.** On a freshly configured workspace, the token cache hasn't warmed up, so `--validate` can fail with `MsalClientException` or `403` while MCP is actually working fine on subsequent real calls. Reserve `--validate` for diagnosing a confirmed failure in Check 1 or Check 2.\r\n\r\n**How to read `--validate` output:**\r\n\r\n- **Look at the GA / Production endpoint (`/api/mcp`) result first.** If this passes, MCP will work for normal plugin usage regardless of what the Preview endpoint reports.\r\n- **A `403 Forbidden` on the Preview endpoint (`/api/mcp_preview`) is expected for most environments.** Preview is opt-in per environment; if your environment hasn't enabled it, the Preview check will always fail. This does not indicate a broken setup.\r\n- **Ignore the overall exit code and the `⚠ Partial success` warning in this case.** The validator returns exit code `1` (failure) unless BOTH `/api/mcp` and `/api/mcp_preview` pass. Because most environments don't enable the Preview endpoint, `--validate` will exit `1` even when MCP is fully functional via the GA endpoint. Focus on per-endpoint results, not the aggregate status.\r\n- **If the GA endpoint (`/api/mcp`) fails:** that's the real signal to investigate — auth, tenant consent, environment allowlist, or endpoint reachability.\r\n\r\n### MCP Server Capabilities\r\n\r\nFor what MCP can and can't do (data CRUD + batch up to 25, table/column creation incl. choice/lookup, `search`/`describe`, file upload/download) versus the SDK / Web API, see the **overview** skill's Tool Capabilities matrix.\r\n\r\nAfter verifying MCP works, tell the user:\r\n\r\n> ✅ Connected to Dataverse at `{DATAVERSE_URL}`. Tools installed, authenticated, MCP live.\r\n>\r\n> You can now:\r\n> - Create tables, columns, and relationships (`dv-metadata`)\r\n> - Write and import data (`dv-data`)\r\n> - Query and analyze data (`dv-query`)\r\n> - Export and promote solutions (`dv-solution`)\r\n>\r\n> To create your first solution, see the `dv-solution` skill.\r\n> To load sample data (accounts, contacts, opportunities), ask: \"Load demo data into my Dataverse environment.\"\r\n\r\n---\r\n\r\n## Supported Agents\r\n\r\nThis plugin's skill files are natively loaded by both **GitHub Copilot CLI** and **Claude Code CLI** when installed as a plugin. No manual context-loading is needed — both agents discover and invoke skills automatically.\r\n\r\nThe PAC CLI commands, Python scripts, and XML templates work identically in both environments.\r\n"
}