Update to Goldsky

{
  "description": "Diagnose and fix broken Goldsky Mirror pipelines. Use this skill whenever a user has a Mirror pipeline that is failing, stuck, terminated, won't start, is in a restart loop, or is blocked by an in-flight request. Also use when the user mentions a specific Mirror pipeline name alongside a problem — even if they don't say 'mirror' explicitly, if they're using `goldsky pipeline` commands (not `goldsky turbo`), this is the right skill. Runs CLI commands directly to check status, read errors, identify root cause, and apply fixes. For YAML syntax or config reference, use /mirror instead. For turbo pipeline problems, use /turbo-doctor instead.",
  "included_files": [
    {
      "relative_path": "evals/trigger-eval.json",
      "size_in_bytes": 3105
    }
  ],
  "name": "mirror-doctor",
  "skill_md_contents": "---\nname: mirror-doctor\ndescription: \"Diagnose and fix broken Goldsky Mirror pipelines. Use this skill whenever a user has a Mirror pipeline that is failing, stuck, terminated, won't start, is in a restart loop, or is blocked by an in-flight request. Also use when the user mentions a specific Mirror pipeline name alongside a problem — even if they don't say 'mirror' explicitly, if they're using `goldsky pipeline` commands (not `goldsky turbo`), this is the right skill. Runs CLI commands directly to check status, read errors, identify root cause, and apply fixes. For YAML syntax or config reference, use /mirror instead. For turbo pipeline problems, use /turbo-doctor instead.\"\n---\n\n# Mirror Pipeline Doctor\n\nDiagnose and fix existing Mirror pipeline problems by running CLI commands, identifying root causes, and executing fixes.\n\n## Boundaries\n\n- Diagnose and fix EXISTING Mirror pipeline problems.\n- Do not build new pipelines — use `/mirror` for config reference or `/turbo-builder` for new Turbo pipelines.\n- Do not serve as a command reference — use `/mirror` for CLI syntax and flag lookups.\n- Do not handle Turbo pipelines — use `/turbo-doctor` for `goldsky turbo` problems.\n- Do not create secrets — use `/secrets` for credential management. But DO check whether secrets exist as part of diagnosis.\n\n## Diagnostic Workflow\n\nFollow these steps in order. Each step builds on the previous one.\n\n### Step 1: Verify Authentication\n\nRun `goldsky project list 2>&1` to confirm the user is logged in.\n\n- **If logged in:** Note the project name and continue.\n- **If not logged in:** Direct the user to `/auth-setup`. Do not proceed until auth works.\n\n### Step 2: Identify the Pipeline\n\nRun `goldsky pipeline list --include-runtime-details 2>&1` to list all Mirror pipelines with their status.\n\nIf the user already named a pipeline, confirm it exists in the list. If not, show the list and ask which pipeline they want to diagnose.\n\nNote both the **desired status** (ACTIVE, INACTIVE, PAUSED) and the **runtime status** (STARTING, RUNNING, FAILING, TERMINATED) — Mirror pipelines have both, and the combination tells the story.\n\n### Step 3: Triage by Status\n\nThe desired + runtime status combination determines the diagnostic path:\n\n| Desired | Runtime | Meaning | Action |\n|---------|---------|---------|--------|\n| ACTIVE | RUNNING | Healthy — pipeline is processing data | Ask user what symptom they're seeing. Proceed to Step 4. |\n| ACTIVE | STARTING | Pipeline is initializing | Ask how long. If >10 min, proceed to Step 4. |\n| ACTIVE | FAILING | Pipeline is encountering errors but hasn't terminated yet | Proceed to Step 4 immediately — this is time-sensitive. |\n| ACTIVE | TERMINATED | **Most common failure.** Pipeline wanted to run but crashed. | Proceed to Step 4. |\n| PAUSED | TERMINATED | User paused the pipeline (snapshot was taken). | Ask if they want to resume: `goldsky pipeline start <name> --from-snapshot last` |\n| INACTIVE | TERMINATED | User stopped the pipeline (no snapshot). | Ask if they want to start: `goldsky pipeline start <name>` |\n\n**ACTIVE + TERMINATED is the most common case.** The pipeline's desired status is ACTIVE (it should be running) but the runtime has terminated due to an error. Focus the diagnosis here.\n\n### Step 4: Gather Diagnostic Data\n\nRun these commands to understand what went wrong:\n\n```bash\n# Get error details and runtime metrics\ngoldsky pipeline monitor <name> 2>&1\n\n# Check for in-flight requests blocking operations\ngoldsky pipeline monitor <name> --update-request 2>&1\n\n# Get the pipeline definition to check for misconfig\ngoldsky pipeline get <name> --definition 2>&1\n\n# Get pipeline info including version\ngoldsky pipeline info <name> 2>&1\n\n# Check available snapshots\ngoldsky pipeline snapshots list <name> 2>&1\n```\n\nRun these in sequence and analyze the output before proceeding. The monitor output is the most important — it shows error messages, records received/written metrics, and runtime status transitions.\n\n### Step 5: Match Error Patterns\n\nBased on the diagnostic data, match against these known patterns:\n\n#### Bad or Missing Secret\n\n**Symptoms:** Pipeline terminates shortly after starting. Monitor shows credential or authentication errors.\n\n**Verify:** Run `goldsky secret list 2>&1` and cross-reference with the `secret_name` values in the pipeline definition from Step 4.\n\n**Fix:**\n1. If the secret doesn't exist, direct the user to `/secrets` to create it.\n2. If the secret exists but credentials are wrong, update it in place with `goldsky secret update`; follow [Rotating Credentials](../secrets/SKILL.md#rotating-credentials) for the required JSON format and credential handling. Do not delete and recreate an in-use secret.\n3. Restart: `goldsky pipeline restart <name> --from-snapshot last`\n\n#### Sink Unreachable\n\n**Symptoms:** Connection timeout, connection refused, or network errors in the monitor output. Pipeline may cycle between FAILING and TERMINATED.\n\n**Common causes:**\n- Firewall not allowing inbound from AWS us-west-2 (Mirror pipelines write from this region)\n- Database is down or restarted\n- Connection pool exhausted\n- Wrong port or host in the secret\n\n**Fix:**\n1. Verify the sink is reachable from us-west-2.\n2. Check that the secret has the correct host, port, and credentials.\n3. Once connectivity is restored, restart: `goldsky pipeline restart <name> --from-snapshot last`\n\n#### Resource Exhaustion\n\n**Symptoms:** Pipeline runs for a while then terminates. Monitor may show high record counts or slow processing. Common during large backfills or pipelines with many sources/JOINs.\n\n**Fix:**\n1. Resize: `goldsky pipeline resize <name> <size>` — sizes are `s`, `m`, `l`, `xl`, `xxl`.\n2. Start small and go up. `s` handles most workloads (up to 300K records/sec, ~8 subgraph sources). Use `l` or larger for big chain backfills or heavy JOINs.\n\n#### In-Flight Request Blocking\n\n**Symptoms:** User tries to update, delete, or restart the pipeline but gets \"Cannot process request, found existing request in-flight.\"\n\n**Diagnose:** `goldsky pipeline monitor <name> --update-request` — this shows what operation is in progress (usually a snapshot).\n\n**Fix:**\n1. If the in-flight operation is a snapshot that's making progress, wait for it.\n2. If it's stuck or unwanted: `goldsky pipeline cancel-update <name>`\n3. Then retry the original operation.\n\n#### Stuck Snapshot\n\n**Symptoms:** Pipeline can't be paused, updated, or restarted because a snapshot creation is taking too long or failing. The `--update-request` monitor shows snapshot progress stuck at a percentage.\n\n**Fix:**\n1. Cancel the stuck snapshot: `goldsky pipeline cancel-update <name>`\n2. Restart without waiting for a new snapshot: `goldsky pipeline restart <name> --from-snapshot last`\n3. If there's no usable snapshot: `goldsky pipeline restart <name> --from-snapshot none` (starts from scratch — warn the user this reprocesses data)\n\n#### Transform SQL Error\n\n**Symptoms:** Pipeline terminates with SQL-related error messages. Could be syntax errors, referencing a non-existent column, or type mismatches.\n\n**Diagnose:** Check the pipeline definition (`goldsky pipeline get <name> --definition`) and look at the `transforms` section.\n\n**Fix:**\n1. Identify the SQL error from the monitor output.\n2. Fix the SQL in the pipeline YAML file.\n3. Validate: `goldsky pipeline validate <file.yaml>`\n4. Reapply: `goldsky pipeline apply <file.yaml> --status ACTIVE --from-snapshot last`\n\nUse `/mirror` for SQL transform syntax reference if needed.\n\n#### Pipeline in Restart Loop\n\n**Symptoms:** Pipeline repeatedly cycles through STARTING → FAILING → TERMINATED. Monitor shows the same error recurring.\n\n**This is usually a symptom of another root cause** — bad secret, sink unreachable, or resource issues. The pipeline keeps trying to start but hits the same wall.\n\n**Fix:**\n1. Identify the underlying error from the monitor (it's usually one of the patterns above).\n2. Fix the root cause first.\n3. Then restart: `goldsky pipeline restart <name> --from-snapshot last`\n\n#### Sink Downtime Cascade\n\n**Symptoms:** Pipeline was running fine, then the sink (database) went down temporarily. Pipeline auto-retried, then restarted its writers, then eventually terminated.\n\n**This is expected behavior** — Mirror handles transient sink errors automatically (retry batch → restart writers → fail after prolonged issues).\n\n**Fix:**\n1. Confirm the sink is back up and healthy.\n2. Restart from the last snapshot: `goldsky pipeline restart <name> --from-snapshot last`\n3. The pipeline will resume from where it left off, not reprocess everything.\n\n### Step 6: Present Diagnosis\n\nAfter identifying the issue, present findings clearly:\n\n```\n## Diagnosis\n\n**Pipeline:** <name>\n**Status:** <desired> + <runtime>\n**Issue:** <one-line summary>\n\n**Root cause:**\n<What's wrong and why>\n\n**Evidence:**\n- <Error message or observation from monitor>\n- <Relevant detail from pipeline definition>\n\n**Recommended fix:**\n1. <Step 1>\n2. <Step 2>\n\n**Prevention:**\n<How to avoid this in the future, if applicable>\n```\n\n### Step 7: Execute Fix\n\nOffer to run the fix commands directly. Always confirm with the user before executing:\n\n- **Restart:** `goldsky pipeline restart <name> --from-snapshot last`\n- **Resize:** `goldsky pipeline resize <name> <size>`\n- **Cancel blocked operation:** `goldsky pipeline cancel-update <name>`\n- **Restart from scratch:** `goldsky pipeline restart <name> --from-snapshot none` (warn: reprocesses data)\n- **Reapply config:** `goldsky pipeline apply <file.yaml> --status ACTIVE --from-snapshot last`\n- **Delete and recreate:** `goldsky pipeline delete <name> -f` then `goldsky pipeline apply <file.yaml> --status ACTIVE` (last resort)\n\nAfter executing, verify recovery by running `goldsky pipeline monitor <name>` and watching for STARTING → RUNNING transition.\n\n## Important Rules\n\n- Always gather data before diagnosing. Never guess at the problem.\n- Check both desired AND runtime status — the combination matters.\n- Confirm with the user before running any destructive commands (delete, restart from scratch).\n- `--from-snapshot last` preserves progress. `--from-snapshot none` starts over. Default to `last` unless there's a reason not to.\n- Transient errors are auto-retried for up to 6 hours. Non-transient errors terminate immediately. If the pipeline terminated quickly after starting, it's likely a config issue (bad secret, wrong SQL), not a transient network blip.\n- If the problem is beyond CLI diagnosis, suggest contacting support@goldsky.com with the pipeline name, error messages, and project ID.\n\n## When Bash is Not Available\n\nIf you don't have the Bash tool, output the diagnostic commands for the user to run, but structure them clearly:\n\n1. Give one command at a time.\n2. Explain what to look for in the output.\n3. Based on their description of the output, proceed with the diagnosis.\n\nThis is the fallback path — always prefer running commands directly when Bash is available.\n\n## Related\n\n- **`/mirror`** — Pipeline YAML configuration, CLI flag reference, sink setup\n- **`/secrets`** — Create and manage sink credentials\n- **`/auth-setup`** — CLI installation and authentication\n- **`/turbo-doctor`** — Diagnose Turbo pipeline problems (not Mirror)\n"
}