Update to Goldsky

{
  "description": "Use this skill when the user asks about Goldsky Mirror pipelines — creating, deploying, operating, or troubleshooting Mirror. Triggers on: 'Mirror pipeline', 'goldsky pipeline apply', 'sync subgraph to database', 'mirror vs turbo', 'direct indexing', 'mirror pipeline YAML', 'mirror pipeline pause/stop/restart'. Also use this skill when the user wants to sync a Goldsky subgraph into a database or message queue — Mirror is the only pipeline product that supports subgraph sources. For new pipelines that don't need a subgraph source, the turbo-builder skill is usually a better fit. Do NOT trigger on 'goldsky turbo' commands or generic 'build a pipeline' requests without subgraph context — those belong to the turbo skills.",
  "included_files": [
    {
      "relative_path": "evals/trigger-eval.json",
      "size_in_bytes": 2651
    }
  ],
  "name": "mirror",
  "skill_md_contents": "---\nname: mirror\ndescription: \"Use this skill when the user asks about Goldsky Mirror pipelines — creating, deploying, operating, or troubleshooting Mirror. Triggers on: 'Mirror pipeline', 'goldsky pipeline apply', 'sync subgraph to database', 'mirror vs turbo', 'direct indexing', 'mirror pipeline YAML', 'mirror pipeline pause/stop/restart'. Also use this skill when the user wants to sync a Goldsky subgraph into a database or message queue — Mirror is the only pipeline product that supports subgraph sources. For new pipelines that don't need a subgraph source, the turbo-builder skill is usually a better fit. Do NOT trigger on 'goldsky turbo' commands or generic 'build a pipeline' requests without subgraph context — those belong to the turbo skills.\"\n---\n\n# Goldsky Mirror Pipelines\n\nMirror is Goldsky's original streaming pipeline product. It reads onchain data from a **source** (a subgraph entity or a direct-indexing dataset), optionally applies **transforms**, and writes the result to a **sink** (your database or message queue).\n\n### Mirror vs Turbo — which should you use?\n\n| | Mirror | Turbo |\n|---|---|---|\n| Subgraph sources | **Yes** | No |\n| Speed & reliability | Good | **Faster, more reliable** |\n| Sink options | See [common Mirror sinks](#sinks) below | See the [Turbo sink catalog](https://docs.goldsky.com/turbo-pipelines/sinks/overview) |\n| Config complexity | Moderate | **Simpler YAML** |\n| Dataset coverage | 130+ chains | 130+ chains, richer catalog |\n\n**Use Turbo unless you need a subgraph source.** Turbo is faster, more reliable, and actively gaining feature parity with Mirror — especially sink support. If you don't have a subgraph requirement, say \"help me build a Turbo pipeline\" and the `/turbo-builder` skill will guide you through a faster setup.\n\n---\n\n## How Mirror Pipelines Work\n\n```\nSource (subgraph entity or direct-indexing dataset)\n  ↓\nTransforms (optional SQL or external handlers)\n  ↓\nSink (PostgreSQL, ClickHouse, Kafka, S3, etc.)\n```\n\nA pipeline is defined in a YAML file (`apiVersion: 3`) and deployed with `goldsky pipeline apply`.\n\n---\n\n## Pipeline YAML Structure\n\nTop-level fields:\n\n| Field | Type | Required | Description |\n| ----- | ---- | -------- | ----------- |\n| `name` | string | yes | Lowercase letters, numbers, hyphens only. Under 50 characters. |\n| `apiVersion` | number | yes | Always `3` |\n| `resource_size` | string | no | `s` (default), `m`, `l`, `xl`, `xxl` |\n| `description` | string | no | Pipeline description |\n| `sources` | object | yes | At least one source |\n| `transforms` | object | no | Use `{}` if none needed |\n| `sinks` | object | yes | At least one sink |\n\n---\n\n## Sources\n\n| Source type | YAML `type` value | Description |\n| ----------- | ----------------- | ----------- |\n| **Subgraph entity** | `subgraph_entity` | Mirror data from Goldsky-hosted subgraphs |\n| **Dataset (direct indexing)** | `dataset` | Raw onchain datasets (blocks, logs, transactions, traces, transfers) |\n\n### Subgraph entity source\n\n```yaml\nsources:\n  subgraph_account:\n    type: subgraph_entity\n    name: account                # Entity name in your subgraph\n    start_at: latest             # \"earliest\" or \"latest\" (default: latest)\n    filter: \"\"                   # Optional SQL WHERE clause for fast scan\n    subgraphs:\n      - name: my-subgraph        # Deployed subgraph name\n        version: 1.0.0\n      - name: my-subgraph-arb    # Cross-chain: add more subgraphs\n        version: 1.0.0\n```\n\n**Fields:** `type` (required: `subgraph_entity`), `name` (required: entity name), `subgraphs` (required: list of `{name, version}`), `start_at` (optional), `filter` (optional), `description` (optional).\n\n### Dataset source\n\n```yaml\nsources:\n  base_logs:\n    type: dataset\n    dataset_name: base.logs      # Use `goldsky dataset list --output json` to discover names\n    version: 1.0.0               # Use `goldsky dataset get <name>` for versions\n    start_at: latest             # \"earliest\" or \"latest\" (default: latest)\n    filter: \"address = '0x...'\"  # Optional — enables Fast Scan for backfills\n```\n\n**Fields:** `type` (required: `dataset`), `dataset_name` (required), `version` (required), `start_at` (optional), `filter` (optional), `description` (optional).\n\n**Fast Scan:** When `filter` is defined on a dataset source with `start_at: earliest`, the filter is pre-applied at the source level, making historical backfill much faster. Use attributes that exist in the dataset schema (`goldsky dataset get <dataset_name>` to check).\n\nSee [docs.goldsky.com/mirror/sources/supported-sources](https://docs.goldsky.com/mirror/sources/supported-sources).\n\n---\n\n## Sinks\n\nCommon Mirror destinations are listed below; this is not an exhaustive count of supported sink types. Compare the specific destination the user needs, rather than inferring product-wide counts from these examples.\n\n| Sink | YAML `type` value | Notes |\n| ---- | ----------------- | ----- |\n| **PostgreSQL** | `postgres` | Most common — OLTP, auto-creates tables, upsert via INSERT ON CONFLICT. Hosted option via NeonDB. |\n| **ClickHouse** | `clickhouse` | OLAP — uses ReplacingMergeTree by default, `append_only_mode` for best performance |\n| **MySQL** | `mysql` | OLTP workloads |\n| **Elasticsearch** | `elasticsearch` | Real-time search and analytics |\n| **Kafka** | `kafka` | High-throughput streaming to a topic, configurable `topic_partitions` |\n| **Object Storage** | `file` | S3, GCS, or R2 — Parquet format, append-only, supports `partition_columns` |\n| **AWS SQS** | `sqs` | Message queuing |\n| **Webhook** | `webhook` | HTTP POST to an external endpoint |\n\nAll sinks writing to user-managed destinations require a **Goldsky Secret** (`secret_name`). Create one with `goldsky secret create`.\n\nSinks support `schema_override` for casting column types at the sink level (e.g., `string` to `jsonb`).\n\n### Sink examples\n\n```yaml\n# PostgreSQL\nsinks:\n  my_pg:\n    type: postgres\n    table: transfers\n    schema: public\n    secret_name: MY_PG_SECRET\n    from: my_transform\n\n# ClickHouse\nsinks:\n  my_ch:\n    type: clickhouse\n    table: transfers\n    database: my_db\n    secret_name: MY_CH_SECRET\n    from: my_source\n\n# Kafka\nsinks:\n  my_kafka:\n    type: kafka\n    topic: accounts\n    topic_partitions: 2\n    secret_name: MY_KAFKA_SECRET\n    from: my_source\n\n# Object Storage (S3/GCS/R2)\nsinks:\n  my_s3:\n    type: file\n    path: s3://bucket/path/\n    format: parquet\n    secret_name: MY_S3_SECRET\n    from: my_source\n\n# SQS\nsinks:\n  my_sqs:\n    type: sqs\n    url: https://sqs.us-east-1.amazonaws.com/123456/my-queue\n    secret_name: MY_SQS_SECRET\n    from: my_source\n```\n\nSee [docs.goldsky.com/mirror/sinks/supported-sinks](https://docs.goldsky.com/mirror/sinks/supported-sinks).\n\n---\n\n## Transforms\n\n| Type | YAML `type` value | Description |\n| ---- | ----------------- | ----------- |\n| **SQL** | _(none — default)_ | Filter, join, or reshape records with SQL |\n| **External handler** | `handler` | POST records to an HTTP endpoint for custom logic |\n\n### SQL transform\n\n```yaml\ntransforms:\n  filtered_logs:\n    sql: SELECT id, block_number, address FROM base_logs WHERE block_number > 1000\n    primary_key: id\n```\n\nSQL transforms reference source or transform names as table names. Supports chaining (one transform reads from another).\n\n**Built-in decode functions:**\n- `_gs_log_decode(abi, topics, data)` — decode raw log events\n- `_gs_tx_decode(abi, input, output)` — decode raw trace/transaction data\n- `_gs_fetch_abi(url, type)` — fetch ABI from URL (etherscan-compatible or raw JSON); fetched once at pipeline start\n\n### External handler transform\n\n```yaml\ntransforms:\n  my_handler:\n    type: handler\n    primary_key: id\n    url: http://example.com/transform\n    from: my_source\n    batch_size: 100              # Records per batch (default: 100)\n    batch_flush_interval: 1s     # Flush interval (default: 1s)\n    payload_columns: [col1,col2] # Optional: send subset of columns\n    headers:                     # Optional custom headers\n      X-Api-Key: my-key\n```\n\n- At-least-once delivery with exponential backoff on failure\n- Max response time: 5 minutes; max connection time: 1 minute\n- Supports `schema_override` for return type casting\n\nSee [docs.goldsky.com/mirror/transforms/sql-transforms](https://docs.goldsky.com/mirror/transforms/sql-transforms).\n\n---\n\n## Full YAML Examples\n\n### Subgraph entity to PostgreSQL\n\n```yaml\nname: my-subgraph-sync\napiVersion: 3\nresource_size: s\nsources:\n  subgraph_transfer:\n    type: subgraph_entity\n    name: Transfer\n    subgraphs:\n      - name: uniswap-v3\n        version: 1.0.0\ntransforms: {}\nsinks:\n  my_postgres:\n    type: postgres\n    table: transfers\n    schema: public\n    secret_name: MY_PG_SECRET\n    from: subgraph_transfer\n```\n\n### Dataset (direct indexing) to PostgreSQL with SQL transform\n\n```yaml\nname: base-logs-filtered\napiVersion: 3\nresource_size: s\nsources:\n  base_logs:\n    type: dataset\n    dataset_name: base.logs\n    version: 1.0.0\n    start_at: earliest\n    filter: \"address = '0x833589fcd6edb6e08f4c7c32d4f71b54bda02913'\"\ntransforms:\n  select_fields:\n    sql: SELECT id, block_number, transaction_hash, data FROM base_logs\n    primary_key: id\nsinks:\n  pg_logs:\n    type: postgres\n    table: base_logs\n    schema: public\n    secret_name: MY_PG_SECRET\n    from: select_fields\n```\n\n> **No subgraph source?** You should almost certainly use Turbo instead — it's faster, more reliable, and has a richer dataset catalog with simpler syntax. Use `/turbo-builder` to get started.\n\n---\n\n## CLI Reference — All Pipeline Commands\n\nGlobal options available on every command: `--token <string>` (CLI auth token), `--color` (colorize output, default true), `-h, --help`.\n\n### `goldsky pipeline apply <config-path>`\n\nCreate or update a pipeline from a YAML config file. Idempotent.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--status` | `ACTIVE \\| INACTIVE \\| PAUSED` | Desired pipeline status |\n| `--from-snapshot` | string | Snapshot to start from: `last`, `new`, `none`, or a snapshot ID. `last` = latest available. `new` = create a fresh snapshot first. `none` = start from scratch. Default: `new` |\n| `--force` | boolean | Skip confirmation prompts (useful for CI) |\n| `--skip-transform-validation` | boolean | Skip transform validation on update |\n| `--save-progress` | boolean | _(deprecated, use `--from-snapshot`)_ Attempt snapshot before applying |\n| `--use-latest-snapshot` | boolean | _(deprecated, use `--from-snapshot`)_ Start from latest snapshot |\n| `--skip-validation` | boolean | _(deprecated)_ Same as `--skip-transform-validation` |\n\n```bash\ngoldsky pipeline apply my-pipeline.yaml --status ACTIVE\ngoldsky pipeline apply my-pipeline.yaml --status ACTIVE --from-snapshot last\ngoldsky pipeline apply my-pipeline.yaml --force   # CI/CD usage\n```\n\n### `goldsky pipeline start <nameOrConfigPath>`\n\nStart a pipeline (equivalent to apply with `--status ACTIVE`).\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--from-snapshot` | string | `last`, `new`, `none`, or snapshot ID |\n| `--use-latest-snapshot` | boolean | _(deprecated, use `--from-snapshot`)_ |\n\n### `goldsky pipeline stop <nameOrConfigPath>`\n\nStop a pipeline **without** taking a snapshot. Sets status to INACTIVE, runtime to TERMINATED.\n\n_No additional flags beyond global options._\n\n### `goldsky pipeline pause <nameOrConfigPath>`\n\nPause a pipeline **with** a snapshot so it can resume from where it left off. Sets status to PAUSED, runtime to TERMINATED.\n\n_No additional flags beyond global options._\n\n### `goldsky pipeline restart <nameOrConfigPath>`\n\nRestart a pipeline without configuration changes. Useful when the sink database was restarted, connection is stuck, etc.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--from-snapshot` | string | **Required.** `last`, `new`, `none`, or snapshot ID |\n| `--disable-monitoring` | boolean | Skip monitoring after restart (default: false) |\n\n```bash\ngoldsky pipeline restart my-pipeline --from-snapshot last\ngoldsky pipeline restart my-pipeline --from-snapshot none  # restart from scratch\n```\n\n### `goldsky pipeline get <nameOrConfigPath>`\n\nGet pipeline configuration and status.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--outputFormat, --output` | `json \\| table \\| yaml` | Output format (default: yaml) |\n| `--definition` | boolean | Print only the pipeline definition (sources, transforms, sinks) |\n| `-v, --version` | string | Pipeline version (default: latest) |\n\n### `goldsky pipeline list`\n\nList all pipelines in the project.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--output, --outputFormat` | `json \\| table \\| yaml` | Output format (default: table) |\n| `--outputVerbosity` | `summary \\| usablewithapplycmd \\| all` | Detail level (default: summary) |\n| `--include-runtime-details` | boolean | Include runtime status and errors (default: false) |\n\n```bash\ngoldsky pipeline list --output json\ngoldsky pipeline list --include-runtime-details\n```\n\n### `goldsky pipeline info <nameOrConfigPath>`\n\nDisplay pipeline information (status, config, runtime details).\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `-v, --version` | string | Pipeline version (default: latest) |\n\n### `goldsky pipeline monitor <nameOrConfigPath>`\n\nMonitor pipeline runtime — status, metrics (records received/written), errors. Refreshes every 10 seconds.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--update-request` | boolean | Monitor an in-flight update request |\n| `--max-refreshes, --maxRefreshes` | number | Max number of data refreshes |\n| `-v, --version` | string | Pipeline version (default: latest) |\n\n### `goldsky pipeline delete <nameOrConfigPath>`\n\nDelete a pipeline permanently.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `-f, --force` | boolean | Force deletion without confirmation prompt (default: false) |\n\n### `goldsky pipeline resize <nameOrConfigPath> <resourceSize>`\n\nChange the compute resources for a pipeline.\n\n| Positional | Description |\n| ---------- | ----------- |\n| `resourceSize` | One of: `s`, `m`, `l`, `xl`, `xxl` (default: `s`) |\n\n```bash\ngoldsky pipeline resize my-pipeline l\n```\n\n### `goldsky pipeline validate [config-path]`\n\nValidate a pipeline YAML config without deploying.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--definition` | string | _(deprecated)_ Inline JSON definition |\n| `--definition-path` | string | _(deprecated)_ Path to JSON/YAML definition |\n\n```bash\ngoldsky pipeline validate my-pipeline.yaml\n```\n\n### `goldsky pipeline export [name]`\n\nExport pipeline configuration.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--all` | boolean | Export configs for all pipelines |\n\n### `goldsky pipeline cancel-update <nameOrConfigPath>`\n\nCancel an in-flight update or snapshot request. Useful when a long-running snapshot blocks a needed update.\n\n_No additional flags beyond global options._\n\n### `goldsky pipeline create <name>` _(interactive/guided)_\n\nGuided CLI experience for creating a pipeline interactively.\n\n| Flag | Type | Description |\n| ---- | ---- | ----------- |\n| `--resource-size, --resourceSize` | `s \\| m \\| l \\| xl \\| xxl` | Resource size (default: s) |\n| `--use-dedicated-ip` | boolean | Use dedicated egress IPs (default: false) |\n| `--skip-transform-validation` | boolean | Skip transform validation |\n| `--status` | `ACTIVE \\| INACTIVE` | _(deprecated, use `pipeline start/stop/pause`)_ |\n| `--description` | string | _(deprecated, use `pipeline apply`)_ |\n| `--definition` | string | _(deprecated, use `pipeline apply`)_ |\n| `--definition-path` | string | _(deprecated, use `pipeline apply`)_ |\n| `--output, --outputFormat` | `json \\| table \\| yaml` | Output format (default: yaml) |\n\n### `goldsky pipeline get-definition <name>` _(deprecated)_\n\nGet a shareable pipeline definition. **Use `goldsky pipeline get <name> --definition` instead.**\n\n### Snapshot Commands\n\n```bash\n# List snapshots for a pipeline\ngoldsky pipeline snapshots list <nameOrConfigPath> [-v <version>]\n\n# Create a snapshot manually\ngoldsky pipeline snapshots create <nameOrConfigPath>\n```\n\n`snapshots list` supports `-v, --version` to filter by pipeline version (default: all versions).\n\n---\n\n## Lifecycle Quick Reference\n\n| Action | Command |\n| ------ | ------- |\n| Deploy / start | `goldsky pipeline apply <file.yaml> --status ACTIVE` |\n| Start (existing) | `goldsky pipeline start <name>` |\n| Pause (with snapshot) | `goldsky pipeline pause <name>` |\n| Stop (no snapshot) | `goldsky pipeline stop <name>` |\n| Restart (no config change) | `goldsky pipeline restart <name> --from-snapshot last` |\n| Update config | `goldsky pipeline apply <file.yaml>` (edit YAML first) |\n| Resize | `goldsky pipeline resize <name> <size>` |\n| Validate YAML | `goldsky pipeline validate <file.yaml>` |\n| Monitor | `goldsky pipeline monitor <name>` |\n| Get config | `goldsky pipeline get <name> --definition` |\n| Export config | `goldsky pipeline export <name>` |\n| Delete | `goldsky pipeline delete <name> -f` |\n| Cancel in-flight op | `goldsky pipeline cancel-update <name>` |\n| List snapshots | `goldsky pipeline snapshots list <name>` |\n| Create snapshot | `goldsky pipeline snapshots create <name>` |\n| List all pipelines | `goldsky pipeline list` |\n\n**Pause vs. Stop:**\n- `pause` — takes a snapshot and suspends the pipeline (status: PAUSED + TERMINATED). Can resume from where it left off.\n- `stop` — stops without taking a snapshot (status: INACTIVE + TERMINATED). Resuming may reprocess data.\n\n**Desired statuses:** ACTIVE, INACTIVE, PAUSED\n**Runtime statuses:** STARTING, RUNNING, FAILING, TERMINATED\n\n---\n\n## Snapshots\n\nSnapshots capture a point-in-time state of a RUNNING pipeline for resumption. They contain progress on reading sources and SQL transform state — **not** sink state.\n\n- **Automatic snapshots** are taken every 4 hours for healthy RUNNING pipelines.\n- **Before updates:** a snapshot is created automatically before applying config changes to a RUNNING pipeline.\n- **On pause:** a snapshot is created when pausing.\n- **Manual:** `goldsky pipeline snapshots create <name>`.\n- **Resume:** only the latest snapshot can be used. For older snapshots, contact support.\n\nThe `--from-snapshot` flag (on `apply`, `start`, `restart`) controls snapshot behavior:\n- `new` — create a fresh snapshot, then start from it (default)\n- `last` — use the latest existing snapshot (no new snapshot)\n- `none` — start from scratch, no snapshot\n- `<snapshot-id>` — use a specific snapshot\n\n---\n\n## Resource Sizing\n\nSet via `resource_size` in YAML or `goldsky pipeline resize <name> <size>`.\n\n| Size | Description |\n| ---- | ----------- |\n| `s` | Default. Handles most use cases, backfill of small chains, up to 300K records/sec, up to ~8 subgraph sources |\n| `m`, `l`, `xl`, `xxl` | Larger compute — for backfilling large chains or large JOINs |\n\nStart small and scale up if needed. Resource size affects pricing.\n\n---\n\n## Networking\n\n- Mirror pipelines write data from **AWS us-west-2**. Ensure your sink allows inbound connections from this region.\n- IP addresses are **dynamic** by default.\n- **Dedicated egress IPs** available on request — use `--use-dedicated-ip` on `pipeline create`, or contact support@goldsky.com.\n- VPC peering available on request.\n- For external handler transforms, deploy close to us-west-2 for best performance (aim for p95 < 100ms).\n\n---\n\n## Dataset Discovery\n\n```bash\n# List available datasets (non-interactive)\ngoldsky dataset list --output json\n\n# Get schema for a specific dataset\ngoldsky dataset get <dataset_name>\n```\n\n---\n\n## Common Questions\n\n**Can Mirror pipelines use subgraphs as a source?**\nYes — this is Mirror's primary advantage over Turbo. Set `type: subgraph_entity` in your source and reference your deployed subgraph.\n\n**Can Mirror handle multiple sources or cross-chain data?**\nYes — define multiple sources in the YAML and use SQL transforms to join or merge them. For subgraphs, you can list multiple subgraphs (different chains) in a single source's `subgraphs` array.\n\n**My pipeline needs more resources / is too slow?**\nRun `goldsky pipeline resize <name> l` (or `xl`, `xxl`). Start small and scale up.\n\n**My pipeline is ACTIVE but TERMINATED — what happened?**\nThe desired status is ACTIVE but the runtime failed (e.g., bad secret, sink unavailable, resource issues). Check errors with `goldsky pipeline monitor <name> --include-runtime-details` or view the dashboard. Fix the issue and restart.\n\n**How do I update a pipeline without losing progress?**\nEdit your YAML and run `goldsky pipeline apply <file.yaml>`. By default, a snapshot is taken before the update is applied. Use `--from-snapshot last` to skip creating a new snapshot and use the latest existing one.\n\n**A long snapshot is blocking my update — what do I do?**\nRun `goldsky pipeline cancel-update <name>` to cancel the in-flight operation, then reapply with `--from-snapshot last` or `--from-snapshot none`.\n\n---\n\n## Related\n\n- **`/turbo-builder`** — Build a new Turbo pipeline (recommended for new projects not using subgraph sources)\n- **`/subgraph-builder`** — Build and deploy the subgraph you want to sync via Mirror\n- **`/secrets`** — Create secrets for sink credentials\n- **`/datasets`** — Browse available dataset names and chain prefixes\n- **Goldsky docs:** [docs.goldsky.com/mirror/introduction](https://docs.goldsky.com/mirror/introduction)\n- **Pipeline config reference:** [docs.goldsky.com/mirror/reference/config-file/pipeline](https://docs.goldsky.com/mirror/reference/config-file/pipeline)\n"
}