Update to Goldsky

{
  "description": "Turbo pipeline YAML reference and architecture guide. Covers: YAML field syntax (start_at, from, version, primary_key), source/transform/sink configuration, validation errors, resource sizing (xs–xxl), architecture decisions (dataset vs kafka, streaming vs job, fan-out vs fan-in, sink selection, pipeline splitting). Triggers on: 'what does field X do', 'what fields does a postgres sink need', 'what resource size', 'should I use kafka or dataset', 'how to structure my pipeline'. For writing transforms, use /turbo-transforms. For end-to-end building, use /turbo-builder.",
  "included_files": [
    {
      "relative_path": "evals/trigger-eval.json",
      "size_in_bytes": 3438
    },
    {
      "relative_path": "references/architecture-patterns.md",
      "size_in_bytes": 7684
    },
    {
      "relative_path": "references/troubleshooting.md",
      "size_in_bytes": 5971
    },
    {
      "relative_path": "references/validation-checklist.md",
      "size_in_bytes": 3511
    },
    {
      "relative_path": "templates/fan-in-pipeline.yaml",
      "size_in_bytes": 3820
    },
    {
      "relative_path": "templates/fan-out-pipeline.yaml",
      "size_in_bytes": 1886
    },
    {
      "relative_path": "templates/filtered-transfers-sql.yaml",
      "size_in_bytes": 1116
    },
    {
      "relative_path": "templates/linear-pipeline.yaml",
      "size_in_bytes": 1590
    },
    {
      "relative_path": "templates/minimal-erc20-blackhole.yaml",
      "size_in_bytes": 635
    },
    {
      "relative_path": "templates/multi-chain-pipeline.yaml",
      "size_in_bytes": 1127
    },
    {
      "relative_path": "templates/multi-chain-templated.yaml",
      "size_in_bytes": 1468
    },
    {
      "relative_path": "templates/multi-sink-pipeline.yaml",
      "size_in_bytes": 2137
    },
    {
      "relative_path": "templates/postgres-output.yaml",
      "size_in_bytes": 1034
    },
    {
      "relative_path": "templates/pubsub-output.yaml",
      "size_in_bytes": 1237
    },
    {
      "relative_path": "templates/solana-transfers.yaml",
      "size_in_bytes": 780
    }
  ],
  "name": "turbo-pipelines",
  "skill_md_contents": "---\nname: turbo-pipelines\ndescription: \"Turbo pipeline YAML reference and architecture guide. Covers: YAML field syntax (start_at, from, version, primary_key), source/transform/sink configuration, validation errors, resource sizing (xs–xxl), architecture decisions (dataset vs kafka, streaming vs job, fan-out vs fan-in, sink selection, pipeline splitting). Triggers on: 'what does field X do', 'what fields does a postgres sink need', 'what resource size', 'should I use kafka or dataset', 'how to structure my pipeline'. For writing transforms, use /turbo-transforms. For end-to-end building, use /turbo-builder.\"\n---\n\n# Turbo Pipeline Configuration & Architecture\n\nYAML configuration reference and architecture guide for Turbo pipelines. For interactive pipeline building, use `/turbo-builder`. For troubleshooting, use `/turbo-doctor`. For transform implementation, use `/turbo-transforms`.\n\n**Decoded contract events are transform output, not a consumable `<chain>.decoded_logs` dataset.** Source EVM contract logs from `<chain>.raw_logs`, then decode them in a transform (see `/turbo-transforms`). Pre-decoded token datasets such as `<chain>.erc20_transfers`, `<chain>.erc721_transfers`, and `<chain>.erc1155_transfers` are separate, available datasets; use `/datasets` to verify chain coverage.\n\n> **CRITICAL:** Always validate YAML with `goldsky turbo validate <file.yaml>` before showing complete pipeline YAML to the user or deploying.\n\n---\n\n## Quick Start\n\n```yaml\nname: my-first-pipeline\nresource_size: s\nsources:\n  transfers:\n    type: dataset\n    dataset_name: base.erc20_transfers\n    version: 1.2.0\n    start_at: latest\ntransforms: {}\nsinks:\n  output:\n    type: blackhole\n    from: transfers\n```\n\n```bash\ngoldsky turbo validate pipeline.yaml   # Validate first\ngoldsky turbo apply pipeline.yaml -i   # Deploy + inspect\n```\n\n## Prerequisites\n\n- **Goldsky CLI** — if `goldsky` is not on PATH, follow `/auth-setup`\n- **Turbo extension** — follow `/auth-setup` and run its bundled installer with `turbo`. Do not trigger the interactive auto-installer. Verify `goldsky turbo --version` succeeds; the published Linux binary requires x64 and glibc 2.39+, and the Mac binary requires Apple Silicon. Windows uses the WSL entry point. An unsupported platform is not a successful installation.\n- **Shell PATH** — restore `export PATH=\"$HOME/.local/bin:$HOME/.goldsky/bin:$PATH\"` in every new shell/tool call; exports do not persist across calls.\n- **Logged in** — follow `/auth-setup` and run `goldsky login` yourself when `goldsky project list` is not authenticated\n- Secrets created for sinks if using PostgreSQL, ClickHouse, Kafka, etc. (see `/secrets`)\n\n---\n\n## Architecture Decisions\n\n### Source Type Selection\n\n| Scenario                          | Source Type | Why                                          |\n| --------------------------------- | ----------- | -------------------------------------------- |\n| Decode contract events from logs  | `dataset`   | Need `raw_logs` + `_gs_log_decode()`         |\n| Track token transfers             | `dataset`   | `erc20_transfers` has structured data        |\n| Historical backfill + live        | `dataset`   | `start_at: earliest` processes history       |\n| Live token balances               | `kafka`     | `latest_balances_v2` is a streaming topic    |\n| Real-time state snapshots         | `kafka`     | Kafka delivers latest state continuously     |\n| Only need new data going forward  | Either      | Dataset with `start_at: latest` or Kafka     |\n\n> **Note:** Kafka sources are used in production but are not documented in official Goldsky docs. Contact Goldsky support for topic names.\n\n### Data Flow Patterns\n\n| Pattern              | When to Use                                          | Template                           |\n| -------------------- | ---------------------------------------------------- | ---------------------------------- |\n| **Linear**           | Single source, single destination, simple processing | `templates/linear-pipeline.yaml`   |\n| **Fan-out**          | One source → multiple sinks (different views/subsets)| `templates/fan-out-pipeline.yaml`  |\n| **Fan-in**           | Multiple event types → UNION ALL → one table         | `templates/fan-in-pipeline.yaml`   |\n| **Multi-chain**      | Same logic across chains (separate pipelines)        | `templates/multi-chain-templated.yaml` |\n\nFor detailed pattern diagrams, YAML examples, and multi-chain deployment guidance, read `references/architecture-patterns.md`.\n\n### Resource Sizing\n\n| Size  | Workers | CPU   | Memory  | When to Use                                                      |\n| ----- | ------- | ----- | ------- | ---------------------------------------------------------------- |\n| `xs`  | —       | 0.4   | 0.5 Gi  | Small datasets, light testing                                    |\n| `s`   | 1       | 0.8   | 1.0 Gi  | Simple filters, single source/sink, low volume (default)         |\n| `m`   | 4       | 1.6   | 2.0 Gi  | Multiple sinks, Kafka streaming, moderate transform complexity   |\n| `l`   | 10      | 3.2   | 4.0 Gi  | Multi-event decoding + UNION ALL, high-volume backfill           |\n| `xl`  | 20      | 6.4   | 8.0 Gi  | Large chain backfills, complex JOINs                             |\n| `xxl` | 40      | 12.8  | 16.0 Gi | Highest throughput; up to 6.3M rows/min                          |\n\nStart small and scale up — defensive sizing avoids wasted resources.\n\n**`resource_size` sizes the pipeline, not the sink — they are independent.** An `l`/`xl` pipeline pointed at an undersized destination produces the \"deployed but writing nothing\" shape: it validates, deploys, reports `Running`, and then fails every write. A Neon free tier (512 MB) will not hold a multi-month backfill of a high-volume chain — it errors `could not extend file because project size limit (512 MB) has been exceeded` and checkpoints time out. Check the destination's storage ceiling against the backfill scope (the start position plus any end bound) before deploying, not after.\n\n### Sink Selection\n\n| Destination          | Sink Type            | Best For                                       |\n| -------------------- | -------------------- | ---------------------------------------------- |\n| Application DB       | `postgres`           | Row-level lookups, joins, application serving  |\n| Real-time aggregates | `postgres_aggregate` | Balances, counters, running totals via triggers|\n| MySQL-compatible DB  | `mysql`              | Existing MySQL stacks, upserts on primary key  |\n| Analytics queries    | `clickhouse`         | Large-scale aggregations, time-series data     |\n| Event processing     | `kafka`              | Downstream consumers, event-driven systems     |\n| AWS messaging        | `sqs_sink`           | Decoupled AWS consumers (standard queues only) |\n| GCP messaging        | `pubsub`             | Google Cloud Pub/Sub topics (Turbo-only)       |\n| Serverless streaming | `s2_sink`            | S2.dev streams, alternative to Kafka           |\n| Notifications        | `webhook`            | Lambda functions, API callbacks, alerts        |\n| Data lake            | `s3_sink`            | Long-term archival, batch processing           |\n| Testing              | `blackhole`          | Validate pipeline without writing data         |\n\nFor full sink field specifications, see [Sinks](https://docs.goldsky.com/turbo-pipelines/sinks/overview) in the docs (one page per sink type).\n\n### Streaming vs Job Mode\n\n| Scenario                             | Mode        | Why                                            |\n| ------------------------------------ | ----------- | ---------------------------------------------- |\n| Real-time dashboard                  | Streaming   | Continuous updates needed                      |\n| Backfill 6 months of history         | Job         | One-time, stops when done                      |\n| Real-time + catch-up on deploy       | Streaming   | `start_at: earliest` does backfill then streams|\n| Export data to S3 once               | Job         | No need for continuous processing              |\n| Webhook notifications on events      | Streaming   | Needs to react as events happen                |\n| Load test with historical data       | Job         | Process and inspect, then discard              |\n\n**Job mode rules:** Runs to completion, auto-deletes ~1hr after finishing. Must delete before redeploying. Cannot pause/resume/restart.\n\n### Pipeline Splitting\n\n**Default to one pipeline with multiple sinks** when the user wants the same source data sent to multiple destinations. A single Turbo pipeline supports multiple sinks natively — each sink has a `from:` field pointing at a source or transform by name, and sinks run independently (one failing does not block the others, and each can have its own `batch_size` / `batch_flush_interval`).\n\n**Do NOT split into separate pipelines just because there are multiple destinations.** Generating one pipeline per sink duplicates the source ingestion, wastes resources, and decouples deployments that should be atomic.\n\n**Split into multiple pipelines only when:** sources are fundamentally different (different chains with independent lifecycles), the destinations need different resource sizes, or the user explicitly asks for separate pipelines.\n\nSee `references/architecture-patterns.md` for fan-out (one source → multiple sinks) and fan-in (UNION ALL) YAML examples, and `templates/multi-sink-pipeline.yaml` / `templates/fan-out-pipeline.yaml` / `templates/fan-in-pipeline.yaml` for ready-to-adapt configs.\n\n---\n\n## Configuration Reference\n\n### Pipeline Structure\n\n```yaml\nname: my-pipeline          # Required: unique identifier (lowercase, hyphens)\nresource_size: s            # Required: xs/s/m/l/xl/xxl\njob: true                   # Optional: one-time batch (default: false = streaming)\n\nsources:\n  source_name:\n    type: dataset           # or: kafka\n    # ... source config\n\ntransforms:                 # Optional\n  transform_name:\n    type: sql               # or: script, handler, dynamic_table\n    # ... transform config\n\nsinks:\n  sink_name:\n    type: postgres           # or: clickhouse, kafka, webhook, s3_sink, etc.\n    # ... sink config\n```\n\n### Source Configuration\n\n#### Dataset Source\n\n```yaml\nsources:\n  my_source:\n    type: dataset\n    dataset_name: <chain>.<dataset_type>\n    version: <version>\n    start_at: latest | earliest    # EVM, NEAR, Bitcoin, Stellar\n    # start_block: <slot_number>   # Solana only — omit to start at the latest slot\n    # end_block: <block_number>    # Solana only: bounded backfill (ignored on EVM)\n    # filter: >-                   # Optional: SQL WHERE for source-level pre-filtering\n    #   address = '0x...' AND block_number >= 10000000\n```\n\n| Field          | Required | Description                                              |\n| -------------- | -------- | -------------------------------------------------------- |\n| `type`         | Yes      | `dataset` for blockchain data                            |\n| `dataset_name` | Yes      | Format: `<chain>.<dataset_type>`                         |\n| `version`      | Yes      | Dataset version (e.g., `1.2.0`)                          |\n| `start_at`     | EVM-family | `latest` or `earliest`; on Stellar also a ledger sequence number. Used by EVM, NEAR, Bitcoin, Stellar. Omitted = `earliest` = full chain history |\n| `start_block`  | No       | Solana only: starting slot. Omitted = latest slot, i.e. no backfill      |\n| `end_block`    | No       | Solana only: stop at this slot. Silently ignored on EVM                  |\n| `filter`       | No       | SQL WHERE clause — pre-filters at ingestion (efficient)  |\n\nUse `filter` for contract addresses and block ranges (coarse pre-filtering). Use transform `WHERE` for fine-grained filtering.\n\nSet the start position explicitly on every dataset source, and say which one you chose. On the `start_at` chains (EVM, NEAR, Bitcoin, Stellar) omitting it does not mean \"start now\": the backend starts from the earliest available data, so the pipeline backfills the full chain history — days of replay and millions of rows before it reaches live data, and the sink has to hold all of it. A block number is not a valid `start_at` value; bound an EVM range with a `block_number` predicate in `filter` (pre-applied at the source), plus `job: true` for a one-shot backfill — `end_block` is silently ignored on EVM. Solana is the opposite default: it uses the numeric `start_block`, and omitting that starts at the latest slot, so a Solana backfill is always something you asked for — bound it with `end_block` or `block_ranges`.\n\nFor chain prefixes and dataset types, see `/datasets`.\n\n#### Kafka Source\n\n```yaml\nsources:\n  my_source:\n    type: kafka\n    topic: base.raw.latest_balances_v2\n```\n\nNo `start_at` or `version` fields. Optional: `filter`, `include_metadata`, `starting_offsets`.\n\n### Transform Configuration\n\n| Type            | Use Case                                       |\n| --------------- | ---------------------------------------------- |\n| `sql`           | Filtering, projections, SQL functions          |\n| `script`        | Custom TypeScript/WASM logic                   |\n| `handler`       | Call external HTTP APIs to enrich              |\n| `dynamic_table` | Lookup tables backed by a database             |\n| `throttle`      | Cap throughput to a fixed records-per-second   |\n\n#### SQL Transform\n\n```yaml\ntransforms:\n  filtered:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT id, sender, recipient, amount\n      FROM source_name\n      WHERE amount > 1000\n```\n\n| Field         | Required | Description                            |\n| ------------- | -------- | -------------------------------------- |\n| `type`        | Yes      | `sql`                                  |\n| `primary_key` | Yes      | Column for uniqueness/ordering         |\n| `sql`         | Yes      | SQL query (reference sources by name)  |\n| `from`        | No       | Override default source (for chaining) |\n\n#### Transform Chaining\n\nChain transforms using `from`:\n\n```yaml\ntransforms:\n  step1:\n    type: sql\n    primary_key: id\n    sql: SELECT * FROM source WHERE amount > 100\n  step2:\n    type: sql\n    primary_key: id\n    from: step1\n    sql: SELECT *, 'processed' as status FROM step1\n```\n\nFor TypeScript, handler, and dynamic table transforms, see `/turbo-transforms`.\n\n#### Throttle Transform\n\nCaps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Throttle does **not** modify data — every input record passes through unchanged. Use it to stay under rate limits of downstream sinks or external APIs, smooth bursty sources, or pace records into HTTP handlers.\n\n```yaml\ntransforms:\n  throttled:\n    type: throttle\n    from: my_source\n    max_batch_size: 100\n    min_batch_interval: 10s\n```\n\n| Field                | Required | Description                                                  |\n| -------------------- | -------- | ------------------------------------------------------------ |\n| `type`               | Yes      | `throttle`                                                   |\n| `from`               | Yes      | Source or transform to read from                             |\n| `max_batch_size`     | No       | Max records per batch                                        |\n| `min_batch_interval` | No       | Minimum time between batches (e.g. `10s`, `500ms`, `1m`)     |\n\nEffective max throughput ≈ `max_batch_size / min_batch_interval` records per second. Throttle limits the **maximum** rate, not the minimum — if upstream is slow, batches will be smaller and arrive less frequently.\n\n**Place throttle close to the bottleneck** (just before the rate-limited sink or handler) so upstream transforms still process at full speed.\n\n### Sink Configuration\n\nQuick examples for common sinks. For full field specs of all sink types, see [Sinks](https://docs.goldsky.com/turbo-pipelines/sinks/overview) in the docs (one page per sink type).\n\n**`primary_key` placement depends on the sink type; it is not limited to transforms.** A SQL transform's key describes its output, while a sink's key controls destination behavior:\n\n| Turbo sink | Sink-level `primary_key` |\n| --- | --- |\n| [PostgreSQL](https://docs.goldsky.com/turbo-pipelines/sinks/postgres) | Optional; enables upserts. Omit for plain inserts. |\n| [MySQL](https://docs.goldsky.com/turbo-pipelines/sinks/mysql) | Optional; enables upserts. Omit for plain inserts. |\n| [ClickHouse](https://docs.goldsky.com/turbo-pipelines/sinks/clickhouse) | Required; sets table ordering and deduplication columns. |\n| [Kafka](https://docs.goldsky.com/turbo-pipelines/sinks/kafka) | Optional; selects message-key columns. If omitted, uses the upstream key when available. |\n\nFor other sink types, check their own field reference before adding or removing `primary_key`. Mirror uses different sink schemas; do not copy Turbo sink fields into Mirror configs.\n\n#### PostgreSQL\n\n```yaml\nsinks:\n  output:\n    type: postgres\n    from: my_transform\n    schema: public\n    table: my_table\n    secret_name: MY_POSTGRES_SECRET\n    primary_key: id\n```\n\n#### ClickHouse\n\n```yaml\nsinks:\n  output:\n    type: clickhouse\n    from: my_transform\n    table: my_table\n    secret_name: MY_CLICKHOUSE_SECRET\n    primary_key: id\n```\n\n#### Pub/Sub (Turbo-only)\n\n```yaml\nsinks:\n  output:\n    type: pubsub\n    from: my_transform\n    topic: my-topic\n    secret_name: MY_PUBSUB_SECRET\n```\n\nThe secret holds a GCP project id and service-account JSON. Topic must already exist in GCP.\n\n#### Blackhole (Testing)\n\n```yaml\nsinks:\n  output:\n    type: blackhole\n    from: my_transform\n```\n\n---\n\n## Checkpoint Behavior\n\n- **Preserved by default** when updating a pipeline\n- **Tied to source names** — renaming a source resets its checkpoint\n- **Tied to pipeline names** — renaming the pipeline resets all checkpoints\n\nTo reset checkpoints: rename the source or pipeline. Warning: this reprocesses all historical data.\n\n---\n\n## Starter Templates\n\n| Template                       | Description                       | Use Case                     |\n| ------------------------------ | --------------------------------- | ---------------------------- |\n| `minimal-erc20-blackhole.yaml` | Simplest pipeline, no credentials | Quick testing                |\n| `filtered-transfers-sql.yaml`  | Filter by contract address        | USDC, specific tokens        |\n| `postgres-output.yaml`         | Write to PostgreSQL               | Production data storage      |\n| `pubsub-output.yaml`           | Write to Google Cloud Pub/Sub     | GCP-based event consumers    |\n| `multi-chain-pipeline.yaml`    | Combine multiple chains           | Cross-chain analytics        |\n| `solana-transfers.yaml`        | Solana SPL tokens                 | Non-EVM chains               |\n| `multi-sink-pipeline.yaml`     | Multiple outputs                  | Archive + alerts + streaming |\n| `linear-pipeline.yaml`         | Simple decode → filter → sink     | Basic linear flow            |\n| `fan-out-pipeline.yaml`        | One source → multiple sinks       | Multi-destination            |\n| `fan-in-pipeline.yaml`         | Multiple events → UNION ALL       | Activity feeds               |\n| `multi-chain-templated.yaml`   | Per-chain pipeline pattern        | Independent chain deploys    |\n\n**Template location:** `templates/` (relative to this skill's directory)\n\n---\n\n## CLI Quick Reference\n\n| Action                  | Command                                                |\n| ----------------------- | ------------------------------------------------------ |\n| Install Goldsky CLI     | Follow `/auth-setup` and use its bundled installer |\n| Install Turbo extension | Follow `/auth-setup` and run its bundled installer with `turbo`; require a successful version check |\n| **Validate (REQUIRED)** | `goldsky turbo validate pipeline.yaml`                 |\n| Deploy/Update           | `goldsky turbo apply pipeline.yaml`                    |\n| Deploy + Inspect        | `goldsky turbo apply pipeline.yaml -i`                 |\n| List pipelines          | `goldsky turbo list`                                   |\n| List datasets           | `goldsky dataset list --output json`                   |\n| List secrets            | `goldsky secret list`                                  |\n\nFor lifecycle commands (pause/resume/restart/delete) and monitoring (inspect/logs), see `/turbo-operations`.\n\n---\n\n## Troubleshooting\n\nSee `references/troubleshooting.md` for CLI hanging, validation errors, and runtime errors.\n\n---\n\n## Related\n\n- **`/turbo-builder`** — Interactive wizard to build pipelines step-by-step\n- **`/turbo-doctor`** — Diagnose and fix pipeline issues\n- **`/turbo-operations`** — Lifecycle commands and monitoring reference\n- **`/turbo-transforms`** — SQL, TypeScript, and dynamic table transform reference\n- **`/datasets`** — Dataset names and chain prefixes\n- **`/secrets`** — Sink credential management\n"
}