{"id":83626,"plugin_id":"plugin_asdk_app_6ad6851613bc819189dbdda8700e1094","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-11-06T00:15:47.341Z","digest":"cfca4dae4aae40afe779c69ab1fff01e474ffea4972a8bd6948416bed4271048","against":null,"payload":{"description":"Turbo pipeline YAML reference and architecture guide. Covers: YAML field syntax (start_at, from, version, primary_key), source/transform/sink configuration, validation errors, resource sizing (xs–xxl), architecture decisions (dataset vs kafka, streaming vs job, fan-out vs fan-in, sink selection, pipeline splitting). Triggers on: 'what does field X do', 'what fields does a postgres sink need', 'what resource size', 'should I use kafka or dataset', 'how to structure my pipeline'. For writing transforms, use /turbo-transforms. For end-to-end building, use /turbo-builder.","included_files":[{"relative_path":"evals/trigger-eval.json","size_in_bytes":3438},{"relative_path":"references/architecture-patterns.md","size_in_bytes":7684},{"relative_path":"references/troubleshooting.md","size_in_bytes":5971},{"relative_path":"references/validation-checklist.md","size_in_bytes":3511},{"relative_path":"templates/fan-in-pipeline.yaml","size_in_bytes":3820},{"relative_path":"templates/fan-out-pipeline.yaml","size_in_bytes":1886},{"relative_path":"templates/filtered-transfers-sql.yaml","size_in_bytes":1116},{"relative_path":"templates/linear-pipeline.yaml","size_in_bytes":1590},{"relative_path":"templates/minimal-erc20-blackhole.yaml","size_in_bytes":635},{"relative_path":"templates/multi-chain-pipeline.yaml","size_in_bytes":1127},{"relative_path":"templates/multi-chain-templated.yaml","size_in_bytes":1468},{"relative_path":"templates/multi-sink-pipeline.yaml","size_in_bytes":2137},{"relative_path":"templates/postgres-output.yaml","size_in_bytes":1034},{"relative_path":"templates/pubsub-output.yaml","size_in_bytes":1237},{"relative_path":"templates/solana-transfers.yaml","size_in_bytes":780}],"name":"turbo-pipelines","skill_md_contents":"---\nname: turbo-pipelines\ndescription: \"Turbo pipeline YAML reference and architecture guide. Covers: YAML field syntax (start_at, from, version, primary_key), source/transform/sink configuration, validation errors, resource sizing (xs–xxl), architecture decisions (dataset vs kafka, streaming vs job, fan-out vs fan-in, sink selection, pipeline splitting). Triggers on: 'what does field X do', 'what fields does a postgres sink need', 'what resource size', 'should I use kafka or dataset', 'how to structure my pipeline'. For writing transforms, use /turbo-transforms. For end-to-end building, use /turbo-builder.\"\n---\n\n# Turbo Pipeline Configuration & Architecture\n\nYAML configuration reference and architecture guide for Turbo pipelines. For interactive pipeline building, use `/turbo-builder`. For troubleshooting, use `/turbo-doctor`. For transform implementation, use `/turbo-transforms`.\n\n**Decoded contract events are transform output, not a consumable `.decoded_logs` dataset.** Source EVM contract logs from `.raw_logs`, then decode them in a transform (see `/turbo-transforms`). Pre-decoded token datasets such as `.erc20_transfers`, `.erc721_transfers`, and `.erc1155_transfers` are separate, available datasets; use `/datasets` to verify chain coverage.\n\n> **CRITICAL:** Always validate YAML with `goldsky turbo validate ` before showing complete pipeline YAML to the user or deploying.\n\n---\n\n## Quick Start\n\n```yaml\nname: my-first-pipeline\nresource_size: s\nsources:\n transfers:\n type: dataset\n dataset_name: base.erc20_transfers\n version: 1.2.0\n start_at: latest\ntransforms: {}\nsinks:\n output:\n type: blackhole\n from: transfers\n```\n\n```bash\ngoldsky turbo validate pipeline.yaml # Validate first\ngoldsky turbo apply pipeline.yaml -i # Deploy + inspect\n```\n\n## Prerequisites\n\n- **Goldsky CLI** — if `goldsky` is not on PATH, follow `/auth-setup`\n- **Turbo extension** — follow `/auth-setup` and run its bundled installer with `turbo`. Do not trigger the interactive auto-installer. Verify `goldsky turbo --version` succeeds; the published Linux binary requires x64 and glibc 2.39+, and the Mac binary requires Apple Silicon. Windows uses the WSL entry point. An unsupported platform is not a successful installation.\n- **Shell PATH** — restore `export PATH=\"$HOME/.local/bin:$HOME/.goldsky/bin:$PATH\"` in every new shell/tool call; exports do not persist across calls.\n- **Logged in** — follow `/auth-setup` and run `goldsky login` yourself when `goldsky project list` is not authenticated\n- Secrets created for sinks if using PostgreSQL, ClickHouse, Kafka, etc. (see `/secrets`)\n\n---\n\n## Architecture Decisions\n\n### Source Type Selection\n\n| Scenario | Source Type | Why |\n| --------------------------------- | ----------- | -------------------------------------------- |\n| Decode contract events from logs | `dataset` | Need `raw_logs` + `_gs_log_decode()` |\n| Track token transfers | `dataset` | `erc20_transfers` has structured data |\n| Historical backfill + live | `dataset` | `start_at: earliest` processes history |\n| Live token balances | `kafka` | `latest_balances_v2` is a streaming topic |\n| Real-time state snapshots | `kafka` | Kafka delivers latest state continuously |\n| Only need new data going forward | Either | Dataset with `start_at: latest` or Kafka |\n\n> **Note:** Kafka sources are used in production but are not documented in official Goldsky docs. Contact Goldsky support for topic names.\n\n### Data Flow Patterns\n\n| Pattern | When to Use | Template |\n| -------------------- | ---------------------------------------------------- | ---------------------------------- |\n| **Linear** | Single source, single destination, simple processing | `templates/linear-pipeline.yaml` |\n| **Fan-out** | One source → multiple sinks (different views/subsets)| `templates/fan-out-pipeline.yaml` |\n| **Fan-in** | Multiple event types → UNION ALL → one table | `templates/fan-in-pipeline.yaml` |\n| **Multi-chain** | Same logic across chains (separate pipelines) | `templates/multi-chain-templated.yaml` |\n\nFor detailed pattern diagrams, YAML examples, and multi-chain deployment guidance, read `references/architecture-patterns.md`.\n\n### Resource Sizing\n\n| Size | Workers | CPU | Memory | When to Use |\n| ----- | ------- | ----- | ------- | ---------------------------------------------------------------- |\n| `xs` | — | 0.4 | 0.5 Gi | Small datasets, light testing |\n| `s` | 1 | 0.8 | 1.0 Gi | Simple filters, single source/sink, low volume (default) |\n| `m` | 4 | 1.6 | 2.0 Gi | Multiple sinks, Kafka streaming, moderate transform complexity |\n| `l` | 10 | 3.2 | 4.0 Gi | Multi-event decoding + UNION ALL, high-volume backfill |\n| `xl` | 20 | 6.4 | 8.0 Gi | Large chain backfills, complex JOINs |\n| `xxl` | 40 | 12.8 | 16.0 Gi | Highest throughput; up to 6.3M rows/min |\n\nStart small and scale up — defensive sizing avoids wasted resources.\n\n**`resource_size` sizes the pipeline, not the sink — they are independent.** An `l`/`xl` pipeline pointed at an undersized destination produces the \"deployed but writing nothing\" shape: it validates, deploys, reports `Running`, and then fails every write. A Neon free tier (512 MB) will not hold a multi-month backfill of a high-volume chain — it errors `could not extend file because project size limit (512 MB) has been exceeded` and checkpoints time out. Check the destination's storage ceiling against the backfill scope (the start position plus any end bound) before deploying, not after.\n\n### Sink Selection\n\n| Destination | Sink Type | Best For |\n| -------------------- | -------------------- | ---------------------------------------------- |\n| Application DB | `postgres` | Row-level lookups, joins, application serving |\n| Real-time aggregates | `postgres_aggregate` | Balances, counters, running totals via triggers|\n| MySQL-compatible DB | `mysql` | Existing MySQL stacks, upserts on primary key |\n| Analytics queries | `clickhouse` | Large-scale aggregations, time-series data |\n| Event processing | `kafka` | Downstream consumers, event-driven systems |\n| AWS messaging | `sqs_sink` | Decoupled AWS consumers (standard queues only) |\n| GCP messaging | `pubsub` | Google Cloud Pub/Sub topics (Turbo-only) |\n| Serverless streaming | `s2_sink` | S2.dev streams, alternative to Kafka |\n| Notifications | `webhook` | Lambda functions, API callbacks, alerts |\n| Data lake | `s3_sink` | Long-term archival, batch processing |\n| Testing | `blackhole` | Validate pipeline without writing data |\n\nFor full sink field specifications, see [Sinks](https://docs.goldsky.com/turbo-pipelines/sinks/overview) in the docs (one page per sink type).\n\n### Streaming vs Job Mode\n\n| Scenario | Mode | Why |\n| ------------------------------------ | ----------- | ---------------------------------------------- |\n| Real-time dashboard | Streaming | Continuous updates needed |\n| Backfill 6 months of history | Job | One-time, stops when done |\n| Real-time + catch-up on deploy | Streaming | `start_at: earliest` does backfill then streams|\n| Export data to S3 once | Job | No need for continuous processing |\n| Webhook notifications on events | Streaming | Needs to react as events happen |\n| Load test with historical data | Job | Process and inspect, then discard |\n\n**Job mode rules:** Runs to completion, auto-deletes ~1hr after finishing. Must delete before redeploying. Cannot pause/resume/restart.\n\n### Pipeline Splitting\n\n**Default to one pipeline with multiple sinks** when the user wants the same source data sent to multiple destinations. A single Turbo pipeline supports multiple sinks natively — each sink has a `from:` field pointing at a source or transform by name, and sinks run independently (one failing does not block the others, and each can have its own `batch_size` / `batch_flush_interval`).\n\n**Do NOT split into separate pipelines just because there are multiple destinations.** Generating one pipeline per sink duplicates the source ingestion, wastes resources, and decouples deployments that should be atomic.\n\n**Split into multiple pipelines only when:** sources are fundamentally different (different chains with independent lifecycles), the destinations need different resource sizes, or the user explicitly asks for separate pipelines.\n\nSee `references/architecture-patterns.md` for fan-out (one source → multiple sinks) and fan-in (UNION ALL) YAML examples, and `templates/multi-sink-pipeline.yaml` / `templates/fan-out-pipeline.yaml` / `templates/fan-in-pipeline.yaml` for ready-to-adapt configs.\n\n---\n\n## Configuration Reference\n\n### Pipeline Structure\n\n```yaml\nname: my-pipeline # Required: unique identifier (lowercase, hyphens)\nresource_size: s # Required: xs/s/m/l/xl/xxl\njob: true # Optional: one-time batch (default: false = streaming)\n\nsources:\n source_name:\n type: dataset # or: kafka\n # ... source config\n\ntransforms: # Optional\n transform_name:\n type: sql # or: script, handler, dynamic_table\n # ... transform config\n\nsinks:\n sink_name:\n type: postgres # or: clickhouse, kafka, webhook, s3_sink, etc.\n # ... sink config\n```\n\n### Source Configuration\n\n#### Dataset Source\n\n```yaml\nsources:\n my_source:\n type: dataset\n dataset_name: .\n version: \n start_at: latest | earliest # EVM, NEAR, Bitcoin, Stellar\n # start_block: # Solana only — omit to start at the latest slot\n # end_block: # Solana only: bounded backfill (ignored on EVM)\n # filter: >- # Optional: SQL WHERE for source-level pre-filtering\n # address = '0x...' AND block_number >= 10000000\n```\n\n| Field | Required | Description |\n| -------------- | -------- | -------------------------------------------------------- |\n| `type` | Yes | `dataset` for blockchain data |\n| `dataset_name` | Yes | Format: `.` |\n| `version` | Yes | Dataset version (e.g., `1.2.0`) |\n| `start_at` | EVM-family | `latest` or `earliest`; on Stellar also a ledger sequence number. Used by EVM, NEAR, Bitcoin, Stellar. Omitted = `earliest` = full chain history |\n| `start_block` | No | Solana only: starting slot. Omitted = latest slot, i.e. no backfill |\n| `end_block` | No | Solana only: stop at this slot. Silently ignored on EVM |\n| `filter` | No | SQL WHERE clause — pre-filters at ingestion (efficient) |\n\nUse `filter` for contract addresses and block ranges (coarse pre-filtering). Use transform `WHERE` for fine-grained filtering.\n\nSet the start position explicitly on every dataset source, and say which one you chose. On the `start_at` chains (EVM, NEAR, Bitcoin, Stellar) omitting it does not mean \"start now\": the backend starts from the earliest available data, so the pipeline backfills the full chain history — days of replay and millions of rows before it reaches live data, and the sink has to hold all of it. A block number is not a valid `start_at` value; bound an EVM range with a `block_number` predicate in `filter` (pre-applied at the source), plus `job: true` for a one-shot backfill — `end_block` is silently ignored on EVM. Solana is the opposite default: it uses the numeric `start_block`, and omitting that starts at the latest slot, so a Solana backfill is always something you asked for — bound it with `end_block` or `block_ranges`.\n\nFor chain prefixes and dataset types, see `/datasets`.\n\n#### Kafka Source\n\n```yaml\nsources:\n my_source:\n type: kafka\n topic: base.raw.latest_balances_v2\n```\n\nNo `start_at` or `version` fields. Optional: `filter`, `include_metadata`, `starting_offsets`.\n\n### Transform Configuration\n\n| Type | Use Case |\n| --------------- | ---------------------------------------------- |\n| `sql` | Filtering, projections, SQL functions |\n| `script` | Custom TypeScript/WASM logic |\n| `handler` | Call external HTTP APIs to enrich |\n| `dynamic_table` | Lookup tables backed by a database |\n| `throttle` | Cap throughput to a fixed records-per-second |\n\n#### SQL Transform\n\n```yaml\ntransforms:\n filtered:\n type: sql\n primary_key: id\n sql: |\n SELECT id, sender, recipient, amount\n FROM source_name\n WHERE amount > 1000\n```\n\n| Field | Required | Description |\n| ------------- | -------- | -------------------------------------- |\n| `type` | Yes | `sql` |\n| `primary_key` | Yes | Column for uniqueness/ordering |\n| `sql` | Yes | SQL query (reference sources by name) |\n| `from` | No | Override default source (for chaining) |\n\n#### Transform Chaining\n\nChain transforms using `from`:\n\n```yaml\ntransforms:\n step1:\n type: sql\n primary_key: id\n sql: SELECT * FROM source WHERE amount > 100\n step2:\n type: sql\n primary_key: id\n from: step1\n sql: SELECT *, 'processed' as status FROM step1\n```\n\nFor TypeScript, handler, and dynamic table transforms, see `/turbo-transforms`.\n\n#### Throttle Transform\n\nCaps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Throttle does **not** modify data — every input record passes through unchanged. Use it to stay under rate limits of downstream sinks or external APIs, smooth bursty sources, or pace records into HTTP handlers.\n\n```yaml\ntransforms:\n throttled:\n type: throttle\n from: my_source\n max_batch_size: 100\n min_batch_interval: 10s\n```\n\n| Field | Required | Description |\n| -------------------- | -------- | ------------------------------------------------------------ |\n| `type` | Yes | `throttle` |\n| `from` | Yes | Source or transform to read from |\n| `max_batch_size` | No | Max records per batch |\n| `min_batch_interval` | No | Minimum time between batches (e.g. `10s`, `500ms`, `1m`) |\n\nEffective max throughput ≈ `max_batch_size / min_batch_interval` records per second. Throttle limits the **maximum** rate, not the minimum — if upstream is slow, batches will be smaller and arrive less frequently.\n\n**Place throttle close to the bottleneck** (just before the rate-limited sink or handler) so upstream transforms still process at full speed.\n\n### Sink Configuration\n\nQuick examples for common sinks. For full field specs of all sink types, see [Sinks](https://docs.goldsky.com/turbo-pipelines/sinks/overview) in the docs (one page per sink type).\n\n**`primary_key` placement depends on the sink type; it is not limited to transforms.** A SQL transform's key describes its output, while a sink's key controls destination behavior:\n\n| Turbo sink | Sink-level `primary_key` |\n| --- | --- |\n| [PostgreSQL](https://docs.goldsky.com/turbo-pipelines/sinks/postgres) | Optional; enables upserts. Omit for plain inserts. |\n| [MySQL](https://docs.goldsky.com/turbo-pipelines/sinks/mysql) | Optional; enables upserts. Omit for plain inserts. |\n| [ClickHouse](https://docs.goldsky.com/turbo-pipelines/sinks/clickhouse) | Required; sets table ordering and deduplication columns. |\n| [Kafka](https://docs.goldsky.com/turbo-pipelines/sinks/kafka) | Optional; selects message-key columns. If omitted, uses the upstream key when available. |\n\nFor other sink types, check their own field reference before adding or removing `primary_key`. Mirror uses different sink schemas; do not copy Turbo sink fields into Mirror configs.\n\n#### PostgreSQL\n\n```yaml\nsinks:\n output:\n type: postgres\n from: my_transform\n schema: public\n table: my_table\n secret_name: MY_POSTGRES_SECRET\n primary_key: id\n```\n\n#### ClickHouse\n\n```yaml\nsinks:\n output:\n type: clickhouse\n from: my_transform\n table: my_table\n secret_name: MY_CLICKHOUSE_SECRET\n primary_key: id\n```\n\n#### Pub/Sub (Turbo-only)\n\n```yaml\nsinks:\n output:\n type: pubsub\n from: my_transform\n topic: my-topic\n secret_name: MY_PUBSUB_SECRET\n```\n\nThe secret holds a GCP project id and service-account JSON. Topic must already exist in GCP.\n\n#### Blackhole (Testing)\n\n```yaml\nsinks:\n output:\n type: blackhole\n from: my_transform\n```\n\n---\n\n## Checkpoint Behavior\n\n- **Preserved by default** when updating a pipeline\n- **Tied to source names** — renaming a source resets its checkpoint\n- **Tied to pipeline names** — renaming the pipeline resets all checkpoints\n\nTo reset checkpoints: rename the source or pipeline. Warning: this reprocesses all historical data.\n\n---\n\n## Starter Templates\n\n| Template | Description | Use Case |\n| ------------------------------ | --------------------------------- | ---------------------------- |\n| `minimal-erc20-blackhole.yaml` | Simplest pipeline, no credentials | Quick testing |\n| `filtered-transfers-sql.yaml` | Filter by contract address | USDC, specific tokens |\n| `postgres-output.yaml` | Write to PostgreSQL | Production data storage |\n| `pubsub-output.yaml` | Write to Google Cloud Pub/Sub | GCP-based event consumers |\n| `multi-chain-pipeline.yaml` | Combine multiple chains | Cross-chain analytics |\n| `solana-transfers.yaml` | Solana SPL tokens | Non-EVM chains |\n| `multi-sink-pipeline.yaml` | Multiple outputs | Archive + alerts + streaming |\n| `linear-pipeline.yaml` | Simple decode → filter → sink | Basic linear flow |\n| `fan-out-pipeline.yaml` | One source → multiple sinks | Multi-destination |\n| `fan-in-pipeline.yaml` | Multiple events → UNION ALL | Activity feeds |\n| `multi-chain-templated.yaml` | Per-chain pipeline pattern | Independent chain deploys |\n\n**Template location:** `templates/` (relative to this skill's directory)\n\n---\n\n## CLI Quick Reference\n\n| Action | Command |\n| ----------------------- | ------------------------------------------------------ |\n| Install Goldsky CLI | Follow `/auth-setup` and use its bundled installer |\n| Install Turbo extension | Follow `/auth-setup` and run its bundled installer with `turbo`; require a successful version check |\n| **Validate (REQUIRED)** | `goldsky turbo validate pipeline.yaml` |\n| Deploy/Update | `goldsky turbo apply pipeline.yaml` |\n| Deploy + Inspect | `goldsky turbo apply pipeline.yaml -i` |\n| List pipelines | `goldsky turbo list` |\n| List datasets | `goldsky dataset list --output json` |\n| List secrets | `goldsky secret list` |\n\nFor lifecycle commands (pause/resume/restart/delete) and monitoring (inspect/logs), see `/turbo-operations`.\n\n---\n\n## Troubleshooting\n\nSee `references/troubleshooting.md` for CLI hanging, validation errors, and runtime errors.\n\n---\n\n## Related\n\n- **`/turbo-builder`** — Interactive wizard to build pipelines step-by-step\n- **`/turbo-doctor`** — Diagnose and fix pipeline issues\n- **`/turbo-operations`** — Lifecycle commands and monitoring reference\n- **`/turbo-transforms`** — SQL, TypeScript, and dynamic table transform reference\n- **`/datasets`** — Dataset names and chain prefixes\n- **`/secrets`** — Sink credential management\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}