{"id":83627,"plugin_id":"plugin_asdk_app_6ad6851613bc819189dbdda8700e1094","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-11-06T00:15:47.385Z","digest":"55baf39e67a268f03858272bbab87da33f151a692ac81a1d1360e4e7a66b0e4e","against":null,"payload":{"description":"Write SQL, TypeScript, and dynamic table transforms for Turbo pipelines. Covers: decoding EVM logs with _gs_log_decode, filtering/casting blockchain data, UNION ALL for combining events, TypeScript/WASM transforms (invoke function), dynamic lookup tables (dynamic_table_check), transform chaining, and Solana decoding. Triggers on: 'decode Transfer events', 'write a SQL transform', 'filter by contract', 'TypeScript transform', 'dynamic table', 'UNION ALL'. For pipeline YAML structure, use /turbo-pipelines. For end-to-end building, use /turbo-builder.","included_files":[{"relative_path":"evals/trigger-eval.json","size_in_bytes":1789},{"relative_path":"references/dynamic-tables.md","size_in_bytes":8492},{"relative_path":"references/evm-patterns.md","size_in_bytes":6029},{"relative_path":"references/solana-patterns.md","size_in_bytes":8942},{"relative_path":"references/typescript-transforms.md","size_in_bytes":7649},{"relative_path":"templates/multi-event-activity-feed.yaml","size_in_bytes":2831},{"relative_path":"templates/throttled-webhook.yaml","size_in_bytes":893}],"name":"turbo-transforms","skill_md_contents":"---\nname: turbo-transforms\ndescription: \"Write SQL, TypeScript, and dynamic table transforms for Turbo pipelines. Covers: decoding EVM logs with _gs_log_decode, filtering/casting blockchain data, UNION ALL for combining events, TypeScript/WASM transforms (invoke function), dynamic lookup tables (dynamic_table_check), transform chaining, and Solana decoding. Triggers on: 'decode Transfer events', 'write a SQL transform', 'filter by contract', 'TypeScript transform', 'dynamic table', 'UNION ALL'. For pipeline YAML structure, use /turbo-pipelines. For end-to-end building, use /turbo-builder.\"\n---\n\n# Turbo Transforms\n\nWrite, understand, and debug SQL, TypeScript, and dynamic table transforms for Turbo pipeline configurations.\n\nIdentify what the user needs (decode, filter, reshape, combine, custom logic, or lookup joins), then use the relevant section below. If generating a complete pipeline YAML (not just a transform snippet), always validate with `goldsky turbo validate` before presenting it to the user.\n\n**Reference files for specialized topics:**\n- `references/evm-patterns.md` — Advanced EVM patterns (decode-once-filter-many, UNION ALL, schema normalization)\n- `references/typescript-transforms.md` — TypeScript/WASM script transforms and handler transforms\n- `references/dynamic-tables.md` — Dynamic table transforms (allowlists, lookup joins)\n- `references/solana-patterns.md` — Solana instruction/log decoding and function examples\n\n---\n\n## Transform Basics\n\n### YAML Structure\n\n```yaml\ntransforms:\n  my_transform:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT id, block_number, address\n      FROM my_source\n      WHERE address = '0xabc...'\n```\n\n### Required Fields\n\n| Field         | Required | Description                                                  |\n| ------------- | -------- | ------------------------------------------------------------ |\n| `type`        | Yes      | `sql`, `script`, `handler`, `dynamic_table`, or `throttle`   |\n| `primary_key` | Yes\\*    | Column used for uniqueness and ordering (\\*not for `throttle`) |\n| `sql`         | Yes      | SQL query (for `sql` type transforms)                        |\n\n### Referencing Data\n\n- Reference **sources** by their YAML key name: `FROM my_source`\n- Reference **other transforms** by their YAML key name: `FROM my_transform`\n- No need for a `from` field in SQL transforms — the `FROM` clause in SQL handles it\n\n### SQL Streaming Limitations\n\nTurbo SQL is powered by Apache DataFusion in streaming mode. The following are **NOT supported**:\n\n- **Joins** — use `dynamic_table` transforms for lookup-style joins instead\n- **Aggregations** (GROUP BY, COUNT, SUM, AVG) — use `postgres_aggregate` sink instead\n- **Window functions** (OVER, PARTITION BY, ROW_NUMBER)\n- **Subqueries** — use transform chaining instead\n- **CTEs** (WITH...AS) **are** supported\n\n### The `_gs_op` Column\n\nEvery record includes a `_gs_op` column that tracks the operation type: `'i'` (insert), `'u'` (update), `'d'` (delete). Preserve this column through transforms for correct upsert semantics in database sinks.\n\n---\n\n## Key SQL Functions\n\nQuick reference for the most-used functions. For the complete function reference, see [SQL transforms](https://docs.goldsky.com/turbo-pipelines/transforms/sql) in the docs.\n\n### `evm_log_decode()` — Decode Raw EVM Logs\n\n```sql\nevm_log_decode(abi_json, topics, data) -> STRUCT<name, event_params>\n-- Aliases: _gs_log_decode, evm_decode_log\n```\n\nReturns `decoded.event_signature` (event name) and `decoded.event_params[N]` (1-indexed parameters as strings).\n\n**Example:**\n```sql\nSELECT\n  _gs_log_decode('[{\"anonymous\":false,\"inputs\":[{\"indexed\":true,\"name\":\"from\",\"type\":\"address\"},{\"indexed\":true,\"name\":\"to\",\"type\":\"address\"},{\"indexed\":false,\"name\":\"value\",\"type\":\"uint256\"}],\"name\":\"Transfer\",\"type\":\"event\"}]',\n    topics, data) AS decoded,\n  id, block_number, transaction_hash, address, block_timestamp\nFROM my_raw_logs\n```\n\n**Tips:** Backtick-escape reserved words (`` `data` ``, `` `decoded` ``). Include all event ABIs in one array. Pre-filter by contract address at the source level for efficiency.\n\n### Other Key Functions\n\n| Function | Purpose |\n| -------- | ------- |\n| `fetch_abi(url, format)` | Fetch ABI from URL (`'raw'` or `'etherscan'` format) |\n| `_gs_keccak256(string)` | Keccak256 hash (returns hex with `0x` prefix) |\n| `xxhash(string)` | Fast non-cryptographic hash for composite keys |\n| `to_u256(value)` / `u256_to_string(u256)` | 256-bit integer math (avoids precision loss) |\n| `dynamic_table_check(table, value)` | Check if value exists in a dynamic lookup table |\n| `to_timestamp(seconds)` | Convert unix seconds to timestamp |\n\n---\n\n## Common Transform Patterns\n\n### Table Aliasing\n\n```sql\nSELECT balance.token_type, balance.owner_address,\n  CAST(`balance` AS STRING) AS balance_amount\nFROM my_balances_source balance\nWHERE balance.token_type = 'ERC_20' AND balance.balance IS NOT NULL\n```\n\n### Simple Filtering\n\n```yaml\ntransforms:\n  usdc_transfers:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT * FROM base_transfers\n      WHERE address = lower('0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913')\n```\n\n### Column Projection and Aliasing\n\n```yaml\ntransforms:\n  clean_transfers:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT id, block_number, to_timestamp(block_timestamp) AS block_time,\n        address AS token_address, sender, recipient, amount\n      FROM my_source\n```\n\n### Type Casting and Numeric Scaling\n\n```sql\n-- Token amount conversions\n(CAST(decoded.event_params[3] AS DOUBLE) / 1e6) AS amount_usdc    -- 6 decimals\n(CAST(decoded.event_params[3] AS DOUBLE) / 1e18) AS amount_eth    -- 18 decimals\nCAST(`balance` AS STRING) AS balance_amount                        -- preserve precision\n```\n\n| Token          | Decimals | Divisor |\n| -------------- | -------- | ------- |\n| USDC, USDT     | 6        | `1e6`   |\n| WBTC           | 8        | `1e8`   |\n| ETH, most ERC-20 | 18    | `1e18`  |\n\n### Conditional Logic with CASE WHEN\n\n```sql\nCASE WHEN decoded.event_params[4] = '0' THEN 'BUY' ELSE 'SELL' END AS side\n```\n\n### Exclusion Filters\n\n```sql\nWHERE decoded.event_params[1] NOT IN ('0x4bfb...', '0xc5d5...', '0x4d97...')\n```\n\n### Transform Chaining\n\nBuild multi-step pipelines where each transform reads from the previous:\n\n```yaml\ntransforms:\n  decoded:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT _gs_log_decode('[...]', topics, data) AS decoded,\n        id, block_number, transaction_hash, address, block_timestamp\n      FROM raw_logs_source\n  transfers:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT id, block_number, decoded.event_params[1] AS from_address,\n        decoded.event_params[2] AS to_address,\n        (CAST(decoded.event_params[3] AS DOUBLE) / 1e18) AS amount\n      FROM decoded\n      WHERE decoded.event_signature = 'Transfer'\n```\n\n### UNION ALL — Combining Multiple Event Types\n\nCombine transforms with identical schemas into a single output:\n\n```yaml\n  all_events:\n    type: sql\n    primary_key: id\n    sql: |\n      SELECT * FROM transfers\n      UNION ALL\n      SELECT * FROM approvals\n```\n\nAll branches must have the same columns in the same order. Use `''` or `0` as placeholders.\n\nFor advanced EVM patterns (decode-once-filter-many, normalizing disparate events, multi-event activity feeds), read `references/evm-patterns.md`.\n\n---\n\n## Source-Level Filtering\n\nFilter data **at the source** before it reaches transforms for efficiency:\n\n```yaml\nsources:\n  my_logs:\n    type: dataset\n    dataset_name: matic.raw_logs\n    version: 1.0.0\n    start_at: earliest\n    filter: >-\n      address IN ('0xabc...', '0xdef...')\n      AND block_number >= 82422949\n```\n\n- Source `filter:` → coarse pre-filtering (contract addresses, block ranges)\n- Transform `WHERE` → fine-grained filtering (event types, parameter values, exclusions)\n\n---\n\n## Sink Batching Configuration\n\n```yaml\nsinks:\n  my_sink:\n    type: clickhouse\n    from: my_transform\n    table: my_table\n    secret_name: MY_SECRET\n    primary_key: id\n    batch_size: 1000\n    batch_flush_interval: 300ms\n```\n\n| Use Case | `batch_flush_interval` | `batch_size` |\n| -------- | ---------------------- | ------------ |\n| Real-time dashboards | `300ms` | `1000` |\n| Moderate throughput | `1000ms` | `1000` |\n| High-volume streaming | `10s` | `100000` |\n\n---\n\n## Debugging Transforms\n\n**\"Unknown source reference\"** — `FROM` clause name doesn't match any source/transform key. Check for typos.\n\n**\"Missing primary_key\"** — Every transform needs `primary_key`. Almost always use `id`.\n\n**\"Column not found\"** — Use `goldsky turbo inspect <pipeline> -n <source_node> -p` to see actual columns.\n\n**Empty results from decoded events:**\n- Verify ABI JSON matches actual contract events\n- Check `decoded.event_signature` matches event name exactly (case-sensitive)\n- Ensure `address` filter matches correct contract\n- Parameters are **1-indexed**: `decoded.event_params[1]`, not `[0]`\n\n**Type mismatch in UNION ALL** — All branches need identical column counts and compatible types.\n\n```bash\ngoldsky turbo inspect <pipeline-name> -n <transform_name> -p\n```\n\n---\n\n## TypeScript / WASM Script Transforms\n\nFor logic SQL can't express: custom parsing, BigInt arithmetic, stateful processing. Schema types: `string`, `uint64`, `int64`, `float64`, `boolean`, `bytes`.\n\nKey rules: define `function invoke(data)`, return `null` to filter, return object matching schema, no async/await or external imports.\n\nSee `references/typescript-transforms.md` for full docs, examples, and the SQL-vs-script decision table. Also includes Handler transforms for HTTP enrichment.\n\n---\n\n## Dynamic Table Transforms\n\nUpdatable lookup tables for allowlists, blocklists, and join-style enrichment — without redeploying. Backed by PostgreSQL (durable) or in-memory (fast). Use `dynamic_table_check('table_name', column)` in SQL.\n\nSee `references/dynamic-tables.md` for full config, backend options, REST API updates, and wallet-tracking example.\n\n---\n\n## Throttle Transform (Flow Control)\n\nCaps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Throttle does **not** modify data — every input record passes through unchanged. Use it to:\n\n- Stay under rate limits of downstream sinks or external APIs\n- Smooth bursty sources into a steady, predictable rate\n- Pace records into an HTTP `handler` so the receiving service is not overwhelmed\n- Slow down processing during development or testing\n\n```yaml\ntransforms:\n  throttled:\n    type: throttle\n    from: my_source       # source or transform\n    max_batch_size: 100\n    min_batch_interval: 10s\n```\n\n| Field                | Required | Description                                              |\n| -------------------- | -------- | -------------------------------------------------------- |\n| `type`               | Yes      | `throttle`                                               |\n| `from`               | Yes      | Source or transform to read from                         |\n| `max_batch_size`     | No       | Max records per batch                                    |\n| `min_batch_interval` | No       | Minimum time between batches (e.g. `10s`, `500ms`, `1m`) |\n\nA batch is flushed downstream when **both** `max_batch_size` records have accumulated **and** `min_batch_interval` has elapsed since the previous batch. Effective max throughput ≈ `max_batch_size / min_batch_interval` records per second.\n\n**Best practices:**\n\n- Place throttle close to the bottleneck (just before the rate-limited sink/handler) so upstream transforms still process at full speed.\n- Tune `max_batch_size` to match the downstream sink's preferred batch size.\n- Throttle is a `transform` — chain other transforms downstream from it the same way you would any other transform (e.g. `from: throttled` in a sink).\n- Throttle does not have a `primary_key` field (records pass through unchanged).\n- Remove throttle in production where possible; it caps throughput by design.\n\n## Solana Transform Patterns\n\nIDL-based instruction decoding, program-specific decoders (Token, System, Stake, Vote), SPL token tracking.\n\nSee `references/solana-patterns.md` for all built-in decoders, full example pipeline, and array/JSON/hex function usage.\n\n---\n\n## Related\n\n- **`/turbo-builder`** — Build and deploy pipelines interactively using these transforms\n- **`/turbo-doctor`** — Diagnose and fix pipeline issues (including transform errors)\n- **`/turbo-pipelines`** — Pipeline YAML configuration and architecture reference\n- **`/turbo-operations`** — Lifecycle commands and monitoring reference\n- **`/datasets`** — Blockchain dataset and chain prefix reference\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}