Update to astronomer-data

{
  "name": "configuring-airflow-language-sdks",
  "description": "Configures Airflow to run language SDK tasks (Java, Go, and future native SDKs) — register a coordinator, map a queue to it, ensure the runtime/artifact on workers, and tune coordinator options. Use when the user wants Airflow to route a queue to a native-language coordinator, asks about the `[sdk]` `coordinators`/`queue_to_coordinator` settings, `AIRFLOW__SDK__COORDINATORS`, `jars_root`, `executables_root` or other coordinator `kwargs`, `task_startup_timeout`, or why their native tasks aren't being picked up. Covers the shared routing mechanism plus per-coordinator options (e.g. JavaCoordinator, ExecutableCoordinator).",
  "included_files": [],
  "skill_md_contents": "---\nname: configuring-airflow-language-sdks\ndescription: Configures Airflow to run language SDK tasks (Java, Go, and future native SDKs) — register a coordinator, map a queue to it, ensure the runtime/artifact on workers, and tune coordinator options. Use when the user wants Airflow to route a queue to a native-language coordinator, asks about the `[sdk]` `coordinators`/`queue_to_coordinator` settings, `AIRFLOW__SDK__COORDINATORS`, `jars_root`, `executables_root` or other coordinator `kwargs`, `task_startup_timeout`, or why their native tasks aren't being picked up. Covers the shared routing mechanism plus per-coordinator options (e.g. JavaCoordinator, ExecutableCoordinator).\n---\n\n# Configuring Airflow for Language SDKs\n\nTo run language SDK tasks, Airflow needs to know two things: which **coordinator** launches the native subprocess, and which **queue** routes to that coordinator. The mechanism is identical across every language SDK — only each coordinator's `classpath` and `kwargs` differ. This skill documents the shared wiring once, then the per-coordinator options. It is platform-neutral: the same settings apply on open-source Airflow and on managed platforms like Astro.\n\n> **Experimental.** The language SDKs are in preview; configuration keys may change.\n\n> For the task code, see **authoring-language-sdk-tasks** (and the per-language authoring skill, e.g. **authoring-java-sdk-tasks**, **authoring-go-sdk-tasks**). For building and shipping the artifact, see the per-language deploy skill (e.g. **deploying-java-sdk-bundles**, **deploying-go-sdk-bundles**).\n\n---\n\n## Prerequisites on the worker\n\n- The **runtime or artifact the SDK needs** must be present on the worker nodes, because the coordinator spawns a native subprocess per task instance. The exact requirement is per-SDK — see [Per-coordinator options](#per-coordinator-options) (the Java SDK needs a **JRE 17+**; the Go SDK needs no language runtime — the bundle is a self-contained native executable, but it must be built for the worker's OS/arch).\n- The compiled/native **artifact(s)** must be reachable on the worker. See the per-language deploy skill.\n- The coordinators ship with the Airflow Task SDK (`apache-airflow-task-sdk`, installed with Airflow). **No extra Python package is required.**\n\n---\n\n## The two settings\n\nBoth live in the `[sdk]` configuration section and apply to every language SDK:\n\n1. **`coordinators`** — a JSON object mapping a coordinator *name* you choose to its implementation (`classpath`) and constructor `kwargs`.\n2. **`queue_to_coordinator`** — a JSON object mapping a task *queue* to a coordinator name.\n\nA task whose stub sets `queue=\"...\"` is handed to the named coordinator, which launches the native subprocess. The coordinator name is arbitrary — it just has to be the same string in both settings. The queue name must match the `queue=` set on the Python `@task.stub`.\n\n### Option A: `airflow.cfg`\n\n```ini\n[sdk]\ncoordinators = {\n  \"java-jdk17\": {\n    \"classpath\": \"airflow.sdk.coordinators.java.JavaCoordinator\",\n    \"kwargs\": {\"jars_root\": [\"/opt/airflow/jars\"]}\n  },\n  \"go\": {\n    \"classpath\": \"airflow.sdk.coordinators.executable.ExecutableCoordinator\",\n    \"kwargs\": {\"executables_root\": [\"/opt/airflow/executable-bundles\"]}\n  }\n}\nqueue_to_coordinator = {\"java\": \"java-jdk17\", \"golang\": \"go\"}\n```\n\n### Option B: environment variables\n\nEach value must be **valid one-line JSON**. This form is convenient for containers, `.env` files, Docker Compose, and Helm.\n\n```bash\nexport AIRFLOW__SDK__COORDINATORS='{\"java-jdk17\": {\"classpath\": \"airflow.sdk.coordinators.java.JavaCoordinator\", \"kwargs\": {\"jars_root\": [\"/opt/airflow/jars\"]}}, \"go\": {\"classpath\": \"airflow.sdk.coordinators.executable.ExecutableCoordinator\", \"kwargs\": {\"executables_root\": [\"/opt/airflow/executable-bundles\"]}}}'\nexport AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{\"java\": \"java-jdk17\", \"golang\": \"go\"}'\n```\n\nThe examples above register **multiple coordinators** at once (one per language) and map a different queue to each — register only the ones you use.\n\n---\n\n## Per-coordinator options\n\nThe `classpath` and `kwargs` are specific to each coordinator. Add a subsection here as new language SDKs land.\n\n### JavaCoordinator\n\n- **`classpath`**: `airflow.sdk.coordinators.java.JavaCoordinator`\n- **Worker runtime**: JRE 17+ (`java` on `PATH`, or set `java_executable`).\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `jars_root` | *(required)* | One or more directories scanned **recursively** for `.jar` files. Accepts a string or a list of strings/paths. The classpath is assembled automatically. |\n| `java_executable` | `\"java\"` | Path to the `java` binary. Defaults to `java` on `$PATH`. |\n| `jvm_args` | `[]` | Extra JVM arguments, e.g. `[\"-Xmx1g\", \"-Dsome.property=value\"]`. |\n| `main_class` | *(auto-detect)* | Explicit entry-point class. If omitted, the coordinator scans `jars_root` for a JAR whose manifest declares `Main-Class`. **Set this explicitly if multiple executable JARs are present** — otherwise the choice is non-deterministic. |\n| `task_startup_timeout` | `10.0` | Seconds to wait for the subprocess to connect after launch. Increase it if JVM startup is slow (constrained hardware, large classpath, first cold start). |\n\n**Java logging via `java.util.logging`.** Of the SDK logging integrations, only JPL and SLF4J are zero-config build dependencies; Log4j 2 and JUL need extra setup — see the logging integration section in **deploying-java-sdk-bundles**. JUL's documented alternative to calling `AirflowJulHandler.setup()` in `main()` is a `logging.properties` file, wired through `jvm_args`:\n\n```ini\n[sdk]\ncoordinators = {\n  \"java-jdk17\": {\n    \"classpath\": \"airflow.sdk.coordinators.java.JavaCoordinator\",\n    \"kwargs\": {\n      \"jars_root\": [\"/opt/airflow/jars\"],\n      \"jvm_args\": [\"-Djava.util.logging.config.file=/opt/airflow/logging.properties\"]\n    }\n  }\n}\n```\n\n### ExecutableCoordinator (Go and other self-contained-executable SDKs)\n\n- **`classpath`**: `airflow.sdk.coordinators.executable.ExecutableCoordinator`\n- **Worker runtime**: none beyond the bundle itself. The bundle is a self-contained native executable (AFBNDL01), so it needs no language runtime, but it must be built for the worker's OS/arch (a mismatch fails with `exec format error`).\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `executables_root` | *(required)* | One or more directories scanned **recursively** for executable bundles (AFBNDL01-trailered native binaries). Accepts a string or a list of strings/paths. Bundles are identified by the trailer magic, not by filename. The coordinator matches an incoming `dag_id` against each bundle's embedded manifest and verifies its integrity hash before launching. |\n| `task_startup_timeout` | `10.0` | Seconds to wait for the subprocess to connect after launch. Increase it if bundle startup is slow (constrained hardware, first cold start). |\n\n*(Future coordinators — for other languages — will list their own `classpath`, runtime, and `kwargs` here.)*\n\n---\n\n## Verifying the configuration\n\n1. Confirm the runtime/artifact is usable where workers run — for the Java SDK, `java -version` via `astro dev bash` or `docker compose exec ...`; for the Go SDK, the packed bundle exists and matches the worker's OS/arch.\n2. Confirm the artifact directory referenced in `kwargs` (e.g. `jars_root`, `executables_root`) actually contains your artifact on the worker filesystem.\n3. Trigger the DAG and open the native task's logs — you should see the subprocess start and your task output.\n\n### Troubleshooting\n\n| Symptom | Likely cause / fix |\n|---------|--------------------|\n| Task fails immediately mentioning coordinator or queue | `coordinators` / `queue_to_coordinator` not valid one-line JSON, or the queue name doesn't match the stub's `queue=`. Fix the JSON and restart. |\n| Runtime not found (e.g. `java: command not found`) | The language runtime isn't on the worker, or the executable path kwarg is wrong. Install the runtime and verify its version. |\n| \"No artifact found\" / \"no DAGs\" / \"no bundle contains dag_id\" | The artifact-directory kwarg points at the wrong place, the artifact isn't there yet, or its `dag_id` doesn't match the stub. Confirm the path and the IDs. |\n| Wrong/ambiguous entry point (Java) | Multiple executable JARs under `jars_root`. Set `main_class` explicitly. |\n| Go bundle is skipped silently | Not a valid AFBNDL01 bundle, or its integrity hash failed (re-pack after any strip/sign/rebuild). |\n| `exec format error` on the Go bundle | Built for a different OS/arch than the worker. Cross-compile with `--goos`/`--goarch` (see **deploying-go-sdk-bundles**). |\n| DAG run hangs at the native task | Raise `task_startup_timeout` (e.g. `30.0`); first-run subprocess startup can be slow. |\n\n---\n\n## Related Skills\n\n- **authoring-language-sdk-tasks**: The shared Python-stub pattern and conceptual model.\n- **authoring-java-sdk-tasks**: Java task code and matching Python stubs.\n- **deploying-java-sdk-bundles**: Build the bundle and put the artifact where the coordinator scans.\n- **authoring-go-sdk-tasks**: Go task code and matching Python stubs.\n- **deploying-go-sdk-bundles**: Build/pack the Go bundle and place it where the coordinator scans.\n- **deploying-airflow**: General deployment of Airflow on Astro, Docker Compose, or Kubernetes.\n"
}