{"id":23752,"plugin_id":"plugins_6ab2f25e4928819184294ebadcbe38ab","kind":"skill","collection_source":null,"comparison_source":null,"observed_at":"2026-09-30T23:17:46.606Z","digest":"6cb43eb51088e0f26c74db33598f211c41487c24467b7f6f2658c5b10d2c9c4d","against":null,"payload":{"name":"dag-factory","description":"Authors Apache Airflow DAGs declaratively from dag-factory YAML configs. Use when building DAGs declaratively from YAML via dag-factory; creating/editing dag-factory templates/YAML configs,reating/editing dag-factory YAML configs, defaults, dynamic tasks, datasets, or callbacks; or validating dag-factory configurations; upgrading or re-pinning dag-factory.","included_files":[{"relative_path":"reference/migration.md","size_in_bytes":3086}],"skill_md_contents":"---\nname: dag-factory\ndescription: Authors Apache Airflow DAGs declaratively from dag-factory YAML configs. Use when building DAGs declaratively from YAML via dag-factory; creating/editing dag-factory templates/YAML configs,reating/editing dag-factory YAML configs, defaults, dynamic tasks, datasets, or callbacks; or validating dag-factory configurations; upgrading or re-pinning dag-factory.\n---\n# DAG Factory\n\nYou are helping a user build Apache Airflow DAGs declaratively with **dag-factory**, a library that turns YAML configuration files into Airflow DAGs. Execute steps in order and prefer the simplest configuration that meets the user's needs.\n\n> **Package**: `dag-factory` on PyPI\n> **Repo**: https://github.com/astronomer/dag-factory\n> **Docs**: https://astronomer.github.io/dag-factory/latest/\n> **Targets**: dag-factory **v1.0+** only. For pre-1.0 projects, see [reference/migration.md](reference/migration.md) before applying any guidance from this skill.\n> **Requires**: Python 3.10+, Airflow 2.4+ (Airflow 3 supported)\n\n## Before Starting\n\nConfirm with the user:\n1. **Airflow version** ≥2.4\n2. **Python version** ≥3.10\n3. **dag-factory version**: this skill targets **v1.0+**. If the project is on <1.0, follow [reference/migration.md](reference/migration.md) to upgrade before continuing.\n4. **Use case**: dag-factory is for declarative, low-code DAG authoring. If the user needs reusable, validated Pythonic templates with Pydantic, suggest **blueprint** instead. If they need full Python flexibility, suggest the **authoring-dags** skill.\n\n---\n\n## Determine What the User Needs\n\n| User Request | Action |\n|--------------|--------|\n| \"Create a YAML DAG\" / \"Convert this Python DAG to YAML\" | Go to **Defining a DAG in YAML** |\n| \"Set up dag-factory in my project\" | Go to **Project Setup** |\n| \"Share defaults across DAGs\" / \"Set start_date once\" | Go to **Defaults** |\n| \"Use a custom operator\" / \"Use KPO / Slack / Snowflake\" | Go to **Custom & Provider Operators** |\n| \"Dynamic / mapped tasks\" / \"expand / partial\" | Go to **Dynamic Task Mapping** |\n| \"Schedule on dataset\" / \"Outlets and inlets\" | Go to **Datasets** |\n| \"Add a callback\" / \"Slack on failure\" | Go to **Callbacks** |\n| \"Use a timetable\" / \"datetime in YAML\" / \"timedelta in YAML\" | Go to **Custom Python Objects (`__type__`)** |\n| \"Lint my YAML\" / \"Validate\" | Go to **Validation Commands** |\n| \"Convert Airflow 2 YAML to Airflow 3\" | Go to **Validation Commands** (`dagfactory convert`) |\n| \"Migrate from dag-factory <1.0\" | See [reference/migration.md](reference/migration.md) |\n| dag-factory errors / troubleshooting | Go to **Troubleshooting** |\n\n---\n\n## Project Setup\n\n### 1. Install the Package\n\nAdd to `requirements.txt`:\n\n```\ndag-factory>=1.0.0\n```\n\ndag-factory **does not** install Airflow providers automatically. Install any provider packages your YAML references (e.g., `apache-airflow-providers-slack`, `apache-airflow-providers-cncf-kubernetes`).\n\n### 2. Create the Loader\n\nCreate `dags/load_dags.py` so Airflow's DAG processor will pick it up:\n\n```python\nimport os\nfrom pathlib import Path\n\nfrom dagfactory import load_yaml_dags\n\nCONFIG_ROOT_DIR = Path(os.getenv(\"CONFIG_ROOT_DIR\", \"/usr/local/airflow/dags/\"))\n\n# Option A: load every *.yml / *.yaml under a folder\nload_yaml_dags(globals_dict=globals(), dags_folder=str(CONFIG_ROOT_DIR))\n\n# Option B: load a single file\n# load_yaml_dags(globals_dict=globals(), config_filepath=str(CONFIG_ROOT_DIR / \"my_dag.yml\"))\n\n# Option C: load from an in-Python dict\n# load_yaml_dags(globals_dict=globals(), config_dict={...})\n```\n\n`globals_dict=globals()` is required so generated DAG objects are registered into the module namespace where Airflow can discover them.\n\n### 3. Verify Installation\n\n```bash\ndagfactory --version\n```\n\n---\n\n## Defining a DAG in YAML\n\nEach top-level YAML key (other than `default`) defines a DAG. The key becomes the `dag_id`. **Use the list format for `tasks` and `task_groups`** — it is the recommended format since v1.0.0.\n\n```yaml\n# dags/example_dag_factory.yml\ndefault:\n  default_args:\n    start_date: 2024-11-11\n\nbasic_example_dag:\n  default_args:\n    owner: \"custom_owner\"\n  description: \"this is an example dag\"\n  schedule: \"0 3 * * *\"\n  catchup: false\n  task_groups:\n    - group_name: \"example_task_group\"\n      tooltip: \"this is an example task group\"\n      dependencies: [task_1]\n  tasks:\n    - task_id: \"task_1\"\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo 1\"\n    - task_id: \"task_2\"\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo 2\"\n      dependencies: [task_1]\n    - task_id: \"task_3\"\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo 3\"\n      dependencies: [task_1]\n      task_group_name: \"example_task_group\"\n```\n\n### Key Fields\n\n| Field | Where | Purpose |\n|-------|-------|---------|\n| `default` | top-level | Shared DAG-level args applied to every DAG in this file |\n| `default_args` | DAG or `default` block | Standard Airflow `default_args` (owner, retries, start_date, ...) |\n| `schedule` | DAG | Cron expression, preset (`@daily`), Dataset list, or `__type__` timetable |\n| `catchup` / `description` / `tags` | DAG | Standard Airflow DAG kwargs |\n| `tasks` | DAG | List of task dicts; each requires `task_id` and `operator` |\n| `operator` | task | **Full import path** to operator class (e.g. `airflow.operators.bash.BashOperator`) |\n| `dependencies` | task / task_group | List of upstream `task_id`s or `group_name`s |\n| `task_groups` | DAG | List of group dicts; each requires `group_name` |\n| `task_group_name` | task | Assigns a task to a task group |\n\nTasks do **not** need to be ordered by dependency in the YAML — dag-factory resolves the DAG topology.\n\n### Dictionary Format (Legacy)\n\nPre-1.0 dictionary format (where `tasks` is a dict keyed by `task_id`) still works for backward compatibility, but prefer the list format for new code.\n\n---\n\n## Defaults\n\nThere are four ways to set defaults, in **precedence order** (highest first):\n\n1. `default_args` / DAG-level keys inside an individual DAG\n2. The top-level `default:` block in the same YAML file\n3. `defaults_config_dict=` argument to `load_yaml_dags`\n4. A `defaults.yml` (or `defaults.yaml`) file via `defaults_config_path=` (or auto-detected next to the DAG YAML)\n\n> Note: loader argument names and several other field names changed in v1.0.0. See [reference/migration.md](reference/migration.md) if you're working on an older project.\n\n### `default` Block in the Same File\n\nPowerful for templating multiple DAGs from one file:\n\n```yaml\ndefault:\n  default_args:\n    owner: \"data-team\"\n    start_date: 2025-01-01\n    retries: 2\n  catchup: false\n  schedule: \"@daily\"\n\ndag_one:\n  description: \"first DAG\"\n  tasks:\n    - task_id: t1\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo one\"\n\ndag_two:\n  description: \"second DAG\"\n  tasks:\n    - task_id: t1\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo two\"\n```\n\n### `defaults.yml` File\n\nPlace a `defaults.yml` next to the DAG YAML, or point `defaults_config_path` at a parent directory. dag-factory **merges** all `defaults.yml` files walking up the directory tree, with the file closest to the DAG YAML winning. DAG-level args (e.g. `schedule`, `catchup`) go at the root of `defaults.yml`; per-task defaults go under `default_args`.\n\n```yaml\n# defaults.yml\nschedule: 0 1 * * *\ncatchup: false\ndefault_args:\n  start_date: '2024-12-31'\n  owner: data-team\n```\n\n---\n\n## Custom & Provider Operators\n\nReference any operator by its **full Python import path**. dag-factory passes all other task keys as kwargs to that operator.\n\n```yaml\ntasks:\n  - task_id: begin\n    operator: airflow.providers.standard.operators.empty.EmptyOperator\n  - task_id: make_bread\n    operator: customized.operators.breakfast_operators.MakeBreadOperator\n    bread_type: 'Sourdough'\n```\n\nThe operator's package must be installed and importable. For Airflow 3, prefer `airflow.providers.standard.operators.*` over the legacy `airflow.operators.*` paths — the `dagfactory convert` CLI rewrites these automatically.\n\n### KubernetesPodOperator\n\nSpecify the operator path and pass kwargs directly. As of v1.0, dag-factory no longer does legacy type casting — use `__type__` for nested k8s objects.\n\n```yaml\ntasks:\n  - task_id: hello-world-pod\n    operator: airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator\n    image: \"python:3.12-slim\"\n    cmds: [\"python\", \"-c\"]\n    arguments: [\"print('hi')\"]\n    name: example-pod\n    namespace: default\n    container_resources:\n      __type__: kubernetes.client.models.V1ResourceRequirements\n      limits: {cpu: \"1\", memory: \"1024Mi\"}\n      requests: {cpu: \"0.5\", memory: \"512Mi\"}\n```\n\n---\n\n## Dynamic Task Mapping\n\nUse `expand` and `partial` keys on a task to map dynamically. dag-factory has two distinct ways to reference an upstream task's output:\n\n- **`task_id.output`** — XCom-style reference, used inside `expand` `op_args` / `op_kwargs` (and the equivalent kwargs of other operators).\n- **`+task_id`** — bare value reference, used when the value sits directly under `expand` (e.g. `expand: {number: +numbers_list}`) or as a TaskFlow decorator argument.\n\nDon't mix them: `+request` won't resolve inside `op_args`, and `request.output` won't resolve as a bare `expand` value.\n\n```yaml\ndynamic_task_map:\n  default_args:\n    start_date: 2025-01-01\n  schedule: \"0 3 * * *\"\n  tasks:\n    - task_id: request\n      operator: airflow.providers.standard.operators.python.PythonOperator\n      python_callable_name: make_list\n      python_callable_file: $CONFIG_ROOT_DIR/expand_tasks.py\n    - task_id: process\n      operator: airflow.providers.standard.operators.python.PythonOperator\n      python_callable_name: consume_value\n      python_callable_file: $CONFIG_ROOT_DIR/expand_tasks.py\n      partial:\n        op_kwargs:\n          fixed_param: \"test\"\n      expand:\n        op_args: request.output    # XCom-style — used inside op_args / op_kwargs\n      dependencies: [request]\n```\n\nBare-value form (TaskFlow `decorator` tasks, or any non-`op_args` mapping):\n\n```yaml\ntasks:\n  - task_id: numbers_list\n    decorator: airflow.sdk.definitions.decorators.task\n    python_callable: sample.build_numbers_list\n  - task_id: double_number\n    decorator: airflow.sdk.definitions.decorators.task\n    python_callable: sample.double\n    expand:\n      number: +numbers_list   # + resolves to upstream task `numbers_list`'s XComArg\n```\n\nFor named map indices (Airflow 2.9+), set `map_index_template: \"{{ task.custom_mapping_key }}\"` and have the callable assign `context[\"custom_mapping_key\"]`.\n\n**Tested patterns**: simple mapping, task-generated mapping, repeated mapping, `partial`, multiple-parameter mapping, `map_index_template`.\n**Unsupported / untested**: mapping over task groups, zipping, transforming expanding data.\n\n---\n\n## Datasets\n\nUse `inlets` / `outlets` on tasks to declare dataset producers, and a list of dataset URIs as `schedule` to consume them.\n\n```yaml\nproducer_dag:\n  default_args:\n    start_date: '2024-01-01'\n  schedule: \"0 5 * * *\"\n  catchup: false\n  tasks:\n    - task_id: task_1\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo 1\"\n      outlets: ['s3://bucket_example/raw/dataset1.json']\n\nconsumer_dag:\n  default_args:\n    start_date: '2024-01-01'\n  schedule: ['s3://bucket_example/raw/dataset1.json']\n  catchup: false\n  tasks:\n    - task_id: task_1\n      operator: airflow.operators.bash.BashOperator\n      bash_command: \"echo 'consumer'\"\n```\n\n### Conditional Dataset Scheduling (Airflow 2.9+ / dag-factory 0.22+)\n\nNesting the logical operators `__and__` / `__or__` under `datasets` key.\n\n```yaml\nschedule:\n  datasets:\n    __or__:\n      - __and__:\n          - s3://bucket-cjmm/raw/dataset_custom_1\n          - s3://bucket-cjmm/raw/dataset_custom_2\n      - s3://bucket-cjmm/raw/dataset_custom_3\n```\n\n---\n\n## Callbacks\n\nThree styles, all valid at the DAG, TaskGroup, or Task level (or under `default_args`):\n\n### 1. String pointing to a callable\n\n```yaml\n- task_id: task_1\n  operator: airflow.operators.bash.BashOperator\n  bash_command: \"echo task_1\"\n  on_failure_callback: include.custom_callbacks.output_standard_message\n```\n\nWith kwargs:\n\n```yaml\n- task_id: task_2\n  operator: airflow.operators.bash.BashOperator\n  bash_command: \"echo task_2\"\n  on_success_callback:\n    callback: include.custom_callbacks.output_custom_message\n    param1: \"Task status\"\n    param2: \"Successful!\"\n```\n\n### 2. File path + function name (no kwargs)\n\n```yaml\n- task_id: task_3\n  operator: airflow.operators.bash.BashOperator\n  bash_command: \"echo task_3\"\n  on_retry_callback_name: output_standard_message\n  on_retry_callback_file: /usr/local/airflow/include/custom_callbacks.py\n```\n\n### 3. Provider callbacks\n\n```yaml\n- task_id: task_4\n  operator: airflow.operators.bash.BashOperator\n  bash_command: \"echo task_4\"\n  on_failure_callback:\n    callback: airflow.providers.slack.notifications.slack.send_slack_notification\n    slack_conn_id: slack_conn_id\n    text: \":red_circle: Task Failed.\"\n    channel: \"#channel\"\n```\n\nThe provider package must be installed.\n\n---\n\n## Custom Python Objects (`__type__`)\n\nFor anything that isn't a simple scalar — `datetime`, `timedelta`, `Asset`, timetables, k8s objects — use the generalized object syntax:\n\n```yaml\nstart_date:\n  __type__: datetime.datetime\n  year: 2025\n  month: 1\n  day: 1\n\nexecution_timeout:\n  __type__: datetime.timedelta\n  hours: 1\n\nschedule:\n  __type__: airflow.timetables.trigger.CronTriggerTimetable\n  cron: \"0 1 * * 3\"\n  timezone: UTC\n```\n\n- `__type__` is the **full import path** to the class\n- `__args__` is a list of positional arguments\n- Other keys become keyword arguments\n- For lists of typed objects, use `__type__: builtins.list` with an `items:` key\n\n### Reserved Keys\n\nDon't use these YAML keys for your own data — dag-factory reserves them: `__type__`, `__args__`, `__join__`, `__and__`, `__or__`. The key `items` is also reserved when used inside a `__type__: builtins.list` block — don't add a custom field named `items` to a typed list construction.\n\n---\n\n## Validation Commands\n\nAfter installing, the `dagfactory` CLI is on PATH:\n\n| Command | When to Use |\n|---------|-------------|\n| `dagfactory --version` | Confirm install / version |\n| `dagfactory lint <path>` | Validate YAML syntax for a file or directory |\n| `dagfactory lint <path> --verbose` | Show a per-file table of results |\n| `dagfactory convert <path>` | Show diffs to migrate Airflow 2 → 3 import paths |\n| `dagfactory convert <path> --override` | Apply the conversions in place |\n\n### Validation Workflow\n\n```bash\n# 1. Lint YAML\ndagfactory lint dags/\n\n# 2. Have Airflow parse to catch operator/import errors\n#    (Astro CLI users)\nastro dev parse\n```\n\n`dagfactory lint` only checks YAML syntax — operator import errors and missing kwargs surface at Airflow parse time.\n\n---\n\n## Troubleshooting\n\n### \"Operator not found\" / `ModuleNotFoundError`\n\n**Cause**: Provider package not installed, or wrong import path.\n\n**Fix**: Install the provider (`pip install apache-airflow-providers-...`) and verify the path. For Airflow 3, run `dagfactory convert` to update legacy `airflow.operators.*` paths to `airflow.providers.standard.operators.*`.\n\n### YAML parses but the DAG doesn't appear in Airflow\n\n**Cause**: Loader file missing or `globals_dict=globals()` not passed.\n\n**Fix**: Ensure a Python file in `dags/` calls `load_yaml_dags(globals_dict=globals(), ...)`. Check `astro dev parse` (or `airflow dags list-import-errors`) for parse errors.\n\n### \"Argument is not JSON-serializable\" / wrong kwarg type\n\n**Cause**: A scalar string is being passed where a Python object is expected (e.g. `start_date: \"2025-01-01\"` for a field that needs `datetime`).\n\n**Fix**: Use `__type__: datetime.datetime` (or `datetime.timedelta` etc.) per **Custom Python Objects**.\n\n### Conditional dataset schedule ignored\n\n**Cause**: Airflow <2.9, dag-factory <0.22, or using legacy `!and`/`!or` keys.\n\n**Fix**: Upgrade and rename to `__and__` / `__or__`.\n\n### Multiple `defaults.yml` not merging as expected\n\n**Cause**: `defaults_config_path` not pointing at a parent directory of the DAG YAML.\n\n**Fix**: Set `defaults_config_path` to the highest ancestor folder you want included; dag-factory walks the tree from DAG file → ancestor and merges in that order, with files closer to the DAG winning.\n\n---\n\n## Verification Checklist\n\nBefore finishing, verify with the user:\n\n- [ ] `dagfactory lint dags/` passes\n- [ ] Loader file exists in `dags/` and calls `load_yaml_dags(globals_dict=globals(), ...)`\n- [ ] Required Airflow providers are in `requirements.txt`\n- [ ] DAG appears in Airflow UI without import errors\n\n---\n\n## Related Skills\n\n- **authoring-dags** — Writing Airflow DAGs in pure Python with `af` CLI validation. Use when YAML can't express what you need.\n- **testing-dags**: For testing DAGs, debugging failures, and the test -> fix -> retest loop\n- **debugging-dags**: For troubleshooting failed DAGs\n\n## Reference\n\n- GitHub: https://github.com/astronomer/dag-factory\n- Docs: https://astronomer.github.io/dag-factory/latest/\n- PyPI: https://pypi.org/project/dag-factory/\n- Migration Guide: https://astronomer.github.io/dag-factory/latest/migration_guide/\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}