{"id":23749,"plugin_id":"plugins_6ab2f25e4928819184294ebadcbe38ab","kind":"skill","collection_source":null,"comparison_source":null,"observed_at":"2026-09-30T23:17:46.489Z","digest":"bb100dff70531a9ab7458eeac9f4c51d6aab61ffc4b7fa9ca1d663ff282afec0","against":null,"payload":{"name":"cosmos-dbt-core","description":"Turns a dbt Core project into an Airflow DAG/TaskGroup using Astronomer Cosmos. Use turning a dbt Core project into an Airflow DAG or TaskGroup with Astronomer Cosmos. Before implementing, verify dbt engine, warehouse, Airflow version, execution environment, DAG vs TaskGroup, and manifest availability.","included_files":[{"relative_path":"reference/cosmos-config.md","size_in_bytes":13145}],"skill_md_contents":"---\nname: cosmos-dbt-core\ndescription: Turns a dbt Core project into an Airflow DAG/TaskGroup using Astronomer Cosmos. Use turning a dbt Core project into an Airflow DAG or TaskGroup with Astronomer Cosmos. Before implementing, verify dbt engine, warehouse, Airflow version, execution environment, DAG vs TaskGroup, and manifest availability.\n---\n\n# Cosmos + dbt Core: Implementation Checklist\n\nExecute steps in order. Prefer the simplest configuration that meets the user's constraints.\n\n> **Version note**: This skill targets Cosmos 1.11+ and Airflow 3.x. If the user is on Airflow 2.x, adjust imports accordingly (see Appendix A).\n>\n> **Reference**: Latest stable: https://pypi.org/project/astronomer-cosmos/\n\n> **Before starting**, confirm: (1) dbt engine = Core (not Fusion → use **cosmos-dbt-fusion**), (2) warehouse type, (3) Airflow version, (4) execution environment (Airflow env / venv / container), (5) DbtDag vs DbtTaskGroup vs individual operators, (6) manifest availability.\n\n---\n\n## 1. Configure Project (ProjectConfig)\n\n| Approach | When to use | Required param |\n|----------|-------------|----------------|\n| Project path | Files available locally | `dbt_project_path` |\n| Manifest only | `dbt_manifest` load | `manifest_path` + `project_name` |\n\n```python\nfrom cosmos import ProjectConfig\n\n_project_config = ProjectConfig(\n    dbt_project_path=\"/path/to/dbt/project\",\n    # manifest_path=\"/path/to/manifest.json\",  # for dbt_manifest load mode\n    # project_name=\"my_project\",  # if using manifest_path without dbt_project_path\n    # install_dbt_deps=False,  # if deps precomputed in CI\n)\n```\n\n## 2. Choose Parsing Strategy (RenderConfig)\n\nPick ONE load mode based on constraints:\n\n| Load mode | When to use | Required inputs | Constraints |\n|-----------|-------------|-----------------|-------------|\n| `dbt_manifest` | Large projects; containerized execution; fastest | `ProjectConfig.manifest_path` | Remote manifest needs `manifest_conn_id` |\n| `dbt_ls` | Complex selectors; need dbt-native selection | dbt installed OR `dbt_executable_path` | Can also be used with containerized execution |\n| `dbt_ls_file` | dbt_ls selection without running dbt_ls every parse | `RenderConfig.dbt_ls_path` | `select`/`exclude` won't work |\n| `automatic` (default) | Simple setups; let Cosmos pick | (none) | Falls back: manifest → dbt_ls → custom |\n\n> **CRITICAL**: Containerized execution (`DOCKER`/`KUBERNETES`/etc.)\n\n```python\nfrom cosmos import RenderConfig, LoadMode\n\n_render_config = RenderConfig(\n    load_method=LoadMode.DBT_MANIFEST,  # or DBT_LS, DBT_LS_FILE, AUTOMATIC\n)\n```\n\n---\n\n## 3. Choose Execution Mode (ExecutionConfig)\n\n> **Reference**: See **[reference/cosmos-config.md](reference/cosmos-config.md#execution-modes-executionconfig)** for detailed configuration examples per mode.\n\nPick ONE execution mode:\n\n| Execution mode | When to use | Speed | Required setup |\n|----------------|-------------|-------|----------------|\n| `WATCHER` | Fastest; single `dbt build` visibility | Fastest | dbt adapter in env OR `dbt_executable_path` or dbt Fusion |\n| `WATCHER_KUBERNETES` | Fastest isolated method; single `dbt build` visibility | Fast | dbt installed in container |\n| `LOCAL` + `DBT_RUNNER` | dbt + adapter in the same Python installation as Airflow | Fast | dbt 1.5+ in `requirements.txt` |\n| `LOCAL` + `SUBPROCESS` | dbt + adapter available in the Airflow deployment, in an isolated Python installation | Medium | `dbt_executable_path` |\n| `AIRFLOW_ASYNC` | BigQuery + long-running transforms | Fast | Airflow ≥2.8; provider deps |\n| `KUBERNETES` | Isolation between Airflow and dbt | Medium | Airflow ≥2.8; provider deps |\n| `VIRTUALENV` | Can't modify image; runtime venv | Slower | `py_requirements` in operator_args |\n| Other containerized approaches | Support Airflow and dbt isolation | Medium | container config |\n\n```python\nfrom cosmos import ExecutionConfig, ExecutionMode\n\n_execution_config = ExecutionConfig(\n    execution_mode=ExecutionMode.WATCHER,  # or LOCAL, VIRTUALENV, AIRFLOW_ASYNC, KUBERNETES, etc.\n)\n```\n\n---\n\n## 4. Configure Warehouse Connection (ProfileConfig)\n\n> **Reference**: See **[reference/cosmos-config.md](reference/cosmos-config.md#profileconfig-warehouse-connection)** for detailed ProfileConfig options and all ProfileMapping classes.\n\n### Option A: Airflow Connection + ProfileMapping (Recommended)\n\n```python\nfrom cosmos import ProfileConfig\nfrom cosmos.profiles import SnowflakeUserPasswordProfileMapping\n\n_profile_config = ProfileConfig(\n    profile_name=\"default\",\n    target_name=\"dev\",\n    profile_mapping=SnowflakeUserPasswordProfileMapping(\n        conn_id=\"snowflake_default\",\n        profile_args={\"schema\": \"my_schema\"},\n    ),\n)\n```\n\n### Option B: Existing profiles.yml\n\n> **CRITICAL**: Do not hardcode secrets; use environment variables.\n\n```python\nfrom cosmos import ProfileConfig\n\n_profile_config = ProfileConfig(\n    profile_name=\"my_profile\",\n    target_name=\"dev\",\n    profiles_yml_filepath=\"/path/to/profiles.yml\",\n)\n```\n\n---\n\n## 5. Configure Testing Behavior (RenderConfig)\n\n> **Reference**: See **[reference/cosmos-config.md](reference/cosmos-config.md#testing-behavior-renderconfig)** for detailed testing options.\n\n| TestBehavior | Behavior |\n|--------------|----------|\n| `AFTER_EACH` (default) | Tests run immediately after each model (default) |\n| `BUILD` | Combine run + test into single `dbt build` |\n| `AFTER_ALL` | All tests after all models complete |\n| `NONE` | Skip tests |\n\n```python\nfrom cosmos import RenderConfig, TestBehavior\n\n_render_config = RenderConfig(\n    test_behavior=TestBehavior.AFTER_EACH,\n)\n```\n\n---\n\n## 6. Configure operator_args\n\n> **Reference**: See **[reference/cosmos-config.md](reference/cosmos-config.md#operator_args-configuration)** for detailed operator_args options.\n\n```python\n_operator_args = {\n    # BaseOperator params\n    \"retries\": 3,\n\n    # Cosmos-specific params\n    \"install_deps\": False,\n    \"full_refresh\": False,\n    \"quiet\": True,\n\n    # Runtime dbt vars (XCom / params)\n    \"vars\": '{\"my_var\": \"{{ ti.xcom_pull(task_ids=\\'pre_dbt\\') }}\"}',\n}\n```\n\n---\n\n## 7. Assemble DAG / TaskGroup\n\n### Option A: DbtDag (Standalone)\n\n```python\nfrom cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig, RenderConfig\nfrom cosmos.profiles import SnowflakeUserPasswordProfileMapping\nfrom pendulum import datetime\n\n_project_config = ProjectConfig(\n    dbt_project_path=\"/usr/local/airflow/dbt/my_project\",\n)\n\n_profile_config = ProfileConfig(\n    profile_name=\"default\",\n    target_name=\"dev\",\n    profile_mapping=SnowflakeUserPasswordProfileMapping(\n        conn_id=\"snowflake_default\",\n    ),\n)\n\n_execution_config = ExecutionConfig()\n_render_config = RenderConfig()\n\nmy_cosmos_dag = DbtDag(\n    dag_id=\"my_cosmos_dag\",\n    project_config=_project_config,\n    profile_config=_profile_config,\n    execution_config=_execution_config,\n    render_config=_render_config,\n    operator_args={},\n    start_date=datetime(2025, 1, 1),\n    schedule=\"@daily\",\n)\n```\n\n### Option B: DbtTaskGroup (Inside Existing DAG)\n\n```python\nfrom airflow.sdk import dag, task  # Airflow 3.x\n# from airflow.decorators import dag, task  # Airflow 2.x\nfrom airflow.models.baseoperator import chain\nfrom cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig, ExecutionConfig, RenderConfig\nfrom pendulum import datetime\n\n_project_config = ProjectConfig(dbt_project_path=\"/usr/local/airflow/dbt/my_project\")\n_profile_config = ProfileConfig(profile_name=\"default\", target_name=\"dev\")\n_execution_config = ExecutionConfig()\n_render_config = RenderConfig()\n\n@dag(start_date=datetime(2025, 1, 1), schedule=\"@daily\")\ndef my_dag():\n    @task\n    def pre_dbt():\n        return \"some_value\"\n\n    dbt = DbtTaskGroup(\n        group_id=\"dbt_project\",\n        project_config=_project_config,\n        profile_config=_profile_config,\n        execution_config=_execution_config,\n        render_config=_render_config,\n    )\n\n    @task\n    def post_dbt():\n        pass\n\n    chain(pre_dbt(), dbt, post_dbt())\n\nmy_dag()\n```\n\n### Option C: Use Cosmos operators directly\n\n```python\nimport os\nfrom datetime import datetime\nfrom pathlib import Path\nfrom typing import Any\n\nfrom airflow import DAG\n\ntry:\n    from airflow.providers.standard.operators.python import PythonOperator\nexcept ImportError:\n    from airflow.operators.python import PythonOperator\n\nfrom cosmos import DbtCloneLocalOperator, DbtRunLocalOperator, DbtSeedLocalOperator, ProfileConfig\nfrom cosmos.io import upload_to_aws_s3\n\nDEFAULT_DBT_ROOT_PATH = Path(__file__).parent / \"dbt\"\nDBT_ROOT_PATH = Path(os.getenv(\"DBT_ROOT_PATH\", DEFAULT_DBT_ROOT_PATH))\nDBT_PROJ_DIR = DBT_ROOT_PATH / \"jaffle_shop\"\nDBT_PROFILE_PATH = DBT_PROJ_DIR / \"profiles.yml\"\nDBT_ARTIFACT = DBT_PROJ_DIR / \"target\"\n\nprofile_config = ProfileConfig(\n    profile_name=\"default\",\n    target_name=\"dev\",\n    profiles_yml_filepath=DBT_PROFILE_PATH,\n)\n\n\ndef check_s3_file(bucket_name: str, file_key: str, aws_conn_id: str = \"aws_default\", **context: Any) -> bool:\n    \"\"\"Check if a file exists in the given S3 bucket.\"\"\"\n    from airflow.providers.amazon.aws.hooks.s3 import S3Hook\n\n    s3_key = f\"{context['dag'].dag_id}/{context['run_id']}/seed/0/{file_key}\"\n    print(f\"Checking if file {s3_key} exists in S3 bucket...\")\n    hook = S3Hook(aws_conn_id=aws_conn_id)\n    return hook.check_for_key(key=s3_key, bucket_name=bucket_name)\n\n\nwith DAG(\"example_operators\", start_date=datetime(2024, 1, 1), catchup=False) as dag:\n    seed_operator = DbtSeedLocalOperator(\n        profile_config=profile_config,\n        project_dir=DBT_PROJ_DIR,\n        task_id=\"seed\",\n        dbt_cmd_flags=[\"--select\", \"raw_customers\"],\n        install_deps=True,\n        append_env=True,\n    )\n\n    check_file_uploaded_task = PythonOperator(\n        task_id=\"check_file_uploaded_task\",\n        python_callable=check_s3_file,\n        op_kwargs={\n            \"aws_conn_id\": \"aws_s3_conn\",\n            \"bucket_name\": \"cosmos-artifacts-upload\",\n            \"file_key\": \"target/run_results.json\",\n        },\n    )\n\n    run_operator = DbtRunLocalOperator(\n        profile_config=profile_config,\n        project_dir=DBT_PROJ_DIR,\n        task_id=\"run\",\n        dbt_cmd_flags=[\"--models\", \"stg_customers\"],\n        install_deps=True,\n        append_env=True,\n    )\n\n    clone_operator = DbtCloneLocalOperator(\n        profile_config=profile_config,\n        project_dir=DBT_PROJ_DIR,\n        task_id=\"clone\",\n        dbt_cmd_flags=[\"--models\", \"stg_customers\", \"--state\", DBT_ARTIFACT],\n        install_deps=True,\n        append_env=True,\n    )\n\n    seed_operator >> run_operator >> clone_operator\n    seed_operator >> check_file_uploaded_task\n```\n\n### Setting Dependencies on Individual Cosmos Tasks\n\n```python\nfrom cosmos import DbtDag, DbtResourceType\nfrom airflow.sdk import task, chain\n\nwith DbtDag(...) as dag:\n    @task\n    def upstream_task():\n        pass\n\n    _upstream = upstream_task()\n\n    for unique_id, dbt_node in dag.dbt_graph.filtered_nodes.items():\n        if dbt_node.resource_type == DbtResourceType.SEED:\n            my_dbt_task = dag.tasks_map[unique_id]\n            chain(_upstream, my_dbt_task)\n```\n\n---\n\n## 8. Safety Checks\n\nBefore finalizing, verify:\n\n- [ ] Execution mode matches constraints (AIRFLOW_ASYNC → BigQuery only)\n- [ ] Warehouse adapter installed for chosen execution mode\n- [ ] Secrets via Airflow connections or env vars, NOT plaintext\n- [ ] Load mode matches execution (complex selectors → dbt_ls)\n- [ ] Airflow 3 asset URIs if downstream DAGs scheduled on Cosmos assets (see Appendix A)\n\n---\n\n## Appendix A: Airflow 3 Compatibility\n\n### Import Differences\n\n| Airflow 3.x | Airflow 2.x |\n|-------------|-------------|\n| `from airflow.sdk import dag, task` | `from airflow.decorators import dag, task` |\n| `from airflow.sdk import chain` | `from airflow.models.baseoperator import chain` |\n\n### Asset/Dataset URI Format Change\n\nCosmos ≤1.9 (Airflow 2 Datasets):\n```\npostgres://0.0.0.0:5434/postgres.public.orders\n```\n\nCosmos ≥1.10 (Airflow 3 Assets):\n```\npostgres://0.0.0.0:5434/postgres/public/orders\n```\n\n> **CRITICAL**: Update asset URIs when upgrading to Airflow 3.\n\n---\n\n## Appendix B: Operational Extras\n\n### Caching\n\nCosmos caches artifacts to speed up parsing. Enabled by default.\n\nReference: https://astronomer.github.io/astronomer-cosmos/configuration/caching.html\n\n### Memory-Optimized Imports\n\n```bash\nAIRFLOW__COSMOS__ENABLE_MEMORY_OPTIMISED_IMPORTS=True\n```\n\nWhen enabled:\n```python\nfrom cosmos.airflow.dag import DbtDag  # instead of: from cosmos import DbtDag\n```\n\n### Artifact Upload to Object Storage\n\n```bash\nAIRFLOW__COSMOS__REMOTE_TARGET_PATH=s3://bucket/target_dir/\nAIRFLOW__COSMOS__REMOTE_TARGET_PATH_CONN_ID=aws_default\n```\n\n```python\nfrom cosmos.io import upload_to_cloud_storage\n\nmy_dag = DbtDag(\n    # ...\n    operator_args={\"callback\": upload_to_cloud_storage},\n)\n```\n\n### dbt Docs Hosting\n\nCosmos serves dbt docs in the Airflow UI. The config depends on your Airflow major\nversion (each uses a different UI plugin system) — it is not a free single-vs-multi choice:\n\n| Airflow         | Config                                                            | Scope                | Since          |\n|-----------------|------------------------------------------------------------------|----------------------|----------------|\n| 2 (FAB plugin)  | `DBT_DOCS_DIR` (+ `DBT_DOCS_CONN_ID`, `DBT_DOCS_INDEX_FILE_NAME`) | Single project       | Cosmos 1.4.0+  |\n| 3.1+ (FastAPI)  | `DBT_DOCS_PROJECTS` (JSON)                                        | One or more projects | Cosmos 1.11.0+ |\n\nAirflow 2:\n\n```bash\nAIRFLOW__COSMOS__DBT_DOCS_DIR=\"path/to/docs\"                   # local path or S3/GCS/Azure/HTTP URI; defaults to the dbt target/ folder\nAIRFLOW__COSMOS__DBT_DOCS_CONN_ID=\"my_conn_id\"                 # optional; for cloud storage\nAIRFLOW__COSMOS__DBT_DOCS_INDEX_FILE_NAME=\"static_index.html\"  # optional; only if docs built with --static\n```\n\nAirflow 3.1+:\n\n```bash\nAIRFLOW__COSMOS__DBT_DOCS_PROJECTS='{\n    \"my_project\": {\n        \"dir\": \"s3://bucket/docs/\",\n        \"index\": \"index.html\",\n        \"conn_id\": \"aws_default\",\n        \"name\": \"My Project\"\n    }\n}'\n```\n\nPick by Airflow version, not project count. The single-project settings are the Airflow 2\npath; Cosmos publishes no deprecation notice for them — do not describe them as \"legacy\"\nor \"deprecated.\"\n\nReference: https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html\n\n---\n\n## Related Skills\n\n- **cosmos-dbt-fusion**: For dbt Fusion projects (not dbt Core)\n- **authoring-dags**: General DAG authoring patterns\n- **testing-dags**: Testing DAGs after creation\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}