Update to astronomer-data

{
  "name": "deploying-airflow",
  "description": "Deploys Airflow DAGs and projects. Use when deploying Airflow or answering anything about deployment - deploying DAGs/projects, pushing code, setting up CI/CD, deploying to production or deployment strategies for Airflow.",
  "included_files": [],
  "skill_md_contents": "---\nname: deploying-airflow\ndescription: Deploys Airflow DAGs and projects. Use when deploying Airflow or answering anything about deployment - deploying DAGs/projects, pushing code, setting up CI/CD, deploying to production or deployment strategies for Airflow.\n---\n\n# Deploying Airflow\n\nThis skill covers deploying Airflow DAGs and projects to production, whether using Astro (Astronomer's managed platform) or open-source Airflow on Docker Compose or Kubernetes.\n\n**Choosing a path:** Astro is a good fit for managed operations and faster CI/CD. For open-source, use Docker Compose for dev and the Helm chart for production.\n\n---\n\n## Astro (Astronomer)\n\nAstro provides CLI commands and GitHub integration for deploying Airflow projects.\n\n### Deploy Commands\n\n| Command | What It Does |\n|---------|--------------|\n| `astro deploy` | Full project deploy — builds Docker image and deploys DAGs |\n| `astro deploy --dags` | DAG-only deploy — pushes only DAG files (fast, no image build) |\n| `astro deploy --image` | Image-only deploy — pushes only the Docker image (for multi-repo CI/CD) |\n| `astro deploy --dbt` | dbt project deploy — deploys a dbt project to run alongside Airflow |\n\n### Full Project Deploy\n\nBuilds a Docker image from your Astro project and deploys everything (DAGs, plugins, requirements, packages):\n\n```bash\nastro deploy\n```\n\nUse this when you've changed `requirements.txt`, `Dockerfile`, `packages.txt`, plugins, or any non-DAG file.\n\n### DAG-Only Deploy\n\nPushes only files in the `dags/` directory without rebuilding the Docker image:\n\n```bash\nastro deploy --dags\n```\n\nThis is significantly faster than a full deploy since it skips the image build. Use this when you've only changed DAG files and haven't modified dependencies or configuration.\n\n### Image-Only Deploy\n\nPushes only the Docker image without updating DAGs:\n\n```bash\nastro deploy --image\n```\n\nThis is useful in multi-repo setups where DAGs are deployed separately from the image, or in CI/CD pipelines that manage image and DAG deploys independently.\n\n### dbt Project Deploy\n\nDeploys a dbt project to run with Cosmos on an Astro deployment:\n\n```bash\nastro deploy --dbt\n```\n\n### GitHub Integration\n\nAstro supports branch-to-deployment mapping for automated deploys:\n\n- Map branches to specific deployments (e.g., `main` -> production, `develop` -> staging)\n- Pushes to mapped branches trigger automatic deploys\n- Supports DAG-only deploys on merge for faster iteration\n\nConfigure this in the Astro UI under **Deployment Settings > CI/CD**.\n\n### CI/CD Patterns\n\nCommon CI/CD strategies on Astro:\n\n1. **DAG-only on feature branches**: Use `astro deploy --dags` for fast iteration during development\n2. **Full deploy on main**: Use `astro deploy` on merge to main for production releases\n3. **Separate image and DAG pipelines**: Use `--image` and `--dags` in separate CI jobs for independent release cycles\n\n### Deploy Queue\n\nWhen multiple deploys are triggered in quick succession, Astro processes them sequentially in a deploy queue. Each deploy completes before the next one starts.\n\n### Reference\n\n- [Astro Deploy Documentation](https://www.astronomer.io/docs/astro/deploy-code)\n\n---\n\n## Open-Source: Docker Compose\n\nDeploy Airflow using the official Docker Compose setup. This is recommended for learning and exploration — for production, use Kubernetes with the Helm chart (see below).\n\n### Prerequisites\n\n- Docker and Docker Compose v2.14.0+\n- The official `apache/airflow` Docker image\n\n### Quick Start\n\nDownload the official Airflow 3 Docker Compose file:\n\n```bash\ncurl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'\n```\n\nThis sets up the full Airflow 3 architecture:\n\n| Service | Purpose |\n|---------|---------|\n| `airflow-apiserver` | REST API and UI (port 8080) |\n| `airflow-scheduler` | Schedules DAG runs |\n| `airflow-dag-processor` | Parses and processes DAG files |\n| `airflow-worker` | Executes tasks (CeleryExecutor) |\n| `airflow-triggerer` | Handles deferrable/async tasks |\n| `postgres` | Metadata database |\n| `redis` | Celery message broker |\n\n### Minimal Setup\n\nFor a simpler setup with LocalExecutor (no Celery/Redis), create a `docker-compose.yaml`:\n\n```yaml\nx-airflow-common: &airflow-common\n  image: apache/airflow:3  # Use the latest Airflow 3.x release\n  environment: &airflow-common-env\n    AIRFLOW__CORE__EXECUTOR: LocalExecutor\n    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow\n    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'\n    AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags\n  volumes:\n    - ./dags:/opt/airflow/dags\n    - ./logs:/opt/airflow/logs\n    - ./plugins:/opt/airflow/plugins\n  depends_on:\n    postgres:\n      condition: service_healthy\n\nservices:\n  postgres:\n    image: postgres:16\n    environment:\n      POSTGRES_USER: airflow\n      POSTGRES_PASSWORD: airflow\n      POSTGRES_DB: airflow\n    volumes:\n      - postgres-db-volume:/var/lib/postgresql/data\n    healthcheck:\n      test: [\"CMD\", \"pg_isready\", \"-U\", \"airflow\"]\n      interval: 10s\n      retries: 5\n      start_period: 5s\n\n  airflow-init:\n    <<: *airflow-common\n    entrypoint: /bin/bash\n    command:\n      - -c\n      - |\n        airflow db migrate\n        airflow users create \\\n          --username admin \\\n          --firstname Admin \\\n          --lastname User \\\n          --role Admin \\\n          --email admin@example.com \\\n          --password admin\n    depends_on:\n      postgres:\n        condition: service_healthy\n\n  airflow-apiserver:\n    <<: *airflow-common\n    command: airflow api-server\n    ports:\n      - \"8080:8080\"\n    healthcheck:\n      test: [\"CMD\", \"curl\", \"--fail\", \"http://localhost:8080/health\"]\n      interval: 30s\n      timeout: 10s\n      retries: 5\n      start_period: 30s\n\n  airflow-scheduler:\n    <<: *airflow-common\n    command: airflow scheduler\n\n  airflow-dag-processor:\n    <<: *airflow-common\n    command: airflow dag-processor\n\n  airflow-triggerer:\n    <<: *airflow-common\n    command: airflow triggerer\n\nvolumes:\n  postgres-db-volume:\n```\n\n> **Airflow 3 architecture note**: The webserver has been replaced by the **API server** (`airflow api-server`), and the **DAG processor** now runs as a standalone process separate from the scheduler.\n\n### Common Operations\n\n```bash\n# Start all services\ndocker compose up -d\n\n# Stop all services\ndocker compose down\n\n# View logs\ndocker compose logs -f airflow-scheduler\n\n# Restart after requirements change\ndocker compose down && docker compose up -d --build\n\n# Run a one-off Airflow CLI command\ndocker compose exec airflow-apiserver airflow dags list\n```\n\n### Installing Python Packages\n\nAdd packages to `requirements.txt` and rebuild:\n\n```bash\n# Add to requirements.txt, then:\ndocker compose down\ndocker compose up -d --build\n```\n\nOr use a custom Dockerfile:\n\n```dockerfile\nFROM apache/airflow:3  # Pin to a specific version (e.g., 3.1.7) for reproducibility\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt\n```\n\nUpdate `docker-compose.yaml` to build from the Dockerfile:\n\n```yaml\nx-airflow-common: &airflow-common\n  build:\n    context: .\n    dockerfile: Dockerfile\n  # ... rest of config\n```\n\n### Environment Variables\n\nConfigure Airflow settings via environment variables in `docker-compose.yaml`:\n\n```yaml\nenvironment:\n  # Core settings\n  AIRFLOW__CORE__EXECUTOR: LocalExecutor\n  AIRFLOW__CORE__PARALLELISM: 32\n  AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 16\n\n  # Email\n  AIRFLOW__EMAIL__EMAIL_BACKEND: airflow.utils.email.send_email_smtp\n  AIRFLOW__SMTP__SMTP_HOST: smtp.example.com\n\n  # Connections (as URI)\n  AIRFLOW_CONN_MY_DB: postgresql://user:pass@host:5432/db\n```\n\n---\n\n## Open-Source: Kubernetes (Helm Chart)\n\nDeploy Airflow on Kubernetes using the official Apache Airflow Helm chart.\n\n### Prerequisites\n\n- A Kubernetes cluster\n- `kubectl` configured\n- `helm` installed\n\n### Installation\n\n```bash\n# Add the Airflow Helm repo\nhelm repo add apache-airflow https://airflow.apache.org\nhelm repo update\n\n# Install with default values\nhelm install airflow apache-airflow/airflow \\\n  --namespace airflow \\\n  --create-namespace\n\n# Install with custom values\nhelm install airflow apache-airflow/airflow \\\n  --namespace airflow \\\n  --create-namespace \\\n  -f values.yaml\n```\n\n### Key values.yaml Configuration\n\n```yaml\n# Executor type\nexecutor: KubernetesExecutor  # or CeleryExecutor, LocalExecutor\n\n# Airflow image (pin to your desired version)\ndefaultAirflowRepository: apache/airflow\ndefaultAirflowTag: \"3\"  # Or pin: \"3.1.7\"\n\n# Git-sync for DAGs (recommended for production)\ndags:\n  gitSync:\n    enabled: true\n    repo: https://github.com/your-org/your-dags.git\n    branch: main\n    subPath: dags\n    wait: 60  # seconds between syncs\n\n# API server (replaces webserver in Airflow 3)\napiServer:\n  resources:\n    requests:\n      cpu: \"250m\"\n      memory: \"512Mi\"\n    limits:\n      cpu: \"500m\"\n      memory: \"1Gi\"\n  replicas: 1\n\n# Scheduler\nscheduler:\n  resources:\n    requests:\n      cpu: \"500m\"\n      memory: \"1Gi\"\n    limits:\n      cpu: \"1000m\"\n      memory: \"2Gi\"\n\n# Standalone DAG processor\ndagProcessor:\n  enabled: true\n  resources:\n    requests:\n      cpu: \"250m\"\n      memory: \"512Mi\"\n    limits:\n      cpu: \"500m\"\n      memory: \"1Gi\"\n\n# Triggerer (for deferrable tasks)\ntriggerer:\n  resources:\n    requests:\n      cpu: \"250m\"\n      memory: \"512Mi\"\n    limits:\n      cpu: \"500m\"\n      memory: \"1Gi\"\n\n# Worker resources (CeleryExecutor only)\nworkers:\n  resources:\n    requests:\n      cpu: \"500m\"\n      memory: \"1Gi\"\n    limits:\n      cpu: \"2000m\"\n      memory: \"4Gi\"\n  replicas: 2\n\n# Log persistence\nlogs:\n  persistence:\n    enabled: true\n    size: 10Gi\n\n# PostgreSQL (built-in)\npostgresql:\n  enabled: true\n\n# Or use an external database\n# postgresql:\n#   enabled: false\n# data:\n#   metadataConnection:\n#     user: airflow\n#     pass: airflow\n#     host: your-rds-host.amazonaws.com\n#     port: 5432\n#     db: airflow\n```\n\n### Upgrading\n\n```bash\n# Upgrade with new values\nhelm upgrade airflow apache-airflow/airflow \\\n  --namespace airflow \\\n  -f values.yaml\n\n# Upgrade to a new Airflow version\nhelm upgrade airflow apache-airflow/airflow \\\n  --namespace airflow \\\n  --set defaultAirflowTag=\"<version>\"\n```\n\n### DAG Deployment Strategies on Kubernetes\n\n1. **Git-sync** (recommended): DAGs are synced from a Git repository automatically\n2. **Persistent Volume**: Mount a shared PV containing DAGs\n3. **Baked into image**: Include DAGs in a custom Docker image\n\n### Useful Commands\n\n```bash\n# Check pod status\nkubectl get pods -n airflow\n\n# View scheduler logs\nkubectl logs -f deployment/airflow-scheduler -n airflow\n\n# Port-forward the API server\nkubectl port-forward svc/airflow-apiserver 8080:8080 -n airflow\n\n# Run a one-off CLI command\nkubectl exec -it deployment/airflow-scheduler -n airflow -- airflow dags list\n```\n\n---\n\n## Related Skills\n\n- **setting-up-astro-project**: For initializing a new Astro project\n- **managing-astro-local-env**: For local development with `astro dev`\n- **authoring-dags**: For writing DAGs before deployment\n- **testing-dags**: For testing DAGs before deployment\n"
}