Update to astronomer-data

{
  "name": "tracing-upstream-lineage",
  "description": "Trace upstream data lineage. Use when the user asks where data comes from, what feeds a table, upstream dependencies, data sources, or needs to understand data origins.",
  "included_files": [],
  "skill_md_contents": "---\nname: tracing-upstream-lineage\ndescription: Trace upstream data lineage. Use when the user asks where data comes from, what feeds a table, upstream dependencies, data sources, or needs to understand data origins.\n---\n\n# Upstream Lineage: Sources\n\nTrace the origins of data - answer \"Where does this data come from?\"\n\n## Lineage Investigation\n\n### Step 1: Identify the Target Type\n\nDetermine what we're tracing:\n- **Table**: Trace what populates this table\n- **Column**: Trace where this specific column comes from\n- **DAG**: Trace what data sources this DAG reads from\n\n### Step 2: Find the Producing DAG\n\nTables are typically populated by Airflow DAGs. Find the connection:\n\n1. **Search DAGs by name**: Use `af dags list` and look for DAG names matching the table name\n   - `load_customers` -> `customers` table\n   - `etl_daily_orders` -> `orders` table\n\n2. **Explore DAG source code**: Use `af dags source <dag_id>` to read the DAG definition\n   - Look for INSERT, MERGE, CREATE TABLE statements\n   - Find the target table in the code\n\n3. **Check DAG tasks**: Use `af tasks list <dag_id>` to see what operations the DAG performs\n\n### On Astro\n\nIf you're running on Astro, the **Lineage tab** in the Astro UI provides visual lineage exploration across DAGs and datasets. Use it to quickly trace upstream dependencies without manually searching DAG source code.\n\n### On OSS Airflow\n\nUse DAG source code and task logs to trace lineage (no built-in cross-DAG UI).\n\n### Step 3: Trace Data Sources\n\nFrom the DAG code, identify source tables and systems:\n\n**SQL Sources** (look for FROM clauses):\n```python\n# In DAG code:\nSELECT * FROM source_schema.source_table  # <- This is an upstream source\n```\n\n**External Sources** (look for connection references):\n- `S3Operator` -> S3 bucket source\n- `PostgresOperator` -> Postgres database source\n- `SalesforceOperator` -> Salesforce API source\n- `HttpOperator` -> REST API source\n\n**File Sources**:\n- CSV/Parquet files in object storage\n- SFTP drops\n- Local file paths\n\n### Step 4: Build the Lineage Chain\n\nRecursively trace each source:\n\n```\nTARGET: analytics.orders_daily\n    ^\n    +-- DAG: etl_daily_orders\n            ^\n            +-- SOURCE: raw.orders (table)\n            |       ^\n            |       +-- DAG: ingest_orders\n            |               ^\n            |               +-- SOURCE: Salesforce API (external)\n            |\n            +-- SOURCE: dim.customers (table)\n                    ^\n                    +-- DAG: load_customers\n                            ^\n                            +-- SOURCE: PostgreSQL (external DB)\n```\n\n### Step 5: Check Source Health\n\nFor each upstream source:\n- **Tables**: Check freshness with the **checking-freshness** skill\n- **DAGs**: Check recent run status with `af dags stats`\n- **External systems**: Note connection info from DAG code\n\n## Lineage for Columns\n\nWhen tracing a specific column:\n\n1. Find the column in the target table schema\n2. Search DAG source code for references to that column name\n3. Trace through transformations:\n   - Direct mappings: `source.col AS target_col`\n   - Transformations: `COALESCE(a.col, b.col) AS target_col`\n   - Aggregations: `SUM(detail.amount) AS total_amount`\n\n## Output: Lineage Report\n\n### Summary\nOne-line answer: \"This table is populated by DAG X from sources Y and Z\"\n\n### Lineage Diagram\n```\n[Salesforce] --> [raw.opportunities] --> [stg.opportunities] --> [fct.sales]\n                        |                        |\n                   DAG: ingest_sfdc         DAG: transform_sales\n```\n\n### Source Details\n\n| Source | Type | Connection | Freshness | Owner |\n|--------|------|------------|-----------|-------|\n| raw.orders | Table | Internal | 2h ago | data-team |\n| Salesforce | API | salesforce_conn | Real-time | sales-ops |\n\n### Transformation Chain\nDescribe how data flows and transforms:\n1. Raw data lands in `raw.orders` via Salesforce API sync\n2. DAG `transform_orders` cleans and dedupes into `stg.orders`\n3. DAG `build_order_facts` joins with dimensions into `fct.orders`\n\n### Data Quality Implications\n- Single points of failure?\n- Stale upstream sources?\n- Complex transformation chains that could break?\n\n### Related Skills\n- Check source freshness: **checking-freshness** skill\n- Debug source DAG: **debugging-dags** skill\n- Trace downstream impacts: **tracing-downstream-lineage** skill\n- Add manual lineage annotations: **annotating-task-lineage** skill\n- Build custom lineage extractors: **creating-openlineage-extractors** skill\n"
}