Update to DataHub Cloud

{
  "description": "Ground text-to-SQL work in DataHub catalog evidence. Use when a user asks to write, draft, debug, or execute SQL; answer a data question that requires SQL; calculate a metric; query named tables; or investigate SQL results with DataHub MCP tools available. Always begin with find_sql_context, even when the user already supplied tables or dataset URNs.",
  "included_files": [],
  "name": "datahub-sql-workflow",
  "skill_md_contents": "---\nname: datahub-sql-workflow\nargument-hint: \"[the question the query should answer]\"\ndescription: Ground text-to-SQL work in DataHub catalog evidence. Use when a user asks to write, draft, debug, or execute SQL; answer a data question that requires SQL; calculate a metric; query named tables; or investigate SQL results with DataHub MCP tools available. Always begin with find_sql_context, even when the user already supplied tables or dataset URNs.\nlicense: Apache-2.0\ncompatibility: Requires DataHub MCP tools (find_sql_context and catalog metadata tools); SQL execution engine optional\nmetadata:\n  author: datahub\n  version: \"2.2\"\n---\n\n# DataHub SQL Workflow\n\n## This plugin is MCP-only\n\nThere is no DataHub CLI here. This plugin declares one MCP server and nothing\nelse, so wherever this skill shows a `datahub ...` command, use the MCP tool with\nthe same function instead:\n\n| CLI shown below | MCP tool |\n| --- | --- |\n| `datahub search` | `search` |\n| `datahub get` | `get_entities` |\n| `datahub lineage` | `get_lineage`, or `get_lineage_paths_between` for a path |\n| `datahub graphql` | no equivalent — the operation is unavailable, say so |\n| `datahub check` | `get_me` |\n\nTool names are prefixed by the server (`mcp__datahub__search`). MCP tools are\nself-documenting, so read their schemas for parameter names rather than mapping\nCLI flags across literally. Where a section describes a CLI-only capability with\nno MCP tool, treat that capability as unavailable rather than improvising.\n\nGround every query in DataHub evidence. Treat business context as the authority\nfor meaning, catalog metadata as the authority for physical shape, and historical\nSQL context as evidence of analyst practice.\n\nRequire `find_sql_context` and DataHub metadata tools. If it is still unavailable,\nstop and ask the user to enable the DataHub MCP tools — do not fall back to any other\nevidence source (other discovery tools, local files, memory, web).\n\nTreat every other tool as capability-dependent: if one is unavailable,\ndisclose the limitation and continue with the supported steps; never\nreplace missing evidence with guesses.\n\n## 1. Find SQL context first\n\nCall `find_sql_context(question=<user's complete question>)` before any other\ncatalog, drafting, probing, or execution tool. Do this even when the user names\ntables or supplies Dataset URNs.\n\nRead the response by shape and follow its `message`:\n\n- Treat `user_edited` matches and their `instructions` as authoritative. They\n  may intentionally contain no datasets, patterns, or snippets.\n- Prefer curated `external:*` matches over generated history when they conflict.\n- With usable matches, use their patterns and datasets as primary candidates.\n  Cross-check `suggested_tables`; suggestions can appear even for a strong match.\n- With no usable match but suggested tables, inspect those Dataset URNs and\n  follow the message's drafting recommendation.\n- With neither usable matches nor suggestions, continue business-context and\n  catalog discovery. Call the drafting tool only with concrete Dataset URNs.\n- If the message reports a persisted-anchor metadata retrieval error, retry\n  `find_sql_context`. Do not reinterpret that failure as an anchor miss.\n\nIf two or more usable matches name disjoint datasets for the same metric or\nquestion, resolve the tie through business meaning (step 2). Prefer a\ndedicated metric or fact table over a same-named attribute column on an\nentity table, and present both candidates if the tie survives.\n\nGenerated matches can contain partial document fragments. Call\n`grep_documents(pattern=\".*\", start_offset=..., context_chars=...)` only when a\nreturned offset can recover context needed for the query.\n\nInterpret `shared_snippets` as modeled sibling semantics, not proof of literal\nwarehouse values. Treat `suggested_tables[].evidence.source == \"both\"` as useful\ncorroboration from independent discovery surfaces, not automatic correctness.\n\n## 1a. Route schema-discovery questions away from anchors\n\nSome questions ask about catalog structure rather than about data: which tables\nexist in a schema, what columns a table has, or what values a column takes.\nAnchors and curated documents cannot answer these — anchors describe query\npatterns, and per-table documentation does not enumerate a schema.\n\nWhen the question is schema discovery, skip the curated-document step below and\nanswer from `search`, `get_entities`, and `list_schema_fields`. Spending a\ndocument fan-out here costs context and cannot succeed.\n\n## 1b. Read curated documentation\n\n`find_sql_context` reads **only** documents whose subtype is `Semantic Anchor` —\nthe ones DataHub generates from query history. Every other document in the\ncatalog is customer-authored and invisible to it. Those are frequently where\njoin keys, SCD and latest-row rules, unit conventions, and \"do not use this\ntable\" warnings actually live.\n\nAfter `find_sql_context`, make these `search_documents` calls in order:\n\n**Call 1 — question-keyed search** (finds concept-level documentation):\n\n```\nsearch_documents(\n  query=<user's complete question>,\n  semantic_query=<user's complete question>,\n  filter='subtype != \"Semantic Anchor\"',\n  num_results=10,\n)\n```\n\n**Calls 2–4 — per-table keyword searches** (finds table-specific documentation):\n\nExtract the distinct table short names from `matches[].datasets` URNs (the\nlast segment after the final dot — e.g., `db.schema.MY_TABLE` → `MY_TABLE`).\nFor each of the top 3 distinct table names, call:\n\n```\nsearch_documents(\n  query=<TABLE_SHORT_NAME>,\n  filter='subtype != \"Semantic Anchor\"',\n  num_results=3,\n)\n```\n\nDo **not** pass `semantic_query` in the per-table calls — keyword matching on\nthe table name reliably finds table-specific documentation.\n\nIf any negated filter returns nothing, re-run that call with no `filter` and\ndiscard hits whose `subType` is `Semantic Anchor`. Some deployments drop negated\nclauses from the semantic leg, which silently reduces the call to keyword-only.\n\nFrom the combined results across all calls, hydrate up to **three** documents\ntotal with `grep_documents` — not three per call, and not a fourth extra read.\nChoose by `subType` and title: prefer documents whose title names one of the\ncandidate tables and whose `subType` indicates table documentation (e.g.,\n`Context`) over notebook-style documents.\n\nCount the strongest question-keyed non-anchor table document toward that cap,\nand fully read it before choosing a source table when its title or matched\ntext covers the requested grain or measures, even when anchors did not name\nthat table. If competing curated documents describe different grains, compare\nthem before selecting.\n\nWhen a governed table already provides the requested measures at the requested\ngrain, use its documented native columns instead of reconstructing them from\nlower-grain tables.\n\nThese table-specific documents frequently contain routing instructions that\nredirect you to a governed table. When a curated document says to prefer a\ndifferent table for the concept you are querying, follow that routing — search\nfor documentation on the redirected table too, and use the governed table as\nthe primary candidate.\n\nWhen retrieved evidence conflicts, rank it: user-edited match instructions,\nthen curated documentation, then generated (non-user-edited) anchors.\nAn anchor is distilled from what analysts have historically run, so a mistake\nrepeated often enough becomes a pattern. A curated document is the organization\nstating what is correct. When a curated document and a generated anchor differ\non any element — table choice, column choice, join key, filter, guard ordering,\nor units — follow the document and treat the generated pattern as corrected.\n\nThis applies to a pattern's mechanics, not only its table selection:\n\n- If a document names a native column for a value the anchor pattern derives\n  from other columns, select the documented column. A derived substitute\n  changes results even when it looks equivalent.\n- If a document specifies an order between operations that the pattern applies\n  differently — deduplicating to a latest version before filtering deleted\n  rows, say — use the documented order. The same predicates in a different\n  order can select different rows.\n- If a document states a unit or conversion the pattern omits, apply it.\n\nTwo limits on that precedence:\n\n- Routing advice (\"prefer table X instead\") states the default lane. It does not\n  override an explicit requirement in the question — freshness, a named table,\n  or a grain the preferred table cannot serve. When the question forces a\n  departure from documented routing, say so and give the reason.\n- When a curated document and live catalog metadata disagree — a documented\n  column is absent from the schema, say — state the disagreement and resolve it\n  before writing SQL. Never silently pick one.\n\n## 2. Establish business meaning\n\nSearch business context after the first call when SQL context is weak or\nabsent, or whenever the canonical definition remains uncertain.\n\nBusiness-context search is also required when:\n\n- usable matches disagree with each other or with `suggested_tables` about\n  which datasets to use; or\n- the leading candidate table lives outside the modeled analytics schemas.\n\nAn empty `message` means the top anchor's _text_ scored well against the\nquestion. It does not mean the anchor names the right tables, or all of them.\nDo not read it as permission to skip the curated-document step in 1b.\n\nBefore drafting, name every table the answer requires and confirm each one\nappears in evidence you actually retrieved — `matches[].datasets`,\n`suggested_tables`, `standard_filters_by_table`, or a curated document. A\nrequired table that appears in none of them is unverified; say so rather than\ninventing its columns.\n\n`search_documents` can also return anchor documents (subtype \"Semantic\nAnchor\"); skip those here — `find_sql_context` already provided them. Focus on\nglossary terms, domain alignment, and data products instead, using `search`\nwith an `entity_type` filter.\n\nIf a document or glossary definition names a table or calculation, follow it\nunless live evidence exposes a concrete conflict. A catalog table that looks\nmore specific, newer, or better-named than the documented one is not by\nitself a reason to deviate — verify with metadata before overriding. When\ndocumentation and catalog results disagree, state the disagreement and\nresolve it before writing SQL. When no business definition exists, state the\ngap and ask the user — do not fill it with an inferred interpretation.\n\nPrefer datasets that belong to a matching domain or data product over\nidentically-named tables outside them — data products mark the curated,\ngoverned query surfaces.\n\n## 3. Verify candidate datasets\n\nWhen a strong, unambiguous match provides a pattern with sufficient column\nand filter detail to draft SQL, go straight to step 5. Run the verification\nsteps below when the anchor pattern alone is not enough to draft\nconfidently: columns or join keys are unclear, the message is non-empty\n(weak or no match), matches and suggestions name different tables, a curated\ndocument contradicts the anchor, or the query requires joining multiple tables.\n\nFor every requested output column, identify the authoritative table and exact\nfield that supplies it. A table can be canonical for one purpose without being\ncanonical for every column it carries. Do not replace an entity label or\nlifecycle field with a similarly named column from a bridge or lookup table\nwhen evidence assigns that output to the canonical entity table or direct\nfield. Treat tables and joins in the closest matching SQL pattern as a\nchecklist: investigate any omitted canonical join before simplifying it away.\nDo not invent `COALESCE` fallbacks or other derivations when documentation is\nsilent; nullable lifecycle fields can encode state.\n\n1. Call `get_entities` on the candidate URNs. Read the metadata as intent\n   signals: description, ownership, tags, glossary terms, domain, data\n   product, table type, partition or clustering keys. Compare candidates on\n   these signals, not by name.\n2. Use targeted `list_schema_fields` calls to confirm relevant columns, types,\n   and grain.\n3. Prefer a governed table already at the requested grain over reconstructing\n   the same metric from raw or event-level data. Schema naming conventions\n   vary by org — treat a source-schema location as a hypothesis, not a\n   conclusion.\n4. Confirm that an \"all X\" question is not answered from a segmented subset.\n5. Verify every proposed join key on both sides. Do not add a speculative inner\n   join that could silently discard unmatched rows. When a curated document\n   names a non-obvious join key, use it rather than the same-named column.\n6. When resolving a user-provided name or search token without evidence of the\n   exact stored value, use a case-insensitive contains predicate rather than\n   copying an equality predicate from historical SQL. Use equality only when\n   curated documentation or `declared_enum_values` confirms the exact value.\n7. After `list_schema_fields` on the chosen table, disposition every\n   lifecycle and validity column it exposes — deletion markers, state or\n   status columns, snapshot or partition dates, latest-row flags. Apply a\n   guard only when the question's intended population, a standard-filter\n   advisory, a curated document, or an anchor pattern requires it; otherwise\n   record the column as considered and omitted.\n\nUse `standard_filters_by_table` from `find_sql_context` throughout verification:\n\n- Apply applicable guards and date shapes unless the user explicitly overrides\n  them.\n- Preserve the exact JSON scalar type, casing, and whitespace of\n  `declared_enum_values`.\n- Treat observed `enum_values` as samples, not an exhaustive allowed set.\n- Treat absent advisories as incomplete, not as evidence of no filters;\n  response budgeting can omit lower-support details.\n\n## 4. Run targeted probe queries\n\nThis step requires a SQL execution tool. If none is available, check\nDataHub for data profiles or sample data on the candidate datasets via\n`get_entities` — these can resolve column-value, null-rate, and\ncardinality questions without a live query. If neither execution nor\nprofiles are available, skip to step 5 and note any assumptions that a\nprobe would have resolved.\n\nRun a probe only when its result could materially change the table, join,\nfilter, grain, or time-window decision — skip it when metadata is already\ndecisive.\n\nRecommend the cheapest row-shape probe first:\n\n```sql\nSELECT <needed_columns>\nFROM <fully_qualified_table>\nLIMIT 1\n```\n\nUse named columns when known. Use `SELECT * ... LIMIT 1` only when metadata\ncannot identify the relevant fields. Omit `LIMIT 1` from aggregates that\nalready return one row.\n\nUse other minimal read-only probes as needed:\n\n- `COUNT(*)` or small grouped counts to test filter viability or grain;\n- `COUNT(DISTINCT key)` and duplicate checks to test uniqueness;\n- null counts or small grouped distributions to inspect candidate fields;\n- `MIN`/`MAX` timestamps to check coverage and freshness;\n- matched and unmatched counts to test join coverage;\n- comparable aggregates to distinguish otherwise plausible tables.\n\nSelect only required fields, apply known guards, and constrain verified\npartitions when appropriate. Never use a probe to manufacture a business rule.\nTreat empty results, unexpected magnitudes, errors, and timeouts as evidence\nabout access, freshness, schema drift, table type, or candidate suitability.\n\nIf authoritative context and observed schema or data drift apart — the\ndefinition's filter returns nothing, a named column is missing or behaves\ndifferently than described, or the answer requires an assumption the\ndefinition does not cover — use read-only probes only to characterize the\ndifference. Stop before the final answer query. Quote the definition\nexactly, name the drift in one sentence, offer two or three plain-language\ninterpretations, and ask which matches the user's intent.\n\nAllow at most three diagnostic rounds. Make each round test a new hypothesis;\ndo not guess-and-retry.\n\n## 5. Draft and verify SQL\n\nDraft directly from a verified anchor pattern when it clearly fits. Call\n`draft_sql_for_tables` only when `find_sql_context`'s message explicitly\nrecommends it — a viable anchor pattern is always preferred over a\ngenerated draft.\n\nPass the complete question, verified Dataset URNs, and actual SQL platform.\nTreat the result as an untrusted draft. Inspect its confidence, explanation,\nassumptions, ambiguities, suggested clarifications, tables used, and semantic\nmodel summary. An empty SQL string is a failed draft.\n\nVerify every table, field, join, literal, predicate, and aggregation against the\nevidence gathered above. Reconcile the draft with `standard_filters_by_table`:\nthe tool's internal injection is best-effort, so add missing required predicates\nand remove duplicates. Reconcile against the anchor pattern the same way:\ncarry every guard predicate the pattern applies into the final query, at the\nsame scope the pattern applies it, or record why it is intentionally\ndropped. Apply the same reconciliation to any required filter a curated\ndocument states — and where a document and an anchor pattern disagree about a\npredicate, its scope, or its order, the document wins.\n\nMatch the answer's shape to the question:\n\n- A present-tense or point-in-time question pins to the latest valid\n  snapshot and returns a single result; produce a trend or per-period\n  breakdown only when the question asks for one.\n- Default to the minimal query that answers the question. Add a join only\n  when a required output column cannot come from the chosen table, and be\n  able to state which requirement forces each join.\n\nBefore execution, ensure every predicate traces to the user's question,\nauthoritative business context, anchor instructions, a curated document, a\nstandard-filter advisory, a verified join, or a probe finding that will be\nreported. Confirm that the aggregation grain matches the question.\n\n## 6. Execute safely and report\n\nExecute only a single read-only `SELECT` statement, including read-only CTEs.\nReject DDL, DML, stored procedures, and side-effecting functions even if the\ndrafting tool merely lowers confidence instead of blocking them.\n\nExecute the final query unless the user requested draft-only output. Do not carry\nan exploratory `LIMIT 1` into the final query unless the user requested one row\nor a sample. If execution fails, re-ground the next attempt in catalog evidence\nor a targeted probe.\n\nTreat a `truncated` result as a sample. Never compute complete totals or other\nfinal aggregates from truncated rows; perform those calculations in SQL.\n\nReturn:\n\n- the answer or execution limitation;\n- the final SQL;\n- every source your answer relies on — datasets, curated documents, glossary\n  terms, domains, data products — cited as a markdown link\n  `[display name](urn:li:...)` using the URN a tool returned. For dataset\n  tables the SQL touches, cite the dataset entity URN (`urn:li:dataset:...`).\n  If you also relied on a curated document about that table, cite both the\n  dataset and the document — they are separate entities;\n- probe findings that changed the decision;\n- any table used without corroborating evidence;\n- assumptions and unresolved ambiguity.\n\nSeparate facts from documentation, facts from catalog metadata, and your own\ninferences; never present an inference as a fact.\n\nIn draft-only mode, omit execution but retain context discovery, verification,\ntargeted probes when needed, ambiguity handling, and source reporting.\n\nReport any discrepancies, gaps, or missing metadata discovered during the\nworkflow via `note_metadata_observation` — this includes missing glossary\ndefinitions, wrong or outdated descriptions, anchor-vs-catalog conflicts,\ncurated-document-vs-anchor conflicts, and missing column documentation. The\ntool is fire-and-forget and does not block the answer.\n"
}