Update to Nimble

{
  "name": "nimble-databricks-data-products",
  "description": "Builds Databricks data products from live web data, end to end: discovers the right Nimble\nweb-data agents, scrapes into Delta tables, and produces an AI/BI dashboard and/or a deployed\nDatabricks App — a table → dashboard → app workflow, for production data products or quick demos.\nUse whenever a request pairs live or scraped web data WITH a Databricks destination — e.g. \"scrape\nAmazon/Walmart prices into a Delta table and build a dashboard\", \"load Zillow/Instagram/Maps/search\nresults into Databricks and build a dashboard or app\", \"showcase Nimble + Databricks to a prospect\".\nPrefer it over nimble-web-expert or competitor-intel when the data lands in Databricks. Do NOT use\nfor one-off web fetches or CSV exports with no Databricks destination — use nimble-web-expert\ninstead. Do NOT use for competitor or company research briefings — use competitor-intel or\ncompany-deep-dive instead. Do NOT use for generic Databricks work with no Nimble/web-data angle —\nuse the official databricks-* skills instead.\n",
  "included_files": [
    {
      "relative_path": "assets/nimble-logo.png",
      "size_in_bytes": 25427
    },
    {
      "relative_path": "references/app-cookbook.md",
      "size_in_bytes": 4795
    },
    {
      "relative_path": "references/branding.md",
      "size_in_bytes": 2447
    },
    {
      "relative_path": "references/dashboard-cookbook.md",
      "size_in_bytes": 5919
    },
    {
      "relative_path": "references/databricks-skills.md",
      "size_in_bytes": 1944
    },
    {
      "relative_path": "references/install-nimble-integration.md",
      "size_in_bytes": 2544
    },
    {
      "relative_path": "references/nimble-agents.md",
      "size_in_bytes": 11840
    },
    {
      "relative_path": "references/preflight.md",
      "size_in_bytes": 3425
    },
    {
      "relative_path": "scripts/build_dashboard.py",
      "size_in_bytes": 13611
    },
    {
      "relative_path": "scripts/ingest.sh",
      "size_in_bytes": 2390
    }
  ],
  "skill_md_contents": "---\nname: nimble-databricks-data-products\ndescription: |\n  Builds Databricks data products from live web data, end to end: discovers the right Nimble\n  web-data agents, scrapes into Delta tables, and produces an AI/BI dashboard and/or a deployed\n  Databricks App — a table → dashboard → app workflow, for production data products or quick demos.\n  Use whenever a request pairs live or scraped web data WITH a Databricks destination — e.g. \"scrape\n  Amazon/Walmart prices into a Delta table and build a dashboard\", \"load Zillow/Instagram/Maps/search\n  results into Databricks and build a dashboard or app\", \"showcase Nimble + Databricks to a prospect\".\n  Prefer it over nimble-web-expert or competitor-intel when the data lands in Databricks. Do NOT use\n  for one-off web fetches or CSV exports with no Databricks destination — use nimble-web-expert\n  instead. Do NOT use for competitor or company research briefings — use competitor-intel or\n  company-deep-dive instead. Do NOT use for generic Databricks work with no Nimble/web-data angle —\n  use the official databricks-* skills instead.\nallowed-tools:\n  - Bash(databricks:*)\n  - Bash(python3:*)\n  - Bash(bash:*)\n  - Bash(jq:*)\n  - Bash(npm:*)\n  - Read\n  - Write\n  - Edit\n  - AskUserQuestion\n  - Skill\nmetadata:\n  author: Nimbleway\n  version: 1.6.1\n  category: data-platforms\n---\n\n# Nimble on Databricks — data products builder\n\nTurn a natural-language brief like\n`pricing analysis on dog products from walmart and amazon` into working Databricks data products:\n**discover agents → ingest live web search data into Delta → build dashboard and/or app → deliver links.**\nEqually at home for a quick demo or a real, reusable data product.\n\nYou are the orchestrator. Databricks mechanics are delegated to the official `databricks-*`\nskills (see `references/databricks-skills.md`); this skill owns the **Nimble glue and the gaps**\n(agent discovery, ingestion-from-agents, the AI/BI dashboard JSON, branding).\n\n## Golden rules\n\n- **Discover, don't assume.** Read agent names via `nimble_agent_list()`, input params via\n  `nimble_agent_describe('<agent>')`, and output fields by probing one call (`to_json(parsing[0])`) —\n  never hardcode from memory (Amazon search takes `keyword`, not `query`).\n- **Probe before fanning out.** Run one call per source first to learn its localization flag, field\n  names, and value formats — sources differ (some return numeric prices, others currency strings).\n- **One statement per Statements API call.** Multiple `;`-separated statements in one call are a parse error.\n- **Each Bash call is a fresh shell.** Env vars and `cd` don't persist — set them inline. See `references/preflight.md`.\n- **Fail fast, then confirm.** Run Phase 0 preflight first; recommend a warehouse + writable schema, then confirm before writing.\n- **Always ask the deliverable.** Table / +dashboard / +app is a per-run choice.\n- **Branding is always on, neutral.** \"Powered by Nimble\" + light theme + yellow accent. See `references/branding.md`.\n- **Leave artifacts in place.** No teardown.\n- **Show your work and the headline.** End with URLs and the one-sentence insight (e.g. the price gap).\n\n## Workflow\n\nTrack these as todos so nothing is skipped.\n\n### Phase 0 — Preflight (read-only, fail fast)\nLean on the **`databricks-core`** skill for the generic checks.\n1. `databricks current-user me` → confirm auth; capture the username (for the default schema).\n2. Find a **RUNNING** SQL warehouse: `databricks warehouses list`. Prefer one already RUNNING; if none, offer to start one.\n3. **Integration gate** — confirm these exist:\n   `nimble_integration.tools.{nimble_search, nimble_extract, nimble_agent_run, nimble_agent_list, nimble_agent_describe}`.\n   Quick check: `databricks functions list nimble_integration tools`.\n   **If missing → STOP** and walk the user through `references/install-nimble-integration.md`\n   (Nimble cookbook). Do not try to auto-install.\n4. **Recommend + confirm** the target: a warehouse and a writable `catalog.schema`\n   (default `users.<username>`). Verify writability — some shared catalogs deny `CREATE TABLE`.\n   Present the recommendation and let the user confirm or override before writing.\n\nDetails + exact commands: `references/preflight.md`.\n\n### Phase 1 — Interpret the brief + clarify (AskUserQuestion)\nParse the brief into: **domain/entity · search terms · sources · analysis goal**.\nThen ask (batch into one AskUserQuestion call):\n- **Deliverable** — always ask: table / table + dashboard / table + dashboard + app.\n- **Sources** — confirm the agents you matched (e.g. Amazon + Walmart SERP).\n- **Volume** — default ~8–10 search terms, ~100+ rows/source.\n\nKeep the brief's intent (the \"analysis goal\") — it picks the Phase 4 template and the headline.\n\n### Phase 2 — Discover agents + map a unified schema\nSee `references/nimble-agents.md`.\n1. `nimble_agent_list()` via SQL, filter by the source/domain keywords.\n2. For each chosen agent: `nimble_agent_describe('<name>')` → read its input params (required ones,\n   exact names, localization/pagination flags). Output fields come from the §2.5 probe, not here.\n3. Design **one unified table** with a `source` column + a normalized core\n   (`product_name, price, currency, rating, review_count, brand, url, …`), keeping only fields the\n   chosen agents actually emit. Multi-source comparison hinges on the shared columns.\n\n### Phase 3 — Ingest (control table + one set-based call)\nSee `references/nimble-agents.md` for the full SQL. Drive ingestion from a **control table**, not\nper-keyword files — it's reproducible and expandable (add a row, re-run).\n0. **Probe ONE call per source first (fail fast).** Before fanning out, run a single\n   `nimble_agent_run` per source and check: status, the real field names, the localization flag, and\n   whether a price casts cleanly. This catches the Walmart-class surprises (localization, currency-\n   string prices, `product_price` vs `price`) in ~40s instead of after a wasted full round. Highest-\n   leverage step — see `nimble-agents.md` §2.5.\n1. Create a **control (queries) table** `<schema>.<table>_queries` (source, agent, keyword,\n   params_json, localization, enabled) and seed one row per (source × term). `params_json` uses each\n   agent's **real** param name (from `input_properties`); set **localization per agent** (e.g.\n   `amazon_serp` true, `walmart_serp` false).\n2. Create the **unified results table** (`source` column + normalized core + `raw VARIANT`).\n3. Run **one INSERT** that calls `nimble_agent_run(q.agent, q.params_json, q.localization)` via a\n   correlated `LATERAL` join over the control table, with a `/*+ REPARTITION(N) */` hint (N ≈ enabled\n   rows, kept modest — high parallelism can trip API rate limits) so the agent calls run in parallel.\n   It's one long statement → run it async with `bash scripts/ingest.sh <WH> ingest.sql`.\n4. **Reconcile against the control table** (LEFT JOIN): a term that lands no items returns an empty\n   result, and a correlated LATERAL drops empty rows — so reconcile to confirm every source is\n   covered. If a source shows 0, re-check its localization flag (per-agent) and casts before\n   building; see `nimble-agents.md` §6 for the diagnostic order.\n\n### Phase 4 — Build the deliverable(s)\nChoose a **template** from the matched agents' `vertical`/`entity_type`:\n\n| Vertical | Dashboard/app shape |\n|----------|---------------------|\n| Ecommerce (SERP/PDP/CLP) | KPIs; listings & avg price by source/keyword; sponsored share; price-vs-rating scatter; product table with Open links; multi-source → comparison bars + best-effort item-level price gap |\n| Social | volume/engagement by account/post; top-content table; like/follower distributions |\n| Real Estate | price & price/sqft; listings by location; beds/baths breakdowns |\n| Maps / Local | avg rating; review counts; places table |\n| LLM / AEO | source/answer presence; share-of-voice; citation table |\n| _fallback_ | KPIs + 2 categorical bars + the raw table (works off any `output_schema`) |\n\n**Comparison depth (hybrid):** always build the aggregate/category comparison; *additionally* try\nbest-effort item-level matching across sources (normalize brand + key tokens). If confident matches\nexist, add a \"same-product price gap\" view; otherwise keep the aggregate comparison and note that\nitem-level matching wasn't confident.\n\n- **Dashboard** → use `scripts/build_dashboard.py` (compact spec → valid `serialized_dashboard`,\n  create + publish). It bakes in every Lakeview gotcha. Read `references/dashboard-cookbook.md` for\n  the spec format and recipes.\n- **App** → follow `references/app-cookbook.md` (delegates scaffold/deploy to `databricks-apps`;\n  adds the Nimble-specific SQL, branding, and the numeric-string / light-mode gotchas).\n- **Branding** → `references/branding.md` (always applied).\n\n### Phase 5 — Verify, deliver & share\n- Publish the dashboard / confirm the app is `RUNNING`; collect URLs.\n- Summarize what was built and the **headline insight** (the comparison takeaway).\n- **Offer to share** the dashboard/app link — if a Slack or Notion connector is available, offer to\n  post it there (Slack = the link + headline; Notion = a short dated page). Mention once; don't nag.\n- **Suggest next steps** with sibling skills, e.g. `competitor-intel` / `company-deep-dive` for\n  business signals on the brands surfaced, or `nimble-web-expert` for a one-off deeper pull.\n- Offer iterations (more charts, item-level matching, theming, a scheduled refresh job).\n\n## Reference map\n- `references/databricks-skills.md` — which official `databricks-*` skill to use per phase.\n- `references/install-nimble-integration.md` — setup when the integration gate fails.\n- `references/preflight.md` — auth, warehouse, writable-schema discovery (exact commands).\n- `references/nimble-agents.md` — discovery, schema mapping, ingestion SQL + gotchas.\n- `references/dashboard-cookbook.md` — Lakeview JSON recipes + every gotcha (authoritative).\n- `references/app-cookbook.md` — AppKit demo app glue + gotchas.\n- `references/branding.md` — \"Powered by Nimble\", logo, colors.\n- `scripts/ingest.sh` — async statement fan-out + poll.\n- `scripts/build_dashboard.py` — compact spec → create + publish a dashboard.\n- `assets/nimble-logo.png` — the Nimble mark for app branding.\n"
}