Update to Rippit

{
  "description": "Analyze support conversations to find where an AI chatbot underperforms and human agents excel, turn those gaps into a prioritized knowledge-base build list, and (when a help center is connected) reconcile the gaps against existing documentation. Works with any helpdesk, any chatbot, and any help center. Use when the user asks to: analyze chatbot / bot gaps, compare bot vs agent (human) performance, run a deflection analysis, understand why the bot escalates or hands off, decide what the bot should be able to handle, find knowledge-base or documentation gaps, or improve / train the chatbot.",
  "included_files": [],
  "name": "rippit-chatbot-analysis",
  "skill_md_contents": "---\nname: rippit-chatbot-analysis\ndescription: >-\n  Analyze support conversations to find where an AI chatbot underperforms and human\n  agents excel, turn those gaps into a prioritized knowledge-base build list, and (when a\n  help center is connected) reconcile the gaps against existing documentation. Works with\n  any helpdesk, any chatbot, and any help center. Use when the user asks to: analyze\n  chatbot / bot gaps, compare bot vs agent (human) performance, run a deflection analysis,\n  understand why the bot escalates or hands off, decide what the bot should be able to\n  handle, find knowledge-base or documentation gaps, or improve / train the chatbot.\n---\n\n# Bot deflection gap analysis\n\nFind where the chatbot fails but humans succeed, and turn that into concrete actions:\nnew/updated help articles, and (if a help center is connected) a coverage reconciliation\nthat separates \"content doesn't exist\" from \"content exists but the bot can't find it.\"\n\nThis skill is intentionally **generic**. Customers run different helpdesks (Zendesk,\nIntercom, Salesforce, Gladly, Kustomer, Front, …), different chatbots (Fin, Zendesk bot,\nAda, a custom LLM agent, …), and different help centers (Intercom Articles, Zendesk Guide,\na public site, or none). **Never assume** how any of these are wired. Confirm the three\nthings below before doing analysis.\n\n## Analysis engine\n\nUse the Rippit MCP tools as the analysis engine: `list_data_sources`, `describe_table`,\n`describe_column`, `read_table`, `aggregate_table`, `pivot_table`, `create_worksheet`,\n`enrich_worksheet` + `get_enrich_status`, `find_quotes`, and `get_report_guide`. The\nworksheet-enrich path is the core: it reads each full conversation transcript and adds the\nper-row judgments this analysis needs (topic, resolution, satisfaction, escalation driver).\n\nCost of enrichment scales with **rows, not columns** — every column in one `enrich_worksheet`\ncall is filled in a single LLM pass per row. So batch all needed columns into ONE call.\n\n---\n\n## MANDATORY: confirm these three things first\n\nDo not skip. Use the `AskUserQuestion` tool (one question per item is fine). These are the\nonly parts that vary by customer, and getting them wrong invalidates the whole analysis.\n\n### 1. The data source\n- Call `list_data_sources` and look for the table of support conversations/tickets. There\n  are often several (per-helpdesk tables, prior worksheets, raw tables). **Do not guess.**\n- Present the candidate table(s) and ask the user to confirm which one holds the chatbot\n  conversations to analyze. If multiple channels exist, confirm whether to filter to one\n  source (e.g. only `intercom`) or analyze all.\n\n### 2. How to distinguish bot-handled vs bot-escalated-to-human\nThis is customer- and tool-specific. `describe_table` / `describe_column` first, then ask\nthe user to confirm the exact rule. Common signals:\n- **Tags** (e.g. `fin handled`, `fin escalated`, `bot resolved`, `handoff`, `agent bot`).\n- **A team/assignee field** (bot team vs a human team), or presence of any human agent in\n  an `agents`/participants column.\n- **An escalation column** or status.\n- **Handled by bot only** = the bot engaged and the conversation never reached a human.\n- **Escalated** = the bot handed off to a human.\n\n**Verify the rule empirically before trusting it.** A frequent trap: the \"bot handled\" tag\nis a *superset* that also covers escalated conversations (i.e. escalated ⊂ handled). Check\noverlap with an `aggregate_table` count using `CONTAINS_ALL` on the two tags. If they\noverlap, define **bot-only = handled AND NOT escalated**, and **handoff = escalated**.\nReport the two population sizes back to the user and confirm they look right.\n\n### 3. Whether a help center is connected (for doc reconciliation)\n- Ask the user whether their help center / knowledge base is connected as a tool (e.g. an\n  Intercom, Zendesk Guide, or Help Scout connector with article-listing tools).\n- If **yes**: you will run the reconciliation step (compare gaps to existing articles).\n- If **no**: skip reconciliation and deliver the gap list + KB build kit only. Offer that\n  they can connect the help center later to get the coverage comparison.\n\nAlso confirm a **time window** and whether to sample (use judgment; last 30–90 days is\ntypical). Prefer the full population if it's under ~3–5k conversations; otherwise sample\nrepresentatively with `sampleRows`.\n\n---\n\n## Process\n\n### Step A — Scope and size\nConfirm the three items above. Then use `aggregate_table` to count bot-only vs handoff\nconversations in the chosen window so you can size the enrichment.\n\n### Step B — Build the worksheet\n`create_worksheet` over the conversations in scope, filtered to the bot-touched population\n(both bot-only and handoff), for the confirmed window. Keep all source columns so nothing\nblocks later enrichment. The distinguishing tag/field stays on the worksheet so you can\nsplit the two groups in aggregation later.\n\n### Step C — Enrich (one call, all columns)\nAdd these columns with `enrich_worksheet` (adapt names/labels to the customer's product,\nbut keep the intent). Each judgment is made against the **full transcript**.\n\n- **contact_reason** — `unspecified_enum` (~15–20): the customer's primary reason for\n  contacting support, as a concise topic.\n- **issue_resolved** — enum: `resolved` / `partially_resolved` / `not_resolved` / `unknown`.\n  Was the underlying issue actually resolved by the end of the thread?\n- **customer_satisfied** — enum: `satisfied` / `neutral` / `dissatisfied` / `unknown`,\n  judged from the customer's own words and tone at the end.\n- **handled_well** — enum: `excellent` / `adequate` / `poor` / `unknown`: accuracy,\n  completeness, empathy, efficiency, independent of who handled it.\n- **escalation_driver** — enum: `not_escalated` / `account_specific_action` /\n  `knowledge_gap` / `technical_bug` / `complex_multistep` / `customer_demanded_human` /\n  `billing_or_contract` / `feature_request` / `other`. For handoffs, the primary reason the\n  bot couldn't resolve it. (`knowledge_gap` = the answer existed and a human simply knew it;\n  this is the prime documentation opportunity.)\n\nPoll `get_enrich_status` until `done: true`.\n\n### Step D — Compare bot vs human\nCall `get_report_guide` once before writing the final report. Then quantify the gap:\n- Resolution, satisfaction, and handling quality for **bot-only** vs **handoff**\n  (filter each group by the confirmed distinguishing rule). Use `aggregate_table` /\n  `pivot_table`.\n- Same cuts **by contact_reason** to find topics where the bot fails but humans resolve/\n  satisfy well.\n- **Escalation drivers** overall and by topic, with human resolution + satisfaction per\n  driver — this reveals *where humans do exceptionally well*, i.e. the closeable gaps.\n\nWatch for the common story: the bot's resolution rate may trail humans only slightly, while\nits **satisfaction** trails badly because many bot chats end with **no closure signal**\n(customer disengages). Call this out if the data shows it — it's a conversation-design fix\n(add \"did that solve it?\" confirmations), not only a knowledge fix.\n\n### Step E — Drill the knowledge gaps into doc topics\nCreate a **child worksheet** filtered to `escalation_driver = knowledge_gap` and enrich it\n(one call) with:\n- **kb_topic** — `unspecified_enum` (~20): the specific, article-sized documentation topic.\n- **question_asked** — string: the customer's question, phrased as an FAQ question.\n- **resolution_answer** — string: the substance of the answer that resolved it (steps,\n  settings, rules, limits) — raw material for an article; `NO CLEAR ANSWER` if none.\n- **answerable_by_docs** — enum: `yes` / `partial` / `no` (could a good article have\n  resolved it without a staff-only backend action?).\n\nAggregate `kb_topic` (with `answerable_by_docs`) to rank the article opportunities. Read\nback the Q/A per top topic (via a subagent if the read is large) and assemble a\n**KB Build Kit**: topics ranked into tiers by volume, each with its real Q→A pairs. Also\nsurface the **cross-cutting question shapes** that recur across topics (e.g. \"how long until\nX updates?\" → one canonical latency reference; \"why can't I see X?\" → one permissions guide;\n\"is there an API/export for X?\" → one capabilities matrix). These punch above their weight.\n\n### Step F — Reconcile against the help center (only if connected)\nPull the **entire** article library (paginate — help centers often exceed one page; check\n`total_pages`/`total_count`). List responses can be very large; when a tool result spills to\na file, parse it with a subagent so bulk text stays out of context. For each gap topic mark\n**Covered / Partial / Missing** (published only), naming the matching article(s). Then flag\nthe failure modes that make a bot miss content that exists:\n- **Published-but-empty articles** (title, no body) — a bot \"finds\" them and returns nothing.\n- **Draft-only content** — the answer is written but not live/retrievable.\n- **Naming/synonym mismatches** — the customer's term differs from the article's term.\n- **Sub-question gaps** inside otherwise-covered topics — augment with Build Kit Q/A.\n\nThe key reframe this step enables: distinguish \"content doesn't exist\" (write it) from\n\"content exists but the bot can't retrieve it\" (fix stubs, publish drafts, add synonyms,\ninvestigate the bot's KB indexing). Don't recommend writing articles that already exist.\n\n### Step G — Deliver\nWrite the report inline per `get_report_guide` (findings-led, visual cards for multi-cut\nfindings, a prioritized action table). Produce the **KB Build Kit** as a document\n(e.g. .docx) containing the ranked topics and their Q→A pairs. If reconciliation ran,\ninclude the coverage table and an action list ordered cheapest-first (fill empty stubs →\nadd synonyms → publish drafts → augment sub-questions → write net-new → audit bot retrieval).\nLink the worksheet(s) so the user can open the underlying data.\n\n---\n\n## Pitfalls & verification\n- **Confirm, don't assume** the data source, the bot/handoff rule, and the help-center\n  connection — every customer differs. This is the whole point of the skill.\n- **Verify the bot/handoff split empirically** (tag overlap) before analyzing.\n- **Batch enrichment** into one call per worksheet; poll to completion; never ask the user\n  to check back later.\n- **Never hand-compute** report numbers — take each figure from an aggregate/pivot call.\n- **Always include a catch-all** (`unknown`/`not enough info`) in enum enrichments.\n- **Paginate the full article library** and handle spilled-to-file tool results via a\n  subagent.\n- Sanity-check that enriched group sizes match the tag/field counts before trusting cuts.\n"
}