← Better StackCONTENT HISTORY

Update to Better Stack

Snapshot Nov 11, 2026 · 06:08 UTC · version 1.1.3

Collection source: downloaded plugin package.

WHAT CHANGED

First saved snapshot

No earlier snapshot is available to establish a change.

Save Better Stack and check future changes in Following.

Up to 3 plugins free · no card required.

Compare saved observations

Download comparison JSON
Full technical diff · 0 changed fields
Full snapshot data
{
  "description": "Investigate a Better Stack incident, alert or outage report end to end. Find the incident, check who is on call, drill into the metrics, logs, traces and errors around it, correlate with recent releases, and post a short situation report. Use when someone asks why something is down, slow or erroring, mentions a Better Stack incident, monitor or alert, or when Claude is working as an on-call first responder in an incident channel (for example in Claude Tag).",
  "included_files": [],
  "name": "investigate-incident",
  "skill_md_contents": "---\nname: investigate-incident\ndescription: Investigate a Better Stack incident, alert or outage report end to end. Find the incident, check who is on call, drill into the metrics, logs, traces and errors around it, correlate with recent releases, and post a short situation report. Use when someone asks why something is down, slow or erroring, mentions a Better Stack incident, monitor or alert, or when Claude is working as an on-call first responder in an incident channel (for example in Claude Tag).\n---\n\n# Investigate an incident with Better Stack\n\nYou are the first responder. Tell the humans what is broken, since when, how bad it is and the most likely root cause, backed by data from Better Stack. Stay read-only unless someone in the thread explicitly asks you to act.\n\n## Core principle: drill down\n\nStart broad, find the anomaly, then narrow the focus. Aggregated data hides details.\n\n```\nBROAD SCAN (hours/days) → FIND ANOMALY → NARROW WINDOW (minutes) → COMPARE WINDOWS\n```\n\n- Time-series queries aggregate into buckets, so a 2-minute spike disappears in 1-hour buckets. When something looks interesting, query it again with a narrower window and smaller buckets.\n- Check metrics before logs. Metrics quickly show what is anomalous and when (\"memory spiked at 14:32\"). Use that timing to target the log search.\n\n## Phase 1: Orient\n\n- **Clarify the symptom:** slow, erroring, down, or intermittent?\n- **Establish the timeline:** when did it start, and was it sudden or gradual?\n- **Check the existing context:**\n  - Given an incident link or ID, call `incident`, `incident_timeline` and `incident_comments`. Otherwise call `incidents` with `status: [\"ongoing\", \"acknowledged\"]`, narrowed with `cause`, `monitor_id` or `from`/`to`.\n  - Call `chart_alerts` to see which telemetry alerts are firing.\n  - For a monitor incident, call `monitor`, then `monitor_response_times` and `monitor_availability`, to tell a total outage from a regional one or a latency-only one.\n  - Call `on_calls` and `on_call` to see who is on call. If the incident is escalating, call `escalation_policy` to see who gets paged next.\n- Don't repeat what the incident comments already say.\n\n## Phase 2: Scope\n\n- Call `sources` to find the affected service's sources, then `source` for its details, and `source_fields` to see which fields exist.\n- For errors, call `applications` and then `errors` for the affected application.\n\nForm a hypothesis based on the symptom:\n\n| Symptom | Primary suspects | Start with |\n| --- | --- | --- |\n| Slow responses | CPU, memory, DB queries, network | Metrics: latency, resource saturation |\n| Errors / 5xx | App bugs, dependency failures, OOM | Errors → metrics → logs |\n| Intermittent issues | Resource contention, connection limits, GC | Metric spikes + log correlation |\n| Complete outage | Crash, network partition, config change | Monitor checks, recent log events |\n| After a deploy | Recent code, config or dependency changes | `releases`, then the diff if GitHub is connected |\n\n## Phase 3: Investigate\n\n**Before writing any SQL, call the help tool for that data type.** The query syntax is specific and will fail without it:\n\n- logs and spans: `query_help` (with `source_type: \"logs\"` or `\"spans\"`)\n- metrics: `metrics_schema` (filter by name, e.g. `*latency*`, `*memory*`) and `metrics_query_help`\n- errors: `errors_query_help` (for custom error analytics beyond `errors` and `error`)\n\nThen run the SQL with `query`. Bound every query in time, and call `query_windows` before querying more than a few hours.\n\n**Metrics:**\n- Resource saturation: CPU, memory, disk, network\n- Application: request rate, error rate, latency (p50/p95/p99)\n- Dependencies: DB connection pool, queue depth\n\n**Logs and spans:**\n- Error clustering: which messages are new or spiking? Group by message or error type over time rather than reading raw lines first.\n- Log level distribution: count by level per minute to find the error spike.\n- Timeline reconstruction: what happened right before the symptom?\n- Spans: latency and error rate by service and operation, to find the slow or failing dependency.\n- If a filter (level, service) returns nothing, retry without it. Not every log has a level field, and service names vary. Never conclude \"no errors\" from one filtered query.\n\n**Errors:**\n- In `errors`, look for patterns that are `new` or `reoccurred` inside the window, then call `error` on the top ones for stack traces, releases and affected users.\n\n**Recent changes:**\n- Call `releases` for the application. A release shortly before the onset is the first suspect.\n- If a GitHub connection is available, compare the code between the last good release and the first bad one.\n\n## Phase 4: Conclude\n\n**Five whys.** Keep asking why until the answer is actionable:\n\n- ✗ \"Connections timed out\" → why?\n- ✗ \"Pool exhausted\" → why?\n- ✗ \"Slow queries held connections\" → why?\n- ✓ \"Missing index on users.email\" → actionable\n\n**Before concluding, verify:**\n- The cause preceded the effect (check the timestamps).\n- The mechanism is clear (an A → B → C chain).\n- The alternatives are ruled out.\n- The finding isn't also present in the baseline.\n\n## Hypothesis discipline\n\n- **Never work with a single hypothesis.** Generate 2–3 competing explanations before investigating. For each one, ask what would confirm it and what would disprove it.\n- **Actively look for disconfirming evidence.** Finding what you expected proves nothing.\n\n## Cross-verification\n\nNever conclude from a single data source. Logs explain what happened; metrics confirm whether it happened and when.\n\n- Redis OOM in the logs → did Redis memory actually spike in the metrics?\n- DB timeouts in the logs → check the connection pool and DB latency metrics.\n- CPU spike in the metrics → what triggered it in the logs (a deploy, a traffic spike)?\n- Errors starting at 14:00 → check the releases around 14:00.\n\n## Noise vs signal\n\n| Finding | Verify with | It's noise if… |\n| --- | --- | --- |\n| Cron job in the logs | Count by hour over 24h | Same count every hour |\n| Errors exist | Error-rate baseline | Errors are present in healthy periods too |\n| CPU high | CPU, memory and GC timeline | CPU rose after memory |\n\n**Rule: if it exists in the baseline, it's not the cause.**\n\nWhen you have an anomaly window, compare three windows: **before** (what's normal), **during** (what's different) and **after** (did it recover, and is there a new steady state).\n\n## When the investigation stalls\n\n1. Wrong service? Check the dependencies and upstream services in the spans.\n2. Wrong time window? Ask when it started.\n3. Is aggregation hiding it? Use smaller buckets and specific hosts.\n4. Metrics normal? Try the logs. Logs normal? Try the traces.\n\nDon't hide dead ends, because ruling things out is progress. If you're stuck, say: \"I've checked X, Y and Z, and all are normal. To go further I need the exact time window or a specific example (request ID, URL), and any recent changes I should know about.\"\n\n## Write the situation report\n\nPost one compact message in the thread, and update it as you learn more rather than posting walls of text:\n\n- **Status:** what is broken and for whom, whether it's ongoing or recovering, and when it started (UTC)\n- **Impact:** the affected monitors, regions, services and users, with numbers\n- **Timeline:** the key events in order\n- **Root cause:** the A → B → C chain and how confident you are, or the competing hypotheses still open\n- **Evidence:** the specific data points (timestamps, values, hosts), with links to the incident, error and dashboard pages from the tool results\n- **Next steps:** a specific recommendation for a human to check or do. Suggest it; don't do it yourself.\n\nSummarize trends instead of dumping raw data. Say clearly what you couldn't check (a missing source, a permission error) instead of guessing.\n\n## Act only when asked\n\nThese tools change real state and notify people: `acknowledge_incident`, `resolve_incident`, `escalate_incident`, `create_incident_comment` and `create_status_page_report`. Use them only when a person in the thread asks for that specific action, and confirm what you did.\n"
}

SHA-256 of public snapshot: 5f0fc3dabcec625c5b9817087933b00c79a2c455b726c48b18108aecf654f279