{"id":53872,"plugin_id":"plugins_6ac84bac80b081918c305a1574fbd186","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-10-24T13:55:15.249Z","digest":"957135aa1cadb7e84caa3b6cb326d12804f9ca14be74bf396eb06398eca7d86d","against":null,"payload":{"description":"Makes systems debuggable and reliably operable — instrumentation, alerting that is worth waking for, service objectives, and learning from failure. Use this to instrument a service, fix alerting that is ignored, set error budgets or reliability targets, prepare for on-call, or run a blameless post-incident review.","included_files":[{"relative_path":"references/sources.md","size_in_bytes":1770}],"name":"observability-and-reliability","skill_md_contents":"---\nname: observability-and-reliability\ndescription: Makes systems debuggable and reliably operable — instrumentation, alerting that is worth waking for, service objectives, and learning from failure. Use this to instrument a service, fix alerting that is ignored, set error budgets or reliability targets, prepare for on-call, or run a blameless post-incident review.\n---\n\n# Observability and reliability\n\nMonitoring tells you a thing you predicted is happening. Observability lets you ask a question you\ndid not anticipate. Production failures are mostly the unanticipated kind.\n\n## Instrument for questions you have not thought of yet\n\nEmit structured events with enough context to slice afterwards — request identifiers, user or tenant,\nversion, dependency, outcome, duration. Free-text logs are unsearchable at volume and become\nexpensive noise.\n\nPropagate a correlation identifier across every hop. Without it, a distributed system is a set of\nindependent stories and reconstructing one request is manual archaeology.\n\nMeasure what the user experiences at the percentile they experience it. A p50 latency graph is\nmostly a graph of the people who were not affected.\n\n## Alert on symptoms, not causes\n\nAlert when users are affected or imminently will be. High CPU is not an alert; requests failing or\nslowing is. Cause-based alerting produces pages for conditions the system handled and no page for\nnovel failures that hurt.\n\nEvery alert must be **actionable, urgent and specific**. If the recipient's honest response is to\nlook and close it, delete the alert — it is training the on-call to ignore the page, and the ignored\npage is eventually the real one.\n\nAlert fatigue is the actual reliability risk in most organizations. Fewer, better alerts beat\ncoverage.\n\n## Objectives and error budgets\n\nSet service level objectives from what users need, then treat the remainder as a budget to spend.\nThis converts a sterile argument between shipping and stability into arithmetic: budget remaining\nmeans ship, budget exhausted means the next work is reliability.\n\nKeep the internal objective tighter than any external commitment made through\n`operations:service-level-management`, so you find out before the customer does.\n\n## Learn from incidents\n\nPost-incident review exists to find what made the failure possible and hard to detect, not who\ntouched it last. Human error is a starting question, never the finding: what made the error easy,\nand why did nothing catch it?\n\nTrack the time to *detect* separately from time to resolve. Long detection is an observability\ndefect, and it is the part that repeats.\n\nProduce a small number of real actions with owners and dates. A review generating fifteen actions\ngenerates none.\n\n## Sources\n\n`references/sources.md` in this skill lists the outside authorities that settle the questions\nhere — what each one is authoritative for, and what you may do with it. Check them before\nanswering on anything they cover, and cite what you used. Most are free to read and not free\nto reproduce; the use note on each is binding.\n\n## Tooling\n\nMetrics and traces: Datadog, Grafana with Prometheus, New Relic, Honeycomb, and similar.\nErrors: Sentry, Rollbar, and similar. Logs: Elastic, OpenSearch, Loki, Splunk, and similar.\n\nOn-call and incident management: PagerDuty, Opsgenie, incident.io, FireHydrant, and similar.\n\nInstrument with OpenTelemetry wherever you can. Vendor-specific instrumentation is the part\nthat makes leaving expensive.\n\n## Never\n\n- Page a human for something they cannot act on.\n- Alert on a cause when you can alert on the symptom.\n- Report reliability as an average when users experience the tail.\n- Close an incident review with the finding that someone was careless.\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}