← Files Honeycomb (EU)ARCHIVED FILE
skills/otel-instrumentation/references/custom-instrumentation.md
17.7 KB · Oct 3, 2026 · 06:27 UTC
# Custom Instrumentation Patterns
Detailed patterns for adding custom instrumentation beyond auto-instrumentation.
## When to Add Custom Instrumentation
Auto-instrumentation covers:
- HTTP server/client requests
- Database queries
- gRPC calls
- Message queue operations
Add custom instrumentation for:
- Business logic (checkout flow, payment processing)
- Cache operations
- Internal function calls that matter
- Custom attributes with business context
## Pattern: Adding Context to Auto-Instrumented Spans
The most impactful custom instrumentation. No new spans needed — just add
attributes to existing spans.
### Go
```go
func handleCheckout(w http.ResponseWriter, r *http.Request) {
span := trace.SpanFromContext(r.Context())
span.SetAttributes(
attribute.String("user.id", getUserID(r)),
attribute.Float64("cart.total", cart.Total()),
attribute.Int("cart.items", cart.ItemCount()),
attribute.String("payment.method", cart.PaymentMethod()),
)
// ... rest of handler
}
```
### Python
```python
@app.route("/checkout", methods=["POST"])
def handle_checkout():
span = trace.get_current_span()
span.set_attribute("user.id", get_user_id())
span.set_attribute("cart.total", cart.total)
span.set_attribute("cart.items", cart.item_count)
span.set_attribute("payment.method", cart.payment_method)
# ... rest of handler
```
### Node.js
```javascript
app.post("/checkout", (req, res) => {
const span = trace.getActiveSpan();
span.setAttribute("user.id", req.user.id);
span.setAttribute("cart.total", cart.total);
span.setAttribute("cart.items", cart.itemCount);
span.setAttribute("payment.method", cart.paymentMethod);
// ... rest of handler
});
```
## Pattern: Wrapping Business Logic in Custom Spans
Create spans around operations you want to see in the trace waterfall.
> The `RecordError`/`record_exception` calls in the compatibility snippets below show the
> legacy span-event exception API. For new code, prefer the **Exception Events with the Logs API**
> pattern below; retain the legacy call only when required by an existing SDK, query, or migration.
### Go
```go
func processPayment(ctx context.Context, order *Order) error {
tracer := otel.Tracer("checkout-service")
ctx, span := tracer.Start(ctx, "process-payment")
defer span.End()
span.SetAttributes(
attribute.String("order.id", order.ID),
attribute.Float64("order.total", order.Total),
attribute.String("payment.provider", order.PaymentProvider),
)
result, err := paymentGateway.Charge(ctx, order)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
span.SetAttributes(attribute.String("payment.transaction_id", result.TransactionID))
return nil
}
```
### Python
```python
def process_payment(order):
tracer = trace.get_tracer("checkout-service")
with tracer.start_as_current_span("process-payment") as span:
span.set_attribute("order.id", order.id)
span.set_attribute("order.total", order.total)
span.set_attribute("payment.provider", order.payment_provider)
try:
result = payment_gateway.charge(order)
span.set_attribute("payment.transaction_id", result.transaction_id)
except Exception as e:
span.record_exception(e)
span.set_status(StatusCode.ERROR, str(e))
raise
```
## Pattern: Exception Events with the Logs API
For new exception instrumentation, emit a Logs API record while the relevant span is active.
Use the standard `exception.*` fields, ERROR severity, and `event.name="exception"`. Set span
status and low-cardinality span dimensions separately so humans can aggregate operation failures
without copying stack traces onto spans.
```text
log event (while span is active):
event.name = "exception"
body = "exception"
severity = ERROR
exception.type, exception.message, exception.stacktrace, exception.escaped
containing span:
status = ERROR
error = true
exception.slug = "err-payment-provider-timeout" # static, optional, low-cardinality
```
Honeycomb correlates the log to the active trace and renders it as a `span_event` annotation.
The event row carries `trace.trace_id` and `trace.parent_id`, but Logs API exception fields are
not hoisted onto the containing span. Query the event row for the diagnostic payload, then use
its `trace.trace_id` with `get_trace(show_events=true)` to inspect the surrounding trace.
The legacy `record_exception` / `RecordError` APIs remain useful for compatibility and existing
instrumentation. Do not assume their `name=exception` shape applies to Logs API events: Logs API
uses `event.name=exception` (and often `body=exception`) with `meta.signal_type=log`.
**SDK implementation note:** use the language's supported Logs API or logging bridge, and verify
that the record carries the active context. Python's current app-facing path is the stdlib
`logging` bridge; Go passes the active context explicitly to `logger.Emit(ctx, record)`; Node
requires an installed context manager and an explicit active context; Java attaches context from
`span.makeCurrent()`. Do not emit the exception after the span scope closes.
### Optional compatibility pattern: exception-promoting LogRecordProcessor
Honeycomb's historical span-event path can promote exception fields onto the parent span. A
trace-correlated Logs API exception does not receive that promotion automatically. If an existing
span-oriented query surface must be preserved, implement a custom **LogRecordProcessor**, registered
before the batch/export processor:
```text
on_emit(log_record, resolved_context):
if log_record.event_name != "exception":
return
span = span_from_context(resolved_context)
if span is absent or not recording:
return
# Promote only fields needed by span-level queries.
span.set_attribute("error", true)
span.set_attribute("error.type", log_record["exception.type"])
span.set_attribute("exception.type", log_record["exception.type"])
if log_record has "exception.slug":
span.set_attribute("exception.slug", log_record["exception.slug"])
if configured for legacy compatibility:
span.set_attribute("exception.message", log_record["exception.message"])
span.set_attribute("exception.stacktrace", log_record["exception.stacktrace"])
# Leave the log record unchanged; it remains the diagnostic source of truth.
```
The processor should be synchronous and must run while the span is still mutable. It should no-op
when there is no valid recording span, preserve the original log record, and never invent fields
that were not emitted. Treat `exception.message` and especially `exception.stacktrace` promotion
as an explicit compatibility option because they are high-cardinality and potentially large.
This is not a standalone `SpanProcessor`: span processors do not receive log records. A span
processor could only participate through a separate shared registry, which adds races, cleanup,
and lifecycle complexity. The exact processor and context APIs are language-specific:
- Go: use the explicit context passed to `logger.Emit(ctx, record)`.
- Java: use the resolved log context; do not reconstruct context after async queueing.
- Node.js: ensure the context manager is installed and use the record's explicit context.
- Python: preserve the active context through the stdlib logging bridge and Logs SDK processor.
Use this only as a migration aid. Agents should still query Logs API exception rows for full
`exception.*` diagnostics and treat promoted span fields as instrumentation-dependent.
## Pattern: Recording Events Within a Span
For non-exception milestones and state changes, use the Logs API for new point-in-time events
when the language SDK supports it. Keep legacy span events where the SDK has no usable Logs API
or compatibility requires them:
```python
with tracer.start_as_current_span("process-order") as span:
span.add_event("validating_order", {"order.id": order.id})
if not validate(order):
span.add_event("validation_failed", {"reason": "invalid_address"})
raise ValidationError()
span.add_event("charging_payment", {"amount": order.total})
charge(order)
span.add_event("order_completed", {"order.id": order.id})
```
## Pattern: Linking Related Traces
When an async job is triggered by a request, link them:
```python
# In the message consumer:
from opentelemetry.trace import Link
def process_message(message):
# Extract the producing span's context from the message
producer_context = extract_context(message.headers)
with tracer.start_as_current_span(
"process-message",
links=[Link(producer_context, {"link.reason": "triggered_by"})],
) as span:
span.set_attribute("message.id", message.id)
# ... process message
```
## Pattern: Timing Attributes on Parent Spans
Put important sub-operation durations as attributes on the parent span instead of
creating child spans for everything.
> **Anti-pattern warning:** Wrapping absolutely everything in its own span is the most
> common failure mode when engineers first get access to tracing tools. You have to
> design the structure of your data for the way you want to query it.
Child spans are helpful for waterfall visualization of a single request, but they're
difficult to query across *all* requests. Timing attributes on a single span are
easier to query and work directly with tools like BubbleUp — which can immediately
surface "that group of requests was slow because authentication took 10 seconds."
### Go
```go
func handleRequest(w http.ResponseWriter, r *http.Request) {
span := trace.SpanFromContext(r.Context())
// Time authentication
authStart := time.Now()
user, err := authenticate(r)
authDur := time.Since(authStart)
span.SetAttributes(attribute.Float64("auth.duration_ms", float64(authDur.Milliseconds())))
// Time payload parsing
parseStart := time.Now()
payload, err := parsePayload(r)
parseDur := time.Since(parseStart)
span.SetAttributes(attribute.Float64("payload_parse.duration_ms", float64(parseDur.Milliseconds())))
// ... rest of handler
}
```
### Python
```python
@app.route("/api/resource", methods=["POST"])
def handle_request():
span = trace.get_current_span()
# Time authentication
auth_start = time.monotonic()
user = authenticate(request)
span.set_attribute("auth.duration_ms", (time.monotonic() - auth_start) * 1000)
# Time payload parsing
parse_start = time.monotonic()
payload = parse_payload(request)
span.set_attribute("payload_parse.duration_ms", (time.monotonic() - parse_start) * 1000)
# ... rest of handler
```
### Node.js
```javascript
app.post("/api/resource", async (req, res) => {
const span = trace.getActiveSpan();
// Time authentication
const authStart = performance.now();
const user = await authenticate(req);
span.setAttribute("auth.duration_ms", performance.now() - authStart);
// Time payload parsing
const parseStart = performance.now();
const payload = await parsePayload(req);
span.setAttribute("payload_parse.duration_ms", performance.now() - parseStart);
// ... rest of handler
});
```
**When to use this pattern:**
- The operation is important to understanding request latency
- You want to GROUP BY or BubbleUp on the timing alongside other parent span attributes
- The alternative (a child span) would require JOINs for cross-request analysis
**When a child span is still better:**
- The operation makes downstream calls you also want to trace
- You need to see the operation in the waterfall view for single-request debugging
- The operation has its own rich set of attributes worth capturing
## Pattern: Exception Slugs
Tag each error throw site with a unique static string (`exception.slug`). This creates
a low-cardinality, greppable identifier that connects dashboards directly to code. Keep this
attribute on the operation span even when full exception details are emitted as a Logs API event.
The legacy `RecordError`/`record_exception` calls shown in this section are compatibility examples;
they are not a requirement for new Logs API instrumentation.
### Go
```go
func processPayment(ctx context.Context, order *Order) error {
span := trace.SpanFromContext(ctx)
result, err := stripe.Charge(ctx, order)
if err != nil {
// Static string — not dynamically generated
// Consider enforcing this with custom lint rules
span.SetAttributes(
attribute.String("exception.slug", "err-stripe-charge-failed"),
attribute.Bool("error", true),
)
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
if !result.Approved {
span.SetAttributes(
attribute.String("exception.slug", "err-payment-declined"),
attribute.Bool("error", true),
)
return ErrPaymentDeclined
}
return nil
}
```
### Python
```python
def process_payment(order):
span = trace.get_current_span()
try:
result = stripe.charge(order)
except stripe.CardError as e:
span.set_attribute("exception.slug", "err-stripe-card-error")
span.set_attribute("error", True)
span.record_exception(e)
span.set_status(StatusCode.ERROR, str(e))
raise
except stripe.APIError as e:
span.set_attribute("exception.slug", "err-stripe-api-unavailable")
span.set_attribute("error", True)
span.record_exception(e)
span.set_status(StatusCode.ERROR, str(e))
raise
```
### Node.js
```javascript
async function processPayment(order) {
const span = trace.getActiveSpan();
try {
const result = await stripe.charges.create(order);
if (!result.approved) {
span.setAttribute("exception.slug", "err-payment-declined");
span.setAttribute("error", true);
throw new PaymentDeclinedError();
}
} catch (err) {
if (!span.attributes?.["exception.slug"]) {
span.setAttribute("exception.slug", "err-stripe-call-failed");
}
span.setAttribute("error", true);
span.recordException(err);
span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
throw err;
}
}
```
**Why this pattern matters:**
- **Greppable:** Search your codebase for the exact slug string to find the throw site
- **Low-cardinality GROUP BY:** Safe to use in `GROUP BY exception.slug` queries
- **Gap detection:** Any failed request *without* an `exception.slug` reveals places
where your error handling could be improved — it's easy to find errors you didn't
anticipate
**Query — find unhandled errors (missing slugs):**
```
VISUALIZE COUNT
WHERE error = true AND exception.slug = NULL
GROUP BY http.route
```
## Pattern: Async Request Summaries
Roll up child operation statistics onto the parent span to identify outlier requests
without needing to count child spans manually.
### Go
```go
type RequestStats struct {
mu sync.Mutex
pgQueryCount int
pgQueryDurMs float64
httpReqCount int
httpReqDurMs float64
}
func (s *RequestStats) RecordPgQuery(dur time.Duration) {
s.mu.Lock()
defer s.mu.Unlock()
s.pgQueryCount++
s.pgQueryDurMs += float64(dur.Milliseconds())
}
func (s *RequestStats) RecordHTTPReq(dur time.Duration) {
s.mu.Lock()
defer s.mu.Unlock()
s.httpReqCount++
s.httpReqDurMs += float64(dur.Milliseconds())
}
func (s *RequestStats) SetOnSpan(span trace.Span) {
s.mu.Lock()
defer s.mu.Unlock()
span.SetAttributes(
attribute.Int("stats.postgres_query_count", s.pgQueryCount),
attribute.Float64("stats.postgres_query_duration_ms", s.pgQueryDurMs),
attribute.Int("stats.http_requests_count", s.httpReqCount),
attribute.Float64("stats.http_requests_duration_ms", s.httpReqDurMs),
)
}
```
### Python
```python
class RequestStats:
def __init__(self):
self.pg_query_count = 0
self.pg_query_duration_ms = 0.0
self.http_req_count = 0
self.http_req_duration_ms = 0.0
def record_pg_query(self, duration_ms):
self.pg_query_count += 1
self.pg_query_duration_ms += duration_ms
def record_http_request(self, duration_ms):
self.http_req_count += 1
self.http_req_duration_ms += duration_ms
def set_on_span(self, span):
span.set_attribute("stats.postgres_query_count", self.pg_query_count)
span.set_attribute("stats.postgres_query_duration_ms", self.pg_query_duration_ms)
span.set_attribute("stats.http_requests_count", self.http_req_count)
span.set_attribute("stats.http_requests_duration_ms", self.http_req_duration_ms)
# Usage in a request handler:
@app.route("/api/resource")
def handle():
stats = RequestStats()
# Pass stats to DB and HTTP client wrappers...
# At end of request:
stats.set_on_span(trace.get_current_span())
```
**Why this pattern matters:**
- A request that makes 742 database queries is almost certainly doing something wrong
- Without summary stats, these outliers are invisible — you'd need to count child spans
per trace manually
- HEATMAP of `stats.postgres_query_count` instantly reveals bimodal distributions
and outliers
**Query — database queries per request:**
```
VISUALIZE HEATMAP(stats.postgres_query_count)
WHERE service.name = "api-service"
```
## Attribute Naming Best Practices
- Use dot-separated namespaces: `user.id`, `order.total`, `cache.hit`
- Follow OTel semantic conventions where they exist
- Create your own namespace for custom attributes: `app.`, `mycompany.`
- Keep attribute values low-cardinality where possible (for GROUP BY)
- High-cardinality is fine for debugging (trace IDs, user IDs, order IDs)
SHA-256: 7bc2480e76f2e35b6d295fb4bf4637b3c93a34d9068e975ae7607a9fa4d6cabd