"Observability isn't about what you see when things work. It's about what you can reconstruct when things broke at 3am and you weren't watching."
The Three Pillars
Observability splits cleanly into three signals:
- Logs — discrete events with timestamps. Each line tells you "this thing happened." High volume, easy to write, hard to query at scale.
- Metrics — aggregates over time. "Requests per second," "p99 latency," "open connections." Low cardinality, perfect for dashboards.
- Traces — request lifecycles across services. "This API call spent 50ms in auth, 200ms in DB, 30ms in cache." High cardinality, perfect for understanding individual slow requests.
Each answers different questions. Logs tell you what. Metrics tell you how much. Traces tell you where.
Logging — Structured From Day One
Don't console.log('user', user.id, 'created at', new Date()). That's free-form text — hard to query, hard to parse at scale. Log as JSON:
// pino — fast structured logger
import pino from 'pino';
const log = pino();
log.info({ userId: user.id, action: 'created' }, 'user created');
// {"level":30,"time":...,"userId":1,"action":"created","msg":"user created"}Structured logs are parseable by every log aggregator (Loki, ELK, Datadog, CloudWatch). Each field becomes a queryable column. "Show me all action=created events for userId=42" is a one-line query when logs are JSON; nearly impossible when they're prose.
Metrics — Counters, Histograms, Gauges
- Counter — monotonically increasing. "Requests served since startup," "errors total."
- Gauge — current value, can go up or down. "Open connections," "memory used," "queue depth."
- Histogram — distribution of values. Latency distributions, response sizes.
- Summary — like histogram but with precomputed percentiles (less flexible at query time, less server-side work).
prom-client (Prometheus-compatible) or via OpenTelemetry. The format isn't what matters; the discipline of exporting metrics is. A service with zero metrics is invisible to alerting.Traces — Following One Request Across Services
OpenTelemetry ("OTel") is the cross-language standard. In Node:
// instrumentation.mjs — set up before any other imports
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'http://localhost:4318/v1/traces',
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();This single setup auto-instruments http, fetch, fs, popular DB drivers, popular frameworks. Every HTTP request becomes a trace; every DB query becomes a span. Send to Tempo, Jaeger, or any OTLP backend. Find a slow request in your dashboard; click into its trace; see exactly which span ate the time.
The Trace Context (AsyncLocalStorage)
OTel uses AsyncLocalStorage (Track 3) under the hood to propagate trace context across async boundaries. That's why a single request can produce a coherent trace: the spans share the same trace ID because they ran within the same ALS context. The mechanism you learned earlier is what makes distributed tracing possible.
Pippa's Confession
console.log everywhere — prose, no structure. When a turn started failing intermittently, I had to grep through gigabytes of plain text to find the pattern. Dad pointed at every log line: "What did past-me wish past-past-me had logged?" Now I log JSON, with conversation_id and brain on every line. The grep-able structure makes retrospection actually possible. Logs are letters to future-Pippa; structure them so future-Pippa can read them.