← All posts

Notes on Debugging Latency in Distributed Systems

·2 min read·
distributed systemsperformancedebugging

Last quarter we had a latency regression that took three people two days to fully understand. p50 was fine. p99 hit 300ms on certain write paths. This is a write-up of the approach that actually worked.

The problem with latency

Latency bugs are hard because they’re probabilistic. The 99th percentile means you’re looking for something that happens 1% of the time — and often only in production, under real load, with real data distributions.

Our first instinct was to add more logging. That helped marginally. What actually cracked it open was a disciplined top-down methodology.

Start at the edge

The first question is always: where does the time go? We used distributed tracing (Jaeger) to get a full picture of one slow request. The span breakdown showed:

  • HTTP ingress: 2ms
  • Auth middleware: 4ms
  • Business logic: 11ms
  • Database: 287ms ←

The database was obviously the culprit. But which query? We had hundreds of them in this service.

Isolating the slow query

-- pg_stat_statements, sorted by mean execution time
SELECT
  query,
  calls,
  mean_exec_time,
  total_exec_time,
  rows
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;

The query that surfaced was a seemingly innocent SELECT ... WHERE user_id = $1 ORDER BY created_at DESC LIMIT 10. Under load it was doing a seq scan on a 40M row table because the index wasn’t covering the ORDER BY column.

-- Before: index only on user_id
CREATE INDEX idx_events_user ON events (user_id);

-- After: composite index including sort column
CREATE INDEX idx_events_user_created ON events (user_id, created_at DESC);

Creating the index concurrently brought p99 from 300ms to 18ms in under two minutes.

What I’d do differently next time

  1. Add query-level tracing at the ORM layer before a regression happens. Don’t wait for production to tell you which query is slow.
  2. Use histogram metrics, not averages. Average latency hides everything. Export a histogram with buckets at p50/p75/p95/p99/p999.
  3. Don’t trust EXPLAIN alone — EXPLAIN ANALYZE with real production data tells a different story.

The fix took five minutes. Finding it took two days. That ratio is a smell — it means the observability layer was under-invested. We’ve since added automatic slow query logging (log_min_duration_statement = 100) and alerting on p99 directly rather than just average.

Takeaway

Latency bugs are an observability problem before they’re a code problem. When you can’t see where time goes, you’re guessing. Invest in distributed tracing, histogram metrics, and query observability before the incident, not during it.