Library · Operations

Observability: knowing before the customer does.

By Mokshify Engineering · Reviewed by the team · Updated 17 Jul 2026 · 6 min read

The problem

Monitoring tells you a thing you predicted might break, broke. Observability is being able to ask questions you did not predict - why is checkout slow for one tenant, on one endpoint, since one deploy? The 99.98% uptime we publish is a monitoring number; the ability to keep it is observability.

The three signals, prioritised honestly

Structured logs first. JSON lines with request id, tenant id, route, duration, outcome - centralised and queryable. For a product-scale system this answers 80% of unpredicted questions, at the lowest cost. Policy: secrets and personal data are scrubbed at the emitter, never at the viewer (security practices).
Metrics for the dashboards that gate. Rate, errors, duration per service; queue depth and job failures (the async half); cache hit rates (the caching half); cost per service so bills cannot drift silently.
Traces when architecture earns them. Distributed tracing pays off with many services; a modular monolith gets most of the value from a request id threaded through logs. Adopt tracing when the request path actually spans processes - not because a vendor diagram looked incomplete.

Alerting that respects humans

Page on symptoms, not causes - users cannot log in, error budget burning - so one incident is one page, not forty.
Every alert has an owner and an action. An alert nobody can act on is noise wearing a siren; we delete those in review.
Budgets over thresholds. An error budget burning fast pages at 2am; burning slowly makes Monday’s agenda. This distinction is most of alert sanity.

Advantages & disadvantages

Advantages: incidents found by systems instead of customers; post-incident notes (blameless, published to clients) built from evidence, not memory; performance work targeted by data. Disadvantages: log volume is a real bill - retention tiers are a decision, not a default; instrumentation is code that must be maintained; dashboards rot unless they are used in reviews - ours are, weekly.

Technology selection

Cloud-native collectors and managed log stores by default (least ops), OpenTelemetry conventions where tracing is warranted (portability, per the same logic as our IaC), and the discipline that every service ships with its dashboard on day one - monitoring before users.