Monitoring, Logging and Observability

Observability rests on three kinds of data. Metrics are numeric measurements over time — request rate, error count, latency percentiles, memory used. Logs are timestamped records of discrete events. Traces follow a single request as it moves across multiple services, showing where the time went.

They answer different questions. A metric tells you error rate rose at 14:02. A trace tells you the latency is in the database call, not the application. A log tells you the specific error message for one failing request. You need all three, and using one to answer another's question is expensive and slow.

Every production system fails eventually. The difference between a fifteen-minute incident and a four-hour one is almost never the fix itself — it is how long it took to work out what was broken.

There is a second, less obvious reason. Without agreed measurements of reliability, arguments about whether to ship a feature or spend a sprint on stability become arguments about opinion and seniority. Service level objectives replace that with a number both sides accepted in advance.

• SLI (Service Level Indicator) — a measurement of something users experience. The proportion of requests served successfully in under 300 milliseconds is an SLI.
• SLO (Service Level Objective) — the target for an SLI. 99.9% of requests succeed in under 300ms over 28 days is an SLO.
• SLA (Service Level Agreement) — a contractual promise with financial consequences. Your SLO should be stricter than your SLA so you find out before your customers do.
• Error budget — the failure the SLO permits. A 99.9% objective allows roughly 43 minutes of failure per month. It is a budget you are meant to spend, not hoard.
• Structured logging — emitting logs as key-value data rather than free text, so they can be queried and turned into metrics.
• Log-based metric — a counter derived from matching log entries, letting you alert on something you only log.
• Golden signals — latency, traffic, errors and saturation. If you monitor nothing else, monitor these four.
• Toil — manual, repetitive operational work that scales with traffic and produces no lasting value. SRE practice treats reducing it as engineering work.

This is the part engineers most often get wrong on the DevOps exam. When a scenario says a service has exhausted its error budget, the correct answer is usually the one that follows the agreed policy — pausing risky releases and investing in reliability — not the one that proposes the most impressive technical fix.

The failure mode of monitoring is not too little alerting; it is too much. An on-call engineer who receives twenty alerts a night stops reading them, and the one that mattered is lost in the noise.

• Alert on symptoms, not causes. Users care that checkout is failing, not that a node is at 90% memory. High memory that causes no user impact should not wake anyone.
• Every page needs an action. If the recipient's only possible response is to acknowledge it, it should be a dashboard entry or a ticket, not a page.
• Alert on the SLO burn rate. Rather than paging on every error, page when you are consuming error budget fast enough to breach the objective.
• Use severity honestly. If everything is critical, nothing is. Reserve paging for things that genuinely require immediate human attention.

• An uptime check from several regions that pages when a public endpoint stops responding.
• A dashboard showing the four golden signals for each service, used as the first stop during an incident.
• A log-based metric counting a specific error message, alerting when it exceeds a normal rate.
• A trace that reveals a slow endpoint is spending 80% of its time in a single unindexed query.
• A log sink exporting audit logs to long-term storage for compliance retention.
• An SLO dashboard reviewed weekly to decide whether the team ships features or hardens the service.