system-design · intermediate

SLIs, SLOs, and Error Budgets — Measure Reliability Like a Product

Start here

Teams argue endlessly about whether the service is “stable enough” until they share numbers tied to user experience.

Three linked ideas solve that:

1. Service Level Indicator (SLI) — a carefully defined measurement of success from the user’s point of view.
2. Service Level Objective (SLO) — a target for that measurement over a time window (for example 99.9% over 28 days).
3. Error budget — the amount of allowed failure implied by the SLO. When the budget is spent, the team prioritizes reliability over risky launches.

This lesson builds on availability and latency vs throughput. It does not require you to work as a Site Reliability Engineer (SRE) already.

What you will learn

1. Define SLI, SLO, and error budget in plain English.
2. Pick user-centric indicators instead of vanity CPU charts.
3. Set a target and window, then compute the budget.
4. Use burn rates and alerts that fire before the month is ruined.
5. Work a complete checkout example with numbers.
6. Connect budgets to release policy without becoming a pure freeze cult.
7. Avoid averages that hide painful tails.

What you should know first

Words you need before we begin

Simple story: the city bus promise

A bus company publishes:

“95% of buses depart within 5 minutes of the schedule, measured over each calendar month.”

• SLI: fraction of trips that departed within 5 minutes.
• SLO: 95% for the month.
• Error budget: 5% of trips may be late. If they burn that in week one, they stop adding new routes and fix operations.

The problem without SLIs and SLOs

Opinion-based reliability

Engineering says “we were fine.” Support says “checkout is broken every Monday.” Leadership hears two stories and picks the louder one.

Metric theater

Dashboards full of CPU, memory, and request counts. None answer: did the user complete the job they came for?

Feature freeze only after disaster

Teams ship until a major outage forces a panic freeze. There is no graduated response when reliability is eroding.

SLOs create a shared dial: green budget → normal change; burning budget → slow down and harden.

Step-by-step explanation

Step 1 — Choose a user journey, not a machine

Bad SLI: “CPU < 70%.”
Users do not buy CPU headroom.

Better SLI candidates:

• Checkout success (non-5xx, money captured, order id returned)
• Login success within 2 seconds
• Video start within 3 seconds

Step 2 — Write the SLI as a fraction

Common form:

\[
\text{SLI} = \frac{\text{good events}}{\text{valid events}}
\]

You must define valid:

• Exclude client 400s caused by user typos if those are not your failure.
• Exclude load tests if they are not user traffic.
• Include timeouts that the user experienced as failure.

Among authenticated checkout attempts that passed client-side validation, the fraction that returned HTTP 2xx with an orderId within 300ms at the API edge or completed successfully under 300ms—pick one precise definition and stick to it.

For a first lesson, use:

good = response is 2xx, body contains orderId, and server duration ≤ 300ms valid = requests to POST /checkout that are authenticated and not load-test tagged

Step 3 — Set an SLO with a window

Examples:

• 99.9% good events over a rolling 28 days
• 99% of jobs complete within 5 minutes over calendar month

Step 4 — Compute the error budget

If SLO = 99.9% over 28 days and you expect 2,000,000 valid checkouts:

• Allowed bad events ≈ \(0.001 \times 2{,}000{,}000 = 2{,}000\)

Step 5 — Alert on burn, not only on “SLO failed at month end”

If you only page when the 28-day SLO is already breached, you learn too late.

Burn alerts ask: at the current failure rate, will we exhaust the budget too quickly?

Simple intuition: a one-hour outage that spends a huge fraction of monthly budget should page immediately, even if the 28-day number still looks “okay” for a few days.

Step 6 — Attach a policy to the budget

Examples of policy (customize to culture):

This is not punishment. It is explicit product prioritisation.

Step 7 — Separate SLOs from SLAs

• SLO: internal target guiding engineering.
• SLA: legal/customer contract, often looser than internal SLO so you have margin.

Visual mental model

flowchart TB
  U[User journeys] --> SLI[SLI definitions]
  SLI --> SLO[SLO targets + windows]
  SLO --> EB[Error budget]
  EB --> Policy[Release and priority policy]
  Policy --> Work[Features vs reliability work]
  Work --> U
  SLI --> Alert[Burn alerts]
  Alert --> Policy

Learning question: If the budget is empty, what should happen to a non-critical redesign launch?

Caption: SLIs measure; SLOs target; budgets decide pace.

Complete worked example: notes app sync API

Starting situation

A notes app sync API (POST /sync) must feel reliable for freelancers who write during client meetings.

Product promise (marketing, not yet engineering): “your notes are basically always available.”

Constraints

• ~5,000,000 valid sync requests per 28-day window
• Team of 8 engineers shipping weekly
• Dependencies: auth service, primary database
• Cannot afford full freeze every week

Decisions

Execution notes

1. Instrument SLI at the edge with consistent labels (valid=true/false).
2. Build a dashboard: current SLI, budget remaining, recent burn.
3. Run a game day: kill a DB replica; confirm burn alert fires.
4. Write the policy in the team handbook so product managers see it before quarter planning.

Failure behavior

A bad deploy causes 2% failure for 30 minutes at peak (high QPS). Budget spent in that half hour may equal days of normal noise. Policy: rollback first, then post-incident, then reliability tasks before the next feature train.

Outcome

The team ships fewer “optional” migrations during warm budget weeks and stops arguing from anecdote. Limitations: the SLI ignores client bugs and offline mobile queues—those need separate product metrics.

How it works in production

Ownership

• Service owner defines SLIs/SLOs with product.
• Platform/SRE often helps with measurement pipelines and multi-window burn alerts.
• On-call responds to burn pages with rollback/mitigation runbooks.

Good operations

• SLI event pipelines that survive partial dashboard outages
• Multi-window burn alerts (short window + long window)
• Error budget reports in weekly ops review
• Capacity and load-shedding plans when latency SLIs degrade under load
• Production readiness reviews that ask “what is the SLI for this new path?”

Anti-patterns

• 20 SLOs nobody reads
• SLOs on internal queue depth with no user mapping
• Paging on raw CPU
• Setting 99.99% because it “sounds enterprise” without staffing

Failure modes

Trade-offs

Compare with related concepts

Common misunderstandings

1. “SLO means zero downtime.”

1. “We have metrics, so we have SLOs.”

1. “Burning budget means people failed.”

1. “99.99% is always better.”

1. “Latency averages are fine.”

Check your understanding

1. Define SLI, SLO, and error budget without acronyms.
2. Why is “CPU low” a weak SLI for checkout?
3. If SLO is 99.9% and you have 1,000,000 valid events, roughly how many failures are budgeted?
4. What is a burn alert for?
5. How does an SLA differ from an SLO?

Practice

1. Write an SLI for “user can open their most recent note.”
2. Pick 99% vs 99.9% for a hobby blog vs a payments API; justify.
3. Draft a three-row budget policy table for your team.
4. Given 10,000 bad events allowed and an incident that caused 4,000, what conversations should happen next week?
5. Critique this SLO: “Average latency under 200ms.”

Revision summary

• SLI measures user-visible success.
• SLO sets a target over a window.
• Error budget is the allowed failure implied by the SLO.
• Alert on budget burn, not only final breach.
• Tie budgets to change policy with product agreement.
• Prefer a few journey-based SLIs over metric sprawl.

Glossary

Abbreviations and terminology

• SLI — Service Level Indicator
• SLO — Service Level Objective
• SLA — Service Level Agreement
• SRE — Site Reliability Engineering
• p95 / p99 — 95th / 99th percentile

What to learn next

1. Availability — nines and downtime math.
2. Observability and DORA — signals and delivery metrics.
3. Incident command — when burn becomes an incident.
4. Production readiness reviews — ship gates.
5. Tail latency and load shedding — protect SLIs under load.

FAQ from first-time learners

Q: Who sets the SLO number?
A: Product and engineering together. Engineering explains cost; product explains user need.

Q: Do internal tools need SLOs?
A: If people depend on them to ship, yes—often looser than customer-facing paths.

Q: What if dependencies cannot support our SLO?
A: Your SLO cannot exceed what dependencies realistically allow without isolation, caching, or graceful degradation. Design or renegotiate.

Track: Reliability and Operations

Series: Reliability & SRE Practice

1. Availability — Nines, Error Budgets, and Redundancy
2. SLIs, SLOs, and Error Budgets — Measure Reliability Like a Product (this guide)
3. Capacity Planning for Backend Services
4. Tail Latency and Load Shedding — Surviving Peak Traffic Overload
5. Production-Readiness Reviews (PRRs)
6. Incident Command for Backend Teams

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab