SLO-Driven Deployments: Shipping Without Fear

Most teams treat reliability and delivery as separate concerns. The SRE team owns the dashboards; the platform team owns the pipeline. The result is a tug-of-war where speed and stability appear to be in opposition. They aren’t. The missing piece is a deployment policy that reads from the same telemetry as your on-call engineer.

Error budgets are a policy, not a poster

An SLO of 99.9% availability over 30 days gives you an error budget of roughly 43 minutes. That budget is spendable. When you have budget, deploys are unrestricted. When you’re burning it faster than you earn it, the pipeline should slow you down — automatically, without a Slack argument.

The trick is making the budget readable by the deployment system, not just by humans staring at Grafana.

Wiring the budget into the pipeline

We expose the current error budget burn rate as a metric the CI runner can query before promoting a canary. If the burn rate for the target service exceeds 2x its steady-state budget, the canary is not promoted and the release is held.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: slo-burn-gate
spec:
  args:
    - name: service-name
  metrics:
    - name: error-budget-burn
      interval: 1m
      successCondition: result[0] < 2.0
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              service="{{args.service-name}}",
              status=~"5.."
            }[5m]))
            /
            on(service)
            (0.001 * sum(rate(http_requests_total{
              service="{{args.service-name}}"
            }[5m])))

Canary analysis is a question, not a verdict

A canary that passes is not proof the release is good — it’s evidence the release is not obviously bad on the signals you chose. Choose your signals carefully. We default to RED metrics (Rate, Errors, Duration) plus a business KPI. A deployment can keep p99 latency flat while quietly tanking conversion; the KPI catches what the infra metrics miss.

• Rate: requests per second, compared against the canary baseline.
• Errors: 5xx ratio and application-level error counters.
• Duration: p99 and p95 latency, not just the mean.
• Business KPI: the one metric your CEO would miss if it dropped.

A canary that passes is not proof the release is good — it’s evidence the release is not obviously bad.

The cultural shift

The hardest part is not the tooling. It’s the conversation where you tell a team that their deploy was held by a machine, not a person. Frame the error budget as spendable currency: when you’re under budget, no one asks questions; when you’re over, the system — not a gatekeeper — asks you to slow down. Engineers stop fighting the process because the process is legible.

Once the policy is automated, speed and stability stop being in tension. They become the same graph, read from different axes.