← All blogs

FullStack Engineering CourseLesson 16 of 20 18 min

Observability and Reliability: OpenTelemetry, SLOs and Disaster Recovery

Monitoring and observability

Monitoring answers questions you thought of in advance: is CPU high, is the error rate above 1%? Observability is the ability to answer new questions about a running system from the data it produces, such as "why are claims from one payer slow since 10:40, but only for one tenant?". A system is observable when it emits enough detailed, connected data to investigate problems nobody predicted.

That data comes as three main signals:

SignalWhat it isAnswersExample
LogsTimestamped records of individual eventsWhat exactly happened in this case?"Claim 42 rejected by payer P7: missing diagnosis code"
MetricsNumbers measured over time, aggregatedHow much, how often, how fast, overall?Requests per second, 95th percentile latency, queue length
TracesThe path of one request through all the services it touched, with timingsWhere did the time go, and which service failed?A claim submission: API 40 ms, eligibility call 900 ms, database 15 ms

The signals are most useful when they are connected. Every log line and every metric sample should carry the trace ID of the request it belongs to, so you can jump from an alert (metric) to a slow request (trace) to the exact error (log).

Structured logging

A structured log stores values as named fields instead of only as text. In .NET, write log messages as templates with named placeholders, not with string interpolation:

C#
// Structured: ClaimId, PayerCode and Amount are stored as separate, searchable fields.
logger.LogInformation("Claim {ClaimId} submitted to payer {PayerCode} for {Amount}", claim.Id, payerCode, claim.Amount);

// Not structured: one opaque string, and the formatting runs even when the level is disabled.
logger.LogInformation($"Claim {claim.Id} submitted to payer {payerCode} for {claim.Amount}");

// Hot paths: source-generated logging avoids boxing and parsing the template at run time.
public static partial class Log
{
    [LoggerMessage(Level = LogLevel.Warning, Message = "Payer {PayerCode} rejected claim {ClaimId}: {Reason}")]
    public static partial void PayerRejected(ILogger logger, string payerCode, int claimId, string reason);
}

Guidelines:

  • Use levels consistently: Error for failures that need attention, Warning for unexpected but handled situations, Information for important business events, Debug and Trace for detail you turn on only when investigating.
  • Add context once with scopes (the correlation ID middleware in lesson 4), so every line in a request carries it.
  • Never log secrets, tokens, passwords or full personal and health data. Log IDs, not contents, and mask anything sensitive. Logs are copied widely and kept for a long time.
  • Logs cost money to collect and store. Keep Information-level logs meaningful, sample very noisy logs, and set retention periods.
  • Serilog is a popular library for structured logging with many outputs (sinks); with OpenTelemetry, the built-in ILogger can export to the same backends.

Metrics

Metrics are cheap to store and fast to query, which makes them the basis for dashboards and alerts. The main types:

  • Counter: a number that only goes up, such as claims submitted or errors. You look at its rate (per second).
  • Histogram: a distribution of values, such as request durations, from which percentiles are calculated.
  • Gauge (an up-down counter or observable gauge in .NET): a current value that goes up and down, such as queue length or active connections.

Two simple frameworks tell you what to measure:

  • RED, for every service and endpoint: Rate (requests per second), Errors (failed requests per second) and Duration (latency percentiles).
  • USE, for every resource (CPU, memory, connection pools, queues): Utilisation (how busy), Saturation (how much work is waiting) and Errors.

Use percentiles, not averages, for latency. If 99 requests take 100 ms and one takes 10 seconds, the average (about 200 ms) looks fine while one user in a hundred waits 10 seconds. The 95th and 99th percentiles (p95, p99) show what the slowest users experience.

C#
// Custom business metrics with the .NET Meter API.
public sealed class ClaimMetrics
{
    private readonly Counter<long> _submitted;
    private readonly Histogram<double> _adjudicationSeconds;

    public ClaimMetrics(IMeterFactory meters)
    {
        var meter = meters.Create("Claims");
        _submitted = meter.CreateCounter<long>("claims.submitted", unit: "{claim}");
        _adjudicationSeconds = meter.CreateHistogram<double>("claims.adjudication.duration", unit: "s");
    }

    public void Submitted(string payerCode) =>
        _submitted.Add(1, new KeyValuePair<string, object?>("payer", payerCode));   // a tag with few possible values

    public void Adjudicated(TimeSpan duration) => _adjudicationSeconds.Record(duration.TotalSeconds);
}

Keep tag values to a small, fixed set (payer, endpoint, status code). Each unique combination of tag values creates a separate time series. Tags such as claim ID or user ID (high cardinality) create millions of series, which makes metrics systems slow and expensive. Put such IDs in logs and traces instead.

Distributed tracing

A trace follows one request across all services. It is made of spans: one span per unit of work (an incoming HTTP request, an outgoing call, a database query), each with a start time, a duration, a parent span and tags. The trace ID travels between services in the W3C traceparent HTTP header (and in message properties for queues), which is called context propagation.

In .NET, a span is an Activity, created from an ActivitySource. ASP.NET Core, HttpClient and most Azure SDKs create activities automatically; you add your own for important internal steps:

C#
public sealed class PricingService(IPayerGateway payer)
{
    private static readonly ActivitySource Source = new("Claims");

    public async Task<Money> PriceAsync(ClaimRecord claim, CancellationToken ct)
    {
        using Activity? activity = Source.StartActivity("PriceClaim");   // a child span of the current request
        activity?.SetTag("claim.id", claim.Id);
        activity?.SetTag("payer.code", claim.PayerCode);

        var contract = await payer.GetContractAsync(claim.PayerCode, ct);   // HttpClient adds its own child span
        var price = contract.Price(claim);

        activity?.SetTag("claim.price", price.Amount);
        return price;
    }
}

OpenTelemetry

OpenTelemetry (OTel) is the open, vendor-neutral standard for producing and exporting logs, metrics and traces. You instrument the code once, and send the data to any backend that supports the OpenTelemetry protocol (OTLP): Azure Monitor, Grafana, Jaeger, Elastic, Datadog and others. Changing the backend means changing configuration, not code.

C#
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("claims-api"))         // the service name shown in every backend
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()                         // incoming requests
        .AddHttpClientInstrumentation()                         // outgoing calls
        .AddSource("Claims")                                    // our own ActivitySource
        .AddOtlpExporter())
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()                            // GC, thread pool, memory (lessons 1 and 2)
        .AddMeter("Claims")                                     // our own Meter
        .AddOtlpExporter())
    .WithLogging(l => l.AddOtlpExporter());

// Or, to send everything to Azure Monitor / Application Insights with one call:
// builder.Services.AddOpenTelemetry().UseAzureMonitor();   // package Azure.Monitor.OpenTelemetry.AspNetCore

In larger setups, services send their data to an OpenTelemetry Collector: a separate process that receives, filters, samples and forwards telemetry to one or more backends. It keeps backend credentials and processing rules out of the application. Sampling (keeping only a share of traces, while always keeping errors and slow ones) controls cost at high volume.

.NET Aspire, Microsoft's stack for building distributed .NET apps, sets up OpenTelemetry by default and includes a local dashboard that shows logs, metrics and traces while you develop.

Backends: Azure Monitor, Prometheus and Grafana, ELK

BackendWhat it isQuery language
Azure Monitor / Application InsightsAzure's managed platform for logs, metrics and traces, with application maps, failure analysis and alertsKQL (Kusto Query Language)
PrometheusAn open-source metrics database that scrapes (pulls) metrics from servicesPromQL
GrafanaDashboards and alerts over many data sources (Prometheus, Loki for logs, Tempo for traces, Azure Monitor)Depends on the source
ELK / Elastic StackElasticsearch (storage and search), Logstash (processing) and Kibana (exploration), mainly for logsKQL (Kibana's own) and Lucene syntax
// Application Insights (KQL): claim submissions per 5 minutes, with p95 latency and failures
requests
| where timestamp > ago(1h) and name == "POST /api/v1/claims"
| summarize count(), p95 = percentile(duration, 95), failures = countif(success == false) by bin(timestamp, 5m)
| render timechart
# Prometheus (PromQL): p95 request duration for claims-api over the last 5 minutes
histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="claims-api"}[5m])))

Alerts that people trust

An alert should wake someone up only when users are affected or soon will be, and only when a person needs to act. Too many alerts teach people to ignore them (alert fatigue).

  • Alert on symptoms users feel (error rate, latency, failed claim submissions), not on every possible cause (CPU at 80%). Use causes on dashboards for investigation.
  • Make every alert actionable, and link it to a runbook (a short guide: what this alert means, how to check, how to fix or mitigate).
  • Route by urgency: page someone for user-facing outages; send everything else to a ticket or a channel for working hours.
  • Review alerts after every incident: remove the ones that fired without need, and add the ones that were missing.

SLIs, SLOs, SLAs and error budgets

Reliability targets turn "the system should be reliable" into numbers that guide decisions:

  • SLI (service level indicator): a measurement of user experience, such as the share of claim submissions that succeed within 1 second.
  • SLO (service level objective): the target for an SLI over a period, such as 99.9% of submissions succeed within 1 second, measured over 30 days.
  • SLA (service level agreement): a contract with customers, with penalties if it is missed. It should be looser than the SLO, so you notice problems before they become breaches.

The error budget is the amount of failure the SLO allows. A 99.9% target over a 30-day month (43,200 minutes) allows 0.1% of failures, about 43 minutes of full downtime or the equivalent in failed requests. The budget turns reliability into a shared decision:

  • While budget remains, the team ships features at normal speed, accepting some risk.
  • When the budget is nearly used up, the team slows releases and spends time on reliability work.
  • Alert on the burn rate (how fast the budget is being used), not on single errors. For example, page someone when the budget is being used 14 times faster than normal over the last hour, which would use it all in about two days.

Choose SLO targets from what users actually need. 100% is never the right target: it is impossible, and chasing it stops all change. Every extra nine (99.9% to 99.99%) costs much more in redundancy and engineering.

Incidents and postmortems

  • Define severity levels, so everyone knows how quickly to respond to what.
  • During an incident, give clear roles: an incident commander who coordinates, people who investigate, and one person who communicates with stakeholders.
  • Mitigate first, then find the root cause: roll back, fail over or switch off a feature flag to restore service, then investigate.
  • Afterwards, write a blameless postmortem: a timeline, the impact, the contributing causes, what went well, and action items with owners. Blameless means focusing on how the system and process allowed the failure, not on who made a mistake, so people report problems honestly.

Disaster recovery

Disaster recovery (DR) plans for losing a whole data centre or region, or for data being corrupted or deleted. Two numbers define what is needed:

  • RPO (recovery point objective): how much data may be lost, measured in time. An RPO of 5 minutes means you can lose at most the last 5 minutes of changes.
  • RTO (recovery time objective): how long the service may be down before it is restored.
StrategyHow it worksTypical RPO / RTOCost
Backup and restoreRegular backups copied to another region; rebuild everything there after a disasterHours / hours to daysLowest
Pilot lightData replicated continuously to a second region; minimal or no compute running there until neededMinutes / tens of minutes to hoursLow
Warm standbyA scaled-down copy of the whole system running in the second region, scaled up on failoverSeconds to minutes / minutesMedium
Active-active (multi-region)Full copies in two or more regions all serving trafficNear zero / near zeroHighest, plus the hardest data consistency problems

Azure building blocks for DR:

  • Availability zones: separate data centres within one region. Zone-redundant services (App Service, Azure SQL, Storage, Service Bus Premium) survive the loss of one zone with no work from you. Use them first; they cover the most common failures cheaply.
  • Azure SQL failover groups: continuous replication to another region, with a single connection endpoint that moves to the secondary on failover.
  • Geo-redundant storage (GRS) for Blob Storage, and geo-disaster recovery for Service Bus namespaces.
  • Azure Front Door: sends users to a healthy region automatically.
  • Infrastructure as code (lesson 15), so a second region can be built quickly and identically.

Active-active across regions is hard mainly because of data. Two regions accepting writes to the same records at the same time can conflict, so either route each tenant or record to one home region, or use a database built for multi-region writes (such as Azure Cosmos DB) with a clear conflict policy. Many systems use active-passive for writes and serve reads from both regions.

A DR plan that has never been tested is only a hope. Run regular failover drills, restore backups to check they work, and measure the real RPO and RTO.

Check your understanding

QuestionShort answer
Logs, metrics or traces?Logs for what happened in one case, metrics for trends and alerts, traces for where time went across services.
Why structured logging?Values are stored as named fields, so they can be searched and aggregated; templates also avoid formatting cost when the level is off.
Why percentiles instead of averages?Averages hide the slow tail; p95 and p99 show what the slowest users experience.
What is high cardinality, and why avoid it in metrics?Tags with many unique values (such as user ID) create huge numbers of time series; put them in logs and traces.
What is context propagation?Passing the trace ID between services (the traceparent header) so all spans join one trace.
Why OpenTelemetry?One vendor-neutral way to instrument code; the backend can change through configuration.
What should alerts be based on?Symptoms users feel, with a runbook, and routed by urgency.
SLI, SLO or SLA?SLI is the measurement, SLO is the internal target, SLA is the contract; the SLA should be looser than the SLO.
How big is a 99.9% monthly error budget?About 43 minutes of downtime in a 30-day month, or the equivalent in failed requests.
RPO or RTO?RPO is how much data you may lose; RTO is how long you may be down.
Why test disaster recovery?Untested backups and failovers often fail when needed; drills reveal the real RPO and RTO.

Summary

  • Emit connected logs, metrics and traces, with the trace ID on everything.
  • Log with structured templates, never log secrets or sensitive data, and control volume and retention.
  • Measure RED for services and USE for resources, use percentiles, and keep metric tags low in cardinality.
  • Instrument with OpenTelemetry and export to Azure Monitor, Prometheus and Grafana, or Elastic.
  • Alert on user-facing symptoms with runbooks; define SLOs and manage releases with error budgets.
  • Handle incidents with clear roles, mitigate first, and learn through blameless postmortems.
  • Set RPO and RTO per system, use availability zones first, choose a DR strategy by cost and need, and test it.

The next lesson covers testing: the test pyramid and honeycomb, xUnit, Testcontainers, contract testing with Pact, end-to-end tests with Playwright, load testing with k6, chaos engineering, mutation testing and security testing.

Found this useful? Share it: LinkedIn X

Comments