FullStack Engineering CourseLesson 15 of 20 29 min
DevOps: Containers, Kubernetes, Infrastructure as Code and CI/CD
From commit to production
DevOps is the set of practices that let a team change software quickly and safely: every change is built, tested and deployed by automation, environments are created from code, and releases are small, frequent and easy to undo. This lesson follows a change from the developer's commit to production:
- Package the app as a container image (Docker).
- Describe the infrastructure as code (Terraform or Bicep).
- Build, test and deploy with a pipeline (GitHub Actions, Azure DevOps, Jenkins), or let the cluster pull changes from Git (GitOps with Argo CD).
- Release safely: trunk-based development, feature flags, blue-green and canary deployments, and database changes that do not break running code.
- Measure how well it works with the DORA metrics.
Containers and Docker
A container image is a package that holds an application and everything it needs to run: the .NET runtime, libraries and configuration defaults. A container is a running instance of an image. Because the same image runs on a laptop, in testing and in production, "it works on my machine" problems mostly disappear.
Images are built in layers, one per instruction in the Dockerfile, and unchanged layers are reused from cache. A multi-stage Dockerfile builds the app in a large image that has the SDK, then copies only the output into a small runtime image:
# Stage 1: build with the full SDK
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY ["Claims.Api/Claims.Api.csproj", "Claims.Api/"]
RUN dotnet restore "Claims.Api/Claims.Api.csproj" # cached until the project file changes
COPY . .
RUN dotnet publish "Claims.Api/Claims.Api.csproj" -c Release -o /app/publish --no-restore
# Stage 2: run on a small "chiseled" image: no shell, no package manager, runs as a non-root user
FROM mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled AS final
WORKDIR /app
COPY --from=build /app/publish .
EXPOSE 8080
ENTRYPOINT ["dotnet", "Claims.Api.dll"]Good practices for images:
- Copy the project files and restore before copying the rest of the source, so the slow restore step is cached when only code changes.
- Use small runtime images (such as Microsoft's chiseled images): faster to pull, and fewer components that could have vulnerabilities.
- Run as a non-root user. .NET images listen on port 8080 by default so they do not need root.
- Add a .dockerignore file so bin, obj, .git and local secrets never enter the image.
- Tag each image with an immutable version, such as the semantic version plus the Git commit SHA (claims-api:1.4.2-a1b2c3d). Never deploy latest: you cannot tell what is running or roll back to it reliably.
- Build the image once and promote the same image through every environment, so what was tested is exactly what goes to production.
- Store images in a private registry (Azure Container Registry), scan them for known vulnerabilities, and sign them so only trusted images can run.
- Never put secrets in an image. Configuration and secrets come from the environment at run time.
The .NET SDK can also build images without a Dockerfile: dotnet publish /t:PublishContainer creates an optimised image directly from the project.
Kubernetes and Helm
Kubernetes runs containers across a cluster of machines (nodes), and keeps the actual state matching the desired state you describe in YAML. The main objects:
| Object | What it does |
|---|---|
| Pod | One or more containers that run together; the smallest unit Kubernetes schedules |
| Deployment | Keeps a set number of identical pods running, and replaces them gradually when the image changes |
| Service | A stable internal address and load balancing for a set of pods |
| Ingress (or Gateway API) | Routes external HTTP traffic to services |
| ConfigMap and Secret | Configuration and secret values passed to pods (prefer Key Vault with the CSI driver over plain Kubernetes secrets) |
| Namespace | Groups and isolates resources, for example per team or environment |
| HorizontalPodAutoscaler | Adds or removes pods based on CPU, memory or custom metrics; KEDA adds event-based scaling |
apiVersion: apps/v1
kind: Deployment
metadata:
name: claims-api
spec:
replicas: 3
selector:
matchLabels: { app: claims-api }
template:
metadata:
labels: { app: claims-api }
spec:
containers:
- name: claims-api
image: claimsacr.azurecr.io/claims-api:1.4.2-a1b2c3d # immutable tag
ports: [{ containerPort: 8080 }]
resources:
requests: { cpu: 250m, memory: 256Mi } # what the scheduler reserves
limits: { memory: 512Mi } # the pod is restarted if it uses more
livenessProbe:
httpGet: { path: /health/live, port: 8080 }
readinessProbe:
httpGet: { path: /health/ready, port: 8080 }Always set resource requests (so the scheduler places pods sensibly) and memory limits (so one leaking pod cannot starve its node), and use separate liveness and readiness probes (lesson 7).
Helm packages Kubernetes YAML into charts: templates plus a values file per environment. One chart deploys the same app to dev, test and production with different replica counts, hostnames and resources. A deployment is then helm upgrade --install claims-api ./chart -f values-prod.yaml.
Infrastructure as code: Terraform and Bicep
Infrastructure as code (IaC) describes cloud resources (resource groups, databases, app services, networks) in files kept in Git, instead of creating them by clicking in the portal. Environments become reproducible, every change is reviewed in a pull request, and drift (manual changes in the portal) can be detected.
| Terraform | Bicep | |
|---|---|---|
| Clouds | Any cloud and many SaaS products, through providers | Azure only |
| State | Keeps a state file mapping code to real resources; store it remotely (an Azure Storage account) with locking | No state file; Azure itself is the source of truth |
| Preview changes | terraform plan | what-if |
| New Azure features | Supported once the provider adds them (AzAPI covers new ones immediately) | Available on day one |
| Fits | Multi-cloud or mixed platforms; teams that already use it | Azure-only teams that want simplicity |
// Bicep: an App Service with a managed identity
param env string
param location string = resourceGroup().location
resource plan 'Microsoft.Web/serverfarms@2023-12-01' = {
name: 'asp-claims-${env}'
location: location
sku: { name: 'P1v3' }
kind: 'linux'
properties: { reserved: true }
}
resource app 'Microsoft.Web/sites@2023-12-01' = {
name: 'app-claims-${env}'
location: location
identity: { type: 'SystemAssigned' }
properties: {
serverFarmId: plan.id
httpsOnly: true
siteConfig: { linuxFxVersion: 'DOTNETCORE|10.0', healthCheckPath: '/health/ready' }
}
}Whichever tool you use: one set of modules for all environments with a variables file per environment, a plan or what-if step on every pull request, apply only after review and approval, and no manual changes in the portal (or, if they happen, bring them back into code).
CI/CD pipelines
Continuous integration (CI) builds and tests every change automatically, on every pull request and merge. Continuous delivery (CD) deploys every successful build to an environment automatically, with production either automatic (continuous deployment) or behind an approval. A typical pipeline:
- On every pull request: restore, build, run unit tests, check formatting and code analysis, scan dependencies for vulnerabilities, and run the IaC plan.
- On merge to main: build and scan the container image once, tag it with the commit SHA, push it to the registry.
- Deploy that image to a test environment automatically, and run integration and smoke tests (quick checks that the main paths work).
- Deploy the same image to production, after an approval if required, using a safe strategy (blue-green or canary, below).
- Watch the health checks and error rates after deployment, and roll back automatically if they degrade.
# GitHub Actions: build once, then promote through environments
name: claims-api
on:
push: { branches: [main] }
pull_request:
permissions:
id-token: write # lets the workflow log in to Azure with OIDC: no stored cloud secrets
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with: { dotnet-version: '10.0.x' }
- run: dotnet test --configuration Release
- uses: azure/login@v2
if: github.ref == 'refs/heads/main'
with:
client-id: ${{ vars.AZURE_CLIENT_ID }}
tenant-id: ${{ vars.AZURE_TENANT_ID }}
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
- name: Build and push the image (once)
if: github.ref == 'refs/heads/main'
run: |
az acr login --name claimsacr
docker build -t claimsacr.azurecr.io/claims-api:${{ github.sha }} .
docker push claimsacr.azurecr.io/claims-api:${{ github.sha }}
deploy-test:
needs: build
if: github.ref == 'refs/heads/main'
environment: test
runs-on: ubuntu-latest
steps:
- run: echo "deploy claims-api:${{ github.sha }} to test, then run smoke tests"
deploy-prod:
needs: deploy-test
environment: production # approval rules are configured on this environment
runs-on: ubuntu-latest
steps:
- run: echo "deploy the same image claims-api:${{ github.sha }} to production"Azure DevOps pipelines (YAML with stages, environments and approvals) and Jenkins follow the same structure. Whatever the tool, log in to Azure with workload identity federation (OIDC) instead of stored secrets, protect the main branch with required reviews and checks, and keep the pipeline definition in the repository with the code.
GitOps with Argo CD
In a normal pipeline, the pipeline pushes changes into the cluster. GitOps reverses this. A Git repository holds the desired state of every environment (the Kubernetes manifests or Helm values, including image tags). A tool running inside the cluster, such as Argo CD or Flux, watches the repository and pulls the cluster into line with it:
- To deploy, change the image tag in the environment's folder and merge a pull request. Argo CD applies it.
- To roll back, revert the commit.
- If someone changes the cluster by hand, Argo CD detects the drift and can put it back automatically.
- The pipeline no longer needs credentials for the cluster, which reduces risk; the cluster pulls rather than being pushed to.
- Git history becomes the audit log of every change to every environment.
Branching: trunk-based development
How a team uses branches has a large effect on delivery speed. In trunk-based development, everyone integrates small changes into the main branch (the trunk) at least daily, through short-lived branches and pull requests. The trunk is always releasable. Unfinished features are merged but hidden behind feature flags.
GitFlow, the older alternative, keeps long-lived develop, release and feature branches. Long-lived branches drift apart, and merging them becomes slow and risky ("merge hell"). Research behind the DORA metrics links trunk-based development with better delivery performance, so it is the usual recommendation. Short-lived release branches are still useful for software shipped to customers in versions, such as mobile apps.
Feature flags
A feature flag is a switch, checked at run time, that turns a piece of code on or off without deploying. It separates deployment (putting code into production) from release (letting users see it). Kinds of flag:
- Release flags: hide unfinished work merged into the trunk. Remove them once the feature is fully live.
- Operational flags (kill switches): turn off an expensive or failing feature quickly during an incident.
- Experiment flags: show different variants to different users and measure the results (A/B testing).
- Permission flags: enable features for specific customers, tenants or plans.
// Azure App Configuration holds the flags; Microsoft.FeatureManagement evaluates them.
builder.Configuration.AddAzureAppConfiguration(o =>
o.Connect(new Uri("https://appcs-claims.azconfig.io"), new DefaultAzureCredential())
.UseFeatureFlags());
builder.Services.AddFeatureManagement();
static async Task<IResult> PriceClaim(int id, IFeatureManager features, IPricingEngine legacy, IPricingEngine2 next)
{
var price = await features.IsEnabledAsync("NewPricingEngine") // can be on for 5% of tenants, or one tenant
? await next.PriceAsync(id)
: await legacy.PriceAsync(id);
return TypedResults.Ok(price);
}Flags are technical debt with a timer. Give each release flag an owner and an expiry date, and remove it (with the old code path) once the feature is fully rolled out.
Deployment strategies
| Strategy | How it works | Rollback | Trade-offs |
|---|---|---|---|
| Rolling | Replace instances a few at a time | Roll forward or redeploy the old version | Simple; old and new versions run side by side for a while |
| Blue-green | Deploy the new version (green) next to the current one (blue), test it, then switch all traffic at once (App Service slot swap) | Switch back instantly | Needs double capacity during the switch; all users move at once |
| Canary | Send a small share of traffic (for example 5%) to the new version, watch errors and latency, then increase step by step | Send all traffic back to the old version | Limits the impact of a bad release; needs good metrics and traffic splitting |
| Ring-based | Release to groups in turn: internal users, then early-adopter tenants, then everyone | Stop at the current ring | Common for multi-tenant SaaS; often combined with feature flags |
Database changes need the most care, because during any of these strategies the old and new versions of the code run at the same time against one database. Use expand and contract (also called parallel change) so every step works with both versions:
- Expand: add the new column or table, without removing anything. Old code ignores it.
- Deploy code that writes to both the old and new columns, and reads from the old one.
- Backfill: copy existing data into the new column.
- Deploy code that reads from the new column.
- Contract: once no running code uses the old column, remove it in a later release.
Never rename or drop a column in the same release as the code that stops using it. Run migrations as a separate, controlled pipeline step, not automatically when each app instance starts.
DORA metrics
The DORA research programme (DevOps Research and Assessment) identified four key metrics that predict both delivery speed and stability:
| Metric | Measures | Shows |
|---|---|---|
| Deployment frequency | How often code is deployed to production | Speed |
| Lead time for changes | Time from a commit to that commit running in production | Speed |
| Change failure rate | The share of deployments that cause a failure needing a fix or rollback | Stability |
| Failed deployment recovery time | How long it takes to recover from a failed deployment (earlier called time to restore service) | Stability |
The research consistently finds that speed and stability go together: teams that deploy small changes often also fail less and recover faster. Track the metrics per team to spot bottlenecks (slow reviews, flaky tests, manual approvals), not to compare or rank people.
Check your understanding
| Question | Short answer |
|---|---|
| Image or container? | An image is the package; a container is a running instance of it. |
| Why a multi-stage Dockerfile? | Build with the SDK, run on a small runtime image, so the final image is smaller and safer. |
| Why not deploy the latest tag? | You cannot tell which version is running or roll back reliably; use immutable tags such as the commit SHA. |
| Why build once and promote? | So the image tested in earlier environments is exactly the one that reaches production. |
| What are requests and limits in Kubernetes? | Requests reserve resources for scheduling; limits cap what a container may use. |
| Terraform or Bicep? | Terraform for multi-cloud and when you want its ecosystem (it keeps state); Bicep for Azure-only teams (no state, new features first). |
| What is GitOps? | Git holds the desired state of each environment, and a tool in the cluster pulls the cluster into line with it. |
| Why trunk-based development? | Small, frequent merges avoid long-lived branches that drift apart; unfinished work hides behind flags. |
| Deployment or release? | Deployment puts code into production; release lets users see it, often controlled by feature flags. |
| Blue-green or canary? | Blue-green switches all traffic at once with instant rollback; canary moves traffic gradually while watching metrics. |
| What is expand and contract? | Changing a database in steps that work with both old and new code: add, migrate and backfill, then remove later. |
| What are the DORA metrics? | Deployment frequency, lead time for changes, change failure rate and failed deployment recovery time. |
Summary
- Package apps as small, non-root container images with immutable tags, built once and promoted unchanged.
- Run on Kubernetes with requests, limits and proper probes, packaged with Helm, or on a managed container service.
- Describe all infrastructure in Terraform or Bicep, with plans reviewed in pull requests.
- Automate build, test, scan and deploy, with OIDC logins and approvals where needed; consider GitOps with Argo CD for Kubernetes.
- Practise trunk-based development and separate deployment from release with feature flags.
- Release with blue-green or canary deployments, and change databases with expand and contract.
- Track the DORA metrics to improve flow, not to judge people.
The next lesson covers observability and reliability: OpenTelemetry, logs, metrics and traces, Application Insights and Azure Monitor, Prometheus and Grafana, SLOs and error budgets, and disaster recovery with RPO, RTO and multi-region failover.
Comments