Chapter 27: Containerization and Local Orchestration

Containerization packages your application and its dependencies into a portable unit. Local orchestration then runs and coordinates multiple of those units reliably on your machine.

With containerization, you bundle code, libraries, runtime, and system tools into a self-contained image that behaves the same on every developer laptop and server.

Local orchestration builds on that by managing multiple containers at once: your web app, database, background workers, and supporting services can all be started, stopped, and wired together with a single command. Together, they give you a realistic “mini production” environment on your local machine, making development, testing, and debugging much more predictable.

By the end of this chapter, you’ll learn what containers are, why teams rely on them, and meet Docker: an open platform for developing, shipping, and running applications using containers.

In this chapter you’ll use Docker to build, run, and orchestrate your services locally so your stack runs the same way on every machine.

Your News Aggregator API from Chapter 26 works perfectly on your laptop. FastAPI serves requests quickly. PostgreSQL stores articles efficiently. Rate limiting protects against abuse. You run python main.py, test the endpoints in your browser, and everything responds correctly. The code is production-ready.

Then you share the repository with a teammate. They clone it, install dependencies, and run the application. Five errors appear immediately. PostgreSQL isn't installed. Environment variables are missing. The Python version is wrong. Database migrations haven't run. What took you three seconds to start takes them three hours to debug. This is the "works on my machine" crisis that containerization solves.

The problem isn't your code. The problem is implicit dependencies. Your laptop has PostgreSQL 15 installed. Your teammate has PostgreSQL 14. Your Python is 3.11. Theirs is 3.10. Your .env file has API keys configured. Theirs doesn't. Hundreds of environmental differences create hundreds of potential failures. Professional developers don't debug these differences manually. They package applications with everything needed to run identically anywhere.

This chapter teaches containerization: packaging your application with its runtime, dependencies, and configuration into a standardized container. You'll containerize your News API, orchestrate it with PostgreSQL and Redis using Docker Compose, optimize performance with caching, and start your entire production-ready stack with one command: docker compose up. By the end, sharing your application means sharing a repository that works immediately for everyone, and deploying to AWS in Chapter 28 becomes a straightforward process.

This chapter takes you from a laptop-only Python API to a fully containerized, multi-service stack that runs identically on any machine. Here's the journey:

By the end of this chapter, you'll be able to:

• Explain what containers are and the problems they solve versus virtual machines and manual deployment
• Install Docker and verify your development environment is ready for containerization
• Write production-ready Dockerfiles using multi-stage builds to minimize image size
• Build Docker images and run containers with proper configuration and networking
• Orchestrate multi-service applications (API + PostgreSQL + Redis) with Docker Compose
• Configure environment variables, volumes, and networks for local development stacks
• Profile application performance and identify optimization opportunities before scaling

You'll containerize your News Aggregator API from Chapter 26, transforming it from a laptop-only application into a portable, production-ready stack. The current state requires manual setup: install PostgreSQL, configure environment variables, run migrations, start the API server. The target state runs with one command on any machine with Docker installed.

Your containerized stack includes three services working together. The News API container runs your FastAPI application with all Python dependencies packaged inside. The PostgreSQL container provides the database with persistent data storage that survives restarts. The Redis container adds an in-memory caching layer that reduces database load and improves response times from 650ms to 5ms for cached requests.

Docker Compose orchestrates these three containers, managing networking so they can communicate, configuring environment variables, and ensuring services start in the correct order. When you run docker compose up, all three containers start together. Your API is accessible at http://localhost:8000, exactly like Chapter 26, but now teammates can clone your repository and start the entire stack without installing anything except Docker.

This preparation is essential for Chapter 28. AWS deployment requires containerized applications. By containerizing locally first, you'll understand how containers work, optimize performance, and debug issues in a familiar environment before deploying to the cloud. The patterns you learn here (Dockerfile optimization, environment configuration, service orchestration) apply directly to production deployments on AWS, Azure, or Google Cloud.

Before we wrap the News API in containers, we need to know whether the bottleneck is the code, the database, or the upstream news provider. That’s what performance profiling will tell us. This isn’t optional preparation. It’s professional discipline. Containers package your application exactly as it exists today. If you containerize slow code, you get slow containers running on expensive servers. Adding more servers doesn't fix inefficient code. It multiplies the cost.

Consider this scenario: a startup launches their API, notices slow response times, and scales from 2 servers to 10 servers at $500 per month. Response times improve slightly, but the fundamental problem remains. They profile the application, discover one database query missing an index, add the index, and drop to 2 servers at $100 per month. The issue wasn't scale. It was a fixable performance bottleneck.

Professional developers optimize before scaling. They establish baseline performance metrics, identify bottlenecks, implement fixes, and verify improvements before containerization and deployment. This section profiles your News API to measure current performance, identify where time is spent, and establish targets for optimization. You'll add timing middleware, run load tests, and document baseline metrics that guide caching strategy in Section 6.

The first step is measuring how long requests take. Your News API fetches articles from NewsAPI and Guardian, saves them to PostgreSQL, and returns results. How long does this take? Without measurement, you're guessing. Add timing middleware that logs request duration for every endpoint.

Make: Add timing middleware to your News API. Create a new file called middleware/timing.py in your project:

import time
from fastapi import Request
from starlette.middleware.base import BaseHTTPMiddleware


class TimingMiddleware(BaseHTTPMiddleware):
    """Middleware that logs request processing time."""
    
    async def dispatch(self, request: Request, call_next):
        # Record start time
        start_time = time.time()
        
        # Process the request
        response = await call_next(request)
        
        # Calculate duration
        duration = time.time() - start_time
        duration_ms = duration * 1000  # Convert to milliseconds
        
        # Log the timing
        print(f"{request.method} {request.url.path} - {duration_ms:.2f}ms")
        
        # Add timing header to response
        response.headers["X-Process-Time"] = f"{duration_ms:.2f}ms"
        
        return response

Now register this middleware in your main.py. Add these lines after creating your FastAPI app:

from middleware.timing import TimingMiddleware

app = FastAPI(title="News Aggregator API")

# Add timing middleware
app.add_middleware(TimingMiddleware)

Check: Start your News API and make requests to the /articles endpoint. Watch your terminal output:

$ uvicorn main:app --reload
INFO:     Started server process
INFO:     Waiting for application startup.
INFO:     Application startup complete.

GET /articles?category=technology - 687.23ms
GET /articles?category=business - 742.15ms
GET /articles - 821.47ms
GET /articles?source=newsapi - 694.82ms

Every request takes 650-850ms. That's slow. Users expect responses under 200ms for good experiences. Let's understand why this takes so long.

Your /articles endpoint performs multiple operations. Each operation takes time. To optimize effectively, break down where the 700ms is spent.

External API calls: Your endpoint fetches from NewsAPI and Guardian. Each call takes approximately 200ms for network round-trip, API processing, and response parsing. Two sources means 400ms minimum just waiting for external APIs.

Database operations: After fetching articles, your code saves them to PostgreSQL (checking for duplicates, inserting new records). Then it queries the database to return results. Database operations add another 150-200ms depending on the number of articles.

Application processing: Normalizing different API response formats, building your standardized response, and serializing to JSON adds 50-100ms.

Total: 400ms (external APIs) + 200ms (database) + 100ms (processing) = 700ms. This matches your observed performance. The bottleneck is clear: external API calls dominate request time.

The problem multiplies with concurrent users. If 100 users request articles simultaneously, your API makes 200 external API calls (100 users × 2 sources). External APIs will rate limit you. NewsAPI might limit you to 100 requests per hour on free tiers. You'd exhaust your quota in 30 seconds.

The solution is caching. The first request takes 700ms to fetch from external APIs and save to the database. Subsequent requests for the same category within 5 minutes return cached data from memory in 5ms. Instead of 200 external API calls for 100 concurrent users, you make 2 calls (one per source) and serve 98 requests from cache.

This pattern is universal in API development: expensive operations (external API calls, complex database queries, heavy computations) are cached so repeated requests serve from memory instead of repeating the expensive work. You'll implement this caching strategy in Section 6 using Redis.

Measuring one request at a time shows average performance. Load testing simulates multiple concurrent users, revealing how your API behaves under realistic traffic. You need a load testing tool that sends many requests quickly.

Make: Install hey, a simple load testing tool. On macOS, use Homebrew:

brew install hey

On Linux or Windows, download the binary from the hey GitHub releases page. Verify installation:

$ hey --version
hey version 0.1.4

Check: Run a load test against your unoptimized API. Make sure your API is running (uvicorn main:app), then execute:

hey -n 100 -c 10 http://localhost:8000/articles?category=technology

This command sends 100 total requests (-n 100) with 10 concurrent requests at a time (-c 10). The output shows detailed performance statistics:

Summary:
  Total:        8.2341 secs
  Slowest:      0.9523 secs
  Fastest:      0.6432 secs
  Average:      0.7234 secs
  Requests/sec: 12.15

Response time histogram:
  0.643 [1]     |
  0.674 [8]     |■■■■
  0.705 [32]    |■■■■■■■■■■■■■■■■
  0.736 [28]    |■■■■■■■■■■■■■■
  0.767 [15]    |■■■■■■■
  0.798 [9]     |■■■■
  0.829 [4]     |■■
  0.860 [2]     |■
  0.891 [0]     |
  0.922 [0]     |
  0.952 [1]     |

Latency distribution:
  10% in 0.6789 secs
  25% in 0.6921 secs
  50% in 0.7123 secs
  75% in 0.7456 secs
  90% in 0.7892 secs
  95% in 0.8234 secs
  99% in 0.9523 secs

Extract: Document these baseline metrics. Your unoptimized API handles approximately 12 requests per second with an average response time of 723ms. The 95th percentile (slowest 5% of requests) takes 823ms. These numbers establish your optimization target.

After implementing Redis caching in Section 6, you'll run this exact test again. The goal: increase throughput from 12 requests/second to over 400 requests/second, and reduce average response time from 723ms to under 10ms for cached requests. That's a 35x performance improvement from adding one service to your stack.

Use this quiz to check your understanding. Try to answer each question out loud or in a notebook before expanding the explanation. If you get stuck, that's a signal to revisit the relevant section.

Containers are lightweight, portable packages that bundle an application with everything it needs to run. Each container includes your app code, its runtime, system libraries, and other dependencies, so it behaves the same way in any environment.

In a traditional setup without containers, your application depends on whatever machine it runs on to provide its runtime and libraries. It expects Python to be installed, specific Linux libraries to be present, and certain system tools to be available. If any of those are missing or the wrong version, your app crashes or behaves unpredictably.

A container changes this by packing its own internal filesystem.

Think of containers like shipping containers for software. A physical shipping container standardizes how cargo moves: the same container travels by truck, train, or ship without repacking the contents. In the same way, a software container standardizes how your application runs: the same image can start on your Mac, a teammate's Linux laptop, or an AWS server without reconfiguration.

Just like physical shipping containers standardize global transport, software containers standardize application deployment across any environment, from a local laptop to the cloud.

Unlike virtual machines, which bundle a full guest operating system for each app, containers reuse the host operating system kernel and only package your application and its dependencies. You still get isolation because each container sees its own filesystem, processes, and network view, but with far less overhead in disk space and startup time.

In Section 1, Background, you saw how your News API ran perfectly on your laptop but fell apart on a teammate’s machine. The code was identical; the environment was not. That gap between your setup and theirs is what containers exist to close.

Traditional solutions involve lengthy setup documentation: "Install Python 3.11. Install PostgreSQL 15. Set these environment variables. Run these migrations. Install these system libraries." Each step is a potential failure point. Teammates spend hours configuring environments instead of writing code, and production servers need careful, error-prone environment matching.

Containers solve this by packaging the environment with the application. Instead of asking everyone to recreate your laptop manually, you ship an image that already includes Python 3.11, your dependencies, and the correct configuration. Your teammate runs one command (docker compose up) and gets the same environment you developed in.

This benefit extends beyond local development. When you deploy to AWS, you deploy the same image that runs on your laptop. No custom setup scripts. No production-only surprises. If it works locally in the container, it behaves the same way in staging and production because the environment travels with the code.

Virtual machines and containers both provide isolation, but their architectures differ fundamentally. Understanding this difference explains why containers became the industry standard for application deployment.

VMs package entire operating systems (GBs), while containers share the host OS kernel and package only applications and dependencies (MBs).

Virtual machines package complete operating systems. A VM running Ubuntu includes the full Ubuntu OS, kernel, system services, and utilities. Running three applications in three VMs means three complete operating systems consuming memory and disk space. Each VM is gigabytes in size. Starting a VM takes minutes because you're booting an entire operating system. VMs provide strong isolation because each runs its own kernel, but resource overhead is substantial.

Containers package applications and dependencies, sharing the host kernel. A container running your Python application includes Python, your code, and pip packages. It shares the host Linux kernel. Running three applications in three containers means one kernel, three isolated application environments. Each container is megabytes, not gigabytes. Starting a container takes seconds because you're starting a process, not booting an OS. Containers provide sufficient isolation for most applications with minimal overhead.

Size comparison: A VM image for a Python web application might be 2-4 GB (full Ubuntu installation plus application). A container image for the same application is typically 200-400 MB (minimal base image plus application dependencies).

Startup time comparison: VMs take 30-60 seconds to boot. Containers start in 1-3 seconds. This speed advantage makes containers ideal for scaling (starting additional instances quickly during traffic spikes) and development (restarting quickly after code changes).

Resource usage: VMs allocate fixed memory and CPU. A VM configured with 2 GB RAM reserves that memory even if the application uses 500 MB. Containers share host resources dynamically. Three containers might collectively use 1 GB of the host's 8 GB RAM, leaving 7 GB available for other work.

Containers aren't always necessary. Simple scripts, exploratory notebooks, and personal tools run fine without containerization. Understanding when containers add value prevents over-engineering.

You need containers when:

Docker Desktop provides everything you need: the Docker engine (runs containers), Docker Compose (orchestrates multi-container applications), and a GUI for managing containers. Installation is straightforward but platform-specific.

For macOS: Download Docker Desktop from docker.com/products/docker-desktop. Open the downloaded DMG file and drag Docker to Applications. Launch Docker Desktop from Applications. You'll see a whale icon in your menu bar when Docker is running.

For Windows: Download Docker Desktop for Windows. The installer requires Windows 10 or 11 with WSL 2 (Windows Subsystem for Linux). Follow the installation wizard. Docker Desktop will prompt you to enable WSL 2 if it's not already configured. Restart when prompted. Launch Docker Desktop from the Start menu.

For Linux: Install Docker Engine using your distribution's package manager. On Ubuntu:

sudo apt-get update
sudo apt-get install docker.io docker-compose
sudo systemctl start docker
sudo systemctl enable docker

Verify installation: Open a terminal and run:

docker --version

You should see output like Docker version 24.0.6, build ed223bc. Then verify Docker Compose:

docker compose version

Expected output: Docker Compose version v2.23.0. If both commands work, your Docker installation is complete and ready for containerization.

Docker is an open platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure so you can deliver software quickly. At its core, Docker provides a way to package an application and all its dependencies (libraries, runtime, system tools, code) into a single unit called a container.

If your application works in a Docker container on your laptop, it will work in a Docker container on a server, on AWS, or on a colleague's machine. It effectively eliminates environmental inconsistencies.

To use Docker effectively, you need to understand the relationships between its core components. Docker operates using three main pillars that you will interact with constantly.

The Docker Architecture

Docker uses a client-server architecture to manage these components:

• Client: The command line tool (docker build, docker run) you use on your terminal.
• Daemon (Server): A background process running on your computer that does the heavy lifting (building, running, and distributing containers).
• Registry: A cloud storage for images (like Docker Hub). This is where you download official images (like postgres or python) instead of writing them from scratch.

A Dockerfile is a text file containing instructions for building a Docker image. Each instruction creates a layer in the image. Let's build a simple Dockerfile for your News API, then improve it with professional patterns.

Make: Create a file named Dockerfile (no extension) in your project root:

# Start from official Python image
FROM python:3.11-slim

# Set working directory inside container
WORKDIR /app

# Copy requirements file
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Expose port 8000
EXPOSE 8000

# Command to run the application
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Check: Build the Docker image. From your project directory (where the Dockerfile is located), run:

docker build -t news-api .

The -t news-api flag names your image "news-api". The . tells Docker to use the current directory as the build context (where to find the Dockerfile and your code). You'll see output showing each instruction executing:

[+] Building 23.4s (11/11) FINISHED
 => [internal] load build definition from Dockerfile
 => [internal] load .dockerignore
 => [internal] load metadata for docker.io/library/python:3.11-slim
 => [1/6] FROM docker.io/library/python:3.11-slim
 => [internal] load build context
 => [2/6] WORKDIR /app
 => [3/6] COPY requirements.txt .
 => [4/6] RUN pip install --no-cache-dir -r requirements.txt
 => [5/6] COPY . .
 => [6/6] exporting to image
 => => naming to docker.io/library/news-api
Successfully built news-api

Now run a container from your image:

docker run -p 8000:8000 --env-file .env news-api

The -p 8000:8000 flag maps port 8000 inside the container to port 8000 on your host machine. The --env-file .env loads environment variables from your .env file. Your API starts inside the container, and you can access it at http://localhost:8000 just like before.

The basic Dockerfile works but produces a large image. Your image likely measures 400-500 MB because it includes build tools, source files, and cached data that aren't needed to run the application. Multi-stage builds solve this by using multiple FROM statements to build in one stage and copy only necessary artifacts to a minimal final stage.

Make: Replace your basic Dockerfile with this optimized version:

# ============================================
# Stage 1: Builder - Install dependencies
# ============================================
FROM python:3.11-slim as builder

WORKDIR /app

# Install system dependencies needed for Python packages
RUN apt-get update && apt-get install -y \
    gcc \
    postgresql-client \
    && rm -rf /var/lib/apt/lists/*

# Copy and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# ============================================
# Stage 2: Runtime - Minimal final image
# ============================================
FROM python:3.11-slim

WORKDIR /app

# Install only runtime dependencies (no build tools)
RUN apt-get update && apt-get install -y \
    libpq5 \
    && rm -rf /var/lib/apt/lists/*

# Copy Python packages from builder stage
COPY --from=builder /root/.local /root/.local

# Copy application code
COPY . .

# Make sure scripts in .local are usable
ENV PATH=/root/.local/bin:$PATH

# Create non-root user for security
RUN useradd -m -u 1000 apiuser && \
    chown -R apiuser:apiuser /app
USER apiuser

# Expose port
EXPOSE 8000

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2)"

# Start application
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Check: Rebuild your image with this optimized Dockerfile:

docker build -t news-api:optimized .

Compare image sizes:

docker images | grep news-api

news-api        optimized    abc123def456    2 minutes ago    187MB
news-api        latest       xyz789ghi012    15 minutes ago   452MB

Extract: The optimized image is 187 MB versus 452 MB for the basic version. You've reduced image size by 58% while improving security (runs as non-root user) and adding health checks. Smaller images mean faster deployments, lower bandwidth costs, and quicker container starts.

Every instruction in a Dockerfile creates a layer. Docker caches these layers to speed up subsequent builds. Understanding layer caching is crucial for fast development workflows. Poor Dockerfile structure means rebuilding everything on every code change. Good structure means rebuilding only what changed.

How layer caching works: Docker processes your Dockerfile from top to bottom. For each instruction, it checks if it has a cached layer from a previous build with identical inputs. If the instruction and its inputs haven't changed, Docker reuses the cached layer. If anything changed, Docker executes the instruction and invalidates all subsequent cached layers.

Example of poor layer ordering:

FROM python:3.11-slim
COPY . .                          # Copies ALL files
RUN pip install -r requirements.txt  # Installs dependencies

This structure copies all application code before installing dependencies. Every time you change any Python file (which happens constantly during development), Docker invalidates the layer and reinstalls all dependencies. Installing dependencies takes 60 seconds. You reinstall them dozens of times per day for no reason.

Example of good layer ordering:

FROM python:3.11-slim
COPY requirements.txt .           # Copy ONLY requirements
RUN pip install -r requirements.txt  # Install dependencies (cached)
COPY . .                          # Copy application code

This structure copies requirements.txt first, installs dependencies, then copies application code. When you change your Python code, Docker invalidates only the final COPY . . layer. The dependency installation layer remains cached. Rebuilds take 2 seconds instead of 60 seconds because dependencies don't reinstall unless requirements.txt changes.

Figure 4.1: Optimized layer ordering prevents invalidation cascades: only changed layers rebuild, while dependency installation remains cached.

When you run docker build, Docker sends your entire project directory (the build context) to the Docker daemon. If your project includes large files like node_modules/, .git/, test data, or local databases, Docker unnecessarily copies gigabytes of data before building. This slows builds dramatically.

A .dockerignore file works like .gitignore, telling Docker which files to exclude from the build context. Create one in your project root:

# Git repository data
.git
.gitignore

# Python cache and virtual environments
__pycache__
*.pyc
*.pyo
*.pyd
.Python
venv/
env/
.venv

# Development databases
*.db
*.sqlite3
music_time_machine.db

# IDE and editor files
.vscode/
.idea/
*.swp
*.swo
*~

# Documentation
*.md
docs/

# Test files
tests/
test_*.py
*_test.py

# Environment files (security risk if copied)
.env.local
.env.*.local

# Docker files (don't need to copy these into container)
Dockerfile
.dockerignore
docker-compose.yml

# Logs and temporary files
*.log
logs/
tmp/

This exclusion list dramatically reduces build context size. Your project might be 500 MB on disk but only 10 MB is sent to Docker for building. Builds start faster, and you reduce the risk of accidentally copying sensitive files into images.

Use this quiz to check your understanding. Try to answer each question out loud or in a notebook before expanding the explanation. If you get stuck, that's a signal to revisit the relevant section.

Docker Compose is a tool for defining and running multi-container Docker applications. Instead of managing containers individually with long docker run commands, you define your entire application stack in a YAML file (docker-compose.yml) and start everything with one command: docker compose up.

Your News API needs three services running together: the FastAPI application, PostgreSQL database, and Redis cache. Without Docker Compose, you'd open three terminal windows and run three commands with complex arguments for networking, environment variables, and volumes. Docker Compose orchestrates all three services, manages networking so they can communicate, configures environment variables, and ensures services start in the correct order.

Imagine starting your News API stack manually without Docker Compose. You'd run these commands in separate terminal windows:

docker run --name postgres-db \
  -e POSTGRES_USER=newsapi \
  -e POSTGRES_PASSWORD=secretpassword \
  -e POSTGRES_DB=news_aggregator \
  -p 5432:5432 \
  -v postgres_data:/var/lib/postgresql/data \
  postgres:15

docker run --name redis-cache \
  -p 6379:6379 \
  redis:7-alpine

docker run --name news-api \
  --link postgres-db:postgres \
  --link redis-cache:redis \
  -e DATABASE_URL=postgresql://newsapi:secretpassword@postgres:5432/news_aggregator \
  -e REDIS_URL=redis://redis:6379 \
  -e NEWSAPI_KEY=your_key_here \
  -e GUARDIAN_API_KEY=your_key_here \
  -p 8000:8000 \
  news-api:optimized

This workflow is tedious and error-prone. You must remember the exact commands, type them correctly, manage three terminal windows, and stop services individually when done. If you restart your computer, you repeat everything. Teammates need documentation explaining every flag and environment variable. This doesn't scale.

Docker Compose replaces these three commands with one: docker compose up. All configuration lives in docker-compose.yml, a version-controlled file that documents your entire stack. Teammates clone your repository and run one command. Everything works.

Make: Create a docker-compose.yml file in your project root. This file defines all three services (API, PostgreSQL, Redis) and their configuration:

version: '3.8'

services:
  # PostgreSQL Database
  postgres:
    image: postgres:15-alpine
    container_name: news-postgres
    environment:
      POSTGRES_USER: newsapi
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: news_aggregator
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U newsapi"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - news-network

  # Redis Cache
  redis:
    image: redis:7-alpine
    container_name: news-redis
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5
    networks:
      - news-network

  # News API Application
  api:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: news-api
    environment:
      DATABASE_URL: postgresql://newsapi:${POSTGRES_PASSWORD}@postgres:5432/news_aggregator
      REDIS_URL: redis://redis:6379
      NEWSAPI_KEY: ${NEWSAPI_KEY}
      GUARDIAN_API_KEY: ${GUARDIAN_API_KEY}
    ports:
      - "8000:8000"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - news-network
    restart: unless-stopped

# Named volumes for persistent data
volumes:
  postgres_data:
  redis_data:

# Custom network for service communication
networks:
  news-network:
    driver: bridge

Check: Before starting the stack, create a .env file for sensitive credentials. Docker Compose loads these automatically:

POSTGRES_PASSWORD=your_secure_password_here
NEWSAPI_KEY=your_newsapi_key_here
GUARDIAN_API_KEY=your_guardian_api_key_here

Now start your entire stack with one command:

docker compose up

Docker Compose builds your API image (if not already built), pulls PostgreSQL and Redis images, creates containers, and starts all three services. You'll see logs from all services in one terminal:

[+] Running 4/4
 ✔ Network news-network      Created
 ✔ Container news-postgres   Started
 ✔ Container news-redis      Started
 ✔ Container news-api        Started

news-postgres  | PostgreSQL init process complete; ready for start up.
news-redis     | Ready to accept connections
news-api       | INFO:     Started server process
news-api       | INFO:     Waiting for application startup.
news-api       | INFO:     Application startup complete.
news-api       | INFO:     Uvicorn running on http://0.0.0.0:8000

Extract: Your entire stack is running. Open http://localhost:8000/docs in your browser to verify the API works. PostgreSQL stores articles. Redis will cache them (after you implement caching in Section 6). Everything runs in isolated containers but communicates through the news-network bridge network.

From your host machine, use localhost:8000 to reach containers. Inside containers, use service names like "postgres" for container-to-container communication.

Your docker-compose.yml references environment variables using ${VARIABLE_NAME} syntax. This pattern separates configuration from code, preventing secrets from appearing in version control. Let's understand this critical security practice.

Bad practice: Hardcoding secrets in docker-compose.yml:

environment:
  POSTGRES_PASSWORD: MySecretPassword123
  NEWSAPI_KEY: abc123def456ghi789

You commit this file to git. Everyone who clones your repository sees your passwords and API keys. If your repository is public, anyone on the internet sees them. Attackers scan GitHub for exposed credentials and exploit them within hours.

Good practice: Reference environment variables from .env file:

environment:
  POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
  NEWSAPI_KEY: ${NEWSAPI_KEY}

Your .gitignore includes .env, so secrets never reach git. Instead, provide a template file:

# Copy this file to .env and fill in your actual values
POSTGRES_PASSWORD=changeme
NEWSAPI_KEY=get_from_newsapi.org
GUARDIAN_API_KEY=get_from_theguardian.com

Teammates copy .env.example to .env and add their credentials. Each developer has their own .env file that never gets committed. Production environments use different credentials stored in secure secret management systems (AWS Secrets Manager, environment variables in deployment platforms).

Docker Compose provides commands for managing your multi-container application throughout development. These commands control the entire stack together rather than managing containers individually.

Start the stack:

docker compose up

This builds images if needed, creates containers, and starts all services. Logs from all containers appear in your terminal. Use -d flag for detached mode (runs in background):

docker compose up -d

Stop the stack:

docker compose down

This stops all containers and removes them. Named volumes (database data) persist. If you want to delete volumes too (starting fresh):

docker compose down -v

View logs:

docker compose logs

For live logs (follows new output): docker compose logs -f. For logs from specific service: docker compose logs api.

Rebuild after code changes:

docker compose up --build

The --build flag forces rebuilding images even if they already exist. Use this after changing application code or dependencies.

Execute commands inside containers:

docker compose exec api python manage.py migrate
docker compose exec postgres psql -U newsapi -d news_aggregator

This runs commands inside running containers. Useful for database migrations, debugging, or inspecting data.

Use this quiz to check your understanding. Try to answer each question out loud or in a notebook before expanding the explanation. If you get stuck, that's a signal to revisit the relevant section.

Redis (REmote DIctionary Server) is an in-memory data store that functions as a cache, database, and message broker. Unlike PostgreSQL which stores data on disk, Redis keeps data in RAM (memory), enabling extremely fast reads and writes. Think of it as a high-speed key-value store where you set a key ("articles:technology") and retrieve it milliseconds later.

For caching, Redis excels because memory access is 100-1000x faster than disk access. A PostgreSQL query reading from disk takes 50-150ms. A Redis lookup reading from RAM takes 1-5ms. This speed difference transforms application performance when you cache frequently accessed data.

Redis stores data in RAM for 1-5ms lookups versus PostgreSQL's 50-150ms disk reads—a 100-1000x speed advantage.

Redis runs as a separate server process that listens on a network port. Your web app opens a connection to this server and sends simple commands. Inside Redis, everything is stored in an in-memory dictionary that maps keys to values, similar to a Python dict.

At a basic level, you write data with SET key value and read it with GET key. Your code chooses meaningful keys like "articles:technology" or "user:42:profile", and Redis keeps the corresponding values in RAM. Because the lookup happens in memory instead of on disk, Redis can return the value in a few milliseconds.

For caching, the typical pattern is called cache-aside: your application checks Redis first, and only falls back to the slower data source (like PostgreSQL or an external API) if the key is missing. On a cache hit, Redis returns the value immediately. On a cache miss, your app retrieves the data from the authoritative source, stores it in Redis for next time, and returns it to the user. Redis supports automatic expiration using commands like SETEX key ttl value, which sets a key with a time-to-live (TTL) in seconds.

Here's the sequence in practice: when the first user requests technology articles, Redis has no cached data yet (cache miss). Your app fetches from the external APIs, stores the result in Redis with a 5-minute expiry, and returns the articles to the user. When the second user makes the same request 30 seconds later, Redis finds the key (cache hit) and returns the cached articles instantly without calling the external APIs. This continues for 5 minutes until the TTL expires. The next request after expiration becomes a cache miss, triggering a fresh fetch that repopulates the cache. The pattern repeats: one expensive operation followed by many fast cached responses.

Remember Section 2's performance profiling: 12 requests per second with 723ms average response time. Your News API takes 700ms per request because it fetches from NewsAPI and Guardian on every request. If 100 users request technology articles within 5 minutes, you make 200 external API calls fetching identical data. This wastes your API quota, adds latency, and risks rate limiting.

Without caching: 100 users trigger 200 external API calls. With Redis: the same requests use just 2 API calls and 98 instant cache hits.

Caching solves this by storing results temporarily. The first user requesting technology articles triggers external API calls (700ms response time). The result is cached in Redis for 5 minutes. The next 99 users requesting technology articles get cached data from memory (5ms response time). You make 2 external API calls instead of 200, and 99% of users get nearly instant responses.

The trade-off is data freshness. Cached articles might be 5 minutes old. For news, this is acceptable. Breaking news from 4 minutes ago is still relevant. For real-time stock prices or live sports scores, this delay is unacceptable. Understanding when caching is appropriate is a critical engineering judgment.

Cache when:

Data changes infrequently but is accessed frequently. News articles from the past hour change rarely but are requested hundreds of times. Product catalogs update daily but are viewed constantly. User profiles change weekly but are loaded on every page view.

Retrieving data is expensive. External API calls, complex database queries, or heavy computations that produce the same result for multiple users. If computing the result once and reusing it saves resources, cache it.

Slight staleness is acceptable. Five-minute-old news is fine. Ten-minute-old weather data is reasonable. Users understand these aren't real-time updates.

Don't cache when:

Data must be absolutely current. Bank account balances, inventory counts, authentication tokens. Serving stale data causes incorrect behavior or security issues.

Data is unique per user and rarely reused. If every user requests different data that's never requested again, caching provides no benefit. You're storing data in memory that's never reused.

Retrieval is already fast. If your database query takes 5ms, caching might save 3ms. The complexity of cache management outweighs the benefit.

Your Docker Compose stack already includes Redis. Now add caching logic to your News API. The pattern: check cache first, return cached data if available, otherwise fetch from external APIs and cache the result.

Make: Install the Redis Python client:

pip install redis

Add redis>=5.0.0 to your requirements.txt file manually. While pip freeze > requirements.txt works if your virtual environment is clean, it's safer to add the package explicitly. If you accidentally run pip freeze outside your venv or with global packages installed, you'll pollute your requirements file with dozens of unrelated dependencies.

Create a caching module with a decorator that handles caching logic:

import redis
import json
import os
from functools import wraps


# Connect to Redis
redis_client = redis.from_url(
    os.getenv("REDIS_URL", "redis://localhost:6379"),
    decode_responses=True  # Automatically decode bytes to strings
)


def cached(expire_seconds=300):
    """
    Caching decorator for functions.
    
    Args:
        expire_seconds: How long to cache results (default: 5 minutes)
    
    Usage:
        @cached(expire_seconds=300)
        def fetch_articles(category):
            # Expensive operation here
            return articles
    """
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            # Generate cache key from function name and arguments
            cache_key = f"{func.__name__}:{args}:{kwargs}"
            
            # Try to get cached result
            cached_result = redis_client.get(cache_key)
            if cached_result:
                print(f"Cache HIT for {cache_key}")
                return json.loads(cached_result)
            
            # Cache miss: call the actual function
            print(f"Cache MISS for {cache_key}")
            result = func(*args, **kwargs)
            
            # Store result in cache
            redis_client.setex(
                cache_key,
                expire_seconds,
                json.dumps(result)
            )
            
            return result
        
        return wrapper
    return decorator


# Track cache statistics
def get_cache_stats():
    """Get Redis cache statistics."""
    info = redis_client.info("stats")
    return {
        "hits": info.get("keyspace_hits", 0),
        "misses": info.get("keyspace_misses", 0),
        "keys": redis_client.dbsize()
    }

Now apply caching to your article fetching function. Update main.py:

from cache import cached, get_cache_stats
from news_sources import fetch_all_sources


@cached(expire_seconds=300)  # Cache for 5 minutes
def get_cached_articles(category: str = None, source: str = None):
    """
    Fetch articles with caching.
    First call: Fetches from external APIs (slow).
    Subsequent calls: Returns from Redis cache (fast).
    """
    articles = fetch_all_sources(category=category)
    
    # Filter by source if specified
    if source:
        articles = [a for a in articles if a["source"] == source]
    
    return articles


@app.get("/articles")
async def get_articles(
    category: Optional[str] = None,
    source: Optional[str] = None
):
    """Get articles with automatic caching."""
    articles = get_cached_articles(category=category, source=source)
    
    return {
        "articles": articles,
        "total": len(articles),
        "cached": True  # All responses after first are cached
    }


@app.get("/cache/stats")
async def cache_statistics():
    """Get cache performance statistics."""
    stats = get_cache_stats()
    
    hit_rate = 0
    total = stats["hits"] + stats["misses"]
    if total > 0:
        hit_rate = (stats["hits"] / total) * 100
    
    return {
        "hits": stats["hits"],
        "misses": stats["misses"],
        "total_keys": stats["keys"],
        "hit_rate_percent": round(hit_rate, 2)
    }

Check: Rebuild and restart your stack:

docker compose down
docker compose up --build

Make your first request to http://localhost:8000/articles?category=technology. Watch your terminal logs:

Cache MISS for get_cached_articles:('technology',):{}
GET /articles?category=technology - 687.34ms

Make the same request again immediately:

Cache HIT for get_cached_articles:('technology',):{}
GET /articles?category=technology - 4.23ms

Extract: The first request took 687ms (fetching from external APIs). The second request took 4ms (served from Redis cache). That's a 162x speedup. Check cache statistics at http://localhost:8000/cache/stats:

{
  "hits": 15,
  "misses": 3,
  "total_keys": 3,
  "hit_rate_percent": 83.33
}

After several requests, your cache hit rate stabilizes around 80-90%. This means 80-90% of requests are served from cache, dramatically reducing external API usage and improving response times.

Section 2 established baseline performance: 12 requests per second with 723ms average response time. Now test your cached API to measure the improvement.

Check: Run the same load test from Section 2:

hey -n 100 -c 10 http://localhost:8000/articles?category=technology

Summary:
  Total:        0.2387 secs
  Slowest:      0.0234 secs
  Fastest:      0.0032 secs
  Average:      0.0084 secs
  Requests/sec: 419.02

Response time histogram:
  0.003 [1]     |
  0.005 [12]    |■■■■■■
  0.007 [38]    |■■■■■■■■■■■■■■■■■■
  0.009 [25]    |■■■■■■■■■■■■
  0.011 [14]    |■■■■■■■
  0.013 [7]     |■■■
  0.015 [2]     |■
  0.017 [0]     |
  0.019 [0]     |
  0.021 [0]     |
  0.023 [1]     |

Latency distribution:
  10% in 0.0041 secs
  25% in 0.0056 secs
  50% in 0.0078 secs
  75% in 0.0095 secs
  90% in 0.0124 secs
  95% in 0.0142 secs
  99% in 0.0234 secs

Extract: Compare the results:

Before caching:

• Requests per second: 12.15
• Average response time: 723ms
• 95th percentile: 823ms

After caching:

• Requests per second: 419.02 (35x improvement)
• Average response time: 8.4ms (86x improvement)
• 95th percentile: 14.2ms (58x improvement)

Adding Redis caching transformed your API from handling 12 requests per second to 419 requests per second. The same hardware now serves 35 times more users. Average response time dropped from 723ms to 8.4ms, making the API feel instant. This is the power of caching done right.

These improvements come with one simple architectural addition: an in-memory cache. You didn't change application logic significantly. You didn't optimize database queries. You didn't add servers. You added caching for expensive operations (external API calls), and performance increased by orders of magnitude.

Use this quiz to check your understanding. Try to answer each question out loud or in a notebook before expanding the explanation. If you get stuck, that's a signal to revisit the relevant section.

You started this chapter with a News Aggregator API that worked perfectly on your laptop but required manual setup for teammates: install PostgreSQL, configure environment variables, run migrations, and start the server. The "works on my machine" problem made sharing your application tedious and deployment risky.

You containerized the application using Docker, packaging your Python code, runtime, dependencies, and configuration into a standardized image that runs identically anywhere. Multi-stage builds optimized image size from 450 MB to 187 MB. Layer caching enables fast rebuilds during development.

You orchestrated your multi-service stack (FastAPI application, PostgreSQL database, Redis cache) using Docker Compose, transforming three terminal windows and three complex commands into one command: docker compose up. Health checks ensure services start in the correct order. Named volumes preserve database data between restarts. Environment variables keep secrets out of version control.

You profiled application performance before optimizing, establishing baseline metrics (12 requests/second, 723ms average response time). You implemented Redis caching for expensive external API calls, improving throughput by 35x (419 requests/second) and reducing average response time by 86x (8.4ms). Load testing verified the improvements quantitatively.

Your News API now runs in a portable, production-ready stack. Teammates clone your repository and run docker compose up to start the full application immediately. The containerized stack is ready for AWS deployment in Chapter 28, where you'll push images to the cloud, run containers at scale, and provision managed databases.

Use this quiz to check your understanding of the entire chapter. Try to answer each question out loud or in a notebook before expanding the explanation. If you get stuck, that's a signal to revisit the relevant section.

Your containerized News API runs perfectly on your laptop. Docker Compose makes development effortless. You've optimized performance with Redis caching. One command starts the full stack. But everything still only exists on your laptop. Close your laptop and the API disappears. Want to show employers? You need to record a video or schedule a live demo. Need to handle real user traffic? Your laptop can't stay awake 24/7.

Chapter 28 takes your containerized application and deploys it to Amazon Web Services (AWS). You'll learn to push containers to the cloud using AWS ECR (Elastic Container Registry), making your Docker images accessible from anywhere. You'll run containers at scale using AWS ECS (Elastic Container Service) with Fargate, where AWS handles the infrastructure and you manage the application.

You'll replace local PostgreSQL and Redis with AWS RDS and ElastiCache—production-grade, automatically backed up, multi-availability-zone databases that scale independently of your application. You'll configure AWS Application Load Balancer to distribute requests across multiple containers, enabling horizontal scaling and zero-downtime deployments when you update code.

You'll build CI/CD pipelines with GitHub Actions so git push automatically tests, builds, and deploys your application with no manual steps. You'll set up CloudWatch logging and alarms to track application health, debug issues, and respond to problems before users notice. You'll implement auto-scaling policies that add containers during traffic spikes and remove them during quiet periods, optimizing costs.

By Chapter 28's end, your News API will be production-deployed on AWS with auto-scaling, monitoring, and automated deployment. You'll have a live URL you can share with recruiters, demonstrating not just coding ability but infrastructure competency. You'll understand how containerized applications scale from local development to global production deployment.

The transition from "works on my laptop" to "serves thousands of users globally" is the final step in your journey from script writer to professional API developer. You've built the application. You've containerized it. Now you'll deploy it at scale.