Project 3: Resilient Background Processing with Python, Redis, and Health Checks

Introduction: Building Resilient Background Services

In modern applications, many tasks are too time-consuming to execute synchronously within a user’s request. Think about sending email notifications, processing large image uploads, generating complex reports, or performing data analytics. Blocking the user interface for these operations leads to poor user experience and can cause timeouts. This is where background processing comes in.

This chapter guides you through building a resilient background worker system using Docker Compose. We’ll set up a Python worker that processes jobs from a Redis queue, ensuring data persistence and implementing health checks for service reliability. By the end, you’ll have a robust, containerized system capable of offloading heavy tasks, significantly improving your application’s responsiveness and fault tolerance.

Planning & Design: Asynchronous Task Execution

The core problem we’re solving is how to execute long-running or resource-intensive tasks without directly impacting the main application’s performance or user experience. Our solution involves a classic message queue pattern: a producer pushes tasks onto a queue, and one or more consumers (workers) pull tasks from the queue and process them asynchronously.

Architecture Overview

Our system will consist of three main logical components, all orchestrated by Docker Compose:

Producer (Simulated): A simple Python script that simulates a web application pushing jobs (e.g., “process image”, “send email”) onto a Redis queue.
Redis: An in-memory data store used as our message broker. It will store the job queue and ensure jobs persist even if the worker temporarily fails.
Python Worker: A Python application that continuously monitors the Redis queue, fetches pending jobs, performs a simulated “heavy computation,” and marks the job as complete.

This setup decouples the task creation from task execution, making the system more scalable and fault-tolerant.

Key Docker Concepts Applied

In this project, we’ll leverage several critical Docker concepts:

Docker Compose: To define and run our multi-service application (Redis and Python Worker) as a single unit.
Custom Dockerfile: For our Python worker, enabling us to install dependencies and configure the environment. We’ll use multi-stage builds for efficiency.
Docker Volumes: To ensure Redis data (our job queue) persists even if the Redis container is stopped or restarted.
Docker Networks: Docker Compose automatically sets up a default network, allowing services to communicate by their service names (e.g., redis).
Environment Variables: To configure our worker (e.g., the Redis host) without hardcoding values.
Health Checks: To monitor the operational status of our worker service, allowing Docker Compose to react if a service becomes unhealthy.
Non-root User: Running containers with a dedicated, non-root user for enhanced security.

Step-by-Step Implementation

Let’s start building our background processing system. We’ll create a new directory for this project.

mkdir project3-background-worker
cd project3-background-worker

Step 1: Set up the Python Worker Application

First, we’ll create the Python application that will act as our worker.

Create a directory named worker inside project3-background-worker:

mkdir worker

Inside the worker directory, create a file named requirements.txt:

project3-background-worker/worker/requirements.txt

redis==5.0.1

⚡ Quick Note: We’re using redis==5.0.1. The redis-py library is the official Python client for Redis, providing a straightforward API for interacting with Redis servers. We specify a version to ensure reproducibility.

Next, create the main worker script, app.py:

project3-background-worker/worker/app.py

import os
import time
import redis
import json

# Retrieve Redis connection details from environment variables
REDIS_HOST = os.getenv('REDIS_HOST', 'localhost')
REDIS_PORT = int(os.getenv('REDIS_PORT', 6379))
REDIS_DB = int(os.getenv('REDIS_DB', 0))

# Connect to Redis
try:
    r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB, decode_responses=True)
    r.ping()
    print(f"Connected to Redis at {REDIS_HOST}:{REDIS_PORT}")
except redis.exceptions.ConnectionError as e:
    print(f"Could not connect to Redis: {e}")
    exit(1)

# Function to simulate a heavy task
def process_job(job_data):
    print(f"Processing job: {job_data['id']} - Type: {job_data['type']}")
    # Simulate work by sleeping for a random duration
    time.sleep(job_data.get('duration', 5))
    print(f"Job {job_data['id']} completed after {job_data.get('duration', 5)} seconds.")
    return f"Job {job_data['id']} processed."

# Main worker loop
if __name__ == "__main__":
    print("Python Worker started. Waiting for jobs...")
    while True:
        # Blocking pop from the 'task_queue' list in Redis
        # This will wait indefinitely until a job is available
        _, job_json = r.brpop('task_queue', timeout=60) # timeout is for graceful shutdown in real apps

        if job_json:
            job_data = json.loads(job_json)
            try:
                process_job(job_data)
            except Exception as e:
                print(f"Error processing job {job_data.get('id', 'unknown')}: {e}")
        else:
            print("No job in queue, waiting...")
        time.sleep(1) # Small delay to prevent busy-waiting if brpop times out

Explanation of app.py:

Environment Variables: REDIS_HOST, REDIS_PORT, REDIS_DB are loaded from environment variables, making the worker configurable without code changes. Defaults are provided for local testing.
Redis Connection: It attempts to connect to Redis and pings it to verify the connection. If connection fails, the worker exits.
process_job: This function simulates a long-running task using time.sleep(). In a real application, this would be where your actual business logic (e.g., image resizing, API calls) resides.
Worker Loop: The while True loop continuously fetches jobs.
r.brpop('task_queue', timeout=60): This is a crucial Redis command. BRPOP (blocking right pop) removes and returns the last element from the task_queue list. If the list is empty, it blocks the connection until an element is available or the timeout (60 seconds here) is reached. This is efficient as it doesn’t busy-wait.
JSON Handling: Jobs are expected to be JSON strings, which are then parsed.
Error Handling: Basic try-except block around job processing to prevent a single bad job from crashing the worker.

Step 2: Create the Dockerfile for the Python Worker

Now, let’s containerize our Python worker. We’ll use a multi-stage build to create a lean production image.

Create a file named Dockerfile inside the worker directory:

project3-background-worker/worker/Dockerfile

# Stage 1: Build stage
# Use a specific Python version for consistency.
# Python 3.12-slim-bookworm is a good balance of features and size for 2026.
FROM python:3.12-slim-bookworm AS builder

# Set working directory
WORKDIR /app

# Install build dependencies if needed (e.g., for some pip packages)
# For 'redis-py', often no specific build dependencies are strictly required
# but it's good practice to include them if your project grows.
# RUN apt-get update && apt-get install -y --no-install-recommends \
#     build-essential \
#     && rm -rf /var/lib/apt/lists/*

# Copy requirements file and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Stage 2: Production stage
# Use a minimal base image for the final runtime
FROM python:3.12-slim-bookworm

# Set working directory
WORKDIR /app

# Create a non-root user for security
RUN adduser --system --group appuser
USER appuser

# Copy only the installed dependencies and application code from the builder stage
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY app.py .

# Command to run the worker
CMD ["python", "app.py"]

Explanation of Dockerfile:

FROM python:3.12-slim-bookworm AS builder: Starts with a slim Python image (based on Debian Bookworm) for the build stage. This is a current, lightweight base image as of 2026-08-06.
WORKDIR /app: Sets the working directory inside the container.
COPY requirements.txt . & RUN pip install ...: Copies the requirements.txt and installs Python dependencies. Using --no-cache-dir reduces image size.
FROM python:3.12-slim-bookworm: The second stage uses the same minimal base image.
RUN adduser --system --group appuser & USER appuser: Crucially, this creates a non-root appuser and switches to it. Running containers as non-root is a fundamental security best practice, preventing potential privilege escalation if vulnerabilities are exploited.
COPY --from=builder ...: This copies only the installed Python packages from the builder stage to the production stage.
COPY app.py .: Copies our worker application script. Since the build context is ./worker and WORKDIR is /app, app.py refers to project3-background-worker/worker/app.py.
CMD ["python", "app.py"]: Defines the command that runs when the container starts.

Step 3: Define Services with Docker Compose

Now, let’s orchestrate our Redis and Python worker services using Docker Compose.

Create a file named docker-compose.yaml in the root project3-background-worker directory:

project3-background-worker/docker-compose.yaml

# Use a recent Compose file format for Docker Compose v2.40.3
version: '3.8'

services:
  redis:
    image: redis:7.2.5-alpine # A lightweight and stable Redis version as of 2026-08-06
    container_name: project3_redis
    ports:
      - "6379:6379" # Expose Redis port for local testing/debugging
    volumes:
      - redis_data:/data # Persist Redis data to a named volume
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s

  worker:
    build:
      context: ./worker # Build context is the 'worker' directory
      dockerfile: Dockerfile
    container_name: project3_worker
    environment:
      # Pass Redis connection details to the worker
      REDIS_HOST: redis # Use the service name 'redis' for inter-container communication
      REDIS_PORT: 6379
      REDIS_DB: 0
    depends_on:
      redis:
        condition: service_healthy # Ensure Redis is healthy before starting the worker
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f app.py || exit 1"] # Check if the Python process is running
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s
    restart: on-failure # Automatically restart if the worker crashes

volumes:
  redis_data: # Define the named volume for Redis persistence

Explanation of docker-compose.yaml:

version: '3.8': Specifies the Compose file format version. 3.8 is a common and robust choice compatible with Docker Compose 2.40.3.
redis service:
- image: redis:7.2.5-alpine: Uses the official redis image, specifically a lightweight Alpine Linux variant for a smaller footprint. Version 7.2.5 is a stable release as of 2026-08-06.
- ports: - "6379:6379": Maps the container’s Redis port to the host’s port, allowing you to connect to Redis directly from your host if needed.
- volumes: - redis_data:/data: Mounts a named Docker volume (redis_data) to the /data directory inside the Redis container. This is where Redis persists its data (RDB snapshots, AOF logs), ensuring that our job queue is not lost if the container restarts.
- healthcheck: Defines how Docker Compose can determine if the Redis service is healthy. It uses redis-cli ping to check connectivity. start_period gives Redis time to initialize.
worker service:
- build: context: ./worker: Tells Docker Compose to build the image for this service using the Dockerfile found in the ./worker directory.
- environment: Passes the REDIS_HOST, REDIS_PORT, and REDIS_DB environment variables to the worker container. Notice REDIS_HOST: redis – Docker Compose automatically sets up internal DNS resolution, so containers can reach each other using their service names.
- depends_on: redis: condition: service_healthy: This is a critical feature. It ensures that the worker service will only start once the redis service is reported as healthy by its health check. This prevents the worker from trying to connect to a Redis instance that isn’t fully ready.
- healthcheck: For the worker, we use pgrep -f app.py to check if our Python script is actively running. If pgrep fails (returns a non-zero exit code), the health check fails.
- restart: on-failure: Configures the worker container to automatically restart if it exits with a non-zero status (i.e., crashes). This adds resilience.
volumes: redis_data:: Declares the redis_data named volume, which Docker will manage.

Step 4: Create a Job Producer (for Testing)

To test our worker, we need a way to put jobs into the Redis queue.

Create a file named producer.py in the root project3-background-worker directory:

project3-background-worker/producer.py

import os
import redis
import json
import time
import uuid
import random

# Retrieve Redis connection details from environment variables
# For local execution, assume Redis is on localhost if not containerized
REDIS_HOST = os.getenv('REDIS_HOST', 'localhost')
REDIS_PORT = int(os.getenv('REDIS_PORT', 6379))
REDIS_DB = int(os.getenv('REDIS_DB', 0))

try:
    r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB, decode_responses=True)
    r.ping()
    print(f"Producer connected to Redis at {REDIS_HOST}:{REDIS_PORT}")
except redis.exceptions.ConnectionError as e:
    print(f"Producer could not connect to Redis: {e}")
    exit(1)

def publish_job(job_type, duration=None):
    job_id = str(uuid.uuid4())
    job_data = {
        'id': job_id,
        'type': job_type,
        'timestamp': time.time()
    }
    if duration:
        job_data['duration'] = duration

    job_json = json.dumps(job_data)
    r.lpush('task_queue', job_json) # Push job to the left of the list
    print(f"Published job {job_id} of type '{job_type}' to 'task_queue'.")

if __name__ == "__main__":
    print("Starting job producer...")
    while True:
        # Publish different types of jobs with varying durations
        job_types = ["image_resize", "send_email", "report_generation", "data_cleanup"]
        selected_type = random.choice(job_types)
        job_duration = random.randint(2, 10) # Simulate 2-10 seconds of work

        publish_job(selected_type, job_duration)
        time.sleep(random.randint(1, 3)) # Wait 1-3 seconds before publishing next job

Explanation of producer.py:

Redis Connection: Similar to the worker, it connects to Redis. For this script, we’ll run it from our host machine, so REDIS_HOST defaults to localhost.
publish_job: Creates a unique job ID, constructs a JSON payload, and uses r.lpush('task_queue', job_json) to add the job to the left side of the Redis list. LPUSH is the counterpart to BRPOP for pushing items to the queue.
Loop: Continuously publishes random job types with random durations every few seconds.

Testing & Verification

Now that all our files are in place, let’s bring up our services and verify everything is working.

Build and Run Services: Navigate to the project3-background-worker directory in your terminal and run:
```
docker compose up --build -d
```
- up: Starts the services defined in docker-compose.yaml.
- --build: Forces Docker Compose to rebuild images if their definitions (like our worker/Dockerfile) have changed.
- -d: Runs the containers in detached mode (in the background).
You should see output indicating that the services are being created and started. Pay attention to the condition: service_healthy for redis before worker starts.
Monitor Worker Logs: To see the worker processing jobs, stream its logs:
```
docker compose logs -f worker
```
Initially, the worker will start and print “Python Worker started. Waiting for jobs…” and “No job in queue, waiting…”. This is expected because we haven’t sent any jobs yet.
Send Jobs with the Producer: Open a new terminal window, navigate to project3-background-worker, and run the producer script directly on your host machine:
```
python producer.py
```
You should see output in this terminal indicating jobs being published to Redis.

Observe Worker Processing: Switch back to the terminal where you’re watching the worker logs. You should now see the worker picking up jobs from the queue, printing “Processing job…” and “Job … completed…” messages.

project3_worker_1  | Python Worker started. Waiting for jobs...
project3_worker_1  | Connected to Redis at redis:6379
project3_worker_1  | Processing job: e1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 - Type: image_resize
project3_worker_1  | Job e1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 completed after 5 seconds.
project3_worker_1  | Processing job: f1g2h3i4-j5k6-7890-a1b2-c3d4e5f67890 - Type: send_email
project3_worker_1  | Job f1g2h3i4-j5k6-7890-a1b2-c3d4e5f67890 completed after 8 seconds.

Check Service Health: In another terminal, you can check the health status of your services:

docker compose ps

You should see (healthy) next to both project3_redis and project3_worker. This confirms our health checks are working.

NAME               IMAGE                    COMMAND                  SERVICE    CREATED          STATUS                   PORTS
project3_redis     redis:7.2.5-alpine       "docker-entrypoint.s…"   redis      10 seconds ago   running (healthy)        0.0.0.0:6379->6379/tcp
project3_worker    project3-background-wo…  "python app.py"          worker     8 seconds ago    running (healthy)

Verify Redis Data Persistence (Optional): Stop the redis service:
```
docker compose stop redis
```
Wait a few seconds, then start it again:
```
docker compose start redis
```
The worker should reconnect to redis (after its own restart or connection retry logic). If you stop the producer.py and let the worker clear the queue, then restart redis, any jobs still in the queue before the stop would still be there after the restart. This demonstrates the volume’s effectiveness.
When you’re done, stop and remove the containers:
```
docker compose down
```
This command will stop and remove the containers and networks. Note that by default, docker compose down does not remove named volumes. To also remove the redis_data volume (and thus clear any persistent Redis data), use:
```
docker compose down --volumes
```

Production Considerations

Building a background processing system for production requires more than just functional code.

Scaling Workers: Docker Compose allows you to easily scale your workers. To run multiple worker instances, use:
```
docker compose up --scale worker=3 -d
```
This will start three worker containers, all consuming jobs from the same Redis queue. This is a fundamental way to increase throughput for background tasks.
Robustness and Retries: Our current worker processes a job once. In a real system, jobs might fail due to transient issues (e.g., external API downtime). Implement retry mechanisms (e.g., exponential backoff) and potentially a dead-letter queue (DLQ) for jobs that consistently fail after multiple retries.
Monitoring and Alerting: Integrate logging (e.g., structured JSON logs for easier parsing), metrics (e.g., job processing rates, queue length), and alerting (e.g., notify if queue length is too high or workers are unhealthy). Tools like Prometheus and Grafana are commonly used for this.
Resource Limits: Define CPU and memory limits for your worker containers in docker-compose.yaml to prevent a runaway worker from consuming all host resources:
```
worker:
  # ... other configurations
  deploy:
    resources:
      limits:
        cpus: '0.5' # 50% of a CPU core
        memory: 512M # 512 MB of RAM
```
🧠 Important: These deploy limits are typically honored by orchestrators like Swarm or Kubernetes, but Docker Desktop and docker compose can apply them locally as well.
Security for Redis: While we exposed Redis on 6379 for convenience, in production, Redis should generally not be exposed directly to the public internet. Use a private network, strong passwords (with the requirepass directive in Redis config), or TLS encryption. Docker secrets can be used to pass sensitive passwords securely.

Common Issues & Solutions

Worker Cannot Connect to Redis:
- Issue: You see Could not connect to Redis errors in the worker logs.
- Solution:
  - Verify the REDIS_HOST environment variable in docker-compose.yaml for the worker service. It should be redis (the service name), not localhost.
  - Check if the redis service is actually running and healthy (docker compose ps).
  - Ensure no firewall is blocking internal Docker network communication (less common for Docker Compose default networks).
  - Check Redis logs for any startup errors.
Jobs Aren’t Being Processed (Worker is Idle):
- Issue: producer.py is pushing jobs, but the worker logs show “No job in queue, waiting…”.
- Solution:
  - Confirm producer.py is connected to the correct Redis instance. If running producer.py on the host, ensure Redis is exposed via ports: "6379:6379" in docker-compose.yaml and producer.py is configured to connect to localhost:6379.
  - Check for typos in the queue name (task_queue) in both producer.py and worker/app.py.
  - Verify the worker container is running and healthy (docker compose ps).
Redis Data Loss After Container Restart:
- Issue: Jobs disappear from the queue if you stop and restart the redis container.
- Solution: Ensure the volumes: - redis_data:/data line is correctly configured for the redis service in docker-compose.yaml, and the volumes: redis_data: declaration exists at the top level. This named volume ensures data persistence.

Summary & Next Steps

In this chapter, you’ve successfully built a resilient background processing system using Python, Redis, and Docker Compose. You’ve learned how to:

Containerize a multi-service application.
Implement data persistence using Docker volumes.
Configure inter-container communication.
Use environment variables for flexible configuration.
Add health checks for robust service management.
Apply multi-stage Docker builds and non-root users for security and efficiency.

This project demonstrates a common pattern for building scalable and reliable microservices. The concepts of message queues, workers, and health checks are fundamental in distributed systems.

Next, we’ll explore more advanced Docker concepts, including custom networking, volume types, and image optimization techniques, to further enhance your containerization skills.

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.