Introduction: Beyond Deployment to Operational Excellence

Deploying a containerized application is a significant achievement, but the true test of engineering lies in its operational readiness. This chapter shifts our focus from “making it run” to “keeping it running reliably.” We’ll dive into the critical aspects of observability—logging and metrics—and explore essential operational practices that ensure your Dockerized applications are robust, maintainable, and resilient in production environments.

By the end of this chapter, you will have built a basic, yet functional, observability stack for your sample application. You’ll understand how to instrument your code for effective monitoring, aggregate logs from distributed containers, and visualize key performance indicators (KPIs) and health metrics using industry-standard open-source tools. This foundation will empower you to diagnose issues quickly, prevent outages, and ensure a smooth operational experience.

Why Observability is Non-Negotiable

In modern, distributed systems, especially those leveraging microservices and containers, understanding the internal state of your applications becomes incredibly complex. Services interact asynchronously, failures can cascade, and traditional debugging methods often fall short. Observability is the capability to understand what’s happening inside a system by examining the data it produces. It’s typically categorized into three pillars:

  1. Logs: Discrete, timestamped records of events. These are crucial for understanding specific actions, errors, or state changes within an application’s lifecycle. Think of them as the detailed forensic evidence of your system.
  2. Metrics: Numerical measurements collected over time, representing the health, performance, or resource utilization of a service. Metrics provide aggregated views, trends, and enable proactive monitoring and alerting. Examples include CPU usage, request latency, error rates, and active connections.
  3. Traces: End-to-end representations of requests as they flow through multiple services in a distributed system. Tracing helps pinpoint bottlenecks and latency issues across service boundaries.

For our projects, we will build a strong foundation in structured logging and metrics. Without these, operating a system at scale is like navigating a ship in dense fog—you’re reacting to collisions rather than proactively steering clear.

Project Setup: A Centralized Observability Stack

Before we dive into implementation, let’s establish a clear project structure for our observability stack. We’ll create a root directory, docker-observability-project, and place our docker-compose.yaml file there. Each component (the application, Fluentd, Prometheus, Grafana) will reside in its own subdirectory.

docker-observability-project/
├── docker-compose.yaml
├── my-log-app/
│   ├── app.py
│   ├── Dockerfile
│   └── requirements.txt
├── fluentd/
│   ├── fluent.conf
│   └── Dockerfile
├── prometheus/
│   └── prometheus.yml
└── grafana/
    # (No custom files needed in 'grafana' for this setup, but the directory is good practice)

Action: Create the docker-observability-project directory and the subdirectories listed above.

mkdir docker-observability-project
cd docker-observability-project
mkdir my-log-app fluentd prometheus grafana
touch docker-compose.yaml

Now, let’s start by creating our example application within my-log-app/.

Structured Logging: Making Logs Machine-Readable

Traditional plain-text logs are difficult to parse and analyze at scale. Structured logging, typically in JSON format, embeds rich metadata with each log entry, making it machine-readable, queryable, and highly valuable for analytics.

Emitting Structured Logs from Applications

The best practice for containerized applications is to write logs to stdout (standard output) and stderr (standard error). Docker’s logging drivers can then capture these streams, decoupling your application from the underlying logging infrastructure.

Let’s create a simple Python application that emits structured JSON logs.

1. Create a Python application: Inside my-log-app/, create app.py:

# docker-observability-project/my-log-app/app.py
import logging
import json
import time
import os
import random

# Configure basic logging to stdout. We'll let the application format JSON.
logging.basicConfig(level=logging.INFO, format='%(message)s')
logger = logging.getLogger(__name__)

def log_event(event_type, message, **kwargs):
    """
    Constructs and logs a structured JSON event to stdout.
    """
    log_entry = {
        "timestamp": time.time(),
        "level": "INFO",
        "service": os.getenv("SERVICE_NAME", "unknown-service"),
        "event_type": event_type,
        "message": message,
        **kwargs # Add any additional keyword arguments as context
    }
    logger.info(json.dumps(log_entry))

if __name__ == "__main__":
    count = 0
    while True:
        count += 1
        request_id = f"req-{count:03d}"
        user_id = f"user-{random.randint(100, 999)}"

        log_event(
            "request_processed",
            f"Processing request {request_id}",
            request_id=request_id,
            user_id=user_id,
            duration_ms=round(random.uniform(10, 200), 2)
        )
        if count % 5 == 0:
            log_event(
                "warning",
                f"High load detected for request {request_id}",
                request_id=request_id,
                threshold=0.8,
                current_load=random.uniform(0.7, 0.95)
            )
        time.sleep(1)

This Python script uses the logging module to print JSON strings to stdout. Each log entry includes core fields like timestamp, level, service, event_type, and a message, along with dynamic, context-specific fields like request_id and duration_ms.

2. Create a Dockerfile for the application: Inside my-log-app/, create Dockerfile:

# docker-observability-project/my-log-app/Dockerfile
# Use a multi-stage build for smaller, more secure images
FROM python:3.11-slim-bookworm AS builder

# Install build dependencies
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Final stage for the runtime image
FROM python:3.11-slim-bookworm

WORKDIR /app
# Copy only the installed packages from the builder stage
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY app.py .

# Set environment variable for service identification in logs
ENV SERVICE_NAME="my-log-service"

# Command to run the application
CMD ["python", "app.py"]

This Dockerfile sets up a Python environment and runs app.py. The multi-stage build ensures our final image is lean, containing only necessary runtime components and not build tools.

3. Create requirements.txt: Inside my-log-app/, create requirements.txt. For this basic logging app, it’s empty, but we include it for good practice and future expansion.

# docker-observability-project/my-log-app/requirements.txt
# No external dependencies for basic logging, but good practice to include.

Running the Application with Docker Compose

Now, let’s define our service in docker-compose.yaml to run our my-log-app.

1. Create docker-compose.yaml: At the project root (docker-observability-project/), create docker-compose.yaml:

# docker-observability-project/docker-compose.yaml
version: '3.8'

services:
  log-app:
    build:
      context: ./my-log-app # Build from the my-log-app directory
      dockerfile: Dockerfile
    container_name: log-app-instance
    environment:
      SERVICE_NAME: "my-log-service"
    logging:
      driver: "json-file" # This is the default, explicitly shown for clarity
      options:
        max-size: "10m" # Rotate log files at 10MB
        max-file: "3"   # Keep a maximum of 3 log files

Here, we explicitly specify the json-file logging driver and its options for log rotation. This prevents log files from consuming all disk space on the host.

2. Build and run the container:

cd docker-observability-project
docker compose up -d --build

Verification: To view the logs:

docker logs log-app-instance

You should see JSON log lines streaming to your terminal, confirming the application is running and emitting structured logs. Press Ctrl+C to stop docker logs.

{"timestamp": 1678886400.123, "level": "INFO", "service": "my-log-service", "event_type": "request_processed", "message": "Processing request req-001", "request_id": "req-001", "user_id": "user-123", "duration_ms": 55.23}

This output is clean and ready for ingestion by a log aggregator.

Centralized Logging with Fluentd

While docker logs is useful for immediate inspection, production systems require a centralized log aggregation system. This system collects logs from all containers, stores them efficiently, and makes them searchable and analyzable. Fluentd is a popular open-source data collector for unified logging.

Logging Architecture Overview

flowchart TD AppService[Application Container] -->|Logs to stdout/stderr| DockerEngine[Docker Engine] DockerEngine -->|Forwards logs via Driver| FluentdAgent[Fluentd Container] FluentdAgent -->|Processes and forwards logs| CentralLogStore[Central Log Store] CentralLogStore -->|Stores logs| Kibana[Kibana for Visualization]

Setting up Fluentd with Docker Compose

We’ll use Fluentd as a log shipper. For this example, Fluentd will collect logs from our log-app service and print them to its own stdout, simulating forwarding to a centralized system like Elasticsearch.

1. Create a fluentd configuration: Inside fluentd/, create fluent.conf:

# docker-observability-project/fluentd/fluent.conf
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

# This match rule captures all incoming logs (**)
<match **>
  @type stdout
  # For demonstration, we're outputting to stdout.
  # In a production setup, this would typically be an output plugin like:
  # @type elasticsearch
  # @type s3
  # @type kafka
  <format>
    @type json # Ensure output is JSON for easier parsing
  </format>
</match>

This fluent.conf configures Fluentd to listen for forwarded logs on port 24224 and then output all received logs to its own stdout in JSON format.

2. Create a Dockerfile for Fluentd: Inside fluentd/, create Dockerfile:

# docker-observability-project/fluentd/Dockerfile
FROM fluent/fluentd:v1.16-debian-1.0 # Using a stable Fluentd image as of 2026-08-06

# Install necessary plugins if needed.
# For this basic example, no extra plugins are required.
# Example: RUN gem install fluent-plugin-elasticsearch

We’re using an official Fluentd image from Docker Hub.

3. Update docker-compose.yaml to include Fluentd: Modify your docker-compose.yaml (at the project root) to add the fluentd service and configure log-app to use the fluentd logging driver.

# docker-observability-project/docker-compose.yaml
version: '3.8'

services:
  log-app:
    build:
      context: ./my-log-app
      dockerfile: Dockerfile
    container_name: log-app-instance
    environment:
      SERVICE_NAME: "my-log-service"
    logging:
      driver: "fluentd" # Use the fluentd logging driver
      options:
        fluentd-address: fluentd:24224 # Point to the fluentd service within the Docker network
        tag: "docker.log-app" # A tag for Fluentd to identify logs from this service

  fluentd:
    build:
      context: ./fluentd # Build Fluentd from the 'fluentd' directory
      dockerfile: Dockerfile
    container_name: fluentd-aggregator
    volumes:
      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro # Mount Fluentd config as read-only
    ports:
      - "24224:24224" # Expose Fluentd's forward port to the host for potential external shippers
    command: ["fluentd", "-c", "/fluentd/etc/fluent.conf"]
    # Fluentd needs to be accessible by log-app, Docker Compose handles this via service names

Notice the fluentd-address: fluentd:24224. Within a Docker Compose network, services can refer to each other by their service names (e.g., fluentd). This is a key feature of Docker’s internal DNS.

4. Stop existing containers and run the full stack:

docker compose down # Stop and remove previous containers
docker compose up -d --build

Verification: Check the fluentd container logs:

docker logs fluentd-aggregator

You should now see the structured JSON logs from log-app-instance being received and printed by the fluentd-aggregator container. This confirms your log forwarding is working correctly.

{"json":{"timestamp":1678886400.123,"level":"INFO","service":"my-log-service","event_type":"request_processed","message":"Processing request req-001","request_id":"req-001","user_id":"user-123","duration_ms":55.23},"container_id":"...","container_name":"/log-app-instance","source":"stdout","log_tag":"docker.log-app","stream":"stdout","time":"2026-03-15T10:40:00.123456789Z"}

Fluentd adds its own metadata (like container_id, container_name, log_tag, time) around your application’s JSON log, enriching the data for further analysis.

Metrics Collection with Prometheus

Metrics provide numerical insights into your application’s health and performance over time. Prometheus is an open-source monitoring system that collects metrics from configured targets by “scraping” their HTTP endpoints at regular intervals. It uses a “pull” model, where Prometheus initiates the data collection.

Metrics Architecture Overview

flowchart TD AppService[Application Container] -->|Exposes /metrics endpoint| DockerNetwork[Docker Network] Prometheus[Prometheus Container] -->|Scrapes /metrics| AppService Grafana[Grafana Container] -->|Queries metrics| Prometheus

Instrumenting an Application for Prometheus

Applications need to expose a /metrics endpoint in a Prometheus-compatible text format. For Python, the prometheus_client library simplifies this.

1. Update my-log-app/app.py to expose metrics: Modify my-log-app/app.py to include Prometheus instrumentation.

# docker-observability-project/my-log-app/app.py
import logging
import json
import time
import os
import random
from prometheus_client import start_http_server, Counter, Gauge, Histogram

# Configure basic logging to stdout
logging.basicConfig(level=logging.INFO, format='%(message)s')
logger = logging.getLogger(__name__)

# Prometheus Metrics
# Counter: A cumulative metric that represents a single monotonically increasing counter.
REQUESTS_TOTAL = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint'])
# Histogram: Samples observations (e.g., request durations) and counts them in configurable buckets.
REQUEST_DURATION_SECONDS = Histogram('http_request_duration_seconds', 'HTTP Request duration in seconds', ['method', 'endpoint'])
# Gauge: A metric that represents a single numerical value that can arbitrarily go up and down.
ACTIVE_REQUESTS = Gauge('active_requests', 'Number of active requests')
SERVICE_HEALTH = Gauge('service_health', 'Service health status (1=healthy, 0=unhealthy)')

def log_event(event_type, message, **kwargs):
    """
    Constructs and logs a structured JSON event to stdout.
    """
    log_entry = {
        "timestamp": time.time(),
        "level": "INFO",
        "service": os.getenv("SERVICE_NAME", "unknown-service"),
        "event_type": event_type,
        "message": message,
        **kwargs
    }
    logger.info(json.dumps(log_entry))

if __name__ == "__main__":
    # Start up the Prometheus HTTP server to expose the metrics.
    metrics_port = int(os.getenv("METRICS_PORT", 8000))
    start_http_server(metrics_port)
    logger.info(f"Prometheus metrics exposed on port {metrics_port}")

    count = 0
    SERVICE_HEALTH.set(1) # Initialize service as healthy

    while True:
        count += 1
        method = random.choice(['GET', 'POST', 'PUT'])
        endpoint = random.choice(['/', '/data', '/status', '/users'])

        # Increment active requests gauge
        ACTIVE_REQUESTS.inc()
        # Measure request duration using the histogram context manager
        with REQUEST_DURATION_SECONDS.labels(method, endpoint).time():
            # Simulate processing work
            time.sleep(random.uniform(0.01, 0.2))
            # Increment total requests counter
            REQUESTS_TOTAL.labels(method, endpoint).inc()
        # Decrement active requests gauge
        ACTIVE_REQUESTS.dec()

        request_id = f"req-{count:03d}"
        user_id = f"user-{random.randint(100, 999)}"

        log_event(
            "request_processed",
            f"Processing request {request_id}",
            request_id=request_id,
            user_id=user_id,
            duration_ms=round(random.uniform(10, 200), 2)
        )
        if count % 10 == 0:
            log_event(
                "warning",
                f"High load detected for request {request_id}",
                request_id=request_id,
                threshold=0.8,
                current_load=random.uniform(0.7, 0.95)
            )
            if random.random() < 0.15: # 15% chance to become unhealthy
                SERVICE_HEALTH.set(0)
                log_event("health_status_change", "Service became unhealthy!", status="unhealthy")
            else:
                SERVICE_HEALTH.set(1) # Back to healthy (if it was unhealthy)

        time.sleep(1)

This updated app.py initializes various Prometheus metrics (Counter, Gauge, Histogram), starts an HTTP server to expose them on a configurable port (METRICS_PORT), and then updates these metrics as it simulates processing requests.

2. Update my-log-app/requirements.txt: Add the Prometheus client library dependency.

# docker-observability-project/my-log-app/requirements.txt
prometheus_client==0.20.0 # Latest stable as of 2026-08-06

3. Update my-log-app/Dockerfile: Add the METRICS_PORT environment variable and EXPOSE instruction.

# docker-observability-project/my-log-app/Dockerfile
FROM python:3.11-slim-bookworm AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim-bookworm

WORKDIR /app
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY app.py .

ENV SERVICE_NAME="my-log-service"
ENV METRICS_PORT=8000 # Metrics server will listen on this port
EXPOSE 8000 # Inform Docker that the container listens on this port

CMD ["python", "app.py"]

Setting up Prometheus with Docker Compose

1. Create Prometheus configuration: Inside prometheus/, create prometheus.yml:

# docker-observability-project/prometheus/prometheus.yml
global:
  scrape_interval: 15s # How frequently Prometheus scrapes targets
  evaluation_interval: 15s # How frequently Prometheus evaluates alerting rules

scrape_configs:
  - job_name: 'docker_services'
    # metrics_path defaults to /metrics
    # scheme defaults to http
    static_configs:
      - targets: ['log-app:8000'] # Scrape our log-app service on its metrics port

Here, log-app is the service name defined in docker-compose.yaml. Docker Compose’s internal DNS resolves log-app to the container’s IP address, allowing Prometheus to discover and scrape its metrics endpoint.

2. Update docker-compose.yaml to include Prometheus: Modify your docker-compose.yaml (at the project root) to add the prometheus service and expose the log-app’s metrics port within the Docker network.

# docker-observability-project/docker-compose.yaml
version: '3.8'

services:
  log-app:
    build:
      context: ./my-log-app
      dockerfile: Dockerfile
    container_name: log-app-instance
    environment:
      SERVICE_NAME: "my-log-service"
      METRICS_PORT: 8000
    # Expose metrics port for external access (e.g., for direct testing via localhost:8000/metrics)
    ports:
      - "8000:8000"
    logging:
      driver: "fluentd"
      options:
        fluentd-address: fluentd:24224
        tag: "docker.log-app"

  fluentd:
    build:
      context: ./fluentd
      dockerfile: Dockerfile
    container_name: fluentd-aggregator
    volumes:
      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
    ports:
      - "24224:24224"
    command: ["fluentd", "-c", "/fluentd/etc/fluent.conf"]

  prometheus:
    image: prom/prometheus:v2.49.1 # Latest stable as of 2026-08-06
    container_name: prometheus-server
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro # Mount config read-only
      - prometheus_data:/prometheus # Persistent storage for metrics data
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - "9090:9090" # Expose Prometheus UI to host port 9090

volumes:
  prometheus_data: # Define a named volume for Prometheus data persistence

We’ve added the prometheus service, mapped its configuration file and a persistent data volume, and exposed its UI on port 9090.

3. Stop existing containers and run all services:

docker compose down
docker compose up -d --build

Verification:

  1. Access log-app metrics directly: Open your browser to http://localhost:8000/metrics. You should see a page with Prometheus-formatted metrics, confirming the application is exposing them correctly.
  2. Access Prometheus UI: Open your browser to http://localhost:9090.
  3. Navigate to “Status” -> “Targets”. You should see log-app:8000 listed as “UP”, indicating Prometheus is successfully scraping metrics.
  4. Go to the “Graph” tab. In the expression bar, type http_requests_total and click “Execute”. You should see a graph of the total request count increasing over time. Try other metrics like active_requests or service_health.

Visualization with Grafana

Grafana is an open-source analytics and interactive visualization web application. It connects to various data sources (like Prometheus) and allows you to create powerful dashboards to monitor and analyze your application’s metrics and logs.

Setting up Grafana with Docker Compose

1. Update docker-compose.yaml to add the Grafana service: Modify your docker-compose.yaml (at the project root) to include the grafana service.

# docker-observability-project/docker-compose.yaml
version: '3.8'

services:
  log-app:
    build:
      context: ./my-log-app
      dockerfile: Dockerfile
    container_name: log-app-instance
    environment:
      SERVICE_NAME: "my-log-service"
      METRICS_PORT: 8000
    ports:
      - "8000:8000"
    logging:
      driver: "fluentd"
      options:
        fluentd-address: fluentd:24224
        tag: "docker.log-app"

  fluentd:
    build:
      context: ./fluentd
      dockerfile: Dockerfile
    container_name: fluentd-aggregator
    volumes:
      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
    ports:
      - "24224:24224"
    command: ["fluentd", "-c", "/fluentd/etc/fluent.conf"]

  prometheus:
    image: prom/prometheus:v2.49.1
    container_name: prometheus-server
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:10.3.3 # Latest stable as of 2026-08-06
    container_name: grafana-dashboard
    volumes:
      - grafana_data:/var/lib/grafana # Persistent storage for Grafana data (dashboards, config)
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin # IMPORTANT: Change this in production!
    ports:
      - "3000:3000" # Expose Grafana UI to host port 3000

volumes:
  prometheus_data:
  grafana_data: # Define a named volume for Grafana data persistence

We’ve added the grafana service, mounted a persistent volume for its data, and set up initial admin credentials via environment variables.

2. Stop existing containers and run all services:

docker compose down
docker compose up -d --build

Verification:

  1. Access Grafana UI: Open your browser to http://localhost:3000.
  2. Log in: Use admin for both username and password (as configured). You’ll likely be prompted to change the password immediately.
  3. Add Prometheus as a data source:
    • From the left-hand menu, click “Connections” (or the cog icon for “Configuration”) -> “Data sources”.
    • Click “Add data source” -> Select “Prometheus”.
    • Set “Name”: Prometheus (or any descriptive name).
    • Set “URL”: http://prometheus:9090. Remember, prometheus is the service name within our Docker Compose network.
    • Scroll down and click “Save & test”. You should see a green “Data source is working” message.
  4. Create a dashboard:
    • From the left-hand menu, click “Dashboards” -> “New dashboard” -> “Add a new panel”.
    • In the “Query” tab, ensure your “Prometheus” data source is selected.
    • In the “PromQL” field, enter a metric like http_requests_total.
    • You should see a graph of your application’s total requests.
    • Experiment with other metrics (active_requests, service_health) and different visualization types (Graph, Gauge, Singlestat).

Operational Readiness Checklist: Beyond Observability

While logging and metrics are foundational, a truly production-ready application requires attention to several other operational concerns.

1. Health Checks

Health checks allow Docker and orchestrators (like Kubernetes or Docker Swarm) to determine if a container is truly healthy and ready to serve requests, not just if its main process is running. This is critical for reliable deployments and self-healing systems.

In my-log-app/Dockerfile (Add a HEALTHCHECK instruction): While Docker Compose healthcheck block overrides this, it’s good practice to include it in the Dockerfile for standalone deployments.

# docker-observability-project/my-log-app/Dockerfile (excerpt)
...
EXPOSE 8000

# Define a health check that pings the metrics endpoint
HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=5s \
  CMD curl --fail http://localhost:8000/metrics || exit 1

This check attempts to curl the /metrics endpoint every 30 seconds. If it fails (returns a non-zero exit code) three consecutive times, Docker considers the container unhealthy. start-period gives the application time to initialize before the first check.

In docker-compose.yaml (Use the healthcheck block): This provides more control and overrides the Dockerfile’s HEALTHCHECK.

# docker-observability-project/docker-compose.yaml (excerpt for log-app)
services:
  log-app:
    # ... other configurations ...
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8000/metrics"]
      interval: 30s # Check every 30 seconds
      timeout: 10s  # Fail if check takes longer than 10 seconds
      retries: 3    # Mark unhealthy after 3 consecutive failures
      start_period: 5s # Give the app 5 seconds to warm up before starting checks

Verification: After running docker compose up -d, execute docker ps. You’ll see (healthy) or (unhealthy) in the STATUS column for your log-app-instance.

2. Resource Limits

Setting CPU and memory limits is crucial to prevent “noisy neighbor” issues, where one misbehaving container consumes excessive resources and impacts other services on the same host. It also helps in capacity planning.

In docker-compose.yaml (Add to log-app service):

# docker-observability-project/docker-compose.yaml (excerpt for log-app)
services:
  log-app:
    # ... other configurations ...
    deploy: # Used for swarm mode, but also respected by Docker Compose for resource limits
      resources:
        limits:
          cpus: '0.5' # Max 0.5 CPU core (e.g., 50% of one core)
          memory: 128M # Max 128MB RAM
        reservations:
          cpus: '0.25' # Reserve 0.25 CPU core (guaranteed minimum)
          memory: 64M # Reserve 64MB RAM (guaranteed minimum)

limits are hard caps; if a container tries to exceed them, it might be throttled (CPU) or killed (memory - OOMKilled). reservations are guaranteed minimums. ⚠️ What can go wrong: Under-resourcing can lead to performance degradation and crashes. Over-resourcing wastes expensive infrastructure. Start with reasonable estimates and refine based on observed metrics.

3. Graceful Shutdowns

When a container is stopped or restarted, Docker sends a SIGTERM signal. Applications should be designed to catch this signal, gracefully complete any ongoing tasks, release resources (e.g., close database connections, flush logs), and then exit. If the application doesn’t exit within a grace period, Docker sends a SIGKILL (forceful termination).

In docker-compose.yaml (Add to log-app service):

# docker-observability-project/docker-compose.yaml (excerpt for log-app)
services:
  log-app:
    # ... other configurations ...
    stop_grace_period: 30s # Give the container up to 30 seconds to shut down gracefully

Your application code (e.g., app.py) would need to implement signal handling, but stop_grace_period ensures Docker waits before forceful termination.

4. Backup Strategies

For any service with persistent data (like Prometheus and Grafana, which use named volumes), a robust backup strategy is non-negotiable. While Docker volumes provide persistence across container restarts, they don’t inherently protect against host failure or accidental deletion.

⚡ Real-world insight: Backup strategies typically involve:

  • Volume backups: Using host-level tools to snapshot or copy the volume data.
  • Cloud-specific solutions: If deployed to a cloud provider, leveraging their managed backup services (e.g., AWS EBS snapshots, Azure Disk Backup).
  • Configuration backups: Keeping prometheus.yml, grafana.ini, and dashboard JSON definitions in version control.

5. Alerting

Monitoring tells you what’s happening; alerting tells you when something needs attention. Prometheus integrates with Alertmanager, a separate component that handles deduplicating, grouping, and routing alerts to various notification channels (email, Slack, PagerDuty, etc.).

This is a more advanced topic, but in production, you would configure rules in Prometheus to detect anomalous behavior (e.g., service_health == 0 for more than 5 minutes, or http_requests_total dropping unexpectedly) and have Alertmanager notify the relevant team.

Common Issues & Solutions for Observability

  1. Log Volume Too High / PII in Logs:

    • Issue: Applications logging excessively (e.g., DEBUG level in production) or inadvertently including sensitive information (Personally Identifiable Information - PII, secrets). This can overwhelm logging systems, incur high costs, and create security/compliance risks.
    • Solution:
      • Implement proper log levels (DEBUG, INFO, WARN, ERROR) and configure your application to use appropriate levels for production (e.g., INFO or WARN).
      • Sanitize or redact sensitive data before logging. Use dedicated logging libraries that support structured logging and redaction.
      • ⚡ Real-world insight: For extremely high-volume events, consider log sampling to reduce ingestion costs while retaining statistical data.

  2. Prometheus Scrape Failures (log-app target down):

    • Issue: Prometheus cannot reach the /metrics endpoint of a target service, showing it as “DOWN” in the UI.
    • Solution:
      • Verify the target service (log-app-instance) is running: docker ps.
      • Ensure its metrics port (8000) is correctly exposed and listening within the Docker network.
      • Check the targets configuration in prometheus/prometheus.yml for correct service names and ports. Remember to use the Docker Compose service name (log-app) for internal communication, not localhost.
      • Inspect the target container’s logs (docker logs log-app-instance) for errors related to network binding or metrics server startup.
      • Ensure no firewall rules on the host are blocking internal Docker network communication (uncommon in default Docker setups but possible with custom network configurations).

  3. Resource Exhaustion (OOMKilled):

    • Issue: A container attempts to use more memory than its allocated limits, leading to it being forcefully terminated by the operating system (Out Of Memory Killed - OOMKilled). This results in service instability and restarts.
    • Solution:
      • Monitor container_memory_usage_bytes and container_cpu_usage_seconds_total metrics in Prometheus/Grafana to identify resource hogs and understand typical usage patterns.
      • Adjust deploy.resources.limits in docker-compose.yaml based on observed peak usage, leaving a small buffer.
      • Optimize application code for memory efficiency or CPU usage, if resource consumption is genuinely excessive.
      • ⚠️ What can go wrong: Repeated OOMKills indicate a fundamental issue with resource allocation or application design. It’s a critical alert.

Summary & Next Steps

In this chapter, you’ve moved beyond simply running containers to building a robust, observable, and operationally ready containerized application. You’ve gained hands-on experience with:

  • Structured Logging: Emitting machine-readable JSON logs from your application.
  • Log Aggregation: Using Docker’s fluentd driver and a fluentd service to centralize logs.
  • Metrics Instrumentation: Adding Prometheus-compatible metrics to your application.
  • Metrics Collection: Setting up Prometheus to scrape and store these metrics.
  • Visualization: Configuring Grafana to connect to Prometheus and create insightful dashboards.
  • Operational Practices: Implementing health checks, resource limits, and understanding graceful shutdowns and backup strategies.

This foundational observability stack is indispensable for understanding the behavior of your applications in the wild, enabling proactive problem-solving, and ensuring system reliability. While we’ve used basic configurations, real-world systems often involve more sophisticated setups, including dedicated log storage (e.g., Elasticsearch, Loki), advanced alerting (Prometheus Alertmanager), and distributed tracing with tools like OpenTelemetry.

Next Steps for Your Docker Journey:

  • Explore Cloud Deployment: Learn how to deploy your Docker Compose applications to managed container services in the cloud (e.g., AWS ECS, Azure Container Instances, Google Cloud Run). These platforms often provide integrated logging and monitoring solutions.
  • Advanced Observability: Dive deeper into distributed tracing with OpenTelemetry to track requests across multiple services. Investigate advanced log analysis with tools like Loki for cost-effective log management.
  • CI/CD Integration: Automate the build, test, and deployment of your containerized applications using Continuous Integration/Continuous Delivery (CI/CD) pipelines. This is the next logical step to truly operationalize your projects.

By continuously refining your observability and operational practices, you’ll build more resilient, maintainable, and ultimately, more successful applications that reliably serve your users.

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.