Skip to main content

Container Resource Management

Setting resource limits (covered in Resource Limits) is only half the equation. You also need to monitor actual usage, adjust limits on running containers, and track disk consumption. This lesson covers the operational side of container resource management.

Live Monitoring with docker stats​

docker stats is the real-time dashboard for container resource usage:

docker stats

Output:

CONTAINER ID   NAME     CPU %    MEM USAGE / LIMIT    MEM %    NET I/O          BLOCK I/O        PIDS
a1b2c3d4e5f6   api      2.50%    180MiB / 512MiB      35.16%   1.2MB / 800kB    5MB / 0B         25
f6e5d4c3b2a1   db       0.80%    350MiB / 1GiB        34.18%   500kB / 1.5MB    50MB / 20MB      18
b2c3d4e5f6a1   cache    0.10%    50MiB / 256MiB       19.53%   200kB / 100kB    0B / 0B          4

Understanding the Columns​

ColumnWhat It ShowsWatch For
CPU %CPU usage relative to host capacitySustained 100%+ = throttling likely
MEM USAGE / LIMITCurrent memory / configured limitClose to limit = OOM risk
MEM %Memory as percentage of limitAbove 80% consistently = danger zone
NET I/ONetwork bytes received / sentUnexpected spikes = possible issue
BLOCK I/ODisk reads / writesHigh writes = possible logging issue
PIDSProcess countRapid increase = fork bomb risk

Filtering and Formatting​

# Monitor specific containers
docker stats api db cache

# No-stream: single snapshot (useful in scripts)
docker stats --no-stream

# Custom format
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

# JSON output (for piping to monitoring tools)
docker stats --no-stream --format '{{json .}}'
tip

Use docker stats --no-stream in monitoring scripts and cron jobs. It prints one snapshot and exits, making it easy to parse and log.

Disk Usage with docker system df​

docker system df shows how much disk space Docker is using across images, containers, and volumes:

docker system df

Output:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          15        5         3.2GB     1.8GB (56%)
Containers      8         5         200MB     120MB (60%)
Local Volumes   12        6         5.1GB     2.3GB (45%)
Build Cache     -         -         800MB     800MB

Detailed View​

docker system df -v

This shows individual items with their sizes -- useful for finding the biggest offenders.

Reclaiming Space​

# Remove stopped containers, unused networks, dangling images, and build cache
docker system prune

# Also remove unused volumes (careful -- data loss!)
docker system prune --volumes

# Remove only dangling images
docker image prune

# Remove only stopped containers
docker container prune

# Remove only unused volumes
docker volume prune
danger

docker system prune --volumes deletes unused volumes, which may contain database data. Always verify which volumes are in use before running this command.

Updating Limits on Running Containers​

docker update changes resource limits without restarting a container:

# Increase memory limit
docker update --memory=1g --memory-swap=1g api

# Change CPU limit
docker update --cpus=2.0 api

# Change restart policy
docker update --restart unless-stopped api

# Update multiple containers at once
docker update --memory=512m api worker scheduler

What Can Be Updated Live​

SettingFlagRequires Restart?
Memory limit--memory❌ No
Memory + Swap--memory-swap❌ No
CPU limit--cpus❌ No
CPU shares--cpu-shares❌ No
PID limit--pids-limit❌ No
Restart policy--restart❌ No
Block I/O weight--blkio-weight❌ No
tip

docker update is invaluable during incidents. If a container is being OOM-killed, you can increase its memory limit immediately without downtime.

Resource Monitoring Patterns​

Pattern 1: Quick Health Check Script​

#!/usr/bin/env bash
# health-check.sh -- quick resource overview

echo "=== Container Resources ==="
docker stats --no-stream --format \
  "table {{.Name}}\t{{.CPUPerc}}\t{{.MemPerc}}\t{{.PIDs}}"

echo ""
echo "=== Disk Usage ==="
docker system df

echo ""
echo "=== Containers with high memory (>80%) ==="
docker stats --no-stream --format '{{.Name}} {{.MemPerc}}' | \
  awk '{gsub(/%/,"",$2); if ($2 > 80) print $1, $2"%"}'

Pattern 2: Log Resource Usage Over Time​

#!/usr/bin/env bash
# log-stats.sh -- append stats to a log file every minute

LOG_FILE="/var/log/docker-stats.log"

while true; do
  echo " $(date -Iseconds) " >> "$LOG_FILE"
  docker stats --no-stream --format \
    '{{.Name}},{{.CPUPerc}},{{.MemUsage}},{{.MemPerc}},{{.PIDs}}' \
    >> "$LOG_FILE"
  sleep 60
done

Pattern 3: Alert on High Memory​

#!/usr/bin/env bash
# alert-memory.sh -- warn if any container exceeds 85% memory

THRESHOLD=85

docker stats --no-stream --format '{{.Name}} {{.MemPerc}}' | while read name pct; do
  value=$(echo "$pct" | tr -d '%')
  if (( $(echo "$value > $THRESHOLD" | bc -l) )); then
    echo "WARNING: $name memory at $pct (threshold: ${THRESHOLD}%)"
    # Add notification (email, Slack webhook, etc.)
  fi
done

Container Events​

docker events streams real-time events from the Docker daemon -- useful for monitoring container lifecycle:

# Watch all events
docker events

# Filter by container
docker events --filter container=api

# Filter by event type
docker events --filter event=oom --filter event=die --filter event=restart

# Time range (last hour)
docker events --since 1h

Key events to monitor:

EventMeaningAction
oomContainer hit memory limitIncrease limit or fix memory leak
dieContainer exitedCheck exit code and logs
restartContainer restartedMonitor restart count for storms
killContainer was killedCheck who/what sent the signal

Putting It All Together​

flowchart TD
    A["Set initial limits\n(docker run --memory --cpus)"] --> B["Monitor with\ndocker stats"]
    B --> C{Issues?}
    C -->|"OOM kills"| D["Increase memory\ndocker update --memory"]
    C -->|"CPU throttling"| E["Increase CPU\ndocker update --cpus"]
    C -->|"Disk full"| F["Clean up\ndocker system prune"]
    C -->|"No issues"| G["Document limits\nand continue monitoring"]
    D --> B
    E --> B
    F --> B

    style A fill:#e3f2fd,stroke:#1565c0
    style G fill:#e8f5e9,stroke:#2e7d32
    style D fill:#fff3e0,stroke:#ef6c00
    style E fill:#fff3e0,stroke:#ef6c00
    style F fill:#ffebee,stroke:#c62828

Key Takeaways​

  • docker stats is your real-time resource dashboard -- use --no-stream for scripting.
  • docker system df reveals disk usage across images, containers, and volumes.
  • docker update changes resource limits on running containers without restart -- invaluable during incidents.
  • docker events streams lifecycle events for monitoring OOM kills, crashes, and restarts.
  • Build monitoring scripts to catch high memory usage and disk pressure before they cause outages.

What's Next​