dojo.md 0.2.0 → 0.2.2

package/courses/docker-container-debugging/scenarios/level-3/advanced-debugging-shift.yaml

@@ -0,0 +1,77 @@
+meta:
+  id: advanced-debugging-shift
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Combined advanced debugging shift — diagnose a production Docker environment with daemon, security, networking, storage, and deployment issues simultaneously"
+  tags: [Docker, troubleshooting, combined, shift-simulation, advanced]
+
+state: {}
+
+trigger: |
+  You're called in for an urgent production issue. The Docker host
+  runs 15 services and "everything is degraded":
+
+  $ docker system df
+  TYPE            TOTAL   ACTIVE  SIZE      RECLAIMABLE
+  Images          200+    15      85GB      62GB (72%)
+  Containers      45      15      25GB      18GB
+  Local Volumes   80      20      120GB     85GB (70%)
+  Build Cache     -       -       45GB      45GB
+
+  275GB consumed. Disk is at 94%.
+
+  $ docker stats --no-stream
+  CONTAINER    CPU%    MEM USAGE/LIMIT    MEM%
+  api-1        180%    3.8GiB/4GiB        95%
+  api-2        175%    3.7GiB/4GiB        92%
+  worker-1     85%     1.2GiB/2GiB        60%
+  postgres     25%     2.1GiB/4GiB        52%
+  redis        5%      512MiB/1GiB        50%
+  nginx        2%      128MiB/256MiB      50%
+
+  Investigation reveals 6 interconnected issues:
+
+  1. API containers at 95% memory — memory leak in a recent deployment.
+     The previous image was tagged :latest and overwritten. No rollback
+     available. Must debug and fix forward.
+
+  2. Disk at 94% — 85GB of orphaned volumes from old database migrations
+     that were never cleaned up. 45GB of build cache. Containers can't
+     write logs.
+
+  3. Worker containers failing intermittently — overlay network between
+     worker and Redis has MTU issues. Large payloads fail, small ones
+     succeed. tcpdump shows fragmented packets.
+
+  4. Security audit alert — 3 containers running as root with
+     --privileged flag. One container has a writable root filesystem
+     with a suspicious file: /tmp/.backdoor.sh
+
+  5. Nginx proxy returns 502 for 10% of requests — during rolling
+     updates, traffic is routed to containers that haven't finished
+     starting. No health check-based routing.
+
+  6. Docker daemon itself is consuming 8GB of memory due to a known
+     bug with container event logging. Needs daemon restart but
+     live-restore isn't configured.
+
+  Task: Walk through the complete triage and resolution. Write: the
+  priority order (security incident first!), immediate stabilization
+  steps, disk cleanup (safe vs dangerous commands), network debugging,
+  deployment fix, daemon maintenance, and the post-incident action
+  items for long-term fixes.
+
+assertions:
+  - type: llm_judge
+    criteria: "Priority triage is correct — (1) SECURITY FIRST: investigate the suspicious /tmp/.backdoor.sh file, check for container escape indicators, audit --privileged containers, preserve evidence before cleanup. (2) STABILITY: address 94% disk to prevent cascading failures — safe prune of build cache and dangling images first. (3) MEMORY: restart leaking API containers as immediate mitigation. (4) Fix networking MTU issue. (5) Fix deployment process for zero-downtime. (6) Plan daemon restart"
+    weight: 0.35
+    description: "Priority triage"
+  - type: llm_judge
+    criteria: "Immediate fixes and long-term actions are covered — immediate: quarantine compromised containers (don't delete — preserve for forensics), docker builder prune and docker image prune for safe disk recovery, restart API containers with memory limits, fix MTU on overlay network. Long-term: remove --privileged, implement non-root containers, tag images with versions (not :latest), configure live-restore in daemon.json, add health check-based routing, set up automated prune cron, implement monitoring and alerting"
+    weight: 0.35
+    description: "Fixes and actions"
+  - type: llm_judge
+    criteria: "Post-incident improvements are systematic — deploy monitoring: Prometheus + cAdvisor for container metrics, alert on memory/disk thresholds. Security: scan images in CI, enforce non-root, drop capabilities, read-only filesystems. Operations: automated cleanup policies, image retention limits, volume backup before prune. Deployment: rolling updates with health checks, image versioning, rollback capability. Documentation: runbook for disk emergencies, security incident response, update procedures"
+    weight: 0.30
+    description: "Post-incident improvements"

package/courses/docker-container-debugging/scenarios/level-3/buildkit-optimization.yaml

@@ -0,0 +1,67 @@
+meta:
+  id: buildkit-optimization
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug BuildKit build performance — diagnose slow builds, cache misses, and optimize multi-stage builds with BuildKit features"
+  tags: [Docker, BuildKit, build-cache, multi-stage, optimization, advanced]
+
+state: {}
+
+trigger: |
+  Your Docker image builds take 15+ minutes in CI. You're using BuildKit
+  but not seeing the expected speedup:
+
+  $ DOCKER_BUILDKIT=1 docker build -t myapp .
+  [+] Building 923.4s (18/18)
+   => [internal] load build context                           45.2s
+   => [stage-1  1/5] FROM node:20                             0.0s
+   => [stage-1  2/5] COPY package*.json ./                    0.3s
+   => [stage-1  3/5] RUN npm ci                             412.8s
+   => [stage-1  4/5] COPY . .                                12.1s
+   => [stage-1  5/5] RUN npm run build                      180.5s
+   => [stage-2  1/3] FROM node:20-slim                        0.0s
+   => [stage-2  2/3] COPY --from=stage-1 /app/dist ./dist     2.1s
+   => [stage-2  3/3] COPY --from=stage-1 /app/node_modules    8.4s
+
+  Issues found:
+
+  1. Build context is 2.3GB — .dockerignore missing node_modules, .git,
+     and test fixtures. The 45-second context load is pure waste.
+
+  2. npm ci runs every build because COPY . . invalidates the cache.
+     Layer ordering: dependencies should be installed before copying
+     source code.
+
+  3. node_modules is 800MB because it includes devDependencies.
+     The production image only needs production deps.
+
+  4. No BuildKit cache mounts — npm ci downloads packages fresh each
+     time even when package.json hasn't changed:
+     RUN --mount=type=cache,target=/root/.npm npm ci
+
+  5. The COPY --from copies all of node_modules (800MB) into the final
+     image. Should run npm ci --production in the final stage or use
+     a separate dependency stage.
+
+  After optimization, build drops from 15 minutes to 2 minutes.
+
+  Task: Explain BuildKit optimization. Write: build context management
+  (.dockerignore), layer cache optimization (order matters), BuildKit
+  cache mounts (--mount=type=cache), multi-stage build patterns for
+  minimal images, parallel stage execution in BuildKit, and common
+  Dockerfile anti-patterns that kill build performance.
+
+assertions:
+  - type: llm_judge
+    criteria: "Layer caching strategy is explained — Docker caches each layer; if a layer changes, all subsequent layers are invalidated. Key pattern: copy dependency files first (package.json, requirements.txt), install dependencies, THEN copy source code. This way source code changes don't invalidate the dependency install layer. COPY . . before RUN npm install is an anti-pattern because any source change triggers a full reinstall"
+    weight: 0.35
+    description: "Layer caching"
+  - type: llm_judge
+    criteria: "BuildKit features are covered — cache mounts: RUN --mount=type=cache,target=/root/.npm npm ci (persists npm cache between builds). Secret mounts: RUN --mount=type=secret,id=npmrc (inject secrets without baking into layers). Parallel stage execution: BuildKit builds independent stages concurrently. Build context: .dockerignore should exclude .git, node_modules, test files, docs. Large context = slow builds even if files aren't used"
+    weight: 0.35
+    description: "BuildKit features"
+  - type: llm_judge
+    criteria: "Multi-stage optimization is practical — separate stages for: (1) dependency installation (with dev deps), (2) build/compile, (3) production image (minimal base, production deps only). Use specific base image tags (node:20.11-slim not node:latest). Final image should only contain runtime artifacts. Measure with docker images and docker history. Target: production images should be 10-50x smaller than build images"
+    weight: 0.30
+    description: "Multi-stage optimization"

package/courses/docker-container-debugging/scenarios/level-3/container-filesystem-debugging.yaml

@@ -0,0 +1,70 @@
+meta:
+  id: container-filesystem-debugging
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug container filesystem issues — diagnose storage driver problems, layer corruption, copy-on-write behavior, and container diff analysis"
+  tags: [Docker, filesystem, storage-driver, overlay2, copy-on-write, layers, advanced]
+
+state: {}
+
+trigger: |
+  Your application container is behaving strangely — configuration
+  files appear to have been modified, but nobody changed the image:
+
+  $ docker diff webapp-1
+  C /etc/nginx/nginx.conf
+  A /tmp/session_abc123
+  C /var/log/nginx/access.log
+  A /var/log/nginx/error.log
+  C /app/config/settings.json
+
+  The C (changed) entries show files modified in the container's
+  writable layer. Someone exec'd into the container and made changes
+  directly — these changes aren't in the image and will be lost on
+  container restart.
+
+  Deeper investigation reveals storage driver issues:
+
+  $ docker info | grep "Storage Driver"
+  Storage Driver: overlay2
+
+  $ ls /var/lib/docker/overlay2/
+  (hundreds of directories — one per layer)
+
+  $ du -sh /var/lib/docker/overlay2/
+  89GB
+
+  Problem 1: A container's writable layer grew to 15GB because the
+  application writes temporary files that are never cleaned up. Even
+  though the files are "deleted" inside the container, the overlay2
+  whiteout files still consume space.
+
+  Problem 2: docker commit was used to save a modified container as
+  a new image. This created a massive single layer with all the
+  runtime changes, making the image 3x larger than necessary.
+
+  Problem 3: After an unclean Docker shutdown, some overlay2 layers
+  are corrupted:
+  $ docker inspect broken-container
+  Error: layer not found
+
+  Task: Explain container filesystem debugging. Write: how overlay2
+  works (lower/upper/merged/work directories), copy-on-write behavior,
+  docker diff for change detection, whiteout files, why docker commit
+  is an anti-pattern, layer corruption recovery, and best practices
+  for managing container writable layers.
+
+assertions:
+  - type: llm_judge
+    criteria: "Overlay2 mechanics are explained — overlay2 uses lower directories (read-only image layers), upper directory (container's writable layer), merged directory (unified view), work directory (atomic operations). Copy-on-write: first write to a file copies it from lower to upper, then modifies. Deletes create whiteout files in upper layer (file still exists in lower). docker diff shows C (changed), A (added), D (deleted) in the writable layer"
+    weight: 0.35
+    description: "Overlay2 mechanics"
+  - type: llm_judge
+    criteria: "Layer management issues are covered — container writable layer grows with every write (even temporary files). Deleting files creates whiteout entries, not actual space recovery. docker commit captures the entire writable layer as a new image layer — includes temp files, logs, secrets, and is not reproducible. Always use Dockerfile for image creation, not docker commit. Large writable layers indicate the app should use volumes for data/logs/temp files"
+    weight: 0.35
+    description: "Layer management"
+  - type: llm_judge
+    criteria: "Recovery and best practices are practical — layer corruption: docker system prune can remove orphaned layers. For specific container: docker rm (removes writable layer). For images: docker rmi and re-pull. /var/lib/docker/overlay2 should not be manually modified. Best practices: use volumes for persistent data, tmpfs for temporary data, --read-only filesystem where possible, monitor writable layer size with docker system df -v, set storage-opts for layer size limits"
+    weight: 0.30
+    description: "Recovery and practices"

package/courses/docker-container-debugging/scenarios/level-3/container-security-hardening.yaml

@@ -0,0 +1,74 @@
+meta:
+  id: container-security-hardening
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug container security issues — diagnose privilege escalation, capability problems, seccomp profiles, and rootless container configuration"
+  tags: [Docker, security, rootless, seccomp, capabilities, hardening, advanced]
+
+state: {}
+
+trigger: |
+  A security audit of your Docker deployment flags several issues:
+
+  Finding 1 — Containers running as root:
+  $ docker exec webapp whoami
+  root
+  $ docker exec webapp id
+  uid=0(root) gid=0(root) groups=0(root)
+
+  If the application is compromised, the attacker has root inside the
+  container. With certain misconfigurations, this can lead to host
+  escape.
+
+  Finding 2 — Excessive capabilities:
+  $ docker exec webapp capsh --print
+  Current: cap_chown,cap_dac_override,cap_fsetid,cap_fowner,
+  cap_mknod,cap_net_raw,cap_setgid,cap_setuid,cap_setfcap,
+  cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_kill,cap_audit_write
+
+  Most apps don't need cap_net_raw, cap_sys_chroot, or cap_mknod.
+
+  Finding 3 — --privileged flag used "because it works":
+  docker run --privileged myapp
+  This gives the container ALL capabilities, access to all devices,
+  and disables seccomp/AppArmor. Used as a debugging shortcut but
+  left in production.
+
+  Finding 4 — Writable root filesystem:
+  $ docker exec webapp touch /etc/malicious-config
+  (succeeds — attacker can modify container filesystem)
+
+  Hardened configuration:
+  docker run \
+    --read-only \
+    --tmpfs /tmp \
+    --cap-drop ALL \
+    --cap-add NET_BIND_SERVICE \
+    --security-opt no-new-privileges:true \
+    --security-opt seccomp=custom-profile.json \
+    -u 1000:1000 \
+    myapp
+
+  But this breaks the app — it tries to write to /var/cache and
+  needs to bind to port 80 (requires NET_BIND_SERVICE or port > 1024).
+
+  Task: Explain container security hardening. Write: running as
+  non-root (USER directive, --user flag), Linux capabilities (drop
+  all, add specific), read-only filesystem (with tmpfs exceptions),
+  seccomp profiles, no-new-privileges, why --privileged is dangerous,
+  and the debugging process when hardening breaks applications.
+
+assertions:
+  - type: llm_judge
+    criteria: "Non-root and capabilities are explained — Dockerfile USER directive or --user flag. Running as root inside container is dangerous even with namespace isolation. Drop ALL capabilities, add only what's needed (NET_BIND_SERVICE for port < 1024, CHOWN for file ownership). --privileged gives ALL capabilities plus device access plus disables security modules — never use in production. To find required capabilities: run with all dropped, check error messages, add back one at a time"
+    weight: 0.35
+    description: "Non-root and capabilities"
+  - type: llm_judge
+    criteria: "Filesystem and seccomp are covered — --read-only makes root filesystem immutable (prevents attackers from modifying binaries/config). Use --tmpfs for directories that need writes (/tmp, /var/cache, /var/run). no-new-privileges prevents privilege escalation via setuid binaries. Seccomp profiles restrict syscalls — Docker's default profile blocks ~44 dangerous syscalls. Custom profiles can restrict further. AppArmor/SELinux add mandatory access control"
+    weight: 0.35
+    description: "Filesystem and seccomp"
+  - type: llm_judge
+    criteria: "Debugging hardened containers is practical — common breakages when hardening: app writes to filesystem (add tmpfs), needs specific capability (add one at a time), needs specific syscall blocked by seccomp (check audit log, adjust profile). Use strace to identify required syscalls. Use docker diff to see filesystem changes. Test hardening in staging before production. Security scanning (Trivy, Docker Scout) complements runtime hardening"
+    weight: 0.30
+    description: "Debugging hardened containers"

package/courses/docker-container-debugging/scenarios/level-3/disk-space-management.yaml

@@ -0,0 +1,74 @@
+meta:
+  id: disk-space-management
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug Docker disk space exhaustion — diagnose storage consumption from images, containers, volumes, and build cache, then implement cleanup strategies"
+  tags: [Docker, disk-space, prune, storage, build-cache, cleanup, advanced]
+
+state: {}
+
+trigger: |
+  Production Docker host is critically low on disk space. Containers
+  are failing to start:
+
+  $ docker run myapp
+  Error: no space left on device
+
+  $ df -h /var/lib/docker
+  Filesystem      Size  Used  Avail  Use%
+  /dev/sda1       200G  195G  5G     98%
+
+  $ docker system df
+  TYPE            TOTAL   ACTIVE  SIZE      RECLAIMABLE
+  Images          127     8       45.2GB    38.1GB (84%)
+  Containers      43      8       12.3GB    8.7GB (70%)
+  Local Volumes   56      12      78.5GB    52.3GB (66%)
+  Build Cache     -       -       34.8GB    34.8GB
+
+  170GB used by Docker! Breakdown:
+
+  1. Images (45GB, 84% reclaimable): 127 images, only 8 in use.
+     Old CI builds never cleaned up. Dangling images from failed
+     builds. Multiple versions of the same image.
+     $ docker images --filter "dangling=true" -q | wc -l
+     45
+
+  2. Containers (12GB): Stopped containers with large log files and
+     writable layer changes. docker rm only removes stopped containers.
+
+  3. Volumes (78GB, 66% reclaimable): Orphaned volumes from removed
+     containers. Database test volumes never cleaned up. Named volumes
+     for services that no longer exist.
+     $ docker volume ls --filter "dangling=true" -q | wc -l
+     44
+
+  4. Build cache (34GB): BuildKit cache from months of builds.
+
+  Cleanup strategy (careful — don't delete production data!):
+  $ docker system prune -a --volumes
+  WARNING! This will remove all stopped containers, all networks not
+  used, all images without containers, and all volumes not used.
+  Are you sure? [y/N]
+
+  But this is too aggressive for production! Need selective cleanup.
+
+  Task: Explain Docker disk space management. Write: understanding
+  docker system df output, selective cleanup (prune with filters),
+  dangerous vs safe prune commands, volume backup before cleanup,
+  automated cleanup strategies (cron, CI cleanup), monitoring disk
+  usage, and prevention (log rotation, image retention policies).
+
+assertions:
+  - type: llm_judge
+    criteria: "Disk usage diagnosis is explained — docker system df shows breakdown by type (images, containers, volumes, build cache). RECLAIMABLE shows what can be safely removed. docker system df -v shows detailed per-item usage. Disk consumers: dangling images (untagged), stopped containers (writable layer + logs), orphaned volumes (no container reference), build cache. Check with du -sh /var/lib/docker/* for filesystem-level breakdown"
+    weight: 0.35
+    description: "Disk usage diagnosis"
+  - type: llm_judge
+    criteria: "Selective cleanup is covered — safe: docker image prune (remove dangling only), docker container prune (remove stopped), docker builder prune (clear build cache). Filtered: docker image prune -a --filter 'until=720h' (remove images older than 30 days). DANGEROUS: docker system prune -a --volumes (removes everything unused including volumes with data). Always: backup named volumes before pruning, never prune volumes on database hosts without verification. Use docker volume inspect to check last mount time"
+    weight: 0.35
+    description: "Selective cleanup"
+  - type: llm_judge
+    criteria: "Prevention and automation are practical — automated cleanup: cron job with docker system prune --filter 'until=168h' (weekly, items older than 7 days). CI/CD: clean up after pipeline runs, limit image retention. Log rotation in daemon.json prevents log bloat. Image retention policy: keep only N latest tags. Monitor: alert when /var/lib/docker exceeds 80%. Use docker system events to track resource creation. Separate volume for /var/lib/docker to isolate Docker disk usage from system"
+    weight: 0.30
+    description: "Prevention and automation"

package/courses/docker-container-debugging/scenarios/level-3/docker-api-automation.yaml

@@ -0,0 +1,72 @@
+meta:
+  id: docker-api-automation
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug Docker API and automation issues — diagnose Docker SDK errors, API versioning problems, and container orchestration script failures"
+  tags: [Docker, API, SDK, automation, scripting, REST, advanced]
+
+state: {}
+
+trigger: |
+  Your container management script suddenly breaks after a Docker
+  Engine upgrade:
+
+  import docker
+  client = docker.from_env()
+  containers = client.containers.list(filters={"health": "unhealthy"})
+  for c in containers:
+      c.restart()
+      print(f"Restarted {c.name}")
+
+  Error:
+  docker.errors.APIError: 400 Client Error: Bad Request
+  ("invalid filter 'health'")
+
+  Investigation reveals several API issues:
+
+  1. API version mismatch — the script was written for Docker API 1.44
+     but the server now runs 1.45 which changed the filter syntax:
+     $ docker version --format '{{.Server.APIVersion}}'
+     1.45
+
+     Fix: pin API version in client:
+     client = docker.DockerClient(base_url='unix:///var/run/docker.sock',
+                                   version='1.44')
+
+  2. The script uses client.containers.list() to find containers,
+     but hits a race condition — between listing and restarting,
+     containers may have been removed by another process:
+     docker.errors.NotFound: 404 Client Error: Container not found
+
+  3. Direct API calls via curl for debugging:
+     $ curl --unix-socket /var/run/docker.sock \
+       http://localhost/v1.44/containers/json?filters={"health":["unhealthy"]}
+     $ curl --unix-socket /var/run/docker.sock \
+       http://localhost/v1.44/events?since=1700000000
+
+  4. The Docker socket permissions prevent the script from running
+     without root:
+     $ ls -la /var/run/docker.sock
+     srw-rw---- 1 root docker 0 Dec 01 10:00 /var/run/docker.sock
+     User not in docker group.
+
+  Task: Explain Docker API and automation debugging. Write: Docker
+  Engine API basics (REST over Unix socket), API versioning and
+  negotiation, Docker SDKs (Python, Go), common SDK errors and fixes,
+  race conditions in container management, socket permissions and
+  security, and using curl for raw API debugging.
+
+assertions:
+  - type: llm_judge
+    criteria: "Docker API fundamentals are explained — Docker Engine exposes a REST API over Unix socket (/var/run/docker.sock) or TCP. All Docker CLI commands are API calls under the hood. API versioning: client and server negotiate version. Pin version to avoid breaking changes. curl --unix-socket for raw API debugging. Key endpoints: /containers/json (list), /containers/{id}/start, /containers/{id}/logs, /events (stream), /images/json"
+    weight: 0.35
+    description: "API fundamentals"
+  - type: llm_judge
+    criteria: "SDK debugging is covered — Python SDK (docker-py): docker.from_env() reads DOCKER_HOST and DOCKER_TLS_VERIFY. Common errors: APIError (bad request/version mismatch), NotFound (container removed), ConnectionError (daemon not running). Always handle NotFound for race conditions. Use version parameter to pin API version. Go SDK (moby/moby/client): similar patterns. Check API compatibility: docker version shows client and server API versions"
+    weight: 0.35
+    description: "SDK debugging"
+  - type: llm_judge
+    criteria: "Security and best practices are practical — socket access = root-equivalent access (can mount host filesystem via API). Never expose Docker socket to untrusted containers. Socket permissions: add user to docker group (convenience) or use rootless Docker (security). For automation: handle race conditions (container may disappear), use docker events for reactive management instead of polling, implement retry logic with backoff, log all management actions for audit trail"
+    weight: 0.30
+    description: "Security and practices"

package/courses/docker-container-debugging/scenarios/level-3/docker-daemon-issues.yaml

@@ -0,0 +1,73 @@
+meta:
+  id: docker-daemon-issues
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug Docker daemon issues — diagnose daemon startup failures, storage driver problems, and daemon configuration errors"
+  tags: [Docker, daemon, storage-driver, daemon.json, configuration, advanced]
+
+state: {}
+
+trigger: |
+  The Docker daemon won't start after a configuration change:
+
+  $ sudo systemctl start docker
+  Job for docker.service failed because the control process exited
+  with error code.
+
+  $ sudo journalctl -u docker -e
+  dockerd: unable to configure the Docker daemon with file
+  /etc/docker/daemon.json: invalid character '}' after object key
+
+  $ cat /etc/docker/daemon.json
+  {
+    "storage-driver": "overlay2",
+    "log-driver": "json-file",
+    "log-opts": {
+      "max-size": "10m"
+      "max-file": "3"
+    }
+  }
+
+  Missing comma after "max-size": "10m"! JSON syntax error.
+
+  After fixing the JSON, a different error:
+  $ sudo journalctl -u docker -e
+  dockerd: failed to start daemon: error initializing graphdriver:
+  driver not supported
+
+  The server has a btrfs filesystem but overlay2 was configured as
+  the storage driver. overlay2 requires ext4 or xfs.
+
+  After changing to the correct storage driver:
+  $ sudo systemctl start docker
+  (success, but all containers and images are gone!)
+
+  Changing storage drivers creates a new storage directory. The old
+  images and containers using the previous driver are inaccessible.
+
+  Additional daemon issues to cover:
+  - Disk space exhaustion preventing container creation
+  - iptables conflicts breaking container networking
+  - DNS configuration in daemon.json
+  - Live restore feature for daemon upgrades
+
+  Task: Explain Docker daemon troubleshooting. Write: how to read
+  daemon logs (journalctl, Docker Desktop logs), common daemon.json
+  errors (JSON syntax, invalid options), storage driver compatibility
+  and migration, disk space management (docker system df, prune), and
+  daemon configuration best practices.
+
+assertions:
+  - type: llm_judge
+    criteria: "Daemon log analysis is explained — Linux: journalctl -u docker for systemd logs, Docker Desktop: ~/Library/Logs/Docker/ (Mac). Common startup failures: invalid daemon.json (JSON syntax error), storage driver incompatibility, disk space exhaustion, port conflicts, permission issues on /var/run/docker.sock. Always validate daemon.json with jq before restarting: jq empty /etc/docker/daemon.json"
+    weight: 0.35
+    description: "Daemon log analysis"
+  - type: llm_judge
+    criteria: "Storage driver issues are covered — overlay2 requires ext4/xfs filesystem, btrfs driver requires btrfs filesystem. Changing storage drivers means losing access to existing images/containers (stored in /var/lib/docker). Check current driver: docker info | grep 'Storage Driver'. Check filesystem: df -T /var/lib/docker. Use data-root in daemon.json to change Docker's storage location"
+    weight: 0.35
+    description: "Storage driver issues"
+  - type: llm_judge
+    criteria: "Configuration best practices are practical — keep a backup of working daemon.json, validate JSON before applying, use live-restore: true to keep containers running during daemon restart, configure log rotation globally, set DNS servers if needed, monitor disk space with docker system df. Disk cleanup: docker system prune -a removes unused images, containers, networks, volumes"
+    weight: 0.30
+    description: "Configuration practices"

package/courses/docker-container-debugging/scenarios/level-3/docker-in-docker-ci.yaml

@@ -0,0 +1,69 @@
+meta:
+  id: docker-in-docker-ci
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug Docker-in-Docker and CI/CD pipeline issues — diagnose DinD socket mounting, build context problems, and CI-specific container failures"
+  tags: [Docker, DinD, CI/CD, socket, build-context, advanced]
+
+state: {}
+
+trigger: |
+  Your CI/CD pipeline (GitLab CI) suddenly fails at the Docker build step:
+
+  .gitlab-ci.yml:
+  build:
+    image: docker:24-dind
+    services:
+      - docker:24-dind
+    variables:
+      DOCKER_HOST: tcp://docker:2376
+      DOCKER_TLS_CERTDIR: "/certs"
+    script:
+      - docker build -t myapp:$CI_COMMIT_SHA .
+      - docker push registry.example.com/myapp:$CI_COMMIT_SHA
+
+  Error:
+  Cannot connect to the Docker daemon at tcp://docker:2376.
+  Is the docker daemon running?
+
+  Investigation:
+  1. The DinD service container starts but TLS handshake fails. The
+     DOCKER_TLS_CERTDIR variable must match between the client and
+     service containers. Also needs DOCKER_CERT_PATH and
+     DOCKER_TLS_VERIFY=1.
+
+  2. After fixing TLS, builds work but are extremely slow — every
+     build downloads all base images fresh because the DinD container
+     has no persistent storage.
+
+  3. Alternative approach — mounting the host Docker socket:
+     volumes:
+       - /var/run/docker.sock:/var/run/docker.sock
+     This is faster (shared image cache) but creates security risks:
+     the CI job has full access to the host Docker daemon.
+
+  4. The socket mount approach causes a different bug:
+     docker build -t myapp . uses the CI container's filesystem as
+     build context, but .dockerignore is missing, sending 2GB of
+     node_modules and .git to the daemon.
+
+  Task: Explain Docker-in-Docker and CI/CD debugging. Write: DinD vs
+  socket mounting (trade-offs), TLS configuration for DinD, build
+  caching strategies in CI (BuildKit cache mounts, registry cache),
+  build context optimization, and security considerations for Docker
+  in CI environments.
+
+assertions:
+  - type: llm_judge
+    criteria: "DinD vs socket mounting trade-offs are explained — DinD (docker:dind service): fully isolated, clean state each build, slower (no cache), more secure. Socket mounting (/var/run/docker.sock): uses host daemon, shared image cache (faster), but CI jobs get host-level Docker access (security risk). Socket mount also means containers created by CI are siblings, not children. DinD TLS setup requires matching DOCKER_TLS_CERTDIR, DOCKER_CERT_PATH, DOCKER_HOST with tcp:// and port 2376"
+    weight: 0.35
+    description: "DinD vs socket mounting"
+  - type: llm_judge
+    criteria: "Build caching in CI is covered — BuildKit inline cache: DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1. Registry-based cache: --cache-from registry.example.com/myapp:latest --cache-to type=registry. CI-specific: mount cache volumes for DinD, use --cache-from with previously pushed images. Build context: always use .dockerignore to exclude .git, node_modules, test files. Small context = fast builds"
+    weight: 0.35
+    description: "Build caching in CI"
+  - type: llm_judge
+    criteria: "Security considerations are practical — socket mounting gives full Docker access (can access other containers, mount host filesystem). DinD with --privileged also has elevated privileges. Alternatives: Kaniko (build without Docker daemon), Buildah (daemonless builds), Docker BuildKit with rootless mode. Credential management: use CI variables for registry auth, never hardcode in Dockerfile. Image scanning in CI pipeline before push"
+    weight: 0.30
+    description: "Security considerations"

package/courses/docker-container-debugging/scenarios/level-3/overlay-network-debugging.yaml

@@ -0,0 +1,70 @@
+meta:
+  id: overlay-network-debugging
+  level: 3
+  course: docker-container-debugging
+  type: output
+  description: "Debug Docker overlay and multi-host networking — diagnose Swarm overlay networks, VXLAN issues, and cross-host container communication"
+  tags: [Docker, overlay, Swarm, VXLAN, multi-host, networking, advanced]
+
+state: {}
+
+trigger: |
+  Your Docker Swarm cluster has 3 nodes. Services on different nodes
+  can't communicate even though they're on the same overlay network:
+
+  $ docker network create -d overlay --attachable mynet
+  $ docker service create --name api --network mynet api-image
+  $ docker service create --name db --network mynet postgres:15
+
+  api and db are scheduled on different nodes:
+  $ docker service ps api
+  ID         NAME    NODE    CURRENT STATE
+  abc123     api.1   node-1  Running 5 minutes ago
+
+  $ docker service ps db
+  ID         NAME    NODE    CURRENT STATE
+  def456     db.1   node-2  Running 3 minutes ago
+
+  From api container:
+  $ docker exec api.1 ping db
+  PING db (10.0.1.5): 56 data bytes
+  (no response — packets lost)
+
+  Investigation:
+
+  1. The overlay network uses VXLAN (UDP port 4789) for encapsulation.
+     Node-2's firewall blocks UDP 4789:
+     $ sudo iptables -L -n | grep 4789
+     (no rule — VXLAN traffic is being dropped)
+
+  2. After opening port 4789, pings work but TCP connections to
+     postgres:5432 intermittently fail. MTU issue — VXLAN adds 50
+     bytes of overhead. The default MTU of 1500 causes fragmentation:
+     $ docker exec api.1 ping -s 1450 db
+     (works)
+     $ docker exec api.1 ping -s 1472 db
+     (fails — packet too large with VXLAN overhead)
+
+  3. DNS resolution between services uses Docker's embedded DNS
+     (127.0.0.11). In some cases, the VIP (virtual IP) load balancing
+     sends traffic to a container that hasn't finished starting.
+
+  Task: Explain Docker overlay network debugging. Write: how overlay
+  networks work (VXLAN encapsulation, control plane vs data plane),
+  required ports for Swarm (2377, 7946, 4789), MTU considerations,
+  DNS service discovery in Swarm (VIP vs DNSRR), cross-host debugging
+  tools and techniques, and common overlay network failure modes.
+
+assertions:
+  - type: llm_judge
+    criteria: "Overlay network mechanics are explained — overlay networks use VXLAN to encapsulate Layer 2 frames in UDP packets (port 4789). Docker Swarm requires: TCP 2377 (cluster management), TCP/UDP 7946 (node communication/gossip), UDP 4789 (VXLAN data). If any port is blocked, overlay networking fails. The ingress network handles published port routing (routing mesh). Each service gets a VIP on the overlay network"
+    weight: 0.35
+    description: "Overlay mechanics"
+  - type: llm_judge
+    criteria: "MTU and DNS issues are covered — VXLAN adds 50 bytes overhead, so effective MTU is 1450 (not 1500). Symptoms of MTU issues: small packets work, large transfers fail or hang. Fix: set MTU on overlay network creation (--opt com.docker.network.driver.mtu=1450) or in daemon.json. DNS: services resolve via embedded DNS (127.0.0.11). VIP mode (default) provides single virtual IP with load balancing. DNSRR mode returns all container IPs. VIP can route to starting containers — use health checks"
+    weight: 0.35
+    description: "MTU and DNS"
+  - type: llm_judge
+    criteria: "Debugging techniques are practical — check network connectivity: docker network inspect (shows connected containers and IPs). Use netshoot container on overlay network for debugging. Check VXLAN: tcpdump on host for UDP 4789. Verify port rules: iptables -L -n. Check Swarm gossip: docker node ls (all nodes must be Ready). Common failures: firewall blocking Swarm ports, MTU mismatch, stale network state (docker network disconnect/connect to reset)"
+    weight: 0.30
+    description: "Debugging techniques"