dojo.md 0.1.0 → 0.2.1

package/courses/aws-lambda-debugging/scenarios/level-1/memory-and-oom.yaml

@@ -0,0 +1,70 @@
+meta:
+  id: memory-and-oom
+  level: 1
+  course: aws-lambda-debugging
+  type: output
+  description: "Debug Lambda memory issues — diagnose out-of-memory errors, understand memory-CPU relationship, and right-size Lambda functions"
+  tags: [AWS, Lambda, memory, OOM, right-sizing, configuration, beginner]
+
+state: {}
+
+trigger: |
+  Your Lambda function fails intermittently with large payloads:
+
+  {
+    "errorType": "Runtime.ExitError",
+    "errorMessage": "RequestId: abc-123 Error: Runtime exited with
+    error: signal: killed"
+  }
+
+  CloudWatch REPORT line:
+  REPORT RequestId: abc-123 Duration: 4521.00 ms
+  Billed Duration: 4521 ms Memory Size: 128 MB
+  Max Memory Used: 128 MB
+
+  Max Memory Used equals Memory Size — the function ran out of memory
+  and was killed by the OOM killer. "signal: killed" = SIGKILL from
+  the system.
+
+  The function processes CSV files from S3. Small files work (< 5MB),
+  large files fail because loading the entire file into memory exceeds
+  128MB.
+
+  Fix options:
+  1. Increase memory:
+     aws lambda update-function-configuration \
+       --function-name process-csv --memory-size 512
+
+  2. Stream processing — read the file in chunks instead of loading
+     it entirely into memory.
+
+  Important: Lambda memory also controls CPU allocation!
+  - 128 MB  = ~0.08 vCPU (very slow)
+  - 512 MB  = ~0.3 vCPU
+  - 1769 MB = 1 vCPU
+  - 3538 MB = 2 vCPU
+  - 10240 MB = 6 vCPU (maximum)
+
+  A function at 128MB runs on a fraction of a CPU. Increasing memory
+  to 256MB might actually DECREASE total cost because the function
+  runs 3x faster (more CPU) even though per-ms cost doubles.
+
+  Task: Explain Lambda memory debugging. Write: how to identify OOM
+  errors (signal: killed, Max Memory Used = Memory Size), the
+  memory-CPU relationship, how to right-size memory (AWS Lambda
+  Power Tuning), common memory-hungry operations (loading files,
+  large JSON parsing), and cost optimization through memory tuning.
+
+assertions:
+  - type: llm_judge
+    criteria: "OOM identification is explained — signal: killed or Runtime.ExitError with killed means OOM. Check REPORT line: if Max Memory Used equals or is very close to Memory Size, function ran out of memory. Memory includes: your code, runtime, libraries, and all variables/data in memory. /tmp directory has separate 512MB-10GB allocation (configurable). Lambda memory range: 128MB to 10,240MB in 1MB increments"
+    weight: 0.35
+    description: "OOM identification"
+  - type: llm_judge
+    criteria: "Memory-CPU relationship is covered — Lambda allocates CPU proportional to memory. 1,769MB = 1 full vCPU. At 128MB you get ~7% of a vCPU — CPU-bound tasks are extremely slow. This means: increasing memory for a CPU-bound function makes it faster AND can make it cheaper (shorter duration offsets higher per-ms cost). Use AWS Lambda Power Tuning tool to find optimal memory: tests your function at different memory sizes and plots cost vs duration. Memory is the ONLY performance lever for Lambda — no separate CPU setting"
+    weight: 0.35
+    description: "Memory-CPU relationship"
+  - type: llm_judge
+    criteria: "Right-sizing is practical — use Lambda Power Tuning (open-source Step Functions tool): runs your function at multiple memory configs, shows cost and duration at each. Look for the 'sweet spot' where increasing memory stops improving duration. Common memory-hungry operations: loading entire files into memory (stream instead), parsing large JSON (use streaming JSON parser), image processing (needs 512MB+). Monitor Max Memory Used metric over time. Set memory to ~1.5x the typical Max Memory Used for headroom. /tmp for large file processing (up to 10GB ephemeral storage)"
+    weight: 0.30
+    description: "Right-sizing"

package/courses/aws-lambda-debugging/scenarios/level-2/async-invocation-failures.yaml

@@ -0,0 +1,72 @@
+meta:
+  id: async-invocation-failures
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Debug Lambda async invocation failures — diagnose retry behavior, dead-letter queue issues, and Lambda Destination configuration"
+  tags: [AWS, Lambda, async, DLQ, destinations, retry, intermediate]
+
+state: {}
+
+trigger: |
+  Your S3-triggered Lambda function processes uploaded images. Some
+  images fail processing but you have no idea which ones or why:
+
+  $ aws cloudwatch get-metric-statistics --namespace AWS/Lambda \
+    --metric-name Errors --dimensions Name=FunctionName,Value=process-image
+  Errors: 45 in the last hour
+
+  45 errors but no visibility into which uploads failed. The function
+  is invoked asynchronously (S3 event) — errors don't go back to a
+  caller. Where do they go?
+
+  Async invocation error flow:
+  1. First attempt: function fails
+  2. Lambda waits 1 minute, retries (attempt 2)
+  3. Lambda waits 2 minutes, retries (attempt 3)
+  4. All 3 attempts failed → event is discarded (lost!)
+
+  Without a DLQ or destination, failed events simply disappear.
+
+  Fix 1 — Dead Letter Queue:
+  $ aws lambda update-function-configuration \
+    --function-name process-image \
+    --dead-letter-config TargetArn=arn:aws:sqs:...:image-dlq
+
+  Now failed events go to SQS after all retries exhausted. But the
+  DLQ only contains the original event, not the error details.
+
+  Fix 2 — Lambda Destinations (better):
+  $ aws lambda put-function-event-invoke-config \
+    --function-name process-image \
+    --destination-config '{
+      "OnSuccess": {"Destination": "arn:aws:sqs:...:success-queue"},
+      "OnFailure": {"Destination": "arn:aws:sqs:...:failure-queue"}
+    }'
+
+  Destinations include: the original event, the function response or
+  error, request/response metadata. Much more useful than DLQ.
+
+  Additional issue: MaximumRetryAttempts set to 0 means no retries.
+  But this also means transient errors (temporary network glitch)
+  are not retried. Tradeoff: retries help with transient failures
+  but tripling processing for permanent errors wastes compute.
+
+  Task: Explain async invocation debugging. Write: the retry behavior
+  (timing, attempt count), DLQ vs Lambda Destinations (comparison),
+  configuring retry limits and maximum event age, monitoring async
+  failures, and designing robust async processing pipelines.
+
+assertions:
+  - type: llm_judge
+    criteria: "Async retry behavior is explained — async invocation: Lambda queues event internally, returns 202 immediately. On failure: retries 2 more times (3 total). Delay: ~1 minute between first and second, ~2 minutes between second and third. After all retries: event sent to DLQ/destination or discarded. MaximumRetryAttempts: configure 0-2 retries. MaximumEventAgeInSeconds: discard events older than X seconds (60-21600). Important: retries mean the function must be idempotent — same event may be processed multiple times"
+    weight: 0.35
+    description: "Retry behavior"
+  - type: llm_judge
+    criteria: "DLQ vs Destinations are compared — DLQ: only captures original event on failure, configured on function, supports SQS and SNS only. Lambda Destinations: captures original event + function response/error + metadata, supports SQS/SNS/Lambda/EventBridge, can handle both success and failure. Destinations are preferred (more information, more flexible). DLQ limitation: doesn't tell you WHY it failed. Destination OnFailure: includes error message, stack trace, request context. Use destinations for new functions, DLQ still works for existing setups"
+    weight: 0.35
+    description: "DLQ vs Destinations"
+  - type: llm_judge
+    criteria: "Monitoring and design are practical — monitor: AsyncEventsDropped metric (events lost without DLQ/destination), DeadLetterErrors (failed to send to DLQ). Alert on both! Design pattern: S3 event → Lambda → on failure: SQS DLQ → monitoring Lambda that alerts and logs details. Make functions idempotent: use a deduplication table to prevent reprocessing. For critical events: record event in DynamoDB before processing, mark as processed when done, sweep unprocessed events periodically as backup. Test failure scenarios: deliberately throw errors to verify DLQ/destination works"
+    weight: 0.30
+    description: "Monitoring and design"

package/courses/aws-lambda-debugging/scenarios/level-2/cold-start-optimization.yaml

@@ -0,0 +1,76 @@
+meta:
+  id: cold-start-optimization
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Optimize Lambda cold starts — implement provisioned concurrency, SnapStart, and package optimization to reduce initialization latency"
+  tags: [AWS, Lambda, cold-start, provisioned-concurrency, SnapStart, optimization, intermediate]
+
+state: {}
+
+trigger: |
+  Your Java Lambda function has cold starts of 8-12 seconds. Users
+  hitting the API after an idle period wait 10+ seconds for the first
+  response. This is unacceptable for a user-facing API.
+
+  CloudWatch REPORT for cold start:
+  REPORT RequestId: abc-123 Duration: 450.00 ms
+  Billed Duration: 8950 ms Memory Size: 1024 MB
+  Max Memory Used: 256 MB Init Duration: 8500.00 ms
+
+  Init Duration: 8.5 seconds! The handler only takes 450ms.
+
+  Breakdown of cold start time:
+  - Download code package: 500ms (50MB jar)
+  - Start JVM: 3,000ms
+  - Class loading & dependency injection: 4,500ms
+  - First database connection: 500ms
+
+  Optimization approaches:
+
+  1. SnapStart (Java 11+, .NET 8):
+     Lambda snapshots the initialized function after the first init.
+     Subsequent cold starts restore from snapshot instead of
+     re-initializing. Reduces cold start from 8.5s to ~200ms.
+     $ aws lambda update-function-configuration \
+       --function-name my-func \
+       --snap-start ApplyOn=PublishedVersions
+     Caveat: must use published versions (not $LATEST).
+
+  2. Provisioned Concurrency:
+     Pre-initialize N execution environments. Zero cold starts
+     for up to N concurrent requests.
+     $ aws lambda put-provisioned-concurrency-config \
+       --function-name my-func --qualifier prod \
+       --provisioned-concurrent-executions 10
+     Cost: ~$15/month per provisioned environment (always running).
+
+  3. Package optimization:
+     Reduce jar from 50MB to 15MB (remove unused dependencies).
+     Use GraalVM native image for sub-second JVM start.
+     Use lighter frameworks (Micronaut/Quarkus instead of Spring).
+
+  4. Warm-up pings:
+     CloudWatch Events rule invokes function every 5 minutes.
+     Keeps one environment warm but doesn't prevent cold starts
+     during scaling.
+
+  Task: Explain cold start optimization. Write: SnapStart (how it
+  works, requirements, limitations), provisioned concurrency (cost,
+  auto-scaling, when to use), package optimization techniques,
+  runtime-specific advice (Java vs Node.js vs Python), and how to
+  measure cold start impact on users.
+
+assertions:
+  - type: llm_judge
+    criteria: "SnapStart and provisioned concurrency are explained — SnapStart: snapshots initialized function at publish time, restores on cold start (~200ms instead of seconds). Works with Java 11+ (Corretto), .NET 8. Must publish versions. Cannot combine with provisioned concurrency. Limitations: uniqueness (random/UUID generated during init may be duplicated). Provisioned concurrency: pre-initializes environments, eliminates cold starts entirely for configured capacity. Cost: hourly charge per provisioned unit. Auto-scaling provisioned concurrency via Application Auto Scaling for variable traffic"
+    weight: 0.35
+    description: "SnapStart and provisioned"
+  - type: llm_judge
+    criteria: "Package and runtime optimization are covered — reduce package size: remove unused dependencies, use tree shaking/bundlers. Java: use Micronaut/Quarkus (designed for Lambda), GraalVM native image (sub-100ms starts), avoid Spring Boot (heavy reflection/classpath scanning). Node.js: bundle with esbuild, lazy-load heavy modules. Python: avoid large packages (pandas, numpy) unless needed, use Lambda-optimized versions. General: initialize SDK clients and DB connections outside handler (reused on warm starts). Measure Init Duration in REPORT line"
+    weight: 0.35
+    description: "Package and runtime"
+  - type: llm_judge
+    criteria: "Measurement and decision-making are practical — measure cold start impact: percentage of invocations that are cold starts (check Init Duration presence in CloudWatch Logs Insights). Query: filter @type = 'REPORT' | stats count() as total, sum(@initDuration > 0) as coldStarts. Calculate: if 5% of requests are cold starts with 5s delay, average latency impact is 250ms. Decision: if cold starts affect p99 latency for user-facing APIs → use SnapStart or provisioned concurrency. If async processing (SQS, events) → cold starts usually don't matter. Cost-benefit: provisioned concurrency costs $15/month per unit — compare to user impact of cold starts"
+    weight: 0.30
+    description: "Measurement"

package/courses/aws-lambda-debugging/scenarios/level-2/dynamodb-streams-debugging.yaml

@@ -0,0 +1,70 @@
+meta:
+  id: dynamodb-streams-debugging
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Debug Lambda DynamoDB Streams integration — diagnose stream processing failures, iterator exhaustion, and event ordering issues"
+  tags: [AWS, Lambda, DynamoDB, Streams, event-source, iterator, intermediate]
+
+state: {}
+
+trigger: |
+  Your Lambda function processes DynamoDB Streams events (change data
+  capture) to sync order data to Elasticsearch. It stopped processing:
+
+  $ aws lambda list-event-source-mappings \
+    --function-name sync-to-es --query "EventSourceMappings[].State"
+  ["Enabled"]
+
+  The mapping is enabled but CloudWatch shows no invocations. Checking
+  the event source mapping details:
+
+  $ aws lambda get-event-source-mapping --uuid abc-123
+  {
+    "State": "Enabled",
+    "StateTransitionReason": "USER_INITIATED",
+    "LastProcessingResult": "PROBLEM: Function call failed"
+  }
+
+  LastProcessingResult: "Function call failed" — the function threw
+  an unhandled error. For DynamoDB Streams (and Kinesis), Lambda
+  retries the SAME batch until it succeeds or the record expires
+  (24 hours by default). One bad record blocks the entire shard!
+
+  CloudWatch Logs (from the failing invocation):
+  Processing INSERT event for order ORD-999
+  Error: Cannot read property 'S' of undefined
+  — A DynamoDB record had an unexpected schema (missing field)
+
+  The function was stuck retrying the same bad record for 6 hours,
+  blocking all subsequent stream records.
+
+  Fix — configure error handling:
+  $ aws lambda update-event-source-mapping --uuid abc-123 \
+    --maximum-retry-attempts 3 \
+    --bisect-batch-on-function-error \
+    --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:..."}}'
+
+  - maximum-retry-attempts: stop retrying after 3 attempts
+  - bisect-batch-on-function-error: split batch to isolate bad record
+  - destination on failure: send failed records to SQS for investigation
+
+  Task: Explain DynamoDB Streams + Lambda debugging. Write: how
+  stream processing works (shards, iterators, ordering), the blocking
+  behavior on errors (entire shard stops), error handling configuration
+  (retry limits, bisect, on-failure destination), common stream
+  processing patterns, and monitoring stream consumer lag.
+
+assertions:
+  - type: llm_judge
+    criteria: "Stream processing mechanics are explained — DynamoDB Streams capture INSERT, MODIFY, DELETE events in order per partition key. Lambda polls stream shards via event source mapping. Records include OldImage and/or NewImage (depending on StreamViewType). Processing is ordered within a shard — one batch at a time. If Lambda fails, it retries the SAME batch (blocks shard). Different from SQS: stream records must be processed in order, so failed records block subsequent records"
+    weight: 0.35
+    description: "Stream mechanics"
+  - type: llm_judge
+    criteria: "Error handling and blocking are covered — default behavior: retry forever until record expires (24h) — blocks shard processing. Configure: MaximumRetryAttempts (limit retries), BisectBatchOnFunctionError (split batch to isolate bad record), MaximumRecordAgeInSeconds (skip old records), DestinationConfig OnFailure (send failed records to SQS/SNS for investigation). Without these settings, one bad record stops ALL processing on that shard. Always configure error handling for stream consumers"
+    weight: 0.35
+    description: "Error handling"
+  - type: llm_judge
+    criteria: "Monitoring and patterns are practical — monitor: IteratorAge metric (time between record written and processed — high value means lag). Alert when IteratorAge exceeds acceptable threshold (e.g., 5 minutes). Check: LastProcessingResult in event source mapping for error state. Common patterns: use DynamoDB Streams for CDC (change data capture), search index sync, cross-region replication, audit logging. Design Lambda for idempotency (records may be delivered more than once). Handle schema evolution gracefully (new/missing fields shouldn't crash the function)"
+    weight: 0.30
+    description: "Monitoring and patterns"

package/courses/aws-lambda-debugging/scenarios/level-2/intermediate-debugging-shift.yaml

@@ -0,0 +1,71 @@
+meta:
+  id: intermediate-debugging-shift
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Combined intermediate debugging shift — diagnose a serverless application with event source, VPC, layer, and concurrency issues simultaneously"
+  tags: [AWS, Lambda, troubleshooting, combined, shift-simulation, intermediate]
+
+state: {}
+
+trigger: |
+  You're troubleshooting a serverless order processing system with
+  5 Lambda functions. The system "partially works":
+
+  Architecture:
+  API Gateway → order-api (Lambda)
+  order-api → SQS queue → process-order (Lambda)
+  process-order → DynamoDB + payment-service (Lambda)
+  payment-service → Stripe API (external)
+  DynamoDB Stream → sync-to-search (Lambda) → OpenSearch
+
+  Current issues:
+
+  1. order-api: intermittent 504 errors from API Gateway.
+     REPORT shows Init Duration: 4200ms for cold starts.
+     The function is in a VPC (for RDS access) and uses a heavy
+     Python layer with pandas (unnecessary for this API).
+     Fix: remove pandas layer, add VPC endpoint for DynamoDB,
+     consider provisioned concurrency.
+
+  2. process-order: messages accumulate in SQS, not being processed.
+     Event source mapping shows LastProcessingResult: "OK" but
+     BatchSize is 1 and function processes one message per second.
+     With 500 messages/minute incoming, it can't keep up.
+     Fix: increase batch size to 10, enable batch window of 5s.
+
+  3. payment-service: throws errors but retries succeed.
+     The function calls Stripe API with no timeout configured.
+     Occasionally Stripe takes 15+ seconds, exceeding the Lambda's
+     10-second timeout. On retry (async), it succeeds → duplicate
+     payment! No idempotency key used.
+     Fix: set HTTP timeout < Lambda timeout, use idempotency key.
+
+  4. sync-to-search: completely stuck, IteratorAge growing.
+     One malformed DynamoDB record causes the function to crash,
+     blocking the entire shard. No error handling configured.
+     Fix: configure MaximumRetryAttempts, BisectBatchOnFunctionError,
+     OnFailure destination.
+
+  5. Account-wide: ConcurrentExecutions at 850/1000. No reserved
+     concurrency configured. A batch job running at the same time
+     could push the account to the limit.
+     Fix: set reserved concurrency for critical functions.
+
+  Task: Walk through diagnosing all issues. Write: the triage
+  approach for serverless architectures, the fix for each issue,
+  idempotency patterns, and operational best practices.
+
+assertions:
+  - type: llm_judge
+    criteria: "All five issues are diagnosed and fixed — (1) cold start: remove unnecessary layer, add VPC endpoints, use provisioned concurrency for API-facing function. (2) SQS throughput: increase batch size and batch window for higher processing rate. (3) duplicate payments: set HTTP client timeout < Lambda timeout, use idempotency keys in Stripe API calls. (4) stream blocking: configure retry limits, bisect on error, failure destination for DynamoDB Streams event source. (5) concurrency: set reserved concurrency for critical functions to prevent resource starvation"
+    weight: 0.35
+    description: "All issues fixed"
+  - type: llm_judge
+    criteria: "Idempotency and error handling are covered — idempotency: use unique transaction IDs for payment operations (Stripe supports Idempotency-Key header). Store processing status in DynamoDB before and after processing. Design all Lambda functions to be safely re-invokable. Error handling: different for each invocation pattern — sync (return error to caller), async (retry + DLQ), stream (retry + bisect + destination). Always configure error handling for event sources — defaults are not production-ready"
+    weight: 0.35
+    description: "Idempotency and errors"
+  - type: llm_judge
+    criteria: "Serverless triage approach is systematic — check CloudWatch metrics first: Errors, Throttles, Duration, ConcurrentExecutions. For event sources: check event source mapping state and LastProcessingResult. For streams: check IteratorAge (growing = falling behind). For SQS: check ApproximateAgeOfOldestMessage (lag indicator). Work through the data flow: API → queue → processing → database → stream → search. Fix upstream issues first (they cascade). Test each fix independently. Monitor for 30+ minutes after fixes to confirm stability"
+    weight: 0.30
+    description: "Triage approach"

package/courses/aws-lambda-debugging/scenarios/level-2/lambda-concurrency-management.yaml

@@ -0,0 +1,70 @@
+meta:
+  id: lambda-concurrency-management
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Debug Lambda concurrency issues — diagnose throttling, configure reserved and provisioned concurrency, and protect downstream services"
+  tags: [AWS, Lambda, concurrency, throttling, reserved, scaling, intermediate]
+
+state: {}
+
+trigger: |
+  A marketing campaign sends a blast email with a link to your API.
+  Traffic spikes 10x. Multiple functions start throttling:
+
+  CloudWatch Metrics (account-wide):
+  ConcurrentExecutions: 1,000 (at limit!)
+  Throttles (order-api): 5,200
+  Throttles (user-api): 1,800
+  Throttles (analytics): 450
+
+  All functions compete for the same 1,000 account concurrency pool.
+  The analytics function (low priority) consumed 600 concurrent
+  executions processing a backlog, leaving only 400 for critical
+  order and user APIs.
+
+  Fix 1 — Reserved concurrency (free):
+  $ aws lambda put-function-concurrency \
+    --function-name order-api \
+    --reserved-concurrent-executions 300
+  $ aws lambda put-function-concurrency \
+    --function-name user-api \
+    --reserved-concurrent-executions 200
+  $ aws lambda put-function-concurrency \
+    --function-name analytics \
+    --reserved-concurrent-executions 50
+
+  Now order-api is guaranteed 300 concurrent executions. But:
+  reserved concurrency is also a MAXIMUM — order-api can never
+  exceed 300 even if capacity is available.
+
+  Unreserved pool: 1000 - 300 - 200 - 50 = 450 for all other functions.
+
+  Fix 2 — Downstream protection:
+  Your RDS database has a max_connections of 100. If 300 Lambda
+  instances each open a connection, the database is overwhelmed:
+  "too many connections" error.
+
+  Solution: RDS Proxy pools database connections. Lambda instances
+  connect to RDS Proxy, which maintains a connection pool to the
+  database.
+
+  Task: Explain Lambda concurrency management. Write: account vs
+  function concurrency limits, reserved concurrency (guarantees AND
+  limits), provisioned concurrency (pre-warmed), protecting downstream
+  services (RDS Proxy, SQS buffering), scaling behavior, and how to
+  request limit increases.
+
+assertions:
+  - type: llm_judge
+    criteria: "Concurrency model is explained — account default: 1,000 concurrent executions (soft limit, can increase). All functions share the pool unless reserved concurrency is set. Reserved concurrency: guarantees AND limits a function's concurrent executions (free, no additional cost). Unreserved pool: total limit minus all reserved concurrency — must leave at least 100 unreserved. Provisioned concurrency: pre-initialized environments for instant response (charged per provisioned unit per hour). Scaling: Lambda scales at 500-3000 instances per minute depending on region"
+    weight: 0.35
+    description: "Concurrency model"
+  - type: llm_judge
+    criteria: "Downstream protection is covered — Lambda can scale faster than downstream services. Database: max_connections limit — use RDS Proxy (connection pooling), or limit Lambda concurrency. APIs: use reserved concurrency to match API rate limits. SQS: use as buffer between Lambda and slow services. Pattern: high-concurrency Lambda → SQS → low-concurrency Lambda → database. ElastiCache: use as cache to reduce database load. Always consider downstream capacity when setting Lambda concurrency"
+    weight: 0.35
+    description: "Downstream protection"
+  - type: llm_judge
+    criteria: "Scaling and limits are practical — request limit increase via AWS Support (usually approved within days). Monitor: ConcurrentExecutions, Throttles, UnreservedAccountConcurrency metrics. Alert on throttles (any throttle = degraded service). Scaling behavior: burst (3000 in us-east-1, 1000 in other regions for first minute), then 500 additional per minute. Event source specific: SQS scales up to 1000 batches/minute, Kinesis limited by shard count. Plan capacity: expected peak concurrent executions = (invocations per second) × (average duration in seconds)"
+    weight: 0.30
+    description: "Scaling and limits"

package/courses/aws-lambda-debugging/scenarios/level-2/lambda-layers-debugging.yaml

@@ -0,0 +1,76 @@
+meta:
+  id: lambda-layers-debugging
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Debug Lambda Layer issues — diagnose layer version conflicts, incorrect directory structure, and dependency resolution problems"
+  tags: [AWS, Lambda, layers, dependencies, version-conflicts, intermediate]
+
+state: {}
+
+trigger: |
+  Your Lambda function fails after adding a new layer:
+
+  {
+    "errorType": "Runtime.ImportModuleError",
+    "errorMessage": "Unable to import module 'handler': No module
+    named 'pandas'"
+  }
+
+  But pandas IS in your layer! Checking the layer:
+
+  $ aws lambda get-layer-version --layer-name data-layer --version 3 \
+    --query "Content.Location" --output text | xargs curl -o layer.zip
+  $ unzip -l layer.zip
+  Archive: layer.zip
+    Length  Name
+    ------  ----
+           pandas/
+           pandas/__init__.py
+           ...
+
+  Problem: wrong directory structure! Lambda expects Python
+  dependencies in specific paths:
+  - python/
+  - python/lib/python3.x/site-packages/
+
+  The layer has pandas/ at the root instead of python/pandas/.
+  Lambda can't find it.
+
+  Correct structure:
+  $ mkdir -p python/lib/python3.12/site-packages/
+  $ pip install pandas -t python/lib/python3.12/site-packages/
+  $ zip -r layer.zip python/
+
+  After fixing, a new error:
+  "numpy.core._multiarray_umath: undefined symbol: PyFloat_Type"
+
+  The numpy in the layer was compiled for Python 3.11 but the
+  function uses Python 3.12. Native extensions must match the
+  exact Python version.
+
+  Another issue — layer order matters:
+  Function has layers: [common-utils:v2, data-layer:v3]
+  Both layers have a different version of the 'requests' library.
+  Layer 2 (data-layer) overrides layer 1 (common-utils) because
+  layers are merged in order, with later layers taking precedence.
+
+  Task: Explain Lambda Layer debugging. Write: correct layer
+  directory structure for each runtime, version compatibility
+  (runtime, architecture), layer merge order and conflicts, the
+  250MB unzipped limit (function + all layers), and when to use
+  layers vs bundled dependencies.
+
+assertions:
+  - type: llm_judge
+    criteria: "Layer structure is explained — Python: python/ or python/lib/python3.x/site-packages/. Node.js: nodejs/node_modules/. Both: lib/ or bin/ for shared libraries/executables. Wrong directory structure is the #1 layer issue. Lambda adds layer paths to the runtime's module search path. Must match exactly or imports fail. Check with: unzip -l layer.zip to verify structure before publishing"
+    weight: 0.35
+    description: "Layer structure"
+  - type: llm_judge
+    criteria: "Compatibility and conflicts are covered — runtime version: native extensions (numpy, pandas, Pillow) must match the exact Python version. Build on matching Amazon Linux environment or use Docker. Architecture: layers must match function architecture (x86_64 vs arm64). Layer order: layers merged in order, later layers override earlier ones if they have the same file. Up to 5 layers per function. Total size: function code + all layers ≤ 250MB unzipped. Debugging: print sys.path (Python) or module.paths (Node.js) to see layer paths"
+    weight: 0.35
+    description: "Compatibility"
+  - type: llm_judge
+    criteria: "Usage guidance is practical — use layers for: shared code across multiple functions (utility libraries, SDK wrappers), large dependencies that don't change often (pandas, numpy), custom runtimes. Don't use layers for: function-specific code (bundle with function), frequently updated dependencies (layer versioning adds complexity). Alternative: use esbuild/webpack bundling for smaller packages. Container images eliminate layer complexity entirely. Layer versions are immutable — publish new version for updates, update all functions referencing the old version"
+    weight: 0.30
+    description: "Usage guidance"

package/courses/aws-lambda-debugging/scenarios/level-2/sam-local-debugging.yaml

@@ -0,0 +1,74 @@
+meta:
+  id: sam-local-debugging
+  level: 2
+  course: aws-lambda-debugging
+  type: output
+  description: "Debug Lambda locally with SAM CLI — use sam local invoke, start-api, and generate-event for local development and testing"
+  tags: [AWS, Lambda, SAM, local-testing, debugging, development, intermediate]
+
+state: {}
+
+trigger: |
+  Your development workflow is painful: make a change, deploy to AWS
+  (2 minutes), test, see error, repeat. You need local debugging.
+
+  Setting up SAM CLI:
+
+  $ sam init --runtime python3.12 --app-template hello-world
+  $ sam build
+  $ sam local invoke HelloWorldFunction
+  Invoking app.lambda_handler (python3.12)
+  Building image...
+  START RequestId: local-123
+  {"statusCode": 200, "body": "{\"message\": \"hello world\"}"}
+  END RequestId: local-123
+  REPORT RequestId: local-123 Duration: 5.23 ms
+
+  Works! But now testing with a real event:
+
+  $ sam local generate-event apigateway aws-proxy > event.json
+  $ sam local invoke HelloWorldFunction -e event.json
+  Error: Template file not found: template.yaml
+
+  Wrong directory — SAM needs template.yaml (infrastructure definition).
+
+  Testing API locally:
+  $ sam local start-api
+  Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
+  $ curl http://localhost:3000/hello
+  {"message": "hello world"}
+
+  But environment variables are missing:
+  $ sam local invoke -n env.json HelloWorldFunction
+  # env.json: {"HelloWorldFunction": {"TABLE_NAME": "local-table"}}
+
+  Debugging with breakpoints:
+  $ sam local invoke -d 5858 HelloWorldFunction
+  Debugger listening on ws://127.0.0.1:5858
+  (attach VS Code debugger to port 5858)
+
+  Limitations:
+  - Local execution uses Docker — slightly different from actual Lambda
+  - IAM permissions not enforced locally
+  - VPC networking not simulated
+  - Event source mappings not simulated (must provide events manually)
+
+  Task: Explain SAM CLI local debugging. Write: sam local invoke
+  (function testing), sam local start-api (API testing), generating
+  test events, environment variables and configuration, debugging
+  with breakpoints (VS Code), limitations of local testing, and
+  complementary tools (LocalStack, moto for AWS mocking).
+
+assertions:
+  - type: llm_judge
+    criteria: "SAM local commands are explained — sam local invoke: runs function once with optional event file (-e event.json). sam local start-api: starts local HTTP server that routes to Lambda functions (mirrors API Gateway). sam local start-lambda: starts local Lambda endpoint for SDK testing. sam local generate-event: creates sample events for various triggers (apigateway, s3, sqs, dynamodb, etc.). sam build: compiles/packages code before local testing. All require Docker running locally"
+    weight: 0.35
+    description: "SAM commands"
+  - type: llm_judge
+    criteria: "Debugging and configuration are covered — breakpoint debugging: sam local invoke -d <port> pauses function until debugger attaches. VS Code: launch.json configuration for SAM debugging. Environment variables: -n env.json file or --parameter-overrides. Docker network: --docker-network to connect Lambda to local services (local DynamoDB, PostgreSQL). Hot reloading: --skip-pull-image for faster iteration. Layer testing: layers specified in template.yaml are automatically included in local invocations"
+    weight: 0.35
+    description: "Debugging"
+  - type: llm_judge
+    criteria: "Limitations and alternatives are practical — SAM local limitations: no IAM enforcement (permission errors won't appear), no VPC simulation, no event source mapping simulation, Docker-based execution has slight differences from real Lambda. Alternatives: LocalStack (full AWS service emulation), moto (Python AWS mocking), DynamoDB Local (official local DynamoDB). Best practice: local testing for rapid development iteration, deploy to a dev AWS account for integration testing, use CI/CD for automated testing against real services. Unit tests mock AWS SDK calls for fast feedback"
+    weight: 0.30
+    description: "Limitations"