vybekiit 0.7.0 → 0.7.2

package/dist/global-skills/aws-serverless/references/troubleshooting.md

@@ -1,164 +1,711 @@
 # Serverless Troubleshooting Reference
 
-Error → cause → fix, focused on non-obvious gotchas and exact CLI commands. The five most common fixes are first.
+Actionable error lookup tables: exact error string → cause → fix with CLI commands.
+
+## Contents
+
+- [Quick fixes](#quick-fixes)
+- [Lambda Error Lookup](#lambda-error-lookup)
+- [API Gateway Error Lookup](#api-gateway-error-lookup)
+- [Step Functions Error Lookup](#step-functions-error-lookup)
+- [SAM/CDK Error Lookup](#samcdk-error-lookup)
+- [Timeout Debugging](#timeout-debugging)
+- [OOM Debugging](#out-of-memory-oom-debugging)
+- [Throttling Diagnosis](#throttling-diagnosis)
+- [CloudWatch Logs Insights Queries](#cloudwatch-logs-insights-queries)
+- [X-Ray Tracing](#x-ray-tracing)
+
+---
 
 ## Quick fixes
 
-- **502 Bad Gateway (proxy):** Lambda response must be `{ statusCode: int, headers: {}, body: "string" }`. `body` must be a `JSON.stringify`'d **string**, not an object. The function ran fine — only the response shape was wrong. (String statusCode is coerced. Under **HTTP API** payload format 2.0 a missing statusCode defaults to 200, but under **REST** proxy a missing statusCode is itself a malformed response and triggers the 502.)
-- **CORS errors:** under proxy integration the **Lambda** returns CORS headers on **every** response including errors; the console "Enable CORS" button doesn't apply. HTTP API: use `CorsConfiguration`. CORS is browser-enforced — the API call itself succeeded. **Never use `AllowOrigin: '*'` in production** — wildcard CORS is a security risk (CWE-942); specify your exact domain (e.g. `https://app.example.com`).
-- **504 from API Gateway:** integration timeout (REST 29s default; HTTP 30s max), independent of Lambda's 15-min limit. For long ops: return 202 immediately, process via SQS/Step Functions, poll or use WebSocket.
-- **VPC Lambda can't reach internet:** needs **private** subnet + NAT Gateway in a **public** subnet. A public subnet does NOT give Lambda a public IP. For AWS services use VPC endpoints (free for S3/DynamoDB gateway endpoints).
-- **ImportModuleError / MODULE_NOT_FOUND:** handler path doesn't match file structure, or deps weren't bundled for the right platform. Python: `pip install -r requirements.txt -t ./package --platform manylinux2014_x86_64 --only-binary=:all:`. Use `sam build` for cross-platform packaging.
+### 502 Bad Gateway from API Gateway
+Lambda proxy integration requires `{ statusCode: int, headers: {}, body: "string" }`.
+The `body` must be a string (`JSON.stringify()`), not an object. API Gateway returns 502 when it cannot parse the Lambda response — the function ran successfully but the response shape was wrong. Note: string statusCode (e.g., "200") is silently coerced to integer, and missing statusCode defaults to 200.
+
+### CORS errors
+With Lambda proxy integration, Lambda must return CORS headers — the API Gateway console "Enable CORS" button does not work for Lambda proxy integration. Add `Access-Control-Allow-Origin`, `Access-Control-Allow-Methods`, `Access-Control-Allow-Headers` to every Lambda response including errors. For HTTP API, use the built-in `CorsConfiguration` instead. CORS is enforced by the browser, not the server — missing headers cause the browser to block the response even though the API call succeeded.
+
+### Lambda timeout + API Gateway 504
+API Gateway has a hard integration timeout: REST API default 29s (configurable 50ms–29s; Regional/private APIs can request higher), HTTP API max 30s (can be lowered, cannot be raised). This is independent of Lambda's 15-min limit. The 504 means API Gateway gave up waiting, not that Lambda failed. For long operations, return 202 immediately, process via SQS or Step Functions, poll or use WebSocket for results.
+
+### VPC Lambda cannot reach internet
+Lambda in a VPC needs a **private** subnet + NAT Gateway in a **public** subnet. Placing Lambda in a public subnet does NOT give it a public IP — Lambda never gets a public IP regardless of subnet type because Lambda's network interface is managed by the service and doesn't support public IP assignment. For AWS services only, use VPC endpoints (free for S3 and DynamoDB gateway endpoints).
+
+### ImportModuleError / MODULE_NOT_FOUND
+Handler path doesn't match file structure, or dependencies weren't bundled. Lambda extracts code to `/var/task` and layers to `/opt` — if the handler path doesn't match the file's location relative to `/var/task`, the runtime can't find it. Python: `pip install -r requirements.txt -t ./package --platform manylinux2014_x86_64 --only-binary=:all:`. Node: verify `exports.handler` exists and `node_modules` is included. Use `sam build` to handle cross-platform packaging automatically.
 
 ---
 
-## Lambda errors (gotchas)
+## Lambda Error Lookup
+
+### Runtime.ImportModuleError
+
+**Error:** `Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'lambda_function'`
+**Cause:** Handler references a module missing from the deployment package.
+
+```bash
+pip install -r requirements.txt -t ./package
+cd package && zip -r ../deployment.zip . && cd .. && zip deployment.zip lambda_function.py
+# Or: sam build && sam deploy
+```
+
+### Runtime.HandlerNotFound
+
+**Error:** `Runtime.HandlerNotFound: Handler 'handler' missing on module 'function'`
+**Cause:** File exists but function/method name doesn't match handler setting.
+
+```bash
+aws lambda update-function-configuration --function-name my-func --handler app.lambda_handler
+# Python: file.function  Node: file.export  Java: package.Class::method
+```
+
+### Task timed out
+
+**Error:** `Task timed out after 3.00 seconds`
+**Cause:** Execution exceeded configured timeout. Slow downstream calls, low memory/CPU, or VPC delays.
+
+```bash
+aws lambda update-function-configuration --function-name my-func --timeout 30
+aws lambda update-function-configuration --function-name my-func --memory-size 512
+# Set SDK/HTTP timeouts shorter than Lambda timeout for meaningful errors
+```
+
+### Runtime.OutOfMemory (OOM)
+
+**Error:** `Runtime.OutOfMemory: ... signal: killed` or `Runtime exited without providing a reason`
+**Cause:** Function exceeded allocated memory — kernel sent SIGKILL.
+
+```bash
+# Check REPORT lines: Max Memory Used vs Memory Size
+aws lambda update-function-configuration --function-name my-func --memory-size 1024
+# Stream large files instead of loading into memory; bound global caches
+```
+
+### AccessDeniedException
+
+**Error:** `AccessDeniedException: ... not authorized to perform: lambda:InvokeFunction`
+**Cause:** Calling IAM principal lacks `lambda:InvokeFunction` permission.
+
+```bash
+aws lambda add-permission --function-name my-func \
+  --statement-id AllowInvoke --action lambda:InvokeFunction \
+  --principal s3.amazonaws.com --source-arn arn:aws:s3:::my-bucket
+```
+
+### TooManyRequestsException
+
+**Error:** `TooManyRequestsException: Rate Exceeded.`
+**Cause:** Function exceeded account concurrency limit (default 1,000).
+
+```bash
+aws lambda get-account-settings
+aws service-quotas request-service-quota-increase \
+  --service-code lambda --quota-code L-B99A9384 --desired-value 3000
+aws lambda put-function-concurrency --function-name my-func --reserved-concurrent-executions 100
+```
+
+### InvalidParameterValueException (size)
+
+**Error:** `Unzipped size must be smaller than 262144000 bytes`
+**Cause:** Package exceeds 50 MB zipped / 250 MB unzipped.
+
+```bash
+find ./package -name "*.pyc" -delete && find ./package -name "*.dist-info" -type d -exec rm -rf {} +
+aws lambda publish-layer-version --layer-name my-deps --zip-file fileb://layer.zip --compatible-runtimes python3.13
+# Or upload via S3, or switch to container image packaging (10 GB limit)
+```
+
+### ETIMEDOUT (VPC)
+
+**Error:** `Error: connect ETIMEDOUT 176.32.98.189:443`
+**Cause:** VPC Lambda can't reach internet — missing NAT Gateway or VPC Endpoint.
+
+```bash
+aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=subnet-xxx"
+aws ec2 create-route --route-table-id rtb-xxx --destination-cidr-block 0.0.0.0/0 --nat-gateway-id nat-xxx
+# Or use VPC Endpoints for AWS services:
+aws ec2 create-vpc-endpoint --vpc-id vpc-xxx --service-name com.amazonaws.us-east-1.s3 --route-table-ids rtb-xxx
+```
+
+### MODULE_NOT_FOUND
+
+**Error:** `Error: Cannot find module 'my-module'`
+**Cause:** Node.js dependency missing — not bundled or built on incompatible platform.
+
+```bash
+npm install --production
+sam build --use-container  # for native modules
+unzip -l deployment.zip | grep my-module  # verify inclusion
+```
 
 ### RecursiveInvocationException
 
-A function writes to a resource that re-triggers it (Lambda halts after ~16 loops).
+**Error:** `RecursiveInvocationException: Recursive invocation detected`
+**Cause:** Function writes to a resource that triggers itself again (~16 invocations before halt).
 
 ```bash
-# Emergency stop:
+# Emergency stop
 aws lambda put-function-concurrency --function-name my-func --reserved-concurrent-executions 0
-# Fix: separate input/output buckets, or prefix filters in the trigger config
+# Fix: use separate input/output buckets or prefix filters in trigger config
 ```
 
-### SnapStart errors
+### SnapStart Errors
 
-`SnapStartException` / `SnapStartNotReadyException` / `SnapStartTimeoutException` — init threw, or it uses non-snapshottable resources (open network connections).
+**Error:** `SnapStartException` / `SnapStartNotReadyException` / `SnapStartTimeoutException`
+**Cause:** SnapStart failed during snapshot — init threw exception or uses non-snapshottable resources (e.g., open network connections).
 
 ```bash
 aws lambda get-function --function-name my-func --query 'Configuration.SnapStart'
-# Java:   CRaC hooks — beforeCheckpoint() closes connections, afterRestore() reopens
-# Python: snapshot_restore runtime hooks to re-establish connections after restore
-# .NET:   SnapshotRestore.Register hooks for before-snapshot / after-restore
+# Java: Use CRaC hooks — beforeCheckpoint() to close connections, afterRestore() to reopen
+# Python: Use snapshot_restore runtime hooks to re-establish connections after restore
+# .NET: Use SnapshotRestore register hooks for before-snapshot and after-restore actions
 ```
 
 ### Sandbox.Timedout
 
-Exceeded the timeout. In newer runtimes this covers **both** init-phase and invoke-phase timeouts — a suppressed init failure consumes the invoke timeout. Move heavy init to lazy loading inside the handler; raise `--timeout`/`--memory-size`.
+**Error:** `Sandbox.Timedout`
+**Cause:** Function exceeded its timeout. In newer runtimes, this covers both init-phase and invoke-phase timeouts. A suppressed init failure consumes the invoke timeout.
+
+```bash
+aws lambda update-function-configuration --function-name my-func --timeout 60 --memory-size 1024
+# Move heavy initialization to lazy loading inside the handler
+```
 
 ### ENILimitReachedException
 
-VPC hit its network-interface quota. Two limits can apply — the Lambda-specific Hyperplane ENI-per-VPC quota and the broader VPC ENI-per-Region quota — so check which one was reached (Service Quotas / `describe-account-attributes`).
+**Error:** `ENILimitReachedException`
+**Cause:** VPC reached network interface quota. Lambda Hyperplane ENIs have a default quota of 500 per VPC (see lambda.md); the overall VPC ENI quota is 5,000 per region. Check which limit applies.
 
 ```bash
 aws service-quotas request-service-quota-increase --service-code vpc --quota-code L-DF5E4CA3 --desired-value 10000
-# Consolidate functions onto the same subnet + security group combinations
+# Consolidate functions to use same subnet + security group combinations
 ```
 
-### TooManyRequestsException / throttling
+### InvalidZipFileException
+
+**Error:** `InvalidZipFileException: Could not unzip uploaded file.`
+**Cause:** Invalid ZIP or handler nested in subdirectory instead of at root.
 
 ```bash
-aws lambda get-account-settings
-aws service-quotas request-service-quota-increase --service-code lambda --quota-code L-B99A9384 --desired-value 3000
-aws lambda put-function-concurrency --function-name my-func --reserved-concurrent-executions 100
+unzip -t deployment.zip  # verify integrity
+cd my-folder && zip -r ../deployment.zip . && cd ..  # files at root, not nested
 ```
 
-Throttle behavior by invocation type: **Sync (API GW)** → 429 (may surface as 500); **Async (S3, SNS)** → auto-retries up to 6h; **SQS** → returns to queue, backs off; **Kinesis/DynamoDB Streams** → retries batch, blocks shard.
-
 ### CodeStorageExceededException
 
-Account exceeded its per-Region code storage quota (unzipped; all versions + layers). Check the current limit with `aws service-quotas get-service-quota --service-code lambda --quota-code L-2ACBD22F` (it is adjustable and has changed over time). Delete old versions and unused layers (`aws lambda list-versions-by-function`, `delete-function --qualifier`).
+**Error:** `CodeStorageExceededException: Code storage limit exceeded.`
+**Cause:** Account exceeded 75 GB code storage per region (all versions + layers).
+
+```bash
+aws lambda list-versions-by-function --function-name my-func
+aws lambda delete-function --function-name my-func --qualifier 1
+aws lambda list-layers  # delete unused layers too
+```
 
 ---
 
-## API Gateway errors (gotchas)
+## API Gateway Error Lookup
+
+### Malformed Lambda Proxy Response (502)
+
+**Error:** `Malformed Lambda proxy response` → 502
+**Cause:** Lambda response missing required format — `body` must be a string, response must be a JSON object (not a plain string or array).
+
+```python
+return {"statusCode": 200, "headers": {"Content-Type": "application/json"}, "body": json.dumps({"msg": "ok"})}
+```
+
+```javascript
+return { statusCode: 200, headers: { "Content-Type": "application/json" }, body: JSON.stringify({ msg: "ok" }) };
+```
+
+### Missing Authentication Token (403)
+
+**Error:** `403 Forbidden: Missing Authentication Token`
+**Cause:** URL doesn't match any resource/method, or API not deployed to stage. Usually routing, not auth.
+
+```bash
+aws apigateway create-deployment --rest-api-id abc123 --stage-name prod
+# Verify: https://{api-id}.execute-api.{region}.amazonaws.com/{stage}/{resource}
+```
+
+### Invalid Permissions on Lambda (500)
+
+**Error:** `Invalid permissions on Lambda function`
+**Cause:** API Gateway lacks `lambda:InvokeFunction` permission on the target function.
+
+```bash
+aws lambda add-permission --function-name my-func --statement-id apigw-invoke \
+  --action lambda:InvokeFunction --principal apigateway.amazonaws.com \
+  --source-arn "arn:aws:execute-api:us-east-1:123456789012:api-id/*/GET/resource"
+```
+
+### Endpoint Request Timed Out (504)
+
+**Error:** `Endpoint request timed out` → 504
+**Cause:** Lambda didn't respond within 29s (REST) / 30s (HTTP) integration timeout.
+
+```bash
+aws lambda update-function-configuration --function-name my-func --memory-size 1024
+# For long operations: return 202 immediately, process async, poll for results
+```
+
+### Authorizer Unauthorized (401)
+
+**Error:** `Unauthorized` (401)
+**Cause:** Lambda authorizer returned deny, threw error, or timed out.
+
+```bash
+aws logs tail /aws/lambda/my-authorizer --since 1h --filter-pattern ERROR
+# Verify authorizer returns: { principalId, policyDocument: { Statement: [{ Effect: "Allow" }] } }
+```
+
+### WAF Access Denied (403)
+
+**Error:** `403 Forbidden` with `x-amzn-errortype: ForbiddenException`
+**Cause:** AWS WAF rule matched — IP denylist, rate limit, or injection detection.
+
+```bash
+# Check WAF sampled requests in console to identify blocking rule
+# Test rules in Count mode before switching to Block
+```
 
-| Error | Cause | Fix |
-|---|---|---|
-| `403 Missing Authentication Token` | URL doesn't match a resource/method, or API not deployed | Usually routing, not auth — verify path and `create-deployment` to the stage |
-| `Invalid permissions on Lambda function` (500) | API GW lacks `lambda:InvokeFunction` | `aws lambda add-permission --principal apigateway.amazonaws.com --source-arn "arn:aws:execute-api:...:api-id/*/GET/resource"` (add `--source-account <account-id>` to scope cross-service triggers like S3 to your account) |
-| `Unauthorized` (401) | Lambda authorizer denied/errored/timed out | Check authorizer logs; verify it returns a valid Allow policy |
-| `403` + `x-amzn-errortype: ForbiddenException` | WAF rule matched | Check WAF sampled requests; test rules in Count mode first |
-| 500 with log `status 429` | Lambda throttled, surfaced as 500 | Increase Lambda concurrency |
+### CORS Errors
+
+**Error:** `blocked by CORS policy: No 'Access-Control-Allow-Origin' header`
+**Cause:** Lambda proxy integration must return CORS headers; HTTP APIs can configure at API level.
+
+```yaml
+# SAM Globals
+Globals:
+  Api:
+    Cors:
+      AllowOrigin: "'*'"
+      AllowMethods: "'GET,POST,OPTIONS'"
+      AllowHeaders: "'Content-Type,Authorization'"
+```
+
+```bash
+# HTTP API
+aws apigatewayv2 update-api --api-id abc123 \
+  --cors-configuration AllowOrigins="*",AllowMethods="GET,POST",AllowHeaders="Content-Type"
+```
+
+### Internal Server Error — Lambda Throttled (500)
+
+**Error:** 500 with CloudWatch log `Lambda invocation failed with status 429`
+**Cause:** Lambda throttled but API Gateway surfaces as 500.
+
+```bash
+# Increase Lambda concurrency (see TooManyRequestsException above)
+aws apigateway update-stage --rest-api-id abc123 --stage-name prod \
+  --patch-operations op=replace,path=/*/*/throttling/rateLimit,value=1000
+```
 
 ---
 
-## Step Functions errors
+## Step Functions Error Lookup
+
+### States.TaskFailed
+
+**Error:** `States.TaskFailed`
+**Cause:** Task failed — unhandled Lambda exception, service error, or missing permissions.
+
+```json
+"Retry": [{"ErrorEquals": ["States.TaskFailed","Lambda.ServiceException","Lambda.SdkClientException"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0}],
+"Catch": [{"ErrorEquals": ["States.TaskFailed"], "Next": "HandleError", "ResultPath": "$.error"}]
+```
+
+### States.Timeout
 
-| Error | Cause / Fix |
-|---|---|
-| `States.TaskFailed` | Add `Retry` for `States.TaskFailed`/`Lambda.ServiceException`/`Lambda.SdkClientException`; `Catch` to a handler |
-| `States.Timeout` | Set `TimeoutSeconds` (and `HeartbeatSeconds` for long tasks) |
-| `States.DataLimitExceeded` | Input/output > 256 KiB — **cannot be caught by `States.ALL`**. Store in S3, pass keys |
-| `ExecutionAlreadyExists` | Names unique per state machine for 90 days — use `--name "exec-$(date +%s)"` or omit `--name` |
-| `States.Permissions` | Execution role lacks target-service permission |
+**Error:** `States.Timeout`
+**Cause:** Task exceeded `TimeoutSeconds` or missed `HeartbeatSeconds` deadline.
+
+```json
+{"Type": "Task", "Resource": "arn:aws:lambda:...", "TimeoutSeconds": 300, "HeartbeatSeconds": 60, "Next": "NextState"}
+```
+
+### States.DataLimitExceeded
+
+**Error:** `States.DataLimitExceeded`
+**Cause:** State input/output exceeded 256 KB. Cannot be caught by `States.ALL`.
+
+**Fix:** Store large data in S3, pass only S3 keys between states. Use `InputPath`/`OutputPath` to filter.
+
+### ExecutionAlreadyExists
+
+**Error:** `ExecutionAlreadyExists`
+**Cause:** Execution name must be unique per state machine for 90 days.
+
+```bash
+aws stepfunctions start-execution --state-machine-arn arn:aws:states:... \
+  --name "exec-$(date +%s)" --input '{}'
+# Or omit --name for auto-generated names
+```
+
+### States.Permissions
+
+**Error:** `States.Permissions: insufficient privileges`
+**Cause:** Execution role lacks permission to invoke target service.
+
+```bash
+aws iam list-attached-role-policies --role-name StepFunctionsRole
+# Add lambda:InvokeFunction, dynamodb:PutItem, etc. to the execution role
+```
 
 ---
 
-## SAM/CDK errors
+## SAM/CDK Error Lookup
 
-### Stale build cache
+### Stale Build Cache
+
+**Error:** `sam build` uses old dependencies after updating requirements.txt, or `--clear-cache` flag unrecognized.
+**Cause:** SAM caches build artifacts. There is no `--clear-cache` flag.
 
 ```bash
-sam build --no-cached        # correct flag — there is NO --clear-cache
-rm -rf .aws-sam/cache        # or delete the cache directory
+sam build --no-cached              # Force clean build (correct flag)
+rm -rf .aws-sam/cache              # Or manually delete cache directory
 ```
 
 ### PythonPipBuilder:ResolveDependencies
 
-Version conflicts or missing native libs: `sam build --use-container --no-cached`; use binary wheels (`psycopg2-binary`).
+**Error:** `PythonPipBuilder:ResolveDependencies - pip install returned a non-zero exit code`
+**Cause:** Dependency version conflicts or missing native libraries.
+
+```bash
+sam build --use-container --no-cached
+# Use binary wheels: psycopg2-binary instead of psycopg2
+```
+
+### DockerBuildFailed
+
+**Error:** `DockerBuildFailed: Docker build failed.`
+**Cause:** Docker not running or Dockerfile errors.
+
+```bash
+docker info  # verify running
+sudo systemctl start docker  # start if needed
+```
+
+### Cannot find module 'esbuild'
 
-### CDK NodejsFunction: `Cannot find module 'esbuild'`
+**Error:** `Cannot find module 'esbuild'`
+**Cause:** CDK `NodejsFunction` needs esbuild for bundling.
+
+```bash
+npm install --save-dev esbuild
+```
 
-`npm install --save-dev esbuild`.
+### CREATE_FAILED
 
-### CREATE_FAILED / UPDATE_ROLLBACK_FAILED
+**Error:** `CREATE_FAILED: AWS::Lambda::Function`
+**Cause:** Invalid runtime, missing S3 code, role not ready, or package too large.
 
 ```bash
 aws cloudformation describe-stack-events --stack-name my-stack \
   --query "StackEvents[?ResourceStatus=='CREATE_FAILED'].[LogicalResourceId,ResourceStatusReason]" --output table
-# Rollback also failed (resource manually deleted / perms changed):
+```
+
+### UPDATE_ROLLBACK_FAILED
+
+**Error:** `UPDATE_ROLLBACK_FAILED`
+**Cause:** Update failed and rollback also failed — resource manually deleted or permissions changed.
+
+```bash
+aws cloudformation continue-update-rollback --stack-name my-stack
 aws cloudformation continue-update-rollback --stack-name my-stack --resources-to-skip MyFunction
 ```
 
-### Circular dependency
+### Security Constraints Not Satisfied
+
+**Error:** `Security Constraints Not Satisfied`
+**Cause:** SAM template missing required properties (Handler, Runtime, CodeUri).
+
+```bash
+sam validate --lint
+```
+
+### CDK Bootstrap Required
 
-`Circular dependency between resources: [MyFunction, MyRole, ...]` — break the cycle by giving the function an explicit `FunctionName` and referencing the hardcoded ARN (`!Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:my-function-name"`) instead of `!Ref`/`!GetAtt`, or extract the IAM role/policy into a separate resource.
+**Error:** `This stack uses assets, so the toolkit stack must be deployed`
+**Cause:** Target account/region not bootstrapped.
 
-### Other quick ones
+```bash
+cdk bootstrap aws://123456789012/us-east-1
+```
 
-- `DockerBuildFailed` → `docker info` / start Docker.
-- `Security Constraints Not Satisfied` (missing Handler/Runtime/CodeUri) → `sam validate --lint`.
-- `This stack uses assets, so the toolkit stack must be deployed` → `cdk bootstrap aws://ACCOUNT/REGION`.
+### Circular Dependency
+
+**Error:** `Circular dependency between resources: [MyFunction, MyRole, ...]`
+**Cause:** Resources reference each other in a cycle.
+
+```yaml
+# Break cycle: give the function an explicit name and hardcode the ARN
+MyFunction:
+  Type: AWS::Lambda::Function
+  Properties:
+    FunctionName: my-function-name  # explicit name
+
+MyRole:
+  Type: AWS::IAM::Role
+  Properties:
+    Policies:
+      - PolicyDocument:
+          Statement:
+            - Effect: Allow
+              Action: lambda:InvokeFunction
+              # No ${MyFunction} reference — no implicit dependency
+              Resource: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:my-function-name"
+# Or restructure to eliminate the cycle (extract IAM role/policy into a separate resource)
+```
 
 ---
 
-## Diagnostics
+## Timeout Debugging
+
+```
+Function times out
+├── INIT phase? (Sandbox.Timedout)
+│   ├── YES → Increase timeout + memory, lazy-load heavy deps
+│   └── NO → INVOKE phase
+│       ├── Timeout ≈ avg duration? → Set to 2-3x average
+│       ├── Calling external services? → Set SDK timeouts < Lambda timeout
+│       ├── CPU-bound? → Increase memory (1,769 MB = 1 vCPU)
+│       └── VPC? → Check NAT Gateway / security group / VPC Endpoints
+```
+
+```bash
+aws lambda get-function-configuration --function-name my-func --query '[Timeout,MemorySize]'
+aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration \
+  --dimensions Name=FunctionName,Value=my-func --period 300 --statistics Average Maximum \
+  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S)
+# For percentiles, use a separate call:
+aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration \
+  --dimensions Name=FunctionName,Value=my-func --period 300 --extended-statistics p99 \
+  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S)
+```
+
+---
+
+## Out-of-Memory (OOM) Debugging
+
+```
+Runtime.OutOfMemory / signal: killed
+├── Check REPORT: Max Memory Used ≈ Memory Size? → OOM confirmed
+├── Immediate? → Payload/dependency too large → increase memory
+├── Gradual? → Memory leak → check global vars accumulating across warm invocations
+└── Fix: increase memory, stream large files, bound caches
+```
+
+| Memory (MB) | vCPUs | Use Case |
+|-------------|-------|----------|
+| 128 | ~0.08 | Simple transforms |
+| 512 | ~0.3 | Moderate processing |
+| 1,769 | 1.0 | CPU-intensive single-threaded |
+| 3,538 | 2.0 | Multi-threaded |
+| 10,240 | ~5.8 | Heavy compute, ML inference |
+
+---
+
+## Throttling Diagnosis
+
+| Concept | Default | Notes |
+|---------|---------|-------|
+| Account concurrency | 1,000/region | Request increase via Service Quotas |
+| Reserved concurrency | None | Guarantees AND caps function concurrency |
+| Concurrency scaling rate | 1,000 envs/10s | Per function, uniform across regions |
+
+| Invocation Type | Throttle Behavior |
+|-----------------|-------------------|
+| Synchronous (API GW) | Returns 429 (API GW may show 500) |
+| Async (S3, SNS) | Auto-retries up to 6 hours |
+| SQS trigger | Returns to queue, backs off |
+| Kinesis/DDB Streams | Retries batch, blocks shard |
+
+```bash
+aws lambda get-account-settings
+aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Throttles \
+  --dimensions Name=FunctionName,Value=my-func --period 60 --statistics Sum \
+  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S)
+```
+
+---
+
+## CloudWatch Logs Insights Queries
+
+Run against `/aws/lambda/FUNCTION_NAME`. For API Gateway, use the access log group.
+
+### Cold Starts
+
+```
+filter @type = "REPORT" | filter ispresent(@initDuration)
+| stats count() as coldStarts, avg(@initDuration) as avgInitMs, max(@initDuration) as maxInitMs, pct(@initDuration, 99) as p99InitMs by bin(1h)
+```
+
+### Cold Start Percentage
+
+```
+filter @type = "REPORT"
+| stats count() as total, sum(ispresent(@initDuration)) as coldStarts, sum(ispresent(@initDuration)) * 100.0 / count() as pct by bin(1h)
+```
+
+### Errors by Type
+
+```
+filter @message like /(?i)error|exception/
+| parse @message /(?<errorType>[A-Za-z]+Error|[A-Za-z]+Exception)/
+| stats count() as cnt by errorType | sort cnt desc
+```
+
+### Timeouts
+
+```
+filter @message like /Task timed out/ | stats count() as timeouts by bin(1h) | sort bin desc
+```
+
+### Memory Utilization
+
+```
+filter @type = "REPORT"
+| stats max(@memorySize/1e6) as provisionedMB, avg(@maxMemoryUsed/1e6) as avgUsedMB, max(@maxMemoryUsed/1e6) as maxUsedMB, pct(@maxMemoryUsed/1e6, 99) as p99UsedMB
+```
+
+### Out-of-Memory Detection (>90% memory)
+
+```
+filter @type = "REPORT" | filter @maxMemoryUsed / @memorySize > 0.9
+| fields @timestamp, @requestId, @maxMemoryUsed/1e6 as usedMB, @memorySize/1e6 as allocatedMB | sort @timestamp desc | limit 50
+```
+
+### Overprovisioned Memory (<50% used)
+
+```
+filter @type = "REPORT"
+| stats max(@memorySize/1e6) as provMB, max(@maxMemoryUsed/1e6) as peakMB, max(@maxMemoryUsed)*100.0/max(@memorySize) as pct
+| filter pct < 50
+```
+
+### Memory Growth (Leak Detection)
 
-For systematic timeout investigation (config → logs → metrics → VPC → cold start → memory → downstream), use the **debugging-lambda-timeouts** skill (see SKILL.md routing).
+```
+filter @type = "REPORT" | stats avg(@maxMemoryUsed/1e6) as avgMemMB by bin(5m) | sort bin asc
+```
 
-CloudWatch Logs Insights run against `/aws/lambda/FUNCTION_NAME` (API Gateway uses the access log group). Useful fields on `@type = "REPORT"` lines: `@initDuration` (cold start), `@duration`, `@billedDuration`, `@maxMemoryUsed`, `@memorySize`. Examples:
+### Latency Percentiles
 
 ```
-# Cold start rate + init latency
-filter @type="REPORT" | stats count() as total, sum(ispresent(@initDuration)) as coldStarts,
-  avg(@initDuration) as avgInitMs, pct(@initDuration,99) as p99InitMs by bin(1h)
+filter @type = "REPORT"
+| stats avg(@duration) as avg, pct(@duration,50) as p50, pct(@duration,90) as p90, pct(@duration,95) as p95, pct(@duration,99) as p99, max(@duration) as max by bin(1h)
+```
+
+### Slowest Invocations
 
-# Out-of-memory (>90% used)
-filter @type="REPORT" | filter @maxMemoryUsed/@memorySize > 0.9
-  | fields @timestamp, @requestId, @maxMemoryUsed/1e6 as usedMB | sort @timestamp desc
+```
+filter @type = "REPORT"
+| fields @timestamp, @requestId, @duration, @maxMemoryUsed/1000000 as memMB, ispresent(@initDuration) as coldStart
+| sort @duration desc | limit 20
+```
 
-# Latency percentiles
-filter @type="REPORT" | stats pct(@duration,50) as p50, pct(@duration,90) as p90,
-  pct(@duration,99) as p99, max(@duration) as max by bin(1h)
+### API Gateway 5xx
 
-# API Gateway 5xx (access log group)
+```
 filter status >= 500 | stats count() as errors by status, path, httpMethod | sort errors desc
 ```
 
-### Memory ↔ vCPU (for OOM / CPU tuning)
+### API Gateway 5xx Over Time
+
+```
+filter status >= 500 | stats count() by bin(5m) | sort bin desc
+```
+
+### Throttle Events
 
-| Memory | vCPUs | Use case |
-|---|---|---|
-| 128 MB | ~0.08 | Simple transforms |
-| 512 MB | ~0.3 | Moderate processing |
-| 1,769 MB | 1.0 | CPU-intensive single-threaded |
-| 3,538 MB | 2.0 | Multi-threaded |
-| 10,240 MB | ~5.8 (up to 6 vCPU) | Heavy compute / ML inference |
+```
+filter @message like /Rate Exceeded|TooManyRequestsException|Throttl/
+| fields @timestamp, @requestId, @message | sort @timestamp desc | limit 50
+```
 
-### X-Ray
+### Billed Duration
 
-Enable via SAM `Tracing: Active` (Function) / `TracingEnabled: true` (Api) or CDK `tracing: lambda.Tracing.ACTIVE` (adds `AWSXRayDaemonWriteAccess` automatically). Default sampling: 1 req/s reservoir + 5%. Instrument SDK calls with `patch_all()` (Python) / `captureAWSv3Client` (Node).
+```
+filter @type = "REPORT"
+| stats count() as invocations, sum(@billedDuration)/1000 as totalBilledSec, avg(@billedDuration) as avgBilledMs by bin(1d)
+```
+
+### Error Messages with Request IDs
+
+```
+filter @message like /(?i)error|exception|fail/
+| fields @timestamp, @requestId, @message | sort @timestamp desc | limit 50
+```
+
+---
+
+## X-Ray Tracing
+
+### Enable in SAM
+
+```yaml
+Globals:
+  Function:
+    Tracing: Active
+```
+
+### Enable in CDK
+
+```typescript
+new lambda.Function(this, 'Fn', {
+  tracing: lambda.Tracing.ACTIVE,  // adds AWSXRayDaemonWriteAccess automatically
+});
+```
+
+### Required IAM
+`AWSXRayDaemonWriteAccess` managed policy on the execution role. SAM/CDK add this automatically.
+
+### Default Sampling
+1 request/second (reservoir) + 5% of additional requests.
+
+### Instrument SDK Calls
+
+```python
+from aws_xray_sdk.core import patch_all
+patch_all()
+```
+
+```javascript
+// SDK v3 (Node.js 18+)
+const { captureAWSv3Client } = require('aws-xray-sdk-core');
+const { DynamoDBClient } = require('@aws-sdk/client-dynamodb');
+const ddb = captureAWSv3Client(new DynamoDBClient({}));
+```
+
+### Query Traces
+
+```bash
+aws xray get-trace-summaries --start-time $(date -u -d '1 hour ago' +%s) --end-time $(date -u +%s) \
+  --filter-expression 'service("my-func") AND fault'
+aws xray batch-get-traces --trace-ids "1-xxx-yyy"
+```
+
+### Enable for API Gateway
+
+```yaml
+Resources:
+  MyApi:
+    Type: AWS::Serverless::Api
+    Properties:
+      StageName: prod
+      TracingEnabled: true
+```
+
+### Enable for Step Functions
+
+```yaml
+Resources:
+  MyStateMachine:
+    Type: AWS::Serverless::StateMachine
+    Properties:
+      Tracing:
+        Enabled: true
+```