@decocms/start 0.19.0

package/.cursor/skills/deco-site-memory-debugging/cdp-connection.md

@@ -0,0 +1,222 @@
+# CDP Connection Guide
+
+How to connect to a Deno pod's Chrome DevTools Protocol inspector for memory debugging.
+
+## Prerequisites
+
+- `kubectl` configured with cluster access
+- Python 3 with `websockets` package (`pip3 install websockets`)
+- Pod must be running Deno with inspector enabled (Deco sites expose port 9229 by default)
+
+## Step 1: Identify the Pod
+
+```bash
+# List pods in the site namespace
+kubectl get pods -n sites-<sitename> -o wide
+
+# Check current memory usage
+kubectl top pods -n sites-<sitename>
+```
+
+## Step 2: Port-Forward
+
+```bash
+# Forward local 9229 to pod's inspector port
+kubectl port-forward -n sites-<sitename> <pod-name> 9229:9229
+
+# If port 9229 is busy, use a different local port
+kubectl port-forward -n sites-<sitename> <pod-name> 19229:9229
+```
+
+**Keep this running in a separate terminal.** Port-forwards drop frequently — check if it's alive before running scripts.
+
+## Step 3: Get WebSocket URL
+
+```bash
+curl -s http://127.0.0.1:9229/json/list | jq '.[0].webSocketDebuggerUrl'
+```
+
+This returns something like:
+```
+ws://127.0.0.1:9229/ws/b9cf0f05-6e67-4ad6-865f-f418f6b4856c
+```
+
+**The UUID changes every time the pod restarts.** Always fetch a fresh URL.
+
+## Step 4: Connect via Python
+
+```python
+import asyncio, json, websockets
+
+WS = "ws://127.0.0.1:9229/ws/<UUID>"
+MSG_ID = 0
+
+async def send_cmd(ws, method, params=None):
+    global MSG_ID
+    MSG_ID += 1
+    msg = {"id": MSG_ID, "method": method}
+    if params:
+        msg["params"] = params
+    await ws.send(json.dumps(msg))
+    for _ in range(10000):
+        raw = await asyncio.wait_for(ws.recv(), timeout=30)
+        data = json.loads(raw)
+        if data.get("id") == MSG_ID:
+            return data
+    return None
+
+async def evaluate(ws, expr):
+    r = await send_cmd(ws, "Runtime.evaluate", {
+        "expression": expr,
+        "contextId": 1,  # IMPORTANT: always use contextId: 1
+        "returnByValue": True,
+        "awaitPromise": True,
+        "timeout": 30000,
+    })
+    if r and "result" in r and "result" in r["result"]:
+        return r["result"]["result"].get("value")
+    return None
+
+async def main():
+    async with websockets.connect(WS, max_size=50*1024*1024) as ws:
+        await send_cmd(ws, "Runtime.enable")
+        # Your analysis code here...
+
+asyncio.run(main())
+```
+
+## Common Mistakes and Pitfalls
+
+### 1. Missing `contextId: 1`
+
+**Symptom:** `Runtime.evaluate` returns empty results or "Cannot find default execution context".
+
+**Cause:** After `Runtime.enable`, Deno emits thousands of `Runtime.consoleAPICalled` events that flood the event loop. Without explicit `contextId`, the evaluation may target the wrong context.
+
+**Fix:** Always pass `contextId: 1` in every `Runtime.evaluate` call:
+```python
+await send_cmd(ws, "Runtime.evaluate", {
+    "expression": expr,
+    "contextId": 1,  # <-- THIS IS REQUIRED
+    ...
+})
+```
+
+### 2. queryObjects returns under `objects`, not `result`
+
+**Symptom:** `KeyError: 'result'` when accessing `queryObjects` response.
+
+**Cause:** Deno's V8 inspector returns the array under `result.objects`, not `result.result` like Chrome does.
+
+**Fix:**
+```python
+# WRONG (Chrome-style):
+arr_id = qr["result"]["result"].get("objectId")
+
+# CORRECT (Deno-style):
+arr_id = qr["result"].get("objects", {}).get("objectId")
+```
+
+Full pattern:
+```python
+async def query_objects(ws, proto_expr):
+    proto_r = await send_cmd(ws, "Runtime.evaluate", {
+        "expression": proto_expr,
+        "contextId": 1,
+    })
+    proto_id = proto_r["result"].get("result", {}).get("objectId")
+    if not proto_id:
+        return None
+    qr = await send_cmd(ws, "Runtime.queryObjects", {"prototypeObjectId": proto_id})
+    if not qr or "result" not in qr:
+        return None
+    return qr["result"].get("objects", {}).get("objectId")
+```
+
+### 3. Event flooding drowns responses
+
+**Symptom:** `send_cmd` never receives the response, times out.
+
+**Cause:** `Runtime.enable` triggers a flood of `Runtime.consoleAPICalled` events (can be thousands). The event drain loop in `send_cmd` may exhaust its iteration limit before finding the response.
+
+**Fix:** Increase the drain limit:
+```python
+for _ in range(10000):  # Use 10000, not 100
+    raw = await asyncio.wait_for(ws.recv(), timeout=30)
+    data = json.loads(raw)
+    if data.get("id") == MSG_ID:
+        return data
+```
+
+### 4. Port-forward drops silently
+
+**Symptom:** `ConnectionClosedError: no close frame` or `ConnectionRefusedError`.
+
+**Cause:** `kubectl port-forward` drops connections under load or after inactivity.
+
+**Fix:**
+1. Check if port-forward process is still running
+2. Re-establish port-forward
+3. Get a fresh WebSocket URL (same pod, new connection)
+4. Script should handle reconnection gracefully
+
+### 5. WebSocket message too large
+
+**Symptom:** `PayloadTooBig` error when taking heap snapshots.
+
+**Fix:** Increase `max_size` on the WebSocket connection:
+```python
+async with websockets.connect(WS, max_size=100*1024*1024) as ws:
+```
+
+### 6. Deno 2.x API changes
+
+Some APIs changed in Deno 2.x:
+- `Deno.resources()` → **removed** in Deno 2.x. Use `/proc/self/fd` instead.
+- `caches.keys()` → may not be available as a function. Use `caches.open(name)` and `cache.keys()` instead.
+
+### 7. /proc access requires permissions
+
+**Symptom:** "Requires all access" error when reading `/proc/self/status` or `/proc/self/maps`.
+
+**Cause:** Deno's permission system blocks filesystem reads outside allowed paths.
+
+**Workaround:** Use `Deno.memoryUsage()` instead — it doesn't require extra permissions and gives RSS, heap, and external memory.
+
+### 8. Heap snapshot node_fields empty
+
+**Symptom:** Heap snapshot `snapshot.node_fields` is an empty array.
+
+**Cause:** Deno/V8 stores `node_fields` under `snapshot.meta` or uses a different layout than Chrome.
+
+**Fix:** Infer field count from `len(nodes) / node_count`. Standard V8 uses 6 or 7 fields per node:
+```python
+field_count = len(nodes) // node_count
+# 6 fields: type, name, id, self_size, edge_count, trace_node_id
+# 7 fields: + detachedness
+```
+
+### 9. callFunctionOn for object analysis
+
+After `queryObjects`, use `callFunctionOn` to analyze the returned array:
+```python
+async def call_on(ws, obj_id, func):
+    r = await send_cmd(ws, "Runtime.callFunctionOn", {
+        "objectId": obj_id,
+        "functionDeclaration": func,
+        "returnByValue": True,
+    })
+    if not r or "result" not in r:
+        return None
+    return r["result"].get("result", {}).get("value")
+
+# Example: count Response objects and check bodyUsed
+body_info = await call_on(ws, resp_array_id, """function() {
+    let used = 0, notUsed = 0;
+    for (let i = 0; i < this.length; i++) {
+        if (this[i].bodyUsed) used++;
+        else notUsed++;
+    }
+    return JSON.stringify({used, notUsed});
+}""")
+```

package/.cursor/skills/deco-site-memory-debugging/memory-analysis.md

@@ -0,0 +1,362 @@
+# Memory Analysis Procedures
+
+Step-by-step procedures for analyzing memory in Deno pods via CDP.
+
+## Procedure 1: Quick Memory Check
+
+**Goal:** Determine if memory is a leak or lazy GC. Takes 2 minutes.
+
+```python
+# 1. Get memory BEFORE GC
+mem_before = await evaluate(ws, "JSON.stringify(Deno.memoryUsage())")
+
+# 2. Force GC
+await send_cmd(ws, "HeapProfiler.collectGarbage")
+await asyncio.sleep(0.5)
+await send_cmd(ws, "HeapProfiler.collectGarbage")  # twice for thoroughness
+
+# 3. Get memory AFTER GC
+mem_after = await evaluate(ws, "JSON.stringify(Deno.memoryUsage())")
+```
+
+**Interpretation:**
+- RSS drops >30%? → Lazy GC, not a leak. **Recommend reducing `--max-old-space-size`.**
+- RSS drops <10%? → Real retained memory. Continue to Procedure 2.
+
+**Recommendation for lazy GC:**
+If most memory is reclaimable by GC, the pod doesn't have a leak — V8 is just being lazy about collecting garbage. Reduce the max old space size so GC triggers more frequently:
+
+```bash
+# In the deployment or Deno flags:
+--v8-flags=--max-old-space-size=512
+# or even 256 for small sites
+```
+
+This keeps RSS predictable without affecting performance. V8's incremental GC is fast (typically <10ms pauses) so more frequent runs have negligible impact on request latency.
+
+## Procedure 2: Object Leak Detection
+
+**Goal:** Find leaked Response/Request objects and unconsumed bodies.
+
+### Check Response Objects
+
+```python
+resp_id = await query_objects(ws, "Response.prototype")
+if resp_id:
+    info = await call_on(ws, resp_id, """function() {
+        let used = 0, notUsed = 0;
+        const leakedUrls = [];
+        for (let i = 0; i < this.length; i++) {
+            if (this[i].bodyUsed) used++;
+            else {
+                notUsed++;
+                if (leakedUrls.length < 20)
+                    leakedUrls.push({
+                        url: this[i].url.substring(0, 120),
+                        status: this[i].status
+                    });
+            }
+        }
+        return JSON.stringify({total: this.length, used, notUsed, leakedUrls});
+    }""")
+```
+
+**Interpretation:**
+- `notUsed` < 5? → Normal (in-flight requests)
+- `notUsed` > 50? → **Response body leak.** Bodies are fetched but never consumed (`.text()`, `.json()`, `.arrayBuffer()`, or `.body.cancel()`).
+- Check the URLs to identify which code path is leaking
+
+### Check Request Objects
+
+```python
+req_id = await query_objects(ws, "Request.prototype")
+if req_id:
+    info = await call_on(ws, req_id, """function() {
+        const hosts = {};
+        for (let i = 0; i < this.length; i++) {
+            try {
+                const h = new URL(this[i].url).host;
+                hosts[h] = (hosts[h] || 0) + 1;
+            } catch(e) {}
+        }
+        return JSON.stringify({total: this.length, hosts});
+    }""")
+```
+
+**Interpretation:**
+- Hundreds of Request objects to the same host → possible fetch loop or unbounded cache
+- `localhost` requests → SSR self-fetches (normal for Fresh)
+
+## Procedure 3: ArrayBuffer Analysis
+
+**Goal:** Identify large memory consumers in ArrayBuffers.
+
+```python
+ab_id = await query_objects(ws, "ArrayBuffer.prototype")
+if ab_id:
+    info = await call_on(ws, ab_id, """function() {
+        let totalBytes = 0;
+        const buckets = {
+            '0-1KB': 0, '1-10KB': 0, '10-100KB': 0,
+            '100KB-1MB': 0, '1-10MB': 0, '10MB+': 0
+        };
+        const large = [];
+        for (let i = 0; i < this.length; i++) {
+            const sz = this[i].byteLength;
+            totalBytes += sz;
+            if (sz < 1024) buckets['0-1KB']++;
+            else if (sz < 10240) buckets['1-10KB']++;
+            else if (sz < 102400) buckets['10-100KB']++;
+            else if (sz < 1048576) buckets['100KB-1MB']++;
+            else if (sz < 10485760) buckets['1-10MB']++;
+            else buckets['10MB+']++;
+
+            if (sz > 100000 && large.length < 20) {
+                try {
+                    const preview = new TextDecoder().decode(
+                        new Uint8Array(this[i], 0, Math.min(200, sz))
+                    );
+                    large.push({sizeMB: sz/1024/1024, preview});
+                } catch(e) {
+                    large.push({sizeMB: sz/1024/1024, preview: '(binary)'});
+                }
+            }
+        }
+        return JSON.stringify({
+            count: this.length,
+            totalMB: totalBytes/1024/1024,
+            buckets,
+            large
+        });
+    }""")
+```
+
+**Known ArrayBuffer patterns:**
+- **~304 MB static buffer** — V8/ICU internal data. Always present. Ignore it.
+- **`resourceMetrics` JSON buffers (0.3-0.6 MB each)** — OpenTelemetry export batches accumulating. Minor but grows over time.
+- **Large JSON buffers (>1 MB)** — ProductListingPage or similar API responses. If appearing in PAIRS, might indicate response body read + original buffer both retained.
+- **`data:application/json;base64,...`** — Source maps. Normal, proportional to loaded modules.
+- **`<!DOCTYPE html>...`** — Rendered HTML pages. If many, SSR cache might be unbounded.
+
+## Procedure 4: Heap Snapshot
+
+**Goal:** Get a comprehensive view of all heap objects.
+
+```python
+await send_cmd(ws, "HeapProfiler.enable")
+
+MSG_ID += 1
+snap_id = MSG_ID
+await ws.send(json.dumps({
+    "id": snap_id,
+    "method": "HeapProfiler.takeHeapSnapshot",
+    "params": {"reportProgress": False, "treatGlobalObjectsAsRoots": True}
+}))
+
+chunks = []
+for _ in range(200000):
+    raw = await asyncio.wait_for(ws.recv(), timeout=120)
+    data = json.loads(raw)
+    if data.get("method") == "HeapProfiler.addHeapSnapshotChunk":
+        chunks.append(data["params"]["chunk"])
+    elif data.get("id") == snap_id:
+        break
+
+snapshot = json.loads("".join(chunks))
+```
+
+**Parsing the snapshot:**
+
+```python
+snap_meta = snapshot.get("snapshot", {})
+node_count = snap_meta.get("node_count", 0)
+nodes = snapshot.get("nodes", [])
+strings = snapshot.get("strings", [])
+
+# Infer field count (node_fields may be empty in Deno)
+field_count = len(nodes) // node_count  # typically 6 or 7
+
+# V8 node type indices (standard order):
+# 0=hidden, 1=array, 2=string, 3=object, 4=code,
+# 5=closure, 6=regexp, 7=number, 8=native,
+# 9=synthetic, 10=concatenated string, 11=sliced string,
+# 12=symbol, 13=bigint, 14=object shape
+
+# Aggregate by type
+type_agg = {}
+for i in range(0, node_count * field_count, field_count):
+    node_type = nodes[i]      # index 0 = type
+    name_idx = nodes[i + 1]   # index 1 = name (string table index)
+    self_size = nodes[i + 3]  # index 3 = self_size
+    # aggregate...
+```
+
+**What to look for in the snapshot:**
+- `string` type >100 MB → HTML pages or JSON cached in memory
+- `native` (type 8) → ArrayBuffers (cross-reference with Procedure 3)
+- `closure` count very high → possible listener/callback leak
+- `object` with specific names → identify which data structures hold memory
+
+## Procedure 5: Additional Checks
+
+### Open File Descriptors
+
+```python
+fds = await evaluate(ws, """
+(async () => {
+    try {
+        let count = 0;
+        const types = {socket: 0, pipe: 0, file: 0, other: 0};
+        for await (const entry of Deno.readDir('/proc/self/fd')) {
+            count++;
+            try {
+                const link = await Deno.readLink('/proc/self/fd/' + entry.name);
+                if (link.startsWith('socket:')) types.socket++;
+                else if (link.startsWith('pipe:')) types.pipe++;
+                else if (link.startsWith('/')) types.file++;
+                else types.other++;
+            } catch(e) { types.other++; }
+        }
+        return JSON.stringify({count, types});
+    } catch(e) { return JSON.stringify({error: e.message}); }
+})()
+""")
+```
+
+- 50-100 FDs → normal
+- 500+ FDs → possible connection leak or file handle leak
+
+### Map/Set Objects (potential caches)
+
+```python
+map_id = await query_objects(ws, "Map.prototype")
+if map_id:
+    info = await call_on(ws, map_id, """function() {
+        const large = [];
+        for (let i = 0; i < this.length; i++) {
+            if (this[i].size > 10) {
+                let keys = [];
+                let j = 0;
+                for (const k of this[i].keys()) {
+                    if (j++ >= 3) break;
+                    keys.push(String(k).substring(0, 80));
+                }
+                large.push({size: this[i].size, keys});
+            }
+        }
+        large.sort((a,b) => b.size - a.size);
+        return JSON.stringify({total: this.length, large: large.slice(0, 15)});
+    }""")
+```
+
+### Deno Module Cache Size
+
+```python
+cache_info = await evaluate(ws, """
+(async () => {
+    const denoDir = Deno.env.get('DENO_DIR') || '/app/deco/deno_dir';
+    let count = 0, totalSize = 0;
+    async function walk(dir, depth) {
+        if (depth > 3) return;
+        try {
+            for await (const entry of Deno.readDir(dir)) {
+                if (entry.isDirectory) await walk(dir + '/' + entry.name, depth + 1);
+                else {
+                    count++;
+                    try {
+                        const s = await Deno.stat(dir + '/' + entry.name);
+                        totalSize += s.size;
+                    } catch(e) {}
+                }
+            }
+        } catch(e) {}
+    }
+    await walk(denoDir, 0);
+    return JSON.stringify({denoDir, files: count, sizeMB: totalSize/1024/1024});
+})()
+""")
+```
+
+### Deno Version
+
+```python
+ver = await evaluate(ws, """
+JSON.stringify({
+    deno: Deno.version,
+    pid: Deno.pid,
+    hostname: Deno.hostname(),
+})
+""")
+```
+
+## Procedure 6: LRU Cache Inspection
+
+**Important context:** Deco's LRU cache stores `true` (a boolean) as the value — it's a metadata index for the filesystem cache, NOT storing response bodies in memory. The `calculatedSize` is the sum of tracked Content-Length values (metadata tracking), not actual memory consumption.
+
+```python
+lru = await evaluate(ws, """
+(() => {
+    const results = [];
+    function findLRU(obj, path, depth) {
+        if (depth > 3 || !obj) return;
+        try {
+            for (const key of Object.keys(obj)) {
+                try {
+                    const val = obj[key];
+                    if (val && typeof val === 'object' &&
+                        typeof val.max === 'number' &&
+                        typeof val.size === 'number' &&
+                        typeof val.calculatedSize === 'number') {
+                        results.push({
+                            path: path + '.' + key,
+                            size: val.size,
+                            calcSizeMB: val.calculatedSize / 1024 / 1024,
+                            max: val.max,
+                            maxSizeMB: val.maxSize ? val.maxSize / 1024 / 1024 : null,
+                        });
+                    }
+                    if (depth < 2) findLRU(val, path + '.' + key, depth + 1);
+                } catch(e) {}
+            }
+        } catch(e) {}
+    }
+    findLRU(globalThis, 'globalThis', 0);
+    return JSON.stringify(results);
+})()
+""")
+```
+
+**Interpretation:**
+- `size` = number of entries in the LRU
+- `calculatedSize` = sum of Content-Length values tracked (metadata, NOT actual memory)
+- `max` = maximum number of entries
+- `maxSize` = maximum calculatedSize before eviction
+
+## Summary: What's Normal vs What's a Leak
+
+| Metric | Normal Range | Concern Threshold |
+|--------|-------------|-------------------|
+| RSS after GC | 500-1500 MB | >2 GB or growing continuously |
+| Heap used | 100-300 MB | >500 MB after GC |
+| Response objects (bodyUsed=false) | <10 | >50 |
+| ArrayBuffers (excl. static 304MB) | <100 MB | >500 MB |
+| Open FDs | 50-100 | >500 |
+| Promises | 1000-5000 | >50000 |
+| RSS drop after GC | 10-50% | If <5%, memory is truly retained |
+
+## Typical Healthy Memory Profile (after GC)
+
+For a large Deco/Fresh site (e.g., fila-store, farmrio):
+
+```
+RSS:       ~1500 MB
+├── Heap:  ~150 MB (JS objects)
+├── External: ~350 MB (ArrayBuffers, ~304MB is static V8)
+└── Native: ~1000 MB (Deno runtime + JIT + modules)
+    ├── V8 JIT compiled code: ~300-500 MB
+    ├── Deno Rust runtime: ~200-300 MB
+    ├── Module cache (on disk): ~100-200 MB
+    └── Libraries + thread stacks: ~100 MB
+```
+
+The ~1 GB native gap is expected for apps loading thousands of modules. It's stable and does not grow over time.