mustflow 2.116.4 → 2.117.1

package/templates/default/locales/en/.mustflow/skills/memory-lifetime-review/references/cross-runtime-memory-stream-reproduction-harness.md

@@ -0,0 +1,289 @@
+# Cross-Runtime Memory and Stream Reproduction Harness
+
+Use this reference when a memory, external-buffer, stream-queue, socket, file-descriptor, handle,
+or allocator symptom must be reproduced across Node-compatible runtimes such as Node.js and Bun.
+Use the main skill first for ownership and diagnosis. Use this reference to design the controlled
+process harness and comparison evidence.
+
+## Contents
+
+1. [Freeze the comparison boundary](#1-freeze-the-comparison-boundary)
+2. [Separate observer and target processes](#2-separate-observer-and-target-processes)
+3. [Run explicit lifecycle phases](#3-run-explicit-lifecycle-phases)
+4. [Collect a non-overlapping metric ledger](#4-collect-a-non-overlapping-metric-ledger)
+5. [Change one factor per comparator](#5-change-one-factor-per-comparator)
+6. [Make result identity reproducible](#6-make-result-identity-reproducible)
+7. [Compare slopes, floors, and conservation](#7-compare-slopes-floors-and-conservation)
+8. [Separate runtime, adapter, and operating-system cells](#8-separate-runtime-adapter-and-operating-system-cells)
+9. [Narrow application versus runtime ownership](#9-narrow-application-versus-runtime-ownership)
+10. [Require causal fix evidence](#10-require-causal-fix-evidence)
+11. [Keep commands and artifacts bounded](#11-keep-commands-and-artifacts-bounded)
+12. [Acceptance gate](#12-acceptance-gate)
+
+## 1. Freeze the comparison boundary
+
+Start with one source program that uses only the compatibility surface shared by every target
+runtime. For a Node-compatible HTTP stream symptom, keep the request, response, Buffer,
+EventEmitter, chunk production, consumer delay, cancellation, and accounting logic identical.
+
+Do not replace the application API and runtime in the same comparison. If a native server adapter
+such as `Bun.serve` is suspected, first establish the shared `node:http` comparator. Change only the
+server adapter in the next cell. Otherwise an API difference, application rewrite, and runtime
+difference become one confounded observation.
+
+Freeze before measuring:
+
+- scenario and expected terminal outcome;
+- payload, chunk size, burst size, production rate, consumer rate, and compression;
+- concurrency, connection reuse, retries, cancellation schedule, and total offered work;
+- observation interval, warmup, load duration, quiescence interval, and stop deadline;
+- runtime flags, allocator settings, container limits, CPU quota, and cache state;
+- measurement schema, thresholds, normalization, repetitions, and statistical rule.
+
+Treat an exploratory threshold as a trigger for a narrower experiment, not as a portable proof.
+Queue, retained-memory, and RSS thresholds depend on chunk size, concurrency, runtime, allocator,
+host pressure, and instrumentation overhead.
+
+## 2. Separate observer and target processes
+
+Use a fixed observer process to launch and sample separate server and client target processes. Keep
+the observer runtime and implementation unchanged while target runtimes or versions vary.
+
+The process split should preserve evidence when either target exits, crashes, is cancelled, or is
+forcibly terminated. Use loopback or an equivalently isolated local transport unless the production
+network path is the variable under study. Start every measured run in fresh target processes so
+allocator and runtime state from a previous case does not contaminate the next case.
+
+Keep three ledgers distinct:
+
+| Ledger | Owner | Examples |
+|---|---|---|
+| Observer | Fixed harness process | target identity, OS samples, deadlines, result writing |
+| Server target | Runtime under test | response queues, accepted writes, sockets, request lifecycle |
+| Client target | Runtime under test | readable queues, consumed bytes, cancellation, client sockets |
+
+Do not count observer timers, pipes, handles, listeners, or output buffers as target resources. Do
+not let a killed client destroy the only copy of the server or OS evidence.
+
+## 3. Run explicit lifecycle phases
+
+Use the same ordered phases for every measured run:
+
+| Phase | Required action | Evidence |
+|---|---|---|
+| Baseline | Start targets, complete runtime initialization, perform only configured or manual GC, then sample | Memory floors, sockets, descriptors or handles, listeners, timers |
+| Warmup | Exercise enough work to initialize JIT, pools, parsers, and caches without entering the measured interval | Warm state and any one-time growth |
+| Load | Run exactly one fixed case and sample at a declared cadence | Slopes, peaks, throughput, queue and request maxima |
+| Admission stop | Stop creating new work while keeping targets alive | Closed cohort and remaining in-flight work |
+| Drain or cancellation | Consume, cancel, disconnect, or kill according to the case | Terminal outcomes and resource-release progress |
+| Quiescence | Wait a bounded interval for close, abort, error, native release, and pool expiry | Post-quiescent floor and unresolved owners |
+| Post-GC observations | When supported and authorized, take at least two separated observations | Reachable heap or external retention versus delayed reclamation |
+| Shutdown | End targets through the case-specific path and preserve the observer result | Exit status, final resource counts, incomplete cleanup |
+
+Mark GC support per runtime and run. Exclude unsupported or failed GC phases from GC-retention claims
+instead of treating them as zero. A forced collection is a diagnostic separator, not an application
+cleanup mechanism.
+
+## 4. Collect a non-overlapping metric ledger
+
+Record runtime-internal and independent OS observations together, but preserve their semantics.
+
+| Metric | What it supports | What it does not prove |
+|---|---|---|
+| Runtime RSS | Process resident growth as reported by the runtime | JavaScript leak or allocator cause |
+| OS resident or working set | Independent process-residency observation | Live ownership or reclaimability |
+| `heapUsed` | Managed heap currently used according to the runtime | Buffer, kernel, or all native ownership |
+| `heapTotal` | Managed heap capacity reserved by the runtime | Live heap or leak size |
+| `external` | Runtime-defined memory associated with managed objects | Every native allocation |
+| `arrayBuffers` | Runtime-defined ArrayBuffer and Buffer backing-store memory | Cross-runtime identical accounting |
+| Active socket registry | Application-observed live sockets | Kernel `TIME_WAIT` or every handle |
+| File descriptors | Open files, sockets, pipes, and other descriptors | Exact leaked resource type without classification |
+| Windows handle count | Open process handles across Windows object types | Socket count or Unix-FD equivalence |
+| Active resource types | Runtime resource-type summary that may keep the event loop alive | Concrete handle identity or full OS inventory |
+| Listener and timer ledger | Harness-known registrations and pending scheduled work | Every runtime-internal callback |
+| Readable and writable queue lengths | Visible stream-layer queued bytes or objects | Kernel, TLS, parser, or hidden native queues |
+| Per-request queue maximum | A single request or stream dominating memory | Whole-process memory |
+| Open and terminal event counts | Missing lifecycle settlement or close imbalance | Cause without owner and path evidence |
+
+For Node.js, `arrayBuffers` is included in `external`; do not add them as disjoint totals. Confirm
+equivalent fields against the target runtime version because a Node-compatible API name does not
+guarantee identical engine accounting. In object mode, queue lengths may count objects rather than
+bytes; record the unit.
+
+Track at least produced, accepted, committed or consumed, cancelled, discarded, and released bytes.
+Keep server and client totals separate. A produced-minus-consumed gap can reside in application,
+stream, runtime, kernel, or discarded work, so localize it with per-edge queues and terminal events.
+
+## 5. Change one factor per comparator
+
+Use paired cases that isolate one causal question:
+
+| Comparator | Only changed factor | Question |
+|---|---|---|
+| Paced producer / fast producer | Production interval | Does producer rate trigger the symptom? |
+| Fast consumer / slow consumer | Consumer delay | Does sink speed expose backlog? |
+| Obey backpressure / ignore backpressure | Production admission after backpressure | Is growth application-created buffering? |
+| Finite stream / continuing stream | Terminal production bound | Does cleanup depend on natural completion? |
+| Graceful completion / mid-stream cancel | Cancellation schedule | Does abort cleanup release every owner? |
+| One cancel / repeated reconnect | Lifecycle repetition | Does each completed cycle leave residue? |
+| Graceful peer exit / forced peer loss | Peer termination mode | Does abrupt loss break finalization? |
+| Fixed low / medium / high concurrency | Concurrency | Is cost linear, bounded, or explosive? |
+
+A concurrency ramp is useful for finding a threshold, but it changes load continuously. Use it only
+for exploration. Re-run fixed cells on both sides of the discovered threshold before attributing
+cause.
+
+The ignore-backpressure case is an intentional positive control, not a healthy implementation. It
+should demonstrate that the harness can detect a growing visible queue. The obey-backpressure case
+should plateau within a predeclared envelope and resume after drain.
+
+## 6. Make result identity reproducible
+
+Write a versioned allowlisted result manifest with:
+
+- scenario name and configuration hash;
+- absolute target binary identity represented safely, binary digest, runtime version, and runtime
+  revision or build identifier when available;
+- observer build identity and result-schema version;
+- operating-system release, architecture, CPU count, memory limit, container or cgroup boundary,
+  allocator class, and relevant runtime flags;
+- payload, chunk, burst, production, consumer, concurrency, cancellation, and duration settings;
+- warmup count, measured repetitions, order block, sample cadence, and GC support;
+- terminal outcomes, throughput, queue maxima, slopes, post-quiescent floors, and evidence gaps.
+
+Do not emit unrestricted arguments, environment variables, paths, URLs, headers, payloads, or
+credentials. Store logical path labels or allowlisted environment fields. Treat NDJSON, heap,
+allocation, trace, packet, and native-profile artifacts as sensitive evidence.
+
+Reject a same-case runtime comparison when configuration hashes differ. For an intentional paired
+case, allow the declared factor and reject every undeclared difference. A multi-factor exploratory
+run may guide the next test but cannot prove a cause.
+
+## 7. Compare slopes, floors, and conservation
+
+Compare server and client separately. Use raw samples and report at least:
+
+- runtime and OS residency slopes during the fixed load phase;
+- managed heap, external, and ArrayBuffer slopes and post-quiescent deltas;
+- visible stream-queue slope, maximum total queue, and maximum per-request queue;
+- active sockets, descriptors or handles, listeners, timers, and open-minus-terminal counts;
+- produced, accepted, committed or consumed, cancelled, discarded, and released byte differences;
+- throughput-normalized memory growth when runtimes complete materially different work;
+- median, dispersion, practical threshold, uncertainty decision, and rejected outliers.
+
+Do not call a faster runtime leakier merely because it processed more bytes in the same wall time.
+Preserve absolute memory and memory per completed unit. Likewise, a slower candidate is not fixed if
+it merely produces fewer bytes before the deadline.
+
+Use interleaved or randomized comparison blocks on the same machine when feasible. Avoid running
+every old-runtime sample before every new-runtime sample because temperature, CPU boost, file cache,
+system load, and memory pressure drift with time. Never rerun until a favorable sample appears.
+
+Require conservation within a declared snapshot tolerance. Examples include started equals terminal
+plus in-flight, accepted writes equal settled writes plus pending writes, and reserved bytes equal
+current plus released or terminally discarded bytes.
+
+## 8. Separate runtime, adapter, and operating-system cells
+
+Build the matrix in layers:
+
+1. Compare the same compatibility-source case on the problem runtime and a control runtime.
+2. Compare the problem runtime with the immediately relevant prior or candidate build.
+3. Change only the server adapter, such as compatibility HTTP versus a native server surface.
+4. Change only the protocol layer, such as HTTP versus a raw TCP stream.
+5. Repeat the same bad and good pair on a second representative operating-system cell.
+
+Keep Linux glibc, Linux musl, container or cgroup Linux, macOS architectures, Windows architectures,
+and WSL as distinct environment cells. Compare absolute RSS only within compatible OS,
+architecture, allocator, and resource-limit boundaries. WSL is a Linux kernel environment for this
+purpose, not a Windows IOCP cell.
+
+Record allocator and memory-limit evidence. A stable managed heap with elevated RSS can reflect
+allocator residency or unmeasured native ownership, but this pattern alone does not distinguish the
+two.
+
+## 9. Narrow application versus runtime ownership
+
+Use the smallest next single change supported by the result:
+
+| Observation | Next boundary |
+|---|---|
+| Ignore-backpressure grows on all runtimes; obey case is bounded | Application admission and queue policy |
+| Shared compatibility source fails on only one runtime | That runtime's compatibility HTTP, stream, or socket bridge |
+| Compatibility and native server adapters both fail | Shared stream, socket, allocator, or engine layer |
+| Only the native server adapter fails | Native response sink, abort, and backpressure path |
+| Raw TCP also fails | HTTP parser is below the likely boundary; inspect TCP, buffer, poller, or event loop |
+| Raw TCP is bounded | HTTP request, response, parser, or adapter lifecycle |
+| Fresh Buffer allocation fails but reuse is bounded | Backing-store allocation and release path |
+| Buffer reuse also fails | Request, stream, socket, or hidden queue ownership |
+| Descriptors or handles remain after close | Socket, poll, watcher, file, or process-resource finalization |
+| All owned resources return but RSS remains | Allocator or unmeasured native-memory diagnostics |
+
+Classify the owner as application, library or adapter, runtime, dependency, allocator, operating
+system, or unresolved. A green case at one layer only removes hypotheses that the comparator
+actually held constant.
+
+## 10. Require causal fix evidence
+
+A symptom disappearing in a newer runtime is a candidate signal, not root-cause proof. Large runtime
+or engine changes can alter allocator behavior, HTTP implementation, throughput, scheduling, and
+object lifetime together.
+
+For a permanent-resolution claim, require where feasible:
+
+- exact last-bad and first-good build identities and binary digests;
+- comparable completed work and the healthy control case remaining healthy;
+- a narrowed runtime or dependency change interval;
+- the candidate change applied to the bad base and the symptom removed;
+- the candidate change removed from the good base and the symptom restored;
+- upstream cause identity for vendored engine, allocator, parser, or event-loop changes;
+- a regression guard on bounded queues, post-quiescent ownership, and resource return rather than
+  an RSS-only literal threshold;
+- repeated results and at least one second relevant environment cell.
+
+If patch and revert builds are unavailable, report the result as version-localized or a fix
+candidate. Do not upgrade the claim to causal resolution.
+
+## 11. Keep commands and artifacts bounded
+
+This reference does not authorize launching servers, load generators, child-process campaigns,
+runtime downloads, profilers, packet tools, heap snapshots, dependency installs, network calls, or
+long-running loops. Map execution to configured one-shot command intents. Mark unavailable runtime,
+OS, profiler, or soak cells as manual-only or missing.
+
+When authoring a durable repository harness, follow the repository's supported scripting runtime and
+place it under the repository's declared test or tooling boundary. Give every run a hard deadline,
+closed stdin, output cap, process-tree cleanup, isolated output directory, and maximum memory or
+workload safety stop. Preserve the primary failure if cleanup also fails.
+
+Do not claim an attachment-linked or external harness file exists in the repository unless its bytes
+were supplied and admitted as a source artifact. A prose design can justify a reference contract,
+not an executable harness implementation.
+
+## 12. Acceptance gate
+
+Accept the reproduction design only when:
+
+- observer, server, client, and OS evidence owners are separated;
+- shared-source runtime comparison changes no application behavior;
+- lifecycle phases and closed-cohort termination are explicit;
+- metric units do not overlap or silently change across runtimes;
+- every paired comparator declares exactly one changed factor;
+- result identity includes runtime build and configuration evidence;
+- comparison uses repeated raw samples, dispersion, and completed-work normalization;
+- environment cells are compared within compatible boundaries;
+- application, adapter, runtime, allocator, and OS attribution remains no broader than the evidence;
+- a newer green version is not called causal without last-bad, first-good, and forward or reverse
+  change evidence;
+- unconfigured runtime, load, GC, profiler, packet, and OS diagnostics remain manual-only or missing;
+- sensitive result artifacts have a versioned allowlisted schema and bounded retention.
+
+## Primary references
+
+- Node.js process memory and active-resource documentation
+- Node.js stream backpressure and queue introspection documentation
+- Bun runtime and memory-profiling documentation for the exact target version
+- Operating-system process-memory, descriptor, and handle documentation for the tested cell
+
+Verify version-sensitive behavior against the exact runtime build under test. A compatibility API
+name is not evidence that two engines report memory or resource state identically.

package/templates/default/locales/en/.mustflow/skills/memory-lifetime-review/references/node-memory-resource-backpressure-diagnostics.md

@@ -0,0 +1,369 @@
+# Node Memory, Resource, and Backpressure Diagnostics
+
+Use this reference when a Node.js process shows growing RSS, heap, external memory, buffers,
+descriptors, sockets, native handles, cache residency, or streaming queues, especially when slow
+consumers, retries, clones, transforms, HTTP/2, or HTTP/3 can blur ownership.
+
+Do not force one exclusive root-cause label onto evidence that describes different dimensions. A
+single incident can combine unbounded buffering, external-memory growth, allocator fragmentation,
+and a missed transport close.
+
+## Contents
+
+1. [Classify three independent axes](#1-classify-three-independent-axes)
+2. [Build one synchronized measurement ledger](#2-build-one-synchronized-measurement-ledger)
+3. [Control the reproduction and compare floors](#3-control-the-reproduction-and-compare-floors)
+4. [Distinguish common growth classes](#4-distinguish-common-growth-classes)
+5. [Model request ownership as a DAG](#5-model-request-ownership-as-a-dag)
+6. [Enforce terminal resource invariants](#6-enforce-terminal-resource-invariants)
+7. [Trace backpressure from the final sink](#7-trace-backpressure-from-the-final-sink)
+8. [Check each pipeline boundary](#8-check-each-pipeline-boundary)
+9. [Run stop, plateau, and resume experiments](#9-run-stop-plateau-and-resume-experiments)
+10. [Calculate the memory envelope](#10-calculate-the-memory-envelope)
+11. [Infer the broken boundary](#11-infer-the-broken-boundary)
+12. [Diagnostic acceptance gate](#12-diagnostic-acceptance-gate)
+13. [Primary references](#13-primary-references)
+
+## 1. Classify three independent axes
+
+Classify every observation on all three axes:
+
+| Axis | Question |
+| --- | --- |
+| Storage location | Are the bytes in the V8 heap, ArrayBuffer or Buffer backing stores, other native allocations, memory mappings, allocator residency, or kernel buffers? |
+| Retention cause | Does a request, pending task, stream queue, cache, pool, native operation, retry, branch, or telemetry queue still own the bytes? |
+| Terminal status | Is the owner still performing valid work, stuck in unbounded backlog, terminal but retained, or released while RSS remains resident? |
+
+Call growth a leak only when the expected terminal condition has occurred, an owner or native
+allocation remains live, and retained count or bytes grow with completed lifecycle repetitions under
+comparable load. A queue that drains completely after a live request is not a strict leak, but an
+unbounded queue is still an OOM defect and must not be reported as safe.
+
+One common mixed sequence is:
+
+```text
+slow consumer
+  -> ignored writable backpressure
+  -> unbounded Buffer queue
+  -> external and array-buffer growth
+  -> queue release
+  -> allocator pages remain resident
+  -> external falls while RSS stays high
+```
+
+Do not infer native memory as `rss - heapTotal - external`. The measurements use different
+inclusion, reservation, and residency semantics. In Node, `arrayBuffers` is part of `external`, not
+an independent addend.
+
+## 2. Build one synchronized measurement ledger
+
+Collect comparable samples from the same process and clock domain:
+
+| Layer | Required evidence | Useful expansion |
+| --- | --- | --- |
+| V8 heap | `heapUsed`, `heapTotal`, post-major-GC floor | old space, large-object space, native and detached contexts, global handles |
+| External memory | `external`, `arrayBuffers` | Buffer count, backing-store bytes, parser, codec, crypto, and TLS buffers |
+| Process memory | RSS plus private or committed memory | anonymous, file-backed, shared mappings, PSS, thread stacks, code pages |
+| Resources | descriptor, socket, or Windows handle counts by type | socket state, pool busy/idle/queued, TLS session cache, active resource categories |
+| Application | started, completed, aborted, failed, retried, in-flight | logical request, attempt, branch, and owner counts |
+| Data flow | admitted, accepted, committed, released, cancelled, and discarded bytes per edge | peak queue bytes, chunk size, transform expansion, sink durability boundary |
+| Protocol | TCP queues, HTTP/2 windows and outbound queue, QUIC credit | reset, window update, blocked-frame, cancel, and session or stream counts |
+
+Use V8 heap-space statistics to distinguish live-space growth from reserved capacity. A growing
+native or detached context count is a retention lead, not a complete diagnosis.
+
+On Linux, mapping summaries can separate anonymous, file-backed, and shared residency. On Windows,
+pair Working Set with Private Bytes or Commit and handle counts by type. Treat mapping inspection,
+native heap profilers, heap snapshots, packet capture, and platform handle tools as configured or
+manual-only diagnostics; this reference does not authorize raw commands.
+
+`process.getActiveResourcesInfo()` reports resource types keeping the event loop alive. It is not an
+enumeration of every reference, descriptor, allocation, or leak. Pair it with ownership and
+platform evidence.
+
+## 3. Control the reproduction and compare floors
+
+Fix more than request rate. Record concurrency, body and chunk sizes, compression, protocol,
+retries, cache state, branch count, consumer rate, origin cardinality, runtime build, and allocator.
+
+Run phases in this order:
+
+1. idle baseline;
+2. bounded warmup with fixed origins, keys, and connections;
+3. normal consumer faster than producer;
+4. slow consumer below producer rate;
+5. fully stopped consumer;
+6. admission stop followed by complete drain;
+7. idle wait beyond pool and keep-alive deadlines; and
+8. optional forced GC only in an isolated staging or canary process.
+
+Do not take a production heap snapshot casually. Snapshot generation can block the event loop and
+require substantial extra memory. Use an isolated process or removed-from-traffic instance and
+record the diagnostic impact.
+
+Compare floors and slopes rather than one peak:
+
+```text
+post_gc_slope
+  = change(post-major-GC heapUsed) / change(completed lifecycles)
+
+post_quiescent_slope
+  = change(resource amount after admission stop, drain, and idle expiry)
+    / change(completed lifecycles)
+
+Q_edge(t)
+  = admitted_bytes_edge(t) - released_bytes_edge(t)
+```
+
+Define release as the moment that edge no longer owns the bytes, not the moment it hands them to
+another queue. Keep compressed, decompressed, transformed, and committed byte domains separate.
+Use a predeclared statistical tolerance rather than demanding literal zero from noisy process
+measurements.
+
+## 4. Distinguish common growth classes
+
+| Class | Decisive evidence | Common false conclusion |
+| --- | --- | --- |
+| JavaScript heap retention | Post-GC and post-quiescent live heap grows with completed work; repeated dominator paths lead to a long-lived map, cache, listener, promise continuation, closure, or context | Treating `heapTotal`, warmup, or a bounded cache as live-object growth |
+| Native leak | JavaScript heap, external memory, queues, and handles are stable while still-live native allocation stacks grow | Calling JIT code, mappings, thread stacks, pools, or fragmentation a native leak |
+| External buffer retention | Small wrappers or views retain large backing stores and `arrayBuffers` moves with owned queue bytes | Treating all `external` as Buffer memory or continuing to call it a Buffer leak after external falls |
+| Unbounded buffering | Growth appears under a slow or stopped consumer near producer-minus-consumer rate and drains after consumption resumes | Calling it safe because it eventually drains, or calling it a terminal leak before the request ends |
+| Descriptor, socket, or handle leak | After admission stop, terminal settlement, and pool idle expiry, owned resources fail to return to baseline, often one per abort or error | Counting a bounded warm pool or kernel `TIME_WAIT` as a process-owned leak |
+| Cache growth | A named cache root owns the bytes and growth follows key, origin, tenant, or SNI cardinality; eviction or flush releases them | Assuming TTL alone is a byte bound or limiting entries while values remain unbounded |
+| Allocator fragmentation or residency | Live heap, external memory, queues, and live native allocations stabilize or fall while allocator active-to-resident gap remains high | Declaring fragmentation merely because RSS did not fall immediately after release |
+
+Persistent `CLOSE_WAIT` plus an owned descriptor is a stronger local-close signal than `TIME_WAIT`.
+Confirm the owning process and lifecycle before assigning cause.
+
+## 5. Model request ownership as a DAG
+
+A strict tree is wrong when pools and multiplexed sessions are shared, retries or hedges create
+multiple attempts, and `clone()` or `tee()` creates multiple body branches.
+
+```text
+process
+  -> process caches and telemetry queues
+  -> pool
+       -> connection or session
+            -> protocol stream leased by request attempt
+  -> logical request
+       -> root cancellation and context
+       -> attempt 1
+       -> attempt 2 or hedge
+       -> output branch A
+       -> output branch B
+```
+
+Track at least:
+
+```text
+resource_id
+resource_type
+scope: process | pool | connection | request | attempt | branch | stream
+owner_id
+parent_id
+leased_by_request_ids
+created_at
+created_stack_hash
+state: CREATED | ACTIVE | CLOSING | CLOSED | FAILED
+current_bytes
+close_action
+terminal_reason
+closed_at
+```
+
+Separate logical request, attempt, branch, protocol stream, shared connection, and process cache
+ownership. A request normally returns a lease instead of closing a healthy shared connection.
+Cancelling one HTTP/2 or HTTP/3 request normally resets its stream, not the whole session or
+connection.
+
+Track terminal reasons such as success, client abort, deadline, DNS, connect, TLS, upstream
+headers, upstream body, transform, downstream, retry-superseded, and shutdown. Do not let the
+success path be the only path with balanced counters.
+
+## 6. Enforce terminal resource invariants
+
+For every supported terminal reason, require the applicable invariants:
+
+| Scope | Terminal invariant |
+| --- | --- |
+| Request | Registry, async context, listeners, timers, and request references from cancellation state are zero |
+| Attempt | Fetch or provider task, retry or hedge task, body, parser, and timeout are terminal |
+| Branch | Every clone or tee branch is consumed, cancelled, or errored |
+| Stream | Reader and writer locks are released, pipe tasks settle, and both transform sides reach a terminal state |
+| File | File handle, stream, temporary file, and quota reservation are released |
+| Protocol | HTTP/1 exchange or HTTP/2/3 request stream is terminal |
+| Shared resource | Request lease and reference count decrement exactly once; a healthy shared socket or session need not close |
+| Memory | Request-owned queue bytes reach zero; shared pools and caches remain within declared byte and cardinality limits |
+| Observation | Terminal reason records once and close actions are idempotent |
+
+A useful cleanup order is: atomically select terminal state, abort the root scope, stop producers and
+retries, cancel or abort sources and sinks, await task settlement, release locks, remove timers and
+listeners, close files, return shared leases, and remove the registry entry. A finalizer or GC hook
+is a safety net, not the normal protocol.
+
+Promises require separate treatment from their work. A settled promise does not prove the socket,
+file, fetch, codec, or native operation ended. A pending promise is not automatically a leak unless
+a longer-lived owner retains it or its underlying operation past the intended boundary.
+
+## 7. Trace backpressure from the final sink
+
+Backpressure is a chain of local capacity translations, not one signal travelling unchanged through
+the stack:
+
+```text
+final sink capacity falls
+  -> Web writer readiness blocks or Node write returns false
+  -> transform readable queue fills
+  -> transform writable pressure rises
+  -> body pull or read stops
+  -> parser and body ingress slow
+  -> HTTP/1 socket reading slows
+     or HTTP/2 flow-control credit stops increasing
+     or QUIC stream and connection credit stops increasing
+  -> remote producer rate falls
+```
+
+Instrument each request, attempt, and branch with compatible byte domains:
+
+```text
+wire_encoded_received
+fetch_ingress_encoded
+body_decoded_enqueued
+body_delivered
+transform_input
+transform_output_enqueued
+sink_accepted
+sink_committed
+sink_released
+cancelled_bytes
+discarded_bytes
+```
+
+Keep accepted, committed, and released distinct. A sink that resolves `write()` after copying into
+an internal array has accepted bytes but has not necessarily committed or released them. Compression
+and expansion require separate encoded and decoded ledgers.
+
+## 8. Check each pipeline boundary
+
+| Boundary | Healthy pressure evidence | Failure pattern |
+| --- | --- | --- |
+| Final writable | Web writer readiness remains pending or Node `write()` returns false until drain; local queue plateaus | Adapter resolves before real capacity, ignores false, or grows a background queue |
+| Transform | Transform invocation and output enqueue stop when readable capacity is exhausted; async transform returns its work promise | Background work or side arrays bypass the stream queues; count-based HWM hides huge chunks |
+| Response body and manual pump | New reads stop when the sink blocks; early exit cancels; every write is awaited | Read loop accumulates promises or chunks, releases only a lock, or leaves a clone branch unread |
+| Fetch or client bridge | Body ingress stops after bounded overshoot when delivery stops | Parser, decoder, or bridge continues consuming into user-space storage |
+| HTTP/1 | Application delivery and socket reads stop after bounded transport and parser overshoot | User-space keeps draining the kernel while application Buffers grow; low kernel Recv-Q is misread as health |
+| HTTP/2 | Stream credit stops increasing while connection control frames and healthy sibling streams still progress | Credit is returned when bytes enter an unbounded application queue, or the entire TCP socket is paused and siblings deadlock |
+| HTTP/3 and QUIC | Stream and connection credit stop increasing; blocked telemetry appears; cancellation closes the narrow stream directions | QUIC returns credit before application consumption or one-direction cancellation leaves reassembly state retained |
+| Transport | TCP receive window or QUIC flow control eventually stalls the sender after bounded in-flight bytes | Continued application reads keep transport open while downstream storage grows |
+
+Treat high-water marks as pressure thresholds, not hard byte caps. Use a byte-length queuing strategy
+for byte payloads and include maximum chunk, in-flight operations, protocol windows, and hidden side
+queues in the budget.
+
+Do not pause an entire HTTP/2 transport merely to slow one stream. Continue processing connection
+control frames and healthy siblings; express pressure through stream and connection flow-control
+accounting. Verify runtime-specific behavior against the target version and client implementation.
+
+## 9. Run stop, plateau, and resume experiments
+
+Start with HTTP/1, no compression, no retry or hedge, no cache, no clone or tee, concurrency one,
+fixed incompressible chunks, and an origin faster than the sink. Add one variable at a time only
+after the base experiment passes.
+
+During normal consumption, queues should oscillate within a low bounded range. During slow
+consumption, buffers may fill, but upstream rate should converge toward sink rate. During a full
+stop, require the applicable observations after bounded overshoot:
+
+- writer readiness blocks or Node write returns false;
+- local writable and transform queues plateau;
+- transform invocations and response reads stop;
+- client body ingress stops;
+- HTTP/2 window updates or QUIC credit increases stop for the affected stream; and
+- remote sender throughput stops after in-flight transport data is exhausted.
+
+On resume, require queue drain, writer readiness or drain, transform and body reads, protocol credit,
+and remote sending to resume. A pipeline that stops but never resumes is deadlocked or cancelled,
+not backpressure-correct.
+
+Then add, one at a time: concurrency, HTTP/2, HTTP/3, gzip or Brotli expansion, output-expanding
+transforms, retries, hedges, clones, tees, slow and fast branches, downstream abort, upstream reset,
+transform failure, and graceful shutdown.
+
+## 10. Calculate the memory envelope
+
+Use an explicit envelope such as:
+
+```text
+pipeline_memory_bound
+  ~= concurrency
+     * sum(
+         queue_high_water_mark_bytes
+         + maximum_chunk_bytes
+         + in_flight_operation_bytes
+         + bounded_side_queue_bytes
+       )
+     + protocol_receive_windows
+     + protocol_send_windows
+     + kernel_and_userspace_transport_buffers
+     + clone_branch_multiplier
+     + retry_or_hedge_multiplier
+     + codec_expansion_working_set
+```
+
+The equation is a budgeting model, not proof of allocator RSS. Validate each term from current
+implementation and workload evidence. A count HWM of one permits one arbitrarily large chunk; it is
+not a one-byte budget.
+
+## 11. Infer the broken boundary
+
+| Observation | Likely broken boundary |
+| --- | --- |
+| `sinkCommitted` stops while `sinkAccepted` grows | Sink adapter resolves before durable or capacity-limited commit |
+| `sinkAccepted` stops while transform output grows | Transform ignores readable pressure or uses a side queue |
+| Transform input stops while body delivery grows | Manual pump is not serializing read and write |
+| Body delivery stops while client ingress grows | Parser, decoder, or response bridge is over-reading |
+| Application delivery stops while HTTP/2 DATA and credit continue | Credit is returned at intermediate copy instead of final consumption |
+| Application delivery stops while QUIC credit continues | QUIC integration returns credit before application consumption |
+| Application body stops while socket bytes rise and kernel receive queue stays low | HTTP/1 or client bridge continues draining into user-space buffers |
+| Queues and array buffers fall while RSS remains high | Allocator residency, slabs, pools, or fragmentation, after excluding live native allocation |
+| Bytes stabilize while descriptors or handles rise | Independent lifecycle leak |
+| Descriptors stabilize while live native allocation stacks rise | Native memory leak |
+| Live memory stabilizes while cache bytes rise | Cache policy or cardinality failure |
+| Post-GC heap rises even with normal consumers | JavaScript retention unrelated to backpressure |
+
+## 12. Diagnostic acceptance gate
+
+Call the diagnosis complete only to the evidence level actually established:
+
+- locate growth in at least one measurable storage class;
+- identify the owner or bounded queue responsible for retaining it;
+- prove request, attempt, branch, stream, file, and protocol terminal invariants for every supported
+  success, abort, timeout, retry, error, and shutdown path;
+- show stopped-consumer queues plateau within a precomputed envelope and upstream rate eventually
+  stops;
+- show resume restores drain, protocol credit, and upstream progress;
+- show request-owned descriptors and handles return within predeclared tolerance after pool idle
+  expiry while shared pools remain within their limits;
+- show post-GC or post-quiescent slopes remain within predeclared statistical tolerance; and
+- distinguish fragmentation only after live heap, external, cache, queue, handle, mapping, and native
+  allocation evidence no longer explains resident growth.
+
+If snapshots, native profilers, platform handle inspection, packet capture, qlog, or long-running
+load are not configured, report them as manual-only or missing. RSS growth alone is an observation,
+not a completed diagnosis.
+
+## 13. Primary references
+
+- [Node.js process memory and active resource APIs](https://nodejs.org/api/process.html)
+- [Node.js V8 heap statistics and heap snapshots](https://nodejs.org/api/v8.html)
+- [Node.js Buffer ownership and views](https://nodejs.org/api/buffer.html)
+- [Node.js stream backpressure and high-water marks](https://nodejs.org/api/stream.html)
+- [Node.js HTTP/2 state and flow control](https://nodejs.org/api/http2.html)
+- [WHATWG Streams Standard](https://streams.spec.whatwg.org/)
+- [RFC 9113: HTTP/2](https://www.rfc-editor.org/rfc/rfc9113.html)
+- [RFC 9000: QUIC](https://www.rfc-editor.org/rfc/rfc9000.html)
+
+Use documentation for the target runtime and client version. The current Node documentation and
+standards establish API and protocol semantics; they do not prove a particular library build,
+allocator, kernel, proxy, or application implements the lifecycle correctly.