@zeyue0329/xiaoma-cli 1.16.1-next.1 → 1.17.1-next.0

package/package.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/package.json",
   "name": "@zeyue0329/xiaoma-cli",
-  "version": "1.16.1-next.1",
+  "version": "1.17.1-next.0",
   "description": "XiaoMa Universal AI Agent Framework",
   "keywords": [
     "agile",

package/src/core-skills/module-help.csv

@@ -11,3 +11,4 @@ Core,xiaoma-review-adversarial-general,Adversarial Review,AR,"Use for quality as
 Core,xiaoma-review-edge-case-hunter,Edge Case Hunter Review,ECH,Use alongside adversarial review for orthogonal coverage — method-driven not attitude-driven.,,[path],anytime,,,false,,
 Core,xiaoma-spec,Spec,SP,"Use to distill any intent input (brief, PRD, transcript, brain dump, design folder, mixed multi-source) into a succinct, no-fluff SPEC.md contract + companions that downstream work derives from. Locks the WHAT before the HOW. Works for software, game design, research, editorial, policy, business, anything intent-bearing. Validation mode also available.",,[path],anytime,,,false,{output_folder}/specs/spec-{slug},SPEC.md + companion files
 Core,xiaoma-customize,XiaoMa Customize,BC,"Use when you want to change how an agent or workflow behaves — add persistent facts, swap templates, insert activation hooks, or customize menus. Scans what's customizable, picks the right scope (agent vs workflow), writes the override to _xiaoma/custom/, and verifies the merge. No TOML hand-authoring required.",,,anytime,,,false,{project-root}/_xiaoma/custom,TOML override files
+Core,xiaoma-heap-dump-analysis,Heap Dump Leak Analysis,HDA,Use when you have a JVM .hprof / OOM heap dump and need the true root cause of a memory leak — which collection retains the leaked objects and why they are never released.,,[path to .hprof] [suspected class],anytime,,,false,report located next to the dump,root-cause memory-leak report

package/src/core-skills/xiaoma-heap-dump-analysis/SKILL.md

@@ -0,0 +1,137 @@
+---
+name: xiaoma-heap-dump-analysis
+description: Analyze a JVM heap dump (.hprof) to find the TRUE root cause of a memory leak — not just "which objects are large" but "who retains them and why they are never released". Drives a class histogram, reverse-reference GC-root tracing, optional Eclipse MAT cross-validation, and precise collection/field measurement into a verified report. Use when the user provides a JVM .hprof / heap dump / OOM dump (or says 内存泄漏 / 堆转储 / 堆 dump（.hprof）分析) and asks to analyze it, find the leak, or 找出内存泄漏的真实原因. NOT for thread dumps (jstack), GC logs, or hs_err crash logs — this is heap (.hprof) memory-leak analysis only.
+argument-hint: "[path to .hprof] [optional: suspected class name]"
+---
+
+# Heap Dump Leak Analysis
+
+## Overview
+
+This skill finds the **true root cause** of a JVM heap memory leak from a `.hprof` dump. A histogram alone only tells you *what* objects are numerous — it points at generic containers (`char[]`, `String`, `HashMap$Node`) and rarely names the bug. The real question is **who retains the leaked objects on a GC-root path, and why they are never released.**
+
+The method is **multi-evidence and self-validating**: every conclusion is reached by at least two independent routes (a self-written reverse-reference tracer + Eclipse MAT's dominator tree, plus exact field/collection measurement). You act as a JVM memory forensics specialist.
+
+The bundled `scripts/` are pure-stdlib streaming HPROF parsers (no third-party deps, memory usage independent of object count) and work on multi-GB dumps with tens of millions of objects. **MAT is optional** — the scripts alone can locate the holder chain; MAT is used as cross-validation when available.
+
+**Out of scope (non-goals).** This skill analyzes JVM **heap dumps (`.hprof`)** for memory leaks only. It does **not** handle thread dumps (`jstack` / `*.tdump`), GC logs, or `hs_err_pid*` crash logs — those are different artifacts needing different tooling. The bundled parsers assume the HPROF binary format and now hard-fail with a clear `[err]` if the input doesn't start with the `JAVA PROFILE` magic. If the user hands you one of those other dump/log types, say so and stop rather than forcing it through this pipeline.
+
+## Conventions
+
+- Bare paths (e.g. `scripts/trace_referrers.py`, `resources/methodology.md`) resolve from this skill's root.
+- All scripts run with the system `python3` (3.8+), no dependencies. Run any script with `--help` first.
+- Class names accept dot or slash form: `com.foo.Bar` == `com/foo/Bar`.
+- **Large dumps — run in background and poll.** Every script does a full streaming pass
+  (~1–2 min per few GB; `trace_referrers` does one pass per hop). Don't block the foreground:
+
+  ```bash
+  nohup python3 scripts/trace_referrers.py <dump> <class> --hops 6 > /tmp/trace.out 2>&1 &
+  until grep -q '\[done\]' /tmp/trace.out 2>/dev/null; do sleep 5; done
+  cat /tmp/trace.out
+  ```
+
+  Scripts emit progress to stderr (`[passA]`, `heap segment #N`, per-hop headers) and finish
+  with a `[done]` marker — poll for it.
+
+## Prerequisites & Data Safety — DO THIS FIRST
+
+Before any analysis, run the safety preflight. **These steps prevent the two failures that wasted the most time in practice.**
+
+1. **Locate and verify the dump.** Confirm it is HPROF: the first 13 bytes are `JAVA PROFILE`. Note its size.
+
+2. **⚠️ Special characters in the filename break tools.** Filenames containing `%`, spaces, or other shell/URI-sensitive characters (common when JVMs write `-XX:HeapDumpPath=app_%p.hprof`) cause Eclipse MAT and many launchers to fail silently. **Immediately create a hardlink with a clean name** and use it everywhere downstream:
+   ```bash
+   ln '/path/to/app_%p.hprof' '/path/to/app_clean.hprof'   # hardlink, instant, no extra disk
+   ```
+
+3. **⚠️ Protect the data — the dump may be the only copy.** A heap dump is irreplaceable, and external tooling (uploaders, splitters) may move or delete it mid-analysis. **Create a second hardlink as a backup** so the inode survives even if the original name is removed:
+   ```bash
+   ln '/path/to/app_clean.hprof' ~/app_dump_backup.hprof
+   ```
+   (A hardlink shares the inode — the data lives as long as ANY link exists, at zero extra disk cost.)
+
+4. **Check tooling.** `python3 --version` (required). `java -version` (JDK 11+, only needed if using MAT). MAT presence is optional — see `resources/mat-headless-runbook.md` for install + headless invocation.
+
+5. **If the user named a suspected class**, note it for Stage 2. Otherwise Stage 1's histogram will surface candidates.
+
+## Stages
+
+| # | Stage | Tool | Purpose |
+|---|-------|------|---------|
+| 1 | Histogram | `scripts/hprof_histogram.py` | Find which classes are abnormally numerous / large — especially business & third-party classes |
+| 2 | Reverse trace | `scripts/trace_referrers.py` | Walk *up* from a suspect class to its GC-root holder chain (the "path to GC roots") |
+| 3 | MAT cross-check *(optional)* | `resources/mat-headless-runbook.md` | Run MAT's Leak Suspects + dominator tree headless; corroborate Stage 2 |
+| 4 | Precise measure | `scripts/inspect_objects.py` | Nail the exact entry count of the suspect collection / the exact value of config fields |
+| 5 | Report | — | Synthesize: symptom → multi-evidence → leak chain → mechanism → fix |
+
+### Stage 1: Histogram — what is abnormal
+
+```bash
+python3 scripts/hprof_histogram.py <dump.hprof> --top 50
+```
+Read the four leaderboards. The **business/third-party leaderboards (JDK excluded)** are the most diagnostic — they point at *your* objects, not generic containers. Look for a domain object whose instance count is wildly higher than the number of live "real" things it should represent (e.g. session objects ≫ live TCP connections). That mismatch is the leak signature; that class is the Stage 2 suspect.
+
+Sanity-anchor the count against reality: compare the suspect against `io.netty.channel.socket.nio.NioSocketChannel`, `sun.nio.ch.SocketChannelImpl`, `java.io.FileDescriptor` (live connections), or a domain "online user" object. A large gap = retained zombies.
+
+### Stage 2: Reverse trace — who retains them
+
+```bash
+python3 scripts/trace_referrers.py <dump.hprof> <suspect-class> --hops 6
+```
+Each hop reports who references the current set, **by referrer class and by exact field name**, and flags two GC-root signals:
+- **★ static field holder** — a `static` field of some class directly references the objects (classic static-cache / registry leak).
+- **★ referrer is itself a GC root** — thread, JNI global, sticky class, etc.
+
+Follow the chain until it converges on a single container (e.g. one `ConcurrentHashMap$Node[]` table, one singleton holder) or hits a ★ anchor. That holder + field is the leak's retention point — the collection that should have been pruned but wasn't.
+
+**Expect the object graph to contain cycles** (e.g. `ClientHead.clientsBox → ClientsBox → map → ClientHead`); the reverse-BFS `next` set will balloon in later hops. That is normal — focus on the *first* hops where a single container/field clearly dominates, and on the ★ anchors.
+
+### Stage 3: MAT cross-validation *(OPTIONAL — skip if MAT isn't installed)*
+
+**Quick gate:** if `/Applications/MemoryAnalyzer.app` (or a `ParseHeapDump.sh` on PATH) is
+absent, **skip this stage entirely** — the pure-Python Stages 1/2/4 already locate and prove
+the holder chain. Do this stage only when MAT is available and you want the authoritative
+dominator-tree + retained-size corroboration.
+
+If Eclipse MAT is available, generate the official Leak Suspects report headless and confirm it names the same holder. **Follow `resources/mat-headless-runbook.md` exactly** — the macOS `.app` launcher fails silently (`exit 14`) from a headless shell; you must invoke the Equinox launcher jar directly with a raised `-Xmx`, and the dump filename must be free of special characters (Stage Prereqs already handled this).
+
+MAT's "Problem Suspect 1" (a dominator occupying the bulk of the heap) and its `Node[N]` accumulation point should match the container Stage 2 converged on. Two independent methods agreeing = high confidence.
+
+### Stage 4: Precise measurement — nail it
+
+Turn inference into hard numbers. First list the holder's fields, then measure the collection and/or read config fields:
+```bash
+# List the holder class's fields (decide what to read)
+python3 scripts/inspect_objects.py <dump.hprof> --class <holder-class>
+# Measure the suspect collection(s) — entry count via table capacity + non-empty buckets
+python3 scripts/inspect_objects.py <dump.hprof> --class <holder-class> --map-fields <field1,field2>
+# Read config / state fields (e.g. is a timeout misconfigured?)
+python3 scripts/inspect_objects.py <dump.hprof> --class <config-class> --fields <field1,field2>
+# Measure a STATIC cache/registry map (static fields are GC-root-level holders — a top leak source)
+python3 scripts/inspect_objects.py <dump.hprof> --class <util-class> --static-fields <field1,field2>
+```
+This distinguishes the real leak collection from innocent ones (a sibling map with thousands of entries is not the 100k+ one), and reads the actual runtime config that governs cleanup (heartbeat/timeout values, flags). **Note:** `ConcurrentHashMap.baseCount`/`HashMap.size` can read low under concurrency; trust **non-empty bucket count** + **table capacity (a power of two)** for the real magnitude.
+
+### Stage 5: Report
+
+Synthesize a report in the user's language (Chinese if the user wrote in Chinese). Required structure:
+
+1. **One-line conclusion** — the true root cause in a sentence.
+2. **Heap overview** — file size, object count, app stack (framework versions if visible).
+3. **Evidence (multi-route)** — a table: histogram counts, reverse-trace holder chain, MAT suspect %, exact measured entry counts. Show they agree.
+4. **Leak chain** — `GC root → … → holder.field (collection) → leaked objects → their retained subtree`. Quantify the retained subtree (what fills the heap).
+5. **Mechanism root cause** — *why* the collection is never pruned (missing remove on disconnect, listener never deregistered, unbounded cache, misconfigured timeout, framework bug, …). Cite the measured config/code evidence. Be explicit about what is *proven* vs *inferred*.
+6. **Fix recommendations** — ordered: permanent fix (code/version upgrade), config correction, guardrails (limits/monitoring/alerts), temporary mitigation (heap bump + rolling restart / scheduled cleanup).
+7. **Data-safety note** — if the original file was renamed/deleted, tell the user which hardlink now holds the data.
+
+## Graceful degradation & scaling
+
+- **No MAT / MAT won't run** → Stages 1, 2, 4 (pure-Python) are fully sufficient to locate and prove the holder chain. MAT is corroboration, not a dependency.
+- **Very large dumps / multiple suspects** → run Stage 2 on each suspect; the scripts are streaming and re-runnable. You may fan out independent traces as subagents and synthesize.
+- **Unfamiliar framework** → after Stage 2 names the holder class + field, look up that library's source for the add/remove lifecycle of that collection to explain the *mechanism* (Stage 5). Quote method names.
+
+## Resources
+
+- `resources/methodology.md` — the full five-stage leak-hunting methodology, decision heuristics, and a worked end-to-end example.
+- `resources/mat-headless-runbook.md` — installing Eclipse MAT and running it **headless** (the exit-14 pitfall, Equinox launcher invocation, `-Xmx`, reading the report zips with `textutil`).
+- `resources/hprof-internals.md` — HPROF binary format reference and how the bundled scripts parse it (so you can extend them for a new question).

package/src/core-skills/xiaoma-heap-dump-analysis/resources/hprof-internals.md

@@ -0,0 +1,157 @@
+# HPROF Binary Format — Reference & How the Scripts Parse It
+
+The bundled scripts are self-contained streaming parsers of the HPROF binary format. This
+note documents the format and the parsing strategy so you can **extend the scripts** for a
+new question (e.g. "dump every field of object X", "list the keys of map Y", "histogram of
+retained subtree of class Z").
+
+## File layout
+
+```
+[format string, NUL-terminated]   e.g. "JAVA PROFILE 1.0.1" / "1.0.2"
+[identifier size : u4]            usually 8 (64-bit, no compressed-oop in the dump format)
+[timestamp : u8]
+[record]*                         a flat sequence of top-level records
+```
+
+Each top-level record:
+
+```
+[tag : u1][time : u4][length : u4][body : <length> bytes]
+```
+
+### Top-level tags used
+
+| tag | name | body |
+|-----|------|------|
+| 0x01 | STRING (UTF-8) | `id` + UTF-8 bytes → the string table (class names, field names) |
+| 0x02 | LOAD_CLASS | class serial(u4) + **class object id** + stack serial(u4) + **name string id** |
+| 0x0C | HEAP_DUMP | a container of heap sub-records |
+| 0x1C | HEAP_DUMP_SEGMENT | same as 0x0C, just chunked (large dumps emit several) |
+| 0x2C | HEAP_DUMP_END | marker |
+
+Other tags (UTF8 stack frames/traces, alloc sites, etc.) are skipped via `length`.
+
+## Heap-dump sub-records
+
+Inside a 0x0C/0x1C body is a sequence of sub-records, each led by a 1-byte sub-tag. To stay
+aligned you must consume **exactly** the right number of bytes for every sub-tag — including
+GC-root records you don't care about.
+
+### GC roots (fixed sizes — must skip precisely)
+
+| sub-tag | root kind | after the leading `id`, extra bytes |
+|---------|-----------|-------------------------------------|
+| 0xFF | UNKNOWN | 0 |
+| 0x01 | JNI GLOBAL | + 1 `id` (JNI ref) |
+| 0x02 | JNI LOCAL | + u4 + u4 (= 8) |
+| 0x03 | JAVA FRAME | + u4 + u4 (= 8) |
+| 0x04 | NATIVE STACK | + u4 (= 4) |
+| 0x05 | STICKY CLASS | 0 |
+| 0x06 | THREAD BLOCK | + u4 (= 4) |
+| 0x07 | MONITOR USED | 0 |
+| 0x08 | THREAD OBJECT | + u4 + u4 (= 8) |
+
+The leading `id` of each is the rooted object — collect these into a set to detect
+"referrer is itself a GC root."
+
+### 0x20 CLASS_DUMP (the tricky one)
+
+```
+class object id        : id
+stack trace serial     : u4
+super class object id  : id
+class loader id        : id
+signers id             : id
+protection domain id   : id
+reserved               : id
+reserved               : id
+instance size          : u4
+constant pool count    : u2,  then each: cp-index(u2) + type(u1) + value(size-by-type)
+static fields count    : u2,  then each: name string id(id) + type(u1) + value(size-by-type)
+instance fields count  : u2,  then each: name string id(id) + type(u1)
+```
+
+From CLASS_DUMP the scripts cache: `super` (to walk the chain), the **instance field list**
+`(type, name-id)` in declaration order, and **static field references** (object-typed
+statics — these are GC-root-level holders and how static-cache leaks are detected).
+
+### 0x21 INSTANCE_DUMP
+
+```
+object id           : id
+stack trace serial  : u4
+class object id     : id
+num bytes that follow : u4
+field values        : <num bytes>   (raw, laid out per the class's field layout)
+```
+
+**Field layout order:** the values are this class's instance fields in declaration order,
+**then the super class's, then its super's**, … each field occupying `size-by-type` bytes.
+To read a specific field you compute its byte offset by walking
+`class → super → super…`, summing field sizes; reference fields (type 2) hold an `id`.
+
+### 0x22 OBJECT_ARRAY_DUMP
+
+```
+array object id : id
+stack serial    : u4
+num elements    : u4
+array class id  : id
+elements        : num × id
+```
+
+### 0x23 PRIMITIVE_ARRAY_DUMP
+
+```
+array object id : id
+stack serial    : u4
+num elements    : u4
+element type    : u1
+elements        : num × size-by-type
+```
+
+## Basic type tags & sizes
+
+| tag | type | size |
+|-----|------|------|
+| 2 | object (ref) | `id_size` (usually 8) |
+| 4 | boolean | 1 |
+| 5 | char | 2 |
+| 6 | float | 4 |
+| 7 | double | 8 |
+| 8 | byte | 1 |
+| 9 | short | 2 |
+| 10 | int | 4 |
+| 11 | long | 8 |
+
+## Parsing strategy used by the scripts
+
+- **Streaming, low memory.** Read each top-level record header (9 bytes), then read the
+  body only for records of interest (STRING, LOAD_CLASS, HEAP_DUMP); `seek()` past the rest.
+  Heap-dump bodies are walked sub-record by sub-record over a `memoryview` (no copies).
+- **Class metadata is cached once**, object counts/fields are never all held in memory — so
+  memory use is independent of object count (works on 40M-object dumps).
+- **Reference offsets** (`get_refoffs`) precompute, per class, the byte offsets of all
+  reference-typed fields (self + supers) for fast referrer scanning in `trace_referrers.py`.
+- **Reverse tracing = repeated full streaming passes.** Each hop re-scans the file, testing
+  every object's references against the current target id-set. Slower than building an
+  in-memory graph but uses trivial memory and is robust on huge dumps.
+
+## Extending the scripts
+
+Common one-off questions and where to hook in:
+
+- **Print every field of a specific object id** → in a heap walk, when
+  `INSTANCE_DUMP.object id == target`, decode each field with the class's
+  `layout` (see `inspect_objects.py`'s `read_val` + `layout`).
+- **Enumerate a map's keys/values** → read the map's `table` (a `Node[]`), then for each
+  non-null `Node` follow `key`/`val`/`next`; resolve value object classes via the
+  class-id → name table.
+- **Retained-style grouping** → for a target class, in one pass collect each instance's
+  outgoing refs, then attribute their shallow sizes (approximate; real dominator/retained
+  needs MAT).
+
+Keep the GC-root and CLASS_DUMP skip logic byte-exact — a single miscounted field
+desynchronizes the entire stream. The scripts raise on an unknown sub-tag, which is the
+canary for a misalignment.

package/src/core-skills/xiaoma-heap-dump-analysis/resources/mat-headless-runbook.md

@@ -0,0 +1,108 @@
+# Eclipse MAT — Headless Runbook (and pitfalls)
+
+Eclipse Memory Analyzer (MAT) is the gold standard for `.hprof` analysis: it builds a
+dominator tree and produces a **Leak Suspects** report. This skill uses it only as
+**cross-validation** for the pure-Python scripts — but when it runs, it is authoritative.
+This runbook captures the exact invocation and the traps that make it fail *silently*.
+
+## Install
+
+macOS (Apple Silicon or Intel), via Homebrew:
+
+```bash
+brew install --cask mat          # cask is named "mat" (Eclipse Memory Analyzer)
+# installs to /Applications/MemoryAnalyzer.app
+```
+
+Other platforms: download the **Standalone / RCP** build from
+<https://eclipse.dev/mat/download/> (needs a JDK 17+ on PATH).
+
+## ⚠️ Pitfall 1 — the macOS `.app` launcher exits 14, silently
+
+Running `…/MemoryAnalyzer.app/Contents/Eclipse/ParseHeapDump.sh` (which calls the
+`Contents/MacOS/MemoryAnalyzer` binary) from a **headless shell** (no GUI/WindowServer
+session) fails with **exit code 14, zero output, no index files**. The Cocoa launcher
+forces `-XstartOnFirstThread` and cannot reach the window server.
+
+**Do not fight `ParseHeapDump.sh`.** Invoke the Equinox launcher jar directly with your own
+JDK — fully headless, no GUI bootstrap:
+
+```bash
+ECL=/Applications/MemoryAnalyzer.app/Contents/Eclipse
+LAUNCHER=$(ls "$ECL"/plugins/org.eclipse.equinox.launcher_*.jar)
+
+java -Xmx12g \
+  --add-exports=java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED \
+  -Djava.awt.headless=true \
+  -Dosgi.install.area="file:$ECL/" \
+  -Dosgi.configuration.area=file:/tmp/mat_cfg/ \
+  -jar "$LAUNCHER" \
+  -consoleLog -nosplash -data /tmp/mat_ws \
+  -application org.eclipse.mat.api.parse \
+  '/path/to/dump_clean.hprof' \
+  org.eclipse.mat.api:suspects org.eclipse.mat.api:top_components
+```
+
+Notes:
+
+- `-Dosgi.install.area` / `-Dosgi.configuration.area` are **required** — without them the
+  launcher loads no MAT bundle and the application is a no-op (the same silent exit 14).
+  Point `configuration.area` at a writable dir (the install `configuration/` is read-only).
+- `-Xmx`: give MAT roughly **the dump size or more** (12g for a 2.6 GB dump is comfortable;
+  MAT's indexing is efficient but needs headroom). If you instead use `ParseHeapDump.sh` on
+  Linux/CI, raise `-Xmx` by editing `MemoryAnalyzer.ini`'s `-vmargs` block.
+- Report ids: `org.eclipse.mat.api:suspects` (Leak Suspects),
+  `org.eclipse.mat.api:top_components`, `org.eclipse.mat.api:overview`.
+
+## ⚠️ Pitfall 2 — special characters in the dump filename
+
+`%`, spaces, etc. (common from `-XX:HeapDumpPath=app_%p.hprof`) break Equinox/URI handling
+and cause the *same* silent exit 14. **Always analyze a clean-named hardlink** (see the
+skill's Prerequisites). This also protects the data if the original is later deleted.
+
+## ⚠️ Pitfall 3 — Gatekeeper quarantine (macOS)
+
+A freshly cask-installed app may be quarantined. If launch is blocked:
+
+```bash
+xattr -dr com.apple.quarantine /Applications/MemoryAnalyzer.app
+```
+
+## Outputs
+
+A successful parse writes, next to the dump:
+
+- Index files: `*.index`, `*.idx.index`, `*.domIn.index`, `*.domOut.index`,
+  `*.o2ret.index`, `*.threads`, … (the dominator tree + retained-size indices).
+- Report zips: `<name>_Leak_Suspects.zip`, `<name>_Top_Components.zip`,
+  `<name>_System_Overview.zip`.
+
+The presence of `*.index` files is the reliable "parsing actually started" signal when
+watching progress.
+
+## Reading the report headlessly
+
+The zips are HTML. Convert to text with macOS `textutil` (or any html-to-text tool):
+
+```bash
+cd /tmp && rm -rf leak && mkdir leak && cd leak
+unzip -o '/path/to/<name>_Leak_Suspects.zip' >/dev/null
+textutil -convert txt -stdout index.html | sed '/^[[:space:]]*$/d' | head -120
+```
+
+What to extract from **Leak Suspects**:
+
+- **Problem Suspect 1**: "one instance of `<HolderClass>` … occupies `N` bytes (`P`%) …"
+  → the dominator and its retained-heap share.
+- "The memory is accumulated in one instance of `<X>[capacity]`" → the accumulation point
+  (often a `ConcurrentHashMap$Node[]` / `Object[]` table).
+- "Thread `…` has a local variable … on the shortest path to …" → the GC-root path.
+
+Confirm this holder matches what `trace_referrers.py` converged on. Detailed dominator
+trees and per-object paths live in `pages/*.html` and `_Top_Components` / `_System_Overview`.
+
+## Quick decision
+
+- MAT runs → use it to corroborate + get retained sizes. Strongest evidence.
+- MAT won't run after ~2 attempts → **don't keep fighting it.** The pure-Python Stages 1/2/4
+  fully locate and prove the holder chain on their own.

package/src/core-skills/xiaoma-heap-dump-analysis/resources/methodology.md

@@ -0,0 +1,148 @@
+# Heap Leak Hunting — Methodology
+
+The single idea behind this skill: **a histogram tells you *what* is numerous; it does not
+tell you *who retains it* or *why it is never freed*.** Generic containers (`char[]`,
+`String`, `HashMap$Node`, `Object[]`) always top the histogram and almost never name the
+bug. Root-causing a leak means walking *up the reference graph* to the GC-root holder, then
+explaining the *mechanism* that keeps the collection growing.
+
+Every conclusion should be reached by **at least two independent routes** so it is not an
+artifact of one tool.
+
+---
+
+## Stage 1 — Histogram: find the abnormal class
+
+Run `scripts/hprof_histogram.py`. Ignore the all-classes board first; read the
+**business / third-party board (JDK excluded)**. You are hunting for a *domain* class whose
+instance count is wildly larger than the number of real things it should represent.
+
+**Anchor the count against reality.** Pick objects whose count equals "live work":
+
+- `io.netty.channel.socket.nio.NioSocketChannel`, `sun.nio.ch.SocketChannelImpl`,
+  `java.io.FileDescriptor` → number of live TCP connections / sockets.
+- a domain "online user / active session" object, thread-pool size, etc.
+
+If a session/handler/listener/entry class has, say, 190k instances while live connections
+are 40k and active users are 6k, the ~180k surplus is **retained zombies**. That class is
+your Stage-2 suspect.
+
+Cross-check internal consistency: leaked parents usually drag fixed-ratio children. If
+`SuspectA = N`, look for classes at `≈N`, `≈2N`, `≈k·N` — they confirm a whole subtree
+leaks together (e.g. each leaked session keeps 1 handshake + 2 transport states + 1 store).
+
+---
+
+## Stage 2 — Reverse-reference trace: who retains them
+
+Run `scripts/trace_referrers.py <dump> <suspect-class>`. It implements MAT's
+"path to GC roots" in pure Python: from every instance of the suspect, find referrers via
+**instance fields, array elements, and static fields**, hop by hop, until reaching a
+GC root or a single accumulation container.
+
+Read each hop for:
+
+- **The dominant referrer field.** `('java.util.concurrent.ConcurrentHashMap$Node', 'val')`
+  with a count ≈ suspect count means "they live as *values* of a ConcurrentHashMap."
+  Next hops climb `Node → Node[] table → ConcurrentHashMap → holder.field`.
+- **★ static field holder** — a `static` field directly references the set. This is the
+  smoking gun for static-registry / static-cache leaks; the holder is a GC root by itself.
+- **★ referrer is itself a GC root** — thread (local var / thread object), JNI global,
+  sticky class.
+
+**Convergence is the goal.** You are looking for the hop where many objects funnel into
+*one* container instance (one big `Node[]` table, one singleton holder). That container's
+owning field is the retention point.
+
+**Cycles are expected.** Frameworks wire back-references (child holds a pointer to the
+registry that holds the child), so the reverse-BFS frontier *grows* after a few hops. Don't
+chase the ballooning frontier — read the *early* hops and the ★ anchors.
+
+---
+
+## Stage 3 — MAT cross-validation (optional)
+
+If Eclipse MAT runs (see `mat-headless-runbook.md`), its **Leak Suspects** report should
+independently name the same holder ("Problem Suspect 1: one instance of X occupies N% …
+accumulated in one `Node[capacity]`"). Two unrelated methods agreeing turns a strong
+inference into a confident finding. MAT also gives **retained size** (true cost of the
+subtree), which the histogram's shallow size cannot.
+
+---
+
+## Stage 4 — Precise measurement: turn inference into numbers
+
+Run `scripts/inspect_objects.py` to:
+
+- **Measure the suspect collection's true size** — `--map-fields`. Distinguish the real
+  leak map (100k+ entries, huge power-of-two table) from innocent siblings (thousands).
+- **Read the runtime config that governs cleanup** — `--fields`. Heartbeat/timeout values,
+  feature flags, max sizes. A misconfigured timeout (e.g. 100× the default) often explains
+  *why* cleanup never fires.
+
+> **Trust non-empty bucket count + table capacity, not `size`/`baseCount`.**
+> `ConcurrentHashMap` spreads its counter across `counterCells` under contention, so
+> `baseCount` can read absurdly low (e.g. 552 for a map with ~190k entries). The
+> `Node[]` table capacity (a power of two) and the non-empty bucket count are reliable.
+> The script flags this automatically (⚠️) when `size` and non-empty buckets differ by >10×.
+
+When comparing sibling collections, remember **they may hold different things**: a registry
+keyed by session-id can be huge while a sibling keyed by live channel stays small (it only
+holds currently-connected transports, not every session). Don't expect every map's size to
+equal the live-connection count — judge each against *what it is supposed to hold*. The leak
+is the one whose size has no business being that large.
+
+---
+
+## Stage 5 — Mechanism & report
+
+Locating the holder is half the job; explain **why it grows unbounded**:
+
+- **Static cache / registry never pruned** — entries added on event A, removal depends on
+  event B that doesn't always happen.
+- **Listener / callback never deregistered** — observer holds the subject (or vice-versa).
+- **Connection / session not removed on disconnect** — cleanup depends on a timeout or an
+  explicit close callback that some paths skip.
+- **Unbounded queue / buffer** — producer outruns consumer.
+- **`ThreadLocal` on a pooled thread** — value outlives the request.
+- **`ClassLoader` leak** — a long-lived object pins a webapp/plugin classloader.
+- **Misconfigured timeout / TTL** — cleanup mechanism exists but is effectively disabled
+  (set to 0) or set so large it never fires.
+
+To pin the mechanism, read the owning library's source for the **add/remove lifecycle** of
+that exact collection, and quote the method names. State plainly what is *proven from the
+heap* vs *inferred*.
+
+---
+
+## Worked example (anonymized, real case)
+
+> This is a real, anonymized case shown to illustrate the flow end-to-end. For a *new* dump,
+> derive every number yourself with the scripts — **do not pattern-match your conclusion to
+> this example.** A different leak will have a different holder class, field, and mechanism;
+> the *method* transfers, the answer does not.
+
+**Symptom.** 2.6 GB dump, ~40M objects. Histogram: a Socket.IO session class `ClientHead`
+= 192,639; live `NioSocketChannel` = 41,773; active session-wrapper = 6,530. → ~186k
+zombie sessions.
+
+**Reverse trace.** `ClientHead ← ConcurrentHashMap$Node.val (194,844) ← Node[] table ←
+ClientsBox (singleton) ← GC root (Netty I/O thread → SocketIOChannelInitializer)`. Sibling
+ratios confirmed: `TransportState ≈ 2× ClientHead`, message queues `≈ 2× ClientHead`.
+
+**MAT.** Leak Suspects: "one `ClientsBox` occupies 64.67% … accumulated in
+`ConcurrentHashMap$Node[524288]`." Same holder. ✓
+
+**Measure.** `ClientsBox.uuid2clients`: table 524288, non-empty buckets 160,989 (≈190k);
+`channel2clients`: 2,027; business static maps: 939 / 14,496. → leak is **only**
+`uuid2clients`. Config: `pingTimeout=120000` (2× default), `upgradeTimeout=1,000,000` (100×
+default).
+
+**Mechanism.** `AuthorizeHandler.authorize()` does `clientsBox.addClient()` on handshake;
+removal depends on ping-timeout or explicit disconnect. Only ~18k timeout tasks exist for
+190k sessions → most zombies have no pending cleanup. The blown-up `upgradeTimeout`
+amplifies retention of mid-upgrade/aborted polling connections.
+
+**Fix.** Upgrade the library; restore `upgradeTimeout`/`pingTimeout` to sane values; add
+ingress rate-limit/auth to stop invalid handshakes; monitor `uuid2clients.size`; mitigate
+with heap bump + rolling restart / scheduled stale-session cleanup.