@chrono-meta/fh-gate 1.4.71 → 1.4.73

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. package/.claude/rules/.public-surface-patterns.defaults +44 -0
  2. package/.claude/rules/fh_4axis_gate.md +207 -0
  3. package/.claude-plugin/marketplace.json +2 -2
  4. package/AGENTS.md +26 -2
  5. package/CATALOG.md +59 -0
  6. package/README.ja.md +1 -1
  7. package/README.ko.md +1 -1
  8. package/README.md +1 -1
  9. package/README.zh.md +1 -1
  10. package/knowledge/shared/harness-core/measurement-integrity-checklist.md +10 -0
  11. package/knowledge/shared/learnings/subagent_invocations_log.yaml +554 -0
  12. package/package.json +21 -1
  13. package/plugins/fh-commons/.claude-plugin/plugin.json +1 -1
  14. package/plugins/fh-meta/.claude-plugin/plugin.json +2 -2
  15. package/plugins/fh-meta/skills/context-doctor/SKILL.md +42 -4
  16. package/plugins/fh-meta/skills/context-doctor/SKILL_detail.md +38 -0
  17. package/plugins/fh-meta/skills/salience-splitter/SKILL.md +1 -1
  18. package/scripts/chamber_candidate_collect.sh +223 -0
  19. package/scripts/degrade_direction_scan.sh +222 -0
  20. package/scripts/fh-gate.sh +76 -2
  21. package/scripts/fh_session_load.sh +202 -0
  22. package/scripts/gate_pathspec_check.sh +166 -0
  23. package/scripts/prepush_guard_check.sh +374 -0
  24. package/scripts/psa_scan_lib.sh +153 -0
  25. package/scripts/public_surface_scan_files.sh +157 -0
  26. package/scripts/selfcheck.sh +16 -0
  27. package/scripts/session_close_check.sh +171 -0
  28. package/scripts/test_degrade_scan_shell_probes.sh +185 -0
  29. package/scripts/test_fh_gate_regressions.sh +46 -2
  30. package/scripts/test_prepush_stdin_integrity.sh +119 -0
  31. package/scripts/universal_guard_check.sh +280 -0
  32. package/templates/.claude/rules/mcp_tool_gating.md +157 -0
  33. package/templates/.git-hooks/pre-commit +848 -0
  34. package/templates/.git-hooks/pre-push +585 -0
  35. package/templates/PRE-PUBLISH-CHECKLIST.md +85 -0
  36. package/templates/degrade_direction_scan.sh +222 -0
  37. package/templates/predelete_check.sh +72 -0
  38. package/templates/regression_guard.sh +563 -0
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "fh-commons",
3
- "version": "1.4.71",
3
+ "version": "1.4.73",
4
4
  "engines": {
5
5
  "claudeCode": ">=1.0.0"
6
6
  },
@@ -1,10 +1,10 @@
1
1
  {
2
2
  "name": "fh-meta",
3
- "version": "1.4.71",
3
+ "version": "1.4.73",
4
4
  "engines": {
5
5
  "claudeCode": ">=1.0.0"
6
6
  },
7
- "description": "Hub meta-engineering toolkit — 35 skills + 7 agents. New in 1.4.71: SKILL_detail.md brought inside the 4-axis gate — the gate matched the literal `SKILL.md`, which `SKILL_detail.md` does not contain, leaving 27.7% of the skill-spec surface ungated (measured; it leaked twice for real). Fix escalated from a name list to a directory scope after an enumeration sweep found a real uncovered file on its first run; anchored by scripts/gate_pathspec_check.sh (known-pair, wired into pre-commit). One-way mirror sync gains a destination-newer abort + mechanically injected mirror banner after two session cards were silently overwritten, anchored by scripts/sync_guard_check.sh. dialogue-harvest gains Step 4-b cross-corpus provenance (absorbed / held-unused / declined) for single-author corpora, where the original provenance labels were free and measured nothing. frontier-digest arxiv leg category-scoped after a relevance drift the staleness rule structurally could not catch (known-pair calibration recorded). templates/CLAUDE.md hub paths switched to {FH_ROOT} so a copied harness stops carrying one operator's absolute paths into org-visible checkouts. New in 1.4.70: dialogue-harvest (mines AI-dialogue logs — sycophancy strip first, induced-vs-independent provenance labeling, EN+KO known-pair calibration shipped); asset-placement-gate Step 0.6 official-corpora check (Claude Cookbook as Tier-0 consult corpus, judged-flag semantics); context-doctor built-in-/doctor-first sister anchor. New in 1.4.48: phantom-quench + steel-quench gain external frontier anchors (arXiv:2607.02052 package-hallucination; arXiv:2607.02057 prompt-coverage-adequacy); README model-flat claim reframed from a per-release point-curve to structural invariants (operation flattens across tiers; depth tier-order fixed within a generation). New in 1.4.47: onboarding step ① surfaces the Mode D companion-store session-start load in the auto-read salience anchor (previously only in the local binding + rules, so a greeting could skip the load). New in 1.4.46: context-doctor gains a command-output axis — routes to a command-output proxy/hook (rtk) to trim verbose CLI stdout, complementing .claudeignore; risk-gated to token-scarce environments (lossy filtering, off gate-input paths). New in 1.4.41: context-doctor 2026 trigger vocab (context engineering/rot/collapse) + phantom-citation hardening; hub measurement-integrity-checklist (cross-model measurement pre-flight: display-name pin/reps≥3/discriminating probe). New in 1.4.40: install-wizard scaffolds the companion store as a queryable wiki (INDEX + session-start read + Raw/Wiki/Conversation ingest axis). New in 1.4.39: auto-decorrelation (cross-family verifier sidecar recruitment, calibration-gated) + video-ingest (capability-routed video ingestion). New in 1.4.37: corpus-grounding-expander + persona-roster-expander (field-harvested verbatim-relay capability skills). New in 1.3.0: public-surface-audit (git-tracked private-token leak scan), field-harvest Mode B session-end auto-trigger, 4-axis gate scope extension (docs/ + AGENTS.md). New in 1.2.0: pipeline-conductor (4-pipeline gated sweep), return-path-gate (chain closure audit), goal-quench (Stop hook + quality gate), steel-quench Wave 5 (multi-model sidecar challenger), 2-layer architecture docs, YAML validation script. Validated cross-CLI: Claude Code, Codex, Gemini.",
7
+ "description": "New in 1.4.73: the mechanical floor now ships — templates/.git-hooks (pre-commit/pre-push) plus every script they actually invoke, their regression anchors, and the .claude/rules data they read. Until now the package shipped the prose that tells you to run the gates while omitting the gates, and 35 shipped-doc references pointed at files the tarball did not contain. Verified by unpacking the tarball, installing the hooks in a scratch repo and committing through them, not by inspecting files[]. degrade_direction_scan gains shell probes: it collected .sh files that no probe could match and reported them clean (known-positive scored 0/4); git hooks were invisible entirely because the extension test ran against the path, not the basename. The field-propagated templates/ copy had drifted from scripts/ and is now byte-pinned by an anchor. Hub meta-engineering toolkit — 35 skills + 7 agents. New in 1.4.71: SKILL_detail.md brought inside the 4-axis gate — the gate matched the literal `SKILL.md`, which `SKILL_detail.md` does not contain, leaving 27.7% of the skill-spec surface ungated (measured; it leaked twice for real). Fix escalated from a name list to a directory scope after an enumeration sweep found a real uncovered file on its first run; anchored by scripts/gate_pathspec_check.sh (known-pair, wired into pre-commit). One-way mirror sync gains a destination-newer abort + mechanically injected mirror banner after two session cards were silently overwritten, anchored by scripts/sync_guard_check.sh. dialogue-harvest gains Step 4-b cross-corpus provenance (absorbed / held-unused / declined) for single-author corpora, where the original provenance labels were free and measured nothing. frontier-digest arxiv leg category-scoped after a relevance drift the staleness rule structurally could not catch (known-pair calibration recorded). templates/CLAUDE.md hub paths switched to {FH_ROOT} so a copied harness stops carrying one operator's absolute paths into org-visible checkouts. New in 1.4.70: dialogue-harvest (mines AI-dialogue logs — sycophancy strip first, induced-vs-independent provenance labeling, EN+KO known-pair calibration shipped); asset-placement-gate Step 0.6 official-corpora check (Claude Cookbook as Tier-0 consult corpus, judged-flag semantics); context-doctor built-in-/doctor-first sister anchor. New in 1.4.48: phantom-quench + steel-quench gain external frontier anchors (arXiv:2607.02052 package-hallucination; arXiv:2607.02057 prompt-coverage-adequacy); README model-flat claim reframed from a per-release point-curve to structural invariants (operation flattens across tiers; depth tier-order fixed within a generation). New in 1.4.47: onboarding step ① surfaces the Mode D companion-store session-start load in the auto-read salience anchor (previously only in the local binding + rules, so a greeting could skip the load). New in 1.4.46: context-doctor gains a command-output axis — routes to a command-output proxy/hook (rtk) to trim verbose CLI stdout, complementing .claudeignore; risk-gated to token-scarce environments (lossy filtering, off gate-input paths). New in 1.4.41: context-doctor 2026 trigger vocab (context engineering/rot/collapse) + phantom-citation hardening; hub measurement-integrity-checklist (cross-model measurement pre-flight: display-name pin/reps≥3/discriminating probe). New in 1.4.40: install-wizard scaffolds the companion store as a queryable wiki (INDEX + session-start read + Raw/Wiki/Conversation ingest axis). New in 1.4.39: auto-decorrelation (cross-family verifier sidecar recruitment, calibration-gated) + video-ingest (capability-routed video ingestion). New in 1.4.37: corpus-grounding-expander + persona-roster-expander (field-harvested verbatim-relay capability skills). New in 1.3.0: public-surface-audit (git-tracked private-token leak scan), field-harvest Mode B session-end auto-trigger, 4-axis gate scope extension (docs/ + AGENTS.md). New in 1.2.0: pipeline-conductor (4-pipeline gated sweep), return-path-gate (chain closure audit), goal-quench (Stop hook + quality gate), steel-quench Wave 5 (multi-model sidecar challenger), 2-layer architecture docs, YAML validation script. Validated cross-CLI: Claude Code, Codex, Gemini.",
8
8
  "author": {
9
9
  "name": "chrono-meta",
10
10
  "email": "chrono-meta@users.noreply.github.com"
@@ -20,7 +20,7 @@ Diagnoses the main causes of session token waste and prescribes immediate remedi
20
20
  3. Not using `/clear` after direction changes → continuing work with accumulated noise
21
21
  4. Verbose CLI output → every `git`/`ls`/build/test call floods context with stdout (a *different* layer from 1–3 — see §Command-Output Reduction)
22
22
 
23
- **Two reduction axes — keep them distinct.** Causes 1–3 are the **context-footprint** axis (what gets read *into* context: files, history). Cause 4 is the **command-output** axis (tokens produced *by* the tools you run). `.claudeignore` cannot touch command output, and a command-output proxy cannot touch file reads — they are complementary, not substitutes.
23
+ **Two reduction axes — keep them distinct.** Causes 1–3 are the **context-footprint** axis (what gets read *into* context: files, history). Cause 4 is the **command-output** axis (tokens produced *by* the tools you run). `.claudeignore` cannot touch command output, and a command-output proxy cannot touch file reads — they are complementary, not substitutes. (Step 3.5 below adds an optional **third**, orthogonal lever — provider-side cache billing — distinct from both.)
24
24
 
25
25
  **Standalone install** — this skill works normally with plugin install only, without cloning the full meta-harness.
26
26
 
@@ -111,6 +111,39 @@ When context is near the limit and you want to *preserve state* rather than rese
111
111
 
112
112
  > Models are tools for allocating the right expertise to complexity. Opus for simple tasks = wasted expertise; Haiku for design decisions = insufficient expertise. Switch models when task nature changes.
113
113
 
114
+ ### Step 3.5. Cache-Boundary Audit (prompt caching, optional third lever)
115
+
116
+ Distinct from both axes above: Steps 1–3 reduce what occupies the context *window*; this
117
+ reduces what the provider re-bills as **fresh input tokens** on each turn by preserving
118
+ prompt-cache hits. **Optional** — run it when a metered/quota-limited backend makes cache
119
+ cost the binding constraint (same gating logic as §Command-Output Reduction: diagnose and
120
+ recommend, do not chase this when tokens are merely plentiful).
121
+
122
+ **Audit checklist**:
123
+ - Is the system prompt / CLAUDE.md content **stable byte-for-byte** across turns in a
124
+ session? Any turn-to-turn diff in the fixed prefix invalidates the cache for that prefix —
125
+ which is in tension with Step 5's CLAUDE.md-compression prescription: compress at session
126
+ boundaries (before a fresh cache is built), never mid-session (which would invalidate one
127
+ already warm).
128
+ - Is session-specific / dynamic content (task state, recent results) placed at the
129
+ **end of the user message**, not interleaved into the system prompt or early context?
130
+
131
+ **Reported, UNCALIBRATED by FH** (2026-07-27 frontier digest, issue #102): input:output token
132
+ ratio **> 10:1** favors cache/context engineering over model-level optimization; **> 50:1**,
133
+ prefix caching dominates; cache-boundary control reportedly raised hit rate ~7%→84% in one
134
+ production case (arXiv:2603.09619 *Context Engineering: From Prompts to Corporate Multi-Agent
135
+ Architecture* · appscale.blog).
136
+
137
+ **What is and is not verified** — the two are separate checks and only one has been run:
138
+ the cited paper's **existence** was confirmed 2026-07-28 (arxiv.org/abs/2603.09619 → HTTP 200,
139
+ title matches; measured alongside a known-real control ID, so the check itself is calibrated).
140
+ The **figures** above were not read out of the paper — they are still traced only to the digest
141
+ comment. Treat the ratio thresholds as illustrative, not a calibrated gate, until the source
142
+ text is read directly.
143
+
144
+ > **Detail**: See `SKILL_detail.md §CacheBoundary` — full citation text and the 403 grounding
145
+ > note — read before citing these figures elsewhere.
146
+
114
147
  ### Step 4. harvest-loop Integration (burst pattern recording)
115
148
 
116
149
  When burst pattern is detected and `tracks/_audit/` exists: locate the latest weekly_audit file and suggest adding the Token Efficiency Check items to it (bash + checklist block in §Step-Bash).
@@ -132,6 +165,8 @@ Run the audit bash (§Step-Bash) and apply thresholds:
132
165
 
133
166
  **Frequency**: When explicitly called with `/context-doctor` or auto-invoked at session start when MEMORY.md is detected at 180+ lines.
134
167
 
168
+ **Cache-boundary note**: apply CLAUDE.md compression (above) at a session boundary, not mid-session — see Step 3.5's cache-invalidation tension if that step is also in scope.
169
+
135
170
  ## Context Hierarchy (L1/L2/L3)
136
171
 
137
172
  Information buried in the middle of a long context window suffers measurable accuracy loss — the "lost in the middle" effect, ~10–30% degradation for mid-context information (see `../../../../knowledge/shared/harness-core/harness_frontier_diagnosis_2026-06-02.md`). The remedy is two-fold: tier the context, and place the most important instructions at the **start and end**, never buried in the middle.
@@ -187,8 +222,8 @@ The reductions above all act on the **context-footprint** axis (files, history r
187
222
 
188
223
  | Environment | Behavior |
189
224
  |---|---|
190
- | Meta-harness cloned (Mode A) | Perform full Steps 1–5 / integrate with harvest-loop files |
191
- | Plugin only (Mode C) | Perform Steps 1–3 / Step 4 output only (no file writes) |
225
+ | Meta-harness cloned (Mode A) | Perform full Steps 1–5 (+ optional Step 3.5) / integrate with harvest-loop files |
226
+ | Plugin only (Mode C) | Perform Steps 1–3 (+ optional Step 3.5, no file writes needed) / Step 4 output only (no file writes) |
192
227
  | External general environment | Focus on `.claudeignore` generation + large file guidance |
193
228
 
194
229
  ## Invocation Triggers
@@ -231,6 +266,7 @@ Explicit invocation (`/context-doctor`) always runs regardless of suppress state
231
266
  - "context diet", "memory audit", "CLAUDE.md is heavy", "MEMORY.md size"
232
267
  - "context engineering", "context rot", "context collapse"
233
268
  - "command output is huge", "verbose output", "rtk", "token killer", "trim command output"
269
+ - "prompt caching", "cache hit rate", "cache boundary", "why are my API costs so high"
234
270
 
235
271
  ### Natural Language Triggers (activates without internal vocabulary)
236
272
 
@@ -247,6 +283,7 @@ Also activates when an external user expresses without token/context terminology
247
283
  | "Context is getting full", "context meter is high" | Approaching context limit | Step 3 — propose Wrap-then-Compact pattern |
248
284
  | "context engineering", "doing context engineering", "context rot setting in" | 2026 industry term for context discipline (Chroma 2025 / Anthropic) | Step 2 + Step 3 |
249
285
  | "every git command dumps a wall of text", "the build output eats my context" | Verbose command output flooding context | §Command-Output Reduction (route to proxy/hook) |
286
+ | "our token bill is high but context looks fine", "cache keeps missing" | Suspected cache-boundary invalidation, not context bloat | Step 3.5 (optional) |
250
287
 
251
288
  ## Three-Doctor Loop Integration
252
289
 
@@ -277,9 +314,10 @@ context-doctor (token/context) · harness-doctor (structure) · sim-conductor (s
277
314
 
278
315
  | Condition | Completion verdict |
279
316
  |---|---|
280
- | Diagnosis results output to conversation (relevant stages among Steps 1~5) | ✅ Diagnosis complete |
317
+ | Diagnosis results output to conversation (relevant stages among Steps 1~5, incl. optional 3.5) | ✅ Diagnosis complete |
281
318
  | `.claudeignore` created or modified + path output | ✅ Prescription complete |
282
319
  | Large file detected with split strategy guidance output | ✅ Step 2 complete |
320
+ | Step 3.5 run: checklist findings output, thresholds labeled UNCALIBRATED per source above | ✅ Cache-boundary audit complete (only when Step 3.5 was triggered) |
283
321
  | "No context structure issues" judgment output | ✅ Health check complete |
284
322
 
285
323
  **This skill's Done When = "diagnosis report output complete"**. Actual resolution of prescription items is in the user's or follow-up work domain and is not included in this skill's completion criteria.
@@ -93,6 +93,44 @@ CC Context Audit Results
93
93
 
94
94
  ---
95
95
 
96
+ ## §CacheBoundary — prompt-caching evidence (Step 3.5, provisional)
97
+
98
+ Full citation for the Step 3.5 evidence line, kept out of the always-read SKILL.md body
99
+ per this file's own role (detail loads on-demand, SKILL.md stays lean).
100
+
101
+ **Source**: forge-harness GitHub issue #102 ("Frontier Digest Log"), 2026-07-27 daily comment,
102
+ itself citing:
103
+ - arXiv:2603.09619 — reported input:output token-ratio thresholds (>10:1 favors context/cache
104
+ engineering over model-level optimization; >50:1, prefix caching dominates) and a
105
+ 41–80% cost reduction / 13–31% TTFT improvement range attributed to prompt caching.
106
+ - appscale.blog/en/blog/context-engineering-production-llm-agents-token-budget-compaction-2026
107
+ — the ~7%→84% cache-hit-rate figure from cache-boundary control (fixed system prompt,
108
+ dynamic content at the user-message end).
109
+
110
+ **Grounding status — split verdict (2026-07-28 re-check).** Two separate questions; only one
111
+ is closed. Do not collapse them:
112
+
113
+ | Question | Verdict | Evidence |
114
+ |---|---|---|
115
+ | Does the cited paper exist? | ✅ **VERIFIED** | `arxiv.org/abs/2603.09619` → HTTP 200, title *"Context Engineering: From Prompts to Corporate Multi-Agent Architecture"*. Measured alongside a known-real control ID (1706.03762 → 200), so the instrument is calibrated for this check. |
116
+ | Do the paper/blog actually state these figures? | ❌ **UNVERIFIED** | The source text was never read. The numbers are traced only to the digest comment. |
117
+
118
+ ⚠️ **Instrument note (kept because it nearly produced a wrong verdict).** The first re-check
119
+ used the arXiv **API** (`export.arxiv.org/api/query`) and returned empty for the target — but
120
+ also empty for the known-real control, i.e. the instrument was dead, not the paper missing.
121
+ An absence measured without a live control is not evidence. Switching to the `abs` page gave a
122
+ working control and flipped the existence verdict to VERIFIED. Original 403 note below for
123
+ provenance.
124
+
125
+ **Original (2026-07-27) note**: both URLs returned HTTP 403 when that session attempted
126
+ to fetch and verify the claimed spans directly (`WebFetch`), matching the same sandbox
127
+ network-policy limitation the daily frontier-digest routine logged on every run this week
128
+ (WebSearch fallback, no direct API access). The figures above are therefore traced only to
129
+ the digest comment's own text — a real, read GitHub source — and are **not** independently
130
+ re-verified against the original paper/post. Per the Instrument-Calibration doctrine
131
+ (`CLAUDE.md`), do not treat the 10:1 / 50:1 thresholds as a calibrated gate and do not cite
132
+ the ~7%→84% figure elsewhere without re-fetching from an environment with access first.
133
+
96
134
  ## §Headroom — Tooling (external option)
97
135
 
98
136
  The compression pass in SKILL.md is tool-agnostic, but a concrete, reversible, local-first implementation exists: **Headroom** (`github.com/chopratejas/headroom`, open source, v0.22). It compresses tool outputs, logs, files, and RAG chunks before they reach the LLM — **vendor/coverage-reported** at 60–95% fewer tokens with the same answers ([The Register, 2026-05-31](https://www.theregister.com/ai-ml/2026/05/31/netflix-wiz-creates-app-to-slash-ai-bills-then-open-sources-it/5248702); figures unverified by FH). General token-efficiency basis: `../../../../knowledge/shared/harness-core/harness_frontier_diagnosis_2026-06-02.md`.
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: salience-splitter
3
- description: Splits an over-loaded always-loaded context asset — a SKILL.md, CLAUDE.md, or memory index — into a lean always-loaded layer + an on-demand layer, using a governance-semantic criterion (not length, but when the content is needed), connected by imperative pointers. Based on paper §9.5 Protocol-Priority Split pattern. Diagnoses, classifies, splits, and verifies in one pass. Renamed from skill-splitter (old name still routes here). Triggers: "SKILL.md too large", "split this skill", "skill is bloated", "skill file too long", "CLAUDE.md 너무 커".
3
+ description: 'Splits an over-loaded always-loaded context asset — a SKILL.md, CLAUDE.md, or memory index — into a lean always-loaded layer + an on-demand layer, using a governance-semantic criterion (not length, but when the content is needed), connected by imperative pointers. Based on paper §9.5 Protocol-Priority Split pattern. Diagnoses, classifies, splits, and verifies in one pass. Renamed from skill-splitter (old name still routes here). Triggers: "SKILL.md too large", "split this skill", "skill is bloated", "skill file too long", "CLAUDE.md 너무 커".'
4
4
  user-invocable: true
5
5
  allowed-tools: ["Read", "Write", "Edit", "Bash", "Grep", "Glob"]
6
6
  model: sonnet
@@ -0,0 +1,223 @@
1
+ #!/usr/bin/env bash
2
+ # chamber_candidate_collect.sh — chamber candidate discovery: 6-source converge → dedup → rank → screen.
3
+ #
4
+ # The second half of the discovery entrance (the first half = chamber_candidate_screen.sh, the reinvention
5
+ # first-pass). Design home: tracks/_meta/fh_signal_2026-07-14_self-dev.md (design v2, 2-family verified).
6
+ #
7
+ # WHAT IT DOES: pulls candidate signals from the 6 sources, deduplicates the same gap arriving multiple
8
+ # ways (2-family finding: harvest-loop/fh_signal/UAP observe the SAME session → triple-listing), ranks by
9
+ # source-diversity + frequency, and runs each survivor through the reinvention screener. Output = a ranked
10
+ # candidate queue for the operator's HITL "run the chamber?" decision (judged uncertainty/failure-cost
11
+ # filters stay with the human — this tool does the mechanizable converge/dedup/rank/screen).
12
+ #
13
+ # CANDIDATE-EMISSION CONVENTION (why a convention, not source-specific parsing): the 2-family review
14
+ # flagged per-source parsing as fragile/hand-wave. So a source declares a candidate with ONE robust line:
15
+ # CHAMBER-CANDIDATE: <one-line description of the capability/project to incubate>
16
+ # Any of the 6 sources adds this line when it surfaces something chamber-worthy. Sources adopt it
17
+ # incrementally — until they do, the queue is honestly SPARSE (the design's "6-source volume = n=0,
18
+ # measured only after wiring"; this tool IS that wiring, and its first run measures the volume).
19
+ #
20
+ # NOT MECHANIZED HERE (stays judged/HITL, by design): uncertainty + failure-cost filters (operator judges
21
+ # the ranked survivors); chamber injection (goal-quench budget gate + HITL). This tool ranks; it does not
22
+ # admit. A high rank is a suggestion, never an auto-run.
23
+ #
24
+ # Usage: bash scripts/chamber_candidate_collect.sh [--min-jaccard N] (default 50)
25
+ # exit 0 = ran (ranked queue on stdout; empty queue is a valid honest result)
26
+ # exit 2 = harness error (FH root not found)
27
+
28
+ set -uo pipefail
29
+
30
+ FH="$(cd "$(dirname "$0")/.." && pwd)"
31
+ if [ ! -d "$FH/tracks/_meta" ] || [ ! -d "$FH/plugins" ]; then
32
+ echo "❌ FH root not found at '$FH' — run from the FH repo." >&2
33
+ exit 2
34
+ fi
35
+ SCREEN="$FH/scripts/chamber_candidate_screen.sh"
36
+ MINJ=50
37
+ [ "${1:-}" = "--min-jaccard" ] && MINJ="${2:-50}"
38
+ case "$MINJ" in ''|*[!0-9]*) echo "❌ --min-jaccard must be an integer 0-100 (got '$MINJ')" >&2; exit 2 ;; esac
39
+
40
+ WORK="${TMPDIR:-/tmp}/cand_collect.$$"
41
+ # fail-CLOSED on workdir failure (Axis-2 challenger MED): an unwritable TMPDIR must NOT masquerade as the
42
+ # design's valid "honest-empty queue" — a genuine infra failure and "no markers surfaced" are otherwise
43
+ # indistinguishable. Route infra failure to exit 2 (harness error), never to green-empty.
44
+ mkdir -p "$WORK" 2>/dev/null || { echo "❌ cannot create workdir under '${TMPDIR:-/tmp}' — cannot collect (NOT an empty queue)" >&2; exit 2; }
45
+ trap 'rm -rf "$WORK"' EXIT
46
+ RAW="$WORK/raw"; : > "$RAW" 2>/dev/null || { echo "❌ cannot write workfile (NOT an empty queue)" >&2; exit 2; }
47
+
48
+ # --- LAYER 1: 6-source converge. Each source = a label + a glob; grep the convention line. ---
49
+ # (source label · file glob) — sources that don't exist yet simply contribute nothing (fail-visible below).
50
+ _pull() { # $1=source-label $2..=files
51
+ local label="$1"; shift
52
+ local f
53
+ for f in "$@"; do
54
+ [ -f "$f" ] || continue
55
+ # convention line: "CHAMBER-CANDIDATE: <desc>" (case-insensitive marker, desc after the colon)
56
+ grep -inE '^[[:space:]]*CHAMBER-CANDIDATE:' "$f" 2>/dev/null \
57
+ | sed -E 's/^[0-9]+:[[:space:]]*CHAMBER-CANDIDATE:[[:space:]]*//I' \
58
+ | while IFS= read -r desc; do
59
+ [ -n "$desc" ] && printf '%s\t%s\n' "$label" "$desc" >> "$RAW"
60
+ done
61
+ done
62
+ }
63
+ SRC_SEEN=0
64
+ for spec in \
65
+ "harness-doctor:$FH/tracks/_meta/*harness_doctor*.md" \
66
+ "harvest-loop:$FH/tracks/_audit/*.md" \
67
+ "fh-signal:$FH/tracks/_meta/fh_signal_*.md" \
68
+ "field-harvest:$FH/tracks/_contrib/*.md" \
69
+ "frontier-digest:$FH/tracks/_meta/frontier_digest_*.md" \
70
+ "uap:$FH/tracks/_meta/user_adaptation_profile.md" ; do
71
+ label="${spec%%:*}"; glob="${spec#*:}"
72
+ # word-split the glob deliberately (globbing); nullglob-safe via the -f check in _pull
73
+ files=$(ls $glob 2>/dev/null || true)
74
+ [ -n "$files" ] && SRC_SEEN=$((SRC_SEEN+1))
75
+ # shellcheck disable=SC2086
76
+ _pull "$label" $files
77
+ done
78
+
79
+ NRAW=$(grep -c . "$RAW" 2>/dev/null); NRAW=${NRAW:-0}
80
+ echo "── chamber candidate collect ──"
81
+ echo "sources present: $SRC_SEEN/6 · raw candidate markers found: $NRAW"
82
+ if [ "$NRAW" -eq 0 ]; then
83
+ echo "queue EMPTY — no 'CHAMBER-CANDIDATE:' markers in any source yet (convention not adopted / nothing surfaced)."
84
+ echo " This is an honest result, not an error: the discovery volume is n=0 until sources emit the marker."
85
+ echo " (Add 'CHAMBER-CANDIDATE: <desc>' to a harness-doctor / fh_signal / frontier-digest / harvest entry.)"
86
+ exit 0
87
+ fi
88
+
89
+ # --- keyword signature per candidate (for dedup) ---
90
+ STOP='^(that|this|from|into|over|when|what|will|your|there|their|then|than|with|chamber|candidate|skill|agent|tool|does|done|after|before|which|about|through)$'
91
+ _sig() { printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9가-힣' ' ' | tr -s ' ' '\n' \
92
+ | awk 'length($0)>=4' | grep -vE "$STOP" | sort -u; }
93
+ # kill-side signature for the seen-filter: keeps 2+ char tokens (a slug's meaningful short tokens like
94
+ # "qa"/"ui" are dropped by _sig's ≥4 filter → an all-short slug yields an EMPTY sig → kkn=0 → the old
95
+ # match never fired → a KILLed candidate re-entered the queue SILENTLY, Axis-2 MED-3b). Drops stopwords only.
96
+ _ksig() { printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9가-힣' ' ' | tr -s ' ' '\n' \
97
+ | awk 'length($0)>=2' | grep -vE "$STOP" | sort -u; }
98
+
99
+ # --- LAYER 2: dedup by greedy jaccard clustering ---
100
+ # cluster files: $WORK/cluster.N holds "source<TAB>desc" lines; $WORK/sig.N holds the union keyword sig.
101
+ NC=0
102
+ idx=0
103
+ while IFS=$'\t' read -r label desc; do
104
+ idx=$((idx+1))
105
+ _sig "$desc" > "$WORK/candsig.$idx"
106
+ merged=0
107
+ c=1
108
+ while [ "$c" -le "$NC" ]; do
109
+ inter=$(comm -12 "$WORK/sig.$c" "$WORK/candsig.$idx" 2>/dev/null | grep -c . || true)
110
+ union=$(sort -u "$WORK/sig.$c" "$WORK/candsig.$idx" 2>/dev/null | grep -c . || true)
111
+ jac=0; [ "${union:-0}" -gt 0 ] && jac=$(( inter * 100 / union ))
112
+ if [ "$jac" -ge "$MINJ" ]; then
113
+ printf '%s\t%s\n' "$label" "$desc" >> "$WORK/cluster.$c"
114
+ sort -u "$WORK/sig.$c" "$WORK/candsig.$idx" > "$WORK/sig.$c.tmp" && mv "$WORK/sig.$c.tmp" "$WORK/sig.$c"
115
+ merged=1; break
116
+ fi
117
+ c=$((c+1))
118
+ done
119
+ if [ "$merged" -eq 0 ]; then
120
+ NC=$((NC+1))
121
+ printf '%s\t%s\n' "$label" "$desc" > "$WORK/cluster.$NC"
122
+ cp "$WORK/candsig.$idx" "$WORK/sig.$NC"
123
+ fi
124
+ done < "$RAW"
125
+
126
+ # --- seen-filter: pull already-KILLed candidates from the G4 run ledger so a re-listed marker for a
127
+ # candidate the chamber already killed does NOT re-enter the main queue (the run-#3 real-use gap). It is
128
+ # EXCLUDED from the ranked queue but SURFACED in a trailing section (re-emit trigger visibility — a KILL
129
+ # is revisitable once its measured observation lands, so we don't silently erase it). Degrade: no ledger
130
+ # → no seen-filter (fail-visible; equals the pre-seen behavior, safe). ---
131
+ LEDGER="$FH/tracks/_chamber/INDEX.md"
132
+ KILLED="$WORK/killed"; : > "$KILLED"
133
+ if [ -f "$LEDGER" ]; then
134
+ # KILL rows: a run-log table data row (`| #N | date | candidate | VERDICT | ... |`) whose VERDICT
135
+ # field ($5) says KILL. Scope the KILL test to the verdict field — NOT a whole-line grep, which would
136
+ # match "kill" inside "skill" in any row's carry-text (Axis-2 HIGH-1: an EMIT row folding a sliver
137
+ # "into goal-quench skill" would be misread as KILL). Candidate = $4; strip backticks/asterisks/space.
138
+ awk -F'|' '$0 ~ /^\| *#/ && NF>=5 && toupper($5) ~ /KILL/ {print $4}' "$LEDGER" 2>/dev/null \
139
+ | sed 's/[`*]//g; s/^ *//; s/ *$//' | grep -v '^$' >> "$KILLED"
140
+ fi
141
+ NKILL=$(grep -c . "$KILLED" 2>/dev/null); NKILL=${NKILL:-0}
142
+ SEENOUT="$WORK/seen_out"; : > "$SEENOUT"
143
+
144
+ # --- LAYER 2: rank each cluster = source-diversity*2 + frequency; then screen for reinvention ---
145
+ RANKED="$WORK/ranked"; : > "$RANKED"
146
+ c=1
147
+ while [ "$c" -le "$NC" ]; do
148
+ freq=$(grep -c . "$WORK/cluster.$c" 2>/dev/null); freq=${freq:-0}
149
+ ndiv=$(cut -f1 "$WORK/cluster.$c" | sort -u | grep -c . || true)
150
+ srcs=$(cut -f1 "$WORK/cluster.$c" | sort -u | paste -sd, -)
151
+ rep=$(head -1 "$WORK/cluster.$c" | cut -f2-)
152
+ score=$(( ndiv * 2 + freq ))
153
+ # seen-filter: does this candidate match an already-KILLed ledger entry? Two decorrelated tests, either
154
+ # sufficient: (1) kill-name keyword recall ≥60% (the common multi-token case); (2) normalized-substring
155
+ # fallback — the flattened slug (alnum-only, ≥5 chars) appearing in the flattened candidate — which
156
+ # closes the all-short-token silent-re-entry hole (MED-3b) that recall alone leaves open. Residual
157
+ # (documented, accepted): a paraphrased re-listing (router→routing) can still under-recall (MED-4), and
158
+ # a generic slug can over-exclude — but over-exclusion is VISIBLE in the SEEN-KILLED section (operator
159
+ # catches a wrong match), whereas silent re-entry was invisible. Visible-imprecise ≻ silent-miss.
160
+ if [ "$NKILL" -gt 0 ]; then
161
+ _ksig "$rep" > "$WORK/repsig"
162
+ repflat=$(printf '%s' "$rep" | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9')
163
+ seen_match=""
164
+ while IFS= read -r kname; do
165
+ [ -z "$kname" ] && continue
166
+ _ksig "$kname" > "$WORK/ksig"
167
+ kkn=$(grep -c . "$WORK/ksig" 2>/dev/null || true); kkn=${kkn:-0}
168
+ matched=0; [ "$kkn" -gt 0 ] && matched=$(comm -12 "$WORK/ksig" "$WORK/repsig" 2>/dev/null | grep -c . || true)
169
+ kpct=0; [ "$kkn" -gt 0 ] && kpct=$(( matched * 100 / kkn ))
170
+ kflat=$(printf '%s' "$kname" | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9')
171
+ if [ "$kkn" -gt 0 ] && [ "$kpct" -ge 60 ]; then seen_match="$kname"; break; fi
172
+ if [ "${#kflat}" -ge 5 ] && printf '%s' "$repflat" | grep -qF "$kflat"; then seen_match="$kname"; break; fi
173
+ done < "$KILLED"
174
+ if [ -n "$seen_match" ]; then
175
+ printf '%s\t%s\n' "$seen_match" "$rep" >> "$SEENOUT"
176
+ c=$((c+1)); continue
177
+ fi
178
+ fi
179
+ # reinvention first-pass on the representative description. Capture BOTH the verdict AND the matched
180
+ # anchor the screener already emits ("top asset overlap: X") so a DUPLICATE-CANDIDATE is ACTIONABLE
181
+ # ("DUP:skill:goal-quench") instead of a uniform non-discriminating flag (Axis-2 challenger Axis-5).
182
+ verdict="?"
183
+ if [ -x "$SCREEN" ] || [ -f "$SCREEN" ]; then
184
+ sout=$(bash "$SCREEN" "$rep" 2>/dev/null)
185
+ verdict=$(printf '%s\n' "$sout" | grep -oE 'VERDICT: [A-Z-]+' | head -1 | sed 's/VERDICT: //')
186
+ [ -z "$verdict" ] && verdict="?"
187
+ if [ "$verdict" = "DUPLICATE-CANDIDATE" ]; then
188
+ anchor=$(printf '%s\n' "$sout" | grep -oE 'top asset overlap: [^ ]+' | head -1 | sed 's/top asset overlap: //')
189
+ [ -n "$anchor" ] && verdict="DUP:$anchor"
190
+ fi
191
+ fi
192
+ printf '%03d\t%s\t%s\t%s\t%s\n' "$score" "$ndiv" "$srcs" "$verdict" "$rep" >> "$RANKED"
193
+ c=$((c+1))
194
+ done
195
+
196
+ echo "deduped candidates: $NC (from $NRAW raw markers)"
197
+ echo ""
198
+ printf '%-5s %-6s %-24s %-18s %s\n' "SCORE" "SRCS" "SOURCES" "REINVENTION" "CANDIDATE"
199
+ printf '%-5s %-6s %-24s %-18s %s\n' "-----" "------" "------------------------" "------------------" "---------"
200
+ sort -rn "$RANKED" | while IFS=$'\t' read -r score ndiv srcs verdict rep; do
201
+ # drop leading zeros for display
202
+ printf '%-5d %-6d %-24s %-18s %s\n' "$((10#$score))" "$ndiv" "$srcs" "$verdict" "$rep"
203
+ done
204
+ echo ""
205
+ echo "NOTE: SCORE = source-diversity×2 + frequency. REINVENTION is a first-pass flag (DUPLICATE-CANDIDATE"
206
+ echo "= HITL KILL review, not auto-drop). Uncertainty/failure-cost stay JUDGED — operator picks which"
207
+ echo "survivors actually enter the chamber (goal-quench budget gate + HITL). This tool ranks; it never admits."
208
+
209
+ # seen-filter trailing section: candidates the G4 ledger already KILLed are shown here (NOT in the ranked
210
+ # queue) so a re-listed marker does not silently re-enter the chamber, yet the KILL stays visible (a KILL
211
+ # is revisitable once its re-emit trigger lands — see the run's EMISSION_VERDICT for that condition).
212
+ if [ -s "$SEENOUT" ]; then
213
+ echo ""
214
+ echo "SEEN-KILLED — excluded from the queue (G4 ledger $LEDGER already KILLed these):"
215
+ while IFS=$'\t' read -r kn rep; do
216
+ [ -z "$rep" ] && continue
217
+ echo " · \"$rep\" ↔ killed as \"$kn\" (재등재 스킵; re-emit only when that run's trigger condition is met)"
218
+ done < "$SEENOUT"
219
+ elif [ ! -f "$LEDGER" ]; then
220
+ echo ""
221
+ echo "SEEN-FILTER: skipped — no G4 ledger at $LEDGER (fail-visible; no seen-exclusion applied this run)."
222
+ fi
223
+ exit 0