leerness 1.35.11 → 1.35.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,29 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.35.13 — 2026-07-06 — 테스트 3-tier: test:core 중간 티어(플래그십 행위 스위트) 신설 (UR-0014)
4
+
5
+ **사용자 지시 "codex와 협업" 라운드 ③ — e2e 3-tier 분리.** GPT5.5 평가가 "npm test 가 300초 내 미완료"라 지적한 후속. 전체 e2e.js(384 케이스, ~12분)를 리스크 있게 쪼개는 대신, **누락된 중간 티어를 별도 파일로 신설**(릴리스 게이트 무손상).
6
+
7
+ ### 3-tier 확립
8
+ - **`test:fast`** = selftest + `smoke.js` — 명령이 크래시 없이 도는가(clean 프로젝트), <1분. 개발 루프. (기존)
9
+ - **`test:core`** (신규) = selftest + `scripts/e2e-core.js` — 플래그십(verify-claim/gate/contract/scan)이 *적대적* 입력을 실제로 거부하고 정직 입력을 통과시키는가, **10 케이스 ~16초**. pre-commit / 빠른 CI. smoke("명령이 도는가")보다 깊고 full("전부")보다 20배 빠른 "가치제안 작동 확인" 티어.
10
+ - **`test`** = selftest + `e2e.js`(384) — 전체, ~12분. 릴리스 게이트. (기존)
11
+ - 부수: `test:smoke` 가 잘못 `e2e.js`(full)를 가리키던 네이밍 버그 → `smoke.js` 로 정정.
12
+
13
+ ### 검증
14
+ - selftest **272** (신규: 3티어 스크립트 정합 + e2e-core.js 존재 가드). `test:core` 10/10 (~16초). full e2e **384** (무영향 — e2e-core 는 별도 파일). README 기여자 노트를 3티어로 갱신.
15
+
16
+ ## 1.35.12 — 2026-07-06 — codex 협업 라운드 ②: clean-room 평가 문서 갱신 + codex 정직성 리뷰 8건 반영 (UR-0011)
17
+
18
+ **사용자 지시 "codex와 협업" 라운드 ② — clean-room 평가 문서 갱신.** `docs/clean-room-evaluations.md` 가 1.19.0(selftest 231)에서 멈춰 stale → 이번 세션 자체 적대적 헌트(verify-claim FP/gate/contract)를 추가하고, codex(gpt-5.5 xhigh)를 **정직성 감사관**으로 병렬 실행해 과장 표현을 교정.
19
+
20
+ ### 갱신
21
+ - 평가를 **"에이전트 클린룸 평가"(behavior-only)** 와 **"메인테이너 자체 적대적 헌트"(self-administered, 가장 덜 독립적)** 로 명확히 구분 — 1.35.9/10/11 헌트를 정직 outcome(FP 1건 발견·수정 / 0 product bug / codex 9건 중 3확정·1환각기각)으로 표에 추가. 한계 섹션에 contract heuristic 한계(UR-0018) + scan secrets 포지셔닝 추가.
22
+ - **codex 정직성 리뷰 8건 전부 반영**(협업 검수 결과): "every evaluation"→"most", "real development"→"development exercises", "Independently"→"Separately", "External reviews"→"maintainer-run(external models≠외부 인간)", "Codex independent review"→"separate Codex review", "converged on 0 bugs"→"0 bugs found in these probes", 그리고 핵심 — "reliably catches ... without false-failing"의 **자기모순 교정**(문서 자신이 1.35.9에서 FP를 찾았으므로 "no false failures"는 보장이 아니라 검사 중인 속성이라 명시).
23
+
24
+ ### 검증
25
+ - selftest **271** (기존 clean-room 문서 가드를 정직성 마커 강화: clean-room + heuristic + **self-administered** + 재현 레시피). full e2e (문서-only 변경, 무영향). 문서 변경이라 런타임 동작 불변.
26
+
3
27
  ## 1.35.11 — 2026-07-06 — codex 협업 contract verify 적대적 헌트: FP 3종 수정 ($ 필드 정규식 + bracket export + 코드펜스 예제)
4
28
 
5
29
  **사용자 지시 "codex와 협업" 라운드 ① — contract verify 적대적 헌트.** codex(gpt-5.5 xhigh 독립 분석)와 Claude(프로브+맹신X 검수) 병렬로 FN/FP/crash 사냥. codex 9건 제기 → 재현으로 검증: 3건 확정 수정, 3건 by-design/heuristic 한계(백로그), 3건은 이미 수정한 `$` 정규식 계열, 1건은 codex 환각(재현 기각).
package/README.md CHANGED
@@ -95,7 +95,7 @@ For secrets, pair the gate with a **dedicated scanner** in the same workflow —
95
95
 
96
96
  Be honest with yourself before you depend on this: leerness is **early and largely solo-maintained**, developed mostly through autonomous AI rounds — so its own `selftest` + e2e suites are the primary quality signal, and external adoption is still small. Don't make it load-bearing on faith: **pin a version**, and treat the differentiated slice — `verify-claim` + the CI `gate` as a required check — as the part worth relying on.
97
97
 
98
- (Contributor note: `npm test` is the full release gate selftest + the entire e2e suite, **10+ minutes by design**. For a quick dev loop use `npm run test:fast` selftest + smoke, under a minute.)
98
+ (Contributor note — three test tiers, fastest to slowest: `npm run test:fast` = selftest + smoke (commands run without crashing, <1 min, dev loop); `npm run test:core` = selftest + a flagship behavioral suite (verify-claim / gate / contract / scan actually reject bad input and pass honest input, ~20 s, pre-commit / quick CI); `npm test` = selftest + the entire e2e suite (**10+ minutes by design**, the release gate).)
99
99
 
100
100
  The asymmetry is what makes a trial reasonable anyway: MIT, **0 runtime dependencies**, offline-first, and all state is plain files in *your* repo. Lock-in is near zero — if it doesn't earn its place, remove the tool and your `task`/`decision`/`lesson` files stay. (For secret scanning specifically, mature dedicated tools like gitleaks/trufflehog exist — use those if you need a hard guarantee; leerness's `scan secrets` is a convenience guard, not a replacement.)
101
101
 
@@ -125,7 +125,7 @@ MIT
125
125
  <!-- leerness:project-readme:start -->
126
126
  ## Leerness Project Harness
127
127
 
128
- 이 프로젝트는 Leerness v1.35.11 하네스를 사용합니다. AI 에이전트는 작업 전 `leerness handoff`로 컨텍스트를 적재하고, 작업 후 `leerness check`/`leerness audit`/`leerness session close`를 수행해야 합니다.
128
+ 이 프로젝트는 Leerness v1.35.13 하네스를 사용합니다. AI 에이전트는 작업 전 `leerness handoff`로 컨텍스트를 적재하고, 작업 후 `leerness check`/`leerness audit`/`leerness session close`를 수행해야 합니다.
129
129
 
130
130
  ### 정체성 — AI 에이전트 운영 레이어 (UR-0030)
131
131
 
@@ -179,7 +179,7 @@ leerness memory restore decision <date|title>
179
179
 
180
180
  ### MCP server (외부 AI 통합)
181
181
 
182
- Leerness v1.35.11는 stdio JSON-RPC MCP server를 내장합니다 — Claude Code · Cursor · Codex CLI 등 외부 AI에 **86개 도구**를 노출:
182
+ Leerness v1.35.13는 stdio JSON-RPC MCP server를 내장합니다 — Claude Code · Cursor · Codex CLI 등 외부 AI에 **86개 도구**를 노출:
183
183
 
184
184
  ```jsonc
185
185
  // 카테고리별
@@ -200,7 +200,7 @@ Leerness v1.35.11는 stdio JSON-RPC MCP server를 내장합니다 — Claude Cod
200
200
  `<<autonomous-loop-dynamic>>` 신호만 보내면 AI가:
201
201
  1) 다음 라운드 후보 선정 → 2) 코드 변경 → 3) stress-v* 신규 작성 + 누적 회귀 → 4) e2e 219/219 → 5) npm pack + git tag + GitHub release → 6) main 자동 push (1.9.140+) → 7) session close → 8) 다음 라운드 예약.
202
202
 
203
- 현재 누적: **70 라운드 (1.9.40 → 1.35.11)** · 매 라운드 GitHub release/태그 생성 · _reports/는 비공개 보존.
203
+ 현재 누적: **70 라운드 (1.9.40 → 1.35.13)** · 매 라운드 GitHub release/태그 생성 · _reports/는 비공개 보존.
204
204
 
205
205
  ### 성능 가이드 (1.9.140 측정)
206
206
 
@@ -238,6 +238,6 @@ leerness release pack --close --auto-main-push
238
238
  - `.harness/session-handoff.md`: 다음 세션 인수인계 (자동 작성)
239
239
  - `.harness/lessons.md` / `decisions.md` / `rules.md`: 영구 메모리 (5 surface)
240
240
 
241
- Last synced by Leerness v1.35.11: 2026-07-06
241
+ Last synced by Leerness v1.35.13: 2026-07-06
242
242
  <!-- leerness:project-readme:end -->
243
243
 
package/bin/leerness.js CHANGED
@@ -32,7 +32,7 @@ const { _evidenceQuality, _parseEvidenceStats, _shellGuardAnalyze, _claimFileInG
32
32
  // 1.9.295 (UR-0025 4단계): 정적 데이터 카탈로그 모듈 분리 (비파괴, require-based).
33
33
  const { CAPABILITY_SURFACE, POWERFUL_COMMANDS, ADAPTERS, REUSE_CATEGORIES, REUSE_CHECKLIST, _DEFAULT_PLATFORM_CONSTRAINTS, _DEFAULT_DOMAIN_CATALOG, _TOOL_CATALOG, _LSP_LANG_PATTERNS, OPTIMISM_PATTERNS, BUILT_IN_PERSONAS, STRINGS, BUILTIN_CATALOG, ROADMAP_STATUS_LABEL, ROADMAP_STATUS_COLOR, SECRET_PATTERNS, MERGE_OVERWRITE_FILES, MINIMAL_SKIP_KEYS, REQUIRED_WORKSPACE_FILES, KEYWORD_STOPWORDS, SKILL_CATALOG_PRESETS } = require('../lib/catalogs'); // 1.9.344/368/369 (UR-0025): catalog 분리 · 1.11.4 (UR-0007): _TOOL_CATALOG
34
34
 
35
- const VERSION = '1.35.11';
35
+ const VERSION = '1.35.13';
36
36
 
37
37
  // 1.9.290 (UR-0037, Codex gpt-5.5 #4 수렴): CLI 전용 부작용은 require 시 실행하지 않는다.
38
38
  // 이전: warning listener 제거 / NODE_OPTIONS 변경 / chcp IIFE 가 top-level 즉시 실행 → require('harness') 시 호스트 프로세스 오염.
@@ -2935,6 +2935,14 @@ function _selfTestCases() {
2935
2935
  const claims6 = body.includes("step('verify-claims'") && body.includes("has('--claims')");
2936
2936
  return has5 && claims6;
2937
2937
  } },
2938
+ { name: '3-tier 테스트 (1.35.13, UR-0014): package.json test:fast/test:core/test 3티어 + scripts/e2e-core.js 존재', run: () => {
2939
+ let pkg; try { pkg = require('../package.json'); } catch { return false; }
2940
+ const s = pkg.scripts || {};
2941
+ // fast=smoke, core=e2e-core, full=e2e — 3티어 스크립트 정합.
2942
+ const tiersOk = /smoke\.js/.test(s['test:fast'] || '') && /e2e-core\.js/.test(s['test:core'] || '') && /scripts\/e2e\.js/.test(s.test || '');
2943
+ const coreExists = exists(path.join(path.dirname(__filename), '..', 'scripts', 'e2e-core.js'));
2944
+ return tiersOk && coreExists;
2945
+ } },
2938
2946
  { name: '자체 contract 적대적 헌트 + codex 교차 (1.35.11): _parseImplExports bracket(#8) + _parseContractSpec 코드펜스 예제 제외(#9) — FP 2종 행위검사($ 필드 FP는 e2e cvdollar 담당)', run: () => {
2939
2947
  const p = require('../lib/pure-utils');
2940
2948
  const bracket = p._parseImplExports('exports["foo"] = function(){};\n').includes('foo'); // #8: bracket export 인식
@@ -3823,11 +3831,12 @@ function _selfTestCases() {
3823
3831
  && eq(F([]), [])
3824
3832
  && eq(F(['a.css', 'b.md', 'c.js', 'd.test.js']).length, 2); // 최대 2개 클러터 방지
3825
3833
  } },
3826
- { name: 'GPT-5.5 평가 #5 (1.19.1, UR-0009): 클린룸 평가 문서 공개 + 한계 명시 (행위)', run: () => {
3834
+ { name: 'GPT-5.5 평가 #5 (1.19.1, UR-0009) + 정직성 calibration (1.35.12): 클린룸 문서 공개 + 한계 명시 + self-administered 라벨 (행위)', run: () => {
3827
3835
  const dp = path.join(path.dirname(__filename), '..', 'docs', 'clean-room-evaluations.md');
3828
3836
  if (!exists(dp)) return true; // 패키지에 없으면 스킵(설치본 안전)
3829
3837
  const d = read(dp);
3830
- return /AI clean-room evaluations/i.test(d) && /heuristic, not semantic/i.test(d) && /npm i leerness@/.test(d);
3838
+ // 1.35.12: 정직성 마커 강화 — clean-room 프레이밍 + heuristic 한계 + self-administered(독립 아님) + 재현 레시피.
3839
+ return /clean-room/i.test(d) && /heuristic, not semantic/i.test(d) && /self-administered/i.test(d) && /npm i leerness@/.test(d);
3831
3840
  } },
3832
3841
  { name: 'CLI 영어화 Phase 1 (1.20.2, UR-0010): _uiLang 해석(flag>env>manifest>ko) + 첫화면 _t 적용 (행위+소스)', run: () => {
3833
3842
  const save = process.argv; const saveEnv = process.env.LEERNESS_LANG;
@@ -1,67 +1,96 @@
1
- # Clean-room evaluations / 클린룸 평가 기록
2
-
3
- > **What this is, honestly.** These are **AI clean-room evaluations** — independent agents installing the
4
- > published npm package into fresh temp directories and exercising it from behavior only (no access to the
5
- > source tree), including adversarial attempts to defeat leerness's own verifier. They are **not**
6
- > third-party human security audits or peer-reviewed benchmarks. We publish them so the README claim
7
- > "verified by independent clean-room evaluations" is checkable rather than a marketing line.
8
- >
9
- > 정직하게 말하면: 아래는 **AI 클린룸 평가**입니다 독립 에이전트가 게시된 npm 패키지를 임시 폴더에
10
- > 설치해 **소스 접근 없이 행위만으로** 검증하고, leerness 검증기 자체를 무력화하려는 적대 시도까지 포함합니다.
11
- > 제3자 인간 보안 감사나 동료심사 벤치마크가 **아닙니다**. README "독립 클린룸 평가로 검증" 주장을
12
- > 확인 가능하게 만들기 위해 공개합니다.
13
-
14
- ## Methodology / 방법론
15
-
16
- Every evaluation follows the same shape:
17
-
18
- ```bash
19
- W=$(mktemp -d); cd "$W"; npm init -y
20
- npm i leerness@<version> # the PUBLISHED package, not the working tree
21
- LB="node node_modules/leerness/bin/leerness.js"
22
- # ... drive the CLI by behavior, assert on exit codes + output, clean up ...
23
- ```
24
-
25
- - **Published-artifact only** installs from npm, never reads the repo source. This is what an external user gets.
26
- - **Behavior-asserted** — checks process exit codes and stdout/stderr, not internal state.
27
- - **Adversarial** — a portion of every pass actively tries to make a *false* "done" claim pass verification
28
- (comment-only stubs, empty-export shells, `assert(true)` fake tests, inflated test counts, language tricks).
29
- - **Both directions (맹신 X / "trust nothing")** — confirmed real defects AND rejected false alarms; a finding
30
- counts only if reproduced on the published package.
31
-
32
- ## Evaluations run / 수행한 평가
33
-
34
- | # | Round | Scope | Outcome |
35
- |---|---|---|---|
36
- | 1 | Universal-harness field study (5-axis) | Real Python / Node / Rust development, agent-swap handoff, adversarial attack on the verifier | Verdict "conditionally yes"; found 5 gaps (2×P1, 3×P2) — all closed in 1.17.2–1.17.6, shipped as the 1.18.0 stable minor |
37
- | 2 | 1.18.0 re-verification (published) | Independently re-ran the 5 gaps against the **published** 1.18.0 | 4 closed; surfaced a new P1 (Windows `--test-cmd python` blocked → false FAIL) → fixed in 1.18.1 |
38
- | 3 | 1.18.2 adversarial stub workflow | Bypass-hunters + false-positive-hunters vs the empty-shell stub detector | 9 one-keyword bypasses found (Object.freeze, async fn, inline comment, …) and closed; **0 false positives** across ~45 legitimate patterns |
39
- | 4 | 1.19.0 published-artifact re-verification | Installed `leerness@1.19.0` from npm; smoke-tested headline fixes | `--test-cmd python`, empty-shell rejection, `lens`, selftest 231 — all pass on the published package |
40
-
41
- ## What these evaluations do NOT establish / 한계 (정직)
42
-
43
- leerness's verification is **heuristic, not semantic**. These evaluations show it reliably catches *obvious
44
- false-done claims* (missing files, empty shells, fake/unlinked tests, inflated counts) without false-failing
45
- honest work. They do **not** prove:
46
-
47
- - the implementation satisfies the actual requirement, or business logic is correct;
48
- - tests are sufficient or well-designed;
49
- - absence of security vulnerabilities or production-runtime correctness.
50
-
51
- Known heuristic gaps tracked for an AST/token-based redesign: multi-arg call-expression empty objects
52
- (`Object.assign({}, {})`), Python `def …: pass` / `...` / `raise NotImplementedError`, multi-language empty
53
- bodies. **Code coverage / mutation testing and third-party reproduction remain open** (the e2e is a
54
- spawn-based runner, which standard coverage tools do not instrument cleanly).
55
-
56
- ## Reproduce it yourself / 직접 재현
57
-
58
- ```bash
59
- W=$(mktemp -d); cd "$W"; npm init -y && npm i leerness@latest
60
- LB="node node_modules/leerness/bin/leerness.js"
61
- $LB init "$W" --yes
62
- # fake "done": claim an implementation that does not exist
63
- $LB task add "Implement X"
64
- $LB task update T-0001 --status done --evidence "x.js done, 5 tests passed"
65
- $LB verify-claim T-0001 # exit 1 claim rejected (x.js missing, no tests ran)
66
- $LB selftest # core-function integrity self-check
67
- ```
1
+ # Clean-room evaluations / 클린룸 평가 기록
2
+
3
+ > **What this is, honestly.** These are **AI evaluations run by the project itself** — an agent installs the
4
+ > published npm package into fresh temp directories and exercises it from behavior only (no access to the
5
+ > source tree), including adversarial attempts to defeat leerness's own verifier. They are **self-administered**
6
+ > (the maintainer's AI, sometimes cross-checked by a second model such as Codex), **not** third-party human
7
+ > security audits or peer-reviewed benchmarks, and **not** independent in the "unaffiliated reviewer" sense.
8
+ > We publish them so the README claim "checkable by clean-room evaluation" is verifiable rather than a
9
+ > marketing lineyou can re-run the recipe below yourself.
10
+ >
11
+ > 정직하게 말하면: 아래는 **프로젝트가 직접 수행한 AI 평가**입니다 에이전트가 게시된 npm 패키지를 빈 임시
12
+ > 폴더에 설치해 **소스 접근 없이 행위만으로** 검증하고, leerness 검증기 자체를 무력화하려는 적대 시도까지
13
+ > 포함합니다. 이는 **자기-수행**(메인테이너의 AI, 때때로 Codex 같은 2차 모델이 교차검증)이며, 제3자 인간 보안
14
+ > 감사·동료심사 벤치마크가 **아니고**, "무관한 외부 리뷰어"라는 의미의 독립도 **아닙니다**. README 의 "클린룸
15
+ > 평가로 확인 가능" 주장을 검증 가능하게 만들기 위해 공개하며, 아래 레시피로 직접 재현할 수 있습니다.
16
+
17
+ ## Methodology / 방법론
18
+
19
+ Most evaluations follow this general shape:
20
+
21
+ ```bash
22
+ W=$(mktemp -d); cd "$W"; npm init -y
23
+ npm i leerness@<version> # the PUBLISHED package, not the working tree
24
+ LB="node node_modules/leerness/bin/leerness.js"
25
+ # ... drive the CLI by behavior, assert on exit codes + output, clean up ...
26
+ ```
27
+
28
+ - **Published-artifact only** installs from npm, never reads the repo source. This is what an external user gets.
29
+ - **Behavior-asserted** — checks process exit codes and stdout/stderr, not internal state.
30
+ - **Adversarial (both directions)** many passes include attempts to make a *false* "done" claim pass
31
+ verification (comment-only stubs, empty-export shells, `assert(true)` fake tests, inflated test counts) **and**
32
+ to make an *honest* claim wrongly fail (false-positive hunting). A finding counts only if reproduced on the
33
+ published package; agent-proposed findings are re-derived, not trusted (맹신 X / "trust nothing").
34
+
35
+ ## Agent clean-room evaluations / 에이전트 클린룸 평가 (behavior-only, published artifact)
36
+
37
+ | # | Round | Scope | Outcome |
38
+ |---|---|---|---|
39
+ | 1 | Universal-harness field study (5-axis) | Python / Node / Rust development exercises, agent-swap handoff, adversarial attack on the verifier | Verdict "conditionally yes"; found 5 gaps (2×P1, 3×P2) — all closed in 1.17.2–1.17.6, shipped as the 1.18.0 stable minor |
40
+ | 2 | 1.18.0 re-verification (published) | Separately re-ran the 5 gaps against the **published** 1.18.0 | 4 closed; surfaced a new P1 (Windows `--test-cmd python` blocked → false FAIL) → fixed in 1.18.1 |
41
+ | 3 | 1.18.2 adversarial stub workflow | Bypass-hunters + false-positive-hunters vs the empty-shell stub detector | 9 one-keyword bypasses found (Object.freeze, async fn, inline comment, …) and closed; **0 false positives** across ~45 legitimate patterns |
42
+ | 4 | 1.19.0 published-artifact re-verification | Installed `leerness@1.19.0` from npm; smoke-tested headline fixes | `--test-cmd python`, empty-shell rejection, `lens`, selftest — all pass on the published package |
43
+
44
+ Maintainer-run multi-model reviews (external models GPT-class + Claude driven against a published build and
45
+ reproduced before fixing) have also shaped many rounds; the confirmed findings are tracked in CHANGELOG under
46
+ the "Nth 외부평가 / 외부리뷰" entries. "External" here means the *models*, not unaffiliated human reviewers.
47
+
48
+ ## Maintainer adversarial hunts / 메인테이너 자체 적대적 헌트 (self-administered, published artifact)
49
+
50
+ These are the **least independent** kind — the maintainer's AI hunting its own flagship surfaces, cross-checked
51
+ with Codex where noted. They are listed for transparency, not as third-party validation. Every finding was
52
+ reproduced on the published package; every fix re-verified after publish.
53
+
54
+ | Version | Surface | Method | Outcome (honest) |
55
+ |---|---|---|---|
56
+ | 1.35.9 | `verify-claim` declared-pass gate | 12-probe FP/FN matrix (single / `--json` / `--all` / `gate --claims` × reporter/growth/lenient) | 1 real false-positive found (a non-test `N/M passing` ratio wrongly gated) and fixed; 0 bypasses |
57
+ | 1.35.10 | `gate` (the CI guardrail) | 8-probe matrix (clean/placeholder/secret/false-done/missing-file) | **0 product bugs**; 2 probe-flagged "FN"s were probe artifacts, refuted by reproduction; added a gate-level security regression guard |
58
+ | 1.35.11 | `contract verify` (spec ↔ impl) | Claude probes + a separate Codex (gpt-5.5) review, cross-checked | Codex raised 9 candidates → **3 confirmed & fixed** (`$`-field regex FP, bracket-export FP, code-fence-example FP), **1 refuted as a hallucination**, the rest documented heuristic limits |
59
+
60
+ The pattern worth noting: two of the three surfaces had **0 product bugs found in these probes** only after a
61
+ dedicated false-positive hunt, and in each round a portion of the AI-proposed findings did **not** reproduce and
62
+ were dropped. That is the point of publishing this — the checks are fallible, the probe matrices are finite, and
63
+ the checks are themselves under adversarial test, including their own false-alarm behavior.
64
+
65
+ ## What these evaluations do NOT establish / 한계 (정직)
66
+
67
+ leerness's verification is **heuristic, not semantic**. These evaluations suggest it catches many *obvious
68
+ false-done claims* (missing files, empty shells, fake/unlinked tests, inflated counts), with no false failures
69
+ in the tested cases — though the false-positive hunts have themselves found and fixed a real false-positive
70
+ (1.35.9), so "no false failures" is a property under test, not a guarantee. They do **not** prove:
71
+
72
+ - the implementation satisfies the actual requirement, or business logic is correct;
73
+ - tests are sufficient or well-designed;
74
+ - absence of security vulnerabilities or production-runtime correctness.
75
+
76
+ Known heuristic gaps tracked for an AST/token-based redesign (UR-0016): multi-arg call-expression empty objects
77
+ (`Object.assign({}, {})`), Python `def …: pass` / `...` / `raise NotImplementedError`, multi-language empty
78
+ bodies; and for `contract verify` (UR-0018) a field/function present only in a **comment or string** still
79
+ passes the grep-presence check, and a function exported as a **non-function value** passes (contract checks
80
+ *presence*; `verify-claim` checks *substance* — a deliberate split). `scan secrets` is a convenience guard, not
81
+ a replacement for a dedicated scanner (gitleaks/trufflehog). **Code coverage / mutation testing and third-party
82
+ reproduction remain open** (the e2e is a spawn-based runner, which standard coverage tools do not instrument
83
+ cleanly).
84
+
85
+ ## Reproduce it yourself / 직접 재현
86
+
87
+ ```bash
88
+ W=$(mktemp -d); cd "$W"; npm init -y && npm i leerness@latest
89
+ LB="node node_modules/leerness/bin/leerness.js"
90
+ $LB init "$W" --yes
91
+ # fake "done": claim an implementation that does not exist
92
+ $LB task add "Implement X"
93
+ $LB task update T-0001 --status done --evidence "x.js done, 5 tests passed"
94
+ $LB verify-claim T-0001 # exit 1 — claim rejected (x.js missing, no tests ran)
95
+ $LB selftest # core-function integrity self-check (exit 0 when the install is intact)
96
+ ```
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "leerness",
3
- "version": "1.35.11",
3
+ "version": "1.35.13",
4
4
  "description": "Leerness: 비파괴 마이그레이션, 자동 버전 감지·업데이트, 계획/진행/핸드오프 자동화, 게으름·시크릿·인코딩 자동 가드, Claude Code 슬래시 통합을 갖춘 한국어 우선 AI 개발 하네스.",
5
5
  "keywords": [
6
6
  "leerness",
@@ -48,8 +48,9 @@
48
48
  ],
49
49
  "scripts": {
50
50
  "test": "node ./bin/leerness.js --version && node ./bin/leerness.js selftest && node ./scripts/e2e.js",
51
+ "test:core": "node ./bin/leerness.js --version && node ./bin/leerness.js selftest && node ./scripts/e2e-core.js",
51
52
  "test:fast": "node ./bin/leerness.js selftest && node ./scripts/smoke.js",
52
- "test:smoke": "node ./scripts/e2e.js",
53
+ "test:smoke": "node ./scripts/smoke.js",
53
54
  "prepack": "node ./bin/leerness.js readme sync . && node ./bin/leerness.js --version"
54
55
  },
55
56
  "publishConfig": {
@@ -0,0 +1,82 @@
1
+ #!/usr/bin/env node
2
+ 'use strict';
3
+
4
+ // 1.35.13 (UR-0014): 3-tier 테스트의 중간 티어 — critical-path 행위(behavioral) 스위트.
5
+ // test:fast (smoke.js) — 명령이 크래시 없이 도는가만 clean 프로젝트에서 확인, <1분. 개발 루프.
6
+ // test:core (이 파일) — 플래그십(verify-claim/gate/contract/scan)이 *적대적* 입력을 실제로 거부/통과하는가, ~1-2분. pre-commit / 빠른 CI.
7
+ // test (e2e.js) — 전체 384 케이스, ~12분, 릴리스 게이트.
8
+ // smoke 는 "명령이 도는가"만 본다. 이 core 는 "가치제안(거짓완료·시크릿·계약위반 차단)이 실제로 작동하는가"를 행위로 검증한다.
9
+ // 전체 e2e.js 를 건드리지 않는 별도 파일 — 릴리스 게이트 회귀 위험 0. 실패 시 exit 1 (CI 친화). 사용: npm run test:core
10
+
11
+ const fs = require('fs'), os = require('os'), path = require('path'), cp = require('child_process');
12
+ process.env.LEERNESS_OFFLINE = process.env.LEERNESS_OFFLINE || '1';
13
+ const CLI = path.resolve(__dirname, '..', 'bin', 'leerness.js');
14
+ let failed = 0, total = 0; const t0 = Date.now();
15
+
16
+ function run(dir, args) { return cp.spawnSync(process.execPath, [CLI, ...args, '--path', dir], { encoding: 'utf8', timeout: 40000 }); }
17
+ function assert(label, cond) { total++; process.stdout.write(`${cond ? '✓' : '✗'} ${label}\n`); if (!cond) failed++; }
18
+ function fresh() { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'leerness-core-')); cp.spawnSync(process.execPath, [CLI, 'init', d, '--yes', '--language', 'ko'], { encoding: 'utf8', timeout: 30000 }); return d; }
19
+
20
+ console.log('# leerness core (test:core) — flagship behavioral guarantees');
21
+
22
+ // (1) verify-claim: 거짓완료(없는 파일) 거부, 정직완료 통과
23
+ {
24
+ const d = fresh();
25
+ run(d, ['task', 'add', 'ghost']);
26
+ run(d, ['task', 'update', 'T-0002', '--status', 'done', '--evidence', 'ghost.js implemented; 5/5 passed']);
27
+ assert('verify-claim rejects false-done (missing file → exit 1)', run(d, ['verify-claim', 'T-0002']).status === 1);
28
+ fs.writeFileSync(path.join(d, 'real.js'), 'module.exports = { f: () => 1 };\n');
29
+ run(d, ['task', 'add', 'real']);
30
+ run(d, ['task', 'update', 'T-0003', '--status', 'done', '--evidence', 'real.js implemented']);
31
+ assert('verify-claim passes truthful done (exit 0)', run(d, ['verify-claim', 'T-0003']).status === 0);
32
+ fs.rmSync(d, { recursive: true, force: true });
33
+ }
34
+
35
+ // (2) verify-claim: 부풀린 테스트 카운트 거부 (1.35.7/9 플래그십)
36
+ {
37
+ const d = fresh();
38
+ fs.writeFileSync(path.join(d, 'x.js'), 'module.exports = { f: () => 1 };\n');
39
+ fs.writeFileSync(path.join(d, 'x.test.js'), 'test();\ntest();\n');
40
+ run(d, ['task', 'add', 'x']);
41
+ run(d, ['task', 'update', 'T-0002', '--status', 'done', '--evidence', 'x.js, x.test.js; 3/3 passed']);
42
+ assert('verify-claim rejects inflated pass (claim 3/3 vs run 2/2 → exit 1)', run(d, ['verify-claim', 'T-0002', '--run-tests', '--test-cmd', 'echo Tests: 2 passed, 2 total']).status === 1);
43
+ fs.rmSync(d, { recursive: true, force: true });
44
+ }
45
+
46
+ // (3) gate: 클린 통과 / 커밋 시크릿 차단 / --claims 거짓완료 차단
47
+ {
48
+ const d = fresh();
49
+ run(d, ['handoff', '']);
50
+ assert('gate clean project → exit 0', run(d, ['gate', '.']).status === 0);
51
+ fs.writeFileSync(path.join(d, 'leak.js'), 'const k = "AKIAJQXMP7RZ2KL9WXYZ";\nmodule.exports = k;\n');
52
+ assert('gate blocks committed secret → exit 1', run(d, ['gate', '.']).status === 1);
53
+ fs.rmSync(path.join(d, 'leak.js'), { force: true });
54
+ run(d, ['task', 'add', 'ghost']);
55
+ run(d, ['task', 'update', 'T-0002', '--status', 'done', '--evidence', 'ghost.js implemented; 5/5 passed']);
56
+ assert('gate --claims blocks false-done → exit 1', run(d, ['gate', '.', '--claims']).status === 1);
57
+ fs.rmSync(d, { recursive: true, force: true });
58
+ }
59
+
60
+ // (4) contract verify: 누락 함수 감지 / 충족 통과
61
+ {
62
+ const d = fresh();
63
+ fs.writeFileSync(path.join(d, 's.md'), '- charge()\n');
64
+ fs.writeFileSync(path.join(d, 'miss.js'), 'module.exports = { other: 1 };\n');
65
+ assert('contract catches missing function → exit 1', run(d, ['contract', 'verify', path.join(d, 's.md'), path.join(d, 'miss.js')]).status === 1);
66
+ fs.writeFileSync(path.join(d, 'ok.js'), 'function charge(){}\nmodule.exports = { charge };\n');
67
+ assert('contract passes when satisfied → exit 0', run(d, ['contract', 'verify', path.join(d, 's.md'), path.join(d, 'ok.js')]).status === 0);
68
+ fs.rmSync(d, { recursive: true, force: true });
69
+ }
70
+
71
+ // (5) scan secrets: 커밋 시크릿 차단 / 클린 통과
72
+ {
73
+ const d = fresh();
74
+ assert('scan secrets clean → exit 0', run(d, ['scan', 'secrets', '.']).status === 0);
75
+ fs.writeFileSync(path.join(d, 'k.js'), 'const k = "AKIAJQXMP7RZ2KL9WXYZ";\n');
76
+ assert('scan secrets committed key → exit 1', run(d, ['scan', 'secrets', '.']).status === 1);
77
+ fs.rmSync(d, { recursive: true, force: true });
78
+ }
79
+
80
+ const dur = ((Date.now() - t0) / 1000).toFixed(1);
81
+ console.log(`\nCore result: ${total - failed}/${total} passed · ${dur}s`);
82
+ if (failed > 0) process.exit(1);