leerness 1.35.11 → 1.35.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/README.md +5 -5
- package/bin/leerness.js +12 -3
- package/docs/clean-room-evaluations.md +96 -67
- package/package.json +3 -2
- package/scripts/e2e-core.js +82 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,29 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.35.13 — 2026-07-06 — 테스트 3-tier: test:core 중간 티어(플래그십 행위 스위트) 신설 (UR-0014)
|
|
4
|
+
|
|
5
|
+
**사용자 지시 "codex와 협업" 라운드 ③ — e2e 3-tier 분리.** GPT5.5 평가가 "npm test 가 300초 내 미완료"라 지적한 후속. 전체 e2e.js(384 케이스, ~12분)를 리스크 있게 쪼개는 대신, **누락된 중간 티어를 별도 파일로 신설**(릴리스 게이트 무손상).
|
|
6
|
+
|
|
7
|
+
### 3-tier 확립
|
|
8
|
+
- **`test:fast`** = selftest + `smoke.js` — 명령이 크래시 없이 도는가(clean 프로젝트), <1분. 개발 루프. (기존)
|
|
9
|
+
- **`test:core`** (신규) = selftest + `scripts/e2e-core.js` — 플래그십(verify-claim/gate/contract/scan)이 *적대적* 입력을 실제로 거부하고 정직 입력을 통과시키는가, **10 케이스 ~16초**. pre-commit / 빠른 CI. smoke("명령이 도는가")보다 깊고 full("전부")보다 20배 빠른 "가치제안 작동 확인" 티어.
|
|
10
|
+
- **`test`** = selftest + `e2e.js`(384) — 전체, ~12분. 릴리스 게이트. (기존)
|
|
11
|
+
- 부수: `test:smoke` 가 잘못 `e2e.js`(full)를 가리키던 네이밍 버그 → `smoke.js` 로 정정.
|
|
12
|
+
|
|
13
|
+
### 검증
|
|
14
|
+
- selftest **272** (신규: 3티어 스크립트 정합 + e2e-core.js 존재 가드). `test:core` 10/10 (~16초). full e2e **384** (무영향 — e2e-core 는 별도 파일). README 기여자 노트를 3티어로 갱신.
|
|
15
|
+
|
|
16
|
+
## 1.35.12 — 2026-07-06 — codex 협업 라운드 ②: clean-room 평가 문서 갱신 + codex 정직성 리뷰 8건 반영 (UR-0011)
|
|
17
|
+
|
|
18
|
+
**사용자 지시 "codex와 협업" 라운드 ② — clean-room 평가 문서 갱신.** `docs/clean-room-evaluations.md` 가 1.19.0(selftest 231)에서 멈춰 stale → 이번 세션 자체 적대적 헌트(verify-claim FP/gate/contract)를 추가하고, codex(gpt-5.5 xhigh)를 **정직성 감사관**으로 병렬 실행해 과장 표현을 교정.
|
|
19
|
+
|
|
20
|
+
### 갱신
|
|
21
|
+
- 평가를 **"에이전트 클린룸 평가"(behavior-only)** 와 **"메인테이너 자체 적대적 헌트"(self-administered, 가장 덜 독립적)** 로 명확히 구분 — 1.35.9/10/11 헌트를 정직 outcome(FP 1건 발견·수정 / 0 product bug / codex 9건 중 3확정·1환각기각)으로 표에 추가. 한계 섹션에 contract heuristic 한계(UR-0018) + scan secrets 포지셔닝 추가.
|
|
22
|
+
- **codex 정직성 리뷰 8건 전부 반영**(협업 검수 결과): "every evaluation"→"most", "real development"→"development exercises", "Independently"→"Separately", "External reviews"→"maintainer-run(external models≠외부 인간)", "Codex independent review"→"separate Codex review", "converged on 0 bugs"→"0 bugs found in these probes", 그리고 핵심 — "reliably catches ... without false-failing"의 **자기모순 교정**(문서 자신이 1.35.9에서 FP를 찾았으므로 "no false failures"는 보장이 아니라 검사 중인 속성이라 명시).
|
|
23
|
+
|
|
24
|
+
### 검증
|
|
25
|
+
- selftest **271** (기존 clean-room 문서 가드를 정직성 마커 강화: clean-room + heuristic + **self-administered** + 재현 레시피). full e2e (문서-only 변경, 무영향). 문서 변경이라 런타임 동작 불변.
|
|
26
|
+
|
|
3
27
|
## 1.35.11 — 2026-07-06 — codex 협업 contract verify 적대적 헌트: FP 3종 수정 ($ 필드 정규식 + bracket export + 코드펜스 예제)
|
|
4
28
|
|
|
5
29
|
**사용자 지시 "codex와 협업" 라운드 ① — contract verify 적대적 헌트.** codex(gpt-5.5 xhigh 독립 분석)와 Claude(프로브+맹신X 검수) 병렬로 FN/FP/crash 사냥. codex 9건 제기 → 재현으로 검증: 3건 확정 수정, 3건 by-design/heuristic 한계(백로그), 3건은 이미 수정한 `$` 정규식 계열, 1건은 codex 환각(재현 기각).
|
package/README.md
CHANGED
|
@@ -95,7 +95,7 @@ For secrets, pair the gate with a **dedicated scanner** in the same workflow —
|
|
|
95
95
|
|
|
96
96
|
Be honest with yourself before you depend on this: leerness is **early and largely solo-maintained**, developed mostly through autonomous AI rounds — so its own `selftest` + e2e suites are the primary quality signal, and external adoption is still small. Don't make it load-bearing on faith: **pin a version**, and treat the differentiated slice — `verify-claim` + the CI `gate` as a required check — as the part worth relying on.
|
|
97
97
|
|
|
98
|
-
(Contributor note: `npm test`
|
|
98
|
+
(Contributor note — three test tiers, fastest to slowest: `npm run test:fast` = selftest + smoke (commands run without crashing, <1 min, dev loop); `npm run test:core` = selftest + a flagship behavioral suite (verify-claim / gate / contract / scan actually reject bad input and pass honest input, ~20 s, pre-commit / quick CI); `npm test` = selftest + the entire e2e suite (**10+ minutes by design**, the release gate).)
|
|
99
99
|
|
|
100
100
|
The asymmetry is what makes a trial reasonable anyway: MIT, **0 runtime dependencies**, offline-first, and all state is plain files in *your* repo. Lock-in is near zero — if it doesn't earn its place, remove the tool and your `task`/`decision`/`lesson` files stay. (For secret scanning specifically, mature dedicated tools like gitleaks/trufflehog exist — use those if you need a hard guarantee; leerness's `scan secrets` is a convenience guard, not a replacement.)
|
|
101
101
|
|
|
@@ -125,7 +125,7 @@ MIT
|
|
|
125
125
|
<!-- leerness:project-readme:start -->
|
|
126
126
|
## Leerness Project Harness
|
|
127
127
|
|
|
128
|
-
이 프로젝트는 Leerness v1.35.
|
|
128
|
+
이 프로젝트는 Leerness v1.35.13 하네스를 사용합니다. AI 에이전트는 작업 전 `leerness handoff`로 컨텍스트를 적재하고, 작업 후 `leerness check`/`leerness audit`/`leerness session close`를 수행해야 합니다.
|
|
129
129
|
|
|
130
130
|
### 정체성 — AI 에이전트 운영 레이어 (UR-0030)
|
|
131
131
|
|
|
@@ -179,7 +179,7 @@ leerness memory restore decision <date|title>
|
|
|
179
179
|
|
|
180
180
|
### MCP server (외부 AI 통합)
|
|
181
181
|
|
|
182
|
-
Leerness v1.35.
|
|
182
|
+
Leerness v1.35.13는 stdio JSON-RPC MCP server를 내장합니다 — Claude Code · Cursor · Codex CLI 등 외부 AI에 **86개 도구**를 노출:
|
|
183
183
|
|
|
184
184
|
```jsonc
|
|
185
185
|
// 카테고리별
|
|
@@ -200,7 +200,7 @@ Leerness v1.35.11는 stdio JSON-RPC MCP server를 내장합니다 — Claude Cod
|
|
|
200
200
|
`<<autonomous-loop-dynamic>>` 신호만 보내면 AI가:
|
|
201
201
|
1) 다음 라운드 후보 선정 → 2) 코드 변경 → 3) stress-v* 신규 작성 + 누적 회귀 → 4) e2e 219/219 → 5) npm pack + git tag + GitHub release → 6) main 자동 push (1.9.140+) → 7) session close → 8) 다음 라운드 예약.
|
|
202
202
|
|
|
203
|
-
현재 누적: **70 라운드 (1.9.40 → 1.35.
|
|
203
|
+
현재 누적: **70 라운드 (1.9.40 → 1.35.13)** · 매 라운드 GitHub release/태그 생성 · _reports/는 비공개 보존.
|
|
204
204
|
|
|
205
205
|
### 성능 가이드 (1.9.140 측정)
|
|
206
206
|
|
|
@@ -238,6 +238,6 @@ leerness release pack --close --auto-main-push
|
|
|
238
238
|
- `.harness/session-handoff.md`: 다음 세션 인수인계 (자동 작성)
|
|
239
239
|
- `.harness/lessons.md` / `decisions.md` / `rules.md`: 영구 메모리 (5 surface)
|
|
240
240
|
|
|
241
|
-
Last synced by Leerness v1.35.
|
|
241
|
+
Last synced by Leerness v1.35.13: 2026-07-06
|
|
242
242
|
<!-- leerness:project-readme:end -->
|
|
243
243
|
|
package/bin/leerness.js
CHANGED
|
@@ -32,7 +32,7 @@ const { _evidenceQuality, _parseEvidenceStats, _shellGuardAnalyze, _claimFileInG
|
|
|
32
32
|
// 1.9.295 (UR-0025 4단계): 정적 데이터 카탈로그 모듈 분리 (비파괴, require-based).
|
|
33
33
|
const { CAPABILITY_SURFACE, POWERFUL_COMMANDS, ADAPTERS, REUSE_CATEGORIES, REUSE_CHECKLIST, _DEFAULT_PLATFORM_CONSTRAINTS, _DEFAULT_DOMAIN_CATALOG, _TOOL_CATALOG, _LSP_LANG_PATTERNS, OPTIMISM_PATTERNS, BUILT_IN_PERSONAS, STRINGS, BUILTIN_CATALOG, ROADMAP_STATUS_LABEL, ROADMAP_STATUS_COLOR, SECRET_PATTERNS, MERGE_OVERWRITE_FILES, MINIMAL_SKIP_KEYS, REQUIRED_WORKSPACE_FILES, KEYWORD_STOPWORDS, SKILL_CATALOG_PRESETS } = require('../lib/catalogs'); // 1.9.344/368/369 (UR-0025): catalog 분리 · 1.11.4 (UR-0007): _TOOL_CATALOG
|
|
34
34
|
|
|
35
|
-
const VERSION = '1.35.
|
|
35
|
+
const VERSION = '1.35.13';
|
|
36
36
|
|
|
37
37
|
// 1.9.290 (UR-0037, Codex gpt-5.5 #4 수렴): CLI 전용 부작용은 require 시 실행하지 않는다.
|
|
38
38
|
// 이전: warning listener 제거 / NODE_OPTIONS 변경 / chcp IIFE 가 top-level 즉시 실행 → require('harness') 시 호스트 프로세스 오염.
|
|
@@ -2935,6 +2935,14 @@ function _selfTestCases() {
|
|
|
2935
2935
|
const claims6 = body.includes("step('verify-claims'") && body.includes("has('--claims')");
|
|
2936
2936
|
return has5 && claims6;
|
|
2937
2937
|
} },
|
|
2938
|
+
{ name: '3-tier 테스트 (1.35.13, UR-0014): package.json test:fast/test:core/test 3티어 + scripts/e2e-core.js 존재', run: () => {
|
|
2939
|
+
let pkg; try { pkg = require('../package.json'); } catch { return false; }
|
|
2940
|
+
const s = pkg.scripts || {};
|
|
2941
|
+
// fast=smoke, core=e2e-core, full=e2e — 3티어 스크립트 정합.
|
|
2942
|
+
const tiersOk = /smoke\.js/.test(s['test:fast'] || '') && /e2e-core\.js/.test(s['test:core'] || '') && /scripts\/e2e\.js/.test(s.test || '');
|
|
2943
|
+
const coreExists = exists(path.join(path.dirname(__filename), '..', 'scripts', 'e2e-core.js'));
|
|
2944
|
+
return tiersOk && coreExists;
|
|
2945
|
+
} },
|
|
2938
2946
|
{ name: '자체 contract 적대적 헌트 + codex 교차 (1.35.11): _parseImplExports bracket(#8) + _parseContractSpec 코드펜스 예제 제외(#9) — FP 2종 행위검사($ 필드 FP는 e2e cvdollar 담당)', run: () => {
|
|
2939
2947
|
const p = require('../lib/pure-utils');
|
|
2940
2948
|
const bracket = p._parseImplExports('exports["foo"] = function(){};\n').includes('foo'); // #8: bracket export 인식
|
|
@@ -3823,11 +3831,12 @@ function _selfTestCases() {
|
|
|
3823
3831
|
&& eq(F([]), [])
|
|
3824
3832
|
&& eq(F(['a.css', 'b.md', 'c.js', 'd.test.js']).length, 2); // 최대 2개 클러터 방지
|
|
3825
3833
|
} },
|
|
3826
|
-
{ name: 'GPT-5.5 평가 #5 (1.19.1, UR-0009): 클린룸
|
|
3834
|
+
{ name: 'GPT-5.5 평가 #5 (1.19.1, UR-0009) + 정직성 calibration (1.35.12): 클린룸 문서 공개 + 한계 명시 + self-administered 라벨 (행위)', run: () => {
|
|
3827
3835
|
const dp = path.join(path.dirname(__filename), '..', 'docs', 'clean-room-evaluations.md');
|
|
3828
3836
|
if (!exists(dp)) return true; // 패키지에 없으면 스킵(설치본 안전)
|
|
3829
3837
|
const d = read(dp);
|
|
3830
|
-
|
|
3838
|
+
// 1.35.12: 정직성 마커 강화 — clean-room 프레이밍 + heuristic 한계 + self-administered(독립 아님) + 재현 레시피.
|
|
3839
|
+
return /clean-room/i.test(d) && /heuristic, not semantic/i.test(d) && /self-administered/i.test(d) && /npm i leerness@/.test(d);
|
|
3831
3840
|
} },
|
|
3832
3841
|
{ name: 'CLI 영어화 Phase 1 (1.20.2, UR-0010): _uiLang 해석(flag>env>manifest>ko) + 첫화면 _t 적용 (행위+소스)', run: () => {
|
|
3833
3842
|
const save = process.argv; const saveEnv = process.env.LEERNESS_LANG;
|
|
@@ -1,67 +1,96 @@
|
|
|
1
|
-
# Clean-room evaluations / 클린룸 평가 기록
|
|
2
|
-
|
|
3
|
-
> **What this is, honestly.** These are **AI
|
|
4
|
-
> published npm package into fresh temp directories and
|
|
5
|
-
> source tree), including adversarial attempts to defeat leerness's own verifier. They are **
|
|
6
|
-
>
|
|
7
|
-
>
|
|
8
|
-
>
|
|
9
|
-
>
|
|
10
|
-
>
|
|
11
|
-
>
|
|
12
|
-
>
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
- **
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
|
38
|
-
|
|
39
|
-
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
1
|
+
# Clean-room evaluations / 클린룸 평가 기록
|
|
2
|
+
|
|
3
|
+
> **What this is, honestly.** These are **AI evaluations run by the project itself** — an agent installs the
|
|
4
|
+
> published npm package into fresh temp directories and exercises it from behavior only (no access to the
|
|
5
|
+
> source tree), including adversarial attempts to defeat leerness's own verifier. They are **self-administered**
|
|
6
|
+
> (the maintainer's AI, sometimes cross-checked by a second model such as Codex), **not** third-party human
|
|
7
|
+
> security audits or peer-reviewed benchmarks, and **not** independent in the "unaffiliated reviewer" sense.
|
|
8
|
+
> We publish them so the README claim "checkable by clean-room evaluation" is verifiable rather than a
|
|
9
|
+
> marketing line — you can re-run the recipe below yourself.
|
|
10
|
+
>
|
|
11
|
+
> 정직하게 말하면: 아래는 **프로젝트가 직접 수행한 AI 평가**입니다 — 에이전트가 게시된 npm 패키지를 빈 임시
|
|
12
|
+
> 폴더에 설치해 **소스 접근 없이 행위만으로** 검증하고, leerness 검증기 자체를 무력화하려는 적대 시도까지
|
|
13
|
+
> 포함합니다. 이는 **자기-수행**(메인테이너의 AI, 때때로 Codex 같은 2차 모델이 교차검증)이며, 제3자 인간 보안
|
|
14
|
+
> 감사·동료심사 벤치마크가 **아니고**, "무관한 외부 리뷰어"라는 의미의 독립도 **아닙니다**. README 의 "클린룸
|
|
15
|
+
> 평가로 확인 가능" 주장을 검증 가능하게 만들기 위해 공개하며, 아래 레시피로 직접 재현할 수 있습니다.
|
|
16
|
+
|
|
17
|
+
## Methodology / 방법론
|
|
18
|
+
|
|
19
|
+
Most evaluations follow this general shape:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
W=$(mktemp -d); cd "$W"; npm init -y
|
|
23
|
+
npm i leerness@<version> # the PUBLISHED package, not the working tree
|
|
24
|
+
LB="node node_modules/leerness/bin/leerness.js"
|
|
25
|
+
# ... drive the CLI by behavior, assert on exit codes + output, clean up ...
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
- **Published-artifact only** — installs from npm, never reads the repo source. This is what an external user gets.
|
|
29
|
+
- **Behavior-asserted** — checks process exit codes and stdout/stderr, not internal state.
|
|
30
|
+
- **Adversarial (both directions)** — many passes include attempts to make a *false* "done" claim pass
|
|
31
|
+
verification (comment-only stubs, empty-export shells, `assert(true)` fake tests, inflated test counts) **and**
|
|
32
|
+
to make an *honest* claim wrongly fail (false-positive hunting). A finding counts only if reproduced on the
|
|
33
|
+
published package; agent-proposed findings are re-derived, not trusted (맹신 X / "trust nothing").
|
|
34
|
+
|
|
35
|
+
## Agent clean-room evaluations / 에이전트 클린룸 평가 (behavior-only, published artifact)
|
|
36
|
+
|
|
37
|
+
| # | Round | Scope | Outcome |
|
|
38
|
+
|---|---|---|---|
|
|
39
|
+
| 1 | Universal-harness field study (5-axis) | Python / Node / Rust development exercises, agent-swap handoff, adversarial attack on the verifier | Verdict "conditionally yes"; found 5 gaps (2×P1, 3×P2) — all closed in 1.17.2–1.17.6, shipped as the 1.18.0 stable minor |
|
|
40
|
+
| 2 | 1.18.0 re-verification (published) | Separately re-ran the 5 gaps against the **published** 1.18.0 | 4 closed; surfaced a new P1 (Windows `--test-cmd python` blocked → false FAIL) → fixed in 1.18.1 |
|
|
41
|
+
| 3 | 1.18.2 adversarial stub workflow | Bypass-hunters + false-positive-hunters vs the empty-shell stub detector | 9 one-keyword bypasses found (Object.freeze, async fn, inline comment, …) and closed; **0 false positives** across ~45 legitimate patterns |
|
|
42
|
+
| 4 | 1.19.0 published-artifact re-verification | Installed `leerness@1.19.0` from npm; smoke-tested headline fixes | `--test-cmd python`, empty-shell rejection, `lens`, selftest — all pass on the published package |
|
|
43
|
+
|
|
44
|
+
Maintainer-run multi-model reviews (external models — GPT-class + Claude — driven against a published build and
|
|
45
|
+
reproduced before fixing) have also shaped many rounds; the confirmed findings are tracked in CHANGELOG under
|
|
46
|
+
the "Nth 외부평가 / 외부리뷰" entries. "External" here means the *models*, not unaffiliated human reviewers.
|
|
47
|
+
|
|
48
|
+
## Maintainer adversarial hunts / 메인테이너 자체 적대적 헌트 (self-administered, published artifact)
|
|
49
|
+
|
|
50
|
+
These are the **least independent** kind — the maintainer's AI hunting its own flagship surfaces, cross-checked
|
|
51
|
+
with Codex where noted. They are listed for transparency, not as third-party validation. Every finding was
|
|
52
|
+
reproduced on the published package; every fix re-verified after publish.
|
|
53
|
+
|
|
54
|
+
| Version | Surface | Method | Outcome (honest) |
|
|
55
|
+
|---|---|---|---|
|
|
56
|
+
| 1.35.9 | `verify-claim` declared-pass gate | 12-probe FP/FN matrix (single / `--json` / `--all` / `gate --claims` × reporter/growth/lenient) | 1 real false-positive found (a non-test `N/M passing` ratio wrongly gated) and fixed; 0 bypasses |
|
|
57
|
+
| 1.35.10 | `gate` (the CI guardrail) | 8-probe matrix (clean/placeholder/secret/false-done/missing-file) | **0 product bugs**; 2 probe-flagged "FN"s were probe artifacts, refuted by reproduction; added a gate-level security regression guard |
|
|
58
|
+
| 1.35.11 | `contract verify` (spec ↔ impl) | Claude probes + a separate Codex (gpt-5.5) review, cross-checked | Codex raised 9 candidates → **3 confirmed & fixed** (`$`-field regex FP, bracket-export FP, code-fence-example FP), **1 refuted as a hallucination**, the rest documented heuristic limits |
|
|
59
|
+
|
|
60
|
+
The pattern worth noting: two of the three surfaces had **0 product bugs found in these probes** only after a
|
|
61
|
+
dedicated false-positive hunt, and in each round a portion of the AI-proposed findings did **not** reproduce and
|
|
62
|
+
were dropped. That is the point of publishing this — the checks are fallible, the probe matrices are finite, and
|
|
63
|
+
the checks are themselves under adversarial test, including their own false-alarm behavior.
|
|
64
|
+
|
|
65
|
+
## What these evaluations do NOT establish / 한계 (정직)
|
|
66
|
+
|
|
67
|
+
leerness's verification is **heuristic, not semantic**. These evaluations suggest it catches many *obvious
|
|
68
|
+
false-done claims* (missing files, empty shells, fake/unlinked tests, inflated counts), with no false failures
|
|
69
|
+
in the tested cases — though the false-positive hunts have themselves found and fixed a real false-positive
|
|
70
|
+
(1.35.9), so "no false failures" is a property under test, not a guarantee. They do **not** prove:
|
|
71
|
+
|
|
72
|
+
- the implementation satisfies the actual requirement, or business logic is correct;
|
|
73
|
+
- tests are sufficient or well-designed;
|
|
74
|
+
- absence of security vulnerabilities or production-runtime correctness.
|
|
75
|
+
|
|
76
|
+
Known heuristic gaps tracked for an AST/token-based redesign (UR-0016): multi-arg call-expression empty objects
|
|
77
|
+
(`Object.assign({}, {})`), Python `def …: pass` / `...` / `raise NotImplementedError`, multi-language empty
|
|
78
|
+
bodies; and for `contract verify` (UR-0018) a field/function present only in a **comment or string** still
|
|
79
|
+
passes the grep-presence check, and a function exported as a **non-function value** passes (contract checks
|
|
80
|
+
*presence*; `verify-claim` checks *substance* — a deliberate split). `scan secrets` is a convenience guard, not
|
|
81
|
+
a replacement for a dedicated scanner (gitleaks/trufflehog). **Code coverage / mutation testing and third-party
|
|
82
|
+
reproduction remain open** (the e2e is a spawn-based runner, which standard coverage tools do not instrument
|
|
83
|
+
cleanly).
|
|
84
|
+
|
|
85
|
+
## Reproduce it yourself / 직접 재현
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
W=$(mktemp -d); cd "$W"; npm init -y && npm i leerness@latest
|
|
89
|
+
LB="node node_modules/leerness/bin/leerness.js"
|
|
90
|
+
$LB init "$W" --yes
|
|
91
|
+
# fake "done": claim an implementation that does not exist
|
|
92
|
+
$LB task add "Implement X"
|
|
93
|
+
$LB task update T-0001 --status done --evidence "x.js done, 5 tests passed"
|
|
94
|
+
$LB verify-claim T-0001 # exit 1 — claim rejected (x.js missing, no tests ran)
|
|
95
|
+
$LB selftest # core-function integrity self-check (exit 0 when the install is intact)
|
|
96
|
+
```
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "leerness",
|
|
3
|
-
"version": "1.35.
|
|
3
|
+
"version": "1.35.13",
|
|
4
4
|
"description": "Leerness: 비파괴 마이그레이션, 자동 버전 감지·업데이트, 계획/진행/핸드오프 자동화, 게으름·시크릿·인코딩 자동 가드, Claude Code 슬래시 통합을 갖춘 한국어 우선 AI 개발 하네스.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"leerness",
|
|
@@ -48,8 +48,9 @@
|
|
|
48
48
|
],
|
|
49
49
|
"scripts": {
|
|
50
50
|
"test": "node ./bin/leerness.js --version && node ./bin/leerness.js selftest && node ./scripts/e2e.js",
|
|
51
|
+
"test:core": "node ./bin/leerness.js --version && node ./bin/leerness.js selftest && node ./scripts/e2e-core.js",
|
|
51
52
|
"test:fast": "node ./bin/leerness.js selftest && node ./scripts/smoke.js",
|
|
52
|
-
"test:smoke": "node ./scripts/
|
|
53
|
+
"test:smoke": "node ./scripts/smoke.js",
|
|
53
54
|
"prepack": "node ./bin/leerness.js readme sync . && node ./bin/leerness.js --version"
|
|
54
55
|
},
|
|
55
56
|
"publishConfig": {
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
|
|
4
|
+
// 1.35.13 (UR-0014): 3-tier 테스트의 중간 티어 — critical-path 행위(behavioral) 스위트.
|
|
5
|
+
// test:fast (smoke.js) — 명령이 크래시 없이 도는가만 clean 프로젝트에서 확인, <1분. 개발 루프.
|
|
6
|
+
// test:core (이 파일) — 플래그십(verify-claim/gate/contract/scan)이 *적대적* 입력을 실제로 거부/통과하는가, ~1-2분. pre-commit / 빠른 CI.
|
|
7
|
+
// test (e2e.js) — 전체 384 케이스, ~12분, 릴리스 게이트.
|
|
8
|
+
// smoke 는 "명령이 도는가"만 본다. 이 core 는 "가치제안(거짓완료·시크릿·계약위반 차단)이 실제로 작동하는가"를 행위로 검증한다.
|
|
9
|
+
// 전체 e2e.js 를 건드리지 않는 별도 파일 — 릴리스 게이트 회귀 위험 0. 실패 시 exit 1 (CI 친화). 사용: npm run test:core
|
|
10
|
+
|
|
11
|
+
const fs = require('fs'), os = require('os'), path = require('path'), cp = require('child_process');
|
|
12
|
+
process.env.LEERNESS_OFFLINE = process.env.LEERNESS_OFFLINE || '1';
|
|
13
|
+
const CLI = path.resolve(__dirname, '..', 'bin', 'leerness.js');
|
|
14
|
+
let failed = 0, total = 0; const t0 = Date.now();
|
|
15
|
+
|
|
16
|
+
function run(dir, args) { return cp.spawnSync(process.execPath, [CLI, ...args, '--path', dir], { encoding: 'utf8', timeout: 40000 }); }
|
|
17
|
+
function assert(label, cond) { total++; process.stdout.write(`${cond ? '✓' : '✗'} ${label}\n`); if (!cond) failed++; }
|
|
18
|
+
function fresh() { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'leerness-core-')); cp.spawnSync(process.execPath, [CLI, 'init', d, '--yes', '--language', 'ko'], { encoding: 'utf8', timeout: 30000 }); return d; }
|
|
19
|
+
|
|
20
|
+
console.log('# leerness core (test:core) — flagship behavioral guarantees');
|
|
21
|
+
|
|
22
|
+
// (1) verify-claim: 거짓완료(없는 파일) 거부, 정직완료 통과
|
|
23
|
+
{
|
|
24
|
+
const d = fresh();
|
|
25
|
+
run(d, ['task', 'add', 'ghost']);
|
|
26
|
+
run(d, ['task', 'update', 'T-0002', '--status', 'done', '--evidence', 'ghost.js implemented; 5/5 passed']);
|
|
27
|
+
assert('verify-claim rejects false-done (missing file → exit 1)', run(d, ['verify-claim', 'T-0002']).status === 1);
|
|
28
|
+
fs.writeFileSync(path.join(d, 'real.js'), 'module.exports = { f: () => 1 };\n');
|
|
29
|
+
run(d, ['task', 'add', 'real']);
|
|
30
|
+
run(d, ['task', 'update', 'T-0003', '--status', 'done', '--evidence', 'real.js implemented']);
|
|
31
|
+
assert('verify-claim passes truthful done (exit 0)', run(d, ['verify-claim', 'T-0003']).status === 0);
|
|
32
|
+
fs.rmSync(d, { recursive: true, force: true });
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
// (2) verify-claim: 부풀린 테스트 카운트 거부 (1.35.7/9 플래그십)
|
|
36
|
+
{
|
|
37
|
+
const d = fresh();
|
|
38
|
+
fs.writeFileSync(path.join(d, 'x.js'), 'module.exports = { f: () => 1 };\n');
|
|
39
|
+
fs.writeFileSync(path.join(d, 'x.test.js'), 'test();\ntest();\n');
|
|
40
|
+
run(d, ['task', 'add', 'x']);
|
|
41
|
+
run(d, ['task', 'update', 'T-0002', '--status', 'done', '--evidence', 'x.js, x.test.js; 3/3 passed']);
|
|
42
|
+
assert('verify-claim rejects inflated pass (claim 3/3 vs run 2/2 → exit 1)', run(d, ['verify-claim', 'T-0002', '--run-tests', '--test-cmd', 'echo Tests: 2 passed, 2 total']).status === 1);
|
|
43
|
+
fs.rmSync(d, { recursive: true, force: true });
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
// (3) gate: 클린 통과 / 커밋 시크릿 차단 / --claims 거짓완료 차단
|
|
47
|
+
{
|
|
48
|
+
const d = fresh();
|
|
49
|
+
run(d, ['handoff', '']);
|
|
50
|
+
assert('gate clean project → exit 0', run(d, ['gate', '.']).status === 0);
|
|
51
|
+
fs.writeFileSync(path.join(d, 'leak.js'), 'const k = "AKIAJQXMP7RZ2KL9WXYZ";\nmodule.exports = k;\n');
|
|
52
|
+
assert('gate blocks committed secret → exit 1', run(d, ['gate', '.']).status === 1);
|
|
53
|
+
fs.rmSync(path.join(d, 'leak.js'), { force: true });
|
|
54
|
+
run(d, ['task', 'add', 'ghost']);
|
|
55
|
+
run(d, ['task', 'update', 'T-0002', '--status', 'done', '--evidence', 'ghost.js implemented; 5/5 passed']);
|
|
56
|
+
assert('gate --claims blocks false-done → exit 1', run(d, ['gate', '.', '--claims']).status === 1);
|
|
57
|
+
fs.rmSync(d, { recursive: true, force: true });
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
// (4) contract verify: 누락 함수 감지 / 충족 통과
|
|
61
|
+
{
|
|
62
|
+
const d = fresh();
|
|
63
|
+
fs.writeFileSync(path.join(d, 's.md'), '- charge()\n');
|
|
64
|
+
fs.writeFileSync(path.join(d, 'miss.js'), 'module.exports = { other: 1 };\n');
|
|
65
|
+
assert('contract catches missing function → exit 1', run(d, ['contract', 'verify', path.join(d, 's.md'), path.join(d, 'miss.js')]).status === 1);
|
|
66
|
+
fs.writeFileSync(path.join(d, 'ok.js'), 'function charge(){}\nmodule.exports = { charge };\n');
|
|
67
|
+
assert('contract passes when satisfied → exit 0', run(d, ['contract', 'verify', path.join(d, 's.md'), path.join(d, 'ok.js')]).status === 0);
|
|
68
|
+
fs.rmSync(d, { recursive: true, force: true });
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
// (5) scan secrets: 커밋 시크릿 차단 / 클린 통과
|
|
72
|
+
{
|
|
73
|
+
const d = fresh();
|
|
74
|
+
assert('scan secrets clean → exit 0', run(d, ['scan', 'secrets', '.']).status === 0);
|
|
75
|
+
fs.writeFileSync(path.join(d, 'k.js'), 'const k = "AKIAJQXMP7RZ2KL9WXYZ";\n');
|
|
76
|
+
assert('scan secrets committed key → exit 1', run(d, ['scan', 'secrets', '.']).status === 1);
|
|
77
|
+
fs.rmSync(d, { recursive: true, force: true });
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
const dur = ((Date.now() - t0) / 1000).toFixed(1);
|
|
81
|
+
console.log(`\nCore result: ${total - failed}/${total} passed · ${dur}s`);
|
|
82
|
+
if (failed > 0) process.exit(1);
|