patina-cli 3.11.0 → 4.0.0

package/docs/research/2025-rebaseline-plan.md

@@ -2,7 +2,8 @@
 
 Status: protocol template, no external detector/vendor claims yet.  
 Owner: maintainers.  
-Related issues: #155, #163, #170, #172, #213.
+Related issues: #155, #157, #160, #303, plus #158/#159 for evaluator follow-up. The #156 adversarial MPS gate is now in-tree.
+Korean source inventory: `docs/research/ko-2025-corpus-sources.md`.
 
 Patina's checked-in benchmark is a deterministic regression corpus. It is useful
 for catching tokenizer, threshold, lexicon, and fixture drift. It is not enough
@@ -24,12 +25,29 @@ Minimum matrix:
 | generators | at least one GPT-family, Claude-family, Gemini-family, and open-weight model |
 | sample size | start at 25 paragraphs per language × class × register cell before publishing claims |
 
+The 25-paragraph matrix is the first intake target for checking coverage holes.
+It is not the public benchmark gate. The stricter claim gate remains the
+`process/pattern-freshness.md` requirement: at least three generator families
+across at least two languages with n≥100 per claim cell and binomial 95%
+confidence intervals.
+
+
+## Execution order
+
+1. Start with Korean calibration (#303/#157): collect natural Korean controls for academic/종결-다, blog, product-doc, and community registers before changing KO thresholds again.
+2. Then run the model-era rebaseline (#155): score the fixed manifest across at least three generator families and at least two languages.
+3. Keep lexicon freshness separate from public catch-rate claims: English #160 remine now has per-entry provenance and ≥4× hot/cold lift evidence; rerun it during quarterly reviews, and do the same for KO/ZH/JA when paired corpora exist.
+4. Use the same manifest for cross-judge and blinded-panel follow-ups (#158/#159) instead of creating separate incompatible samples; keep the in-tree adversarial MPS gate current as a companion check.
+
 ## Data rules
 
 - Use redistributable prompts and generated text; do not check private user text into the repo.
 - Store full text only when redistribution is allowed. Otherwise store hashes, metadata, and metrics.
 - Keep generation metadata: model, provider, date, prompt id, decoding params, language, register, and any editing pass.
 - Separate detector-facing evaluation from rewrite-quality evaluation.
+- For Korean 2025+ rows, start from `docs/research/ko-2025-corpus-sources.md`
+  and run the local intake helper before any public report:
+  `npm run benchmark:rebaseline:intake -- --input artifacts/rebaseline-2025/intake.local.jsonl --require-source-review`.
 
 ## Metrics
 
@@ -37,7 +55,7 @@ For deterministic suspect-zone detection:
 
 - accuracy, precision, recall, F1
 - Wilson 95% confidence interval per language and register
-- detector sub-signal breakdown: burstiness, MATTR, lexicon
+- detector sub-signal breakdown: burstiness, MATTR, lexicon, koDiagnostics
 - expected metric ranges for checked-in fixtures
 
 For rewrite quality:
@@ -60,6 +78,11 @@ Recommended local/private layout before publishing sanitized summaries:
 
 ```text
 artifacts/rebaseline-2025/
+├── README.md                  # tracked scaffold and privacy rules
+├── intake.example.jsonl       # tracked smoke fixture, not evidence
+├── intake.local.jsonl         # gitignored local working input
+├── manifest.public.jsonl      # gitignored sanitized output for local review
+├── human-controls.public.jsonl # tracked hash-only KO web candidate manifest
 ├── prompts.jsonl
 ├── generations.jsonl          # only redistributable text
 ├── generations.private.jsonl  # gitignored/private text if needed
@@ -71,6 +94,52 @@ artifacts/rebaseline-2025/
 
 Checked-in public summaries should live under `docs/benchmarks/` after review.
 
+## Manifest scaffold
+
+Use the checked-in example manifest to validate the metadata contract before
+collecting a larger corpus:
+
+```bash
+npm run benchmark:rebaseline
+npm run benchmark:rebaseline:report
+node scripts/rebaseline-summary.mjs --input tests/quality/rebaseline-manifest.example.jsonl --json
+npm run benchmark:rebaseline:intake -- --input artifacts/rebaseline-2025/intake.example.jsonl --dry-run
+npm run benchmark:rebaseline:intake -- --input artifacts/rebaseline-2025/intake.local.example.jsonl --dry-run --require-source-review
+npm run benchmark:rebaseline:web -- --target-per-register 50 --max-per-source 12 --collected-at 2026-05-22
+npm run benchmark:rebaseline:score -- --input artifacts/rebaseline-2025/private/web-human-controls.generated.private.jsonl --output artifacts/rebaseline-2025/human-controls.public.jsonl --scored-at 2026-05-22
+node scripts/rebaseline-summary.mjs --input artifacts/rebaseline-2025/human-controls.public.jsonl --json
+```
+
+The manifest row schema is intentionally metadata-first:
+
+- required: `sample_id`, `language`, `class`, `register`, `model_family`,
+  `provider`, `model`, `generated_at`, `prompt_id`, `decoding`, `postprocess`,
+  `redistribution`, `text_hash`
+- optional: `text` only when redistribution permits it, `patina_score`,
+  `expected_hot`, `predicted_hot`, `source_url`, `source_license`,
+  `source_review`, `reviewer_notes`
+
+`text_hash` uses the same `sha256:<hex>` style as runtime manifests. If a row
+contains `text`, the validator checks that the digest matches. If a row is
+`metadata-only`, `private`, or `no-redistribution`, full text must stay out of
+the repository.
+
+`npm run benchmark:rebaseline:report` writes the sanitized summary to
+`docs/benchmarks/rebaseline-latest.md` and `.json`. The default report now uses
+the #155 claim-ready 2026 manifest; `tests/quality/rebaseline-manifest.example.jsonl`
+remains the small `BLOCKED` fixture for proving the gate fails incomplete
+evidence.
+
+The tracked `artifacts/rebaseline-2025/human-controls.public.jsonl` file is a
+250-row Korean web candidate manifest for validating the provenance and hash-only
+path, balanced at 50 rows per tracked register. It contains no raw text;
+score/outcome fields are deterministic false-positive evidence only and must not
+be used as a threshold or README performance claim by themselves.
+
+Use the false-positive form for person-written samples that should feed the
+human/natural side of future calibration:
+<https://github.com/devswha/patina/issues/new?template=false_positive.yml>.
+
 ## Publication gate
 
 Do not publish competitive claims until the report includes:

package/docs/research/2026-rebaseline.md

@@ -0,0 +1,102 @@
+# 2026 modern-model rebaseline
+
+Status: claim-ready for the checked-in deterministic benchmark surface as of 2026-05-22.
+Related issues: #155, #157, #160, #303.
+
+This page records the first #155-compatible rebaseline after the HC3-era claim was retired. It does not replace the broader protocol in [`2025-rebaseline-plan.md`](2025-rebaseline-plan.md); it is the current public claim surface for Korean and English only.
+
+## Inputs
+
+| input | rows | public text | notes |
+|---|---:|---:|---|
+| `artifacts/rebaseline-2025/private/modern-generations.private.jsonl` | 600 | 0 | Locally generated raw text from logged-in CLI surfaces; kept ignored under `private/`. |
+| `artifacts/rebaseline-2025/human-controls.public.jsonl` | 100 selected from 250 | 0 | Korean public-web human controls, hash-only and already scored. |
+| `artifacts/rebaseline-2025/private/hape-en.private.jsonl` | 100 selected from HAP-E human rows | 0 | English HAP-E human controls, transformed to hash-only public rows. |
+| `artifacts/rebaseline-2025/rebaseline-2026.scored.public.jsonl` | 800 | 0 | Public scored manifest: metadata, hashes, deterministic scores, and outcomes only. |
+
+Modern-model positive cells are balanced at n=100 for each language × family:
+
+| language | model family | local surface/model |
+|---|---|---|
+| en | gpt-family | `codex-cli` / `gpt-5.5` |
+| en | claude-family | `claude-cli` / `claude-sonnet-4-6` |
+| en | gemini-family | `gemini-cli` / `gemini-2.5-pro` |
+| ko | gpt-family | `codex-cli` / `gpt-5.5` |
+| ko | claude-family | `claude-cli` / `claude-sonnet-4-6` |
+| ko | gemini-family | `gemini-cli` / `gemini-2.5-pro` |
+
+The model rows are ordinary assistant completions requested through CLI tools, not API-temperature-controlled lab completions. Treat the result as a deterministic Patina detector rebaseline for these local surfaces, not as an authorship study.
+
+## Claim gate
+
+`npm run benchmark:rebaseline:claim-manifest -- --scored-at 2026-05-22` writes the public-safe manifest, then:
+
+```bash
+node scripts/rebaseline-summary.mjs \
+  --input artifacts/rebaseline-2025/rebaseline-2026.scored.public.jsonl \
+  --write \
+  --basename rebaseline-latest \
+  --require-claim-ready
+```
+
+Latest report: [`docs/benchmarks/rebaseline-latest.md`](../benchmarks/rebaseline-latest.md).
+
+| gate | result |
+|---|---:|
+| generator families with n≥100 positive cells | 3 |
+| languages with n≥100 positive cells | 2 |
+| qualified positive language × family cells | 6 |
+| natural/human languages with n≥100 | 2 |
+| complete expected/predicted outcome rows | 800 |
+| public raw text committed | 0 |
+
+## Headline metrics
+
+All intervals below are Wilson 95% confidence intervals.
+
+| metric | value |
+|---|---:|
+| Overall AI catch rate | 67.3% [63.5–71.0%] |
+| Overall human-control false-positive rate | 16.0% [11.6–21.7%] |
+| Precision | 92.7% |
+| F1 | 0.780 |
+| TP/FP/FN/TN | 404/32/196/168 |
+
+## Catch rate by language × model family
+
+| language | model family | n | catch rate | 95% CI | caught/missed |
+|---|---|---:|---:|---:|---:|
+| en | claude-family | 100 | 74.0% | 64.6%–81.6% | 74/26 |
+| en | gemini-family | 100 | 79.0% | 70.0%–85.8% | 79/21 |
+| en | gpt-family | 100 | 77.0% | 67.8%–84.2% | 77/23 |
+| ko | claude-family | 100 | 68.0% | 58.3%–76.3% | 68/32 |
+| ko | gemini-family | 100 | 62.0% | 52.2%–70.9% | 62/38 |
+| ko | gpt-family | 100 | 44.0% | 34.7%–53.8% | 44/56 |
+
+## False-positive rate by language
+
+| language | n | FP rate | 95% CI | FP/TN |
+|---|---:|---:|---:|---:|
+| en | 100 | 14.0% | 8.5%–22.1% | 14/86 |
+| ko | 100 | 18.0% | 11.7%–26.7% | 18/82 |
+
+## Interpretation
+
+- The old README headline, “91% Korean / 76% English,” is no longer the current claim. It came from smaller HC3-era/paired fixtures and overstated current Korean catch behavior.
+- English remains around the previous HC3 headline on these modern CLI samples (74–79% per family), but Korean is materially lower, especially the GPT-family cell at 44%.
+- Human-control false positives remain inside the existing ≤25% acceptance envelope overall, but register-level review is still needed before any threshold tightening.
+- This is an editing-hotspot detector benchmark. It must not be presented as a reliable author-attribution or detector-bypass guarantee.
+
+## Reproducibility notes
+
+- Raw generated and HAP-E text stays in ignored `artifacts/rebaseline-2025/private/` files. The committed manifest stores `sha256` digests, metadata, Patina scores, and outcome labels only.
+- The generation helper is `scripts/rebaseline-generate-modern.mjs`; the public-safe combiner is `scripts/rebaseline-build-claim-manifest.mjs`.
+- HAP-E is used only for English human controls in this report. The previous #160 HAP-E lexicon-mining note still applies: HAP-E is useful, but it is not a substitute for Korean or 2026 model positives.
+- `docs/benchmarks/rebaseline-latest.json` is the machine-readable companion to the Markdown report.
+
+## Remaining research work
+
+1. Add lightly/heavily edited-AI rows so rewrite behavior is tested separately from raw assistant completions.
+2. Add non-KO/EN languages once public controls and generated positives reach the same n≥100 cell gate.
+3. Repeat the generation on API surfaces with explicit decoding parameters if a stricter lab-style claim is needed.
+4. Review Korean GPT-family misses before changing thresholds; the current result argues for targeted KO diagnostics rather than global score inflation.

package/docs/research/adversarial-mps.md

@@ -0,0 +1,41 @@
+# Adversarial MPS audit
+
+This report checks whether a rewrite can preserve explicit meaning anchors while still looking AI-like. It is a repo-owned adversarial fixture set, not a public model-performance claim.
+
+Fixture source: `tests/quality/adversarial-mps/fixtures.jsonl`
+
+## Summary
+
+- Fixtures: 10
+- Passing adversarial cases: 10/10
+- Minimum anchor-MPS proxy: 100.0
+- Minimum deterministic AI score: 100.0
+- Gate: MPS proxy ≥90 and deterministic AI score ≥60.
+
+## Results
+
+| id | lang | register | MPS proxy | AI score | hot paragraphs | status |
+|---|---|---|---:|---:|---:|---|
+| adv-mps-ko-01 | ko | marketing | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-ko-02 | ko | technical | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-ko-03 | ko | academic | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-ko-04 | ko | product-doc | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-ko-05 | ko | policy | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-en-01 | en | marketing | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-en-02 | en | technical | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-en-03 | en | academic | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-en-04 | en | support | 100.0 | 100.0 | 1/1 | pass |
+| adv-mps-en-05 | en | strategy | 100.0 | 100.0 | 1/1 | pass |
+
+## Interpretation
+
+The audit confirms the known gap: an anchor-preservation floor can pass text that still retains AI-marker density. MPS should remain a meaning-safety floor, not a humanness score. A complementary anti-gaming check should penalize repeated AI-marker recurrence after rewrite, especially when MPS is high.
+
+## Proposed MPS-v2 companion check
+
+Keep MPS unchanged for semantic safety, then add an independent recurrence gate:
+
+1. Score the original and rewritten text with deterministic `analyzeText`.
+2. If `MPS ≥ 90` and rewritten AI score remains `≥ 60`, mark the candidate as `style_not_improved`.
+3. In Ouroboros selection, prefer candidates that pass MPS and lower the AI score; do not let high MPS alone rescue a visibly AI-like rewrite.
+4. Report preserved anchors and recurring AI markers separately so users can decide whether to edit more or keep the register.

package/docs/research/ai-human-metrics.md

@@ -1,7 +1,7 @@
 # AI 글과 사람 글 차이의 지표화 — patina 벤치마크 조사 노트
 
 작성일: 2026-05-20  
-상태: 조사/설계 노트, 구현 전 제안
+상태: 조사/설계 노트, 부분 구현 — deterministic benchmark와 opt-in live quality scaffold 반영
 
 ## 1. 질문
 
@@ -18,9 +18,9 @@ patina의 벤치마크에서 측정하려는 것은 단순히 “이 글의 출
 | 층 | 파일 | 현재 측정값 | 성격 |
 |---|---|---|---|
 | 결정론적 benchmark | `tests/quality/benchmark.mjs` | accuracy, precision, recall, F1, TP/FP/FN/TN | LLM 호출 없음. stylometry/lexicon hot 판정 회귀 테스트 |
-| future live quality regression | proposed `tests/quality/live-quality.mjs` | AI-likeness, MPS, fidelity, PASS/WARN/ERROR | 모델 호출 필요. rewrite 품질과 의미 보존 점검. 현재 main 기준 구현 전 follow-up |
+| opt-in live quality scaffold | `tests/quality/live-quality.mjs` | before/after AI-likeness, deterministic meaning proxy, safe_gain, PASS/WARN/FAIL | 기본 실행은 모델 호출 없이 skip. `OPENCODE_AVAILABLE=1`일 때 rewrite 품질과 의미 보존 proxy 점검 |
 | scoring spec | `core/scoring.md` | AI-likeness, fidelity, combined score, MPS | 점수 정의의 기준 문서 |
-| stylometry spec | `core/stylometry.md` | burstiness CV, MATTR, lexicon density | deterministic signal 정의 |
+| stylometry spec | `core/stylometry.md` | burstiness CV, MATTR, lexicon density, koDiagnostics | deterministic signal 정의 |
 
 현재 deterministic hot 판정은 다음 OR 규칙이다.
 
@@ -28,18 +28,28 @@ patina의 벤치마크에서 측정하려는 것은 단순히 “이 글의 출
 paragraph is SUSPECT iff
   burstiness_band == "low"
   OR MATTR_band == "low"
-  OR lexicon_density > threshold
+  OR (lexicon_density > threshold AND lexicon_min_hits is satisfied)
+  OR koDiagnostics.hot == true
 ```
 
-현재 레포의 로컬 `tests/quality/results.json` 스냅샷은 ko/en/zh/ja fixture 34개 기준 전체 accuracy 1.0이다. 다만 이 corpus는 작고 설계된 synthetic/curated fixture이므로, 일반화 성능 증거로 과신하면 안 된다.
+`burstiness_band`는 단락 문장이 3개 이상일 때만 부여한다. 2문장 이하 CV는
+진단값으로만 남기고, 그 값만으로는 hot 판정을 만들지 않는다.
+한국어/중국어/일본어 lexicon은 단일 hit를 hot으로 보지 않고 audit hint로만 남긴다.
+짧은 전문/정책 문단에서 보통 명사 하나가 걸리는 오탐을 줄이기 위한 보수적 가드다.
+
+한국어(`lang=ko`)는 `spacing`, `comma`, `posDiversity` 진단값을 추가로 남긴다.
+`posDiversity`는 형태소 분석기가 아니라 조사/어미 suffix class proxy다. 세 지표가 모두
+보수적 임계값을 넘을 때만 `koDiagnostics.hot`으로 OR 규칙에 들어간다.
+
+현재 공개 benchmark 스냅샷은 ko/en/zh/ja fixture 39개 기준 전체 accuracy 1.0이다. 다만 이 corpus는 작고 설계된 synthetic/curated fixture이므로, 일반화 성능 증거로 과신하면 안 된다.
 
 ### 2.1 단기 benchmark 한계와 rebaseline 계획 (#155/#162)
 
 현재 체크인된 deterministic report는 회귀 테스트로는 유용하지만, 공개 성능 주장으로 쓰기에는 아직 좁다.
 
-- 표본 수: ko/en/zh/ja suspect-zone fixture 34개. 언어·장르·출처별 신뢰구간을 낼 만큼 크지 않다.
+- 표본 수: ko/en/zh/ja suspect-zone fixture 39개. 언어·장르·출처별 신뢰구간을 낼 만큼 크지 않다.
 - 모델 시대성: 2025+ GPT/Claude/Gemini/Llama/Qwen 계열 생성문 rebaseline은 아직 별도 follow-up이다.
-- 통계 보고: `docs/benchmarks/latest.md`는 fixture 수, lang/class sample size, Wilson 95% CI를 공개한다. bootstrap interval과 threshold sweep은 아직 없다.
+- 통계 보고: `docs/benchmarks/latest.md`는 fixture 수, lang/class sample size, Wilson 95% CI와 `signal_score` 기반 ROC-AUC / PR-AUC / best-F1 threshold 진단을 공개한다. bootstrap interval은 아직 없다.
 - 범위: 현재 수치는 stylometry/lexicon hot 판정 회귀이며, rewrite 품질·MPS·fidelity의 live 품질 점수가 아니다.
 
 따라서 README/benchmark 문구는 “현재 fixture에서 회귀가 통과했다”로 해석해야 하며, “새 모델·새 장르에서도 100% 탐지한다”는 의미가 아니다. 다음 rebaseline에서는 최소한 언어 × class × register별 sample size를 고정하고, confidence interval과 threshold sweep을 함께 게시한다.
@@ -66,7 +76,7 @@ MATTR는 길이에 민감한 TTR 문제를 완화하기 위해 moving window를
 
 - 근거: Covington & McFall의 MATTR 논문은 단순 TTR의 길이 의존 문제를 지적하고 moving-average 방식을 제안한다.
 - patina 관찰: 영어에서는 보조 신호, 한국어에서는 어절 토큰화 때문에 MATTR가 과대평가되어 약한 신호다.
-- patina 판단: **현행 유지. 한국어에서는 핵심 신호로 보지 않는다.**
+- patina 판단: **현행 유지. 한국어에서는 주요 신호로 보지 않는다.**
 
 ### 3.3 AI-favored lexicon density
 
@@ -80,7 +90,7 @@ density = matched_ai_lexicon_entries / token_count * 1000
 - 약점: 도메인별 정상 전문어와 AI 포장어를 혼동할 수 있다.
 - patina 판단: **언어/도메인별 calibration이 필요하다.** 특히 한국어 lexicon은 paired corpus mining 방식이 효과적이었다.
 
-### 3.4 패턴 기반 severity score
+### 3.4 패턴 severity score
 
 patina의 가장 중요한 차별점은 “AI detection”보다 “AI writing pattern editing”에 있다. 따라서 다음 패턴들은 detector feature이면서 동시에 rewrite target이다.
 
@@ -91,13 +101,13 @@ patina의 가장 중요한 차별점은 “AI detection”보다 “AI writing p
 - 출처 없는 권위 주장
 - viral-hook score-only rhetoric
 
-patina 판단: **패턴 기반 점수는 계속 중심축이어야 한다.** 통계 지표는 패턴 스캔의 보조 신호로 둔다.
+patina 판단: **패턴 점수는 계속 중심축이어야 한다.** 통계 지표는 패턴 스캔의 보조 신호로 둔다.
 
-### 3.5 언어모델 확률 기반 지표
+### 3.5 언어모델 확률 지표
 
 대표 연구:
 
-| 접근 | 핵심 아이디어 | patina 적용성 |
+| 접근 | 주요 아이디어 | patina 적용성 |
 |---|---|---|
 | [GLTR](https://arxiv.org/abs/1906.04043) | token probability, rank, entropy로 “너무 예측 가능한 선택”을 시각화 | 설명 가능. 단, local LM 또는 API logprob 필요 |
 | [DetectGPT](https://arxiv.org/abs/2301.11305) | 생성 텍스트가 log-probability curvature의 특정 영역에 놓인다는 가설 | 강력하지만 perturbation/LM logprob 비용 큼 |
@@ -118,13 +128,15 @@ patina 판단: **기본 benchmark에는 넣지 말고 optional research track으
 - sentence opener diversity
 - POS-like rough proxy: 명사형 종결, 수동 표현, nominalization suffix
 
-patina 판단: **다음 deterministic feature 후보 1순위.** 외부 모델 없이 구현 가능하고, pattern/lexicon과 다른 축이다.
+patina 판단: **한국어부터 보수적 composite로 시작했다.** 외부 모델 없이 구현 가능하고,
+pattern/lexicon과 다른 축이라 유망하지만, 쉼표 없음 같은 단일 특징은 hot 판정에 넣지 않는다.
+spacing/comma/suffix proxy가 동시에 맞을 때만 `koDiagnostics.hot`을 켠다.
 
 ## 4. 탐지 연구에서 얻을 주의사항
 
 ### 4.1 detector는 일반적으로 OOD에 약하다
 
-[M4](https://arxiv.org/abs/2305.14902)는 multi-generator, multi-domain, multilingual 환경에서 detector의 generalization이 어렵다고 보고한다. [RAID](https://arxiv.org/abs/2405.07940)는 sampling strategy, adversarial attack, unseen model 변화에 기존 detector들이 쉽게 속는다고 보고한다.
+[M4](https://arxiv.org/abs/2305.14902)는 multi-generator, multi-domain, multilingual 맥락에서 detector의 generalization이 어렵다고 보고한다. [RAID](https://arxiv.org/abs/2405.07940)는 sampling strategy, adversarial attack, unseen model 변화에 기존 detector들이 쉽게 속는다고 보고한다.
 
 patina 벤치마크도 같은 함정이 있다. synthetic fixture에서 100%가 나와도 다음 상황에서는 깨질 수 있다.
 
@@ -136,7 +148,7 @@ patina 벤치마크도 같은 함정이 있다. synthetic fixture에서 100%가
 
 ### 4.2 단일 “AI 확률”로 말하면 안 된다
 
-OpenAI도 기존 AI classifier를 낮은 정확도 때문에 중단했고, 짧은 텍스트·비영어·OOD·편집된 AI 글에서 취약하다고 명시했다. 공식 문서의 핵심 메시지는 “주요 의사결정 도구로 쓰지 말라”는 것이다.
+OpenAI도 기존 AI classifier를 낮은 정확도 때문에 중단했고, 짧은 텍스트·비영어·OOD·편집된 AI 글에서 취약하다고 명시했다. 공식 문서의 주요 메시지는 “주요 의사결정 도구로 쓰지 말라”는 것이다.
 
 patina도 `AI-generated probability`가 아니라 다음처럼 표현해야 한다.
 
@@ -150,7 +162,7 @@ patina도 `AI-generated probability`가 아니라 다음처럼 표현해야 한
 - watermark: 특정 생성 시스템이 사전에 심은 신호 검출
 - patina: 결과 텍스트의 문체적 AI 신호 축소
 
-따라서 watermark는 참고 연구로만 두고, patina benchmark의 핵심 지표로 넣지 않는다.
+따라서 watermark는 참고 연구로만 두고, patina benchmark의 주요 지표로 넣지 않는다.
 
 ## 5. patina에 맞는 지표 설계안
 
@@ -196,10 +208,10 @@ deterministic_ai_likeness =
 
 | Feature | 초기 weight | 이유 |
 |---|---:|---|
-| pattern severity | 0.35 | patina의 핵심 목적과 직접 연결 |
+| pattern severity | 0.35 | patina의 주요 목적과 직접 연결 |
 | lexicon density | 0.20 | 설명 가능하고 calibration 경험 있음 |
-| burstiness CV | 0.20 | 현재 ko/en/zh/ja suspect-zone fixture에서 유효한 핵심 deterministic 신호 |
-| function-word divergence | 0.15 | 다음 확장 후보. 주제 독립성이 높음 |
+| burstiness CV | 0.20 | 현재 ko/en/zh/ja suspect-zone fixture에서 유효한 주요 deterministic 신호 |
+| function-word divergence | 0.15 | 한국어 suffix proxy부터 보수적 composite로 시작. 주제 독립성이 높음 |
 | punctuation/opening rhythm | 0.05 | 가벼운 보조 신호 |
 | MATTR | 0.05 | 한국어에서 약하므로 낮게 시작 |
 
@@ -237,7 +249,7 @@ safe_gain = max(0, humanization_gain) * (meaning_safety / 100)
 
 #### Stability
 
-LLM 기반 rewrite는 비결정적이므로 같은 fixture를 N회 돌린 분산도 필요하다.
+LLM rewrite는 비결정적이므로 같은 fixture를 N회 돌린 분산도 필요하다.
 
 ```text
 score_stability = stddev(after_ai_likeness over N runs)
@@ -306,13 +318,13 @@ notes: |
 ### Phase 1 — 보고 강화, 의존성 없음
 
 - `tests/quality/results.json`에 feature vector를 더 상세히 저장
-- threshold sweep과 ROC-AUC/PR-AUC 계산 추가
+- 더 큰 2025+ corpus에서 threshold sweep과 ROC-AUC/PR-AUC 재계산
 - per-register/per-language/per-class 요약 추가
 - `tests/quality/README.md`에 “AI-likeness이지 provenance가 아님” 명시
 
 ### Phase 2 — corpus 확장
 
-- synthetic/curated fixture 34개에서 real-world sampled fixture로 확장
+- synthetic/curated fixture 39개에서 real-world sampled fixture로 확장
 - human false positive register를 먼저 늘린다
 - 목표: 언어별 최소 100 human + 100 AI paragraph
 
@@ -357,7 +369,7 @@ patina가 따라야 할 방향은 범용 AI detector가 아니다. 더 강한
 4. **단일 AI 확률을 주장하지 않는다.**
 5. 무거운 LM-probability detector는 optional research track으로 격리한다.
 
-즉, patina benchmark의 핵심 지표는 다음 세 문장으로 요약할 수 있다.
+즉, patina benchmark의 주요 지표는 다음 세 문장으로 요약할 수 있다.
 
 ```text
 얼마나 AI스럽게 읽히는가?

package/docs/research/human-eval-panel.md

@@ -0,0 +1,42 @@
+# Blinded human evaluation panel plan
+
+Status: study design ready; panel not run.
+Related issue: #159.
+
+This plan keeps human preference work separate from deterministic scoring. It should run only with texts that can be shown to reviewers and redistributed or summarized under consent.
+
+## Research question
+
+Can blinded readers tell which text is the AI-like draft and which one is the Patina rewrite, and do they prefer the rewrite without seeing meaning loss?
+
+## Minimum pilot
+
+- 30 paired samples;
+- 5 raters per sample;
+- language and register recorded for each pair;
+- randomized A/B order;
+- no model or tool names shown to raters;
+- reviewer consent and redistribution notes stored outside public fixtures unless publishable.
+
+## Rater task
+
+For each pair, ask:
+
+1. Which version reads more natural for the stated context?
+2. Did either version lose a key fact, number, name, or caveat?
+3. Which version would you send with light edits?
+4. Free-text note, optional.
+
+## Report shape
+
+| metric | output |
+|---|---|
+| naturalness preference | Patina / original / tie counts with confidence interval |
+| meaning concern | rate of reported fact/caveat loss |
+| register split | results by language and register |
+| rater agreement | kappa or raw agreement, depending on labels |
+| exclusions | samples removed and why |
+
+## Privacy rule
+
+Do not commit reviewer names, private comments, or no-redistribution source text. Public reports can include aggregate counts and short examples only when the source license and reviewer consent allow it.

package/docs/research/judge-agreement.md

@@ -0,0 +1,24 @@
+# Cross-judge agreement plan
+
+Status: CLI shortcut removed during surface simplification; full matrix blocked on evaluator budget.
+Related issue: #158.
+
+A score is more useful when the judge is not from the same model family as the suspected generator. Patina no longer exposes a per-run CLI warning for this; cross-family independence belongs in an explicit evaluation matrix rather than everyday score UX.
+
+## Full matrix gate
+
+The full issue is still open until a report covers:
+
+- 3 generator families × 3 judge families × 30 samples;
+- shared prompts and fixed sample ids;
+- pairwise agreement table;
+- Krippendorff alpha or Cohen/Fleiss kappa where the labels support it;
+- a note when a judge is evaluating its own family.
+
+## Matrix template
+
+| sample set | generator | judge | n | hot agree | hot disagree | agreement |
+|---|---|---|---:|---:|---:|---:|
+| pending | pending | pending | 0 | 0 | 0 | n/a |
+
+Do not fill this table with synthetic numbers. Use it only after the manifest and judge outputs exist.

package/docs/research/ko-2025-corpus-sources.md

@@ -0,0 +1,135 @@
+# KO/2025+ Corpus Source Inventory
+
+Verified: 2026-05-22. This inventory turns the Korean rebaseline blocker
+(#303/#157/#155/#160) into an executable intake plan. It is not a public
+performance claim.
+
+## Decision
+
+Use a **metadata-first corpus**:
+
+1. keep raw text in `artifacts/rebaseline-2025/` or another private store;
+2. commit only redistributable examples, hashes, metadata, and aggregate reports;
+3. do not publish catch-rate claims until `docs/research/2025-rebaseline-plan.md`
+   reaches its n≥100 public claim gate.
+
+That lets Patina use current Korean sources without copying restricted corpora
+into the repository.
+
+## Source matrix
+
+| source | role | evidence | repo policy | first intake |
+|---|---|---|---|---|
+| KatFish / KatFishNet | Korean AI-vs-human seed set; especially useful for #303 punctuation/spacing checks | ACL 2025/arXiv paper says KatFish covers human and four-LLM Korean text across three genres; the public GitHub repo contains `katfish_dataset/*.jsonl` but no repo-level license metadata is exposed | Treat raw rows as **license-review** until a license/redistribution decision is recorded. Aggregate metrics are OK; raw rows stay private. | `docs/benchmarks/katfish-ko-latest.md` reports aggregate-only calibration: Patina KO diagnostics improve KatFish catch rate by +15.9 pp with +0 public-web human-control FP rows. |
+| 모두의 말뭉치 | human Korean controls for product-doc, news/editorial, dialogue, summary registers | NIKL lists 2024/2025 corpora and requires corpus application/approval fields | Do not commit raw text. Store local extracts privately and commit only hashes/metrics unless approval explicitly allows publication. | Apply for research/evaluation use; start with 25 human-control paragraphs after approval. |
+| 한국어 학습자 말뭉치 | false-positive stress test for learner/second-language Korean, not a normal human baseline | Official site describes learner writing/speech corpora and notes original learner material is privacy-protected; Data.go.kr lists research-purpose distribution limits | Use as a separate FP envelope. Do not blend with native Korean controls. Commit metadata/hashes only. | Add `class: natural-human`, `register: learner-writing` only after schema/register decision, or map to `academic-summary` with reviewer note. |
+| HAERAE-HUB/KOREAN-SyntheticText-1.5B | broad synthetic Korean AI-like pool | Hugging Face dataset page shows text parquet with 1.55M rows | Synthetic side only. Check dataset card/license before committing full text; otherwise hash-only. | Sample short paragraphs for lexicon mining candidates, then manually review before pattern changes. |
+| Maintainer-generated 2026 prompts | controlled GPT/Claude/Gemini/open-weight model-era rows | Generated from repo-owned prompts and reproducible metadata | Preferred public seed when provider terms and prompt contents allow redistribution. Keep prompts public; keep vendor UI copies private if unsure. | Generate 5 rows each for GPT-family, Claude-family, Gemini-family, and open-weight across blog/product-doc/chat-update. |
+| Community false-positive submissions | real Patina FP cases | GitHub false-positive issue template captures language/register/score output | Use only with explicit fixture permission. Strip account/private context by default. | Convert accepted issues into hash-only rows first; promote to fixture only after permission. |
+| Public Korean web pages (Korea.kr, Toss Tech, Kakao/Naver/Toss docs, KISTEP/KEI/NRF, Seoul OpenGov) | natural-human Korean pilot controls for source/provenance workflow | Public pages with visible source URLs and licensing/copyright guidance where available | Commit only hash-only metadata until page-level redistribution and attribution are reviewed. Keep raw extracts in ignored `artifacts/rebaseline-2025/private/`. | `artifacts/rebaseline-2025/human-controls.public.jsonl` contains 250 scored hash-only candidate rows, with 50 rows each for blog, product-doc, academic-summary, technical-how-to, and chat-update; positive AI-like rows still block threshold work. |
+
+## Intake commands
+
+Use the local workspace scaffold:
+
+```bash
+npm run benchmark:rebaseline:intake -- \
+  --input artifacts/rebaseline-2025/intake.local.jsonl \
+  --public-output artifacts/rebaseline-2025/manifest.public.jsonl \
+  --private-output artifacts/rebaseline-2025/private/generations.private.jsonl \
+  --require-source-review
+
+node scripts/rebaseline-summary.mjs \
+  --input artifacts/rebaseline-2025/manifest.public.jsonl \
+  --json
+```
+
+The intake helper computes missing `text_hash` values. If redistribution is not
+public, it strips `text` from the public manifest and writes the full row to the
+private output path.
+
+Tracked starter files:
+
+- `artifacts/rebaseline-2025/prompts.template.jsonl` — repo-owned prompt
+  anchors for Korean academic/종결-다, product-doc, blog, chat/update,
+  technical-how-to, and edited-AI rows.
+- `artifacts/rebaseline-2025/intake.local.example.jsonl` — 25 metadata-only
+  rows matching the pilot buckets below. The hashes are placeholders; replace
+  them locally before treating the file as evidence.
+- `artifacts/rebaseline-2025/sources.ko-public.jsonl` — tracked public-source
+  inventory for hash-only web collection.
+- `artifacts/rebaseline-2025/human-controls.public.jsonl` — 250 web-sourced
+  Korean natural-human candidate rows, balanced at 50 rows for each tracked
+  register. It is hash-only and validates the collection path; absent AI-like
+  cells still block public claim changes.
+
+To refresh public-web candidates from the tracked source inventory:
+
+```bash
+npm run benchmark:rebaseline:web -- \
+  --input artifacts/rebaseline-2025/sources.ko-public.jsonl \
+  --output artifacts/rebaseline-2025/private/web-human-controls.generated.private.jsonl \
+  --target-per-register 50 \
+  --max-per-source 12 \
+  --collected-at 2026-05-22
+
+npm run benchmark:rebaseline:score -- \
+  --input artifacts/rebaseline-2025/private/web-human-controls.generated.private.jsonl \
+  --output artifacts/rebaseline-2025/human-controls.public.jsonl \
+  --scored-at 2026-05-22
+
+node scripts/rebaseline-summary.mjs \
+  --input artifacts/rebaseline-2025/human-controls.public.jsonl \
+  --json
+
+npm run benchmark:katfish-ko -- --write --basename katfish-ko-latest
+```
+
+To score those candidates without committing raw text:
+
+```bash
+npm run benchmark:rebaseline:score -- \
+  --input artifacts/rebaseline-2025/private/web-human-controls.generated.private.jsonl \
+  --output artifacts/rebaseline-2025/human-controls.public.jsonl \
+  --scored-at 2026-05-22
+```
+
+## Remaining Korean pilot holes
+
+The 250-row public-web pilot proves the collection and scoring path and fills the native-human register coverage gate for #157. Use the original 25-row skeleton only as a local intake template; future rows should fill the positive and comparison holes instead:
+
+| bucket | remaining need | notes |
+|---|---:|---|
+| native human controls | 0 for the five tracked registers | Current tracked split is n=50 each for academic-summary, product-doc, chat-update, blog, and technical-how-to. |
+| self-generated AI-like | n≥100 per GPT/Claude/Gemini/open-weight claim cell | Keep prompt ids, model ids, decoding, and provider terms notes; KatFish aggregate closes only the KO diagnostic calibration, not public claim coverage. |
+| lightly/heavily edited AI | at least one light and one heavy edit per target register | Preserve before/after hashes and edit policy. |
+| KatFish aggregate comparison | complete for #303 | Raw rows stay private; tracked output is aggregate-only. |
+| FP submissions / learner stress | separate reviewed envelope | Do not blend learner Korean into the native-human baseline. |
+
+Exit criteria for the pilot:
+
+- `npm run benchmark:rebaseline:intake -- --input artifacts/rebaseline-2025/intake.local.jsonl --dry-run --require-source-review`
+  passes before any rows are scored.
+- `node scripts/rebaseline-summary.mjs --input <manifest>` validates with no errors.
+- every row has `source_review` or `reviewer_notes` explaining redistribution
+  status when raw text is absent;
+- no threshold or README catch-rate claim changes are made from the pilot alone;
+- findings are posted back to #303/#157/#155 before #160 lexicon mining starts.
+
+## Source links
+
+- KatFishNet paper: <https://arxiv.org/abs/2503.00032>
+- KatFishNet repository: <https://github.com/Shinwoo-Park/katfishnet>
+- 모두의 말뭉치 request page: <https://kli.korean.go.kr/corpus/main/requestMain.do>
+- 모두의 말뭉치 introduction: <https://kli.korean.go.kr/m/introduce/corpusIntroduce.do>
+- 한국어 학습자 말뭉치: <https://kcorpus.korean.go.kr/index/goIntroduceSite.do>
+- 한국어 학습자 말뭉치 Data.go.kr metadata: <https://www.data.go.kr/data/15094033/fileData.do>
+- HAERAE Korean SyntheticText: <https://huggingface.co/datasets/HAERAE-HUB/KOREAN-SyntheticText-1.5B>
+- KOGL introduction: <https://www.kogl.or.kr/info/introduce.do>
+- KOGL license guide: <https://www.kogl.or.kr/info/license.do>
+- Korea.kr policy article: <https://www.korea.kr/news/policyNewsView.do?newsId=148959377>
+- MCST KOGL type guide: <https://www.mcst.go.kr/site/s_open/kogl/koglType.jsp>
+- MCST copyright Q&A: <https://www.mcst.go.kr/site/s_policy/copyright/question/question17.jsp>
+- Seoul OpenGov copyright policy: <https://opengov.seoul.go.kr/copyright>
+- Tracked public-web source inventory:
+  `artifacts/rebaseline-2025/sources.ko-public.jsonl`