asciidoc-pubkit 0.1.2 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: de07af1758fdba08a7099b9e66f63b14084d0dad6cba8437498f0ddd0577b425
4
- data.tar.gz: 8474d883c10c23f1221ff85cbd023d781c84f93dd55436a542c0d304c0a8f424
3
+ metadata.gz: c946ae67aae9ea8e6e21f5464b61daec42a18488c5aca5fbe434921f13dad824
4
+ data.tar.gz: 47ca9c1414c78699dbb07ebe3bfdb07460cc132a8a10c642392d147eb2247b90
5
5
  SHA512:
6
- metadata.gz: 6983524a7967131b469360bebd417125d7eea14cdf8054d14c928940528d685ee50fd0c402ca354073a55e643217ed931ddda3ca3e402d252d02e4030b7b3a9e
7
- data.tar.gz: 8e8111fea8d32dfb1449c48cfc8d8f2df22e58dec14a44a55c6dcb6596a6bd3e87aa86fbd596cfc332a6ca49c9d1452f5dd34b23126137d9359a081585e1ae56
6
+ metadata.gz: f8e1382976343a206f04363bdfb0896e3633605f239c2fbfb46afacaaf11415a93757a418c09d9fc449700742b15268f69883a36e0c7e200f9f3e73b4f4a0f22
7
+ data.tar.gz: 3f0395078d442df25b730cde44186cb1868c3b68f7ea292438a2ad8871179a30198524fcd846c5ea6f2ce84d65d90f01b9d4db829f012b6e47e68f4b2051d3da
data/CHANGELOG.md CHANGED
@@ -1,5 +1,52 @@
1
1
  # Changelog
2
2
 
3
+ ## Unreleased
4
+
5
+ ## 0.6.1 — 2026-10-01
6
+
7
+ - Add `review score FILE` for direct AsciiDoc manuscript scoring without a
8
+ session. Report a candidate-density indicator by default and optionally invoke
9
+ a local Codex or Claude CLI for readability ratings with `--agent`. Preserve
10
+ manuscript files, coverage limits, explicit tokenizer selection, and
11
+ `meaning_verified: false`.
12
+ - Add a development-only prose benchmark with a fixed AsciiDoc corpus, scored
13
+ candidate coverage and locations, and a saved MeCab baseline before detector
14
+ changes. Keep quality judgments separate from detection performance.
15
+ - Add explicit local Codex/Claude rewrite and judge trials with fact checklists,
16
+ blinded document labels, evaluator calibration, repeated-run statistics, and
17
+ compatibility checks for baseline comparisons. Keep benchmark AI trials
18
+ separate from manuscript scoring and the default test suite.
19
+
20
+ - Extend the shared Japanese prose criteria with evidence-based metaphor review,
21
+ distinctions between instructions, behavior, and capabilities, and separate
22
+ checks for information loss and unsupported additions.
23
+ - Add limited metaphorical-operation phrases as contextual review candidates,
24
+ with a Japanese evaluation corpus for detection and manual meaning review.
25
+ - Introduce yomiyasu in the README references.
26
+
27
+ ## 0.6.0 — 2026-09-30
28
+
29
+ - Add `writing criteria` and `writing prompt` commands backed by one packaged
30
+ Japanese technical prose guide. Include that guide in saved review prompts.
31
+ - Organize the CLI into `writing` and `review` groups and accept `--lang ja` on
32
+ both. Reject unsupported languages explicitly, including document attributes.
33
+ - Save the shared criteria and language with each review session, and update the
34
+ book writing and review skills to use the generated prompts.
35
+ - Refine Japanese review questions for redundant qualifications and generic
36
+ recaps, and document the editorial sources and contextual limits.
37
+
38
+ ## 0.1.3 — 2026-09-29
39
+
40
+ - Consolidate Japanese review candidates for terminology, compressed noun
41
+ relationships, operations, and negative qualifications in the packaged rules.
42
+ - Suppress literal candidates contained in longer matched terms, preserving
43
+ allow-list handling, and recognize sahen potential forms before noun rules.
44
+ - Reduce single-file review prompts by emitting each paragraph once, sharing
45
+ context metadata, and compacting JSON; retain all candidates and review settings.
46
+ - Confirm replacement of existing review sessions in interactive scans.
47
+ - Add `review scan --yes` (`-y`) and `--no-input` for batch execution.
48
+ - Preserve previous sessions until replacement scan artifacts are ready.
49
+
3
50
  ## 0.1.2 — Released
4
51
 
5
52
  - Package default Japanese review rules as a validated UTF-8 YAML file.
data/README.md CHANGED
@@ -5,25 +5,38 @@
5
5
 
6
6
  # asciidoc-pubkit
7
7
 
8
- A toolkit for authoring, reviewing, and publishing AsciiDoc books.
8
+ A toolkit for authoring and reviewing AsciiDoc books.
9
9
 
10
- The initial release reviews Japanese running prose, generates contextual review
11
- prompts for coding agents, and checks edited manuscripts against a saved baseline.
10
+ Version 0.6.1 provides shared Japanese technical writing criteria, an authoring
11
+ prompt, contextual review prompts, manuscript scoring, and baseline verification
12
+ for edited manuscripts.
12
13
  The CLI, diagnostics, documentation, and generated instructions are in English.
13
14
  Japanese text is retained in manuscript excerpts, rule dictionaries, and fixtures.
14
15
 
15
16
  ## Status
16
17
 
17
- This is an initial implementation. It does not invoke an AI service, automatically
18
- rewrite manuscripts, or publish books. EPUB, image, and book scaffolding commands
19
- are planned extensions, not available features.
18
+ The CLI does not automatically rewrite manuscripts or publish books. EPUB, image,
19
+ and book scaffolding commands are planned extensions, not available features.
20
20
 
21
21
  Ruby 3.2 or later is required. Asciidoctor is installed as a gem dependency.
22
22
  Starting with version 0.1.1, the default review backend requires the external MeCab
23
23
  command and a UTF-8 IPADIC dictionary. Node.js and textlint are not required.
24
24
  Explicit `--tokenizer literal` mode provides limited phrase matching without MeCab.
25
25
 
26
- Version 0.1.1 includes morphological analysis. Version 0.1.0 uses literal matching.
26
+ Version 0.6.1 also provides `review score FILE` for a manuscript score.
27
+ It uses local candidate analysis by default; `--agent codex` or `--agent claude`
28
+ explicitly asks an installed CLI to evaluate readability and may connect to its
29
+ configured model provider. No scoring mode edits manuscripts.
30
+
31
+ Version 0.1.1 added morphological analysis. Version 0.1.0 used literal matching.
32
+
33
+ The CLI has `writing` and `review` command groups. Both default to `--lang ja`.
34
+ Language-specific criteria live under `data/writing/<language>/`; review rules
35
+ use `data/review-rules.<language>.yml`. Writing and review have separate lists
36
+ of supported languages, so a future writing language need not imply review
37
+ support. Other languages are reserved for future implementations and now
38
+ return an explicit error. A review session saves its language and the criteria
39
+ used to generate its prompt.
27
40
 
28
41
  ## Install from RubyGems
29
42
 
@@ -39,6 +52,13 @@ Ruby 3.2 or later is required. RubyGems installs the required Ruby dependencies;
39
52
  no repository clone or Node.js installation is needed. MeCab and IPADIC must be
40
53
  installed separately when using version 0.1.1 or later in the default mode.
41
54
 
55
+ Generate a writing prompt without an existing manuscript:
56
+
57
+ ```sh
58
+ asciidoc-pubkit writing criteria --lang ja
59
+ asciidoc-pubkit writing prompt --lang ja --output writing-prompt.md
60
+ ```
61
+
42
62
  Run the review workflow from your manuscript directory:
43
63
 
44
64
  ```sh
@@ -60,7 +80,7 @@ Add the gem to your project's `Gemfile` to manage its version with Bundler:
60
80
 
61
81
  ```ruby
62
82
  source 'https://rubygems.org'
63
- gem 'asciidoc-pubkit', '~> 0.1.2'
83
+ gem 'asciidoc-pubkit', '~> 0.6.1'
64
84
  ```
65
85
 
66
86
  Then install dependencies and run the CLI through Bundler:
@@ -82,17 +102,88 @@ git clone https://github.com/cybergarage/asciidoc-pubkit.git
82
102
  cd asciidoc-pubkit
83
103
  bundle install
84
104
  gem build asciidoc-pubkit.gemspec
85
- gem install ./asciidoc-pubkit-0.1.2.gem
105
+ gem install ./asciidoc-pubkit-0.6.1.gem
86
106
  asciidoc-pubkit --version
87
107
  ```
88
108
 
89
109
  The gem name and CLI name are `asciidoc-pubkit`; the Ruby require path is
90
110
  `asciidoc_pubkit`, and the namespace is `AsciidocPubkit`.
91
111
 
112
+ ## Writing commands
113
+
114
+ `writing criteria` prints the packaged common criteria. `writing prompt` adds
115
+ task instructions around those same criteria. Neither command needs an AsciiDoc
116
+ file, a review session, MeCab, or network access. Both accept `--lang ja` and
117
+ `--output FILE`; output defaults to stdout, and an existing output file is never
118
+ overwritten. The prompt asks the agent to read project instructions and evidence;
119
+ it does not supply source facts or authorize edits. Book-specific voice and
120
+ format still come from the book. The OSS-only prose style profile remains with
121
+ the book workflow, not the shared default.
122
+
123
+ ```sh
124
+ asciidoc-pubkit writing criteria --lang ja
125
+ asciidoc-pubkit writing prompt --lang ja --output writing-prompt.md
126
+ ```
127
+
128
+ For example, `--lang en` exits with an unsupported-language error. No English
129
+ review or writing criteria are shipped in 0.6.1.
130
+
92
131
  For a small trial, use `examples/book.adoc` as the scan input. Its Japanese
93
132
  paragraphs deliberately contain review candidates; its code block must remain
94
133
  unchanged.
95
134
 
135
+ ## Score an AsciiDoc manuscript
136
+
137
+ Pass any AsciiDoc entry file directly; no review session or gold annotations are
138
+ required. Output defaults to a short English report with a score out of 100.
139
+
140
+ ```sh
141
+ asciidoc-pubkit review score book.adoc
142
+ asciidoc-pubkit review score book.adoc --json --output score.json
143
+ asciidoc-pubkit review score book.adoc --agent codex
144
+ asciidoc-pubkit review score book.adoc --agent claude --json
145
+ ```
146
+
147
+ Without `--agent`, `candidate-density-v1` reports a mechanical review indicator:
148
+ `max(0, 100 - 5 * candidates * 1000 / characters)`, rounded to three decimals.
149
+ Characters are non-whitespace Unicode characters in source-mapped running prose
150
+ after inline masking. Every finding, including informational candidates, counts
151
+ once. Lower scores indicate more detected review cues per 1,000 characters, not
152
+ proven defects or AI authorship. Short texts are especially sensitive to a single
153
+ candidate. No unmasked prose yields `null` rather than a perfect score.
154
+ Successful scoring exits 0 regardless of the score; invalid input, configuration,
155
+ analyzer, evaluator, or output failures exit 2. Verification exit codes are
156
+ unchanged. The JSON report includes candidate counts, findings, coverage notices,
157
+ analysis identity, and the formula. Rule, allow-list, and tokenizer changes can
158
+ change the score even when the manuscript is unchanged.
159
+
160
+ With `--agent`, `llm-readability-v1` displays an evaluator score instead. Sentence
161
+ clarity and paragraph coherence are each rated 1–3 per paragraph and mapped to
162
+ 0, 50, and 100. The displayed score averages these two dimensions across scored
163
+ paragraphs with equal paragraph weights. JSON also retains the individual
164
+ ratings, English reasons, source locations, evaluator model and CLI version,
165
+ rubric fingerprint, and the mechanical candidate-density score. These ordinal
166
+ averages are descriptive summaries, not validated interval measurements. This
167
+ single-document evaluation has no reference manuscript or evidence checklist;
168
+ it does not measure fact retention, technical accuracy, or meaning preservation.
169
+ `meaning_verified` remains `false` in both modes.
170
+
171
+ Scoring uses the same configuration discovery, custom rules, glossary, allows,
172
+ style, language, base directory, include handling, and prose coverage as scanning.
173
+ It accepts `--only`, `--config`, `--rules`, `--base-dir`, `--tokenizer`, `--style`,
174
+ `--lang ja`, and repeated `--attribute` options. Excluded, unmapped, and protected
175
+ blocks are not prose-scored. Coverage notices remain important when interpreting
176
+ a score. MeCab/IPADIC is the default; literal mode must be selected explicitly.
177
+
178
+ AI evaluation requires an installed, authenticated Codex or Claude CLI, and may
179
+ send the selected paragraph text and headings to its configured provider and
180
+ consume account usage. It receives no candidate findings or source file paths.
181
+ `--model` selects the evaluator model, and `--timeout` sets the invocation timeout
182
+ (default 300 seconds); both require `--agent`. Temporary evaluation files are
183
+ removed after scoring. Use `--json --output FILE` to retain the result. Output
184
+ files are never overwritten, and existing output is rejected before invoking an
185
+ evaluator. A file with no unmasked prose is not sent to an evaluator.
186
+
96
187
  ## Run from a local checkout
97
188
 
98
189
  Use the `run` Make target to execute the checkout without installing the
@@ -163,7 +254,7 @@ asciidoc-pubkit review verify .pubkit/review --output verification.json
163
254
  ```
164
255
 
165
256
  All three commands leave manuscript files unchanged. Only the agent edits them.
166
- Output paths must not already exist. Use a new session directory for a new pass.
257
+ `prompt` and `verify` output files must not already exist.
167
258
  Without `--output`, `prompt` writes Markdown and `verify` writes JSON to stdout.
168
259
  `scan` defaults to `.pubkit/review` and prints a short summary.
169
260
 
@@ -175,6 +266,27 @@ asciidoc-pubkit review scan chapter.adoc --style desu-masu
175
266
  asciidoc-pubkit review scan book.adoc --attribute edition=print --base-dir .
176
267
  ```
177
268
 
269
+ If the session directory already exists, an interactive terminal asks
270
+ `Replace it? [y/N]`. Only `y` or `yes` (case-insensitive) replaces it; any other
271
+ answer or end of input cancels with exit status 2. Replacement resets the review
272
+ baseline and removes old session contents. Use a different `--output` path to
273
+ keep the previous review pass. The old session remains intact if scanning fails
274
+ before the replacement is installed.
275
+
276
+ For batch execution:
277
+
278
+ ```sh
279
+ # Answer yes automatically and replace an existing review session.
280
+ asciidoc-pubkit review scan book.adoc --yes # short form: -y
281
+ # Never ask; exit with status 2 if the output already exists.
282
+ asciidoc-pubkit review scan book.adoc --no-input
283
+ ```
284
+
285
+ Non-terminal stdin also disables prompting. `--yes --no-input` permits
286
+ replacement without reading stdin. These options apply to `scan` only;
287
+ regular files, symlinks, directories without a review session layout, and
288
+ sessions containing the current manuscript sources cannot be replaced.
289
+
178
290
  An entrypoint or a standalone chapter can be scanned. Includes and conditionals
179
291
  are processed by Asciidoctor. Match the publishing build's attributes and base
180
292
  directory to review the intended edition. `--only` selects one included source
@@ -213,13 +325,24 @@ asciidoc-pubkit review prompt .pubkit/review --mode diagnose
213
325
  for findings and proposed revisions without editing. Both modes require the agent
214
326
  to distinguish **revise**, **keep**, and **needs-evidence** decisions.
215
327
 
216
- The Markdown includes every selected paragraph, neighboring selected paragraphs,
217
- heading context, candidates, saved settings, and preservation instructions. It
218
- also asks for contextual review of paragraphs with no machine matches. Initial
328
+ The single Markdown output includes every selected paragraph once, in document
329
+ order, with the shared criteria saved at scan time, candidates, saved settings,
330
+ and preservation instructions. Paragraph
331
+ text uses fenced text blocks; metadata uses compact JSON. File and heading context
332
+ is shared by consecutive paragraphs, and candidates inherit their file and
333
+ paragraph ID. Adjacent entries provide neighboring context without repeating text.
334
+ The prompt asks the agent to read manageable ranges with neighboring paragraphs
335
+ at boundaries and track completed paragraph IDs and candidate decisions. It also
336
+ requires review of paragraphs with no machine matches. Large manuscripts can
337
+ still exceed a context window if the entire prompt is loaded at once. Initial
219
338
  support is for local prose correction, not chapter reorganization.
220
339
 
221
340
  Source hashes are checked before generating a prompt. Modified sources, modified
222
341
  session artifacts, and incompatible session versions require a fresh scan.
342
+ `review scan`, `review prompt`, and `review verify` accept `--lang ja`; prompt and
343
+ verify check the value against the session language. A non-Japanese language in
344
+ `--lang`, `review.language`, or an explicit AsciiDoc `:lang:` attribute is rejected.
345
+ An omitted `:lang:` attribute uses the selected review language.
223
346
 
224
347
  ### Verify
225
348
 
@@ -297,16 +420,17 @@ use an absolute path for an explicit executable override.
297
420
  | --- | --- | --- |
298
421
  | `abstract-reference` | hint | Ask what an abstract noun refers to |
299
422
  | `weak-predicate` | hint | Ask whether an operation's purpose or result is clear |
300
- | `contextual-phrase` | hint | Review referents, assumptions, and qualifications while preserving negation |
301
- | `generic-framing` | hint | Review generic introductions and emphasis |
423
+ | `contextual-phrase` | hint | Check whether a qualification changes interpretation or action; preserve necessary negation |
424
+ | `generic-framing` | hint | Check whether an introduction or recap adds a claim, scope, or consequence |
302
425
  | `vague-degree` | hint | Ask what depth, level, scope, or comparison is intended |
303
426
  | `repeated-ending` | info | Identify three consecutive sentences with the same detected ending |
304
427
  | `glossary-variant` | warning | Identify project-specific terminology variants |
305
428
  | `style-candidate` | hint | Check selected polite/plain endings against an explicit style |
306
429
 
307
430
  MeCab mode matches noun and adjective tokens and dictionary forms of verbs.
308
- Sahen predicates are matched as a noun followed by the verb for "do"; standalone
309
- sahen nouns are not treated as verbal predicates. Auxiliary sequences retain
431
+ Sahen predicates are matched as a noun followed by `する` or the potential
432
+ `できる`; standalone sahen nouns are not treated as verbal predicates. A configured
433
+ sahen predicate takes precedence over a matching standalone noun rule. Auxiliary sequences retain
310
434
  negation, past tense, passive forms, and progressive forms in the reported surface.
311
435
  The added reach predicate is negative-only; the existing handling predicate is
312
436
  reviewed in both affirmative and negative forms. Glossary variants and generic
@@ -315,6 +439,39 @@ literal matching and suppress overlapping morphological candidates. Compound
315
439
  nouns are matched across adjacent noun tokens. Selection and narrowing verbs
316
440
  include potential forms; predicate surfaces also preserve causative auxiliaries.
317
441
 
442
+ Default terms group components, actors, relationships, and resources under
443
+ `abstract-reference`; operations under `weak-predicate`; and compressed noun
444
+ relationships, referents, sufficiency, and negative qualifications under
445
+ `contextual-phrase`. `木` and `余白` invite a context-dependent katakana terminology
446
+ review, not mandatory replacement. `明示選択` and `欠落理由` invite checking whether
447
+ `明示的選択` and `欠落した理由` clarify the intended relationship. Shared phrases such
448
+ as `ではありません` cover longer qualifications without listing every sentence.
449
+ `generic-framing` also flags `このように` and `要するに` as possible recaps.
450
+ These are candidates for contextual review, not banned expressions. A necessary
451
+ condition, uncertainty, or distinction must survive a revision; a redundant
452
+ disclaimer can instead be removed or folded into a more precise main claim.
453
+
454
+ Metaphorical-operation candidates include limited exact surfaces of
455
+ `地味に効く`, `静かに壊れる`, `時間を溶かす`, and `側に倒す`, including selected polite,
456
+ past, and connective forms listed in the packaged YAML. They use contextual
457
+ phrase matching in both tokenizers; this is not complete inflection coverage or
458
+ syntactic analysis. Bare verbs such as `効く`, `壊れる`, `溶かす`, and `倒す` are
459
+ not added as general metaphor candidates. Identify the actual effect, policy,
460
+ work, or failure state from evidence rather than applying a fixed replacement.
461
+ Keep valid technical meanings and necessary negation.
462
+
463
+ The shared criteria distinguish reader instructions, actual system behavior,
464
+ and available capabilities. They also separate readability from checks for
465
+ missing information and unsupported additions. Sentence length, punctuation,
466
+ and list density are contextual cues, not fixed acceptance thresholds. These
467
+ updates are included in 0.6.1; start a new session to use updated criteria and rules.
468
+
469
+ Literal term matching suppresses matches strictly contained in a longer matched
470
+ term, across categories. The longer term also suppresses contained matches when
471
+ it is allowed; separate occurrences remain candidates. Glossary and style checks
472
+ are independent. In MeCab mode, predicates outside a contextual phrase can still
473
+ be reported separately.
474
+
318
475
  Morphological candidates include `lemma`, `part_of_speech`, `negative`, and
319
476
  `detector` alongside the original `match`, line, and column. Negation detection
320
477
  covers common IPADIC negative auxiliaries; it is not full semantic analysis of
@@ -327,12 +484,12 @@ reports analyzer changes instead of treating results from different dictionaries
327
484
  as directly comparable. Prompt generation uses saved evidence and does not need
328
485
  MeCab. Changed rules or dictionary settings require a new scan.
329
486
 
330
- Sessions from earlier tool versions are not compatible with 0.1.2. Keep the original baseline for
487
+ Sessions from earlier tool versions are not compatible with 0.6.1. Keep the original baseline for
331
488
  an ongoing review and finish it with the original version, or start a new review
332
489
  pass in a different directory:
333
490
 
334
491
  ```sh
335
- asciidoc-pubkit review scan book.adoc --output .pubkit/review-0.1.2
492
+ asciidoc-pubkit review scan book.adoc --output .pubkit/review-0.6.1
336
493
  ```
337
494
 
338
495
  Severity describes review priority, not proof of an error. There is no AI-authorship
@@ -365,11 +522,121 @@ analysis of inflections, negative predicates, Unicode positions, and long lines.
365
522
  Install MeCab and UTF-8 IPADIC before running the complete test suite.
366
523
  GitHub Actions is configured for Ruby 3.2, 3.3, 3.4, and 4.0 on Linux.
367
524
 
368
- ## License
525
+ [`test/fixtures/prose_evaluation.ja.json`](test/fixtures/prose_evaluation.ja.json)
526
+ contains original Japanese technical prose for candidate regression and manual
527
+ revision evaluation. Automated tests check metaphor candidates in both
528
+ tokenizers, source positions, inline exclusions, and allow lists. The revision
529
+ pairs are manual evaluation cases, not an automated semantic checker or a
530
+ measured AI benchmark. Compare source and revision separately for readability,
531
+ information loss, and unsupported additions, using each case's review axes and
532
+ expected disposition. Cover conditions, negation, numbers, versions, actors,
533
+ causes, implementation requirements, and necessary repetition. Record the
534
+ model, prompt, source evidence, and human judgments when evaluating actual
535
+ generated revisions; fewer findings alone do not establish an improvement.
536
+
537
+ ### Development prose benchmark
538
+
539
+ The checkout provides `script/evaluate-prose` for measuring future detector and
540
+ prompt changes. This is a development tool, not an installed gem command; the
541
+ benchmark CLI calls are separate from the optional `review score --agent` path.
542
+ The fixed
543
+ [`prose_benchmark.ja.adoc`](test/fixtures/prose_benchmark.ja.adoc) contains 13
544
+ original editorial paragraphs with intentionally vague framing and metaphors,
545
+ inflection variants, and valid technical prose controls. Its
546
+ [`annotations`](test/fixtures/prose_benchmark.ja.json) contain 11 review targets
547
+ and separate source fact checklists. It is a small synthetic regression corpus,
548
+ not an AI authorship dataset or a representative measure of prose quality.
549
+
550
+ Run the deterministic benchmark before and after a change:
369
551
 
370
- Copyright 2026 CyberGarage.
552
+ ```sh
553
+ bundle exec ruby script/evaluate-prose --output .pubkit/evaluation/before
554
+ # After changing the detector or prompt:
555
+ bundle exec ruby script/evaluate-prose --output .pubkit/evaluation/after \
556
+ --compare .pubkit/evaluation/before/report.json
557
+ ```
371
558
 
372
- Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE).
559
+ The report measures candidate precision, recall, F1, and location accuracy on a
560
+ 0–100 scale for `generic-framing`, metaphorical operations, and `repeated-ending`.
561
+ Other rule findings are counted separately and are not judged against these
562
+ annotations. Detection matches a target's paragraph and kind; location accuracy
563
+ then checks the one-based source position, represented internally as a zero-based
564
+ Unicode character offset within that paragraph. The annotated repeated-ending
565
+ location is the matching ending of the third sentence in the consecutive run.
566
+ A candidate can be correctly detected even when the reviewer should keep it.
567
+ Zero denominators are reported as `null`, not perfect scores. Duplicate detections
568
+ count as false positives. These scores measure the detector, not the manuscript.
569
+
570
+ The checked-in [`MeCab baseline`](benchmark/baseline-mecab.json), captured before
571
+ the planned detector changes, has precision 100, recall 81.818, F1 90, and location
572
+ accuracy 77.778. It misses two inflected metaphor phrases and mislocates two
573
+ repeated-ending candidates. Reports record source, criteria, rules, corpus, and
574
+ dictionary fingerprints. Comparisons reject changed corpora, scoring versions,
575
+ analyzers, or dictionaries. Reports with configuration fingerprints also reject
576
+ changed benchmark configuration. A fixed empty configuration
577
+ keeps these trials independent of auto-discovered manuscript settings.
578
+ Record your own baseline when dictionary fingerprints
579
+ differ. `--tokenizer literal` explicitly selects a separate limited-mode run;
580
+ there is no automatic fallback, and literal results do not validate MeCab.
581
+
582
+ ### Optional local AI evaluation
583
+
584
+ Use an installed, authenticated Codex or Claude CLI to perform rewrite and judge
585
+ trials explicitly. A local CLI may call its configured remote model provider and
586
+ consume account usage; this is not an offline model test. Normal `rake test`
587
+ does not run either CLI. For example:
588
+
589
+ ```sh
590
+ bundle exec ruby script/evaluate-prose --agent codex --repeats 3 \
591
+ --output .pubkit/evaluation/ai-before
592
+ bundle exec ruby script/evaluate-prose --agent codex --repeats 3 \
593
+ --output .pubkit/evaluation/ai-after \
594
+ --compare .pubkit/evaluation/ai-before/report.json
595
+ # Or use Claude:
596
+ bundle exec ruby script/evaluate-prose --agent claude \
597
+ --output .pubkit/evaluation/claude-before
598
+ ```
599
+
600
+ Each trial generates the actual saved review prompt, requests paragraph revisions
601
+ as JSON, and runs the existing mechanical verifier. A fresh judge invocation
602
+ evaluates both original and revised prose with neutral A/B labels, alternating
603
+ their order across trials. The judge sees source evidence but no findings, tool
604
+ revision labels, or expected calibration answers. The writer receives the source
605
+ fact checklist in addition to the review prompt, so this measures an evidence-
606
+ assisted workflow rather than every real-world manuscript review.
607
+
608
+ Sentence clarity and paragraph coherence are separately rated 1–3 and displayed
609
+ as 0, 50, or 100 before averaging. These ordinal averages are descriptive
610
+ summaries, not validated interval measurements. Fact retention measures the
611
+ percentage of annotated source facts preserved; addition-free and substitution-
612
+ free paragraph percentages remain separate. Known acceptable, incorrect, and
613
+ unsupported revisions from `prose_evaluation.ja.json` calibrate the judge using
614
+ accuracy and balanced accuracy across the three dispositions. Inspect poor
615
+ calibration before interpreting quality scores. No combined AI-likeness score or
616
+ semantic pass is produced: `meaning_verified` remains `false`.
617
+ Mechanical verification results and their pass rate are also reported separately.
618
+ Quality scores from a trial with preservation violations remain visible for
619
+ diagnosis; a completed evaluation does not authorize accepting that revision.
620
+
621
+ One trial is a smoke test; use at least three to inspect variation. Reports retain
622
+ individual scores, means, population standard deviations, and ranges. They do
623
+ not perform a significance test. Same-model judging can favor the writer's style;
624
+ `--judge-agent` and `--judge-model` select a separate evaluator. `--model` selects
625
+ the writer model; omitted model options use each CLI's default. AI comparisons
626
+ require the same reported writer and judge models, CLI versions, rubric, and
627
+ calibration corpus. CLI defaults can change, so retain the recorded model IDs
628
+ and use explicit model options for controlled comparisons. Score changes can
629
+ reflect sampling variation as well as implementation changes.
630
+
631
+ The runner uses argument arrays rather than shell interpolation, read-only Codex
632
+ execution or tool-disabled Claude execution, and a per-invocation timeout
633
+ (`--timeout`, default 300 seconds). The CLI may still load its user configuration;
634
+ use consistent settings for comparisons. Current adapters require the flags
635
+ shown by `codex exec --help` and `claude --help`; older CLI versions may need an
636
+ update. Missing CLIs, authentication errors, timeouts, and malformed responses
637
+ fail explicitly. Existing output directories are never replaced. Partial
638
+ artifacts remain available after failures. Results, prompts, responses, and review
639
+ sessions belong under ignored `.pubkit/`; do not commit private evaluation runs.
373
640
 
374
641
  ## Customize review rules
375
642
 
@@ -411,7 +678,7 @@ rejected. YAML aliases and object tags are not supported.
411
678
  | `schema_version` | Integer `1` |
412
679
  | `terms` | Mapping containing all five categories listed below |
413
680
  | `verbs` | Mapping from canonical verb forms to nonempty arrays of MeCab/IPADIC dictionary forms; each dictionary form belongs to only one canonical form |
414
- | `sahen` | Array of nouns matched with a following `する` verb |
681
+ | `sahen` | Array of nouns matched with a following `する` or potential `できる` verb |
415
682
  | `negative_only` | Array of canonical predicates restricted to negative forms; use the noun plus `する` for sahen predicates |
416
683
  | `compound_nouns` | Array of terms matched across contiguous noun tokens; each must also appear in `abstract-reference.terms` |
417
684
 
@@ -436,3 +703,47 @@ Prompt generation and verification use the saved contents, even if the original
436
703
  YAML file is subsequently edited or removed. Start a new session to apply rule
437
704
  changes. Older sessions without a rule snapshot fall back to the currently
438
705
  installed default file; start a new session for reproducible custom-rule reviews.
706
+
707
+ ## References
708
+
709
+ The shared criteria and default questions draw on the following editorial
710
+ sources. Their stylistic choices differ, including whether headings should state
711
+ conclusions. Candidate detection therefore leaves contextual judgment, document
712
+ structure, and final edits to the reviewer.
713
+
714
+ - [natural-japanese](https://github.com/coji/natural-japanese): separates mechanical
715
+ detection from contextual judgment and identifies repetitive framing.
716
+ - [yomiyasu](https://github.com/nanaism/yomiyasu):
717
+ informs evidence-based review of
718
+ metaphorical operations, distinctions between instructions and capabilities,
719
+ and separate assessment of meaning preservation and readability. The criteria
720
+ and regression examples here are independently written; the upstream skill,
721
+ Python linter, and benchmark passages are not bundled. Fixed style thresholds
722
+ and an AI-likeness score are not adopted.
723
+ - [日本語技術文書の文章規範](https://gist.github.com/k16shikano/fd287c3133457c4fd8f5601d34aa817d):
724
+ informs paragraph logic, evidence scope, and meaningful uncertainty.
725
+ - [AI臭い文章とは何なのか](https://speakerdeck.com/nasuvitz/ai-kusai-bunshou-toha-nanina-no-ka):
726
+ provides examples of unnecessary contrast, abstract referents, and paired short
727
+ sentences.
728
+
729
+ The development evaluation adapts ideas from the following primary research;
730
+ its engineering audience, corpus, and rubric do not reproduce those benchmarks.
731
+ Research texts and datasets are not redistributed here.
732
+
733
+ - [Evaluation of Document-Level Text Simplification in Japanese](https://aclanthology.org/2026.lrec-1.85/)
734
+ (Yamashita et al., LREC 2026): separates information preservation from sentence
735
+ and document simplicity and evaluates LLM judgments against human annotations.
736
+ Its elementary-school audience and Wikipedia corpus differ from this project.
737
+ - [Evaluating Factuality in Text Simplification](https://aclanthology.org/2022.acl-long.506/)
738
+ (Devaraj et al., ACL 2022): distinguishes insertion, deletion, and substitution
739
+ errors, motivating separate checks for unsupported additions and lost facts.
740
+ - [Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation](https://aclanthology.org/2024.readi-1.1/)
741
+ (Cripwell et al., READI 2024): motivates separate readability and preservation
742
+ scores rather than a single aggregate. SARI-style overlap metrics are not
743
+ adopted because lexical agreement alone is insufficient for this prose review.
744
+
745
+ ## License
746
+
747
+ Copyright 2026 CyberGarage.
748
+
749
+ Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE).