asciidoc-pubkit 0.1.2 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +47 -0
- data/README.md +334 -23
- data/data/review-rules.ja.yml +190 -26
- data/data/writing/ja/criteria.md +223 -0
- data/lib/asciidoc_pubkit/cli.rb +62 -11
- data/lib/asciidoc_pubkit/document.rb +5 -0
- data/lib/asciidoc_pubkit/language.rb +16 -0
- data/lib/asciidoc_pubkit/local_evaluator.rb +72 -0
- data/lib/asciidoc_pubkit/rule_set.rb +5 -0
- data/lib/asciidoc_pubkit/rules.rb +26 -13
- data/lib/asciidoc_pubkit/score.rb +120 -0
- data/lib/asciidoc_pubkit/session.rb +85 -17
- data/lib/asciidoc_pubkit/settings.rb +3 -3
- data/lib/asciidoc_pubkit/writing.rb +33 -0
- data/lib/asciidoc_pubkit.rb +5 -1
- metadata +10 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c946ae67aae9ea8e6e21f5464b61daec42a18488c5aca5fbe434921f13dad824
|
|
4
|
+
data.tar.gz: 47ca9c1414c78699dbb07ebe3bfdb07460cc132a8a10c642392d147eb2247b90
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f8e1382976343a206f04363bdfb0896e3633605f239c2fbfb46afacaaf11415a93757a418c09d9fc449700742b15268f69883a36e0c7e200f9f3e73b4f4a0f22
|
|
7
|
+
data.tar.gz: 3f0395078d442df25b730cde44186cb1868c3b68f7ea292438a2ad8871179a30198524fcd846c5ea6f2ce84d65d90f01b9d4db829f012b6e47e68f4b2051d3da
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,52 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
## 0.6.1 — 2026-10-01
|
|
6
|
+
|
|
7
|
+
- Add `review score FILE` for direct AsciiDoc manuscript scoring without a
|
|
8
|
+
session. Report a candidate-density indicator by default and optionally invoke
|
|
9
|
+
a local Codex or Claude CLI for readability ratings with `--agent`. Preserve
|
|
10
|
+
manuscript files, coverage limits, explicit tokenizer selection, and
|
|
11
|
+
`meaning_verified: false`.
|
|
12
|
+
- Add a development-only prose benchmark with a fixed AsciiDoc corpus, scored
|
|
13
|
+
candidate coverage and locations, and a saved MeCab baseline before detector
|
|
14
|
+
changes. Keep quality judgments separate from detection performance.
|
|
15
|
+
- Add explicit local Codex/Claude rewrite and judge trials with fact checklists,
|
|
16
|
+
blinded document labels, evaluator calibration, repeated-run statistics, and
|
|
17
|
+
compatibility checks for baseline comparisons. Keep benchmark AI trials
|
|
18
|
+
separate from manuscript scoring and the default test suite.
|
|
19
|
+
|
|
20
|
+
- Extend the shared Japanese prose criteria with evidence-based metaphor review,
|
|
21
|
+
distinctions between instructions, behavior, and capabilities, and separate
|
|
22
|
+
checks for information loss and unsupported additions.
|
|
23
|
+
- Add limited metaphorical-operation phrases as contextual review candidates,
|
|
24
|
+
with a Japanese evaluation corpus for detection and manual meaning review.
|
|
25
|
+
- Introduce yomiyasu in the README references.
|
|
26
|
+
|
|
27
|
+
## 0.6.0 — 2026-09-30
|
|
28
|
+
|
|
29
|
+
- Add `writing criteria` and `writing prompt` commands backed by one packaged
|
|
30
|
+
Japanese technical prose guide. Include that guide in saved review prompts.
|
|
31
|
+
- Organize the CLI into `writing` and `review` groups and accept `--lang ja` on
|
|
32
|
+
both. Reject unsupported languages explicitly, including document attributes.
|
|
33
|
+
- Save the shared criteria and language with each review session, and update the
|
|
34
|
+
book writing and review skills to use the generated prompts.
|
|
35
|
+
- Refine Japanese review questions for redundant qualifications and generic
|
|
36
|
+
recaps, and document the editorial sources and contextual limits.
|
|
37
|
+
|
|
38
|
+
## 0.1.3 — 2026-09-29
|
|
39
|
+
|
|
40
|
+
- Consolidate Japanese review candidates for terminology, compressed noun
|
|
41
|
+
relationships, operations, and negative qualifications in the packaged rules.
|
|
42
|
+
- Suppress literal candidates contained in longer matched terms, preserving
|
|
43
|
+
allow-list handling, and recognize sahen potential forms before noun rules.
|
|
44
|
+
- Reduce single-file review prompts by emitting each paragraph once, sharing
|
|
45
|
+
context metadata, and compacting JSON; retain all candidates and review settings.
|
|
46
|
+
- Confirm replacement of existing review sessions in interactive scans.
|
|
47
|
+
- Add `review scan --yes` (`-y`) and `--no-input` for batch execution.
|
|
48
|
+
- Preserve previous sessions until replacement scan artifacts are ready.
|
|
49
|
+
|
|
3
50
|
## 0.1.2 — Released
|
|
4
51
|
|
|
5
52
|
- Package default Japanese review rules as a validated UTF-8 YAML file.
|
data/README.md
CHANGED
|
@@ -5,25 +5,38 @@
|
|
|
5
5
|
|
|
6
6
|
# asciidoc-pubkit
|
|
7
7
|
|
|
8
|
-
A toolkit for authoring
|
|
8
|
+
A toolkit for authoring and reviewing AsciiDoc books.
|
|
9
9
|
|
|
10
|
-
|
|
11
|
-
|
|
10
|
+
Version 0.6.1 provides shared Japanese technical writing criteria, an authoring
|
|
11
|
+
prompt, contextual review prompts, manuscript scoring, and baseline verification
|
|
12
|
+
for edited manuscripts.
|
|
12
13
|
The CLI, diagnostics, documentation, and generated instructions are in English.
|
|
13
14
|
Japanese text is retained in manuscript excerpts, rule dictionaries, and fixtures.
|
|
14
15
|
|
|
15
16
|
## Status
|
|
16
17
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
are planned extensions, not available features.
|
|
18
|
+
The CLI does not automatically rewrite manuscripts or publish books. EPUB, image,
|
|
19
|
+
and book scaffolding commands are planned extensions, not available features.
|
|
20
20
|
|
|
21
21
|
Ruby 3.2 or later is required. Asciidoctor is installed as a gem dependency.
|
|
22
22
|
Starting with version 0.1.1, the default review backend requires the external MeCab
|
|
23
23
|
command and a UTF-8 IPADIC dictionary. Node.js and textlint are not required.
|
|
24
24
|
Explicit `--tokenizer literal` mode provides limited phrase matching without MeCab.
|
|
25
25
|
|
|
26
|
-
Version 0.
|
|
26
|
+
Version 0.6.1 also provides `review score FILE` for a manuscript score.
|
|
27
|
+
It uses local candidate analysis by default; `--agent codex` or `--agent claude`
|
|
28
|
+
explicitly asks an installed CLI to evaluate readability and may connect to its
|
|
29
|
+
configured model provider. No scoring mode edits manuscripts.
|
|
30
|
+
|
|
31
|
+
Version 0.1.1 added morphological analysis. Version 0.1.0 used literal matching.
|
|
32
|
+
|
|
33
|
+
The CLI has `writing` and `review` command groups. Both default to `--lang ja`.
|
|
34
|
+
Language-specific criteria live under `data/writing/<language>/`; review rules
|
|
35
|
+
use `data/review-rules.<language>.yml`. Writing and review have separate lists
|
|
36
|
+
of supported languages, so a future writing language need not imply review
|
|
37
|
+
support. Other languages are reserved for future implementations and now
|
|
38
|
+
return an explicit error. A review session saves its language and the criteria
|
|
39
|
+
used to generate its prompt.
|
|
27
40
|
|
|
28
41
|
## Install from RubyGems
|
|
29
42
|
|
|
@@ -39,6 +52,13 @@ Ruby 3.2 or later is required. RubyGems installs the required Ruby dependencies;
|
|
|
39
52
|
no repository clone or Node.js installation is needed. MeCab and IPADIC must be
|
|
40
53
|
installed separately when using version 0.1.1 or later in the default mode.
|
|
41
54
|
|
|
55
|
+
Generate a writing prompt without an existing manuscript:
|
|
56
|
+
|
|
57
|
+
```sh
|
|
58
|
+
asciidoc-pubkit writing criteria --lang ja
|
|
59
|
+
asciidoc-pubkit writing prompt --lang ja --output writing-prompt.md
|
|
60
|
+
```
|
|
61
|
+
|
|
42
62
|
Run the review workflow from your manuscript directory:
|
|
43
63
|
|
|
44
64
|
```sh
|
|
@@ -60,7 +80,7 @@ Add the gem to your project's `Gemfile` to manage its version with Bundler:
|
|
|
60
80
|
|
|
61
81
|
```ruby
|
|
62
82
|
source 'https://rubygems.org'
|
|
63
|
-
gem 'asciidoc-pubkit', '~> 0.1
|
|
83
|
+
gem 'asciidoc-pubkit', '~> 0.6.1'
|
|
64
84
|
```
|
|
65
85
|
|
|
66
86
|
Then install dependencies and run the CLI through Bundler:
|
|
@@ -82,17 +102,88 @@ git clone https://github.com/cybergarage/asciidoc-pubkit.git
|
|
|
82
102
|
cd asciidoc-pubkit
|
|
83
103
|
bundle install
|
|
84
104
|
gem build asciidoc-pubkit.gemspec
|
|
85
|
-
gem install ./asciidoc-pubkit-0.1.
|
|
105
|
+
gem install ./asciidoc-pubkit-0.6.1.gem
|
|
86
106
|
asciidoc-pubkit --version
|
|
87
107
|
```
|
|
88
108
|
|
|
89
109
|
The gem name and CLI name are `asciidoc-pubkit`; the Ruby require path is
|
|
90
110
|
`asciidoc_pubkit`, and the namespace is `AsciidocPubkit`.
|
|
91
111
|
|
|
112
|
+
## Writing commands
|
|
113
|
+
|
|
114
|
+
`writing criteria` prints the packaged common criteria. `writing prompt` adds
|
|
115
|
+
task instructions around those same criteria. Neither command needs an AsciiDoc
|
|
116
|
+
file, a review session, MeCab, or network access. Both accept `--lang ja` and
|
|
117
|
+
`--output FILE`; output defaults to stdout, and an existing output file is never
|
|
118
|
+
overwritten. The prompt asks the agent to read project instructions and evidence;
|
|
119
|
+
it does not supply source facts or authorize edits. Book-specific voice and
|
|
120
|
+
format still come from the book. The OSS-only prose style profile remains with
|
|
121
|
+
the book workflow, not the shared default.
|
|
122
|
+
|
|
123
|
+
```sh
|
|
124
|
+
asciidoc-pubkit writing criteria --lang ja
|
|
125
|
+
asciidoc-pubkit writing prompt --lang ja --output writing-prompt.md
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
For example, `--lang en` exits with an unsupported-language error. No English
|
|
129
|
+
review or writing criteria are shipped in 0.6.1.
|
|
130
|
+
|
|
92
131
|
For a small trial, use `examples/book.adoc` as the scan input. Its Japanese
|
|
93
132
|
paragraphs deliberately contain review candidates; its code block must remain
|
|
94
133
|
unchanged.
|
|
95
134
|
|
|
135
|
+
## Score an AsciiDoc manuscript
|
|
136
|
+
|
|
137
|
+
Pass any AsciiDoc entry file directly; no review session or gold annotations are
|
|
138
|
+
required. Output defaults to a short English report with a score out of 100.
|
|
139
|
+
|
|
140
|
+
```sh
|
|
141
|
+
asciidoc-pubkit review score book.adoc
|
|
142
|
+
asciidoc-pubkit review score book.adoc --json --output score.json
|
|
143
|
+
asciidoc-pubkit review score book.adoc --agent codex
|
|
144
|
+
asciidoc-pubkit review score book.adoc --agent claude --json
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Without `--agent`, `candidate-density-v1` reports a mechanical review indicator:
|
|
148
|
+
`max(0, 100 - 5 * candidates * 1000 / characters)`, rounded to three decimals.
|
|
149
|
+
Characters are non-whitespace Unicode characters in source-mapped running prose
|
|
150
|
+
after inline masking. Every finding, including informational candidates, counts
|
|
151
|
+
once. Lower scores indicate more detected review cues per 1,000 characters, not
|
|
152
|
+
proven defects or AI authorship. Short texts are especially sensitive to a single
|
|
153
|
+
candidate. No unmasked prose yields `null` rather than a perfect score.
|
|
154
|
+
Successful scoring exits 0 regardless of the score; invalid input, configuration,
|
|
155
|
+
analyzer, evaluator, or output failures exit 2. Verification exit codes are
|
|
156
|
+
unchanged. The JSON report includes candidate counts, findings, coverage notices,
|
|
157
|
+
analysis identity, and the formula. Rule, allow-list, and tokenizer changes can
|
|
158
|
+
change the score even when the manuscript is unchanged.
|
|
159
|
+
|
|
160
|
+
With `--agent`, `llm-readability-v1` displays an evaluator score instead. Sentence
|
|
161
|
+
clarity and paragraph coherence are each rated 1–3 per paragraph and mapped to
|
|
162
|
+
0, 50, and 100. The displayed score averages these two dimensions across scored
|
|
163
|
+
paragraphs with equal paragraph weights. JSON also retains the individual
|
|
164
|
+
ratings, English reasons, source locations, evaluator model and CLI version,
|
|
165
|
+
rubric fingerprint, and the mechanical candidate-density score. These ordinal
|
|
166
|
+
averages are descriptive summaries, not validated interval measurements. This
|
|
167
|
+
single-document evaluation has no reference manuscript or evidence checklist;
|
|
168
|
+
it does not measure fact retention, technical accuracy, or meaning preservation.
|
|
169
|
+
`meaning_verified` remains `false` in both modes.
|
|
170
|
+
|
|
171
|
+
Scoring uses the same configuration discovery, custom rules, glossary, allows,
|
|
172
|
+
style, language, base directory, include handling, and prose coverage as scanning.
|
|
173
|
+
It accepts `--only`, `--config`, `--rules`, `--base-dir`, `--tokenizer`, `--style`,
|
|
174
|
+
`--lang ja`, and repeated `--attribute` options. Excluded, unmapped, and protected
|
|
175
|
+
blocks are not prose-scored. Coverage notices remain important when interpreting
|
|
176
|
+
a score. MeCab/IPADIC is the default; literal mode must be selected explicitly.
|
|
177
|
+
|
|
178
|
+
AI evaluation requires an installed, authenticated Codex or Claude CLI, and may
|
|
179
|
+
send the selected paragraph text and headings to its configured provider and
|
|
180
|
+
consume account usage. It receives no candidate findings or source file paths.
|
|
181
|
+
`--model` selects the evaluator model, and `--timeout` sets the invocation timeout
|
|
182
|
+
(default 300 seconds); both require `--agent`. Temporary evaluation files are
|
|
183
|
+
removed after scoring. Use `--json --output FILE` to retain the result. Output
|
|
184
|
+
files are never overwritten, and existing output is rejected before invoking an
|
|
185
|
+
evaluator. A file with no unmasked prose is not sent to an evaluator.
|
|
186
|
+
|
|
96
187
|
## Run from a local checkout
|
|
97
188
|
|
|
98
189
|
Use the `run` Make target to execute the checkout without installing the
|
|
@@ -163,7 +254,7 @@ asciidoc-pubkit review verify .pubkit/review --output verification.json
|
|
|
163
254
|
```
|
|
164
255
|
|
|
165
256
|
All three commands leave manuscript files unchanged. Only the agent edits them.
|
|
166
|
-
|
|
257
|
+
`prompt` and `verify` output files must not already exist.
|
|
167
258
|
Without `--output`, `prompt` writes Markdown and `verify` writes JSON to stdout.
|
|
168
259
|
`scan` defaults to `.pubkit/review` and prints a short summary.
|
|
169
260
|
|
|
@@ -175,6 +266,27 @@ asciidoc-pubkit review scan chapter.adoc --style desu-masu
|
|
|
175
266
|
asciidoc-pubkit review scan book.adoc --attribute edition=print --base-dir .
|
|
176
267
|
```
|
|
177
268
|
|
|
269
|
+
If the session directory already exists, an interactive terminal asks
|
|
270
|
+
`Replace it? [y/N]`. Only `y` or `yes` (case-insensitive) replaces it; any other
|
|
271
|
+
answer or end of input cancels with exit status 2. Replacement resets the review
|
|
272
|
+
baseline and removes old session contents. Use a different `--output` path to
|
|
273
|
+
keep the previous review pass. The old session remains intact if scanning fails
|
|
274
|
+
before the replacement is installed.
|
|
275
|
+
|
|
276
|
+
For batch execution:
|
|
277
|
+
|
|
278
|
+
```sh
|
|
279
|
+
# Answer yes automatically and replace an existing review session.
|
|
280
|
+
asciidoc-pubkit review scan book.adoc --yes # short form: -y
|
|
281
|
+
# Never ask; exit with status 2 if the output already exists.
|
|
282
|
+
asciidoc-pubkit review scan book.adoc --no-input
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Non-terminal stdin also disables prompting. `--yes --no-input` permits
|
|
286
|
+
replacement without reading stdin. These options apply to `scan` only;
|
|
287
|
+
regular files, symlinks, directories without a review session layout, and
|
|
288
|
+
sessions containing the current manuscript sources cannot be replaced.
|
|
289
|
+
|
|
178
290
|
An entrypoint or a standalone chapter can be scanned. Includes and conditionals
|
|
179
291
|
are processed by Asciidoctor. Match the publishing build's attributes and base
|
|
180
292
|
directory to review the intended edition. `--only` selects one included source
|
|
@@ -213,13 +325,24 @@ asciidoc-pubkit review prompt .pubkit/review --mode diagnose
|
|
|
213
325
|
for findings and proposed revisions without editing. Both modes require the agent
|
|
214
326
|
to distinguish **revise**, **keep**, and **needs-evidence** decisions.
|
|
215
327
|
|
|
216
|
-
The Markdown includes every selected paragraph,
|
|
217
|
-
|
|
218
|
-
|
|
328
|
+
The single Markdown output includes every selected paragraph once, in document
|
|
329
|
+
order, with the shared criteria saved at scan time, candidates, saved settings,
|
|
330
|
+
and preservation instructions. Paragraph
|
|
331
|
+
text uses fenced text blocks; metadata uses compact JSON. File and heading context
|
|
332
|
+
is shared by consecutive paragraphs, and candidates inherit their file and
|
|
333
|
+
paragraph ID. Adjacent entries provide neighboring context without repeating text.
|
|
334
|
+
The prompt asks the agent to read manageable ranges with neighboring paragraphs
|
|
335
|
+
at boundaries and track completed paragraph IDs and candidate decisions. It also
|
|
336
|
+
requires review of paragraphs with no machine matches. Large manuscripts can
|
|
337
|
+
still exceed a context window if the entire prompt is loaded at once. Initial
|
|
219
338
|
support is for local prose correction, not chapter reorganization.
|
|
220
339
|
|
|
221
340
|
Source hashes are checked before generating a prompt. Modified sources, modified
|
|
222
341
|
session artifacts, and incompatible session versions require a fresh scan.
|
|
342
|
+
`review scan`, `review prompt`, and `review verify` accept `--lang ja`; prompt and
|
|
343
|
+
verify check the value against the session language. A non-Japanese language in
|
|
344
|
+
`--lang`, `review.language`, or an explicit AsciiDoc `:lang:` attribute is rejected.
|
|
345
|
+
An omitted `:lang:` attribute uses the selected review language.
|
|
223
346
|
|
|
224
347
|
### Verify
|
|
225
348
|
|
|
@@ -297,16 +420,17 @@ use an absolute path for an explicit executable override.
|
|
|
297
420
|
| --- | --- | --- |
|
|
298
421
|
| `abstract-reference` | hint | Ask what an abstract noun refers to |
|
|
299
422
|
| `weak-predicate` | hint | Ask whether an operation's purpose or result is clear |
|
|
300
|
-
| `contextual-phrase` | hint |
|
|
301
|
-
| `generic-framing` | hint |
|
|
423
|
+
| `contextual-phrase` | hint | Check whether a qualification changes interpretation or action; preserve necessary negation |
|
|
424
|
+
| `generic-framing` | hint | Check whether an introduction or recap adds a claim, scope, or consequence |
|
|
302
425
|
| `vague-degree` | hint | Ask what depth, level, scope, or comparison is intended |
|
|
303
426
|
| `repeated-ending` | info | Identify three consecutive sentences with the same detected ending |
|
|
304
427
|
| `glossary-variant` | warning | Identify project-specific terminology variants |
|
|
305
428
|
| `style-candidate` | hint | Check selected polite/plain endings against an explicit style |
|
|
306
429
|
|
|
307
430
|
MeCab mode matches noun and adjective tokens and dictionary forms of verbs.
|
|
308
|
-
Sahen predicates are matched as a noun followed by
|
|
309
|
-
sahen nouns are not treated as verbal predicates.
|
|
431
|
+
Sahen predicates are matched as a noun followed by `する` or the potential
|
|
432
|
+
`できる`; standalone sahen nouns are not treated as verbal predicates. A configured
|
|
433
|
+
sahen predicate takes precedence over a matching standalone noun rule. Auxiliary sequences retain
|
|
310
434
|
negation, past tense, passive forms, and progressive forms in the reported surface.
|
|
311
435
|
The added reach predicate is negative-only; the existing handling predicate is
|
|
312
436
|
reviewed in both affirmative and negative forms. Glossary variants and generic
|
|
@@ -315,6 +439,39 @@ literal matching and suppress overlapping morphological candidates. Compound
|
|
|
315
439
|
nouns are matched across adjacent noun tokens. Selection and narrowing verbs
|
|
316
440
|
include potential forms; predicate surfaces also preserve causative auxiliaries.
|
|
317
441
|
|
|
442
|
+
Default terms group components, actors, relationships, and resources under
|
|
443
|
+
`abstract-reference`; operations under `weak-predicate`; and compressed noun
|
|
444
|
+
relationships, referents, sufficiency, and negative qualifications under
|
|
445
|
+
`contextual-phrase`. `木` and `余白` invite a context-dependent katakana terminology
|
|
446
|
+
review, not mandatory replacement. `明示選択` and `欠落理由` invite checking whether
|
|
447
|
+
`明示的選択` and `欠落した理由` clarify the intended relationship. Shared phrases such
|
|
448
|
+
as `ではありません` cover longer qualifications without listing every sentence.
|
|
449
|
+
`generic-framing` also flags `このように` and `要するに` as possible recaps.
|
|
450
|
+
These are candidates for contextual review, not banned expressions. A necessary
|
|
451
|
+
condition, uncertainty, or distinction must survive a revision; a redundant
|
|
452
|
+
disclaimer can instead be removed or folded into a more precise main claim.
|
|
453
|
+
|
|
454
|
+
Metaphorical-operation candidates include limited exact surfaces of
|
|
455
|
+
`地味に効く`, `静かに壊れる`, `時間を溶かす`, and `側に倒す`, including selected polite,
|
|
456
|
+
past, and connective forms listed in the packaged YAML. They use contextual
|
|
457
|
+
phrase matching in both tokenizers; this is not complete inflection coverage or
|
|
458
|
+
syntactic analysis. Bare verbs such as `効く`, `壊れる`, `溶かす`, and `倒す` are
|
|
459
|
+
not added as general metaphor candidates. Identify the actual effect, policy,
|
|
460
|
+
work, or failure state from evidence rather than applying a fixed replacement.
|
|
461
|
+
Keep valid technical meanings and necessary negation.
|
|
462
|
+
|
|
463
|
+
The shared criteria distinguish reader instructions, actual system behavior,
|
|
464
|
+
and available capabilities. They also separate readability from checks for
|
|
465
|
+
missing information and unsupported additions. Sentence length, punctuation,
|
|
466
|
+
and list density are contextual cues, not fixed acceptance thresholds. These
|
|
467
|
+
updates are included in 0.6.1; start a new session to use updated criteria and rules.
|
|
468
|
+
|
|
469
|
+
Literal term matching suppresses matches strictly contained in a longer matched
|
|
470
|
+
term, across categories. The longer term also suppresses contained matches when
|
|
471
|
+
it is allowed; separate occurrences remain candidates. Glossary and style checks
|
|
472
|
+
are independent. In MeCab mode, predicates outside a contextual phrase can still
|
|
473
|
+
be reported separately.
|
|
474
|
+
|
|
318
475
|
Morphological candidates include `lemma`, `part_of_speech`, `negative`, and
|
|
319
476
|
`detector` alongside the original `match`, line, and column. Negation detection
|
|
320
477
|
covers common IPADIC negative auxiliaries; it is not full semantic analysis of
|
|
@@ -327,12 +484,12 @@ reports analyzer changes instead of treating results from different dictionaries
|
|
|
327
484
|
as directly comparable. Prompt generation uses saved evidence and does not need
|
|
328
485
|
MeCab. Changed rules or dictionary settings require a new scan.
|
|
329
486
|
|
|
330
|
-
Sessions from earlier tool versions are not compatible with 0.1.
|
|
487
|
+
Sessions from earlier tool versions are not compatible with 0.6.1. Keep the original baseline for
|
|
331
488
|
an ongoing review and finish it with the original version, or start a new review
|
|
332
489
|
pass in a different directory:
|
|
333
490
|
|
|
334
491
|
```sh
|
|
335
|
-
asciidoc-pubkit review scan book.adoc --output .pubkit/review-0.1
|
|
492
|
+
asciidoc-pubkit review scan book.adoc --output .pubkit/review-0.6.1
|
|
336
493
|
```
|
|
337
494
|
|
|
338
495
|
Severity describes review priority, not proof of an error. There is no AI-authorship
|
|
@@ -365,11 +522,121 @@ analysis of inflections, negative predicates, Unicode positions, and long lines.
|
|
|
365
522
|
Install MeCab and UTF-8 IPADIC before running the complete test suite.
|
|
366
523
|
GitHub Actions is configured for Ruby 3.2, 3.3, 3.4, and 4.0 on Linux.
|
|
367
524
|
|
|
368
|
-
|
|
525
|
+
[`test/fixtures/prose_evaluation.ja.json`](test/fixtures/prose_evaluation.ja.json)
|
|
526
|
+
contains original Japanese technical prose for candidate regression and manual
|
|
527
|
+
revision evaluation. Automated tests check metaphor candidates in both
|
|
528
|
+
tokenizers, source positions, inline exclusions, and allow lists. The revision
|
|
529
|
+
pairs are manual evaluation cases, not an automated semantic checker or a
|
|
530
|
+
measured AI benchmark. Compare source and revision separately for readability,
|
|
531
|
+
information loss, and unsupported additions, using each case's review axes and
|
|
532
|
+
expected disposition. Cover conditions, negation, numbers, versions, actors,
|
|
533
|
+
causes, implementation requirements, and necessary repetition. Record the
|
|
534
|
+
model, prompt, source evidence, and human judgments when evaluating actual
|
|
535
|
+
generated revisions; fewer findings alone do not establish an improvement.
|
|
536
|
+
|
|
537
|
+
### Development prose benchmark
|
|
538
|
+
|
|
539
|
+
The checkout provides `script/evaluate-prose` for measuring future detector and
|
|
540
|
+
prompt changes. This is a development tool, not an installed gem command; the
|
|
541
|
+
benchmark CLI calls are separate from the optional `review score --agent` path.
|
|
542
|
+
The fixed
|
|
543
|
+
[`prose_benchmark.ja.adoc`](test/fixtures/prose_benchmark.ja.adoc) contains 13
|
|
544
|
+
original editorial paragraphs with intentionally vague framing and metaphors,
|
|
545
|
+
inflection variants, and valid technical prose controls. Its
|
|
546
|
+
[`annotations`](test/fixtures/prose_benchmark.ja.json) contain 11 review targets
|
|
547
|
+
and separate source fact checklists. It is a small synthetic regression corpus,
|
|
548
|
+
not an AI authorship dataset or a representative measure of prose quality.
|
|
549
|
+
|
|
550
|
+
Run the deterministic benchmark before and after a change:
|
|
369
551
|
|
|
370
|
-
|
|
552
|
+
```sh
|
|
553
|
+
bundle exec ruby script/evaluate-prose --output .pubkit/evaluation/before
|
|
554
|
+
# After changing the detector or prompt:
|
|
555
|
+
bundle exec ruby script/evaluate-prose --output .pubkit/evaluation/after \
|
|
556
|
+
--compare .pubkit/evaluation/before/report.json
|
|
557
|
+
```
|
|
371
558
|
|
|
372
|
-
|
|
559
|
+
The report measures candidate precision, recall, F1, and location accuracy on a
|
|
560
|
+
0–100 scale for `generic-framing`, metaphorical operations, and `repeated-ending`.
|
|
561
|
+
Other rule findings are counted separately and are not judged against these
|
|
562
|
+
annotations. Detection matches a target's paragraph and kind; location accuracy
|
|
563
|
+
then checks the one-based source position, represented internally as a zero-based
|
|
564
|
+
Unicode character offset within that paragraph. The annotated repeated-ending
|
|
565
|
+
location is the matching ending of the third sentence in the consecutive run.
|
|
566
|
+
A candidate can be correctly detected even when the reviewer should keep it.
|
|
567
|
+
Zero denominators are reported as `null`, not perfect scores. Duplicate detections
|
|
568
|
+
count as false positives. These scores measure the detector, not the manuscript.
|
|
569
|
+
|
|
570
|
+
The checked-in [`MeCab baseline`](benchmark/baseline-mecab.json), captured before
|
|
571
|
+
the planned detector changes, has precision 100, recall 81.818, F1 90, and location
|
|
572
|
+
accuracy 77.778. It misses two inflected metaphor phrases and mislocates two
|
|
573
|
+
repeated-ending candidates. Reports record source, criteria, rules, corpus, and
|
|
574
|
+
dictionary fingerprints. Comparisons reject changed corpora, scoring versions,
|
|
575
|
+
analyzers, or dictionaries. Reports with configuration fingerprints also reject
|
|
576
|
+
changed benchmark configuration. A fixed empty configuration
|
|
577
|
+
keeps these trials independent of auto-discovered manuscript settings.
|
|
578
|
+
Record your own baseline when dictionary fingerprints
|
|
579
|
+
differ. `--tokenizer literal` explicitly selects a separate limited-mode run;
|
|
580
|
+
there is no automatic fallback, and literal results do not validate MeCab.
|
|
581
|
+
|
|
582
|
+
### Optional local AI evaluation
|
|
583
|
+
|
|
584
|
+
Use an installed, authenticated Codex or Claude CLI to perform rewrite and judge
|
|
585
|
+
trials explicitly. A local CLI may call its configured remote model provider and
|
|
586
|
+
consume account usage; this is not an offline model test. Normal `rake test`
|
|
587
|
+
does not run either CLI. For example:
|
|
588
|
+
|
|
589
|
+
```sh
|
|
590
|
+
bundle exec ruby script/evaluate-prose --agent codex --repeats 3 \
|
|
591
|
+
--output .pubkit/evaluation/ai-before
|
|
592
|
+
bundle exec ruby script/evaluate-prose --agent codex --repeats 3 \
|
|
593
|
+
--output .pubkit/evaluation/ai-after \
|
|
594
|
+
--compare .pubkit/evaluation/ai-before/report.json
|
|
595
|
+
# Or use Claude:
|
|
596
|
+
bundle exec ruby script/evaluate-prose --agent claude \
|
|
597
|
+
--output .pubkit/evaluation/claude-before
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
Each trial generates the actual saved review prompt, requests paragraph revisions
|
|
601
|
+
as JSON, and runs the existing mechanical verifier. A fresh judge invocation
|
|
602
|
+
evaluates both original and revised prose with neutral A/B labels, alternating
|
|
603
|
+
their order across trials. The judge sees source evidence but no findings, tool
|
|
604
|
+
revision labels, or expected calibration answers. The writer receives the source
|
|
605
|
+
fact checklist in addition to the review prompt, so this measures an evidence-
|
|
606
|
+
assisted workflow rather than every real-world manuscript review.
|
|
607
|
+
|
|
608
|
+
Sentence clarity and paragraph coherence are separately rated 1–3 and displayed
|
|
609
|
+
as 0, 50, or 100 before averaging. These ordinal averages are descriptive
|
|
610
|
+
summaries, not validated interval measurements. Fact retention measures the
|
|
611
|
+
percentage of annotated source facts preserved; addition-free and substitution-
|
|
612
|
+
free paragraph percentages remain separate. Known acceptable, incorrect, and
|
|
613
|
+
unsupported revisions from `prose_evaluation.ja.json` calibrate the judge using
|
|
614
|
+
accuracy and balanced accuracy across the three dispositions. Inspect poor
|
|
615
|
+
calibration before interpreting quality scores. No combined AI-likeness score or
|
|
616
|
+
semantic pass is produced: `meaning_verified` remains `false`.
|
|
617
|
+
Mechanical verification results and their pass rate are also reported separately.
|
|
618
|
+
Quality scores from a trial with preservation violations remain visible for
|
|
619
|
+
diagnosis; a completed evaluation does not authorize accepting that revision.
|
|
620
|
+
|
|
621
|
+
One trial is a smoke test; use at least three to inspect variation. Reports retain
|
|
622
|
+
individual scores, means, population standard deviations, and ranges. They do
|
|
623
|
+
not perform a significance test. Same-model judging can favor the writer's style;
|
|
624
|
+
`--judge-agent` and `--judge-model` select a separate evaluator. `--model` selects
|
|
625
|
+
the writer model; omitted model options use each CLI's default. AI comparisons
|
|
626
|
+
require the same reported writer and judge models, CLI versions, rubric, and
|
|
627
|
+
calibration corpus. CLI defaults can change, so retain the recorded model IDs
|
|
628
|
+
and use explicit model options for controlled comparisons. Score changes can
|
|
629
|
+
reflect sampling variation as well as implementation changes.
|
|
630
|
+
|
|
631
|
+
The runner uses argument arrays rather than shell interpolation, read-only Codex
|
|
632
|
+
execution or tool-disabled Claude execution, and a per-invocation timeout
|
|
633
|
+
(`--timeout`, default 300 seconds). The CLI may still load its user configuration;
|
|
634
|
+
use consistent settings for comparisons. Current adapters require the flags
|
|
635
|
+
shown by `codex exec --help` and `claude --help`; older CLI versions may need an
|
|
636
|
+
update. Missing CLIs, authentication errors, timeouts, and malformed responses
|
|
637
|
+
fail explicitly. Existing output directories are never replaced. Partial
|
|
638
|
+
artifacts remain available after failures. Results, prompts, responses, and review
|
|
639
|
+
sessions belong under ignored `.pubkit/`; do not commit private evaluation runs.
|
|
373
640
|
|
|
374
641
|
## Customize review rules
|
|
375
642
|
|
|
@@ -411,7 +678,7 @@ rejected. YAML aliases and object tags are not supported.
|
|
|
411
678
|
| `schema_version` | Integer `1` |
|
|
412
679
|
| `terms` | Mapping containing all five categories listed below |
|
|
413
680
|
| `verbs` | Mapping from canonical verb forms to nonempty arrays of MeCab/IPADIC dictionary forms; each dictionary form belongs to only one canonical form |
|
|
414
|
-
| `sahen` | Array of nouns matched with a following `する` verb |
|
|
681
|
+
| `sahen` | Array of nouns matched with a following `する` or potential `できる` verb |
|
|
415
682
|
| `negative_only` | Array of canonical predicates restricted to negative forms; use the noun plus `する` for sahen predicates |
|
|
416
683
|
| `compound_nouns` | Array of terms matched across contiguous noun tokens; each must also appear in `abstract-reference.terms` |
|
|
417
684
|
|
|
@@ -436,3 +703,47 @@ Prompt generation and verification use the saved contents, even if the original
|
|
|
436
703
|
YAML file is subsequently edited or removed. Start a new session to apply rule
|
|
437
704
|
changes. Older sessions without a rule snapshot fall back to the currently
|
|
438
705
|
installed default file; start a new session for reproducible custom-rule reviews.
|
|
706
|
+
|
|
707
|
+
## References
|
|
708
|
+
|
|
709
|
+
The shared criteria and default questions draw on the following editorial
|
|
710
|
+
sources. Their stylistic choices differ, including whether headings should state
|
|
711
|
+
conclusions. Candidate detection therefore leaves contextual judgment, document
|
|
712
|
+
structure, and final edits to the reviewer.
|
|
713
|
+
|
|
714
|
+
- [natural-japanese](https://github.com/coji/natural-japanese): separates mechanical
|
|
715
|
+
detection from contextual judgment and identifies repetitive framing.
|
|
716
|
+
- [yomiyasu](https://github.com/nanaism/yomiyasu):
|
|
717
|
+
informs evidence-based review of
|
|
718
|
+
metaphorical operations, distinctions between instructions and capabilities,
|
|
719
|
+
and separate assessment of meaning preservation and readability. The criteria
|
|
720
|
+
and regression examples here are independently written; the upstream skill,
|
|
721
|
+
Python linter, and benchmark passages are not bundled. Fixed style thresholds
|
|
722
|
+
and an AI-likeness score are not adopted.
|
|
723
|
+
- [日本語技術文書の文章規範](https://gist.github.com/k16shikano/fd287c3133457c4fd8f5601d34aa817d):
|
|
724
|
+
informs paragraph logic, evidence scope, and meaningful uncertainty.
|
|
725
|
+
- [AI臭い文章とは何なのか](https://speakerdeck.com/nasuvitz/ai-kusai-bunshou-toha-nanina-no-ka):
|
|
726
|
+
provides examples of unnecessary contrast, abstract referents, and paired short
|
|
727
|
+
sentences.
|
|
728
|
+
|
|
729
|
+
The development evaluation adapts ideas from the following primary research;
|
|
730
|
+
its engineering audience, corpus, and rubric do not reproduce those benchmarks.
|
|
731
|
+
Research texts and datasets are not redistributed here.
|
|
732
|
+
|
|
733
|
+
- [Evaluation of Document-Level Text Simplification in Japanese](https://aclanthology.org/2026.lrec-1.85/)
|
|
734
|
+
(Yamashita et al., LREC 2026): separates information preservation from sentence
|
|
735
|
+
and document simplicity and evaluates LLM judgments against human annotations.
|
|
736
|
+
Its elementary-school audience and Wikipedia corpus differ from this project.
|
|
737
|
+
- [Evaluating Factuality in Text Simplification](https://aclanthology.org/2022.acl-long.506/)
|
|
738
|
+
(Devaraj et al., ACL 2022): distinguishes insertion, deletion, and substitution
|
|
739
|
+
errors, motivating separate checks for unsupported additions and lost facts.
|
|
740
|
+
- [Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation](https://aclanthology.org/2024.readi-1.1/)
|
|
741
|
+
(Cripwell et al., READI 2024): motivates separate readability and preservation
|
|
742
|
+
scores rather than a single aggregate. SARI-style overlap metrics are not
|
|
743
|
+
adopted because lexical agreement alone is insufficient for this prose review.
|
|
744
|
+
|
|
745
|
+
## License
|
|
746
|
+
|
|
747
|
+
Copyright 2026 CyberGarage.
|
|
748
|
+
|
|
749
|
+
Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE).
|