zero-slop 2.7.0 → 2.7.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +59 -60
- package/SKILL.md +13 -2
- package/bin/zero-slop.mjs +173 -0
- package/data/corpus/must-flag/caveat-template.md +21 -0
- package/data/corpus/must-flag/classifier-scaffolding.md +19 -0
- package/data/corpus/must-flag/label-period-list.md +10 -0
- package/data/corpus/must-flag/manifest.json +64 -0
- package/data/corpus/must-flag/unnamed-validator.md +8 -0
- package/package.json +5 -1
- package/references/eval.md +78 -20
- package/scripts/register.py +120 -7
- package/scripts/slopscore.py +182 -36
package/README.md
CHANGED
|
@@ -5,23 +5,20 @@
|
|
|
5
5
|
<img alt="tests" src="https://img.shields.io/badge/tests-passing-227B5B">
|
|
6
6
|
<img alt="dependencies" src="https://img.shields.io/badge/runtime%20dependencies-0-227B5B">
|
|
7
7
|
<img alt="privacy" src="https://img.shields.io/badge/learning-private-227B5B">
|
|
8
|
-
<img alt="version" src="https://img.shields.io/badge/version-2.7.
|
|
8
|
+
<img alt="version" src="https://img.shields.io/badge/version-2.7.6-72528F">
|
|
9
9
|
</p>
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
Take all the slop out of your AI writing. The #1 agentic anti-slop skill.
|
|
12
|
+
|
|
13
|
+
Score your writing 0 to 100 for AI slop, see the exact phrases behind the number, and edit them out without changing a single fact. Free, MIT, scored offline with no account. Worked examples and the full benchmark are at [zero-slop.ai](https://zero-slop.ai).
|
|
12
14
|
|
|
13
15
|

|
|
14
16
|
|
|
15
17
|
## Problem
|
|
16
18
|
|
|
17
|
-
AI writing has
|
|
18
|
-
"This marks a pivotal moment." Ask an AI to fix it and it sands off the vocabulary and
|
|
19
|
-
cadence that made the writing yours, and rewrites your numbers on the way.
|
|
20
|
-
|
|
19
|
+
AI writing has a distinct tell: "It's not X. It's Y." "Here's the thing nobody tells you."
|
|
21
20
|
|
|
22
|
-
Zero Slop is an Agent Skill and ships no model. Claude, GPT, or another compatible model
|
|
23
|
-
does the editing; Zero Slop supplies the workflow, the meter, and the checks that refuse
|
|
24
|
-
any change to a name, number, quotation or link.
|
|
21
|
+
Zero Slop is an Agent Skill and ships no model. Claude, GPT, or another compatible model does the editing; Zero Slop supplies the workflow and the checks that refuse any change to a name, number, quotation or link. It runs in Claude Code, Cowork, Codex, Cursor, Warp, Zed, and any other harness that reads SKILL.md.
|
|
25
22
|
|
|
26
23
|
## How to install Zero Slop
|
|
27
24
|
|
|
@@ -37,8 +34,15 @@ Or install it with `npx`:
|
|
|
37
34
|
npx skills add manavmishra/ZeroSlop --global
|
|
38
35
|
```
|
|
39
36
|
|
|
37
|
+
Or from the registry, which also installs the scorer as a command:
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
npx zero-slop install # add --harness codex|cursor|opencode|zed
|
|
41
|
+
npx zero-slop score draft.md # score without installing anything
|
|
42
|
+
```
|
|
43
|
+
|
|
40
44
|
ChatGPT users can download [`dist/zero-slop-single-file.md`](dist/zero-slop-single-file.md).
|
|
41
|
-
Claude.ai users can upload [`dist/zero-slop.zip`](dist/zero-slop.zip). `npx skills update zero-slop --global` updates
|
|
45
|
+
Claude.ai users can upload [`dist/zero-slop.zip`](dist/zero-slop.zip). `npx skills update zero-slop --global` updates it later.
|
|
42
46
|
|
|
43
47
|
## How to use Zero Slop
|
|
44
48
|
|
|
@@ -46,40 +50,37 @@ Claude.ai users can upload [`dist/zero-slop.zip`](dist/zero-slop.zip). `npx skil
|
|
|
46
50
|
/zero-slop (your writing)
|
|
47
51
|
```
|
|
48
52
|
|
|
49
|
-
You get the edited draft, a before-and-after score, and the flagged phrases quoted with
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
and drops into CI.
|
|
53
|
+
You get the edited draft, a before-and-after score, and the flagged phrases quoted with why. `/zero-slop inspect (your writing)` reviews without rewriting.
|
|
54
|
+
For a folder, `slopscore.py --batch drafts/ --gate 25` fails the build above the threshold.
|
|
55
|
+
|
|
53
56
|
## The slop that Zero Slop catches
|
|
54
57
|
|
|
55
58
|
290 weighted patterns and a 96-term lexicon, including:
|
|
56
59
|
|
|
57
|
-
1.
|
|
58
|
-
2.
|
|
59
|
-
3.
|
|
60
|
-
4.
|
|
61
|
-
5.
|
|
62
|
-
6.
|
|
63
|
-
7.
|
|
64
|
-
8.
|
|
65
|
-
9.
|
|
66
|
-
10.
|
|
67
|
-
|
|
68
|
-
A reading pass covers
|
|
69
|
-
than any span: one shape repeated seven times, statistics piled into a paragraph,
|
|
60
|
+
1. Binary contrasts: "It's not X. It's Y."
|
|
61
|
+
2. Throat-clearing openers: "Here's the thing," "Let me be clear"
|
|
62
|
+
3. Faux-insight setups: "What nobody tells you," "The part everyone misses"
|
|
63
|
+
4. Colon reveals: "The best part: it learns."
|
|
64
|
+
5. Dramatic fragments: "That's it. That's the whole thing."
|
|
65
|
+
6. Superficial analysis: "highlighting the team's commitment to innovation"
|
|
66
|
+
7. Importance puffery: "marks a pivotal moment," "a testament to"
|
|
67
|
+
8. Weasel attribution: "experts agree," "studies show"
|
|
68
|
+
9. Synonym cycling: the agent, the assistant, the tool, all one thing.
|
|
69
|
+
10. Marketing riders: "robust" and "leverage" score only beside a marketing trigger, so a runbook stays quiet.
|
|
70
|
+
|
|
71
|
+
A reading pass covers defects of the whole document, which no span pattern reaches: one shape repeated seven times, statistics piled into a paragraph,
|
|
70
72
|
paragraphs that shuffle without loss. [`references/eval.md`](references/eval.md) has all
|
|
71
|
-
|
|
73
|
+
80 checks.
|
|
72
74
|
|
|
73
75
|
Human writing scored 9 to 21 in [`data/corpus/must-not-flag/`](data/corpus/must-not-flag/);
|
|
74
76
|
unedited AI drafts averaged 77 across [`bench/examples.json`](bench/examples.json).
|
|
75
|
-
|
|
77
|
+
|
|
76
78
|
|
|
77
79
|
## How it works
|
|
78
80
|
|
|
79
81
|

|
|
80
82
|
|
|
81
|
-
Eight roles form one workflow. Each is a job rather than a service
|
|
82
|
-
several of them, each as its own pass, so nothing grades its own output.
|
|
83
|
+
Eight roles form one workflow. Each is a job rather than a service, run as its own pass so nothing grades its own output. The research supports the checks, not the number eight, which is an engineering choice.
|
|
83
84
|
|
|
84
85
|
| Role | Who does it | What happens |
|
|
85
86
|
|---|---|---|
|
|
@@ -92,7 +93,7 @@ several of them, each as its own pass, so nothing grades its own output.
|
|
|
92
93
|
| 7. Verifier | Local tools and your AI assistant | Compares text with source for facts, meaning, qualifiers, voice, format, structure. |
|
|
93
94
|
| 8. Fresh-eyes finalizer | Fresh AI pass | Reads the verified text as a first-time reader, applying only safe polish. Any final polish restarts the final checks; the same text must return unchanged before release. |
|
|
94
95
|
|
|
95
|
-
|
|
96
|
+
Studies find
|
|
96
97
|
[predictable wording](https://arxiv.org/abs/2301.11305) and
|
|
97
98
|
[overused vocabulary](https://arxiv.org/abs/2406.07016) in machine text, and authorship
|
|
98
99
|
detectors can [misclassify non-native English](https://arxiv.org/abs/2304.02819). Local
|
|
@@ -100,12 +101,10 @@ tools use only Python's standard library.
|
|
|
100
101
|
|
|
101
102
|
## Private learning from your edits
|
|
102
103
|
|
|
103
|
-
|
|
104
|
-
what you kept. Zero Slop watches nothing on its own: no file monitoring, no browser
|
|
104
|
+
Learning starts only when you hand over both versions: the original output and your edited version. Zero Slop watches nothing on its own: no file monitoring, no browser
|
|
105
105
|
hooks, no reaching into where you publish.
|
|
106
106
|
|
|
107
|
-
|
|
108
|
-
single word needs five. Each must be new and stay silent on known-human text. Private
|
|
107
|
+
Private
|
|
109
108
|
data stays under `$ZERO_SLOP_HOME`.
|
|
110
109
|
|
|
111
110
|
This is human-in-the-loop online learning. It never retrains Claude, GPT, or another
|
|
@@ -113,13 +112,13 @@ model, and involves no neural training or RLHF. A profile can exempt existing wa
|
|
|
113
112
|
words when selected by name; it does not learn cadence, tone, or a complete writing
|
|
114
113
|
style.
|
|
115
114
|
|
|
116
|
-
|
|
117
115
|
## What's inside
|
|
118
116
|
|
|
119
117
|
[`SKILL.md`](SKILL.md) has the workflow and [`references/eval.md`](references/eval.md) the
|
|
120
|
-
|
|
118
|
+
80 checks. [`scripts/slopscore.py`](scripts/slopscore.py) is the meter and fact gate,
|
|
121
119
|
with [`scripts/register.py`](scripts/register.py) running the reading pass.
|
|
122
120
|
[`bench/README.md`](bench/README.md) documents every benchmark with its limits.
|
|
121
|
+
[zero-slop.ai](https://zero-slop.ai) has the same reference as browsable pages, plus the [benchmark in full](https://zero-slop.ai/benchmark/).
|
|
123
122
|
|
|
124
123
|
## Evidence
|
|
125
124
|
|
|
@@ -127,7 +126,7 @@ with [`scripts/register.py`](scripts/register.py) running the reading pass.
|
|
|
127
126
|
|
|
128
127
|
We reran Zero Slop, [avoid-ai-writing](https://github.com/conorbronsdon/avoid-ai-writing),
|
|
129
128
|
[no-ai-slop](https://github.com/petergyang/no-ai-slop) and
|
|
130
|
-
[humanizer](https://github.com/blader/humanizer) on the same
|
|
129
|
+
[humanizer](https://github.com/blader/humanizer) on the same samples, each with
|
|
131
130
|
GPT-5.4, high reasoning, batches of three, and its pinned instructions.
|
|
132
131
|
|
|
133
132
|
| Method | Mean writing score ↓ | Passed all Zero Slop checks | Important details kept | Average length change |
|
|
@@ -140,28 +139,31 @@ GPT-5.4, high reasoning, batches of three, and its pinned instructions.
|
|
|
140
139
|
|
|
141
140
|

|
|
142
141
|
|
|
143
|
-
|
|
144
|
-
incumbent. The GPT-5.4 reviewer favored Zero Slop on 13 drafts and avoid-ai-writing on 3,
|
|
145
|
-
with 2 unresolved; the passes agreed on 16 of 18. Our source check cleared 18/18 of our
|
|
146
|
-
rewrites and 16/18 of the incumbent's. On mean score across this second set we lost,
|
|
147
|
-
17.8 to 17.0.
|
|
142
|
+
|
|
148
143
|
|
|
149
144
|

|
|
150
145
|
|
|
146
|
+
Cross-checks the tools didn't build: the AIStoryHub checker's clean
|
|
147
|
+
rates, and a method-hidden quality ranking.
|
|
148
|
+
|
|
149
|
+

|
|
150
|
+
|
|
151
|
+

|
|
152
|
+
|
|
151
153
|
This is a small LLM-reviewed regression study. It measures neither field accuracy nor a
|
|
152
154
|
universal ranking. Drafts, mappings, verdicts, hashes and limits:
|
|
153
155
|
[`bench/incumbent-blind-replay/`](bench/incumbent-blind-replay/). On the 38-item
|
|
154
|
-
editorial panel ([`bench/README.md`](bench/README.md)), v2.6
|
|
155
|
-
frozen document
|
|
156
|
-
|
|
157
|
-
|
|
156
|
+
editorial panel ([`bench/README.md`](bench/README.md)), v2.7.6 matched the prior 84.2% result
|
|
157
|
+
with every frozen document score unchanged, all 18 human controls clear and all 18 search
|
|
158
|
+
cases still caught: the release moves what the gate asks and leaves the meter
|
|
159
|
+
untouched. Median throughput was 2.32% lower across 12 runs, which is local timing
|
|
160
|
+
noise and no kind of speed claim.
|
|
158
161
|
|
|
159
162
|
### Speed
|
|
160
163
|
|
|
161
|
-
On one Apple silicon Mac: 1,000 documents in
|
|
162
|
-
15,201-word document in 0.
|
|
163
|
-
learning pass 0.
|
|
164
|
-
service-level guarantee.
|
|
164
|
+
On one Apple silicon Mac: 1,000 documents in 2.1956 seconds (455.5 per second), a
|
|
165
|
+
15,201-word document in 0.3225 seconds, the slowest stress case in 2.2577 seconds, an
|
|
166
|
+
8,000-word learning pass in 0.1811 seconds. Editing time is excluded; expect different numbers on different machines.
|
|
165
167
|
|
|
166
168
|
### Current models
|
|
167
169
|
|
|
@@ -175,21 +177,19 @@ The pinned [RAID+](https://huggingface.co/datasets/markstanl/RAID-Plus) sample y
|
|
|
175
177
|
| Gemma 3 27B | 1,634 | 21.6 | 30.4% |
|
|
176
178
|
| Llama 3.3 70B | 2,000 | 25.5 | 41.7% |
|
|
177
179
|
|
|
178
|
-
RAID+ labels capture which model produced a text
|
|
180
|
+
RAID+ labels capture which model produced a text, not how well it reads. In
|
|
179
181
|
Beemo, raw responses averaged
|
|
180
|
-
30.2, expert edits 25.3, human answers 20.0. Neither dataset
|
|
181
|
-
|
|
182
|
+
30.2, expert edits 25.3, human answers 20.0. Neither dataset has quality labels.
|
|
182
183
|
|
|
183
184
|
## Where Zero Slop came from
|
|
184
185
|
|
|
185
|
-
Zero Slop
|
|
186
|
-
avoid-ai-writing, adding a
|
|
186
|
+
Zero Slop enhances work done by no-ai-slop, humanizer, de-slop, stop-slop, unslop-text and
|
|
187
|
+
avoid-ai-writing, by adding a slop score, source protection, separate editorial passes,
|
|
187
188
|
private learning, portfolio analysis and release tests.
|
|
188
189
|
|
|
189
190
|

|
|
190
191
|
|
|
191
|
-
The chart records which features each project documents
|
|
192
|
-
quality. Reproduce the shipped checks:
|
|
192
|
+
The chart records which features each project documents. It says nothing about writing quality and is not a claim about which tool writes better. Reproduce by using these tests:
|
|
193
193
|
|
|
194
194
|
```sh
|
|
195
195
|
python3 tests/test_all.py
|
|
@@ -198,7 +198,6 @@ python3 scripts/register.py --selftest
|
|
|
198
198
|
python3 bench/make_charts.py --check
|
|
199
199
|
```
|
|
200
200
|
|
|
201
|
-
|
|
202
201
|
## License
|
|
203
202
|
|
|
204
203
|
MIT
|
package/SKILL.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: zero-slop
|
|
3
3
|
license: MIT
|
|
4
4
|
metadata:
|
|
5
|
-
version: "2.7.
|
|
5
|
+
version: "2.7.6"
|
|
6
6
|
author: manavmishra
|
|
7
7
|
description: Turn drafts into sharp, natural prose or inspect them without rewriting. Zero Slop runs inside the user's existing AI assistant; Claude, GPT, or another compatible model reads and edits in context while local tools point to exact phrases and protect the source. Use when the user asks to humanize or de-slop writing, inspect AI-sounding patterns, fix text that reads like ChatGPT, polish outward-facing prose, draft social or LinkedIn content, or apply a final quality check to prose the agent generated. The workflow preserves facts, voice, and format and learns privately from repeated, reason-labelled human edits.
|
|
8
8
|
---
|
|
@@ -165,7 +165,9 @@ conversation, a linked or supplied sample, or none). A sample-built, named
|
|
|
165
165
|
scoring profile under `$ZERO_SLOP_HOME/voices/` contains only existing
|
|
166
166
|
watchlist-word exceptions. It does not contain the sample or capture the
|
|
167
167
|
writer's cadence, syntax, humor, or tone. Skip code blocks, quotes, and legal
|
|
168
|
-
boilerplate
|
|
168
|
+
boilerplate — but only the quoted or boilerplate words themselves: the authored frame
|
|
169
|
+
around them (labels, emphasis, list geometry) is the writer's prose and stays in
|
|
170
|
+
scope.
|
|
169
171
|
**Record the input format** — pasted text, .md, .docx, .pdf,
|
|
170
172
|
.html, .txt, a JSON field — because the output must come back in that same
|
|
171
173
|
format (step 9). Take a form inventory: decide which parts of the document are
|
|
@@ -449,6 +451,8 @@ the most reader value. `references/rewrite-moves.md` expands each rung.
|
|
|
449
451
|
"decided". Kill participial openers ("Leveraging X, …"). Translate internal
|
|
450
452
|
workflow labels into plain language; never let evaluator or harness language
|
|
451
453
|
leak into reader-facing prose.
|
|
454
|
+
Strong claims the author owns are content, not register: cut an intensifier
|
|
455
|
+
only for a defect you can name in context, never for strength alone.
|
|
452
456
|
Prefer an explicit actor and an active verb when responsibility matters. Keep
|
|
453
457
|
passive voice when the actor is unknown, irrelevant, deliberately withheld, or
|
|
454
458
|
native to the genre; passive voice alone is not evidence of AI writing.
|
|
@@ -595,6 +599,13 @@ format, and non-prose structure. Apply these contextual checks too:
|
|
|
595
599
|
`_coverage` map by dispositioning every paragraph; the verdict fails on any
|
|
596
600
|
paragraph nobody dispositioned, exactly as it fails on an unanswered check.
|
|
597
601
|
A non-zero exit is a failed check.
|
|
602
|
+
- **The delta.** Run
|
|
603
|
+
`python3 <skill-root>/scripts/register.py --delta <original> <final>` and
|
|
604
|
+
answer for what it prints: every inserted run must restate source meaning,
|
|
605
|
+
every cut emphasis word needs a named defect, and every rewritten span passes
|
|
606
|
+
the three direction tests — purpose has not become outcome, agency has not
|
|
607
|
+
moved, a warned future has not become an asserted present. The fact gate
|
|
608
|
+
cannot see any of these; this is where a reframed claim gets caught.
|
|
598
609
|
- **Performed register.** Re-run the step 2 performed-register pass on the exact
|
|
599
610
|
final text and state the counts. An exceeded antithesis budget, or a surviving
|
|
600
611
|
significance-scaffolding sentence, is a failed check: the text returns through
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// The npm package shipped four versions with no bin, no main and no scripts, so
|
|
3
|
+
// `npx zero-slop` did nothing and every real install went through the `skills`
|
|
4
|
+
// CLI against GitHub instead. That left the registry download count measuring
|
|
5
|
+
// mirrors rather than people. This is the executable half: it installs the same
|
|
6
|
+
// runtime the tarball already carries, and runs the scorer without a checkout.
|
|
7
|
+
|
|
8
|
+
import { cp, mkdir, readFile, rm, stat } from "node:fs/promises";
|
|
9
|
+
import { spawn } from "node:child_process";
|
|
10
|
+
import { homedir } from "node:os";
|
|
11
|
+
import { dirname, join, resolve } from "node:path";
|
|
12
|
+
import { fileURLToPath } from "node:url";
|
|
13
|
+
|
|
14
|
+
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
|
15
|
+
const PAYLOAD = ["SKILL.md", "references", "scripts", "data"];
|
|
16
|
+
|
|
17
|
+
// Where each harness reads global skills from. `skills add` knows this mapping
|
|
18
|
+
// too; it is repeated here so the package works without that CLI installed.
|
|
19
|
+
const HARNESS_DIRS = {
|
|
20
|
+
claude: ".claude/skills",
|
|
21
|
+
codex: ".codex/skills",
|
|
22
|
+
cursor: ".cursor/skills",
|
|
23
|
+
opencode: ".config/opencode/skills",
|
|
24
|
+
zed: ".config/zed/skills",
|
|
25
|
+
};
|
|
26
|
+
|
|
27
|
+
async function version() {
|
|
28
|
+
const pkg = JSON.parse(await readFile(join(ROOT, "package.json"), "utf8"));
|
|
29
|
+
return pkg.version;
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
function usage(v) {
|
|
33
|
+
return `zero-slop ${v} — score AI-sounding prose 0-100 and edit it out.
|
|
34
|
+
|
|
35
|
+
Usage
|
|
36
|
+
npx zero-slop install [--harness <name>] [--dir <path>] [--force]
|
|
37
|
+
npx zero-slop score <file>... [-- <slopscore flags>]
|
|
38
|
+
npx zero-slop where
|
|
39
|
+
npx zero-slop --version
|
|
40
|
+
|
|
41
|
+
Install targets
|
|
42
|
+
--harness ${Object.keys(HARNESS_DIRS).join(", ")} (default: claude)
|
|
43
|
+
--dir install into an explicit directory instead
|
|
44
|
+
--force overwrite an existing installation
|
|
45
|
+
|
|
46
|
+
Examples
|
|
47
|
+
npx zero-slop install # ~/.claude/skills/zero-slop
|
|
48
|
+
npx zero-slop install --harness codex
|
|
49
|
+
npx zero-slop score draft.md
|
|
50
|
+
npx zero-slop score drafts/ -- --batch --gate 25
|
|
51
|
+
|
|
52
|
+
Docs: https://zero-slop.ai Source: https://github.com/manavmishra/ZeroSlop`;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
function parseArgs(argv) {
|
|
56
|
+
const flags = {};
|
|
57
|
+
const rest = [];
|
|
58
|
+
let passthrough = [];
|
|
59
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
60
|
+
const a = argv[i];
|
|
61
|
+
if (a === "--") {
|
|
62
|
+
passthrough = argv.slice(i + 1);
|
|
63
|
+
break;
|
|
64
|
+
}
|
|
65
|
+
if (a === "--force") flags.force = true;
|
|
66
|
+
else if (a === "--harness" || a === "--dir") flags[a.slice(2)] = argv[++i];
|
|
67
|
+
else if (a.startsWith("--harness=")) flags.harness = a.split("=")[1];
|
|
68
|
+
else if (a.startsWith("--dir=")) flags.dir = a.split("=")[1];
|
|
69
|
+
else rest.push(a);
|
|
70
|
+
}
|
|
71
|
+
return { flags, rest, passthrough };
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
function targetDir(flags) {
|
|
75
|
+
if (flags.dir) return resolve(flags.dir);
|
|
76
|
+
const harness = flags.harness ?? "claude";
|
|
77
|
+
const base = HARNESS_DIRS[harness];
|
|
78
|
+
if (!base) {
|
|
79
|
+
throw new Error(
|
|
80
|
+
`unknown harness "${harness}". Known: ${Object.keys(HARNESS_DIRS).join(", ")}`,
|
|
81
|
+
);
|
|
82
|
+
}
|
|
83
|
+
return join(homedir(), base, "zero-slop");
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
async function exists(p) {
|
|
87
|
+
try {
|
|
88
|
+
await stat(p);
|
|
89
|
+
return true;
|
|
90
|
+
} catch {
|
|
91
|
+
return false;
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
async function install(flags) {
|
|
96
|
+
const dest = targetDir(flags);
|
|
97
|
+
if ((await exists(dest)) && !flags.force) {
|
|
98
|
+
console.error(
|
|
99
|
+
`zero-slop is already installed at ${dest}\n` +
|
|
100
|
+
`Re-run with --force to overwrite it, or --dir to install elsewhere.`,
|
|
101
|
+
);
|
|
102
|
+
return 1;
|
|
103
|
+
}
|
|
104
|
+
// Replace rather than merge: a stale reference file left behind by an older
|
|
105
|
+
// version is a silent behaviour change, which is the failure this avoids.
|
|
106
|
+
await rm(dest, { recursive: true, force: true });
|
|
107
|
+
await mkdir(dest, { recursive: true });
|
|
108
|
+
for (const entry of PAYLOAD) {
|
|
109
|
+
await cp(join(ROOT, entry), join(dest, entry), { recursive: true });
|
|
110
|
+
}
|
|
111
|
+
console.log(`Installed zero-slop ${await version()} into ${dest}`);
|
|
112
|
+
console.log("Restart your agent, then run: /zero-slop (your writing)");
|
|
113
|
+
return 0;
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
function runScorer(args) {
|
|
117
|
+
return new Promise((resolvePromise) => {
|
|
118
|
+
const script = join(ROOT, "scripts", "slopscore.py");
|
|
119
|
+
const child = spawn("python3", [script, ...args], { stdio: "inherit" });
|
|
120
|
+
child.on("error", (err) => {
|
|
121
|
+
if (err.code === "ENOENT") {
|
|
122
|
+
console.error(
|
|
123
|
+
"python3 was not found on PATH. The scorer is a standard-library " +
|
|
124
|
+
"Python program and needs Python 3 to run.",
|
|
125
|
+
);
|
|
126
|
+
resolvePromise(127);
|
|
127
|
+
return;
|
|
128
|
+
}
|
|
129
|
+
console.error(err.message);
|
|
130
|
+
resolvePromise(1);
|
|
131
|
+
});
|
|
132
|
+
child.on("close", (code) => resolvePromise(code ?? 0));
|
|
133
|
+
});
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
async function main() {
|
|
137
|
+
const argv = process.argv.slice(2);
|
|
138
|
+
const { flags, rest, passthrough } = parseArgs(argv);
|
|
139
|
+
const command = rest[0];
|
|
140
|
+
const v = await version();
|
|
141
|
+
|
|
142
|
+
if (argv.includes("--version") || argv.includes("-v")) {
|
|
143
|
+
console.log(v);
|
|
144
|
+
return 0;
|
|
145
|
+
}
|
|
146
|
+
if (!command || argv.includes("--help") || argv.includes("-h") || command === "help") {
|
|
147
|
+
console.log(usage(v));
|
|
148
|
+
return command || argv.length ? 0 : 0;
|
|
149
|
+
}
|
|
150
|
+
if (command === "install") return install(flags);
|
|
151
|
+
if (command === "where") {
|
|
152
|
+
console.log(targetDir(flags));
|
|
153
|
+
return 0;
|
|
154
|
+
}
|
|
155
|
+
if (command === "score") {
|
|
156
|
+
const files = rest.slice(1);
|
|
157
|
+
if (!files.length && !passthrough.length) {
|
|
158
|
+
console.error("score needs at least one file. See: npx zero-slop --help");
|
|
159
|
+
return 2;
|
|
160
|
+
}
|
|
161
|
+
return runScorer([...files, ...passthrough]);
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
console.error(`unknown command "${command}". See: npx zero-slop --help`);
|
|
165
|
+
return 2;
|
|
166
|
+
}
|
|
167
|
+
|
|
168
|
+
main()
|
|
169
|
+
.then((code) => process.exit(code))
|
|
170
|
+
.catch((err) => {
|
|
171
|
+
console.error(err.message);
|
|
172
|
+
process.exit(1);
|
|
173
|
+
});
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Quarterly metrics review
|
|
2
|
+
|
|
3
|
+
## Traffic
|
|
4
|
+
|
|
5
|
+
Sessions rose from 41,200 to 48,900 after the March relaunch, with mobile
|
|
6
|
+
carrying most of the gain. Neither figure includes bot traffic.
|
|
7
|
+
|
|
8
|
+
## Revenue
|
|
9
|
+
|
|
10
|
+
Subscription revenue reached $212,000 against a $195,000 target, driven by the
|
|
11
|
+
annual-plan discount that ran through April. None of these numbers are audited.
|
|
12
|
+
|
|
13
|
+
## Support
|
|
14
|
+
|
|
15
|
+
Median first-response time fell from 9 hours to 4 after the queue rework. No
|
|
16
|
+
figure here covers weekend tickets.
|
|
17
|
+
|
|
18
|
+
## Hiring
|
|
19
|
+
|
|
20
|
+
Two of the three open platform roles closed in under six weeks. Neither offer
|
|
21
|
+
has been announced.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# What the logging migration cost
|
|
2
|
+
|
|
3
|
+
The economics are simple. Storage is cheap and cardinality is not, so a
|
|
4
|
+
pipeline that indexes every label pays for the labels rather than the bytes.
|
|
5
|
+
|
|
6
|
+
We ingested 4TB a day across 240 services. The dangerous part is that the bill
|
|
7
|
+
arrived monthly while the cardinality grew hourly, so the first three alerts
|
|
8
|
+
looked like billing errors.
|
|
9
|
+
|
|
10
|
+
Sampling debug lines at 1% took ingest to 900GB a day. The gap between the two
|
|
11
|
+
figures has a simple cause: 78% of the volume came from one library's retry
|
|
12
|
+
logging, which nobody had read since the library shipped.
|
|
13
|
+
|
|
14
|
+
The score has limits worth stating. It counts what it can see, and the rest is
|
|
15
|
+
the reader's judgment.
|
|
16
|
+
|
|
17
|
+
One honest caveat before the numbers: this is a single fleet over one quarter,
|
|
18
|
+
and the shape of the saving depends on how your services log. Those are the
|
|
19
|
+
ones that matter when you plan your own migration.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# Release review checklist
|
|
2
|
+
|
|
3
|
+
Before every release the platform team walks the same four gates:
|
|
4
|
+
|
|
5
|
+
1. **Latency.** "p99 under 200ms on the checkout path."
|
|
6
|
+
2. **Rollback.** "One command, under five minutes, no data loss."
|
|
7
|
+
3. **Access.** "No new scopes without a security sign-off."
|
|
8
|
+
4. **Comms.** "Status page drafted before the deploy starts."
|
|
9
|
+
|
|
10
|
+
The quoted bars come from the SRE handbook and have not changed since March.
|
|
@@ -73,6 +73,70 @@
|
|
|
73
73
|
"span": "Different teams, different fixes, same bottleneck."
|
|
74
74
|
}
|
|
75
75
|
]
|
|
76
|
+
},
|
|
77
|
+
{
|
|
78
|
+
"file": "caveat-template.md",
|
|
79
|
+
"expect": [
|
|
80
|
+
{
|
|
81
|
+
"check": "Robotic symmetry",
|
|
82
|
+
"span": "Neither figure includes bot traffic"
|
|
83
|
+
}
|
|
84
|
+
]
|
|
85
|
+
},
|
|
86
|
+
{
|
|
87
|
+
"file": "label-period-list.md",
|
|
88
|
+
"expect": [
|
|
89
|
+
{
|
|
90
|
+
"check": "List-label periods and bare-noun bullets",
|
|
91
|
+
"span": "**Latency.**"
|
|
92
|
+
}
|
|
93
|
+
]
|
|
94
|
+
},
|
|
95
|
+
{
|
|
96
|
+
"file": "unnamed-validator.md",
|
|
97
|
+
"expect": [
|
|
98
|
+
{
|
|
99
|
+
"check": "Weasel attribution",
|
|
100
|
+
"span": "An external audit"
|
|
101
|
+
},
|
|
102
|
+
{
|
|
103
|
+
"check": "Weasel attribution",
|
|
104
|
+
"span": "A\nthird-party checker"
|
|
105
|
+
}
|
|
106
|
+
]
|
|
107
|
+
},
|
|
108
|
+
{
|
|
109
|
+
"file": "classifier-scaffolding.md",
|
|
110
|
+
"expect": [
|
|
111
|
+
{
|
|
112
|
+
"metric": "classifier_scaffolding",
|
|
113
|
+
"span": "The economics are simple"
|
|
114
|
+
},
|
|
115
|
+
{
|
|
116
|
+
"metric": "classifier_scaffolding",
|
|
117
|
+
"span": "The dangerous part is"
|
|
118
|
+
},
|
|
119
|
+
{
|
|
120
|
+
"metric": "classifier_scaffolding",
|
|
121
|
+
"span": "has a simple cause"
|
|
122
|
+
},
|
|
123
|
+
{
|
|
124
|
+
"metric": "classifier_scaffolding",
|
|
125
|
+
"span": "limits worth stating"
|
|
126
|
+
},
|
|
127
|
+
{
|
|
128
|
+
"metric": "classifier_scaffolding",
|
|
129
|
+
"span": "One honest caveat"
|
|
130
|
+
},
|
|
131
|
+
{
|
|
132
|
+
"metric": "classifier_scaffolding",
|
|
133
|
+
"span": "the ones that matter"
|
|
134
|
+
},
|
|
135
|
+
{
|
|
136
|
+
"check": "Interpretive metadiscourse",
|
|
137
|
+
"span": "It counts what it can see, and the rest is the reader's judgment."
|
|
138
|
+
}
|
|
139
|
+
]
|
|
76
140
|
}
|
|
77
141
|
]
|
|
78
142
|
}
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Migration results
|
|
2
|
+
|
|
3
|
+
We moved the billing service to the new queue in April. Median settlement time
|
|
4
|
+
fell from 41 seconds to 9, and the duplicate-charge rate dropped to zero across
|
|
5
|
+
1.2 million transactions.
|
|
6
|
+
|
|
7
|
+
An external audit reviewed the cutover and found no unresolved issues. A
|
|
8
|
+
third-party checker also rated the new pipeline clean on every run it sampled.
|
package/package.json
CHANGED
|
@@ -1,7 +1,10 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "zero-slop",
|
|
3
|
-
"version": "2.7.
|
|
3
|
+
"version": "2.7.6",
|
|
4
4
|
"description": "An Agent Skill that scores AI-sounding prose 0-100 locally and rewrites it without losing a fact. Runs inside Claude Code, Codex, Cursor, Warp, Zed and other SKILL.md agents. Offline, zero dependencies, MIT.",
|
|
5
|
+
"bin": {
|
|
6
|
+
"zero-slop": "bin/zero-slop.mjs"
|
|
7
|
+
},
|
|
5
8
|
"keywords": [
|
|
6
9
|
"agent-skill",
|
|
7
10
|
"agent-skills",
|
|
@@ -32,6 +35,7 @@
|
|
|
32
35
|
"url": "https://zero-slop.ai/about/"
|
|
33
36
|
},
|
|
34
37
|
"files": [
|
|
38
|
+
"bin/",
|
|
35
39
|
"SKILL.md",
|
|
36
40
|
"references/",
|
|
37
41
|
"scripts/",
|
package/references/eval.md
CHANGED
|
@@ -30,6 +30,11 @@ Scoring this file is meaningless: it quotes tells as examples, so the meter flag
|
|
|
30
30
|
own catalogue. That is the documented exception for material quoted as an example, and
|
|
31
31
|
it is a good candidate for `data/corpus/must-not-flag/`.
|
|
32
32
|
|
|
33
|
+
The exception covers only the quoted words. The authored frame around a quotation —
|
|
34
|
+
the label that introduces it, emphasis on that label, the list geometry it sits in —
|
|
35
|
+
is the writer's own prose and every check here applies to it. A line is never exempt
|
|
36
|
+
merely because part of it is quoted.
|
|
37
|
+
|
|
33
38
|
## A. Register, which the meter cannot reach
|
|
34
39
|
|
|
35
40
|
Run every item on every draft, including one that scored clear. Record the counts in
|
|
@@ -77,10 +82,23 @@ the report even when they are zero.
|
|
|
77
82
|
significance: highlighting, underscoring, reflecting, showcasing.
|
|
78
83
|
15. **Importance puffery.** "Marks a pivotal moment." "Stands as a testament." State
|
|
79
84
|
the fact and let the reader weigh it.
|
|
80
|
-
16. **Interpretive metadiscourse
|
|
81
|
-
reader what to notice or how much weight to give it.
|
|
85
|
+
16. **Interpretive metadiscourse. Count per 1,000 words: ___** Lines that step outside
|
|
86
|
+
the subject to tell the reader what to notice or how much weight to give it. Two
|
|
87
|
+
forms, and the flat one is the one that survives every other channel. Theatrical:
|
|
88
|
+
"Here's the detail that matters:" Flat: "The economics are simple." "The dangerous
|
|
89
|
+
part is that slop feels fast." "The score has limits worth stating." "One honest
|
|
90
|
+
caveat before the ranking." Each grades, previews or ranks the point instead of
|
|
91
|
+
delivering it, and each reads as ordinary prose, so a clear writing score is no
|
|
92
|
+
evidence at all. Test every candidate by deletion: if the sentence after it still
|
|
93
|
+
lands, the classifier was scaffolding. Anchored stems are counted for you; the
|
|
94
|
+
shapes no stem reaches are yours to find, because whether a sentence earns its
|
|
95
|
+
claim is a judgment and not a match. A zero on the anchored count is not an
|
|
96
|
+
answer to this check and never closes it: the stems match the shapes they were
|
|
97
|
+
built from, and this family's whole character is that it arrives in new ones.
|
|
82
98
|
17. **Weasel attribution.** "Studies show." "Experts agree." Name the source, or flag
|
|
83
|
-
it for the writer. Never invent one.
|
|
99
|
+
it for the writer. Never invent one. An unnamed validator is the same defect —
|
|
100
|
+
"an external checker", "a third-party audit", "an independent review" — name the
|
|
101
|
+
tool or the auditor, or flag it.
|
|
84
102
|
18. **Adjective inflation and hollow intensifiers.** "a real improvement", "actual
|
|
85
103
|
results", "a genuine breakthrough", "true value". Delete the adjective; the noun
|
|
86
104
|
carries the claim or it does not. Distinct from empty adverbs (next): adverbs pad
|
|
@@ -176,6 +194,17 @@ the report even when they are zero.
|
|
|
176
194
|
47. **Wall-of-text reply.** Paragraphing that hides a sequence the reader needs. A
|
|
177
195
|
long narrative paragraph is not a wall of text merely because it is long; the tell
|
|
178
196
|
is a buried list of steps or options.
|
|
197
|
+
47a. **Compression standing in for editing. `reorder_ratio`: ___** Cutting and
|
|
198
|
+
reordering are different edits and they do not produce the same draft. Subtraction
|
|
199
|
+
leaves every surviving sentence exactly where the model put it, so a shorter draft
|
|
200
|
+
can read more machine-shaped than the one it replaced; a measured case scored the
|
|
201
|
+
original 83 for predictability, the compressed rewrite 92, and the reordered
|
|
202
|
+
rewrite 58. A ratio near zero on a rewrite that cut a lot of words means the ladder
|
|
203
|
+
delivered its lexicon and formatting rungs and skipped its order rung. That is not
|
|
204
|
+
automatically wrong: a draft whose argument already arrives in the right sequence
|
|
205
|
+
should keep it, and shuffling a sound structure to move a number is exactly the
|
|
206
|
+
over-correction hard rule 3 forbids. What the check forbids is not noticing. Say
|
|
207
|
+
which one happened and why the order you shipped is the right one.
|
|
179
208
|
|
|
180
209
|
## C. Fidelity
|
|
181
210
|
|
|
@@ -187,9 +216,27 @@ the report even when they are zero.
|
|
|
187
216
|
51. **Qualifiers survive.** Hedges, scope limits, and caveats carry the same strength
|
|
188
217
|
as the source.
|
|
189
218
|
52. **Claims not reframed.** Same names and numbers can still carry a changed
|
|
190
|
-
emphasis or implication. Compare meaning, not tokens
|
|
219
|
+
emphasis or implication. Compare meaning, not tokens, and run three direction
|
|
220
|
+
tests on every span `register.py --delta` reports as inserted or rewritten:
|
|
221
|
+
purpose has not become outcome ("changed tactics to achieve its objective" is
|
|
222
|
+
not "changed tactics until it reached its objective"); agency has not moved
|
|
223
|
+
("helps teams generate fixes" is not "generates fixes"); a warned future has
|
|
224
|
+
not become an asserted present. These are contextual comparisons against this
|
|
225
|
+
source, never a word list.
|
|
191
226
|
53. **Non-prose untouched.** Code, front matter, tables, blockquotes, identifiers,
|
|
192
227
|
paths, and heading hierarchy intact.
|
|
228
|
+
53a. **Every dropped figure ruled, not defaulted.** The gate protects every number
|
|
229
|
+
it sees and cannot tell a measured fact from a flourish, because the difference
|
|
230
|
+
is whether a source stands behind it and that lives in the document, not in the
|
|
231
|
+
digits. So it hands you each dropped figure with the sentence it came from
|
|
232
|
+
(`figure_evidence`) and waits. Rule on each one: a figure with a source, a unit,
|
|
233
|
+
a baseline or a citation must be restored; an unsourced multiplier or statistic
|
|
234
|
+
doing an intensifier's job -- "the 10x move", "tenfold", "~70% of pilots fail" --
|
|
235
|
+
was right to cut, and you record that ruling rather than letting the gate assume
|
|
236
|
+
it. Never invent a citation to keep a number alive. Unruled means unresolved:
|
|
237
|
+
the default is that the figure stays, so silence here is a failed check and not
|
|
238
|
+
a pass. Both errors are real. Restoring a flourish is how a rewrite ships fake
|
|
239
|
+
precision, and dropping a measurement is how it ships a lie.
|
|
193
240
|
|
|
194
241
|
## D. Voice and readability
|
|
195
242
|
|
|
@@ -207,44 +254,55 @@ the report even when they are zero.
|
|
|
207
254
|
59. **Proportional cutting.** No compression that stripped character. Density is
|
|
208
255
|
information per word, not fewer words.
|
|
209
256
|
60. **Read aloud.** Would this sound natural read to a sharp colleague?
|
|
210
|
-
61. **
|
|
257
|
+
61. **Emphasis survives.** `register.py --delta` lists every intensifier or
|
|
258
|
+
absolute the rewrite cut from this source. Each cut needs a defect named from
|
|
259
|
+
this file, judged in its own context: puffery, hyperbole universal, filler.
|
|
260
|
+
"Too strong" is not a defect, and a falsifiable claim the author owns keeps
|
|
261
|
+
its full strength whatever word carries it. Cutting "changed overnight" from
|
|
262
|
+
a claim the author would defend cost a blind head-to-head once.
|
|
263
|
+
62. **Earned length.** `--delta` lists every run of three or more words the rewrite
|
|
264
|
+
added. Each must restate meaning already in this source for this audience; a
|
|
265
|
+
bridge or an unpacked definition can qualify, scaffolding the author never
|
|
266
|
+
wrote cannot. A rewrite longer than its original defends every insertion or
|
|
267
|
+
loses them.
|
|
268
|
+
63. **No over-correction.** AI slop was not traded for edgy slop: forced hot takes,
|
|
211
269
|
fake first person, performed candor, staccato drama.
|
|
212
270
|
|
|
213
271
|
## E. Form
|
|
214
272
|
|
|
215
|
-
|
|
216
|
-
|
|
273
|
+
64. **Returned in the format it arrived in.** A .docx comes back a .docx.
|
|
274
|
+
65. **Reader language.** No evaluator or harness vocabulary leaked into the copy:
|
|
217
275
|
candidate, artifact, overlay, gate, scorecard, burstiness.
|
|
218
|
-
|
|
276
|
+
66. **Formatting slop.** No emoji headings, decorative bold, or bullets that should be
|
|
219
277
|
prose. Count the headings over one or two sentences: four consecutive H3s each
|
|
220
278
|
wrapping a code block and one line is a section that wants to be one section.
|
|
221
|
-
|
|
279
|
+
67. **Em dashes.** None in short copy. One or two in a long draft only where they
|
|
222
280
|
clearly beat a comma, period, or parenthesis.
|
|
223
|
-
|
|
281
|
+
68. **Hyphenated modifier stacking.** "AI-powered cloud-native data-driven platform".
|
|
224
282
|
Two stacked compound modifiers in one noun phrase is a tell; keep the hyphens
|
|
225
283
|
grammar requires and cut the rest.
|
|
226
|
-
|
|
284
|
+
69. **List-label periods and bare-noun bullets.** Bullets that are fragments ending in
|
|
227
285
|
full stops, or a list of bare noun phrases with no predicate. Either make them
|
|
228
286
|
sentences or make them a real list.
|
|
229
|
-
|
|
287
|
+
70. **Numbered-list inflation.** A numbered list where order carries no meaning, or a
|
|
230
288
|
list padded to a round number. Use a bulleted list, or prose.
|
|
231
|
-
|
|
289
|
+
71. **Chatbot artifacts.** Citation markup leaks, "Regenerate response", tool URL
|
|
232
290
|
parameters such as utm_source, and any assistant scaffolding left in the copy.
|
|
233
|
-
|
|
291
|
+
72. **Attribution accurate.** The report names who did what without guessing which
|
|
234
292
|
model is running, and never implies a Zero Slop service read the draft.
|
|
235
293
|
|
|
236
294
|
## F. Process integrity
|
|
237
295
|
|
|
238
296
|
Role 8 answers this section. It is the check no single-agent eval can make.
|
|
239
297
|
|
|
240
|
-
|
|
298
|
+
73. **Roles stayed separate.** The copy desk, read-aloud pass, verification, and
|
|
241
299
|
fresh-eyes review each ran as a distinct pass.
|
|
242
|
-
|
|
243
|
-
|
|
300
|
+
74. **No self-certification.** No role graded text it generated.
|
|
301
|
+
75. **Counts reported.** Every count in section A appears in the summary, including
|
|
244
302
|
the zeros.
|
|
245
|
-
|
|
303
|
+
76. **The exact final text cleared every check.** Not an earlier draft, not a version
|
|
246
304
|
that was repaired afterward.
|
|
247
|
-
|
|
305
|
+
77. **Role 8 approved without changes.** If it changed anything, roles 5 through 8
|
|
248
306
|
ran again on the revision.
|
|
249
|
-
|
|
307
|
+
78. **Fallbacks named honestly.** If the three-round limit was reached, the report
|
|
250
308
|
says which check failed and does not describe the result as fully verified.
|
package/scripts/register.py
CHANGED
|
@@ -38,6 +38,7 @@ BUDGETS = {
|
|
|
38
38
|
"subtractive_contrast": (6.0, 3),
|
|
39
39
|
"comma_series": (26.0, 8),
|
|
40
40
|
"significance_scaffolding": (0.0, 1),
|
|
41
|
+
"classifier_scaffolding": (1.5, 1),
|
|
41
42
|
"inanimate_agent": (4.0, 2),
|
|
42
43
|
"repeated_openings": (3.0, 2),
|
|
43
44
|
# Added after a three-way audit found eight families the reading pass missed
|
|
@@ -71,6 +72,29 @@ RX_SIGNIFICANCE = re.compile(
|
|
|
71
72
|
re.I,
|
|
72
73
|
)
|
|
73
74
|
|
|
75
|
+
# The mild half of announced significance: a clause that grades, previews or
|
|
76
|
+
# ranks the point instead of delivering it. RX_SIGNIFICANCE above catches the
|
|
77
|
+
# theatrical form ("Here's the detail that matters:"); these are the flat
|
|
78
|
+
# classifiers that read as ordinary prose and scored clear on every channel --
|
|
79
|
+
# "The economics are simple", "the ones that matter", "limits worth stating".
|
|
80
|
+
# Found by running a competing skill over ten already-clean pages, so the
|
|
81
|
+
# ratchet applies: the anchors ship here and data/corpus/must-flag keeps them
|
|
82
|
+
# honest. The shapes no anchor reaches are asked of the reader in eval.md A11,
|
|
83
|
+
# because whether a sentence earns its claim is a judgment, not a match.
|
|
84
|
+
# Wrapped prose is still prose: the separators are \s+ so a tell that happens to
|
|
85
|
+
# straddle a line break is still counted.
|
|
86
|
+
RX_CLASSIFIER = re.compile(
|
|
87
|
+
r"\b(?:the\s+\w+\s+(?:is|are|was|were)\s+simple"
|
|
88
|
+
r"|(?:has|have|had)\s+a\s+simple\s+\w+"
|
|
89
|
+
r"|the\s+(?:dangerous|important|interesting|tricky|hard|scary|surprising|real)"
|
|
90
|
+
r"\s+(?:part|thing|bit|point|question)\s+(?:here\s+)?is"
|
|
91
|
+
r"|worth\s+(?:stating|sitting\s+with|pausing\s+on|dwelling\s+on)"
|
|
92
|
+
r"|the\s+ones?\s+that\s+(?:actually\s+)?matters?"
|
|
93
|
+
r"|(?:one|an)\s+honest\s+(?:caveat|note|admission|answer)"
|
|
94
|
+
r"|that\s+is\s+the\s+(?:finding|part|point)\s+worth)\b",
|
|
95
|
+
re.I,
|
|
96
|
+
)
|
|
97
|
+
|
|
74
98
|
# Inanimate subjects performing human verbs. no-ai-slop catches this family by
|
|
75
99
|
# asking; here it is the lexically anchored subset of it.
|
|
76
100
|
RX_INANIMATE = re.compile(
|
|
@@ -252,6 +276,7 @@ def measure(text: str) -> dict:
|
|
|
252
276
|
subtractive = [" ".join(m.split()) for m in RX_SUBTRACTIVE.findall(prose)]
|
|
253
277
|
series = RX_SERIES.findall(prose)
|
|
254
278
|
significance = [" ".join(m.split()) for m in RX_SIGNIFICANCE.findall(prose)]
|
|
279
|
+
classifier = [" ".join(m.group(0).split()) for m in RX_CLASSIFIER.finditer(prose)]
|
|
255
280
|
inanimate = [" ".join(m.split()) for m in RX_INANIMATE.findall(prose)]
|
|
256
281
|
|
|
257
282
|
openings = sentence_openings(prose)
|
|
@@ -290,6 +315,7 @@ def measure(text: str) -> dict:
|
|
|
290
315
|
"subtractive_contrast": {"count": len(subtractive), "per_1k": per_k(len(subtractive)), "hits": subtractive[:12]},
|
|
291
316
|
"comma_series": {"count": len(series), "per_1k": per_k(len(series))},
|
|
292
317
|
"significance_scaffolding": {"count": len(significance), "per_1k": per_k(len(significance)), "hits": significance[:6]},
|
|
318
|
+
"classifier_scaffolding": {"count": len(classifier), "per_1k": per_k(len(classifier)), "hits": classifier[:6]},
|
|
293
319
|
"inanimate_agent": {"count": len(inanimate), "per_1k": per_k(len(inanimate)), "hits": inanimate[:8]},
|
|
294
320
|
"repeated_openings": {"count": len(repeated), "per_1k": per_k(len(repeated)), "hits": repeated[:6]},
|
|
295
321
|
"paragraph_uniformity": uniformity,
|
|
@@ -319,6 +345,7 @@ LABEL = {
|
|
|
319
345
|
"subtractive_contrast": "Binary contrasts",
|
|
320
346
|
"comma_series": "Comma-series density",
|
|
321
347
|
"significance_scaffolding": "Announced significance",
|
|
348
|
+
"classifier_scaffolding": "Graded not delivered (stems)",
|
|
322
349
|
"inanimate_agent": "Inanimate subjects, human verbs",
|
|
323
350
|
"repeated_openings": "Repeated sentence openings",
|
|
324
351
|
}
|
|
@@ -398,6 +425,13 @@ AUTO_ANSWERED = {
|
|
|
398
425
|
"comma-series density": "comma_series",
|
|
399
426
|
"announced significance": "significance_scaffolding",
|
|
400
427
|
"significance scaffolding": "significance_scaffolding",
|
|
428
|
+
# Interpretive metadiscourse is deliberately NOT here. A density metric can
|
|
429
|
+
# answer "how often does this shape appear"; it cannot answer "does this
|
|
430
|
+
# sentence earn its claim", and the anchored stems below match only the
|
|
431
|
+
# shapes they were built from. Auto-answering the check with them scored 0
|
|
432
|
+
# on a draft carrying "What's easy to miss:" and "This is the insight that
|
|
433
|
+
# changed how I think about..." -- the count silenced the question instead
|
|
434
|
+
# of answering it. The reader answers A16; the stems only give a head start.
|
|
401
435
|
}
|
|
402
436
|
SKIP_SECTIONS = {"C"} # owned by slopscore --fidelity
|
|
403
437
|
|
|
@@ -427,10 +461,18 @@ def load_checks(path: pathlib.Path | None = None) -> list[dict]:
|
|
|
427
461
|
"id": f"{section}{item.group(1)}",
|
|
428
462
|
"section": section,
|
|
429
463
|
"title": title,
|
|
430
|
-
|
|
464
|
+
# Collected as a list and joined below. Rebuilding the string on
|
|
465
|
+
# every continuation line reread the whole ask to append six words.
|
|
466
|
+
"ask": [item.group(3).strip()],
|
|
431
467
|
}
|
|
432
|
-
|
|
433
|
-
|
|
468
|
+
# Continuation lines clear the item number by three spaces on some items
|
|
469
|
+
# and four on others. Testing for four dropped every three-space item and
|
|
470
|
+
# truncated its ask to whatever fit beside the title: A2's ask parsed as
|
|
471
|
+
# the single word "The". Any indent counts now, and a whitespace-only
|
|
472
|
+
# line still falls through to the terminator below instead of appending
|
|
473
|
+
# nothing and holding the item open.
|
|
474
|
+
elif current and line.startswith((" ", "\t")) and line.strip():
|
|
475
|
+
current["ask"].append(line.strip())
|
|
434
476
|
elif current and not line.strip():
|
|
435
477
|
checks.append(current)
|
|
436
478
|
current = None
|
|
@@ -438,6 +480,7 @@ def load_checks(path: pathlib.Path | None = None) -> list[dict]:
|
|
|
438
480
|
checks.append(current)
|
|
439
481
|
|
|
440
482
|
for c in checks:
|
|
483
|
+
c["ask"] = " ".join(c["ask"]).strip()
|
|
441
484
|
low = c["title"].lower()
|
|
442
485
|
c["auto"] = next((v for k, v in AUTO_ANSWERED.items() if k in low), None)
|
|
443
486
|
c["skip"] = c["section"] in SKIP_SECTIONS
|
|
@@ -447,6 +490,7 @@ def load_checks(path: pathlib.Path | None = None) -> list[dict]:
|
|
|
447
490
|
def read_packet(text: str, name: str) -> dict:
|
|
448
491
|
"""Emit the reading brief. The host model answers it; nothing here guesses."""
|
|
449
492
|
prose = prose_of(text)
|
|
493
|
+
checks = load_checks() # one parse feeds all three views below
|
|
450
494
|
paras = []
|
|
451
495
|
for i, para in enumerate(re.split(r"\n\s*\n", prose), 1):
|
|
452
496
|
para = para.strip()
|
|
@@ -473,15 +517,15 @@ def read_packet(text: str, name: str) -> dict:
|
|
|
473
517
|
},
|
|
474
518
|
"questions": [
|
|
475
519
|
{"id": c["id"], "title": c["title"], "ask": c["ask"]}
|
|
476
|
-
for c in
|
|
520
|
+
for c in checks
|
|
477
521
|
if not c["skip"] and not c["auto"]
|
|
478
522
|
],
|
|
479
523
|
"answered_from_measurement": [
|
|
480
524
|
{"id": c["id"], "title": c["title"], "metric": c["auto"]}
|
|
481
|
-
for c in
|
|
525
|
+
for c in checks if c["auto"]
|
|
482
526
|
],
|
|
483
527
|
"handled_by_fidelity_gate": [
|
|
484
|
-
{"id": c["id"], "title": c["title"]} for c in
|
|
528
|
+
{"id": c["id"], "title": c["title"]} for c in checks if c["skip"]
|
|
485
529
|
],
|
|
486
530
|
"paragraphs": paras,
|
|
487
531
|
}
|
|
@@ -545,7 +589,7 @@ def verdict(text: str, answers: dict) -> tuple[int, str]:
|
|
|
545
589
|
out.append("")
|
|
546
590
|
|
|
547
591
|
out.append(" Read by the model:")
|
|
548
|
-
for check in
|
|
592
|
+
for check in checks: # same filtered list built above
|
|
549
593
|
qid = check["id"]
|
|
550
594
|
got = answers.get(qid)
|
|
551
595
|
if not isinstance(got, dict) or got.get("answer") not in ("pass", "fail"):
|
|
@@ -611,6 +655,68 @@ def verdict(text: str, answers: dict) -> tuple[int, str]:
|
|
|
611
655
|
return 0, "\n".join(out)
|
|
612
656
|
|
|
613
657
|
|
|
658
|
+
# Emphasis words: cutting one is sometimes right (puffery) and sometimes voice
|
|
659
|
+
# flattening ("changed overnight" is a falsifiable claim the author owns). The
|
|
660
|
+
# tool cannot tell which, so it reports every cut and the eval demands a defect
|
|
661
|
+
# name for each. "Too strong" is not a defect.
|
|
662
|
+
EMPHASIS = {
|
|
663
|
+
"overnight", "never", "always", "every", "entire", "all", "nothing",
|
|
664
|
+
"worst", "best", "first", "only", "immediately", "instantly", "forever",
|
|
665
|
+
"massive", "enormous", "catastrophically", "obsessively", "extraordinary",
|
|
666
|
+
"unprecedented", "remarkable", "completely", "exactly",
|
|
667
|
+
}
|
|
668
|
+
|
|
669
|
+
|
|
670
|
+
def delta(original: str, rewrite: str) -> dict:
|
|
671
|
+
"""Word-level diff: what the rewrite added that the author never wrote, and
|
|
672
|
+
what emphasis it took away. Insertions are never free; each run must carry
|
|
673
|
+
meaning already in the source. A rewrite 27 words longer than the original
|
|
674
|
+
once lost a blind head-to-head on exactly this."""
|
|
675
|
+
import difflib
|
|
676
|
+
a = original.split()
|
|
677
|
+
b = rewrite.split()
|
|
678
|
+
sm = difflib.SequenceMatcher(a=[w.lower().strip(".,;:!?\"'()") for w in a],
|
|
679
|
+
b=[w.lower().strip(".,;:!?\"'()") for w in b])
|
|
680
|
+
inserted, deleted, cut_emphasis = [], [], []
|
|
681
|
+
for op, i1, i2, j1, j2 in sm.get_opcodes():
|
|
682
|
+
if op in ("insert", "replace") and j2 - j1 >= 3:
|
|
683
|
+
inserted.append(" ".join(b[j1:j2])[:90])
|
|
684
|
+
if op in ("delete", "replace"):
|
|
685
|
+
for w in a[i1:i2]:
|
|
686
|
+
if w.lower().strip(".,;:!?\"'()") in EMPHASIS:
|
|
687
|
+
ctx = " ".join(a[max(0, i1 - 4):min(len(a), i2 + 4)])
|
|
688
|
+
cut_emphasis.append(f"{w} ({ctx[:70]})")
|
|
689
|
+
if op == "delete" and i2 - i1 >= 3:
|
|
690
|
+
deleted.append(" ".join(a[i1:i2])[:90])
|
|
691
|
+
return {
|
|
692
|
+
"original_words": len(a), "rewrite_words": len(b),
|
|
693
|
+
"net": len(b) - len(a),
|
|
694
|
+
"inserted_runs": inserted[:12], "deleted_runs": deleted[:12],
|
|
695
|
+
"cut_emphasis": cut_emphasis[:12],
|
|
696
|
+
}
|
|
697
|
+
|
|
698
|
+
|
|
699
|
+
def render_delta(d: dict) -> str:
|
|
700
|
+
out = [f"Length: {d['original_words']} -> {d['rewrite_words']} words "
|
|
701
|
+
f"({'+' if d['net'] >= 0 else ''}{d['net']})"]
|
|
702
|
+
if d["net"] > 0:
|
|
703
|
+
out.append(" The rewrite is LONGER than the original. Every inserted run below")
|
|
704
|
+
out.append(" must carry meaning already in the source, or it goes.")
|
|
705
|
+
out.append("")
|
|
706
|
+
out.append(f" Inserted runs the author never wrote ({len(d['inserted_runs'])}):")
|
|
707
|
+
for r in d["inserted_runs"] or ["(none)"]:
|
|
708
|
+
out.append(f" + {r}")
|
|
709
|
+
out.append(f" Cut emphasis, each needs a defect name, not 'too strong' ({len(d['cut_emphasis'])}):")
|
|
710
|
+
for r in d["cut_emphasis"] or ["(none)"]:
|
|
711
|
+
out.append(f" - {r}")
|
|
712
|
+
if d["deleted_runs"]:
|
|
713
|
+
out.append(f" Deleted runs ({len(d['deleted_runs'])}):")
|
|
714
|
+
for r in d["deleted_runs"]:
|
|
715
|
+
out.append(f" - {r}")
|
|
716
|
+
return "\n".join(out)
|
|
717
|
+
|
|
718
|
+
|
|
719
|
+
|
|
614
720
|
MUST_FLAG = EVAL_PATH.resolve().parent.parent / "data" / "corpus" / "must-flag"
|
|
615
721
|
|
|
616
722
|
|
|
@@ -736,6 +842,8 @@ def main() -> int:
|
|
|
736
842
|
ap.add_argument("--gate", action="store_true", help="exit 1 when any rate is over budget")
|
|
737
843
|
ap.add_argument("--calibrate", metavar="DIR", help="recompute budgets from a human corpus")
|
|
738
844
|
ap.add_argument("--selftest", action="store_true", help="check the gate against the checklist")
|
|
845
|
+
ap.add_argument("--delta", nargs=2, metavar=("ORIGINAL", "REWRITE"),
|
|
846
|
+
help="what the rewrite inserted, and what emphasis it cut")
|
|
739
847
|
ap.add_argument("--recall", action="store_true", help="verify every recorded miss in data/corpus/must-flag still gets caught")
|
|
740
848
|
ap.add_argument("--read", action="store_true", help="emit the reading brief for the host model")
|
|
741
849
|
ap.add_argument("--verdict", metavar="ANSWERS_JSON", help="gate on the measured rates plus the model's answers")
|
|
@@ -745,6 +853,11 @@ def main() -> int:
|
|
|
745
853
|
return _selftest()
|
|
746
854
|
if args.recall:
|
|
747
855
|
return recall()
|
|
856
|
+
if args.delta:
|
|
857
|
+
a = pathlib.Path(args.delta[0]).read_text(encoding="utf-8", errors="ignore")
|
|
858
|
+
b = pathlib.Path(args.delta[1]).read_text(encoding="utf-8", errors="ignore")
|
|
859
|
+
print(render_delta(delta(a, b)))
|
|
860
|
+
return 0
|
|
748
861
|
if args.calibrate:
|
|
749
862
|
calibrate(args.calibrate)
|
|
750
863
|
return 0
|
package/scripts/slopscore.py
CHANGED
|
@@ -941,7 +941,11 @@ CHANNELS = [
|
|
|
941
941
|
# could check, and things whose invention is the failure the skill forbids.
|
|
942
942
|
FACT_RX = [
|
|
943
943
|
("figure", r"(?<![\w.])\$?\d[\d,]*(?:\.\d+)?\s*(?:%|percent|x|bn|m|k|million|billion)?(?![\w])"),
|
|
944
|
-
|
|
944
|
+
# A name never spans a line break. Allowing \s+ here let a run swallow the
|
|
945
|
+
# paragraph boundary after a heading -- "Leverage\n\nThe", "Mishra\n\nPaste"
|
|
946
|
+
# -- and the invented run then read as a dropped entity in any rewrite that
|
|
947
|
+
# repunctuated the section.
|
|
948
|
+
("name", r"\b(?:[A-Z][a-z]{2,}(?:[ \t]+[A-Z][a-z]+)*)\b"),
|
|
945
949
|
("quote", r"[\u201c\"]([^\u201d\"]{6,120})[\u201d\"]"),
|
|
946
950
|
("url", r"https?://\S+"),
|
|
947
951
|
]
|
|
@@ -1014,6 +1018,46 @@ are artificial bookmark despite hey modern please researchers save unpopular wel
|
|
|
1014
1018
|
NOT_NAME_WORDS = {word.lower() for word in NOT_NAMES} | COMMON_WORDS
|
|
1015
1019
|
|
|
1016
1020
|
|
|
1021
|
+
def _peel_entity(run, prose, other):
|
|
1022
|
+
"""The entity inside a title-case run, or None if the run holds no name.
|
|
1023
|
+
|
|
1024
|
+
A capitalised ordinary word glues itself to the name that follows it --
|
|
1025
|
+
"With Claude", "In March", "At Acme". Discarding the whole run loses the
|
|
1026
|
+
entity, so the rewrite that punctuates the sentence differently gets
|
|
1027
|
+
reported as having dropped a name that is still sitting in it. Peel the
|
|
1028
|
+
leading word and re-test what remains.
|
|
1029
|
+
"""
|
|
1030
|
+
while run:
|
|
1031
|
+
if run in NOT_NAMES or len(run) < 3:
|
|
1032
|
+
return None
|
|
1033
|
+
low = run.lower()
|
|
1034
|
+
tokens = re.findall(r"[a-z]+", low)
|
|
1035
|
+
# A title-cased run made entirely of ordinary sentence words is not an
|
|
1036
|
+
# entity (for example, "Shipped Tuesday").
|
|
1037
|
+
if tokens and all(token in NOT_NAME_WORDS for token in tokens):
|
|
1038
|
+
return None
|
|
1039
|
+
single = " " not in run
|
|
1040
|
+
# A capitalised common word ("Draw", "Usually", "Start"), an adverb
|
|
1041
|
+
# ("Finally"), or a sentence-opening gerund ("Watching") is not an
|
|
1042
|
+
# entity; a real name never is.
|
|
1043
|
+
if single and (low in COMMON_WORDS
|
|
1044
|
+
or low.endswith("ly") or low.endswith("ing")):
|
|
1045
|
+
return None
|
|
1046
|
+
# A word is only a name if it is never used as an ordinary lowercase
|
|
1047
|
+
# word -- not here, and not in the text we compare against.
|
|
1048
|
+
# "Under"/"Shipped" appear lowercased somewhere in normal prose;
|
|
1049
|
+
# "Priya"/"Acme" do not. Strip the capitalised forms first so the
|
|
1050
|
+
# entity cannot vouch for itself.
|
|
1051
|
+
head = run.split()[0]
|
|
1052
|
+
blob = re.sub(r"\b" + re.escape(head) + r"\b", " ", prose + " " + other)
|
|
1053
|
+
if not re.search(r"\b" + re.escape(head.lower()) + r"\b", blob):
|
|
1054
|
+
return run
|
|
1055
|
+
if single:
|
|
1056
|
+
return None
|
|
1057
|
+
run = run.split(" ", 1)[1]
|
|
1058
|
+
return None
|
|
1059
|
+
|
|
1060
|
+
|
|
1017
1061
|
def facts(text, _other=""):
|
|
1018
1062
|
"""Checkable claims in a draft: figures, named entities, quotes, links."""
|
|
1019
1063
|
# URLs contain lowercase forms of the names they point at ("acme.io" made
|
|
@@ -1040,32 +1084,8 @@ def facts(text, _other=""):
|
|
|
1040
1084
|
for m in re.finditer(rx, urls if kind == "url" else prose):
|
|
1041
1085
|
v = (m.group(1) if m.lastindex else m.group(0)).strip()
|
|
1042
1086
|
if kind == "name":
|
|
1043
|
-
|
|
1044
|
-
|
|
1045
|
-
low = v.lower()
|
|
1046
|
-
tokens = re.findall(r"[a-z]+", low)
|
|
1047
|
-
# A title-cased run made entirely of ordinary sentence words is
|
|
1048
|
-
# not an entity (for example, "Shipped Tuesday").
|
|
1049
|
-
if tokens and all(token in NOT_NAME_WORDS for token in tokens):
|
|
1050
|
-
continue
|
|
1051
|
-
# A capitalised common word ("Draw", "Usually", "Start"), an
|
|
1052
|
-
# adverb ("Finally"), or a sentence-opening gerund ("Watching",
|
|
1053
|
-
# "Calling") is not an entity; a real name never is.
|
|
1054
|
-
if " " not in v and (low in COMMON_WORDS
|
|
1055
|
-
or low.endswith("ly") or low.endswith("ing")):
|
|
1056
|
-
continue
|
|
1057
|
-
# A word is only a name if it is never used as an ordinary
|
|
1058
|
-
# lowercase word — not here, and not in the text we compare
|
|
1059
|
-
# against. "Under"/"Shipped" appear lowercased somewhere in
|
|
1060
|
-
# normal prose; "Priya"/"Acme" do not. Multi-word entities keep
|
|
1061
|
-
# their head token for this test.
|
|
1062
|
-
head = v.split()[0]
|
|
1063
|
-
# Is this token ever used as an ordinary lowercase word, here or
|
|
1064
|
-
# in the compared text? Sentence openers are ("under load",
|
|
1065
|
-
# "shipped tuesday"); real names never are. Strip the capitalized
|
|
1066
|
-
# forms first so the entity cannot vouch for itself.
|
|
1067
|
-
blob = re.sub(r"\b" + re.escape(head) + r"\b", " ", prose + " " + other)
|
|
1068
|
-
if re.search(r"\b" + re.escape(head.lower()) + r"\b", blob):
|
|
1087
|
+
v = _peel_entity(v, prose, other)
|
|
1088
|
+
if not v:
|
|
1069
1089
|
continue
|
|
1070
1090
|
if kind == "figure":
|
|
1071
1091
|
v = v.replace(",", "").lstrip("$").rstrip()
|
|
@@ -1263,9 +1283,50 @@ def structure_changes(before, after):
|
|
|
1263
1283
|
return findings
|
|
1264
1284
|
|
|
1265
1285
|
|
|
1266
|
-
|
|
1286
|
+
# An unsourced figure -- "the 10x move", "tenfold", "~70% of pilots fail" -- is
|
|
1287
|
+
# an intensifier wearing a number's clothes. The gate exists to protect facts,
|
|
1288
|
+
# and a figure with no source behind it is not one. Protecting it anyway made
|
|
1289
|
+
# the gate report the honest cut as a dropped fact, and rerank sorts on
|
|
1290
|
+
# fidelity first, so the rewrite that KEPT the fake precision won. That is the
|
|
1291
|
+
# gate preserving slop, which is the opposite of its job.
|
|
1292
|
+
#
|
|
1293
|
+
# Which figures are load-bearing is a contextual judgment, so this script does
|
|
1294
|
+
# not make it. No pattern can separate "fell 40%" from "10x better" reliably --
|
|
1295
|
+
# the difference is whether a source stands behind the number, which lives in
|
|
1296
|
+
# the surrounding document, not in the digits. The tool's job is to hand the
|
|
1297
|
+
# reviewer the evidence; the ruling belongs to the assistant running the
|
|
1298
|
+
# verifier role (SKILL step 7, "Unsourced statistics") or to the writer. A
|
|
1299
|
+
# figure is protected until someone with context says otherwise, so the default
|
|
1300
|
+
# behaviour here is exactly as strict as it was before.
|
|
1301
|
+
|
|
1302
|
+
|
|
1303
|
+
def figure_contexts(text, figures):
|
|
1304
|
+
"""Each figure with the sentence it sits in, so a reviewer can rule on it.
|
|
1305
|
+
|
|
1306
|
+
Evidence, not a verdict: the caller decides whether a dropped figure was a
|
|
1307
|
+
measured fact that must be restored or an unsourced flourish that was right
|
|
1308
|
+
to cut.
|
|
1309
|
+
"""
|
|
1310
|
+
out = {}
|
|
1311
|
+
for figure in figures:
|
|
1312
|
+
for sentence in sentences(text):
|
|
1313
|
+
if re.search(r"\b" + re.escape(figure) + r"\b", sentence, re.I):
|
|
1314
|
+
out[figure] = " ".join(sentence.split())
|
|
1315
|
+
break
|
|
1316
|
+
else:
|
|
1317
|
+
out[figure] = ""
|
|
1318
|
+
return out
|
|
1319
|
+
|
|
1320
|
+
|
|
1321
|
+
def fidelity(before, after, adjudicated=None):
|
|
1267
1322
|
"""Did the rewrite keep every fact, and did it add any?
|
|
1268
1323
|
|
|
1324
|
+
``adjudicated`` is the set of dropped figures a reviewer with context has
|
|
1325
|
+
ruled unsourced, so cutting them is an improvement rather than a loss. It
|
|
1326
|
+
is empty unless someone explicitly rules, which keeps the deterministic
|
|
1327
|
+
default strict: this function never decides on its own that a number was
|
|
1328
|
+
only rhetoric.
|
|
1329
|
+
|
|
1269
1330
|
The benchmark's worst result was a rewrite that invented a feeling the
|
|
1270
1331
|
author never described — the exact thing hard rule 1 forbids — and nothing
|
|
1271
1332
|
in the gate measured it. Preservation is checkable; invention is the half
|
|
@@ -1275,6 +1336,8 @@ def fidelity(before, after):
|
|
|
1275
1336
|
a, b = facts(before, after), facts(after, before)
|
|
1276
1337
|
structure = structure_changes(before, after)
|
|
1277
1338
|
rows, kept_all, invented_any = [], True, False
|
|
1339
|
+
adjudicated = set(adjudicated or ())
|
|
1340
|
+
unsourced, dropped_items = set(), set()
|
|
1278
1341
|
def entity_tokens(entity):
|
|
1279
1342
|
return {w for w in re.findall(r"[a-z]+", entity.lower())
|
|
1280
1343
|
if w not in NOT_NAME_WORDS}
|
|
@@ -1298,10 +1361,26 @@ def fidelity(before, after):
|
|
|
1298
1361
|
dropped = {e for e in a[kind] if not entity_match(e, b[kind])}
|
|
1299
1362
|
added = {e for e in b[kind] if not entity_match(e, a[kind])}
|
|
1300
1363
|
kept = a[kind] - dropped
|
|
1364
|
+
# A word capitalised once at a heading or sentence start reads
|
|
1365
|
+
# exactly like a product name to any lexical rule -- "Embedded
|
|
1366
|
+
# governance", "Models + Context = Leverage". Which one it is
|
|
1367
|
+
# depends on the document, so the reviewer rules and the tool
|
|
1368
|
+
# supplies the sentence rather than guessing.
|
|
1369
|
+
unsourced |= dropped & adjudicated
|
|
1370
|
+
dropped = dropped - adjudicated
|
|
1371
|
+
dropped_items |= dropped
|
|
1301
1372
|
else:
|
|
1302
1373
|
kept = a[kind] & b[kind]
|
|
1303
1374
|
dropped = a[kind] - b[kind]
|
|
1304
1375
|
added = b[kind] - a[kind]
|
|
1376
|
+
if kind == "figure" and dropped:
|
|
1377
|
+
# Figures the reviewer has ruled unsourced were right to cut,
|
|
1378
|
+
# so they stop failing preservation. Nothing is ruled without
|
|
1379
|
+
# that explicit judgment, and adding a figure is still an
|
|
1380
|
+
# invention however it was ruled.
|
|
1381
|
+
unsourced |= dropped & adjudicated
|
|
1382
|
+
dropped = dropped - adjudicated
|
|
1383
|
+
dropped_items |= dropped
|
|
1305
1384
|
if not (a[kind] or b[kind]):
|
|
1306
1385
|
continue
|
|
1307
1386
|
rows.append((kind, kept, dropped, added))
|
|
@@ -1316,7 +1395,55 @@ def fidelity(before, after):
|
|
|
1316
1395
|
kept_all = False
|
|
1317
1396
|
invented_any = invented_any or any(row["added"] for row in structure)
|
|
1318
1397
|
return {"rows": rows, "preserved": kept_all, "invented": invented_any,
|
|
1319
|
-
"interior": new_interior, "structure": structure
|
|
1398
|
+
"interior": new_interior, "structure": structure,
|
|
1399
|
+
"unsourced": unsourced,
|
|
1400
|
+
# The sentence each dropped figure came from, so whoever rules on
|
|
1401
|
+
# it can see whether a source stood behind the number.
|
|
1402
|
+
"figure_evidence": figure_contexts(before, dropped_items)}
|
|
1403
|
+
|
|
1404
|
+
|
|
1405
|
+
def reorder_ratio(before, after):
|
|
1406
|
+
"""How much of the surviving material the rewrite actually moved.
|
|
1407
|
+
|
|
1408
|
+
0.0 means every kept sentence is still in its original order; 1.0 means the
|
|
1409
|
+
order was inverted. Cutting and reordering are different edits with
|
|
1410
|
+
different results: subtraction leaves the surviving prose sitting exactly
|
|
1411
|
+
where the model would have put it, while moving the payoff changes what the
|
|
1412
|
+
reader meets first. Nothing in the gate could tell the two apart, so a
|
|
1413
|
+
compression-only rewrite passed every check the ladder's order rung was
|
|
1414
|
+
supposed to enforce.
|
|
1415
|
+
"""
|
|
1416
|
+
def shingles(text):
|
|
1417
|
+
out = []
|
|
1418
|
+
for sentence in sentences(text):
|
|
1419
|
+
words = {w for w in re.findall(r"[a-z]{4,}", sentence.lower())
|
|
1420
|
+
if w not in NOT_NAME_WORDS}
|
|
1421
|
+
if words:
|
|
1422
|
+
out.append(words)
|
|
1423
|
+
return out
|
|
1424
|
+
|
|
1425
|
+
src, dst = shingles(before), shingles(after)
|
|
1426
|
+
if len(src) < 2 or len(dst) < 2:
|
|
1427
|
+
return 0.0
|
|
1428
|
+
order = []
|
|
1429
|
+
for target in dst:
|
|
1430
|
+
best, best_at = 0.0, None
|
|
1431
|
+
for i, source in enumerate(src):
|
|
1432
|
+
union = len(target | source)
|
|
1433
|
+
overlap = len(target & source) / union if union else 0.0
|
|
1434
|
+
if overlap > best:
|
|
1435
|
+
best, best_at = overlap, i
|
|
1436
|
+
if best >= 0.3 and best_at is not None:
|
|
1437
|
+
order.append(best_at)
|
|
1438
|
+
if len(order) < 2:
|
|
1439
|
+
return 0.0
|
|
1440
|
+
pairs = inversions = 0
|
|
1441
|
+
for i in range(len(order)):
|
|
1442
|
+
for j in range(i + 1, len(order)):
|
|
1443
|
+
pairs += 1
|
|
1444
|
+
if order[i] > order[j]:
|
|
1445
|
+
inversions += 1
|
|
1446
|
+
return round(inversions / pairs, 3) if pairs else 0.0
|
|
1320
1447
|
|
|
1321
1448
|
|
|
1322
1449
|
# The shared rewrite-quality objective. One definition of "a better rewrite",
|
|
@@ -1326,10 +1453,18 @@ def fidelity(before, after):
|
|
|
1326
1453
|
RW_GATE = {"email": 35, "research": 40, "professional": 40}
|
|
1327
1454
|
RW_GATE_DEFAULT = 25
|
|
1328
1455
|
RW_FORMAL = {"research", "professional"}
|
|
1329
|
-
|
|
1330
|
-
|
|
1331
|
-
|
|
1332
|
-
|
|
1456
|
+
# "structure" exists because the other four terms all saturate on a draft that
|
|
1457
|
+
# arrives clean: deslop is ~0 when there is no slop to remove, and gate, rhythm
|
|
1458
|
+
# and length each cap at 1.0, so every candidate scored an identical 0.55 and
|
|
1459
|
+
# the ranking fell through to the fidelity tier. A meter with no opinion about
|
|
1460
|
+
# which rewrite is better is the reason a worse rewrite could win.
|
|
1461
|
+
RW_WEIGHTS = {"deslop": 0.40, "gate": 0.20, "rhythm": 0.12, "length": 0.13,
|
|
1462
|
+
"structure": 0.15}
|
|
1463
|
+
RW_REORDER_FULL = 0.20
|
|
1464
|
+
|
|
1465
|
+
|
|
1466
|
+
def rewrite_score(before_text, after_text, genre=None, data=None,
|
|
1467
|
+
adjudicated=None):
|
|
1333
1468
|
"""Score one rewrite: a soft quality in [0,1] plus its fidelity flags."""
|
|
1334
1469
|
if data is None:
|
|
1335
1470
|
data = load_patterns()
|
|
@@ -1340,14 +1475,25 @@ def rewrite_score(before_text, after_text, genre=None, data=None):
|
|
|
1340
1475
|
clamp = lambda x: max(0.0, min(1.0, x))
|
|
1341
1476
|
deslop = clamp((b_ai - a["ai_likelihood"]) / b_ai)
|
|
1342
1477
|
gate = 1.0 if a["ai_likelihood"] <= RW_GATE.get(genre, RW_GATE_DEFAULT) else 0.0
|
|
1343
|
-
|
|
1478
|
+
# Formal genres score with the rhythm-uniformity penalty switched off,
|
|
1479
|
+
# because an even pulse is native to an abstract rather than a tell. The
|
|
1480
|
+
# objective was still paying for burstiness there, so a casualised abstract
|
|
1481
|
+
# outranked one that kept its register -- the composite penalising formal
|
|
1482
|
+
# writing for being formal, which is the thing --formal exists to stop.
|
|
1483
|
+
rhythm = 1.0 if formal else clamp(a.get("burstiness", 0.0) / 0.45)
|
|
1344
1484
|
bw, aw = len(before_text.split()), len(after_text.split())
|
|
1345
1485
|
length = 1.0 if not bw or aw / bw >= 0.6 else clamp((aw / bw) / 0.6)
|
|
1486
|
+
reorder = reorder_ratio(before_text, after_text)
|
|
1487
|
+
structure = clamp(reorder / RW_REORDER_FULL)
|
|
1346
1488
|
soft = sum(RW_WEIGHTS[k] * v for k, v in
|
|
1347
|
-
{"deslop": deslop, "gate": gate, "rhythm": rhythm,
|
|
1348
|
-
|
|
1489
|
+
{"deslop": deslop, "gate": gate, "rhythm": rhythm,
|
|
1490
|
+
"length": length, "structure": structure}.items())
|
|
1491
|
+
fid = fidelity(before_text, after_text, adjudicated)
|
|
1349
1492
|
return {"soft": round(soft, 4), "deslop": round(deslop, 3), "gate": gate,
|
|
1350
1493
|
"rhythm": round(rhythm, 3), "length": round(length, 3),
|
|
1494
|
+
"structure": round(structure, 3), "reorder": reorder,
|
|
1495
|
+
"unsourced": sorted(fid["unsourced"]),
|
|
1496
|
+
"figure_evidence": fid["figure_evidence"],
|
|
1351
1497
|
"after_ai": a["ai_likelihood"], "before_ai": b["ai_likelihood"],
|
|
1352
1498
|
"burstiness": round(a.get("burstiness", 0.0), 3),
|
|
1353
1499
|
"high_tells": sum(1 for h in a.get("hits", []) if h.get("w", 0) >= 4),
|