zero-slop 2.8.0 → 2.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -13
- package/SKILL.md +1 -1
- package/data/corpus/must-flag/manifest.json +134 -121
- package/data/corpus/must-flag/plural-noun-fragment.md +4 -0
- package/package.json +1 -1
- package/scripts/register.py +182 -28
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
<img alt="tests" src="https://img.shields.io/badge/tests-passing-227B5B">
|
|
6
6
|
<img alt="dependencies" src="https://img.shields.io/badge/runtime%20dependencies-0-227B5B">
|
|
7
7
|
<img alt="privacy" src="https://img.shields.io/badge/learning-private-227B5B">
|
|
8
|
-
<img alt="version" src="https://img.shields.io/badge/version-2.8.
|
|
8
|
+
<img alt="version" src="https://img.shields.io/badge/version-2.8.2-72528F">
|
|
9
9
|
<a href="https://hol.org/guard/plugins"><img alt="Listed in the HOL plugin registry" src="https://img.shields.io/badge/HOL%20registry-listed-2C6E8F"></a>
|
|
10
10
|
<a href="https://github.com/hashgraph-online/awesome-ai-plugins#tools--integrations"><img alt="Listed in awesome-ai-plugins" src="https://img.shields.io/badge/awesome--ai--plugins-listed-2C6E8F"></a>
|
|
11
11
|
<a href="https://zero-slop.ai/try/"><img alt="This README scores 12.0 out of 100 on the slop score, inside the human range" src="https://img.shields.io/badge/slop%20score-12.0%2F100-0f7d55"></a>
|
|
@@ -191,10 +191,9 @@ with [`scripts/register.py`](scripts/register.py) running the reading pass.
|
|
|
191
191
|
|
|
192
192
|
The saved replay ran Zero Slop, [avoid-ai-writing](https://github.com/conorbronsdon/avoid-ai-writing),
|
|
193
193
|
[no-ai-slop](https://github.com/petergyang/no-ai-slop) and
|
|
194
|
-
[humanizer](https://github.com/blader/humanizer)
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
pretend to have regenerated them.
|
|
194
|
+
[humanizer](https://github.com/blader/humanizer) over 18 drafts with GPT-5.4, high
|
|
195
|
+
reasoning, and pinned instructions. Zero Slop's outputs came from v2.5.9; later
|
|
196
|
+
releases rescore those frozen outputs rather than pretend to regenerate them.
|
|
198
197
|
|
|
199
198
|
| Method | Mean writing score ↓ | Passed Zero Slop's local gates | Source check passed | Average length change |
|
|
200
199
|
|---|---:|---:|---:|---:|
|
|
@@ -218,23 +217,44 @@ rates, and a method-hidden quality ranking.
|
|
|
218
217
|
|
|
219
218
|
|
|
220
219
|
This is a small LLM-reviewed regression study. It measures neither field accuracy nor a
|
|
221
|
-
universal ranking. Drafts,
|
|
222
|
-
in [`bench/README.md`](bench/README.md). The separate method-hidden two-way replay used
|
|
220
|
+
universal ranking. Drafts, hashes, and limits are in [`bench/README.md`](bench/README.md). The separate method-hidden two-way replay used
|
|
223
221
|
Zero Slop v2.6.0 and is preserved in
|
|
224
222
|
[`bench/incumbent-blind-replay/`](bench/incumbent-blind-replay/).
|
|
225
223
|
|
|
226
224
|
For the 38-item editorial panel, the current scorer matched the prior 84.2% result.
|
|
227
225
|
All frozen scores stayed unchanged, all 18 human controls remained below the gate,
|
|
228
226
|
and all 18 obvious search cases remained above it. These fixed-sample checks are not
|
|
229
|
-
proof of general accuracy.
|
|
230
|
-
local runs; treat that as machine noise, not a speed claim.
|
|
227
|
+
proof of general accuracy.
|
|
231
228
|
|
|
232
229
|
### Speed
|
|
233
230
|
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
231
|
+
One busy Apple silicon Mac. Meter: 1,000 documents in 2.5035 seconds (399.4 per
|
|
232
|
+
second), 15,201 words in 0.4301 seconds, worst stress case 3.2518 seconds. Reading
|
|
233
|
+
pass, untimed on record until now: 0.8304 seconds for the same 1,000 (1,204.2 per
|
|
234
|
+
second), 0.1359 for the same large document, linear to 96,000 words. Learning pass,
|
|
235
|
+
8,000 words: 0.2027 seconds. Across 24 interleaved runs against 2.7.7 we measured
|
|
236
|
+
0.08% lower median throughput; that is machine noise, not a speed claim. Editing
|
|
237
|
+
time is excluded.
|
|
238
|
+
|
|
239
|
+
### Reading-pass accuracy
|
|
240
|
+
|
|
241
|
+
The reading pass budgets antithesis pairs by frequency, so the count has to be right
|
|
242
|
+
before the budget means anything. It had never been measured. On 58 labelled pairs in
|
|
243
|
+
[`bench/antithesis/`](bench/antithesis/):
|
|
244
|
+
|
|
245
|
+
| Reading pass | 2.8.0 | now |
|
|
246
|
+
|---|---:|---:|
|
|
247
|
+
| Recall, all shapes | 40.0% | 90.0% |
|
|
248
|
+
| Recall, shapes in reach | 44.4% | 100% |
|
|
249
|
+
| Precision | 80.0% | 100% |
|
|
250
|
+
| False positives | 3 | 0 |
|
|
251
|
+
|
|
252
|
+
|
|
253
|
+
|
|
254
|
+
Bare subject swap and the weak isocolon stay out of reach and count against recall: the
|
|
255
|
+
first is a judgment call in `references/tells.md`, the second is identical to ordinary
|
|
256
|
+
parallel prose on every lexical statistic. Maintainer labels on constructed pairs, so
|
|
257
|
+
this is a regression floor, not field accuracy.
|
|
238
258
|
|
|
239
259
|
### Current models
|
|
240
260
|
|
package/SKILL.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: zero-slop
|
|
3
3
|
license: MIT
|
|
4
4
|
metadata:
|
|
5
|
-
version: "2.8.
|
|
5
|
+
version: "2.8.2"
|
|
6
6
|
author: manavmishra
|
|
7
7
|
description: Turn drafts into sharp, natural prose or inspect them without rewriting. Zero Slop runs inside the user's existing AI assistant; Claude, GPT, or another compatible model reads and edits in context while local tools point to exact phrases and protect the source. Use when the user asks to humanize or de-slop writing, inspect AI-sounding patterns, fix text that reads like ChatGPT, polish outward-facing prose, draft social or LinkedIn content, or apply a final quality check to prose the agent generated. The workflow preserves facts, voice, and format and learns privately from repeated, reason-labelled human edits.
|
|
8
8
|
---
|
|
@@ -1,142 +1,155 @@
|
|
|
1
1
|
{
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
},
|
|
17
|
-
{
|
|
18
|
-
"file": "monument-negation.md",
|
|
19
|
-
"expect": [
|
|
20
|
-
{
|
|
21
|
-
"metric": "monument_verb",
|
|
22
|
-
"span": "stands as a testament"
|
|
23
|
-
},
|
|
24
|
-
{
|
|
25
|
-
"metric": "negation_triad",
|
|
26
|
-
"span": "no runbook, no owner, no rollback"
|
|
2
|
+
"_comment": "The ratchet: every miss an audit or a competitor catches becomes a fixture here, in the same change. metric entries are verified by register.py --recall; check entries name the eval.md family a reading pass must catch, and --recall verifies the span exists and the family is a real check.",
|
|
3
|
+
"fixtures": [
|
|
4
|
+
{
|
|
5
|
+
"file": "adjective-inflation.md",
|
|
6
|
+
"expect": [
|
|
7
|
+
{
|
|
8
|
+
"metric": "adjective_inflation",
|
|
9
|
+
"span": "a real improvement"
|
|
10
|
+
},
|
|
11
|
+
{
|
|
12
|
+
"metric": "adjective_inflation",
|
|
13
|
+
"span": "a genuine breakthrough"
|
|
14
|
+
}
|
|
15
|
+
]
|
|
27
16
|
},
|
|
28
17
|
{
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
18
|
+
"file": "monument-negation.md",
|
|
19
|
+
"expect": [
|
|
20
|
+
{
|
|
21
|
+
"metric": "monument_verb",
|
|
22
|
+
"span": "stands as a testament"
|
|
23
|
+
},
|
|
24
|
+
{
|
|
25
|
+
"metric": "negation_triad",
|
|
26
|
+
"span": "no runbook, no owner, no rollback"
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"metric": "negation_triad",
|
|
30
|
+
"span": "Not reviewed, not staged, not announced"
|
|
31
|
+
}
|
|
32
|
+
]
|
|
40
33
|
},
|
|
41
34
|
{
|
|
42
|
-
|
|
43
|
-
|
|
35
|
+
"file": "dangling-thin.md",
|
|
36
|
+
"expect": [
|
|
37
|
+
{
|
|
38
|
+
"metric": "thin_section",
|
|
39
|
+
"span": "Install"
|
|
40
|
+
},
|
|
41
|
+
{
|
|
42
|
+
"metric": "thin_section",
|
|
43
|
+
"span": "Configure"
|
|
44
|
+
},
|
|
45
|
+
{
|
|
46
|
+
"metric": "dangling_pointer",
|
|
47
|
+
"span": "download the installer"
|
|
48
|
+
}
|
|
49
|
+
]
|
|
44
50
|
},
|
|
45
51
|
{
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
},
|
|
58
|
-
{
|
|
59
|
-
"check": "Synonym cycling",
|
|
60
|
-
"span": "the local checker"
|
|
61
|
-
}
|
|
62
|
-
]
|
|
63
|
-
},
|
|
64
|
-
{
|
|
65
|
-
"file": "mic-drop.md",
|
|
66
|
-
"expect": [
|
|
67
|
-
{
|
|
68
|
-
"metric": "verbless_fragment",
|
|
69
|
-
"span": "The review queue."
|
|
70
|
-
},
|
|
71
|
-
{
|
|
72
|
-
"check": "Antithesis pairs",
|
|
73
|
-
"span": "Different teams, different fixes, same bottleneck."
|
|
74
|
-
}
|
|
75
|
-
]
|
|
76
|
-
},
|
|
77
|
-
{
|
|
78
|
-
"file": "caveat-template.md",
|
|
79
|
-
"expect": [
|
|
80
|
-
{
|
|
81
|
-
"check": "Robotic symmetry",
|
|
82
|
-
"span": "Neither figure includes bot traffic"
|
|
83
|
-
}
|
|
84
|
-
]
|
|
85
|
-
},
|
|
86
|
-
{
|
|
87
|
-
"file": "label-period-list.md",
|
|
88
|
-
"expect": [
|
|
89
|
-
{
|
|
90
|
-
"check": "List-label periods and bare-noun bullets",
|
|
91
|
-
"span": "**Latency.**"
|
|
92
|
-
}
|
|
93
|
-
]
|
|
94
|
-
},
|
|
95
|
-
{
|
|
96
|
-
"file": "unnamed-validator.md",
|
|
97
|
-
"expect": [
|
|
98
|
-
{
|
|
99
|
-
"check": "Weasel attribution",
|
|
100
|
-
"span": "An external audit"
|
|
101
|
-
},
|
|
102
|
-
{
|
|
103
|
-
"check": "Weasel attribution",
|
|
104
|
-
"span": "A\nthird-party checker"
|
|
105
|
-
}
|
|
106
|
-
]
|
|
107
|
-
},
|
|
108
|
-
{
|
|
109
|
-
"file": "classifier-scaffolding.md",
|
|
110
|
-
"expect": [
|
|
111
|
-
{
|
|
112
|
-
"metric": "classifier_scaffolding",
|
|
113
|
-
"span": "The economics are simple"
|
|
52
|
+
"file": "referent-cycling.md",
|
|
53
|
+
"expect": [
|
|
54
|
+
{
|
|
55
|
+
"metric": "referent_cluster",
|
|
56
|
+
"span": "local tools"
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
"check": "Synonym cycling",
|
|
60
|
+
"span": "the local checker"
|
|
61
|
+
}
|
|
62
|
+
]
|
|
114
63
|
},
|
|
115
64
|
{
|
|
116
|
-
|
|
117
|
-
|
|
65
|
+
"file": "mic-drop.md",
|
|
66
|
+
"expect": [
|
|
67
|
+
{
|
|
68
|
+
"metric": "verbless_fragment",
|
|
69
|
+
"span": "The review queue."
|
|
70
|
+
},
|
|
71
|
+
{
|
|
72
|
+
"check": "Antithesis pairs",
|
|
73
|
+
"span": "Different teams, different fixes, same bottleneck."
|
|
74
|
+
}
|
|
75
|
+
]
|
|
118
76
|
},
|
|
119
77
|
{
|
|
120
|
-
|
|
121
|
-
|
|
78
|
+
"file": "caveat-template.md",
|
|
79
|
+
"expect": [
|
|
80
|
+
{
|
|
81
|
+
"check": "Robotic symmetry",
|
|
82
|
+
"span": "Neither figure includes bot traffic"
|
|
83
|
+
}
|
|
84
|
+
]
|
|
122
85
|
},
|
|
123
86
|
{
|
|
124
|
-
|
|
125
|
-
|
|
87
|
+
"file": "label-period-list.md",
|
|
88
|
+
"expect": [
|
|
89
|
+
{
|
|
90
|
+
"check": "List-label periods and bare-noun bullets",
|
|
91
|
+
"span": "**Latency.**"
|
|
92
|
+
}
|
|
93
|
+
]
|
|
126
94
|
},
|
|
127
95
|
{
|
|
128
|
-
|
|
129
|
-
|
|
96
|
+
"file": "unnamed-validator.md",
|
|
97
|
+
"expect": [
|
|
98
|
+
{
|
|
99
|
+
"check": "Weasel attribution",
|
|
100
|
+
"span": "An external audit"
|
|
101
|
+
},
|
|
102
|
+
{
|
|
103
|
+
"check": "Weasel attribution",
|
|
104
|
+
"span": "A\nthird-party checker"
|
|
105
|
+
}
|
|
106
|
+
]
|
|
130
107
|
},
|
|
131
108
|
{
|
|
132
|
-
|
|
133
|
-
|
|
109
|
+
"file": "classifier-scaffolding.md",
|
|
110
|
+
"expect": [
|
|
111
|
+
{
|
|
112
|
+
"metric": "classifier_scaffolding",
|
|
113
|
+
"span": "The economics are simple"
|
|
114
|
+
},
|
|
115
|
+
{
|
|
116
|
+
"metric": "classifier_scaffolding",
|
|
117
|
+
"span": "The dangerous part is"
|
|
118
|
+
},
|
|
119
|
+
{
|
|
120
|
+
"metric": "classifier_scaffolding",
|
|
121
|
+
"span": "has a simple cause"
|
|
122
|
+
},
|
|
123
|
+
{
|
|
124
|
+
"metric": "classifier_scaffolding",
|
|
125
|
+
"span": "limits worth stating"
|
|
126
|
+
},
|
|
127
|
+
{
|
|
128
|
+
"metric": "classifier_scaffolding",
|
|
129
|
+
"span": "One honest caveat"
|
|
130
|
+
},
|
|
131
|
+
{
|
|
132
|
+
"metric": "classifier_scaffolding",
|
|
133
|
+
"span": "the ones that matter"
|
|
134
|
+
},
|
|
135
|
+
{
|
|
136
|
+
"check": "Interpretive metadiscourse",
|
|
137
|
+
"span": "It counts what it can see, and the rest is the reader's judgment."
|
|
138
|
+
}
|
|
139
|
+
]
|
|
134
140
|
},
|
|
135
141
|
{
|
|
136
|
-
|
|
137
|
-
|
|
142
|
+
"file": "plural-noun-fragment.md",
|
|
143
|
+
"expect": [
|
|
144
|
+
{
|
|
145
|
+
"metric": "verbless_fragment",
|
|
146
|
+
"span": "Same compound, three identifiers."
|
|
147
|
+
},
|
|
148
|
+
{
|
|
149
|
+
"metric": "verbless_fragment",
|
|
150
|
+
"span": "Same assay, two units."
|
|
151
|
+
}
|
|
152
|
+
]
|
|
138
153
|
}
|
|
139
|
-
|
|
140
|
-
}
|
|
141
|
-
]
|
|
154
|
+
]
|
|
142
155
|
}
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
Every dataset you add widens the space of things that look relevant. Same
|
|
2
|
+
compound, three identifiers. Same assay, two units. Two sources that disagree
|
|
3
|
+
and no rule for which one wins. The model stays exactly as smart. The retrieval
|
|
4
|
+
gets ambiguous, and fluent ambiguity reads as confidence.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "zero-slop",
|
|
3
|
-
"version": "2.8.
|
|
3
|
+
"version": "2.8.2",
|
|
4
4
|
"description": "An Agent Skill that scores AI-sounding prose locally, guides the host AI through an editorial rewrite, and checks changed source details. The local tools run offline with zero dependencies.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"zero-slop": "bin/zero-slop.mjs"
|
package/scripts/register.py
CHANGED
|
@@ -81,22 +81,41 @@ SHORT_FLOORS = {
|
|
|
81
81
|
ANTITHESIS_STOP = frozenset("""
|
|
82
82
|
a an the this that these those it its is are was were be been being am do does did
|
|
83
83
|
to of in on at by for with from as and or but so if then than not no nor yet
|
|
84
|
-
we you they he she i us our your their his her them me my
|
|
84
|
+
we you they he she i us our your their his her them me my there here
|
|
85
85
|
one two three first second next last own same very just only also more most less
|
|
86
|
+
every all each any some many few way thing
|
|
86
87
|
""".split())
|
|
87
88
|
|
|
88
|
-
#
|
|
89
|
-
#
|
|
89
|
+
# Negation, including the contracted forms. "isn't", "don't" and "won't" carry
|
|
90
|
+
# the figure exactly as "is not" does, and the shipped detector matched none of
|
|
91
|
+
# them: "Slop isn't a vibe. It's measurable." is a documented anchor in
|
|
92
|
+
# references/tells.md and it walked straight past.
|
|
93
|
+
NEGATION = re.compile(r"\b(?:not|never|cannot)\b|\w+n['’]t\b", re.I)
|
|
94
|
+
# A copula in both halves is the "X is A. Y is B." frame, which is the marked
|
|
95
|
+
# figure's usual carrier and is absent from ordinary negated prose ("The server
|
|
96
|
+
# did not respond. We restarted it.").
|
|
97
|
+
COPULA = re.compile(r"\b(?:is|are|was|were|be|been|am)\b|\w+['’]s\b|\w+n['’]t\b", re.I)
|
|
90
98
|
RX_MARKED_OPEN = re.compile(r"^\W*not\b", re.I)
|
|
91
|
-
RX_MARKED_CLOSE = re.compile(
|
|
99
|
+
RX_MARKED_CLOSE = re.compile(
|
|
100
|
+
r"\b(?:was|were|is|are|did|does|do|has|have|had|will|can|could|would)\s+not\W*$", re.I)
|
|
101
|
+
# The two template shapes the meter already anchors (this-is-what-looks-like,
|
|
102
|
+
# no-x-had-to). The meter scores them as spans; the register pass has to COUNT
|
|
103
|
+
# them, because tells.md budgets the family by frequency and a span hit is not
|
|
104
|
+
# a count.
|
|
105
|
+
RX_CONTRASTIVE = re.compile(r"^\W*(?:but|yet|however)\b", re.I)
|
|
106
|
+
RX_STOCK_CLOSER = re.compile(r"\bthis is what\b.{0,60}?\blooks? like\b", re.I)
|
|
107
|
+
RX_UNMARKED_REVERSAL = re.compile(r"^\W*no\s+[\w-]+(?:\s+[\w-]+){0,3}\s+had to\b", re.I)
|
|
92
108
|
|
|
93
109
|
|
|
94
|
-
def
|
|
110
|
+
def _tokens(sent: str) -> list[str]:
|
|
111
|
+
return [w.lower() for w in re.findall(r"[A-Za-z][\w'’-]*", sent)]
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def _antithesis_content(tokens: list[str]) -> list[str]:
|
|
95
115
|
"""Content words, with a trailing -s folded away so a verb frame still
|
|
96
116
|
matches when only its agreement changed: 'let' and 'lets' are one frame."""
|
|
97
117
|
out = []
|
|
98
|
-
for word in
|
|
99
|
-
word = word.lower()
|
|
118
|
+
for word in tokens:
|
|
100
119
|
if word in ANTITHESIS_STOP:
|
|
101
120
|
continue
|
|
102
121
|
if len(word) > 3 and word.endswith("s") and not word.endswith("ss"):
|
|
@@ -105,35 +124,102 @@ def _antithesis_content(sent: str) -> list[str]:
|
|
|
105
124
|
return out
|
|
106
125
|
|
|
107
126
|
|
|
127
|
+
def _common_prefix(a: list[str], b: list[str]) -> int:
|
|
128
|
+
n = 0
|
|
129
|
+
for x, y in zip(a, b):
|
|
130
|
+
if x != y:
|
|
131
|
+
break
|
|
132
|
+
n += 1
|
|
133
|
+
return n
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
# A heading or a list lead-in carries no terminal punctuation, so it arrives
|
|
137
|
+
# from _sentences glued to the paragraph beneath it. Rather than change the
|
|
138
|
+
# splitter or prose_of -- both are shared by every family, and moving either one
|
|
139
|
+
# pushed four to eleven documents over budget on rate alone -- this family
|
|
140
|
+
# rejects the glued span itself. Nothing else sees the change.
|
|
141
|
+
RX_SCAFFOLD = re.compile(r"^\s*(?:[#>|]|[-*+]\s|\d+\.\s)|\*\*|\n\s*(?:[#>|]|[-*+]\s|\d+\.\s)")
|
|
142
|
+
|
|
143
|
+
|
|
108
144
|
def antithesis_pairs(prose: str) -> list[str]:
|
|
109
|
-
"""Adjacent balanced sentences
|
|
145
|
+
"""Adjacent balanced sentences where the second lands the twist.
|
|
146
|
+
|
|
147
|
+
Recall is bounded and the bound is a property of the figure, not of the
|
|
148
|
+
implementation. "A meter reports a number. A reader reports a feeling." and
|
|
149
|
+
"The report lists every vendor. The appendix lists every contract." are the
|
|
150
|
+
same construction to every lexical statistic -- same lengths, same one
|
|
151
|
+
shared word, same 0.33 overlap -- and only the first is antithesis. What
|
|
152
|
+
separates them is semantic opposition, which no word count can see. So the
|
|
153
|
+
marked shapes, where a negation anchors the figure, are matched broadly;
|
|
154
|
+
the unmarked ones are matched only where the parallel is strong enough to
|
|
155
|
+
be structural. Bare subject swap stays the reader's call, as tells.md says.
|
|
156
|
+
"""
|
|
110
157
|
out = []
|
|
111
158
|
sents = _sentences(prose)
|
|
112
|
-
for
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
159
|
+
toks = [_tokens(s) for s in sents]
|
|
160
|
+
cont = [_antithesis_content(t) for t in toks]
|
|
161
|
+
# Matches do not overlap. The figure is a pair, so a sentence that has
|
|
162
|
+
# already landed one twist cannot also be the setup for the next: three
|
|
163
|
+
# short consecutive sentences were producing two pairs out of one figure
|
|
164
|
+
# and inflating a rate the budget reads directly.
|
|
165
|
+
consumed = -1
|
|
166
|
+
for i in range(len(sents) - 1):
|
|
167
|
+
if i <= consumed:
|
|
117
168
|
continue
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
169
|
+
first, second = sents[i], sents[i + 1]
|
|
170
|
+
ta, tb = toks[i], toks[i + 1]
|
|
171
|
+
na, nb = len(ta), len(tb)
|
|
172
|
+
if not (2 <= na <= 14 and 1 <= nb <= 14):
|
|
121
173
|
continue
|
|
122
|
-
if
|
|
123
|
-
out.append(f"{first} {second}")
|
|
124
|
-
continue
|
|
125
|
-
head, tail = _antithesis_content(first), _antithesis_content(second)
|
|
126
|
-
if len(head) < 2 or len(tail) < 2:
|
|
127
|
-
continue
|
|
128
|
-
# Identical content is repetition, not antithesis: the figure needs the
|
|
129
|
-
# arguments to have changed.
|
|
130
|
-
if head == tail:
|
|
174
|
+
if RX_SCAFFOLD.search(first) or RX_SCAFFOLD.search(second):
|
|
131
175
|
continue
|
|
176
|
+
head, tail = cont[i], cont[i + 1]
|
|
132
177
|
shared = set(head) & set(tail)
|
|
133
|
-
|
|
134
|
-
|
|
178
|
+
prefix = _common_prefix(ta, tb)
|
|
179
|
+
negated = bool(NEGATION.search(first) or NEGATION.search(second))
|
|
180
|
+
|
|
181
|
+
# Marked, announced at the open or the close of the pair.
|
|
182
|
+
if RX_MARKED_OPEN.match(first) and nb <= 10:
|
|
183
|
+
out.append(f"{first} {second}"); consumed = i + 1; continue
|
|
184
|
+
if RX_MARKED_CLOSE.search(second) and na <= 10:
|
|
185
|
+
out.append(f"{first} {second}"); consumed = i + 1; continue
|
|
186
|
+
# Marked, carried inside the pair. A negation alone is ordinary prose,
|
|
187
|
+
# so the halves also have to be short AND share a frame. One shared
|
|
188
|
+
# topic word is not a frame: "The AI roles supply judgment. A generating
|
|
189
|
+
# role never certifies its own output." shares "role" and is ordinary
|
|
190
|
+
# prose. The frame is a repeated opening, a copula on both sides, or two
|
|
191
|
+
# words in common.
|
|
192
|
+
if negated and na <= 10 and nb <= 8 and (prefix >= 1 or len(shared) >= 2):
|
|
193
|
+
out.append(f"{first} {second}"); consumed = i + 1; continue
|
|
194
|
+
# A copula on both sides is the weakest of the three frames, so it only
|
|
195
|
+
# counts when the halves are staccato-short. "Passwords are never stored
|
|
196
|
+
# in plain text. They are hashed with a per-user salt." is two copulas
|
|
197
|
+
# and a negation and no figure at all.
|
|
198
|
+
if (negated and na <= 7 and nb <= 6
|
|
199
|
+
and COPULA.search(first) and COPULA.search(second)):
|
|
200
|
+
out.append(f"{first} {second}"); consumed = i + 1; continue
|
|
201
|
+
# A contrastive opener on the second half is the twist announced by a
|
|
202
|
+
# conjunction rather than by the negation's position: "That creates
|
|
203
|
+
# speed. But speed is not velocity."
|
|
204
|
+
if (negated and RX_CONTRASTIVE.match(second) and na <= 8 and nb <= 7
|
|
205
|
+
and (shared or COPULA.search(second))):
|
|
206
|
+
out.append(f"{first} {second}"); consumed = i + 1; continue
|
|
207
|
+
# The two template shapes the meter anchors, counted here.
|
|
208
|
+
if RX_STOCK_CLOSER.search(second) or RX_UNMARKED_REVERSAL.match(first):
|
|
209
|
+
out.append(f"{first} {second}"); consumed = i + 1; continue
|
|
210
|
+
# Isocolon: one frame, both arguments swapped. A repeated opening is
|
|
211
|
+
# restatement rather than a swap, but only when the repeated part
|
|
212
|
+
# carries meaning: "Version one shipped in March. Version two shipped in
|
|
213
|
+
# June." repeats the subject and is enumeration, while "A junior
|
|
214
|
+
# engineer reads the error. A senior engineer reads the stack trace."
|
|
215
|
+
# repeats only the article and is the figure. So the test is whether any
|
|
216
|
+
# CONTENT word opens both halves, not whether any token does.
|
|
217
|
+
shared_open = any(w not in ANTITHESIS_STOP for w in ta[:prefix])
|
|
218
|
+
if shared_open or head == tail or len(head) < 2 or len(tail) < 2:
|
|
219
|
+
continue
|
|
135
220
|
if len(shared) >= 2 and len(shared) / min(len(head), len(tail)) >= 0.5:
|
|
136
221
|
out.append(f"{first} {second}")
|
|
222
|
+
consumed = i + 1
|
|
137
223
|
return out
|
|
138
224
|
|
|
139
225
|
|
|
@@ -246,6 +332,74 @@ FINITE_VERB = re.compile(
|
|
|
246
332
|
r"|gate[sd]?|gives?|runs?|gets?|gave|gone)\b"
|
|
247
333
|
r"|\b\w+(?:s|ed|es)\b", re.I)
|
|
248
334
|
|
|
335
|
+
# The -s half of that catch-all cannot tell a verb from a plural noun, and a
|
|
336
|
+
# plural noun in a fragment made the whole fragment invisible: "Same compound,
|
|
337
|
+
# three identifiers." and "Same assay, two units." both read as verbed on
|
|
338
|
+
# `identifiers` and `units`. Since fragments are mostly noun phrases, and noun
|
|
339
|
+
# phrases are mostly plural, the detector was blind to its own commonest shape.
|
|
340
|
+
#
|
|
341
|
+
# A determiner, number or quantifier immediately before an -s word makes it a
|
|
342
|
+
# noun ("three identifiers", "no rules"), not a verb ("the model runs" keeps
|
|
343
|
+
# its verb because "model" is not in this list). Same closed-list device as
|
|
344
|
+
# IMPERATIVE_OPENER above, and measured the same way: on data/corpus/
|
|
345
|
+
# must-not-flag it adds no findings at all.
|
|
346
|
+
NOUN_MARKER = frozenset("""
|
|
347
|
+
a an the this that these those my your his her its our their no some any
|
|
348
|
+
many several few both all each every another other more most much little
|
|
349
|
+
two three four five six seven eight nine ten dozen hundred thousand
|
|
350
|
+
one first second third next last same own other
|
|
351
|
+
of at in on for by with from to into onto over under about across through
|
|
352
|
+
without within per via against between during after before
|
|
353
|
+
""".split())
|
|
354
|
+
# The prepositions are there for the same reason as the determiners: what
|
|
355
|
+
# follows one is a noun, not a verb. "In innovation at scale." has two words
|
|
356
|
+
# that are verbs elsewhere ("scale", and "innovation" is safe), and without
|
|
357
|
+
# this the fragment reads as a sentence. An infinitive after "to" is not a
|
|
358
|
+
# finite verb either, so listing it here is correct rather than convenient.
|
|
359
|
+
|
|
360
|
+
|
|
361
|
+
# The base forms IMPERATIVE_OPENER already knows, reused away from the sentence
|
|
362
|
+
# opening. A plural subject takes a bare verb -- "our engineers ship weekly" --
|
|
363
|
+
# which carries no inflection for the catch-all to find, so without this the
|
|
364
|
+
# fix above turns every such sentence into a fragment.
|
|
365
|
+
BASE_VERBS = frozenset(
|
|
366
|
+
IMPERATIVE_OPENER.pattern
|
|
367
|
+
.split("(?:add|", 1)[1]
|
|
368
|
+
.split(")\\b", 1)[0]
|
|
369
|
+
.replace("|", " ")
|
|
370
|
+
.split()
|
|
371
|
+
) | {"add"} | frozenset("""
|
|
372
|
+
work need mean seem feel matter differ vary exist remain happen occur tend
|
|
373
|
+
cost fail pass fit last agree apply depend belong arrive land stick
|
|
374
|
+
""".split())
|
|
375
|
+
# The extras are the stative and intransitive verbs a plural subject takes and
|
|
376
|
+
# the imperative list has no reason to carry -- you do not tell someone to
|
|
377
|
+
# "matter". "The tools work." was the fragment this produced without them. Each
|
|
378
|
+
# one is also a noun in some context ("the work", "the cost"), which the
|
|
379
|
+
# determiner test in _looks_like_noun already handles.
|
|
380
|
+
|
|
381
|
+
|
|
382
|
+
def _looks_like_noun(sent: str, start: int) -> bool:
|
|
383
|
+
"""Is the word at `start` sitting in a noun phrase rather than a verb slot?"""
|
|
384
|
+
prev = re.findall(r"[A-Za-z][\w'-]*", sent[:start])
|
|
385
|
+
return bool(prev) and prev[-1].lower() in NOUN_MARKER
|
|
386
|
+
|
|
387
|
+
|
|
388
|
+
def _has_finite_verb(sent: str) -> bool:
|
|
389
|
+
for match in FINITE_VERB.finditer(sent):
|
|
390
|
+
word = match.group(0)
|
|
391
|
+
# An explicit auxiliary or an -ed form is a verb wherever it appears.
|
|
392
|
+
if not re.fullmatch(r"\w+(?:s|es)", word, re.I):
|
|
393
|
+
return True
|
|
394
|
+
if not _looks_like_noun(sent, match.start()):
|
|
395
|
+
return True # "the model runs"
|
|
396
|
+
# "three identifiers" is a noun, "one report" is a noun, but "engineers
|
|
397
|
+
# ship" is a verb: the same determiner test decides both.
|
|
398
|
+
for match in re.finditer(r"\b[A-Za-z][\w'-]*\b", sent):
|
|
399
|
+
if match.group(0).lower() in BASE_VERBS and not _looks_like_noun(sent, match.start()):
|
|
400
|
+
return True
|
|
401
|
+
return False
|
|
402
|
+
|
|
249
403
|
|
|
250
404
|
def _sentences(prose: str) -> list[str]:
|
|
251
405
|
return [x.strip() for x in re.split(r"(?<=[.!?])\s+", prose) if x.strip()]
|
|
@@ -261,7 +415,7 @@ def verbless_fragments(prose: str) -> list[str]:
|
|
|
261
415
|
continue
|
|
262
416
|
if IMPERATIVE_OPENER.match(sent):
|
|
263
417
|
continue
|
|
264
|
-
if not
|
|
418
|
+
if not _has_finite_verb(sent):
|
|
265
419
|
out.append(sent)
|
|
266
420
|
return out
|
|
267
421
|
|