@izhimu/pi-caveman 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/README.md +36 -44
- package/extensions/caveman.ts +101 -149
- package/package.json +5 -1
- package/skills/caveman/README.md +52 -48
- package/skills/caveman/SKILL.md +3 -1
- package/skills/caveman-compress/README.md +42 -29
- package/skills/caveman-compress/scripts/compress.py +113 -22
- package/skills/caveman-compress/scripts/validate.py +193 -6
- package/skills/cavecrew/README.md +0 -61
- package/skills/cavecrew/SKILL.md +0 -82
- package/skills/caveman-commit/README.md +0 -44
- package/skills/caveman-commit/SKILL.md +0 -65
- package/skills/caveman-discover/SKILL.md +0 -118
- package/skills/caveman-evidence-review/SKILL.md +0 -144
- package/skills/caveman-explore/SKILL.md +0 -42
- package/skills/caveman-explore/package.json +0 -12
- package/skills/caveman-explore/tests/skill-file.test.mjs +0 -44
- package/skills/caveman-help/README.md +0 -38
- package/skills/caveman-help/SKILL.md +0 -63
- package/skills/caveman-learn/CLAUDE.md +0 -32
- package/skills/caveman-learn/README.md +0 -29
- package/skills/caveman-learn/SKILL.md +0 -69
- package/skills/caveman-learn/package.json +0 -12
- package/skills/caveman-learn/tests/skill-file.test.mjs +0 -44
- package/skills/caveman-manage/SKILL.md +0 -114
- package/skills/caveman-optimize/SKILL.md +0 -114
- package/skills/caveman-review/README.md +0 -33
- package/skills/caveman-review/SKILL.md +0 -55
- package/skills/caveman-setup/SKILL.md +0 -224
- package/skills/caveman-stats/README.md +0 -36
- package/skills/caveman-stats/SKILL.md +0 -12
- package/skills/investigate-first/SKILL.md +0 -16
- package/skills/investigate-first/agents/openai.yaml +0 -4
- package/skills/lean-build/SKILL.md +0 -18
- package/skills/lean-build/agents/openai.yaml +0 -4
- package/skills/migration/SKILL.md +0 -17
- package/skills/migration/agents/openai.yaml +0 -4
- package/skills/safe-refactor/SKILL.md +0 -16
- package/skills/safe-refactor/agents/openai.yaml +0 -4
- package/skills/surgical-patch/SKILL.md +0 -16
- package/skills/surgical-patch/agents/openai.yaml +0 -4
- package/skills/verify-and-stop/SKILL.md +0 -16
- package/skills/verify-and-stop/agents/openai.yaml +0 -4
|
@@ -10,9 +10,11 @@
|
|
|
10
10
|
|
|
11
11
|
---
|
|
12
12
|
|
|
13
|
-
A Claude Code skill that compresses
|
|
13
|
+
A Claude Code skill that compresses project memory files (`CLAUDE.md`, todos,
|
|
14
|
+
preferences) into caveman format, reducing repeated input size.
|
|
14
15
|
|
|
15
|
-
Claude
|
|
16
|
+
Claude loads `CLAUDE.md` on every session start, so large files add repeated
|
|
17
|
+
input tokens. Caveman shortens supported natural-language files.
|
|
16
18
|
|
|
17
19
|
## What It Do
|
|
18
20
|
|
|
@@ -21,11 +23,15 @@ Claude read `CLAUDE.md` on every session start. If file big, cost big. Caveman m
|
|
|
21
23
|
```
|
|
22
24
|
|
|
23
25
|
```
|
|
24
|
-
CLAUDE.md ← compressed (Claude reads
|
|
26
|
+
CLAUDE.md ← compressed (Claude reads smaller file each session)
|
|
25
27
|
CLAUDE.original.md ← human-readable backup (you edit this)
|
|
26
28
|
```
|
|
27
29
|
|
|
28
|
-
Original
|
|
30
|
+
Original remains in data directory rather than next to live file, so skill
|
|
31
|
+
auto-loaders do not read it twice. Path is
|
|
32
|
+
`$XDG_DATA_HOME/caveman-compress/backups/<parent-dir-name>/` on macOS and Linux,
|
|
33
|
+
or `%LOCALAPPDATA%\caveman-compress\backups\<parent-dir-name>\` on Windows. Edit
|
|
34
|
+
`.original.md` there, then run skill again to re-compress.
|
|
29
35
|
|
|
30
36
|
## Benchmarks
|
|
31
37
|
|
|
@@ -33,14 +39,15 @@ Real results on real project files:
|
|
|
33
39
|
|
|
34
40
|
| File | Original | Compressed | Saved |
|
|
35
41
|
|------|----------:|----------:|------:|
|
|
36
|
-
| `claude-md-preferences.md` | 706 | 285 |
|
|
37
|
-
| `project-notes.md` | 1145 | 535 |
|
|
38
|
-
| `claude-md-project.md` | 1122 | 636 |
|
|
39
|
-
| `todo-list.md` | 627 | 388 |
|
|
40
|
-
| `mixed-with-code.md` | 888 | 560 |
|
|
41
|
-
|
|
|
42
|
+
| `claude-md-preferences.md` | 706 | 285 | 59.6% |
|
|
43
|
+
| `project-notes.md` | 1145 | 535 | 53.3% |
|
|
44
|
+
| `claude-md-project.md` | 1122 | 636 | 43.3% |
|
|
45
|
+
| `todo-list.md` | 627 | 388 | 38.1% |
|
|
46
|
+
| `mixed-with-code.md` | 888 | 560 | 36.9% |
|
|
47
|
+
| Average | 898 | 481 | 46% |
|
|
42
48
|
|
|
43
|
-
All validations passed
|
|
49
|
+
All fixture validations passed: headings, code blocks, URLs, and file paths were
|
|
50
|
+
preserved exactly.
|
|
44
51
|
|
|
45
52
|
## Before / After
|
|
46
53
|
|
|
@@ -48,7 +55,7 @@ All validations passed ✅ — headings, code blocks, URLs, file paths preserved
|
|
|
48
55
|
<tr>
|
|
49
56
|
<td width="50%">
|
|
50
57
|
|
|
51
|
-
###
|
|
58
|
+
### Original (706 tokens)
|
|
52
59
|
|
|
53
60
|
> "I strongly prefer TypeScript with strict mode enabled for all new code. Please don't use `any` type unless there's genuinely no way around it, and if you do, leave a comment explaining the reasoning. I find that taking the time to properly type things catches a lot of bugs before they ever make it to runtime."
|
|
54
61
|
|
|
@@ -57,17 +64,20 @@ All validations passed ✅ — headings, code blocks, URLs, file paths preserved
|
|
|
57
64
|
|
|
58
65
|
### <img src="../../docs/assets/dancing-rock.svg" width="20" height="20" alt="rock"/> Caveman (285 tokens)
|
|
59
66
|
|
|
60
|
-
> "Prefer TypeScript strict mode always. No `any` unless unavoidable
|
|
67
|
+
> "Prefer TypeScript strict mode always. No `any` unless unavoidable; comment why if used. Proper types catch bugs early."
|
|
61
68
|
|
|
62
69
|
</td>
|
|
63
70
|
</tr>
|
|
64
71
|
</table>
|
|
65
72
|
|
|
66
|
-
|
|
73
|
+
This fixture produced 59.6% fewer counted tokens. Structural validation passed;
|
|
74
|
+
result does not prove semantic equivalence on other files or models.
|
|
67
75
|
|
|
68
76
|
## Security
|
|
69
77
|
|
|
70
|
-
`caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O
|
|
78
|
+
`caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O
|
|
79
|
+
patterns detected by static analysis. See [SECURITY.md](./SECURITY.md) for why
|
|
80
|
+
these operations exist and how paths are constrained.
|
|
71
81
|
|
|
72
82
|
## Install
|
|
73
83
|
|
|
@@ -76,10 +86,10 @@ Compress is built in with the `caveman` plugin. Install `caveman` once, then use
|
|
|
76
86
|
If you need local files, the compress skill lives at:
|
|
77
87
|
|
|
78
88
|
```bash
|
|
79
|
-
caveman-compress/
|
|
89
|
+
skills/caveman-compress/
|
|
80
90
|
```
|
|
81
91
|
|
|
82
|
-
|
|
92
|
+
Requires Python 3.10 or newer.
|
|
83
93
|
|
|
84
94
|
## Usage
|
|
85
95
|
|
|
@@ -98,8 +108,8 @@ Examples:
|
|
|
98
108
|
|
|
99
109
|
| Type | Compress? |
|
|
100
110
|
|------|-----------|
|
|
101
|
-
| `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` |
|
|
102
|
-
| Extensionless natural language |
|
|
111
|
+
| `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` | Yes |
|
|
112
|
+
| Extensionless natural language | Yes |
|
|
103
113
|
| `.py`, `.js`, `.ts`, `.json`, `.yaml` | ❌ Skip (code/config) |
|
|
104
114
|
| `*.original.md` | ❌ Skip (backup files) |
|
|
105
115
|
|
|
@@ -110,13 +120,13 @@ Examples:
|
|
|
110
120
|
↓
|
|
111
121
|
detect file type (no tokens)
|
|
112
122
|
↓
|
|
113
|
-
Claude compresses (tokens
|
|
123
|
+
Claude compresses (tokens: one call)
|
|
114
124
|
↓
|
|
115
125
|
validate output (no tokens)
|
|
116
126
|
checks: headings, code blocks, URLs, file paths, bullets
|
|
117
127
|
↓
|
|
118
|
-
if errors: Claude fixes cherry-picked issues only (tokens
|
|
119
|
-
does NOT recompress
|
|
128
|
+
if errors: Claude fixes cherry-picked issues only (tokens: targeted fix)
|
|
129
|
+
does NOT recompress; only patches broken parts
|
|
120
130
|
↓
|
|
121
131
|
retry up to 2 times
|
|
122
132
|
↓
|
|
@@ -142,22 +152,25 @@ Caveman compress natural language. It never touch:
|
|
|
142
152
|
|
|
143
153
|
## Why This Matter
|
|
144
154
|
|
|
145
|
-
`CLAUDE.md` loads on
|
|
155
|
+
`CLAUDE.md` loads on every session start. A 1,000-token project memory file adds
|
|
156
|
+
1,000 input tokens each time project opens, or 100,000 across 100 sessions.
|
|
146
157
|
|
|
147
|
-
Caveman
|
|
158
|
+
Caveman reduced counted tokens by about 46% on five listed fixtures. Validators
|
|
159
|
+
confirmed headings, code blocks, URLs, and file paths. They did not establish
|
|
160
|
+
general semantic or task-quality equivalence.
|
|
148
161
|
|
|
149
162
|
```
|
|
150
163
|
┌────────────────────────────────────────────┐
|
|
151
164
|
│ TOKEN SAVINGS PER FILE █████ 46% │
|
|
152
|
-
│
|
|
153
|
-
│
|
|
165
|
+
│ FIXTURES IN TABLE 5 │
|
|
166
|
+
│ STRUCTURAL VALIDATION passed on all │
|
|
154
167
|
│ SETUP TIME █ 1x │
|
|
155
168
|
└────────────────────────────────────────────┘
|
|
156
169
|
```
|
|
157
170
|
|
|
158
171
|
## Part of Caveman
|
|
159
172
|
|
|
160
|
-
This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit
|
|
173
|
+
This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit.
|
|
161
174
|
|
|
162
|
-
-
|
|
163
|
-
-
|
|
175
|
+
- `caveman`: ask Claude to answer in shorter prose
|
|
176
|
+
- `caveman-compress`: shorten supported project-memory files with backups and validation
|
|
@@ -16,9 +16,16 @@ import tempfile
|
|
|
16
16
|
from pathlib import Path
|
|
17
17
|
from typing import List
|
|
18
18
|
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
)
|
|
19
|
+
# Windows consoles default to cp1252, which cannot encode the emoji glyphs in
|
|
20
|
+
# our status lines; replace unencodable characters instead of crashing.
|
|
21
|
+
for _stream in (sys.stdout, sys.stderr):
|
|
22
|
+
try:
|
|
23
|
+
_stream.reconfigure(errors="replace")
|
|
24
|
+
except Exception:
|
|
25
|
+
pass
|
|
26
|
+
|
|
27
|
+
# A fence marker at the start of a line, at CommonMark's 0-3 space indent.
|
|
28
|
+
FENCE_LINE_REGEX = re.compile(r"^\s{0,3}(`{3,}|~{3,})")
|
|
22
29
|
|
|
23
30
|
# YAML frontmatter: starts at file start with --- on its own line, ends with --- on its own line.
|
|
24
31
|
# Captures the entire block (including delimiters and trailing newline) and the body after.
|
|
@@ -106,14 +113,44 @@ def is_sensitive_path(filepath: Path) -> bool:
|
|
|
106
113
|
|
|
107
114
|
|
|
108
115
|
def strip_llm_wrapper(text: str) -> str:
|
|
109
|
-
"""Strip outer ```markdown ... ``` fence when it wraps the
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
116
|
+
r"""Strip an outer ```markdown ... ``` fence when it wraps the ENTIRE output.
|
|
117
|
+
|
|
118
|
+
The wrapper is only real when the first and last fence lines are the SAME
|
|
119
|
+
block. The old regex (``\A\s*(fence)[^\n]*\n(.*)\n\1\s*\Z`` with DOTALL and
|
|
120
|
+
a greedy ``.*``) never checked that: it matched any document that merely
|
|
121
|
+
STARTS and ENDS with a fence line. An ordinary README section —
|
|
122
|
+
```bash npm install``` , prose, ```bash npm test``` — came back with its
|
|
123
|
+
first and last fence markers deleted and its two code blocks merged into
|
|
124
|
+
prose, so validation failed on both the compress and the fix path and the
|
|
125
|
+
section was permanently uncompressible after three paid API calls.
|
|
126
|
+
"""
|
|
127
|
+
lines = text.split("\n")
|
|
128
|
+
first, last = 0, len(lines) - 1
|
|
129
|
+
while first < len(lines) and not lines[first].strip():
|
|
130
|
+
first += 1
|
|
131
|
+
while last > first and not lines[last].strip():
|
|
132
|
+
last -= 1
|
|
133
|
+
if first >= last:
|
|
134
|
+
return text
|
|
135
|
+
opener = FENCE_LINE_REGEX.match(lines[first])
|
|
136
|
+
closer = FENCE_LINE_REGEX.match(lines[last])
|
|
137
|
+
if not opener or not closer:
|
|
138
|
+
return text
|
|
139
|
+
marker = opener.group(1)
|
|
140
|
+
# Closing fence: same character, at least as long, and nothing else on the line.
|
|
141
|
+
if closer.group(1)[0] != marker[0] or len(closer.group(1)) < len(marker):
|
|
142
|
+
return text
|
|
143
|
+
if lines[last].strip() != closer.group(1):
|
|
144
|
+
return text
|
|
145
|
+
# Any fence of the same kind in between means these two are not one block.
|
|
146
|
+
for line in lines[first + 1:last]:
|
|
147
|
+
inner = FENCE_LINE_REGEX.match(line)
|
|
148
|
+
if inner and inner.group(1)[0] == marker[0] and len(inner.group(1)) >= len(marker):
|
|
149
|
+
return text
|
|
150
|
+
return "\n".join(lines[first + 1:last])
|
|
151
|
+
|
|
152
|
+
|
|
153
|
+
def write_text_atomic(path: Path, text: str, newline: str = "\n") -> None:
|
|
117
154
|
"""Write ``text`` to ``path`` atomically as UTF-8.
|
|
118
155
|
|
|
119
156
|
Path.write_text() truncates the destination before encoding the string —
|
|
@@ -122,8 +159,22 @@ def write_text_atomic(path: Path, text: str) -> None:
|
|
|
122
159
|
first, write the bytes to a sibling temp file, fsync, then os.replace()
|
|
123
160
|
so the destination only ever moves from one complete, valid file to
|
|
124
161
|
another. Preserves the original file's permission bits across the swap.
|
|
162
|
+
|
|
163
|
+
``newline`` is the line terminator to emit. Callers pass the terminator
|
|
164
|
+
read_source() found in the source file so a CRLF document stays CRLF —
|
|
165
|
+
text-mode writes translating LF to the platform default rewrote every
|
|
166
|
+
line ending in every file the tool touched (issue #762), and reading the
|
|
167
|
+
bytes ourselves means nothing translates them back.
|
|
125
168
|
"""
|
|
126
|
-
|
|
169
|
+
if newline != "\n":
|
|
170
|
+
# Normalise first: model output can already carry CRLF, and a bare
|
|
171
|
+
# "\n" -> "\r\n" replace would turn those into "\r\r\n".
|
|
172
|
+
text = text.replace("\r\n", "\n").replace("\n", newline)
|
|
173
|
+
write_bytes_atomic(path, text.encode("utf-8"))
|
|
174
|
+
|
|
175
|
+
|
|
176
|
+
def write_bytes_atomic(path: Path, data: bytes) -> None:
|
|
177
|
+
"""Write ``data`` to ``path`` atomically, preserving permission bits."""
|
|
127
178
|
fd, tmp_name = tempfile.mkstemp(
|
|
128
179
|
dir=str(path.parent), prefix=path.name + ".", suffix=".tmp"
|
|
129
180
|
)
|
|
@@ -144,6 +195,40 @@ def write_text_atomic(path: Path, text: str) -> None:
|
|
|
144
195
|
raise
|
|
145
196
|
|
|
146
197
|
|
|
198
|
+
def read_source(filepath: Path) -> tuple[str, str, bytes]:
|
|
199
|
+
"""Read a source file as UTF-8, returning (text, line_terminator, raw_bytes).
|
|
200
|
+
|
|
201
|
+
Decodes strictly. The old errors="ignore" silently DROPPED every byte that
|
|
202
|
+
was not valid UTF-8 — a cp1252-authored file holding `\xe9` for "e-acute"
|
|
203
|
+
lost that byte, the mangled text was what got written to the backup, the
|
|
204
|
+
backup readback compared mangled-to-mangled so verification passed, and
|
|
205
|
+
then the original was overwritten. The bytes were unrecoverable and
|
|
206
|
+
nothing reported a problem (the destructive form of issue #686). A file we
|
|
207
|
+
cannot read exactly is a file we must not rewrite.
|
|
208
|
+
|
|
209
|
+
Line endings are detected from the raw bytes and returned to the caller
|
|
210
|
+
rather than being universal-newline'd away, so write_text_atomic can put
|
|
211
|
+
back what was there (issue #762). A mixed-ending file takes the terminator
|
|
212
|
+
the majority of its lines use — presence of one CRLF is not a mandate to
|
|
213
|
+
rewrite every LF in the document. The raw bytes come back too, so the
|
|
214
|
+
backup can be a byte-for-byte copy rather than a re-rendering.
|
|
215
|
+
"""
|
|
216
|
+
raw = filepath.read_bytes()
|
|
217
|
+
try:
|
|
218
|
+
text = raw.decode("utf-8")
|
|
219
|
+
except UnicodeDecodeError as e:
|
|
220
|
+
raise ValueError(
|
|
221
|
+
f"Refusing to compress {filepath}: not valid UTF-8 "
|
|
222
|
+
f"(byte 0x{raw[e.start]:02x} at offset {e.start}). "
|
|
223
|
+
"Compression rewrites the file in place, and any byte this tool "
|
|
224
|
+
"cannot decode would be destroyed by the round trip. "
|
|
225
|
+
"Convert the file to UTF-8 first."
|
|
226
|
+
) from None
|
|
227
|
+
crlf = text.count("\r\n")
|
|
228
|
+
newline = "\r\n" if crlf * 2 > text.count("\n") else "\n"
|
|
229
|
+
return text.replace("\r\n", "\n").replace("\r", "\n"), newline, raw
|
|
230
|
+
|
|
231
|
+
|
|
147
232
|
def first_nonblank_line(text: str) -> str:
|
|
148
233
|
"""Return the first non-blank line, stripped — used to detect a prose
|
|
149
234
|
preamble smuggled in ahead of the real content (issue #588)."""
|
|
@@ -153,13 +238,19 @@ def first_nonblank_line(text: str) -> str:
|
|
|
153
238
|
return ""
|
|
154
239
|
|
|
155
240
|
|
|
156
|
-
def _write_target(filepath: Path, text: str, backup_path: Path) -> None:
|
|
241
|
+
def _write_target(filepath: Path, text: str | bytes, backup_path: Path, newline: str = "\n") -> None:
|
|
157
242
|
"""Write to the target file, surfacing the backup location if the write
|
|
158
243
|
itself fails. write_text_atomic already leaves the target untouched on
|
|
159
244
|
failure, but the caller still needs to know where the pre-compression
|
|
160
|
-
original lives instead of being left to guess (issue #652).
|
|
245
|
+
original lives instead of being left to guess (issue #652).
|
|
246
|
+
|
|
247
|
+
``bytes`` restore the source verbatim; ``str`` is model output that still
|
|
248
|
+
has to be rendered with the document's line terminator."""
|
|
161
249
|
try:
|
|
162
|
-
|
|
250
|
+
if isinstance(text, bytes):
|
|
251
|
+
write_bytes_atomic(filepath, text)
|
|
252
|
+
else:
|
|
253
|
+
write_text_atomic(filepath, text, newline)
|
|
163
254
|
except Exception:
|
|
164
255
|
print(f"❌ Write to {filepath} failed. Original preserved at backup: {backup_path}")
|
|
165
256
|
raise
|
|
@@ -229,6 +320,7 @@ Compress this markdown into caveman format.
|
|
|
229
320
|
|
|
230
321
|
STRICT RULES:
|
|
231
322
|
- Do NOT modify anything inside ``` code blocks
|
|
323
|
+
- Do NOT modify anything inside a 4-space-indented code block either — those are code too, and they are validated
|
|
232
324
|
- Do NOT modify anything inside inline backticks
|
|
233
325
|
- Preserve ALL URLs exactly
|
|
234
326
|
- Preserve ALL headings exactly
|
|
@@ -301,7 +393,7 @@ def compress_file(filepath: Path) -> bool:
|
|
|
301
393
|
print("Skipping (not natural language)")
|
|
302
394
|
return False
|
|
303
395
|
|
|
304
|
-
original_text = filepath
|
|
396
|
+
original_text, newline, original_raw = read_source(filepath)
|
|
305
397
|
# Store backup outside the source directory so skill auto-loaders don't
|
|
306
398
|
# re-ingest the `.original.md` copy as a live file. Mirror the source's
|
|
307
399
|
# parent-dir name + stem under a platform-aware base to reduce collisions.
|
|
@@ -355,9 +447,8 @@ def compress_file(filepath: Path) -> bool:
|
|
|
355
447
|
# antivirus, disk full), unlink the bad backup and abort instead of
|
|
356
448
|
# leaving the user with a corrupt backup + compressed primary.
|
|
357
449
|
backup_dir.mkdir(parents=True, exist_ok=True)
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
if backup_readback != original_text:
|
|
450
|
+
write_bytes_atomic(backup_path, original_raw)
|
|
451
|
+
if backup_path.read_bytes() != original_raw:
|
|
361
452
|
print(f"❌ Backup write verification failed: {backup_path}")
|
|
362
453
|
print(" In-memory original differs from on-disk backup. Aborting before touching the input file.")
|
|
363
454
|
try:
|
|
@@ -365,7 +456,7 @@ def compress_file(filepath: Path) -> bool:
|
|
|
365
456
|
except OSError:
|
|
366
457
|
pass
|
|
367
458
|
return False
|
|
368
|
-
_write_target(filepath, compressed, backup_path)
|
|
459
|
+
_write_target(filepath, compressed, backup_path, newline)
|
|
369
460
|
|
|
370
461
|
# Step 2: Validate + Retry
|
|
371
462
|
for attempt in range(MAX_RETRIES):
|
|
@@ -383,7 +474,7 @@ def compress_file(filepath: Path) -> bool:
|
|
|
383
474
|
|
|
384
475
|
if attempt == MAX_RETRIES - 1:
|
|
385
476
|
# Restore original on failure
|
|
386
|
-
_write_target(filepath,
|
|
477
|
+
_write_target(filepath, original_raw, backup_path, newline)
|
|
387
478
|
backup_path.unlink(missing_ok=True)
|
|
388
479
|
print("❌ Failed after retries — original restored")
|
|
389
480
|
return False
|
|
@@ -409,6 +500,6 @@ def compress_file(filepath: Path) -> bool:
|
|
|
409
500
|
print(" Possible preamble leak. Skipping this attempt.")
|
|
410
501
|
continue
|
|
411
502
|
|
|
412
|
-
_write_target(filepath, compressed, backup_path)
|
|
503
|
+
_write_target(filepath, compressed, backup_path, newline)
|
|
413
504
|
|
|
414
505
|
return True
|
|
@@ -5,13 +5,40 @@ from pathlib import Path
|
|
|
5
5
|
|
|
6
6
|
URL_REGEX = re.compile(r"https?://[^\s)]+")
|
|
7
7
|
FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
|
|
8
|
+
|
|
9
|
+
# A line that is nothing but a fence marker plus an optional info string, at ANY
|
|
10
|
+
# indentation. Used ONLY to scrub leaked markers before inline-code pairing (see
|
|
11
|
+
# extract_inline_codes) — never for block extraction.
|
|
12
|
+
#
|
|
13
|
+
# Widening FENCE_OPEN_REGEX itself to `\s*` looks like the obvious fix for #820
|
|
14
|
+
# and is a net regression: a lone indented ``` (the natural way to SHOW a fence
|
|
15
|
+
# inside prose) then opens a block that runs to EOF, swallowing real code blocks
|
|
16
|
+
# and silently removing their inline spans from validation. That turns a
|
|
17
|
+
# false-failure bug into a false-PASS bug, and a false PASS overwrites the
|
|
18
|
+
# user's file with unvalidated output.
|
|
19
|
+
FENCE_MARKER_LINE_REGEX = re.compile(r"^\s*(?:`{3,}|~{3,})[^`~]*$")
|
|
20
|
+
|
|
21
|
+
# Cap on how much of a lost/added span is echoed in an error message. Unpaired
|
|
22
|
+
# backticks can make a "span" hundreds of characters of prose; printing it whole
|
|
23
|
+
# is what made #820's failures undiagnosable.
|
|
24
|
+
MAX_REPORTED_SPAN = 60
|
|
8
25
|
HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
|
|
9
26
|
BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
|
|
27
|
+
# Any list item, ordered or not. Four spaces inside a list item is the item's
|
|
28
|
+
# content indentation, never an indented code block.
|
|
29
|
+
LIST_ITEM_REGEX = re.compile(r"^\s*(?:[-*+]|\d+[.)])\s")
|
|
10
30
|
|
|
11
31
|
# crude but effective path detection
|
|
12
32
|
# Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
|
|
13
33
|
PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
|
|
14
34
|
|
|
35
|
+
# PATH_REGEX is crude on purpose and also matches ordinary prose pairs —
|
|
36
|
+
# "pros/cons", "Node/browser", "state/lifecycle". Caveman prose ADDS those
|
|
37
|
+
# constructions freely and dropping one breaks nothing, so only an unambiguous
|
|
38
|
+
# path — a leading ./ ../ / or drive letter, or a dotted filename in the last
|
|
39
|
+
# component — is treated as a hard loss.
|
|
40
|
+
DEFINITE_PATH_REGEX = re.compile(r"^(?:\./|\.\./|/|[A-Za-z]:\\)|[^/\\]*\.[A-Za-z0-9]{1,8}$")
|
|
41
|
+
|
|
15
42
|
|
|
16
43
|
class ValidationResult:
|
|
17
44
|
def __init__(self):
|
|
@@ -79,9 +106,108 @@ def extract_code_blocks(text):
|
|
|
79
106
|
blocks.append("\n".join(block_lines))
|
|
80
107
|
# Unclosed fences are silently skipped — they indicate malformed markdown
|
|
81
108
|
# and including them would cause false-positive validation failures.
|
|
109
|
+
return blocks + extract_indented_code_blocks(text)
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
def extract_indented_code_blocks(text):
|
|
113
|
+
"""CommonMark indented code blocks — 4-space-indented runs outside any fence.
|
|
114
|
+
|
|
115
|
+
Without these, ` kubectl delete pod --all -n prod` was prose to the
|
|
116
|
+
validator: "code blocks preserved exactly" compared empty to empty and
|
|
117
|
+
PASSED while the compressor rewrote the command to
|
|
118
|
+
`kubectl delete pod -n dev`. A clean pass on a mutated destructive command
|
|
119
|
+
is the worst failure this tool has, because it overwrites the user's file.
|
|
120
|
+
|
|
121
|
+
Deliberately conservative about lists: inside a list item, four spaces are
|
|
122
|
+
the item's content indentation, not code, and nested bullets are ordinary
|
|
123
|
+
prose the compressor SHOULD rewrite. A run is only treated as code when the
|
|
124
|
+
document is not inside a list and the run is preceded by a blank line — so
|
|
125
|
+
this adds detections, it never turns existing passes into false failures.
|
|
126
|
+
"""
|
|
127
|
+
blocks = []
|
|
128
|
+
lines = text.split("\n")
|
|
129
|
+
fenced = set()
|
|
130
|
+
for block in extract_fenced_spans(lines):
|
|
131
|
+
fenced.update(block)
|
|
132
|
+
in_list = False
|
|
133
|
+
previous_blank = True
|
|
134
|
+
i = 0
|
|
135
|
+
n = len(lines)
|
|
136
|
+
while i < n:
|
|
137
|
+
line = lines[i]
|
|
138
|
+
stripped = line.strip()
|
|
139
|
+
if i in fenced:
|
|
140
|
+
in_list, previous_blank = in_list, False
|
|
141
|
+
i += 1
|
|
142
|
+
continue
|
|
143
|
+
if not stripped:
|
|
144
|
+
previous_blank = True
|
|
145
|
+
i += 1
|
|
146
|
+
continue
|
|
147
|
+
indent = len(line) - len(line.lstrip(" \t"))
|
|
148
|
+
if LIST_ITEM_REGEX.match(line):
|
|
149
|
+
in_list = True
|
|
150
|
+
elif indent == 0:
|
|
151
|
+
in_list = False
|
|
152
|
+
if not in_list and previous_blank and indent >= 4:
|
|
153
|
+
run = []
|
|
154
|
+
while i < n and i not in fenced:
|
|
155
|
+
current = lines[i]
|
|
156
|
+
if not current.strip():
|
|
157
|
+
# A blank line continues an indented block only if more
|
|
158
|
+
# indented content follows.
|
|
159
|
+
lookahead = i + 1
|
|
160
|
+
while lookahead < n and not lines[lookahead].strip():
|
|
161
|
+
lookahead += 1
|
|
162
|
+
if lookahead < n and lookahead not in fenced and \
|
|
163
|
+
len(lines[lookahead]) - len(lines[lookahead].lstrip(" \t")) >= 4:
|
|
164
|
+
run.extend(lines[i:lookahead])
|
|
165
|
+
i = lookahead
|
|
166
|
+
continue
|
|
167
|
+
break
|
|
168
|
+
if len(current) - len(current.lstrip(" \t")) < 4:
|
|
169
|
+
break
|
|
170
|
+
run.append(current)
|
|
171
|
+
i += 1
|
|
172
|
+
if run:
|
|
173
|
+
blocks.append("\n".join(run))
|
|
174
|
+
previous_blank = False
|
|
175
|
+
continue
|
|
176
|
+
previous_blank = False
|
|
177
|
+
i += 1
|
|
82
178
|
return blocks
|
|
83
179
|
|
|
84
180
|
|
|
181
|
+
def extract_fenced_spans(lines):
|
|
182
|
+
"""Line-index ranges covered by fenced blocks, so indented-code detection
|
|
183
|
+
never reaches inside one."""
|
|
184
|
+
spans = []
|
|
185
|
+
i = 0
|
|
186
|
+
n = len(lines)
|
|
187
|
+
while i < n:
|
|
188
|
+
m = FENCE_OPEN_REGEX.match(lines[i])
|
|
189
|
+
if not m:
|
|
190
|
+
i += 1
|
|
191
|
+
continue
|
|
192
|
+
fence_char = m.group(2)[0]
|
|
193
|
+
fence_len = len(m.group(2))
|
|
194
|
+
start = i
|
|
195
|
+
i += 1
|
|
196
|
+
while i < n:
|
|
197
|
+
close_m = FENCE_OPEN_REGEX.match(lines[i])
|
|
198
|
+
if (
|
|
199
|
+
close_m
|
|
200
|
+
and close_m.group(2)[0] == fence_char
|
|
201
|
+
and len(close_m.group(2)) >= fence_len
|
|
202
|
+
and close_m.group(3).strip() == ""
|
|
203
|
+
):
|
|
204
|
+
i += 1
|
|
205
|
+
break
|
|
206
|
+
i += 1
|
|
207
|
+
spans.append(range(start, i))
|
|
208
|
+
return spans
|
|
209
|
+
|
|
210
|
+
|
|
85
211
|
def extract_urls(text):
|
|
86
212
|
return set(URL_REGEX.findall(text))
|
|
87
213
|
|
|
@@ -101,10 +227,28 @@ def extract_inline_codes(text):
|
|
|
101
227
|
fences indented 1-3 spaces (valid CommonMark). Reuse extract_code_blocks
|
|
102
228
|
(FENCE_OPEN_REGEX-based, indentation-aware) instead so an indented fence's
|
|
103
229
|
body backticks don't leak into inline-code pairing.
|
|
230
|
+
|
|
231
|
+
Any fence-marker line that survives that pass is then blanked (#820). A
|
|
232
|
+
fence indented 4+ spaces — what you get from showing an example inside a
|
|
233
|
+
bullet — is not matched by FENCE_OPEN_REGEX, so extract_code_blocks does
|
|
234
|
+
not remove it and its OWN backticks used to leak in and shift the pairing
|
|
235
|
+
of every following span, making the file permanently uncompressible.
|
|
236
|
+
Blanking just the marker lines fixes that without removing any prose, and
|
|
237
|
+
cannot run away the way a widened fence opener does.
|
|
238
|
+
|
|
239
|
+
The span pattern deliberately still spans newlines. CommonMark permits a
|
|
240
|
+
line ending inside a code span and hard-wrapped markdown produces them, so
|
|
241
|
+
a single-line pattern silently drops real spans — which downgrades a
|
|
242
|
+
deleted or mutated span from error to PASS. Long/garbled spans are a
|
|
243
|
+
presentation problem, handled by truncating in the error message instead.
|
|
104
244
|
"""
|
|
105
245
|
text_without_fences = text
|
|
106
246
|
for block in extract_code_blocks(text):
|
|
107
247
|
text_without_fences = text_without_fences.replace(block, "", 1)
|
|
248
|
+
text_without_fences = "\n".join(
|
|
249
|
+
"" if FENCE_MARKER_LINE_REGEX.match(line) else line
|
|
250
|
+
for line in text_without_fences.split("\n")
|
|
251
|
+
)
|
|
108
252
|
return re.findall(r"`([^`]+)`", text_without_fences)
|
|
109
253
|
|
|
110
254
|
|
|
@@ -115,11 +259,26 @@ def validate_headings(orig, comp, result):
|
|
|
115
259
|
h1 = extract_headings(orig)
|
|
116
260
|
h2 = extract_headings(comp)
|
|
117
261
|
|
|
262
|
+
# Changed heading TEXT is an error, not a warning. Every in-document anchor
|
|
263
|
+
# link points at a heading's slug, so renaming "# Configuration Options" to
|
|
264
|
+
# "# Config" silently breaks all of them — and SKILL.md and CLAUDE.md both
|
|
265
|
+
# state headings are preserved. Only counts used to gate the overwrite, so
|
|
266
|
+
# a run that renamed every heading reported "Validation passed". A level-only
|
|
267
|
+
# change keeps every slug intact and stays a warning.
|
|
118
268
|
if len(h1) != len(h2):
|
|
119
269
|
result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
|
|
270
|
+
return
|
|
120
271
|
|
|
121
|
-
|
|
122
|
-
|
|
272
|
+
t1 = [text for _, text in h1]
|
|
273
|
+
t2 = [text for _, text in h2]
|
|
274
|
+
if t1 != t2:
|
|
275
|
+
lost = [t for t in t1 if t not in t2]
|
|
276
|
+
added = [t for t in t2 if t not in t1]
|
|
277
|
+
result.add_error(f"Heading text/order changed: lost={lost}, added={added}")
|
|
278
|
+
elif h1 != h2:
|
|
279
|
+
# Same text, different level. The outline moved but no anchor broke —
|
|
280
|
+
# slugs come from the text, so links still resolve.
|
|
281
|
+
result.add_warning("Heading levels changed")
|
|
123
282
|
|
|
124
283
|
|
|
125
284
|
def validate_code_blocks(orig, comp, result):
|
|
@@ -139,11 +298,23 @@ def validate_urls(orig, comp, result):
|
|
|
139
298
|
|
|
140
299
|
|
|
141
300
|
def validate_paths(orig, comp, result):
|
|
301
|
+
"""File paths are preserved — an error, never a warning.
|
|
302
|
+
|
|
303
|
+
This validator never called add_error at all, so a compressed file that had
|
|
304
|
+
dropped `src/hooks/caveman-config.js` still reported "Validation passed" and
|
|
305
|
+
the in-place overwrite stood. SKILL.md and CLAUDE.md both promise paths
|
|
306
|
+
survive compression; nothing enforced it.
|
|
307
|
+
"""
|
|
142
308
|
p1 = extract_paths(orig)
|
|
143
309
|
p2 = extract_paths(comp)
|
|
310
|
+
lost = p1 - p2
|
|
311
|
+
added = p2 - p1
|
|
144
312
|
|
|
145
|
-
|
|
146
|
-
|
|
313
|
+
definite = {p for p in lost if DEFINITE_PATH_REGEX.search(p)}
|
|
314
|
+
if definite:
|
|
315
|
+
result.add_error(f"File paths lost: {sorted(definite)}")
|
|
316
|
+
if (lost - definite) or added:
|
|
317
|
+
result.add_warning(f"Path mismatch: lost={sorted(lost)}, added={sorted(added)}")
|
|
147
318
|
|
|
148
319
|
|
|
149
320
|
def validate_bullets(orig, comp, result):
|
|
@@ -160,6 +331,22 @@ def validate_bullets(orig, comp, result):
|
|
|
160
331
|
|
|
161
332
|
|
|
162
333
|
def validate_inline_codes(orig, comp, result):
|
|
334
|
+
def _render_spans(spans):
|
|
335
|
+
"""Render spans for an error message, truncated and newline-escaped.
|
|
336
|
+
|
|
337
|
+
A span may legitimately contain newlines, and an unpaired backtick can
|
|
338
|
+
make one hundreds of characters of prose. Printing those whole is what
|
|
339
|
+
made #820's failures undiagnosable — but the fix belongs here, in
|
|
340
|
+
presentation, not in what counts as a span.
|
|
341
|
+
"""
|
|
342
|
+
out = []
|
|
343
|
+
for span in sorted(spans):
|
|
344
|
+
flat = span.replace("\n", "\\n")
|
|
345
|
+
if len(flat) > MAX_REPORTED_SPAN:
|
|
346
|
+
flat = flat[:MAX_REPORTED_SPAN] + "…"
|
|
347
|
+
out.append(repr(flat))
|
|
348
|
+
return "{" + ", ".join(out) + "}"
|
|
349
|
+
|
|
163
350
|
c1 = Counter(extract_inline_codes(orig))
|
|
164
351
|
c2 = Counter(extract_inline_codes(comp))
|
|
165
352
|
|
|
@@ -170,9 +357,9 @@ def validate_inline_codes(orig, comp, result):
|
|
|
170
357
|
if code in c2 and c2[code] < count:
|
|
171
358
|
lost.add(f"{code} (lost {count - c2[code]} of {count} occurrences)")
|
|
172
359
|
if lost:
|
|
173
|
-
result.add_error(f"Inline code lost: {lost}")
|
|
360
|
+
result.add_error(f"Inline code lost: {_render_spans(lost)}")
|
|
174
361
|
if added:
|
|
175
|
-
result.add_warning(f"Inline code added: {added}")
|
|
362
|
+
result.add_warning(f"Inline code added: {_render_spans(added)}")
|
|
176
363
|
|
|
177
364
|
|
|
178
365
|
# ---------- Main ----------
|