@izhimu/pi-caveman 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/CHANGELOG.md +26 -0
  2. package/README.md +36 -44
  3. package/extensions/caveman.ts +101 -149
  4. package/package.json +5 -1
  5. package/skills/caveman/README.md +52 -48
  6. package/skills/caveman/SKILL.md +3 -1
  7. package/skills/caveman-compress/README.md +42 -29
  8. package/skills/caveman-compress/scripts/compress.py +113 -22
  9. package/skills/caveman-compress/scripts/validate.py +193 -6
  10. package/skills/cavecrew/README.md +0 -61
  11. package/skills/cavecrew/SKILL.md +0 -82
  12. package/skills/caveman-commit/README.md +0 -44
  13. package/skills/caveman-commit/SKILL.md +0 -65
  14. package/skills/caveman-discover/SKILL.md +0 -118
  15. package/skills/caveman-evidence-review/SKILL.md +0 -144
  16. package/skills/caveman-explore/SKILL.md +0 -42
  17. package/skills/caveman-explore/package.json +0 -12
  18. package/skills/caveman-explore/tests/skill-file.test.mjs +0 -44
  19. package/skills/caveman-help/README.md +0 -38
  20. package/skills/caveman-help/SKILL.md +0 -63
  21. package/skills/caveman-learn/CLAUDE.md +0 -32
  22. package/skills/caveman-learn/README.md +0 -29
  23. package/skills/caveman-learn/SKILL.md +0 -69
  24. package/skills/caveman-learn/package.json +0 -12
  25. package/skills/caveman-learn/tests/skill-file.test.mjs +0 -44
  26. package/skills/caveman-manage/SKILL.md +0 -114
  27. package/skills/caveman-optimize/SKILL.md +0 -114
  28. package/skills/caveman-review/README.md +0 -33
  29. package/skills/caveman-review/SKILL.md +0 -55
  30. package/skills/caveman-setup/SKILL.md +0 -224
  31. package/skills/caveman-stats/README.md +0 -36
  32. package/skills/caveman-stats/SKILL.md +0 -12
  33. package/skills/investigate-first/SKILL.md +0 -16
  34. package/skills/investigate-first/agents/openai.yaml +0 -4
  35. package/skills/lean-build/SKILL.md +0 -18
  36. package/skills/lean-build/agents/openai.yaml +0 -4
  37. package/skills/migration/SKILL.md +0 -17
  38. package/skills/migration/agents/openai.yaml +0 -4
  39. package/skills/safe-refactor/SKILL.md +0 -16
  40. package/skills/safe-refactor/agents/openai.yaml +0 -4
  41. package/skills/surgical-patch/SKILL.md +0 -16
  42. package/skills/surgical-patch/agents/openai.yaml +0 -4
  43. package/skills/verify-and-stop/SKILL.md +0 -16
  44. package/skills/verify-and-stop/agents/openai.yaml +0 -4
@@ -10,9 +10,11 @@
10
10
 
11
11
  ---
12
12
 
13
- A Claude Code skill that compresses your project memory files (`CLAUDE.md`, todos, preferences) into caveman format — so every session loads fewer tokens automatically.
13
+ A Claude Code skill that compresses project memory files (`CLAUDE.md`, todos,
14
+ preferences) into caveman format, reducing repeated input size.
14
15
 
15
- Claude read `CLAUDE.md` on every session start. If file big, cost big. Caveman make file small. Cost go down forever.
16
+ Claude loads `CLAUDE.md` on every session start, so large files add repeated
17
+ input tokens. Caveman shortens supported natural-language files.
16
18
 
17
19
  ## What It Do
18
20
 
@@ -21,11 +23,15 @@ Claude read `CLAUDE.md` on every session start. If file big, cost big. Caveman m
21
23
  ```
22
24
 
23
25
  ```
24
- CLAUDE.md ← compressed (Claude reads this — fewer tokens every session)
26
+ CLAUDE.md ← compressed (Claude reads smaller file each session)
25
27
  CLAUDE.original.md ← human-readable backup (you edit this)
26
28
  ```
27
29
 
28
- Original never lost. Backup lives in a data dir, not next to your file — `$XDG_DATA_HOME/caveman-compress/backups/<parent-dir-name>/` (macOS/Linux) or `%LOCALAPPDATA%\caveman-compress\backups\<parent-dir-name>\` (Windows) — so skill auto-loaders don't re-read it as a live file. You can read and edit `.original.md` there. Run skill again to re-compress after edits.
30
+ Original remains in data directory rather than next to live file, so skill
31
+ auto-loaders do not read it twice. Path is
32
+ `$XDG_DATA_HOME/caveman-compress/backups/<parent-dir-name>/` on macOS and Linux,
33
+ or `%LOCALAPPDATA%\caveman-compress\backups\<parent-dir-name>\` on Windows. Edit
34
+ `.original.md` there, then run skill again to re-compress.
29
35
 
30
36
  ## Benchmarks
31
37
 
@@ -33,14 +39,15 @@ Real results on real project files:
33
39
 
34
40
  | File | Original | Compressed | Saved |
35
41
  |------|----------:|----------:|------:|
36
- | `claude-md-preferences.md` | 706 | 285 | **59.6%** |
37
- | `project-notes.md` | 1145 | 535 | **53.3%** |
38
- | `claude-md-project.md` | 1122 | 636 | **43.3%** |
39
- | `todo-list.md` | 627 | 388 | **38.1%** |
40
- | `mixed-with-code.md` | 888 | 560 | **36.9%** |
41
- | **Average** | **898** | **481** | **46%** |
42
+ | `claude-md-preferences.md` | 706 | 285 | 59.6% |
43
+ | `project-notes.md` | 1145 | 535 | 53.3% |
44
+ | `claude-md-project.md` | 1122 | 636 | 43.3% |
45
+ | `todo-list.md` | 627 | 388 | 38.1% |
46
+ | `mixed-with-code.md` | 888 | 560 | 36.9% |
47
+ | Average | 898 | 481 | 46% |
42
48
 
43
- All validations passed ✅ — headings, code blocks, URLs, file paths preserved exactly.
49
+ All fixture validations passed: headings, code blocks, URLs, and file paths were
50
+ preserved exactly.
44
51
 
45
52
  ## Before / After
46
53
 
@@ -48,7 +55,7 @@ All validations passed ✅ — headings, code blocks, URLs, file paths preserved
48
55
  <tr>
49
56
  <td width="50%">
50
57
 
51
- ### 📄 Original (706 tokens)
58
+ ### Original (706 tokens)
52
59
 
53
60
  > "I strongly prefer TypeScript with strict mode enabled for all new code. Please don't use `any` type unless there's genuinely no way around it, and if you do, leave a comment explaining the reasoning. I find that taking the time to properly type things catches a lot of bugs before they ever make it to runtime."
54
61
 
@@ -57,17 +64,20 @@ All validations passed ✅ — headings, code blocks, URLs, file paths preserved
57
64
 
58
65
  ### <img src="../../docs/assets/dancing-rock.svg" width="20" height="20" alt="rock"/> Caveman (285 tokens)
59
66
 
60
- > "Prefer TypeScript strict mode always. No `any` unless unavoidable — comment why if used. Proper types catch bugs early."
67
+ > "Prefer TypeScript strict mode always. No `any` unless unavoidable; comment why if used. Proper types catch bugs early."
61
68
 
62
69
  </td>
63
70
  </tr>
64
71
  </table>
65
72
 
66
- **Same instructions. 60% fewer tokens. Every. Single. Session.**
73
+ This fixture produced 59.6% fewer counted tokens. Structural validation passed;
74
+ result does not prove semantic equivalence on other files or models.
67
75
 
68
76
  ## Security
69
77
 
70
- `caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O patterns detected by static analysis. This is a false positive — see [SECURITY.md](./SECURITY.md) for a full explanation of what the skill does and does not do.
78
+ `caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O
79
+ patterns detected by static analysis. See [SECURITY.md](./SECURITY.md) for why
80
+ these operations exist and how paths are constrained.
71
81
 
72
82
  ## Install
73
83
 
@@ -76,10 +86,10 @@ Compress is built in with the `caveman` plugin. Install `caveman` once, then use
76
86
  If you need local files, the compress skill lives at:
77
87
 
78
88
  ```bash
79
- caveman-compress/
89
+ skills/caveman-compress/
80
90
  ```
81
91
 
82
- **Requires:** Python 3.10+
92
+ Requires Python 3.10 or newer.
83
93
 
84
94
  ## Usage
85
95
 
@@ -98,8 +108,8 @@ Examples:
98
108
 
99
109
  | Type | Compress? |
100
110
  |------|-----------|
101
- | `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` | ✅ Yes |
102
- | Extensionless natural language | ✅ Yes |
111
+ | `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` | Yes |
112
+ | Extensionless natural language | Yes |
103
113
  | `.py`, `.js`, `.ts`, `.json`, `.yaml` | ❌ Skip (code/config) |
104
114
  | `*.original.md` | ❌ Skip (backup files) |
105
115
 
@@ -110,13 +120,13 @@ Examples:
110
120
  ↓
111
121
  detect file type (no tokens)
112
122
  ↓
113
- Claude compresses (tokens — one call)
123
+ Claude compresses (tokens: one call)
114
124
  ↓
115
125
  validate output (no tokens)
116
126
  checks: headings, code blocks, URLs, file paths, bullets
117
127
  ↓
118
- if errors: Claude fixes cherry-picked issues only (tokens — targeted fix)
119
- does NOT recompress — only patches broken parts
128
+ if errors: Claude fixes cherry-picked issues only (tokens: targeted fix)
129
+ does NOT recompress; only patches broken parts
120
130
  ↓
121
131
  retry up to 2 times
122
132
  ↓
@@ -142,22 +152,25 @@ Caveman compress natural language. It never touch:
142
152
 
143
153
  ## Why This Matter
144
154
 
145
- `CLAUDE.md` loads on **every session start**. A 1000-token project memory file costs tokens every single time you open a project. Over 100 sessions that's 100,000 tokens of overhead — just for context you already wrote.
155
+ `CLAUDE.md` loads on every session start. A 1,000-token project memory file adds
156
+ 1,000 input tokens each time project opens, or 100,000 across 100 sessions.
146
157
 
147
- Caveman cut that by ~46% on average. Same instructions. Same accuracy. Less waste.
158
+ Caveman reduced counted tokens by about 46% on five listed fixtures. Validators
159
+ confirmed headings, code blocks, URLs, and file paths. They did not establish
160
+ general semantic or task-quality equivalence.
148
161
 
149
162
  ```
150
163
  ┌────────────────────────────────────────────┐
151
164
  │ TOKEN SAVINGS PER FILE █████ 46% │
152
- │ SESSIONS THAT BENEFIT ██████████ 100% │
153
- │ INFORMATION PRESERVED ██████████ 100% │
165
+ │ FIXTURES IN TABLE 5 │
166
+ │ STRUCTURAL VALIDATION passed on all │
154
167
  │ SETUP TIME █ 1x │
155
168
  └────────────────────────────────────────────┘
156
169
  ```
157
170
 
158
171
  ## Part of Caveman
159
172
 
160
- This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit — making Claude use fewer tokens without losing accuracy.
173
+ This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit.
161
174
 
162
- - **caveman** — make Claude *speak* like caveman (cuts response tokens ~65%)
163
- - **caveman-compress** — make Claude *read* less (cuts context tokens ~46%)
175
+ - `caveman`: ask Claude to answer in shorter prose
176
+ - `caveman-compress`: shorten supported project-memory files with backups and validation
@@ -16,9 +16,16 @@ import tempfile
16
16
  from pathlib import Path
17
17
  from typing import List
18
18
 
19
- OUTER_FENCE_REGEX = re.compile(
20
- r"\A\s*(`{3,}|~{3,})[^\n]*\n(.*)\n\1\s*\Z", re.DOTALL
21
- )
19
+ # Windows consoles default to cp1252, which cannot encode the emoji glyphs in
20
+ # our status lines; replace unencodable characters instead of crashing.
21
+ for _stream in (sys.stdout, sys.stderr):
22
+ try:
23
+ _stream.reconfigure(errors="replace")
24
+ except Exception:
25
+ pass
26
+
27
+ # A fence marker at the start of a line, at CommonMark's 0-3 space indent.
28
+ FENCE_LINE_REGEX = re.compile(r"^\s{0,3}(`{3,}|~{3,})")
22
29
 
23
30
  # YAML frontmatter: starts at file start with --- on its own line, ends with --- on its own line.
24
31
  # Captures the entire block (including delimiters and trailing newline) and the body after.
@@ -106,14 +113,44 @@ def is_sensitive_path(filepath: Path) -> bool:
106
113
 
107
114
 
108
115
  def strip_llm_wrapper(text: str) -> str:
109
- """Strip outer ```markdown ... ``` fence when it wraps the entire output."""
110
- m = OUTER_FENCE_REGEX.match(text)
111
- if m:
112
- return m.group(2)
113
- return text
114
-
115
-
116
- def write_text_atomic(path: Path, text: str) -> None:
116
+ r"""Strip an outer ```markdown ... ``` fence when it wraps the ENTIRE output.
117
+
118
+ The wrapper is only real when the first and last fence lines are the SAME
119
+ block. The old regex (``\A\s*(fence)[^\n]*\n(.*)\n\1\s*\Z`` with DOTALL and
120
+ a greedy ``.*``) never checked that: it matched any document that merely
121
+ STARTS and ENDS with a fence line. An ordinary README section —
122
+ ```bash npm install``` , prose, ```bash npm test``` — came back with its
123
+ first and last fence markers deleted and its two code blocks merged into
124
+ prose, so validation failed on both the compress and the fix path and the
125
+ section was permanently uncompressible after three paid API calls.
126
+ """
127
+ lines = text.split("\n")
128
+ first, last = 0, len(lines) - 1
129
+ while first < len(lines) and not lines[first].strip():
130
+ first += 1
131
+ while last > first and not lines[last].strip():
132
+ last -= 1
133
+ if first >= last:
134
+ return text
135
+ opener = FENCE_LINE_REGEX.match(lines[first])
136
+ closer = FENCE_LINE_REGEX.match(lines[last])
137
+ if not opener or not closer:
138
+ return text
139
+ marker = opener.group(1)
140
+ # Closing fence: same character, at least as long, and nothing else on the line.
141
+ if closer.group(1)[0] != marker[0] or len(closer.group(1)) < len(marker):
142
+ return text
143
+ if lines[last].strip() != closer.group(1):
144
+ return text
145
+ # Any fence of the same kind in between means these two are not one block.
146
+ for line in lines[first + 1:last]:
147
+ inner = FENCE_LINE_REGEX.match(line)
148
+ if inner and inner.group(1)[0] == marker[0] and len(inner.group(1)) >= len(marker):
149
+ return text
150
+ return "\n".join(lines[first + 1:last])
151
+
152
+
153
+ def write_text_atomic(path: Path, text: str, newline: str = "\n") -> None:
117
154
  """Write ``text`` to ``path`` atomically as UTF-8.
118
155
 
119
156
  Path.write_text() truncates the destination before encoding the string —
@@ -122,8 +159,22 @@ def write_text_atomic(path: Path, text: str) -> None:
122
159
  first, write the bytes to a sibling temp file, fsync, then os.replace()
123
160
  so the destination only ever moves from one complete, valid file to
124
161
  another. Preserves the original file's permission bits across the swap.
162
+
163
+ ``newline`` is the line terminator to emit. Callers pass the terminator
164
+ read_source() found in the source file so a CRLF document stays CRLF —
165
+ text-mode writes translating LF to the platform default rewrote every
166
+ line ending in every file the tool touched (issue #762), and reading the
167
+ bytes ourselves means nothing translates them back.
125
168
  """
126
- data = text.encode("utf-8")
169
+ if newline != "\n":
170
+ # Normalise first: model output can already carry CRLF, and a bare
171
+ # "\n" -> "\r\n" replace would turn those into "\r\r\n".
172
+ text = text.replace("\r\n", "\n").replace("\n", newline)
173
+ write_bytes_atomic(path, text.encode("utf-8"))
174
+
175
+
176
+ def write_bytes_atomic(path: Path, data: bytes) -> None:
177
+ """Write ``data`` to ``path`` atomically, preserving permission bits."""
127
178
  fd, tmp_name = tempfile.mkstemp(
128
179
  dir=str(path.parent), prefix=path.name + ".", suffix=".tmp"
129
180
  )
@@ -144,6 +195,40 @@ def write_text_atomic(path: Path, text: str) -> None:
144
195
  raise
145
196
 
146
197
 
198
+ def read_source(filepath: Path) -> tuple[str, str, bytes]:
199
+ """Read a source file as UTF-8, returning (text, line_terminator, raw_bytes).
200
+
201
+ Decodes strictly. The old errors="ignore" silently DROPPED every byte that
202
+ was not valid UTF-8 — a cp1252-authored file holding `\xe9` for "e-acute"
203
+ lost that byte, the mangled text was what got written to the backup, the
204
+ backup readback compared mangled-to-mangled so verification passed, and
205
+ then the original was overwritten. The bytes were unrecoverable and
206
+ nothing reported a problem (the destructive form of issue #686). A file we
207
+ cannot read exactly is a file we must not rewrite.
208
+
209
+ Line endings are detected from the raw bytes and returned to the caller
210
+ rather than being universal-newline'd away, so write_text_atomic can put
211
+ back what was there (issue #762). A mixed-ending file takes the terminator
212
+ the majority of its lines use — presence of one CRLF is not a mandate to
213
+ rewrite every LF in the document. The raw bytes come back too, so the
214
+ backup can be a byte-for-byte copy rather than a re-rendering.
215
+ """
216
+ raw = filepath.read_bytes()
217
+ try:
218
+ text = raw.decode("utf-8")
219
+ except UnicodeDecodeError as e:
220
+ raise ValueError(
221
+ f"Refusing to compress {filepath}: not valid UTF-8 "
222
+ f"(byte 0x{raw[e.start]:02x} at offset {e.start}). "
223
+ "Compression rewrites the file in place, and any byte this tool "
224
+ "cannot decode would be destroyed by the round trip. "
225
+ "Convert the file to UTF-8 first."
226
+ ) from None
227
+ crlf = text.count("\r\n")
228
+ newline = "\r\n" if crlf * 2 > text.count("\n") else "\n"
229
+ return text.replace("\r\n", "\n").replace("\r", "\n"), newline, raw
230
+
231
+
147
232
  def first_nonblank_line(text: str) -> str:
148
233
  """Return the first non-blank line, stripped — used to detect a prose
149
234
  preamble smuggled in ahead of the real content (issue #588)."""
@@ -153,13 +238,19 @@ def first_nonblank_line(text: str) -> str:
153
238
  return ""
154
239
 
155
240
 
156
- def _write_target(filepath: Path, text: str, backup_path: Path) -> None:
241
+ def _write_target(filepath: Path, text: str | bytes, backup_path: Path, newline: str = "\n") -> None:
157
242
  """Write to the target file, surfacing the backup location if the write
158
243
  itself fails. write_text_atomic already leaves the target untouched on
159
244
  failure, but the caller still needs to know where the pre-compression
160
- original lives instead of being left to guess (issue #652)."""
245
+ original lives instead of being left to guess (issue #652).
246
+
247
+ ``bytes`` restore the source verbatim; ``str`` is model output that still
248
+ has to be rendered with the document's line terminator."""
161
249
  try:
162
- write_text_atomic(filepath, text)
250
+ if isinstance(text, bytes):
251
+ write_bytes_atomic(filepath, text)
252
+ else:
253
+ write_text_atomic(filepath, text, newline)
163
254
  except Exception:
164
255
  print(f"❌ Write to {filepath} failed. Original preserved at backup: {backup_path}")
165
256
  raise
@@ -229,6 +320,7 @@ Compress this markdown into caveman format.
229
320
 
230
321
  STRICT RULES:
231
322
  - Do NOT modify anything inside ``` code blocks
323
+ - Do NOT modify anything inside a 4-space-indented code block either — those are code too, and they are validated
232
324
  - Do NOT modify anything inside inline backticks
233
325
  - Preserve ALL URLs exactly
234
326
  - Preserve ALL headings exactly
@@ -301,7 +393,7 @@ def compress_file(filepath: Path) -> bool:
301
393
  print("Skipping (not natural language)")
302
394
  return False
303
395
 
304
- original_text = filepath.read_text(encoding="utf-8", errors="ignore")
396
+ original_text, newline, original_raw = read_source(filepath)
305
397
  # Store backup outside the source directory so skill auto-loaders don't
306
398
  # re-ingest the `.original.md` copy as a live file. Mirror the source's
307
399
  # parent-dir name + stem under a platform-aware base to reduce collisions.
@@ -355,9 +447,8 @@ def compress_file(filepath: Path) -> bool:
355
447
  # antivirus, disk full), unlink the bad backup and abort instead of
356
448
  # leaving the user with a corrupt backup + compressed primary.
357
449
  backup_dir.mkdir(parents=True, exist_ok=True)
358
- write_text_atomic(backup_path, original_text)
359
- backup_readback = backup_path.read_text(encoding="utf-8", errors="ignore")
360
- if backup_readback != original_text:
450
+ write_bytes_atomic(backup_path, original_raw)
451
+ if backup_path.read_bytes() != original_raw:
361
452
  print(f"❌ Backup write verification failed: {backup_path}")
362
453
  print(" In-memory original differs from on-disk backup. Aborting before touching the input file.")
363
454
  try:
@@ -365,7 +456,7 @@ def compress_file(filepath: Path) -> bool:
365
456
  except OSError:
366
457
  pass
367
458
  return False
368
- _write_target(filepath, compressed, backup_path)
459
+ _write_target(filepath, compressed, backup_path, newline)
369
460
 
370
461
  # Step 2: Validate + Retry
371
462
  for attempt in range(MAX_RETRIES):
@@ -383,7 +474,7 @@ def compress_file(filepath: Path) -> bool:
383
474
 
384
475
  if attempt == MAX_RETRIES - 1:
385
476
  # Restore original on failure
386
- _write_target(filepath, original_text, backup_path)
477
+ _write_target(filepath, original_raw, backup_path, newline)
387
478
  backup_path.unlink(missing_ok=True)
388
479
  print("❌ Failed after retries — original restored")
389
480
  return False
@@ -409,6 +500,6 @@ def compress_file(filepath: Path) -> bool:
409
500
  print(" Possible preamble leak. Skipping this attempt.")
410
501
  continue
411
502
 
412
- _write_target(filepath, compressed, backup_path)
503
+ _write_target(filepath, compressed, backup_path, newline)
413
504
 
414
505
  return True
@@ -5,13 +5,40 @@ from pathlib import Path
5
5
 
6
6
  URL_REGEX = re.compile(r"https?://[^\s)]+")
7
7
  FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
8
+
9
+ # A line that is nothing but a fence marker plus an optional info string, at ANY
10
+ # indentation. Used ONLY to scrub leaked markers before inline-code pairing (see
11
+ # extract_inline_codes) — never for block extraction.
12
+ #
13
+ # Widening FENCE_OPEN_REGEX itself to `\s*` looks like the obvious fix for #820
14
+ # and is a net regression: a lone indented ``` (the natural way to SHOW a fence
15
+ # inside prose) then opens a block that runs to EOF, swallowing real code blocks
16
+ # and silently removing their inline spans from validation. That turns a
17
+ # false-failure bug into a false-PASS bug, and a false PASS overwrites the
18
+ # user's file with unvalidated output.
19
+ FENCE_MARKER_LINE_REGEX = re.compile(r"^\s*(?:`{3,}|~{3,})[^`~]*$")
20
+
21
+ # Cap on how much of a lost/added span is echoed in an error message. Unpaired
22
+ # backticks can make a "span" hundreds of characters of prose; printing it whole
23
+ # is what made #820's failures undiagnosable.
24
+ MAX_REPORTED_SPAN = 60
8
25
  HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
9
26
  BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
27
+ # Any list item, ordered or not. Four spaces inside a list item is the item's
28
+ # content indentation, never an indented code block.
29
+ LIST_ITEM_REGEX = re.compile(r"^\s*(?:[-*+]|\d+[.)])\s")
10
30
 
11
31
  # crude but effective path detection
12
32
  # Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
13
33
  PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
14
34
 
35
+ # PATH_REGEX is crude on purpose and also matches ordinary prose pairs —
36
+ # "pros/cons", "Node/browser", "state/lifecycle". Caveman prose ADDS those
37
+ # constructions freely and dropping one breaks nothing, so only an unambiguous
38
+ # path — a leading ./ ../ / or drive letter, or a dotted filename in the last
39
+ # component — is treated as a hard loss.
40
+ DEFINITE_PATH_REGEX = re.compile(r"^(?:\./|\.\./|/|[A-Za-z]:\\)|[^/\\]*\.[A-Za-z0-9]{1,8}$")
41
+
15
42
 
16
43
  class ValidationResult:
17
44
  def __init__(self):
@@ -79,9 +106,108 @@ def extract_code_blocks(text):
79
106
  blocks.append("\n".join(block_lines))
80
107
  # Unclosed fences are silently skipped — they indicate malformed markdown
81
108
  # and including them would cause false-positive validation failures.
109
+ return blocks + extract_indented_code_blocks(text)
110
+
111
+
112
+ def extract_indented_code_blocks(text):
113
+ """CommonMark indented code blocks — 4-space-indented runs outside any fence.
114
+
115
+ Without these, ` kubectl delete pod --all -n prod` was prose to the
116
+ validator: "code blocks preserved exactly" compared empty to empty and
117
+ PASSED while the compressor rewrote the command to
118
+ `kubectl delete pod -n dev`. A clean pass on a mutated destructive command
119
+ is the worst failure this tool has, because it overwrites the user's file.
120
+
121
+ Deliberately conservative about lists: inside a list item, four spaces are
122
+ the item's content indentation, not code, and nested bullets are ordinary
123
+ prose the compressor SHOULD rewrite. A run is only treated as code when the
124
+ document is not inside a list and the run is preceded by a blank line — so
125
+ this adds detections, it never turns existing passes into false failures.
126
+ """
127
+ blocks = []
128
+ lines = text.split("\n")
129
+ fenced = set()
130
+ for block in extract_fenced_spans(lines):
131
+ fenced.update(block)
132
+ in_list = False
133
+ previous_blank = True
134
+ i = 0
135
+ n = len(lines)
136
+ while i < n:
137
+ line = lines[i]
138
+ stripped = line.strip()
139
+ if i in fenced:
140
+ in_list, previous_blank = in_list, False
141
+ i += 1
142
+ continue
143
+ if not stripped:
144
+ previous_blank = True
145
+ i += 1
146
+ continue
147
+ indent = len(line) - len(line.lstrip(" \t"))
148
+ if LIST_ITEM_REGEX.match(line):
149
+ in_list = True
150
+ elif indent == 0:
151
+ in_list = False
152
+ if not in_list and previous_blank and indent >= 4:
153
+ run = []
154
+ while i < n and i not in fenced:
155
+ current = lines[i]
156
+ if not current.strip():
157
+ # A blank line continues an indented block only if more
158
+ # indented content follows.
159
+ lookahead = i + 1
160
+ while lookahead < n and not lines[lookahead].strip():
161
+ lookahead += 1
162
+ if lookahead < n and lookahead not in fenced and \
163
+ len(lines[lookahead]) - len(lines[lookahead].lstrip(" \t")) >= 4:
164
+ run.extend(lines[i:lookahead])
165
+ i = lookahead
166
+ continue
167
+ break
168
+ if len(current) - len(current.lstrip(" \t")) < 4:
169
+ break
170
+ run.append(current)
171
+ i += 1
172
+ if run:
173
+ blocks.append("\n".join(run))
174
+ previous_blank = False
175
+ continue
176
+ previous_blank = False
177
+ i += 1
82
178
  return blocks
83
179
 
84
180
 
181
+ def extract_fenced_spans(lines):
182
+ """Line-index ranges covered by fenced blocks, so indented-code detection
183
+ never reaches inside one."""
184
+ spans = []
185
+ i = 0
186
+ n = len(lines)
187
+ while i < n:
188
+ m = FENCE_OPEN_REGEX.match(lines[i])
189
+ if not m:
190
+ i += 1
191
+ continue
192
+ fence_char = m.group(2)[0]
193
+ fence_len = len(m.group(2))
194
+ start = i
195
+ i += 1
196
+ while i < n:
197
+ close_m = FENCE_OPEN_REGEX.match(lines[i])
198
+ if (
199
+ close_m
200
+ and close_m.group(2)[0] == fence_char
201
+ and len(close_m.group(2)) >= fence_len
202
+ and close_m.group(3).strip() == ""
203
+ ):
204
+ i += 1
205
+ break
206
+ i += 1
207
+ spans.append(range(start, i))
208
+ return spans
209
+
210
+
85
211
  def extract_urls(text):
86
212
  return set(URL_REGEX.findall(text))
87
213
 
@@ -101,10 +227,28 @@ def extract_inline_codes(text):
101
227
  fences indented 1-3 spaces (valid CommonMark). Reuse extract_code_blocks
102
228
  (FENCE_OPEN_REGEX-based, indentation-aware) instead so an indented fence's
103
229
  body backticks don't leak into inline-code pairing.
230
+
231
+ Any fence-marker line that survives that pass is then blanked (#820). A
232
+ fence indented 4+ spaces — what you get from showing an example inside a
233
+ bullet — is not matched by FENCE_OPEN_REGEX, so extract_code_blocks does
234
+ not remove it and its OWN backticks used to leak in and shift the pairing
235
+ of every following span, making the file permanently uncompressible.
236
+ Blanking just the marker lines fixes that without removing any prose, and
237
+ cannot run away the way a widened fence opener does.
238
+
239
+ The span pattern deliberately still spans newlines. CommonMark permits a
240
+ line ending inside a code span and hard-wrapped markdown produces them, so
241
+ a single-line pattern silently drops real spans — which downgrades a
242
+ deleted or mutated span from error to PASS. Long/garbled spans are a
243
+ presentation problem, handled by truncating in the error message instead.
104
244
  """
105
245
  text_without_fences = text
106
246
  for block in extract_code_blocks(text):
107
247
  text_without_fences = text_without_fences.replace(block, "", 1)
248
+ text_without_fences = "\n".join(
249
+ "" if FENCE_MARKER_LINE_REGEX.match(line) else line
250
+ for line in text_without_fences.split("\n")
251
+ )
108
252
  return re.findall(r"`([^`]+)`", text_without_fences)
109
253
 
110
254
 
@@ -115,11 +259,26 @@ def validate_headings(orig, comp, result):
115
259
  h1 = extract_headings(orig)
116
260
  h2 = extract_headings(comp)
117
261
 
262
+ # Changed heading TEXT is an error, not a warning. Every in-document anchor
263
+ # link points at a heading's slug, so renaming "# Configuration Options" to
264
+ # "# Config" silently breaks all of them — and SKILL.md and CLAUDE.md both
265
+ # state headings are preserved. Only counts used to gate the overwrite, so
266
+ # a run that renamed every heading reported "Validation passed". A level-only
267
+ # change keeps every slug intact and stays a warning.
118
268
  if len(h1) != len(h2):
119
269
  result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
270
+ return
120
271
 
121
- if h1 != h2:
122
- result.add_warning("Heading text/order changed")
272
+ t1 = [text for _, text in h1]
273
+ t2 = [text for _, text in h2]
274
+ if t1 != t2:
275
+ lost = [t for t in t1 if t not in t2]
276
+ added = [t for t in t2 if t not in t1]
277
+ result.add_error(f"Heading text/order changed: lost={lost}, added={added}")
278
+ elif h1 != h2:
279
+ # Same text, different level. The outline moved but no anchor broke —
280
+ # slugs come from the text, so links still resolve.
281
+ result.add_warning("Heading levels changed")
123
282
 
124
283
 
125
284
  def validate_code_blocks(orig, comp, result):
@@ -139,11 +298,23 @@ def validate_urls(orig, comp, result):
139
298
 
140
299
 
141
300
  def validate_paths(orig, comp, result):
301
+ """File paths are preserved — an error, never a warning.
302
+
303
+ This validator never called add_error at all, so a compressed file that had
304
+ dropped `src/hooks/caveman-config.js` still reported "Validation passed" and
305
+ the in-place overwrite stood. SKILL.md and CLAUDE.md both promise paths
306
+ survive compression; nothing enforced it.
307
+ """
142
308
  p1 = extract_paths(orig)
143
309
  p2 = extract_paths(comp)
310
+ lost = p1 - p2
311
+ added = p2 - p1
144
312
 
145
- if p1 != p2:
146
- result.add_warning(f"Path mismatch: lost={p1 - p2}, added={p2 - p1}")
313
+ definite = {p for p in lost if DEFINITE_PATH_REGEX.search(p)}
314
+ if definite:
315
+ result.add_error(f"File paths lost: {sorted(definite)}")
316
+ if (lost - definite) or added:
317
+ result.add_warning(f"Path mismatch: lost={sorted(lost)}, added={sorted(added)}")
147
318
 
148
319
 
149
320
  def validate_bullets(orig, comp, result):
@@ -160,6 +331,22 @@ def validate_bullets(orig, comp, result):
160
331
 
161
332
 
162
333
  def validate_inline_codes(orig, comp, result):
334
+ def _render_spans(spans):
335
+ """Render spans for an error message, truncated and newline-escaped.
336
+
337
+ A span may legitimately contain newlines, and an unpaired backtick can
338
+ make one hundreds of characters of prose. Printing those whole is what
339
+ made #820's failures undiagnosable — but the fix belongs here, in
340
+ presentation, not in what counts as a span.
341
+ """
342
+ out = []
343
+ for span in sorted(spans):
344
+ flat = span.replace("\n", "\\n")
345
+ if len(flat) > MAX_REPORTED_SPAN:
346
+ flat = flat[:MAX_REPORTED_SPAN] + "…"
347
+ out.append(repr(flat))
348
+ return "{" + ", ".join(out) + "}"
349
+
163
350
  c1 = Counter(extract_inline_codes(orig))
164
351
  c2 = Counter(extract_inline_codes(comp))
165
352
 
@@ -170,9 +357,9 @@ def validate_inline_codes(orig, comp, result):
170
357
  if code in c2 and c2[code] < count:
171
358
  lost.add(f"{code} (lost {count - c2[code]} of {count} occurrences)")
172
359
  if lost:
173
- result.add_error(f"Inline code lost: {lost}")
360
+ result.add_error(f"Inline code lost: {_render_spans(lost)}")
174
361
  if added:
175
- result.add_warning(f"Inline code added: {added}")
362
+ result.add_warning(f"Inline code added: {_render_spans(added)}")
176
363
 
177
364
 
178
365
  # ---------- Main ----------