@markup-carve/carve-grammars 0.1.7 → 0.1.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +192 -215
  2. package/diff/carve-diff.css +47 -0
  3. package/diff/index.d.ts +14 -0
  4. package/diff/index.js +58 -0
  5. package/highlightjs/carve.js +365 -169
  6. package/package.json +93 -5
  7. package/prism/carve.js +439 -91
  8. package/shiki/carve.css +36 -0
  9. package/shiki/diff.d.ts +9 -0
  10. package/shiki/diff.js +39 -0
  11. package/shiki/index.js +2 -0
  12. package/textmate/carve.tmLanguage.json +1195 -234
  13. package/tiptap/carve-editor.js +73 -0
  14. package/tiptap/carve-kit.js +184 -40
  15. package/tiptap/carve-to-pm.js +39 -16
  16. package/tiptap/editor.css +454 -1
  17. package/tiptap/extensions/carve-abbreviation-definition.js +7 -0
  18. package/tiptap/extensions/carve-abbreviation.js +7 -1
  19. package/tiptap/extensions/carve-attribute-slots.js +53 -7
  20. package/tiptap/extensions/carve-citation-definition.js +1 -1
  21. package/tiptap/extensions/carve-citation.js +20 -1
  22. package/tiptap/extensions/carve-comment.js +9 -4
  23. package/tiptap/extensions/carve-crossref.js +14 -1
  24. package/tiptap/extensions/carve-definition-list.js +3 -2
  25. package/tiptap/extensions/carve-div.js +14 -10
  26. package/tiptap/extensions/carve-embed.js +35 -4
  27. package/tiptap/extensions/carve-figure.js +3 -13
  28. package/tiptap/extensions/carve-footnote-definition.js +2 -2
  29. package/tiptap/extensions/carve-footnote.js +18 -0
  30. package/tiptap/extensions/carve-frontmatter.js +236 -1
  31. package/tiptap/extensions/carve-heading.js +2 -19
  32. package/tiptap/extensions/carve-link-ref-def.js +12 -2
  33. package/tiptap/extensions/carve-literal.js +7 -4
  34. package/tiptap/extensions/carve-math.js +14 -23
  35. package/tiptap/extensions/carve-mention.js +9 -0
  36. package/tiptap/extensions/carve-raw-block.js +3 -1
  37. package/tiptap/extensions/carve-raw-inline.js +7 -4
  38. package/tiptap/extensions/carve-source-preservation.js +17 -0
  39. package/tiptap/extensions/carve-substitution.js +37 -2
  40. package/tiptap/extensions/carve-symbol.js +11 -2
  41. package/tiptap/extensions/carve-unsupported-inline.js +3 -0
  42. package/tiptap/extensions/carve-unsupported.js +3 -0
  43. package/tiptap/extensions/editable-atom-view.js +358 -0
  44. package/tiptap/extensions/panel-bar.js +24 -3
  45. package/tiptap/index.d.ts +7 -0
  46. package/tiptap/index.js +1 -1
  47. package/tiptap/schema-map.json +11 -9
  48. package/tiptap/serializer.js +183 -43
  49. package/tiptap/wire-fixtures.json +28 -9
package/README.md CHANGED
@@ -28,21 +28,25 @@ present here is not a promise that the editor grammars use the same one.
28
28
  npm install @markup-carve/carve-grammars
29
29
  ```
30
30
 
31
- All peer dependencies are optional - install only what you use:
32
- `@tiptap/core` + `@tiptap/starter-kit` (v2 or v3) for the editor, `prismjs` (v1)
33
- for Prism, `highlight.js` (v11) for highlight.js. CI runs the suite against both
34
- Tiptap majors.
31
+ All peer dependencies are optional - install only what you use: the Tiptap
32
+ packages below (v2 or v3) for the editor, `prismjs` (v1) for Prism,
33
+ `highlight.js` (v11) for highlight.js. CI runs the suite against both Tiptap
34
+ majors.
35
+
36
+ `@markup-carve/carve-grammars/tiptap` loads `CarveKit`, which imports several
37
+ standalone Tiptap extensions, so the editor entry needs all of these installed.
38
+ Disabling one through `CarveKit.configure()` does not remove its import:
39
+
40
+ ```bash
41
+ npm install @tiptap/core @tiptap/pm @tiptap/starter-kit \
42
+ @tiptap/extension-{bullet-list,code,code-block,hard-break,heading,highlight,image,link,list-item,ordered-list,subscript,superscript,table,table-cell,table-header,table-row,task-item,task-list,underline}
43
+ ```
35
44
 
36
45
  On Tiptap 3, `CarveKit` disables StarterKit's bundled Underline and Link, since
37
46
  it registers its own (underline carries Carve's `_text_` mapping). Pass
38
47
  `starterKit: { underline: true }` to opt back in, at the cost of a duplicate
39
48
  mark name.
40
49
 
41
- `CarveKit` also pulls in several standalone Tiptap marks/extensions (highlight,
42
- subscript, superscript, underline, link, image, table, task-list); install the
43
- `@tiptap/extension-*` packages you use, or disable them via
44
- `CarveKit.configure({ underline: false, ... })`.
45
-
46
50
  ## Usage
47
51
 
48
52
  ```js
@@ -96,7 +100,7 @@ Without it, the attribute stays literal: `<span abbr="...">`. The mark's
96
100
  `parseHTML` reads back the `<abbr title>` form.
97
101
 
98
102
  The tokens target carve-php's **parser** (the contract: serialized Carve must parse
99
- back to the same elements). Carve's inline syntax differs notably from Djot's:
103
+ back to the same elements). Carve's inline syntax differs from Djot's:
100
104
  emphasis is `/text/` (Djot uses `_`), `_text_` is underline, `~text~` is
101
105
  strikethrough, highlight is `=text=`, and subscript/superscript are the
102
106
  braced `{,text,}` / `{^text^}` only (a bare `,` or `^` is literal text since
@@ -105,8 +109,9 @@ carve #259).
105
109
  Each single-char delimiter has two equivalent forms: a **bare** form
106
110
  (`=text=`) and a **forced brace** form (`{=text=}`) that also works intraword;
107
111
  both parse to the same element. The two columns above list bare / forced.
108
- `serializeToCarve` emits the bare form for `* / _ ~` and the forced `{…}` form
109
- for `= , ^` (round-trip-safe — those delimiters are likelier to be inert bare);
112
+ `serializeToCarve` emits the bare form for `* / _ ~`, or the forced form where a
113
+ letter or digit touches the delimiter, and the forced `{…}` form
114
+ for `= , ^` (round-trip-safe: those delimiters are likelier to be inert bare);
110
115
  `{+…+}` / `{-…-}` (insert / delete) have only the brace form, since `+` / `-`
111
116
  are not emphasis delimiters.
112
117
 
@@ -259,6 +264,59 @@ available as the `editor` CSS part (`carve-editor::part(editor)`). Tiptap stays
259
264
  an implementation detail of the element, although its peer packages must be
260
265
  installed with `carve-grammars`.
261
266
 
267
+ Document front matter appears as a collapsed **Document metadata** card. Expand
268
+ it to edit the common `title`, `lang`, `author`, and `description` fields, or use
269
+ the raw YAML/TOML field for custom metadata. Both paths update the document
270
+ through the editor, so they participate in undo/redo and preserve unknown keys.
271
+
272
+ ### Quick fields
273
+
274
+ The four quick fields are a default, not the contract. A product whose front
275
+ matter means something else describes its own:
276
+
277
+ ```js
278
+ CarveKit.configure({
279
+ carveFrontmatter: {
280
+ fields: [
281
+ { key: 'title', label: 'Note title', placeholder: 'Optional title', inputAttributes: { required: true, maxlength: 120 } },
282
+ { key: 'summary', label: 'Summary', multiline: true },
283
+ ],
284
+ },
285
+ })
286
+ ```
287
+
288
+ A descriptor needs a `key` - the front matter key it reads and writes. `label`
289
+ defaults to the capitalized key, `placeholder` is optional, and `multiline`
290
+ picks a `textarea` instead of an `input`. The list replaces the default one
291
+ entirely, so an empty array renders the raw front matter editor with no quick
292
+ fields at all:
293
+
294
+ ```js
295
+ CarveKit.configure({ carveFrontmatter: { fields: [] } })
296
+ ```
297
+
298
+ A key may appear only once; a second descriptor for it is refused, because the
299
+ controls are held by key and the earlier one would render blank and write
300
+ nowhere.
301
+
302
+ `inputAttributes` carries native validation and input hints: `required`,
303
+ `autocomplete`, `inputmode`, `minlength`, `maxlength`, `pattern` and
304
+ `aria-describedby`. These are enforced: a change that fails the control's own
305
+ `checkValidity()` is reported to the author and never reaches the document.
306
+ Anything else is **refused with a `TypeError`** rather than ignored - the node view owns `name`, `type`, `value`, `disabled` and `readonly`,
307
+ and an event handler set here would bypass the format-aware update path. A
308
+ constraint that is silently dropped reads like a working one that never fires,
309
+ which is why the rejection is loud.
310
+
311
+ The collapsed summary still reads `title` and `lang` out of the document
312
+ whatever the field list says, and writes still go through an editor transaction,
313
+ so undo/redo and unknown-key preservation are unchanged.
314
+ Unsupported-source atoms use the same compact/editable pattern for their exact
315
+ Carve payload. Inline footnotes, cross-references, and citations open target
316
+ pickers, while abbreviation and link-reference definitions use collapsible
317
+ forms. Import `@markup-carve/carve-grammars/tiptap/editor.css` when mounting
318
+ `CarveKit` directly to receive the associated editor chrome and table styling.
319
+
262
320
  ## Syntax highlighting
263
321
 
264
322
  Render Carve source as highlighted HTML on the web. Both grammars cover the full
@@ -364,6 +422,34 @@ can drop the one character a rule is about. The spec corpus is the exception and
364
422
  can afford to be - it marks `tests/corpus/**` as `-text`, so
365
423
  `250-line-endings-and-a-byte-order-mark-3.crv` really does begin `ef bb bf`.
366
424
 
425
+ ### Link destinations inside a bare run
426
+
427
+ A delimiter inside a link destination, its title or an autolink does not close
428
+ the bare run around it, so `/see [x](http://a.b/c) now/` stays one italic run.
429
+ The three grammars follow the spec's productions for what counts: a `](` needs a
430
+ complete label before it (`link_text`), a destination and a title take only
431
+ their own escapes, a title follows exactly one space (`link_title`), a URL
432
+ autolink holds only `url_char`, and an email autolink needs a dotted domain
433
+ ending in letters (`email_autolink`). Anything else closes the run where the
434
+ spec closes it.
435
+
436
+ Known limits, shared by all three unless noted:
437
+
438
+ - A label nested more than four brackets deep, or a destination holding three
439
+ levels of parentheses, is not recognized. A regex cannot count.
440
+ - Prism and highlight.js compile without the `u` flag, so an email autolink
441
+ accepts any non-ASCII character other than whitespace where the spec asks for
442
+ a letter. The TextMate grammar uses `\p{L}`.
443
+
444
+ ### Substitutions
445
+
446
+ A substitution splits at its first arrow outside code, math, a literal, a
447
+ comment or an escape, and each half holds inline content. With no such arrow,
448
+ `{~ ~}` is a forced strikethrough.
449
+
450
+ Known limit: in highlight.js and TextMate, a link or autolink that spans the
451
+ arrow stays whole in the deleted half. `{~[x](u~>v)~>c~}` splits after `u`.
452
+
367
453
  ### Fence words
368
454
 
369
455
  All three surfaces answer `carve` and `crv`. `.crv` is the canonical file
@@ -383,6 +469,33 @@ before resolving it. `tests/lib/aliases.js` holds the required set and
383
469
  `tests/alias-parity-test.js` asserts it on each surface through that surface's
384
470
  own registration API.
385
471
 
472
+ ### Attribute blocks
473
+
474
+ Known limit, shared by all three: a block glued to a bare delimiter is read as
475
+ attached even where the delimiter closes no run. In `` x*{title="`"} y `` the
476
+ braces are text, so the backtick opens a code span, but all three scope an
477
+ attribute block.
478
+
479
+ ### TextMate limits
480
+
481
+ A TextMate rule sees one line at a time, so a few shapes color differently from
482
+ how Carve reads them. vscode-carve declares the same three.
483
+
484
+ - A bare bold run can cross a soft line break, so its opener can't check for a
485
+ closer first. When the only closer-shaped `*` sits inside a code span, or
486
+ there is none, the run colors to the end of its paragraph:
487
+ `` x *a `b* c` d `` renders as text but shows as bold.
488
+ - A braced span whose closer is on a later line is not scoped as a span.
489
+ - A bare italic, underline, strikethrough or highlight run is scoped only when
490
+ its closer is on the same line.
491
+
492
+ ### Prism and highlight.js limits
493
+
494
+ - Prism scopes a bare run only when its closer is on the same line.
495
+ - highlight.js compiles without the `u` flag, so it reads any non-ASCII
496
+ character after a bare closer as a letter. `/a/« b` renders `a` in italics
497
+ but shows as text.
498
+
386
499
  ### Prism
387
500
 
388
501
  The grammar registers itself against the global `Prism`, so `Prism` must be
@@ -452,6 +565,70 @@ Named exports for other setups: `carveGrammar`, `carveLightExtras` /
452
565
  `carveDarkExtras`, `carveLightTheme` / `carveDarkTheme`, `extendTheme`,
453
566
  `carveStylingTransformer`.
454
567
 
568
+ #### Diff presentation with an underlying language
569
+
570
+ Carve keeps a code block's presentation hooks separate from its language. The
571
+ portable source convention for an instructional diff is a `.diff` block
572
+ attribute above a language-tagged fence:
573
+
574
+ ````carve
575
+ {.diff}
576
+ ```js
577
+ - fileIcon.classList.add("icon-file-text");
578
+ + fileIcon.classList.remove("icon-file-text");
579
+ ```
580
+ ````
581
+
582
+ Core HTML preserves both channels as
583
+ `<pre class="diff"><code class="language-js">`. A host can detect that shape
584
+ and invoke the opt-in `diffCodeTransformer()` with the fence language:
585
+
586
+ ```js
587
+ import { diffCodeTransformer } from '@markup-carve/carve-grammars/shiki/diff'
588
+ import '@markup-carve/carve-grammars/shiki/carve.css'
589
+
590
+ const transformers = pre.classList.contains('diff')
591
+ ? [diffCodeTransformer()]
592
+ : []
593
+
594
+ const highlighted = highlighter.codeToHtml(code.textContent, {
595
+ lang: 'javascript',
596
+ theme: 'github-light',
597
+ transformers,
598
+ })
599
+ ```
600
+
601
+ Create a fresh transformer for every code block. It treats `+`, `-`, and a
602
+ space as structural first characters, removes that character while Shiki
603
+ tokenizes the underlying language, then restores it and marks added/removed
604
+ lines. This first version targets compact instructional changes, not complete
605
+ patch files with `@@` hunks or `---` / `+++` file headers. It is deliberately
606
+ not enabled by `carveMarkdown()`, because ordinary code blocks must not lose
607
+ their first character.
608
+
609
+ A host that highlights with **highlight.js, Prism, or nothing** - where Shiki's
610
+ per-line token model is not available - uses the highlighter-agnostic helper
611
+ instead. It takes a per-line highlight callback (or defaults to HTML-escaping)
612
+ and produces the same `line` / `diff add` / `diff remove` / `diff-marker`
613
+ classes:
614
+
615
+ ```js
616
+ import { applyLanguageDiff } from '@markup-carve/carve-grammars/diff'
617
+ import '@markup-carve/carve-grammars/diff/carve-diff.css'
618
+
619
+ document.querySelectorAll('pre.diff > code').forEach((code) => {
620
+ const language = [...code.classList]
621
+ .find((name) => name.startsWith('language-'))
622
+ ?.slice('language-'.length)
623
+ applyLanguageDiff(code, (body) => hljs.highlight(body, { language }).value)
624
+ })
625
+ ```
626
+
627
+ `renderLanguageDiff(code, highlightLine)` returns the HTML string if the host
628
+ manages the DOM itself (a webview building a document string, a server writing
629
+ markup). The marker is stripped before `highlightLine` runs, so the underlying
630
+ language tokenizes the line body without its `+`/`-`/space.
631
+
455
632
  ## Diagram rendering
456
633
 
457
634
  Carve's `FencedRenderExtension` presets emit a `<pre class="LANG">source</pre>`
@@ -536,6 +713,7 @@ graphviz/d2 against a self-hosted server), `onError`, `fetch`.
536
713
  - `astToProseMirror(ast, options?)` - convert an existing `@markup-carve/carve`
537
714
  AST to ProseMirror JSON.
538
715
  - `serializeToCarve(doc)` - serialize an `editor.getJSON()` document to Carve markup.
716
+ - `serializeToCarveWithReport(doc)` - the same, returning `{ source, dropped, degraded }`. `dropped` names what the source could not carry (a mention's display `label` that differs from its `id`), `degraded` a node written as literal text (a mention name with no Carve spelling, such as `Lea Thompson`, becomes `\@Lea Thompson`). A stock Tiptap `mention` node is written like `carveMention`: `id` is the name, a `null` label counts as absent, and `mentionSuggestionChar` is never written.
539
717
  - `escapeCarve(text)` - contextually escape literal Carve syntax in a plain-text run so it round-trips as text (used internally by `serializeToCarve`).
540
718
  - `CarveKit` - the bundled Tiptap extension set.
541
719
  - Individual extensions: `CarveInsert`, `CarveDelete`, `CarveCriticComment`, `CarveDiv`, `CarveSpan`, `CarveFootnote`, `CarveFootnoteDefinition`, `CarveMath`, `CarveEmbed`, `CarveAbbreviation`, `CarveDefinitionList`, `CarveUnsupported`.
@@ -604,207 +782,6 @@ implementation spelled it with underscores.
604
782
  `CarveFootnoteDefinition` is the matching body block, serialized as
605
783
  `[^label]: body`.
606
784
 
607
- ## Tests
608
-
609
- ```bash
610
- npm test
611
- ```
612
-
613
- The suite holds all three grammars to one source of truth: the shared corpus
614
- from the [`markup-carve/carve`](https://github.com/markup-carve/carve) spec,
615
- vendored as the `spec/` git submodule (`git submodule update --init`).
616
-
617
- - `npm run test:coverage` - the coverage matrix. Each grammar (prism,
618
- highlightjs, tiptap) declares a covered-category set and a skip set (with a
619
- reason per skip); the test fails if the two do not partition every corpus
620
- category, so a new spec category forces a deliberate decision.
621
- - `npm run test:snapshot` - golden token snapshots. Each covered `.crv` is
622
- tokenized with Prism's and highlight.js's own tokenizers and the token stream
623
- (type + text) is compared against a committed golden in `tests/snapshots/`.
624
- Refresh intended changes with `npm run snapshots:update`.
625
- - `npm run test:roundtrip` - the Tiptap serializer round-trip. Each covered
626
- `.crv` runs `parse -> ProseMirror JSON -> serializeToCarve -> parse` and the
627
- two parsed ASTs must be identical, catching serializer drift. Categories the
628
- serializer cannot represent are skipped with a reason.
629
-
630
- `npm test` runs all of the above plus the structural grammar and serializer
631
- unit tests. CI runs the same on Node 18, 20 and 22.
632
-
633
- ### The construct ledger: ten surfaces, one derived list
634
-
635
- Carve's syntax lives on ten grammar surfaces - `tree-sitter-carve`, the four
636
- here (TextMate, Prism, highlight.js, Tiptap), `vscode-carve`, `intellij-carve`,
637
- `sublime-carve`, `vim-carve` and `emacs-carve` - and nothing used to measure
638
- them against the same construct list. `{%` comments landed on five of them and
639
- had no rule at all on the other five, with nothing going red
640
- (markup-carve/carve#1239, #284).
641
-
642
- `npm test` now runs `tests/construct-ledger-test.js`, which holds
643
- `tests/lib/construct-ledger.json` to three rules:
644
-
645
- - **The construct list is derived, never written.** `scripts/spec-constructs.mjs`
646
- reads the alternatives of the `block` and `inline_element` productions out of
647
- the spec's normative `spec/resources/grammar.ebnf`. A clause that adds a
648
- construct makes every surface unclassified until someone says what it did
649
- about it. A hand-maintained checklist would go stale in exactly that case,
650
- because whoever forgot the grammar rule also forgot the checklist row.
651
- - **Three columns, not two.** Each construct on each surface is `IMPLEMENTED`
652
- (with the name that surface gives it), `UNSUPPORTED` (with a reason - an empty
653
- reason fails), or `GAP` (with a ticket). A construct in none of them fails the
654
- test. Without the third column a legitimate gap - a Prism tokenizer cannot do
655
- what a tree-sitter grammar does - reads as a defect, and a check with a high
656
- noise floor gets muted.
657
- - **Two axes per construct.** Recognizing a construct and keeping its payload
658
- inert are different questions, and a surface can be right about the first and
659
- wrong about the second: on four editor surfaces `{% *not bold* %}` colored a
660
- bold run inside a comment. Every entry declares which, and the answer is
661
- measured on every run - by tokenizing a sample with a live marker inside the
662
- payload - for Prism, highlight.js and every TextMate grammar whose checkout is
663
- in front of the seeder, and by converting that sample through the bridge for
664
- Tiptap. Those rows cannot rot. The rest are recorded, which is what
665
- `payload: "unmeasured"` says out loud - a state no cell is in any more,
666
- since intellij-carve's thirteen were the last of them (#329).
667
-
668
- One sample per construct is a re-measurement, not a measurement.
669
- `tests/opaque-payload-test.js` generates every payload up to three characters
670
- over each construct's own delimiter alphabet - about ten thousand documents -
671
- and it is what finds this class: of 1325 corpus documents exactly ONE exposed
672
- the fenced-code leak in #309, and a comment leaking only when it spans a line
673
- break is invisible to a single-line sample (#320).
674
-
675
- #### A red row is a question, not a defect
676
-
677
- Three surfaces have now been worked row by row, and the ratio is stable enough
678
- to plan against rather than be surprised by:
679
-
680
- | pass | rows | the instrument | genuinely absent |
681
- | --- | --- | --- | --- |
682
- | `tree-sitter-carve` (#245) | 14 | 9 | 3 |
683
- | `vim-carve` + `sublime-carve` (#318) | 20 | 11 | 6 |
684
- | `textmate`, `prism`, `tiptap`, `vscode-carve` (#307, #308, #310) | 47 | 31 | 9 |
685
-
686
- **Two rows in three are a name the probe cannot reach**, and the third pass also
687
- turned up six rows that read `IMPLEMENTED` and were not. So the first job on a
688
- per-surface ticket is deciding which rows are real, and the answer is expected
689
- to be "most of them are not". A GAP row means *the probe did not find a rule it
690
- recognizes*, which is a different claim from *this surface does not implement
691
- the construct*.
692
-
693
- A fold belongs in `SIGNATURE_OVERRIDES`, which the file calls a per-surface
694
- NAMING table - never in a rule renamed on a surface to satisfy the probe. Where
695
- a surface genuinely cannot express a construct, `UNSUPPORTED` **with a reason**
696
- is the entry; a wrong rule is worse than a missing one.
697
-
698
- #### The instrument can be wrong in the direction that looks green
699
-
700
- Five ways the probe has mis-read a surface, each found by opening the file:
701
-
702
- 1. **It read too little.** `Object.keys(grammar.json.rules)` is 196 of
703
- tree-sitter's 346 names; a `.sublime-syntax` scopes a capture as well as a
704
- match (#315).
705
- 2. **The signature list had holes** - `code_span` is `verbatim` on one surface
706
- and `code_inline` on another (#315, #307).
707
- 3. **One rule implements several constructs**, and one name cannot carry four
708
- through a shared table (#318).
709
- 4. **The evidence named a rule about a different construct.** `carveCommentInline`
710
- is the braced comment and was cited for the trailing one (#318). The same
711
- pair beat the word-boundary rank on the TextMate family, where both
712
- candidates are whole-name hits and *length* picked the wrong one (#307).
713
- 5. **The re-check read the file's prose as evidence.** `every IMPLEMENTED row
714
- cites a name the shipped grammar really carries` asked whether the evidence
715
- was a SUBSTRING of the source. `prism/carve.js` carries the comment "Prism
716
- has no cross-reference token at all" beside the lookbehind that worked
717
- around the absence - so a row citing `cross-ref` re-checked **green** for as
718
- long as the rule was missing (#307). That is the trap the docblock at the
719
- top of `scripts/surface-probe.mjs` describes, and that its extractors were
720
- built to avoid, surviving in the check that verifies their output. It reads
721
- `vocabulary()` now, so the seed and the re-check agree on what a name is.
722
-
723
- The lesson generalizes past this file: **a check that treats a file's prose as
724
- evidence that the file implements what the prose says it does NOT implement
725
- cannot fail.** Match against what a grammar DECLARES, never against its text.
726
-
727
- Re-measure after changing a grammar:
728
-
729
- ```bash
730
- node scripts/seed-construct-ledger.mjs
731
- ```
732
-
733
- That reads the four surfaces here directly. The six in other repositories are
734
- read from checkouts named by environment variables, and any that is not given
735
- keeps the row already recorded, with the commit it was read at:
736
-
737
- ```bash
738
- CARVE_SURFACE_VIM_CARVE=../vim-carve \
739
- CARVE_SURFACE_EMACS_CARVE=../emacs-carve \
740
- node scripts/seed-construct-ledger.mjs
741
- ```
742
-
743
- Statuses are measured; reasons, per-construct notes, tickets and a surface's
744
- `note` are written by hand and carried across a re-measurement, so a re-run
745
- never drops a stated reason. A surface-level `note` is for what the rows cannot
746
- say - why a surface needed no work to reach zero, or what its measured payload
747
- column does NOT cover.
748
-
749
- One thing a re-run does NOT carry: the payload axis of a surface the seeder
750
- cannot tokenize. It is only carried over when the recorded value is already
751
- `inert` or `leaks`, so a construct that moves from `GAP` to `IMPLEMENTED` in the
752
- same run arrives with `payload: "unmeasured"` and a ticket, even when the person
753
- doing the run has just measured it. That is deliberate: a seeder cannot tokenize
754
- a Vim syntax file or an emacs font-lock table, so the alternative is inventing an
755
- answer. Measure the payload as part of the same pass and write `inert` (or
756
- `leaks`, with a note) by hand on those rows.
757
-
758
- It applies to fewer surfaces than it used to. `vscode-carve` and
759
- `intellij-carve` are TextMate grammars, and the seeder loads any of those through
760
- Shiki, so naming their checkout measures the recognition axis and the payload
761
- sample in the same run:
762
-
763
- ```bash
764
- CARVE_SURFACE_VSCODE_CARVE=../vscode-carve node scripts/seed-construct-ledger.mjs
765
- CARVE_SURFACE_INTELLIJ_CARVE=../intellij-carve node scripts/seed-construct-ledger.mjs
766
- ```
767
-
768
- **A `leaks` cell is still hand-written on those surfaces, and the seeder carries
769
- it rather than overwriting it.** The seeder tokenizes ONE sample per construct
770
- and the sweep below generates hundreds, and on all three surfaces measured for
771
- the first time since #320 the sample said every row was inert while the sweep
772
- disagreed - five rows on intellij-carve alone (#329). So a run whose sample says
773
- `inert` leaves a recorded `leaks` alone. That cannot hide a fix: the sweep
774
- asserts every recorded leak STILL leaks, so a fix fails that file, its entry
775
- comes out, and the ledger's own orphan check then fails until the cell is
776
- corrected too.
777
-
778
- ### The payload sweep reaches nine of the ten surfaces
779
-
780
- `tests/opaque-payload-test.js` is the second axis measured over a GENERATED space
781
- rather than one sample per construct, and one sample is not enough. Every surface
782
- measured for the first time so far has had leaks the sample could not see:
783
-
784
- | surface | leaking rows | the sample found |
785
- | --- | --- | --- |
786
- | tree-sitter-carve (#328) | 3 | none of them |
787
- | emacs-carve (#328) | 2 | the `%%%` fence, not the verbatim run |
788
- | intellij-carve (#329) | 4 | none of them |
789
-
790
- Name a checkout and the sweep drives it:
791
-
792
- ```bash
793
- CARVE_SURFACE_TREE_SITTER_CARVE=../tree-sitter-carve \
794
- CARVE_SURFACE_EMACS_CARVE=../emacs-carve \
795
- CARVE_SURFACE_VSCODE_CARVE=../vscode-carve \
796
- CARVE_SURFACE_INTELLIJ_CARVE=../intellij-carve \
797
- node tests/opaque-payload-test.js
798
- ```
785
+ ## Development
799
786
 
800
- tree-sitter needs its native addon built (`npm install` in that checkout), and
801
- emacs-carve needs an `emacs` on PATH - the mode is fontified in one batch Emacs
802
- per sweep row, which is what `prime` on that tokenizer is for. A surface whose
803
- checkout is not named, or not built, is simply not swept.
804
-
805
- A leak on a grammar in ANOTHER repository is a defect this suite can measure and
806
- not fix, so it is recorded in `KNOWN_LEAKS` with the ticket it lives on and
807
- asserted to STILL leak. A fix therefore fails this file and the entry comes out
808
- with the fix, the same arrangement the residual table at the end of it uses for
809
- one document at a time. A surface in THIS repository is deliberately absent from
810
- that table: a leak here is fixable here, so it stays red.
787
+ Contributor setup, testing, and maintenance notes are in the [development guide](docs/development.md).
@@ -0,0 +1,47 @@
1
+ /* Companion CSS for carve-grammars/diff (the highlighter-agnostic helper).
2
+ *
3
+ * Frames the `{.diff}` presentation for output from renderLanguageDiff() /
4
+ * applyLanguageDiff() - a `<pre class="diff has-diff">` whose lines are
5
+ * `<span class="line diff add|remove">` prefixed with a `.diff-marker`. Keyed on
6
+ * `.has-diff` rather than a highlighter class, so it works with highlight.js,
7
+ * Prism, or no highlighter. The Shiki path has its own `../shiki/carve.css`.
8
+ *
9
+ * Hosts may override the variables without depending on a site framework; a
10
+ * VS Code / JetBrains webview points them at the editor's own diff theme. */
11
+ pre.has-diff {
12
+ --carve-diff-add-background: rgb(46 160 67 / 15%);
13
+ --carve-diff-add-marker: #1a7f37;
14
+ --carve-diff-remove-background: rgb(248 81 73 / 15%);
15
+ --carve-diff-remove-marker: #cf222e;
16
+ }
17
+
18
+ pre.has-diff code {
19
+ display: block;
20
+ }
21
+
22
+ pre.has-diff .line {
23
+ display: inline-block;
24
+ width: 100%;
25
+ }
26
+
27
+ pre.has-diff .line.diff.add {
28
+ background: var(--carve-diff-add-background);
29
+ }
30
+
31
+ pre.has-diff .line.diff.remove {
32
+ background: var(--carve-diff-remove-background);
33
+ }
34
+
35
+ pre.has-diff .diff-marker {
36
+ display: inline-block;
37
+ width: 1ch;
38
+ font-weight: 700;
39
+ }
40
+
41
+ pre.has-diff .line.diff.add .diff-marker {
42
+ color: var(--carve-diff-add-marker);
43
+ }
44
+
45
+ pre.has-diff .line.diff.remove .diff-marker {
46
+ color: var(--carve-diff-remove-marker);
47
+ }
@@ -0,0 +1,14 @@
1
+ /** Turn a marker-stripped diff line into HTML (language highlighting or escaping). */
2
+ export type HighlightLine = (body: string) => string
3
+
4
+ /**
5
+ * Render a Carve `{.diff}` language fence to an HTML string of per-line spans,
6
+ * using any highlighter (or none). See the module JSDoc for the contract.
7
+ */
8
+ export function renderLanguageDiff(code: string, highlightLine?: HighlightLine): string
9
+
10
+ /**
11
+ * Apply the diff presentation to a rendered `<pre class="diff"><code>` in the
12
+ * DOM: replace the code's content with per-line diff spans and mark the `<pre>`.
13
+ */
14
+ export function applyLanguageDiff(codeElement: Element, highlightLine?: HighlightLine): void
package/diff/index.js ADDED
@@ -0,0 +1,58 @@
1
+ /**
2
+ * Present a Carve `{.diff}` language fence with any highlighter (or none).
3
+ *
4
+ * Carve core emits `{.diff}` on a language-tagged fence as
5
+ * `<pre class="diff"><code class="language-x">…</code></pre>` with the leading
6
+ * `+`, `-`, and context-space characters left as plain text - applying the
7
+ * presentation is the host's job. The Shiki hosts use `../shiki/diff.js`; this
8
+ * module covers hosts on highlight.js, Prism, or no highlighter, where the
9
+ * per-line token model Shiki provides is not available.
10
+ *
11
+ * The caller supplies a per-line highlighter. The marker is stripped before it
12
+ * runs (so `- old()` tokenizes as `old()`), then restored as a `diff-marker`
13
+ * span, and added/removed lines get `diff add` / `diff remove` classes - the
14
+ * same shape the Shiki transformer produces. Pair with
15
+ * `@markup-carve/carve-grammars/diff/carve-diff.css`.
16
+ *
17
+ * No Node imports: safe to bundle into a browser client or a webview.
18
+ */
19
+
20
+ const MARKER = /^[+\- ]/;
21
+
22
+ const escapeHtml = (value) =>
23
+ value.replace(/[&<>]/g, (character) => (character === '&' ? '&amp;' : character === '<' ? '&lt;' : '&gt;'));
24
+
25
+ /**
26
+ * Render diff code to an HTML string of per-line spans.
27
+ *
28
+ * @param {string} code The code text, with each line's leading marker intact.
29
+ * @param {(body: string) => string} [highlightLine] Turns a marker-stripped line
30
+ * into HTML. Defaults to HTML-escaping (no language highlighting).
31
+ * @returns {string} The `<span class="line …">…</span>` lines, newline-joined.
32
+ */
33
+ export function renderLanguageDiff(code, highlightLine = escapeHtml) {
34
+ return code
35
+ .replace(/\n$/, '')
36
+ .split('\n')
37
+ .map((line) => {
38
+ const marker = MARKER.test(line) ? line[0] : '';
39
+ const body = marker ? line.slice(1) : line;
40
+ const lineClass = marker === '+' ? 'line diff add' : marker === '-' ? 'line diff remove' : 'line';
41
+ const markerSpan = marker ? `<span class="diff-marker">${escapeHtml(marker)}</span>` : '';
42
+ return `<span class="${lineClass}">${markerSpan}${highlightLine(body)}</span>`;
43
+ })
44
+ .join('\n');
45
+ }
46
+
47
+ /**
48
+ * Apply the diff presentation to a rendered `<pre class="diff"><code>` in the
49
+ * DOM: replace the code's content with per-line diff spans and mark the `<pre>`.
50
+ *
51
+ * @param {Element} codeElement The `<code>` inside a `<pre class="diff">`.
52
+ * @param {(body: string) => string} [highlightLine] See {@link renderLanguageDiff}.
53
+ */
54
+ export function applyLanguageDiff(codeElement, highlightLine) {
55
+ const pre = codeElement.parentElement;
56
+ codeElement.innerHTML = renderLanguageDiff(codeElement.textContent || '', highlightLine);
57
+ if (pre) pre.classList.add('has-diff');
58
+ }