mdrefine 0.2.0-rc.2 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (130) hide show
  1. package/CHANGELOG.md +577 -0
  2. package/CODE_OF_CONDUCT.md +30 -0
  3. package/CONTRIBUTING.md +189 -16
  4. package/README.md +100 -154
  5. package/SECURITY.md +15 -13
  6. package/config.schema.json +91 -0
  7. package/dist/cli/diff.d.ts +10 -0
  8. package/dist/cli/diff.js +247 -0
  9. package/dist/cli/diff.js.map +1 -0
  10. package/dist/cli/main.js +270 -150
  11. package/dist/cli/main.js.map +1 -1
  12. package/dist/config/load.js +67 -6
  13. package/dist/config/load.js.map +1 -1
  14. package/dist/config/problems.d.ts +7 -0
  15. package/dist/config/problems.js +208 -0
  16. package/dist/config/problems.js.map +1 -0
  17. package/dist/config/resolve.d.ts +42 -109
  18. package/dist/config/resolve.js +193 -52
  19. package/dist/config/resolve.js.map +1 -1
  20. package/dist/core/directives.d.ts +7 -0
  21. package/dist/core/directives.js +25 -13
  22. package/dist/core/directives.js.map +1 -1
  23. package/dist/core/engine.d.ts +42 -4
  24. package/dist/core/engine.js +518 -79
  25. package/dist/core/engine.js.map +1 -1
  26. package/dist/core/frozen.d.ts +6 -0
  27. package/dist/core/frozen.js +15 -0
  28. package/dist/core/frozen.js.map +1 -0
  29. package/dist/core/scope.d.ts +30 -0
  30. package/dist/core/scope.js +122 -0
  31. package/dist/core/scope.js.map +1 -0
  32. package/dist/core/types.d.ts +202 -23
  33. package/dist/core/types.js.map +1 -1
  34. package/dist/index.d.ts +2 -2
  35. package/dist/index.js.map +1 -1
  36. package/dist/node.d.ts +3 -0
  37. package/dist/node.js +4 -0
  38. package/dist/node.js.map +1 -0
  39. package/dist/rules/blocks.d.ts +34 -0
  40. package/dist/rules/blocks.js +550 -0
  41. package/dist/rules/blocks.js.map +1 -0
  42. package/dist/rules/content.d.ts +2 -0
  43. package/dist/rules/content.js +85 -0
  44. package/dist/rules/content.js.map +1 -0
  45. package/dist/rules/dialects.js +10 -6
  46. package/dist/rules/dialects.js.map +1 -1
  47. package/dist/rules/links.js +302 -6
  48. package/dist/rules/links.js.map +1 -1
  49. package/dist/rules/lists.d.ts +2 -0
  50. package/dist/rules/lists.js +924 -0
  51. package/dist/rules/lists.js.map +1 -0
  52. package/dist/rules/structure.js +127 -1
  53. package/dist/rules/structure.js.map +1 -1
  54. package/dist/rules/style.d.ts +8 -2
  55. package/dist/rules/style.js +1137 -165
  56. package/dist/rules/style.js.map +1 -1
  57. package/dist/syntax/block-identifier.d.ts +18 -0
  58. package/dist/syntax/block-identifier.js +24 -0
  59. package/dist/syntax/block-identifier.js.map +1 -0
  60. package/dist/syntax/obsidian.d.ts +29 -11
  61. package/dist/syntax/obsidian.js +9 -10
  62. package/dist/syntax/obsidian.js.map +1 -1
  63. package/dist/syntax/parse.d.ts +87 -3
  64. package/dist/syntax/parse.js +247 -19
  65. package/dist/syntax/parse.js.map +1 -1
  66. package/dist/workspace/access.d.ts +12 -3
  67. package/dist/workspace/access.js +33 -5
  68. package/dist/workspace/access.js.map +1 -1
  69. package/dist/workspace/discovery.d.ts +85 -0
  70. package/dist/workspace/discovery.js +155 -0
  71. package/dist/workspace/discovery.js.map +1 -0
  72. package/dist/workspace/encoding.d.ts +24 -0
  73. package/dist/workspace/encoding.js +113 -0
  74. package/dist/workspace/encoding.js.map +1 -0
  75. package/dist/workspace/files.d.ts +92 -6
  76. package/dist/workspace/files.js +715 -59
  77. package/dist/workspace/files.js.map +1 -1
  78. package/dist/workspace/git.d.ts +54 -0
  79. package/dist/workspace/git.js +198 -0
  80. package/dist/workspace/git.js.map +1 -0
  81. package/dist/workspace/index.d.ts +58 -3
  82. package/dist/workspace/index.js +136 -37
  83. package/dist/workspace/index.js.map +1 -1
  84. package/dist/workspace/selection.d.ts +2 -2
  85. package/dist/workspace/selection.js +3 -3
  86. package/dist/workspace/selection.js.map +1 -1
  87. package/dist/workspace/slug.js +8 -1
  88. package/dist/workspace/slug.js.map +1 -1
  89. package/dist/workspace/vault.js +3 -1
  90. package/dist/workspace/vault.js.map +1 -1
  91. package/docs/api.md +239 -0
  92. package/docs/benchmarks.md +140 -200
  93. package/docs/changelog.md +1 -0
  94. package/docs/cli.md +281 -89
  95. package/docs/configuration.md +193 -480
  96. package/docs/design.md +163 -284
  97. package/docs/dialects.md +169 -0
  98. package/docs/examples.md +151 -109
  99. package/docs/index.md +21 -37
  100. package/docs/integrations.md +144 -0
  101. package/docs/obsidian.md +91 -115
  102. package/docs/plugins.md +147 -212
  103. package/docs/quick-start.md +68 -74
  104. package/docs/rules.md +1237 -0
  105. package/docs/versions.md +56 -0
  106. package/examples/README.md +17 -14
  107. package/examples/automation/summarize-report.mjs +5 -2
  108. package/examples/library/lint-folder.mjs +5 -4
  109. package/examples/library/lint-workspace.mjs +32 -0
  110. package/examples/plugin/notes/release-checklist.md +2 -2
  111. package/examples/plugin/rules.mjs +17 -23
  112. package/examples/workspaces/obsidian-vault/after/mdtools.config.jsonc +2 -2
  113. package/examples/workspaces/obsidian-vault/before/mdtools.config.jsonc +2 -2
  114. package/examples/workspaces/repository-docs/after/README.md +2 -2
  115. package/examples/workspaces/repository-docs/after/mdtools.config.jsonc +0 -2
  116. package/examples/workspaces/repository-docs/before/README.md +2 -2
  117. package/examples/workspaces/repository-docs/before/docs/guide.md +7 -7
  118. package/examples/workspaces/repository-docs/before/docs/reference.md +3 -1
  119. package/examples/workspaces/repository-docs/before/mdtools.config.jsonc +0 -2
  120. package/package.json +38 -12
  121. package/docs/contributing.md +0 -218
  122. package/docs/releases/0.1.0-beta.1.md +0 -39
  123. package/docs/releases/0.1.0-beta.2.md +0 -83
  124. package/docs/releases/0.1.0-beta.3.md +0 -61
  125. package/docs/releases/0.1.0-beta.4.md +0 -50
  126. package/docs/releases/0.1.0-rc.1.md +0 -99
  127. package/docs/releases/0.2.0-rc.1.md +0 -141
  128. package/docs/releases/0.2.0-rc.2.md +0 -225
  129. package/docs/releases.md +0 -144
  130. package/docs/testing.md +0 -93
package/docs/design.md CHANGED
@@ -1,284 +1,163 @@
1
- # Design and current limitations
2
-
3
- ## Architecture
4
-
5
- The project uses TypeScript, Node.js, and npm. mdast/micromark provide parsing
6
- and syntax extensions. The formatter retains original source and applies
7
- source-positioned edits rather than serializing the whole AST. This allows
8
- disabled formatting rules and opaque constructs to retain their spelling.
9
-
10
- `src/syntax/` owns parsing and Obsidian tokens. `src/config/` owns configuration
11
- and plugin discovery. `src/core/` owns the shared contracts, suppression
12
- directives, diagnostics, and edit engine. `src/rules/` implements built-in
13
- rules. `src/workspace/` resolves targets and handles filesystem
14
- selection/writes. `src/cli/` connects those services; the public `src/index.ts`
15
- library has no implicit filesystem access.
16
-
17
- Lint and format use the same rule settings. A style rule checks a preferred
18
- representation and proposes edits. A problem rule reports defects without
19
- inventing repairs. Inline, block, and document phases separate operations that
20
- would otherwise commonly overlap. Each phase reparses its result and compares a
21
- semantic fingerprint, using resolved destinations for link identity. Formatting
22
- must converge; otherwise the original document is returned with an error
23
- diagnostic.
24
-
25
- ## Safety boundaries
26
-
27
- The semantic fingerprint protects the supported AST model, not every
28
- application's rendered output. It normalizes ordinary prose whitespace and known
29
- callout-marker casing. It does not run Obsidian or GitHub's renderer. Syntax
30
- extensions require regression tests for their own meaning and source
31
- preservation. Tests use synthetic documents, deterministic property checks, and
32
- filesystem/CLI integration cases, including the list-collapse failure that
33
- motivated replacing heuristic reflow; [testing](testing.md) describes the
34
- suites.
35
-
36
- The fingerprint is computed in the document's dialect, so it cannot notice that
37
- the dialect itself is wrong. In CommonMark a table, or a run of footnote
38
- definitions, is an ordinary paragraph: reflow joins its lines, and the
39
- CommonMark meaning of the result is unchanged although no site renders the table
40
- any more. When nothing names a dialect, the tool therefore assumes `github`,
41
- whose syntax the Forgejo, Gitea, and Obsidian dialects build on, and not
42
- CommonMark. For a document that really is plain CommonMark this only makes
43
- formatting more careful: constructs that would have been reflowed or restyled,
44
- such as a table or a formula between `$` signs, are left alone. A vault is the
45
- exception, because no other dialect knows that Obsidian shows a single line
46
- break unless the vault's settings say otherwise: for a document inside a vault,
47
- the CLI assumes `obsidian`. A vault is a folder that contains a `.obsidian`
48
- folder. The CLI already looks for that folder to find the workspace root and the
49
- line-break setting, so no new guess about a document's content is involved. A
50
- document belongs to the nearest vault at or above its own folder, which can lie
51
- below the workspace root or above it, and the line-break setting is read from
52
- that vault. The search upward ends at a repository boundary, a folder that
53
- contains `.git`. Discovery leaves a repository inside a workspace alone, so a
54
- repository checked out inside a vault is not read as notes from the inside
55
- either; without a boundary, every project below a vault would be.
56
- Version 0.2.0-rc.1 and earlier assumed CommonMark everywhere. Link resolution
57
- follows the vault too: an Obsidian link in a vault below the workspace root
58
- reaches only that vault's files, by paths counted from the vault's folder. A
59
- workspace root inside a vault is the one layout where the tool sees only part of
60
- a vault. It does not read the rest: the root is the user's statement of what the
61
- tool may touch. A link whose target the rest of the vault could hold, or outdo,
62
- is reported as not checkable and is never rewritten, by the same marking that an
63
- unreadable directory gives. The assumption is made in one place, where
64
- configuration is resolved: the library takes the dialect to assume as an
65
- argument, since it does not read files, and anything that names a dialect
66
- replaces it. The workspace index uses `github` for link targets when a library
67
- caller gives it no dialect.
68
-
69
- Obsidian callout headers remain intact; supported body prose can reflow directly
70
- below the header or in later paragraphs. The semantic fingerprint also protects
71
- the title/body boundary. Lazy quote continuations and inline syntax spanning
72
- that boundary remain untouched. Highlight, comment, math, code, HTML, and
73
- front-matter contents are preserved. Standalone Obsidian comment blocks may span
74
- blank lines. Inline comments spanning multiple paragraphs are not fully modeled;
75
- suppress formatting around those constructs or use standalone comment blocks.
76
-
77
- The source-preserving wrapper supports ordinary paragraphs, list items, and
78
- blockquote containers. It preserves explicit hard-break paragraphs, lazy/unusual
79
- continuations it cannot safely reconstruct, and paragraphs carrying block IDs.
80
- Long unbreakable atoms can exceed the width. Intraword emphasis retains
81
- asterisks when underscores would change parsing. Formatting width is a target,
82
- not a license to split protected syntax.
83
-
84
- Dollar math is read by a tokenizer of the project's own, with the conditions
85
- that each renderer was observed to apply to `$…$` in running text. The rule for
86
- choosing them is one-sided on purpose. Text that the tool wrongly takes for math
87
- is only left alone, while a formula taken for prose could be rewrapped or have
88
- its `*` restyled. A condition is therefore applied to a dialect only when its
89
- renderer was seen to apply it, and where a renderer is stricter still, the tool
90
- keeps protecting the text. The observations, about 125 cases rendered by GitHub,
91
- Forgejo, and Gitea, are kept with the tests and are compared with the tokenizer
92
- on every run. Obsidian has no renderer that a test or a script can ask, so its
93
- conditions rest on 22 cases that a person looked at in Reading view and in Live
94
- Preview. Where those two views disagree, the dialect follows the one that shows
95
- more as a formula, and what was not looked at is still read as math.
96
-
97
- GitHub alert support normalizes known alert type markers. Footnotes, math,
98
- strikethrough, and autolinks are parsed, but do not each have dedicated style
99
- rules. Literal autolinks come from the syntax extension only. GitHub applies a
100
- second, transform-time autolink pass to files, which links cases such as
101
- `[www.example.com]`, but the GFM tree transform implementing it produces nodes
102
- without source positions, and one such URL made the engine refuse the whole
103
- document. No built-in rule acts on literal autolinks, so formatted output is the
104
- same either way; plugins inspecting link nodes do not see those autolinks.
105
-
106
- Forgejo and Gitea documents parse as GitHub Markdown. Their additional syntax is
107
- protected rather than modeled: paragraphs containing a definition description
108
- line or a `\[` display-math line are not reflowed, `\(...\)` math and `[[...]]`
109
- shortlinks on one line are unbreakable atoms, and a pair split across lines
110
- protects its paragraph because reflow could join it. Emphasis markers inside a
111
- one-line `\(...\)` pair are not changed, and code spans are not joined where
112
- that would put `\(` and `\)` on one line. Gitea parses the backslash math
113
- delimiters only when its instance enables them, so they are protected
114
- regardless. Gitea's exception for underscores before `.py` is protected the same
115
- way: on a line containing `_.py`, emphasis markers, multiline code spans, and
116
- link destinations whose rewrite would change their underscores are left
117
- unchanged, and such a paragraph is reflowed only when all its underscores are
118
- inside code spans and it contains no `$` or `\(` math. Shortlinks are not
119
- resolved as links. Legacy `> **Note**` callouts, table-of-contents front matter,
120
- emoji shortcodes, color previews, and issue or commit references are preserved
121
- unchanged and not modeled. Table alignment currently applies only to top-level
122
- tables and preserves cell contents. Embedded code formatting, metadata mutation,
123
- document generation, external URL fetching, and file renames are outside this
124
- release.
125
-
126
- A Forgejo or Gitea heading's trailing attribute block (`## Title {#id .class}`)
127
- is protected from formatting, because both renderers read it before they parse
128
- inline content. Whether a heading ends in a valid block, and what the block
129
- says, depends on every character from the first `{` of the heading's last text
130
- line on, so emphasis markers and link destinations from there on are left
131
- unchanged. A code span spanning lines of a heading is not joined, because the
132
- block is read from the heading's last text line. A Setext heading converted to
133
- ATX gets a `#` run in front of its block escaped, which would otherwise close
134
- the heading. For these two dialects the semantic fingerprint includes each
135
- block's source text, so an edit from any other rule or plugin that changes a
136
- block, makes one invalid, or creates one is refused. That covers a block
137
- followed by a closing `#` sequence (`## Title {#id} ##`) as well, which only
138
- Gitea before 1.26 reads as attributes. Forgejo, and Gitea before 1.26, also
139
- generate the anchor from that line as written, so the marker and link
140
- destination rules leave the last line of a heading without a custom `id`
141
- unchanged on these dialects, and the fingerprint includes the Forgejo and Gitea
142
- anchors generated from it.
143
-
144
- Obsidian links resolve against indexed paths, extensionless Markdown candidates,
145
- and unique suffixes for internal links. Duplicate candidates remain ambiguous.
146
- Heading fragments use exact text; GitHub uses slugged headings including
147
- duplicate suffixes, and Forgejo uses its own anchor algorithm, which collapses
148
- punctuation runs and keeps Unicode letters. Gitea deletes punctuation like
149
- GitHub but also drops combining marks; Gitea fragments also accept the duplicate
150
- suffixes that versions before 1.26 generated and, for headings containing
151
- `_.py`, the anchor of the literal text that 1.26 and later show. Inline HTML in
152
- a heading contributes its text, but not its tags or comments, to GitHub anchors,
153
- except for the tags that GitHub's tag filter shows as text, such as `<script>`.
154
- Forgejo, and Gitea before 1.26, build anchors from the source of the heading's
155
- last line, with link destinations, emphasis markers, character references, and
156
- tags included and `#` sequences, container prefixes, and a trailing attribute
157
- block left out; Gitea 1.26 and later use the rendered text of every line, in
158
- which `<script>`, `<style>`, `<html>`, and `<head>` tags are shown as text and a
159
- hard line break is a line break, so Gitea fragments accept both. Character
160
- references inside a filtered tag on GitHub are not modeled. Nor, on Gitea, are
161
- the number a footnote reference shows in a heading, or tags or Markdown syntax
162
- inside `<textarea>`, `<plaintext>`, and the other elements whose content HTML
163
- reads as plain text, including an unclosed one, which takes in the rest of the
164
- document. Forgejo and Gitea fragments read a heading's trailing attribute block
165
- (`## Title {#id .class}`) by the rules of goldmark 1.8, which Forgejo 16 and
166
- Gitea 1.26 and later use: an `id` is the heading's only anchor, matched as
167
- written and never numbered, and any other valid block is left out of the text
168
- that generates the anchor. That text comes from a second parse of the target
169
- with the blocks blanked out, because goldmark removes a block before it parses
170
- the heading's inline content. The block stays heading text for other dialects,
171
- rules, and plugins. An `id` written with the `user-content-` prefix that the
172
- renderers add themselves is matched without it. Not modeled are the older
173
- attribute rules of Gitea before 1.26, which for example also read a block
174
- followed by a closing `#` sequence, and attribute values nested more than 64
175
- levels deep, which leave the block as heading text. A heading whose `id` is not
176
- a string (`{id=5}`) has no anchor on Gitea 1.26 and later and makes Forgejo, and
177
- Gitea before 1.26, render nothing for the document; `forgejo/heading-id` reports
178
- it on both dialects, and the anchors are those of Gitea 1.26 and later. Block
179
- identifiers are indexed from trailing text markers. Heading nesting paths,
180
- property aliases, PDF subpaths, and every Obsidian plugin's syntax are not fully
181
- supported. Embeds are preserved as embeds; image dimensions in their aliases
182
- remain intact.
183
-
184
- GitHub, Forgejo, and Gitea store every anchor with the prefix `user-content-`
185
- and add it to a link's fragment unless the fragment already starts with it. A
186
- fragment written with the prefix therefore reaches the same anchor as one
187
- without, and is matched without it; Obsidian fragments stay literal. The three
188
- also keep the `id` attribute of every element and the `name` of an `<a>` in a
189
- document's HTML, prefixed the same way, so those are fragments too, read from
190
- the inline and block HTML with comments dropped; GitHub lowercases them and
191
- matches any fragment lowercased as well, including a heading's. They do not
192
- number later headings. The scan follows HTML tokenization: comments, unclosed
193
- ones to the end of their block, declarations, and the content of `<script>`,
194
- `<textarea>`, and the other raw-text elements are skipped, a `<!--` inside a
195
- quoted attribute value is part of the value, and every value of a repeated
196
- attribute counts, since the renderers disagree on which one they keep. Entity
197
- references in such attributes are not decoded. GitHub and Gitea 1.26 and later
198
- also generate anchors for headings written as HTML, from their text content with
199
- inner tags removed and character references decoded; GitHub numbers them in
200
- document order together with the Markdown headings and also when the tag has an
201
- `id`, Gitea numbers nothing and skips a tag with an `id`, and Forgejo generates
202
- none. Headings inside HTML blocks and inline in a paragraph both count, a
203
- heading's start tag ends a heading that is still open, and one left open at the
204
- end of its HTML is not reported, although the renderers continue it through the
205
- Markdown that follows. Numeric character references are decoded with or without
206
- a semicolon; named ones only with it. GitHub's treatment was taken from a live
207
- README and from the heading filter its rendering derives from, not from a page
208
- with repeated or `id`-bearing HTML headings.
209
-
210
- The workspace indexes eligible paths first and parses target Markdown only when
211
- a fragment is checked. For the rest of that invocation it keeps the target's
212
- anchors and nothing else: the parsed document, which takes many times the memory
213
- of its source, is released as soon as the anchors are known. Versions up to
214
- `0.2.0-rc.1` kept every parsed target until the run ended, so memory grew with
215
- the number of linked documents. Anchors form three groups, one for Obsidian, one
216
- for GitHub and CommonMark, and one for Forgejo and Gitea. A group is computed
217
- the first time a link is checked against one of its dialects, so a GitHub or
218
- Obsidian workspace never reads attribute blocks, which need a second parse, or
219
- the Forgejo and Gitea anchors, and the forges never number anchors GitHub's way.
220
- The command line checks each target against one group. A library host that
221
- checks a target against a second group has it read and parsed once more, which
222
- is cheaper overall than keeping every parsed document or computing every group
223
- for every target. CLI discovery also defers reading Markdown until it is
224
- selected or needed for a fragment. Built-in rule-schema validators are reused
225
- across documents. A suffix lookup index is built on first use instead of
226
- scanning every path for each shortened link. There is no persistent cache,
227
- worker pool, watch service, editor extension, or language server yet. The
228
- library has a pluggable workspace interface for specialized hosts.
229
-
230
- Discovery skips a directory that the system refuses to let it read (`EACCES` or
231
- `EPERM`) and carries on; every other read error is treated as a fault and stops
232
- the run. A directory is read as a whole or not at all: one that can be listed
233
- but not entered, or entered but not listed, is skipped like one that allows
234
- neither, although the first reveals file names and the second would open a file
235
- whose name is known. The index then holds the readable files only. Links whose
236
- path leads into a skipped directory resolve as unreadable rather than missing.
237
- Obsidian's search by note name cannot be made exact: a name with no readable
238
- match is reported as unchecked, a name with one readable match is accepted but
239
- not rewritten, like a note found in a place that Obsidian tries only after a
240
- skipped directory, and a name with several is ambiguous either way. A file is
241
- read only when it is processed or when a link's fragment needs its content, so a
242
- refusal is met late: a selected file becomes an `engine/unreadable-file`
243
- diagnostic that blocks the batch's write phase, and a link target makes the
244
- fragment unchecked for the documents that link to it.
245
- [Paths that cannot be read](configuration.md#paths-that-cannot-be-read)
246
- describes the behavior.
247
-
248
- `npm run benchmark` measures parsing, linting, and formatting of synthetic
249
- repository documentation and a synthetic vault, through the library and through
250
- the CLI, and prints an output hash for comparing implementations.
251
- [Benchmarks](benchmarks.md) explains each measurement and records reference
252
- results. They do not predict a particular workspace's throughput. Measure the
253
- original workload before adding caches or workers.
254
-
255
- Formatting retains the current parsed document within one call, reusing it
256
- between phases and for final diagnostics. Every changed candidate is parsed and
257
- checked against the original semantic fingerprint before becoming the next
258
- phase's input; the original fingerprint is computed only once an edit produces a
259
- candidate. The final pass, in which no phase changes the document, reuses its
260
- style findings and runs only the remaining rules for the final diagnostics.
261
- Suppression directives are scanned only when the source contains their prefix,
262
- and rules traverse the tree with a plain recursive walk rather than a
263
- closure-per-node visitor. No-op edits still undergo range and overlap
264
- validation. This retains the plugin contract that rules must not mutate
265
- documents.
266
-
267
- The benchmark's library measurements compare formatting documents that need
268
- edits with formatting their already-formatted output. They follow a warm-up run,
269
- check outputs and diagnostics, and exclude discovery, I/O, CLI serialization,
270
- and custom plugins; the command-line measurements include them. Run the
271
- benchmark without competing workloads.
272
-
273
- ## Maintenance priorities
274
-
275
- Prioritize reproducible document corruption and renderer disagreement over new
276
- style options. Add a minimal regression case before a fix; verify that disabling
277
- the affected rule preserves the source. Broaden dialect fixtures and resolver
278
- compatibility before promising complete application parity. Benchmark real
279
- synthetic vault sizes before adding caching or worker processes.
280
-
281
- Future changes can add richer link policies, reference-definition formatting,
282
- more container layout, versioned plugin contracts, and optional standalone
283
- distribution. Each should extend the shared configuration and edit contracts
284
- instead of creating a second formatter pipeline.
1
+ # How it works and its limits
2
+
3
+ ## Safety model
4
+
5
+ Formatting does not print a document again from its syntax tree. Each style rule
6
+ proposes edits at positions in the original source, and everything outside those
7
+ edits stays byte for byte as it was. A rule that is turned off therefore leaves
8
+ its constructs as they were written, and syntax that the tool does not
9
+ understand survives unchanged.
10
+
11
+ The edits are applied in three phases, inline, block, and document, so that
12
+ edits that commonly overlap do not meet in one pass. After each phase that
13
+ changes the document, the tool parses the result again and compares its meaning
14
+ with the original: the syntax tree with the whitespace of prose normalized and
15
+ each link compared by the file it resolves to. When the meaning differs, the
16
+ phase tries the edits one rule at a time and leaves out the edits of each rule
17
+ that would change it, all of them, because one edit can depend on another; the
18
+ other rules' edits are kept. Each rule left out gets an `engine/unsafe-format`
19
+ error at the line where the meaning would first differ, as in
20
+ `style/wrap would change the parsed meaning at line 12; none of its edits were applied.`
21
+ Of two edits that overlap, the one that starts first is applied, and the other
22
+ rule runs again in the next pass. The phases repeat until nothing changes, for
23
+ at most eight passes. When rules undo each other or formatting does not settle,
24
+ `format` returns the document as it was with an `engine/unsafe-format` error
25
+ that names them. Either way the run ends with exit code `2`, and
26
+ `format --write` still writes the other documents, and this one when the edits
27
+ that were kept changed it.
28
+
29
+ The comparison is only as good as the parser's reading:
30
+
31
+ - It compares syntax trees, and does not run GitHub's, Forgejo's, Gitea's, or
32
+ Obsidian's renderer.
33
+ - It reads the document in its dialect, so it cannot notice that the dialect is
34
+ wrong. In CommonMark a table is an ordinary paragraph, and joining its lines
35
+ changes nothing that CommonMark sees. This is why the tool [assumes a
36
+ dialect](dialects.md#the-assumed-dialect) with more syntax, not less, when
37
+ none is named.
38
+ - The parser, micromark, follows CommonMark closely but not everywhere. Directly
39
+ below a paragraph, a list item that begins with another list marker, as in
40
+ `- 2. Step` or `- -`, is read as a list item holding text, where CommonMark
41
+ and GitHub read a nested list ([issue
42
+ #191](https://github.com/viell-dev/slop-markdown-tools/issues/191)).
43
+ Formatting cannot change such a document's meaning, because the comparison
44
+ uses the same reading before and after, but rules see text where GitHub shows
45
+ a list, and the rules that would change such a place leave it as written.
46
+ Below an empty list item (`-`), the parser takes the `[x]` that begins the
47
+ next line for a checkbox that belongs to no item, where GitHub shows the text
48
+ `[x]`, so a blank line between them, as `style/block-spacing` would add, is
49
+ refused as a change of meaning. The parser also keeps the spaces and tabs that
50
+ start a code span's next line, where CommonMark and GitHub leave them out, so
51
+ that a span `npm run` continued by an indented `build` reads
52
+ `npm run build`, and it takes a space or line ending off each end of a span
53
+ before its line endings become spaces, where CommonMark takes spaces off
54
+ after. The comparison reads a code span as GitHub shows it, also where
55
+ renderers differ: GitHub keeps the indentation of a lazy continuation line in
56
+ the span, and commonmark.js does not. `style/inline-code` and `style/wrap`
57
+ join a span to that text ([issue
58
+ #265](https://github.com/viell-dev/slop-markdown-tools/issues/265), [issue
59
+ #285](https://github.com/viell-dev/slop-markdown-tools/issues/285), [issue
60
+ #290](https://github.com/viell-dev/slop-markdown-tools/issues/290)). Directly
61
+ below a table that ends in a list item or block quote, the parser reads the
62
+ lines of another table as text, where GitHub shows a table, so `style/wrap`
63
+ and `style/block-spacing` leave text whose second line is a table delimiter
64
+ row as written ([issue
65
+ #283](https://github.com/viell-dev/slop-markdown-tools/issues/283)).
66
+ - Front matter needs a closing line. A first line of `---` or `+++` that no
67
+ later line closes is read as renderers read it, as a thematic break or text,
68
+ with the lists and block quotes below it; the parser's front matter extension
69
+ reads them as paragraphs there, so the tool leaves it out for such a document
70
+ ([issue #278](https://github.com/viell-dev/slop-markdown-tools/issues/278)).
71
+ - Front matter ends at the next `---` or `+++` line, also one inside a code
72
+ block, for the parser, for site generators, and for GitHub's file view when
73
+ the front matter is a YAML mapping. GitHub's comments, and renderers without
74
+ front matter, read the code block. When a line of the front matter opens a
75
+ code block, an HTML block, or display math, and the closing line is content of
76
+ that block in the other reading, the tool reads the document without front
77
+ matter, and an edit has to keep the meaning of both readings. In such a
78
+ document, a rule whose edits would change either, such as joining the lines of
79
+ the front matter, is refused ([issue
80
+ #295](https://github.com/viell-dev/slop-markdown-tools/issues/295)).
81
+
82
+ Which `$…$` is math was observed on the renderers themselves: about 125 cases
83
+ rendered by GitHub, Forgejo, and Gitea, which the tests compare with the tool on
84
+ every run, and 22 cases that a person looked at in both of Obsidian's views. The
85
+ rule is one-sided on purpose: text taken for math is only left alone, while a
86
+ formula taken for prose could be wrapped or restyled, so a condition is applied
87
+ to a dialect only when its renderer was seen to apply it.
88
+
89
+ ## Limits
90
+
91
+ Formatting:
92
+
93
+ - A code span, an image, a link's destination, or a word longer than the line
94
+ stays whole and exceeds the width. The width is a target, not a license to
95
+ split syntax.
96
+ - Paragraphs that wrapping cannot reconstruct safely are left as written;
97
+ [`style/wrap`](rules.md#stylewrap) lists them, and its `reportUnreflowed`
98
+ option reports them.
99
+ - Emphasis inside a word keeps `*` when the configured marker is `_`, because
100
+ `_` cannot mark emphasis there.
101
+ - Tables are aligned only outside lists and block quotes.
102
+ - Footnotes, strikethrough, and literal autolinks are read but have no rules of
103
+ their own. Reference definitions are checked but not rewritten.
104
+ - There is no formatting of code inside code blocks, no change of front matter
105
+ or other metadata, no generation of documents, and no renaming of files.
106
+
107
+ Files:
108
+
109
+ - Only UTF-8 text is read. Another encoding is not guessed, because a guess can
110
+ be wrong and the renderers read Markdown as UTF-8, and it is not decoded with
111
+ replacement characters, which the next edit would write into the file. See
112
+ [files that cannot be read](cli.md#files-that-cannot-be-read-or-written).
113
+ - `format --write` writes into the existing file, which keeps its permissions
114
+ and attributes but is not atomic: a crash or power failure during the write
115
+ can leave the file incomplete; see [what writing
116
+ keeps](cli.md#what-writing-keeps).
117
+
118
+ Syntax:
119
+
120
+ - Obsidian inline comments that span several paragraphs are not fully read;
121
+ suppress formatting around them or use comment blocks. Heading nesting paths
122
+ in links, property aliases, PDF subpaths, and the syntax of community plugins
123
+ are not supported.
124
+ - Forgejo's and Gitea's own syntax, such as definition lists, backslash math,
125
+ and shortlinks, is protected rather than read, and shortlinks are not checked
126
+ as links.
127
+ - Legacy `> **Note**` callouts, table-of-contents front matter, emoji
128
+ shortcodes, color previews, and issue or commit references are kept as text.
129
+ - GitHub reads an escaped `\$` as the start of a formula; the tool does not.
130
+ - GitHub also links bare addresses such as `[www.example.com]` in a second pass
131
+ that leaves no source positions. The tool does not read those links, so
132
+ plugins do not see them; no built-in rule acts on them.
133
+
134
+ Links:
135
+
136
+ - Links to the web are never fetched.
137
+ - Obsidian's resolution by note name is not a complete copy of Obsidian's: a
138
+ missing or ambiguous target is reported, never guessed. A directory is never
139
+ rewritten to a README or index note.
140
+ - When a directory cannot be read, a search by note name cannot be exact. A link
141
+ that finds no readable note is reported as not checkable, and one that finds a
142
+ single note is accepted but never rewritten.
143
+ - With the workspace root inside a vault, links to notes outside the root cannot
144
+ be checked; see [Obsidian
145
+ vaults](obsidian.md#a-workspace-root-inside-a-vault).
146
+ - Heading anchors are modeled as each renderer generates them, with these
147
+ exceptions: character references inside tags that GitHub shows as text; on
148
+ Gitea, the number of a footnote reference in a heading, and Markdown inside
149
+ `<textarea>`, `<plaintext>`, and other elements whose content HTML reads as
150
+ text; the attribute rules of Gitea before 1.26; attribute values nested more
151
+ than 64 levels deep; and character references in HTML `id` and `name`
152
+ attributes, which are not decoded.
153
+
154
+ Scale:
155
+
156
+ - The tool uses one processor core and keeps nothing between runs: there is no
157
+ cache, worker pool, watch mode, editor extension, or language server. Start-up
158
+ takes about a quarter of a second.
159
+ - While a document is parsed, it needs about 70 to 130 times its size in memory,
160
+ and a single paragraph of 100 kB or more full of inline syntax parses slowly;
161
+ one made of tens of thousands of square brackets can take seconds from about
162
+ 50 kB. [Benchmarks](benchmarks.md#limits-of-these-numbers) has the
163
+ measurements.