@portabletext/markdown 2.0.0 → 2.2.0

package/README.md

@@ -56,6 +56,18 @@ const markdown = portableTextToMarkdown([
 # Hello **world**
 ```
 
+**Edit through Markdown without losing keys**
+
+```ts
+import {applyMarkdownEdit, portableTextToMarkdown} from '@portabletext/markdown'
+
+const markdown = portableTextToMarkdown(stored)
+const editedMarkdown = markdown.replace('tomorow', 'tomorrow')
+const edited = applyMarkdownEdit(stored, editedMarkdown)
+// same content as parsing editedMarkdown, with the stored `_key`s
+// kept the way the same edit in an editor would have kept them
+```
+
 ## Supported features
 
 | Feature          | Markdown → Portable Text | Portable Text → Markdown |
@@ -81,32 +93,28 @@ const markdown = portableTextToMarkdown([
 
 ## Round-trip behavior
 
-1. Translation preserves semantics, not source spelling. The first MD→PT→MD pass normalizes Markdown to one canonical spelling: autolinks and reference links become inline links, indented code becomes fenced code, and emphasis, headings, lists, and tables each get one canonical form.
-
-   ```
-   <https://portabletext.org>  ->  [https://portabletext.org](https://portabletext.org)
-   [ref link][id]              ->  [ref link](https://example.com "title")
-   ```
+Converting Markdown to Portable Text and back isn't a lossless mirror:
 
-2. The normalized Markdown is a fixpoint for the constructs in the [Supported features](#supported-features) table above: parsing it and serializing again reproduces it byte-for-byte, pinned by a full-document round-trip test. This doesn't yet extend to plain text that happens to contain literal Markdown punctuation: serialization doesn't escape it, so a second parse reads it back as markup instead of literal text.
+1. Translation preserves semantics, not source spelling: the first MD→PT→MD pass normalizes Markdown to one canonical spelling (autolinks become inline links, indented code becomes fenced code, soft-wrapped lines join into one, and so on).
+2. The normalized Markdown is a fixpoint for plain text and the [Supported features](#supported-features) table: parsing it and serializing again reproduces it byte-for-byte.
+3. MD→PT survival is schema-driven: a construct whose type the schema doesn't declare keeps its content and drops the structure that named it.
+4. PT structures with no Markdown form degrade predictably on PT→MD (extra table header rows flatten into the body, deep or level-skipping lists collapse to relative nesting, unknown marks pass their text through unformatted). Unknown object types round-trip instead: block-level as a ` ```json:object ` fence, inline as a `json:object`-tagged code span, both carrying the value as JSON. A fence or span whose body isn't a JSON object with a `_type` is ordinary code.
+5. Identity does not round-trip for text blocks: keys are regenerated on every parse, and adjacent spans with identical marks merge into one. Unknown objects keep their `_key`. [`applyMarkdownEdit`](#applymarkdownedit) restores stored keys after an edit.
+6. A hard break and a `\n` in a span's text are exclusive counterparts in both directions: a `\n` always renders as hard-break syntax on the way out, and hard-break syntax always becomes `\n` on the way in, never the space a soft wrap joins with.
 
-   ```
-   \*bar\*  ->  *bar*  (serialized unescaped; a second parse reads this as emphasis)
-   ```
+The named exceptions to the fixpoint claim: an explicit-scheme URL or email keeps its text but gains a `link` mark on reparse, and a fuzzy `www.` form does too unless it carries markdown-significant punctuation; a hard break inside a heading splits into a second block on reparse, since an ATX heading is single-line; leading or trailing whitespace that CommonMark's own block parsing trims isn't part of the fixpoint; a `code` object with the reserved language `json:object` loses that language on serialization; and span text ending in `json:object` directly before a code-marked span holding a typed JSON object binds into an inline object on reparse.
 
-3. MD→PT survival is schema-driven. Constructs whose type the schema doesn't declare degrade predictably: they keep their content and drop the structure that named them. Marks drop formatting but keep the text (`**bar**` with no `strong` decorator in the schema becomes a plain span reading `bar`); tables flatten their cell content into top-level blocks; images fall back to their Markdown source as plain text; task-list checkboxes strip to plain list items.
-
-4. PT structures with no Markdown form degrade predictably on PT→MD. GFM tables have one header row, so header rows beyond the first flatten into the body. Deep or level-skipping lists collapse to relative nesting. A list's first item renders at the top level whatever its `level`, and each deeper jump between items indents one step, however many levels it skips. Multi-block table cells join their blocks with spaces. Unknown object types render as a fenced JSON block; unknown marks pass their text through unformatted.
-
-5. Identity does not round-trip. Keys are regenerated on every parse, and adjacent spans with identical marks merge into one.
+See [Markdown round-tripping](https://www.portabletext.org/conversion/markdown-round-tripping/) on the docs site for the full contract and worked examples.
 
 ## Usage
 
-<!-- The schema table, matcher table, supported-features table, and
-     round-trip section have condensed twins on the docs site
+<!-- The schema table, matcher table, and supported-features table have
+     condensed twins on the docs site
      (apps/docs/src/content/docs/conversion/markdown-to-portable-text.mdx).
      Keep them in sync: a claim corrected in one place is stale in the
-     other. -->
+     other. The Round-trip behavior section above is a summary of
+     apps/docs/src/content/docs/conversion/markdown-round-tripping.mdx,
+     which is canonical; keep the two in sync the same way. -->
 
 ### `markdownToPortableText`
 
@@ -270,6 +278,8 @@ The default code block matcher requires the schema type to have a `'code'` field
 
 **Links** support optional titles using `[text](url "title")` syntax. The title is captured in the `'title'` field of the `'link'` annotation.
 
+**Line breaks**: soft-wrapped lines within a paragraph join with a single space; a hard break (two or more trailing spaces, or a backslash, before the newline) becomes a line break within the block.
+
 **Nested lists** are handled automatically. Each list item block includes a `level` property indicating its nesting depth (1 for top-level, 2 for nested, etc.).
 
 **HTML blocks** (like `<div>...</div>`) become `'html'` block objects with the raw HTML in the `'html'` field. Inline HTML is controlled by the `html.inline` option.
@@ -299,7 +309,7 @@ markdownToPortableText(markdown, {
 })
 ```
 
-> **Note:** Checking if the type exists in the schema isn't required, but it's good practice. Returning `undefined` gracefully skips unsupported types.
+> **Note:** Checking if the type exists in the schema isn't required, but it's good practice. Returning `undefined` skips unsupported types.
 
 **Table matcher:** GFM pipe tables convert by default (see [Default behavior](#default-behavior)). Provide your own matcher to map onto a differently-shaped `table` type:
 
@@ -417,6 +427,39 @@ markdownToPortableText(markdown, {
 })
 ```
 
+#### Reporting degradation
+
+Every construct the schema can't represent (a decorator not in the schema, a table with no `table` block object, a task checkbox with no `task` list, and so on) degrades to a lossier shape rather than throwing. A ` ```json:object ` fence or tagged code span that fails to reconstruct (invalid JSON, or no string `_type`) reports too, as `object-carrier-invalid`, even on a schema that would otherwise accept everything: it's the payload that's unusable, not the schema. `onDegradation` observes both, one callback covering all uses. Left unset, the conversion stays silent and returns the lossiest representation it can build, since a library shouldn't log on its own initiative. Passed a function, observe the losses: it's called once, after the whole document has been walked, only when at least one construct degraded, with a report object holding every `Degradation` in encounter order and a canonical grouped `message`. Enforce against lossy output by throwing your own error from inside that callback; the throw propagates out of `markdownToPortableText`.
+
+```ts
+markdownToPortableText(markdown, {
+  onDegradation: ({degradations, message}) => {
+    // degradations: every Degradation, in encounter order
+    // message: the same degradations grouped, snippeted, and sorted by line
+    logger.warn(message)
+  },
+})
+```
+
+Each `Degradation` carries `type`, a human-readable `message`, `line` when a source line is available, and `snippet` (the offending construct's text, truncated to 40 characters) when there's a specific piece of source text to quote. Match on `type`, not `message`: the type is the stable contract, the message can change between releases.
+
+Enforcing means throwing your own error from inside the callback:
+
+```ts
+markdownToPortableText('# heading\n\n**bold**', {
+  schema: compileSchema(defineSchema({})),
+  onDegradation: ({message}) => {
+    throw new Error(message)
+  },
+})
+// throws Error:
+// Markdown could not be converted without loss:
+// - line 1: `#` heading became a normal paragraph: the schema has no `h1` style ("heading")
+// - line 3: Removed bold formatting, kept the text: the schema has no `strong` decorator ("bold")
+```
+
+Repeated declines of the same kind (the same missing decorator on three spans, say) collapse into one line of `message` instead of repeating the sentence: `- Removed bold formatting, kept the text: the schema has no \`strong\` decorator (3×: "a", "b", "c")`. `degradations` stays ungrouped, one entry per occurrence.
+
 ### `portableTextToMarkdown`
 
 ```ts
@@ -506,7 +549,7 @@ The conversion is driven by **Renderers**: functions that render Portable Text e
 
 Unknown types render as JSON code blocks by default; unknown styles, list items, and marks pass through their children.
 
-The default type renderers are collision-safe: because the serializer dispatches on the `_type` name alone, `code`, `html`, `image`, `callout`, and `table` fall back to the `unknownType` renderer (a JSON code block) when a value doesn't match the shape their renderer expects (say, your own differently-shaped `code` type); `horizontal-rule` has no shape to check and always renders `---`. Register your own `types.<name>` renderer to override how any of them serialize, or to handle a same-named type of a different shape.
+The default type renderers are collision-safe: because the serializer dispatches on the `_type` name alone, `code`, `html`, `image`, `callout`, and `table` fall back to the `unknownType` renderer (a JSON code block) when a value doesn't match the shape their renderer expects (say, your own differently-shaped `code` type); `horizontal-rule` has no shape to check and always renders `---`. `image` also falls back when `src` is a string a Markdown parser would refuse (a `javascript:`/`vbscript:`/`file:` URI, or a `data:` URI outside `png`/`gif`/`jpeg`/`webp`), so the value survives as a `json:object` fence instead of reparsing as literal text. Register your own `types.<name>` renderer to override how any of them serialize, or to handle a same-named type of a different shape.
 
 > **Note:** The `underline` renderer is included for Portable Text that uses it, but there's no standard Markdown syntax for underline, so it renders as HTML.
 
@@ -568,17 +611,19 @@ portableTextToMarkdown(blocks, {
 
 #### What renderers receive
 
+One question decides which text field a renderer reads: **are you wrapping markdown, or emitting something else?** `children` is markdown: nested marks are rendered and literal punctuation is backslash-escaped, so it embeds in markdown output as-is (`` `**${children}**` ``). `text` is source: the raw span text exactly as stored, for renderers that emit their own delimiters or non-markdown output (code fences, HTML tags), where escapes would appear literally. The built-in `code` decorator renderer reads `text`; every other built-in reads `children`.
+
 **Block renderers** (`block.*`):
 
 - `value` – the block object
-- `children` – rendered content of the block
+- `children` – rendered content of the block, as markdown (escaped)
 - `index` – position in the blocks array
 
 **Mark renderers** (`marks.*`):
 
 - `value` – the mark definition (for annotations like links)
-- `children` – the rendered marked content
-- `text` – the raw text content (without nested mark rendering)
+- `children` – the rendered marked content, as markdown (escaped)
+- `text` – the raw text content (unescaped, no nested mark rendering)
 - `markType` – the mark type name
 - `markKey` – the mark's key (for annotations)
 
@@ -633,9 +678,43 @@ portableTextToMarkdown(blocks, {
 })
 ```
 
-By default, unknown types render as JSON code blocks, and unknown marks/styles pass through their children unchanged.
+By default, unknown types render as `json:object` fences or tagged code spans that round-trip (see [Round-trip behavior](#round-trip-behavior)), and unknown marks/styles pass through their children unchanged.
+
+#### Gating default renderers on a schema
+
+Going to convert the markdown back with `markdownToPortableText`? Pass the same `schema` to both, and nothing the schema can't rebuild becomes markdown that gets destroyed on the way back: undeclared types travel as `json:object` fences that reparse to the same value.
+
+```ts
+import {compileSchema, defineSchema} from '@portabletext/schema'
+
+const schema = compileSchema(
+  defineSchema({
+    blockObjects: [
+      {
+        name: 'code',
+        fields: [
+          {name: 'code', type: 'string'},
+          {name: 'language', type: 'string'},
+        ],
+      },
+    ],
+  }),
+)
+
+portableTextToMarkdown(blocks, {schema})
+```
+
+A default renderer (`callout`, `code`, `horizontal-rule`, `html`, `image`, `table`) runs only when the schema declares that type at the position the node appears in: `blockObjects` for a block, `inlineObjects` for an inline object. An `image` declared in only one of the two still falls back to `unknownType` at the other position.
+
+The gate reads type names, never field values: declaring a type doesn't validate anything, and a value's fields play no part in which renderer runs. Fields matter on the parse side instead: `markdownToPortableText` filters a construct down to its declared fields, so declare each type with the fields its values carry, or the markdown forms this gate lets through come back rebuilt without them.
+
+An undeclared type falls back to `unknownType`, whose default output is the same `json:object` fence or tagged code span described above, so it round-trips at block and inline positions. Inside a table cell the `json:object` form is the inline code span (a GFM cell is one line, and a fence would be squashed), so an undeclared object in a cell survives too; declared types whose markdown form spans multiple lines (a code block in a cell) still flatten on reparse. Renderers you register in `types` bypass the gate entirely, whether or not the schema declares them.
+
+Without a `schema`, every default renderer stays active.
 
-You can also customize hard break rendering:
+#### Hard breaks
+
+Customize how a hard break (a `\n` inside a span's text) renders:
 
 ```ts
 portableTextToMarkdown(blocks, {
@@ -664,6 +743,102 @@ portableTextToMarkdown(blocks, {
 })
 ```
 
+### `applyMarkdownEdit`
+
+Parsing markdown mints fresh `_key`s for text blocks (see [Round-trip behavior](#round-trip-behavior)), so converting a document to markdown, editing one word, and converting back returns what looks like a full rewrite: comment anchors detach, history churns, and granular patching is impossible. `applyMarkdownEdit` converts edited markdown back to Portable Text and restores stored `_key`s the way the same edit in an editor would have kept them:
+
+```ts
+import {applyMarkdownEdit, portableTextToMarkdown} from '@portabletext/markdown'
+
+const stored = [
+  {
+    _type: 'block',
+    _key: 'b1',
+    style: 'normal',
+    children: [{_type: 'span', _key: 's1', text: 'Ships tomorow.', marks: []}],
+    markDefs: [],
+  },
+]
+
+const markdown = portableTextToMarkdown(stored)
+// markdown === 'Ships tomorow.'; an agent (or anything else) fixes the typo
+const edited = applyMarkdownEdit(stored, 'Ships tomorrow.')
+```
+
+`edited`:
+
+```json
+[
+  {
+    "_type": "block",
+    "_key": "b1",
+    "style": "normal",
+    "children": [
+      {"_type": "span", "_key": "s1", "text": "Ships tomorrow.", "marks": []}
+    ],
+    "markDefs": []
+  }
+]
+```
+
+Keys follow the edit the way they would in an editor, and when the evidence is unclear, a block gets a fresh key rather than a wrong one. The result is a value, not patches; output keys are always unique among siblings; the inputs are never mutated.
+
+#### What keeps its key
+
+- Unchanged and moved blocks. Repeated content pairs in order.
+- A block rewritten in place, like typing over it. Style changes count as rewrites, and an edited table cell keeps the whole table's keys.
+- A split keeps the key on the first non-empty fragment, like pressing enter; a merge keeps the first block's key, like pressing backspace. A soft-wrap join is a merge.
+- A typo fix lands as a text change on the same span, and editing a link's URL keeps its annotation key. Two identical annotations in one block (the same link twice, say) pair in order, like any repeated content.
+- A `json:object` payload keeps the `_key` it carries, unless the payload matches stored content, which keeps the stored key: editing markdown cannot re-key existing content.
+
+#### What gets restored
+
+Markdown cannot carry everything a block stores, so an adopted block gets back what the edit could not have touched:
+
+- A field the dialect cannot express, like a text block's `alignment`, including a whole custom object-array field the dialect drops. A field markdown does express, like `language` on a code block, follows the edit.
+- A custom style, list kind, or decorator markdown has no syntax for.
+- An empty or whitespace-only paragraph, which has no markdown form at all (blank lines are the block separator): it is restored next to its surviving neighbor, and deleted along with that neighbor if the neighbor goes. This covers top-level blocks; an empty paragraph nested inside a table cell or callout content is not restored. An empty heading or list item has a visible markdown form (`## `, `- `) and round-trips like any other block.
+- A block the edit did not touch comes back exactly as stored, span structure and unmappable marks included: an adjacent pair of spans that only differ by a decorator markdown has no syntax for keeps its split rather than merging into the one span a plain parse would produce.
+
+#### When keys reset
+
+- Ambiguity: when an insertion or deletion makes a match unclear, a block keeps its key only on clear evidence; everything else gets a new key, and short blocks near the change are the usual casualties.
+- Indistinguishable edits: replacing a block with unrelated content in the same position keeps its key (the end state is identical to a rewrite), and a whole-document rewrite that keeps the block count pairs blocks in order. That last one is deliberate: a translation keeps every anchor by position, which is the behavior translate flows need. The cost is that a reorder-plus-edit with balanced counts mispairs the same way, block-level and sibling-level alike.
+- Caps: on very large ambiguous edits, evidence gathering is size- and time-capped and degrades to fresh keys rather than waiting, so near the caps, which keys survive can vary with machine speed.
+- Refusal: a stored value that cannot survive its own serialize→parse round trip resets the whole document to the plain conversion, every key fresh except the ones `json:object` payloads carry.
+
+#### Options
+
+The options bag mirrors the two converters, plus a top-level `schema`: `deserialize` takes the rest of `markdownToPortableText`'s options and `serialize` takes the rest of `portableTextToMarkdown`'s. `schema` is taken once and governs both directions, because restoring keys depends on the two serializations agreeing. Pass the same `serialize` options that produced the markdown that was edited. The `deserialize` options apply to the stored value as well as the edited markdown, with two exceptions: new keys come from `deserialize.keyGenerator` (or the built-in generator), and `onDegradation` reports only on the edited markdown.
+
+#### Observing reconciliation
+
+Pass `onReconciliation` to see what happened to every key. The callback fires exactly once per call, synchronously, right before the function returns:
+
+```ts
+applyMarkdownEdit(stored, editedMarkdown, {
+  onReconciliation: (report) => {
+    if (report.keyMatching === 'skipped') {
+      // report.reason is 'round-trip-mismatch' (the stored value cannot
+      // survive its own serialize→parse round trip) or
+      // 'document-too-large'. The returned value is the plain
+      // conversion, every key fresh except the ones `json:object`
+      // payloads carry
+      return
+    }
+    report.preservedKeys // stored keys that survived, with a basis and a path
+    report.keyFallbacks // regions that got fresh keys instead of a guess
+    report.renamedKeys // keys rewritten to keep siblings unique
+  },
+})
+```
+
+Every `key` and `path` in the report matches the returned value exactly, and a path segment is a string field name, a number array index, or `{_key}` for a keyed element, the same convention as editor paths. Which keys survived, the paths, `renamedKeys`, and `keyMatching` are facts of that invocation, safe to branch on. A preserved key's `basis` names the matching method (`'content-unchanged'`, `'content-moved'`, `'content-split'`, `'content-merged'`, `'same-position'`, `'similar-content'`) and is advisory: near the evidence caps it can vary with machine speed, so never branch on it. A node absent from `preservedKeys` was not restored from the stored value, whether its key is fresh or carried by a `json:object` payload. The exported `ReconciliationReport` and `ReconciliationKeyPath` types are `@beta`.
+
+#### Concurrent edits
+
+Reconciling an unchanged serialization returns the stored value byte for byte for the content the edit did not touch, keys, span structure, and unmappable marks included; content the edit did touch still comes back canonicalized, with its keys restored where reconciliation can trace them. `applyMarkdownEdit` does not merge concurrent edits: reconcile against the exact value that produced the markdown, and before writing the result back, check that the stored field still equals that value. If it changed while the markdown was being edited, the edit describes a document that no longer exists, and writing it would silently overwrite the newer changes: serialize the current value and redo the edit instead.
+
 ## License
 
 MIT © [Sanity.io](https://www.sanity.io/)