@uniweb/semantic-parser 1.4.0 → 1.5.0

package/AGENTS.md

@@ -53,7 +53,7 @@ The parser returns a flat content structure:
 ```js
 {
   title: '',       // Main heading
-  pretitle: '',    // `#>` label line(s) or smaller headings above the title
+  pretitle: '',    // `#>` label line(s); also smaller headings above the title
   subtitle: '',    // Line(s) one step below the title (string or array)
   paragraphs: [],
   links: [],       // All link-like entities (including buttons, documents)
@@ -161,7 +161,7 @@ Lists maintain hierarchy through nested structure. The `processListItems()` func
 
 Key patterns — the staircase rule (each heading relates to the one before it):
 
-- **Pretitle**: `#>` label lines (headings with `role: "pretitle"`), and smaller headings stacked directly above a more important one (H3→H1, H6→H5, …). String or array.
+- **Pretitle**: `#>` label lines (headings with `role: "pretitle"`) — what to write. Also smaller headings stacked directly above a more important one (H3→H1, H6→H5, …), the older spelling, read for compatibility. String or array.
 - **Subtitle**: one step below the title, directly adjacent — and each further one-step descent (or same-size repeat) is another subtitle line. String or array.
 - **Items**: a heading two or more steps below the previous one, a step back up, or any heading after body content starts a new group.
 - **Banner Pattern**: an image (with banner role or followed by a heading) at the start of the first group stays with the headline.

package/README.md

@@ -66,7 +66,8 @@ Main content fields are at the top level. The `items` array contains additional
 result = {
   // Header fields (from headings) — each is a string, or an array when the
   // part spans several lines
-  pretitle: "",             // `#>` label line(s), or smaller headings above the title
+  pretitle: "",             // `#>` label line(s) — also smaller headings above
+                            //   the title, the older spelling
   title: "Welcome",         // Main heading
   subtitle: "",             // Line(s) one step below the title
 
@@ -180,7 +181,7 @@ This is the mechanism that creates repeating content groups (cards, features, FA
 Two spellings fill the `pretitle` slot:
 
 - **A `#>` label line** — a heading with `role: "pretitle"`. It names the block that starts next, at any depth, with no level arithmetic; the hash count is carried for round-tripping and means nothing.
-- **Position** — smaller headings stacked above the most important heading of a run (H3 before H1, H6 before H5, …), in any order. `#### / ### / # Title` and `## / ### / # Title` both yield a two-line pretitle.
+- **Position** — smaller headings stacked above the most important heading of a run (H3 before H1, H6 before H5, …), in any order. `#### / ### / # Title` and `## / ### / # Title` both yield a two-line pretitle. ⚖️ This is the older spelling and is read for compatibility; a label line is what to WRITE, because it carries its meaning alone rather than by what follows it.
 
 ### Banner image
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@uniweb/semantic-parser",
-  "version": "1.4.0",
+  "version": "1.5.0",
   "description": "Semantic parser for ProseMirror/TipTap content structures",
   "type": "module",
   "main": "./src/index.js",

package/src/builders/doc.js

@@ -6,6 +6,34 @@
  *
  * Starter content uses plain strings (no HTML marks), so the conversion
  * is straightforward — no need to reverse inline HTML formatting.
+ *
+ * ## ⭐ IT ACCEPTS THE PARSER'S OWN VOCABULARY TOO — and that is a contract
+ *
+ * This builder was written with its own spelling for several concepts, and
+ * `parseContent` emits a DIFFERENT one for the same thing: a link's text is
+ * `text` here and `label` there; an image's address is `src` here and `url`
+ * there; a video's cover is a string here and `{ src | url | identifier }`
+ * there; a list item is a plain string here and a whole content GROUP there.
+ *
+ * ⛔ EVERY ONE OF THOSE MISMATCHES FAILED SILENTLY, which is why the rule is
+ * now written down. Measured 2026-09-16, all four on the shapes a caller would
+ * most naturally reach for:
+ *
+ *   { links:  [{ label, href }] }        → buildDoc returns **null** outright
+ *   { images: [{ url, alt }] }           → an ImageBlock with NO url attr
+ *   { videos: [{ coverImg: '/p.jpg' }] } → the cover parses back as ""
+ *   { lists:  parsed.lists }             → a text node whose `text` is an
+ *                                          OBJECT — an invalid document
+ *
+ * None of them threw. A generator author reaches for `label` and `url` because
+ * those are the names in `docs/reference/component-metadata.md` and the names a
+ * component actually reads off `content` — so the natural spelling was the
+ * broken one, and it produced an empty section rather than an error.
+ *
+ * ⇒ For every concept where the two sides disagree, ACCEPT BOTH SPELLINGS. The
+ * emitted document is unchanged — this only widens what may come in — so the
+ * round-trip assertions above still pin the output. A new field whose name
+ * differs from the parser's needs the same treatment, or it joins this list.
  */
 
 // --- TipTap node builders ---
@@ -27,6 +55,33 @@ function heading(level, text) {
   }
 }
 
+/**
+ * A label line — `#> Text` — which is how a pretitle is written.
+ *
+ * ⛔ NOT A SMALLER HEADING BEFORE THE TITLE. That positional form still parses,
+ * and it is what this builder emitted until 2026-09-16, but it is the old
+ * spelling and it is fragile in a way the label line is not: it means "pretitle"
+ * only by sitting before a bigger heading, so the same line moved, or left
+ * alone, becomes something else. `#>` says what it is wherever it lands.
+ *
+ * ⭐ THE `#` COUNT MEANS NOTHING. `#>`, `##>` and `###>` are the same thing —
+ * content-reader keeps the authored count in `level` purely so a file
+ * round-trips byte-for-byte. We match the level of the title it labels, which is
+ * what an author writing by hand does (`##>` above an `##` item title).
+ *
+ * On the wire it is an ordinary heading carrying `role: 'pretitle'`, so every
+ * consumer that already understands headings keeps working.
+ */
+function labelLine(level, text) {
+  if (!text) return null
+  if (Array.isArray(text)) return text.map(t => labelLine(level, t)).filter(Boolean)
+  return {
+    type: 'heading',
+    attrs: { level, role: 'pretitle' },
+    content: [textNode(text)],
+  }
+}
+
 function paragraph(text) {
   if (!text) return null
   return {
@@ -35,18 +90,23 @@ function paragraph(text) {
   }
 }
 
-function linkParagraph({ text, href, target }) {
-  if (!text || !href) return null
+function linkParagraph({ text, label, href, target }) {
+  // `label` is what parseContent emits and what a component reads off
+  // `content.links[]`; `text` is this builder's original spelling.
+  const caption = text || label
+  if (!caption || !href) return null
   const mark = { type: 'link', attrs: { href } }
   if (target) mark.attrs.target = target
   return {
     type: 'paragraph',
-    content: [{ type: 'text', text, marks: [mark] }],
+    content: [{ type: 'text', text: caption, marks: [mark] }],
   }
 }
 
-function imageBlock({ src, alt = '', caption = '', direction, role, width, height }) {
-  const attrs = { url: src, alt }
+function imageBlock({ src, url, alt = '', caption = '', direction, role, width, height }) {
+  // The node attr is `url` and so is the parsed field; `src` is this builder's
+  // original spelling. Either one in, `url` out.
+  const attrs = { url: src || url, alt }
   if (caption) attrs.caption = caption
   if (direction) attrs.direction = direction
   if (role) attrs.role = role
@@ -56,6 +116,18 @@ function imageBlock({ src, alt = '', caption = '', direction, role, width, heigh
   return { type: 'ImageBlock', attrs }
 }
 
+// A document is STORED as the image row with `role: 'pdf'` — the node the
+// markdown reader writes and the parser reads as a `document` — so that is what
+// this emits. `url` (the parser's spelling) or `src` (this builder's) in.
+function documentNode({ url, src, alt = '', caption = '', preview, author, description }) {
+  const attrs = { src: url || src, alt, role: 'pdf' }
+  if (caption) attrs.caption = caption
+  if (preview) attrs.preview = preview
+  if (author) attrs.author = author
+  if (description) attrs.description = description
+  return { type: 'image', attrs }
+}
+
 function iconNode({ src, url, svg, library, name, size, color }) {
   // UniwebIcon supports multiple source types.
   //
@@ -81,11 +153,16 @@ function iconNode({ src, url, svg, library, name, size, color }) {
   return { type: 'UniwebIcon', attrs }
 }
 
-function videoNode({ src, caption, direction, coverImg }) {
-  const attrs = { src }
+function videoNode({ src, url, caption, direction, coverImg }) {
+  const attrs = { src: src || url }
   if (caption) attrs.caption = caption
   if (direction) attrs.direction = direction
-  if (coverImg) attrs.coverImg = coverImg
+  // ⛔ The editor's `Video` node holds `coverImg` as an OBJECT — the reader is
+  // `makeAssetUrl`, which looks for `.src` / `.url` / `.identifier`. A bare
+  // string passed straight through parses back as "": `info?.src` on a string
+  // is undefined. Normalize here so a caller may write either.
+  const cover = typeof coverImg === 'string' ? { src: coverImg } : coverImg
+  if (cover && (cover.src || cover.url || cover.identifier)) attrs.coverImg = cover
   return { type: 'Video', attrs }
 }
 
@@ -93,17 +170,75 @@ function dividerBlock() {
   return { type: 'DividerBlock' }
 }
 
+/**
+ * A code block for DISPLAY — what lands in `content.snippets`.
+ *
+ * `parseContent` groups one as `{ language, code }`; the element before
+ * grouping carries `{ text, attrs }`. Accept either spelling, and a bare string
+ * for a snippet with no language.
+ */
+function codeBlock(snippet) {
+  const spec = typeof snippet === 'string' ? { code: snippet } : snippet
+  if (!spec || typeof spec !== 'object') return null
+  const text = spec.code ?? spec.text
+  if (typeof text !== 'string' || text === '') return null
+  const attrs = {}
+  if (spec.language) attrs.language = spec.language
+  if (spec.filename) attrs.filename = spec.filename
+  return { type: 'codeBlock', attrs, content: [textNode(text)] }
+}
+
+/**
+ * A tagged data block — what lands in `content.data[tag]`.
+ *
+ * ⛔ IT IS A `dataBlock`, NOT A TAGGED `codeBlock`, and the difference is not
+ * cosmetic. content-reader emits `dataBlock` when a tagged fence PARSED, with
+ * the parsed value on `attrs.data`; it falls back to a tagged `codeBlock`
+ * carrying raw TEXT only when parsing failed. A builder handed a real object
+ * has parsed data by definition, so emitting the failure shape would hand a
+ * component a string where it expects a record.
+ *
+ * ⭐ `language` IS LOAD-BEARING, not decoration. content-writer reads it back to
+ * choose the fence, and the parsed value alone cannot say whether the author
+ * wrote YAML or JSON — without it an author's YAML silently becomes JSON on the
+ * next editor sync. A builder has no author to echo, so it states one.
+ */
+function dataBlock(tag, value, language) {
+  // A null value is the same absence as no key at all — a component reading
+  // `content.data[tag]` gets null either way — so emit no fence rather than one
+  // whose whole payload is `null`.
+  if (!tag || value === undefined || value === null) return null
+  return { type: 'dataBlock', attrs: { tag, language, data: value } }
+}
+
 function bulletList(items) {
   if (!items || !items.length) return null
+  const entries = items.map(listItemText).filter(Boolean)
+  if (!entries.length) return null
   return {
     type: 'bulletList',
-    content: items.map(item => ({
+    content: entries.map(text => ({
       type: 'listItem',
-      content: [paragraph(item)].filter(Boolean),
+      content: [paragraph(text)].filter(Boolean),
     })),
   }
 }
 
+/**
+ * One list entry's text. A generator writes plain strings; `parseContent`
+ * emits a full content GROUP per entry ({ paragraphs, links, … }), and feeding
+ * that back produced a text node whose `text` was an object — an invalid
+ * document that threw nowhere.
+ */
+function listItemText(item) {
+  if (typeof item === 'string') return item
+  if (!item || typeof item !== 'object') return ''
+  if (Array.isArray(item.paragraphs) && item.paragraphs.length) {
+    return item.paragraphs.join(' ')
+  }
+  return item.title || ''
+}
+
 // --- Group builder ---
 
 /**
@@ -113,14 +248,13 @@ function bulletList(items) {
  * @param {number} titleLevel - Heading level for title (1 for main, 2 for items)
  * @returns {Array} Array of TipTap nodes
  */
-function buildGroupNodes(group, titleLevel = 1) {
+function buildGroupNodes(group, titleLevel = 1, options = {}) {
   const nodes = []
 
-  // 1. Headings: pretitle → title → subtitle
-  // Pretitle uses a higher level number (less important) than title
-  // e.g., H3 before H1 — mirrors isPreTitle() in groups.js
+  // 1. Headings: pretitle → title → subtitle.
+  // A pretitle is a LABEL LINE (`#> Text`), at the level of the title it labels.
   if (group.pretitle) {
-    const pre = heading(titleLevel + 2, group.pretitle)
+    const pre = labelLine(titleLevel, group.pretitle)
     if (Array.isArray(pre)) nodes.push(...pre)
     else if (pre) nodes.push(pre)
   }
@@ -171,6 +305,12 @@ function buildGroupNodes(group, titleLevel = 1) {
     }
   }
 
+  if (group.documents) {
+    for (const doc of group.documents) {
+      nodes.push(documentNode(doc))
+    }
+  }
+
   if (group.lists) {
     for (const list of group.lists) {
       const node = bulletList(list)
@@ -178,6 +318,29 @@ function buildGroupNodes(group, titleLevel = 1) {
     }
   }
 
+  if (group.snippets) {
+    for (const snippet of group.snippets) {
+      const node = codeBlock(snippet)
+      if (node) nodes.push(node)
+    }
+  }
+
+  // `data` is a MAP of tag → value, not a list: one tagged fence per key, in
+  // declaration order.
+  //
+  // ⚠️ A `concept_block` (```md:faq) also lands in `content.data`, as
+  // `{ items, sequence }` — a tagged PROSE fence, not data. This does not
+  // reconstruct one: it emits the data fence, whose payload round-trips to the
+  // same value. Prose that was parsed into a document does not come back as
+  // prose here, and a caller rebuilding a concept block wants the markdown.
+  if (group.data && typeof group.data === 'object' && !Array.isArray(group.data)) {
+    const language = options.dataLanguage || 'yaml'
+    for (const [tag, value] of Object.entries(group.data)) {
+      const node = dataBlock(tag, value, language)
+      if (node) nodes.push(node)
+    }
+  }
+
   return nodes
 }
 
@@ -191,22 +354,26 @@ function buildGroupNodes(group, titleLevel = 1) {
  * roundtrips through parseContent() to yield the same structure.
  *
  * @param {Object} content - Content structure (same shape as parseContent output / starter)
+ * @param {Object} [options]
+ * @param {string} [options.dataLanguage='yaml'] - the fence a tagged data block
+ *   is written as. It is recorded on the node because content-writer reads it
+ *   back to choose the fence, and a parsed value cannot say which one it was.
  * @returns {Object|null} TipTap document { type: 'doc', content: [...] }, or null if empty
  */
-function buildDoc(content) {
+function buildDoc(content, options = {}) {
   if (!content) return null
 
   const nodes = []
 
   // Main group content (title level 1)
-  nodes.push(...buildGroupNodes(content, 1))
+  nodes.push(...buildGroupNodes(content, 1, options))
 
   // Items: separated by DividerBlock (mirrors divider-based grouping in groups.js)
   if (content.items && content.items.length > 0) {
     for (const item of content.items) {
       nodes.push(dividerBlock())
       // Item headings use level 2 (one below main H1)
-      nodes.push(...buildGroupNodes(item, 2))
+      nodes.push(...buildGroupNodes(item, 2, options))
     }
   }
 

package/src/processors/groups.js

@@ -15,6 +15,7 @@ function flattenGroup(group) {
         icons: group.body.icons || [],
         lists: group.body.lists || [],
         videos: group.body.videos || [],
+        documents: group.body.documents || [],
         insets: group.body.insets || [],
         snippets: group.body.snippets || [],
         data: group.body.data || {},
@@ -303,6 +304,7 @@ function processGroupContent(elements, options = {}) {
         images: [],
         icons: [],
         videos: [],
+        documents: [],
         insets: [],
         snippets: [],
         paragraphs: [],
@@ -387,6 +389,10 @@ function processGroupContent(elements, options = {}) {
                     body.videos.push(preserveProps);
                     break;
 
+                case "document":
+                    body.documents.push(preserveProps);
+                    break;
+
                 case "link":
                     if (element.children && Array.isArray(element.children))
                         processInlineElements(element.children, body);
@@ -520,19 +526,6 @@ function processGroupContent(elements, options = {}) {
                     });
                     break;
 
-                case "document-group":
-                    // Map documents to links with role=document
-                    element.documents.forEach(doc => {
-                        body.links.push({
-                            href: doc.href || doc.downloadUrl || '',
-                            label: doc.title || '',
-                            role: 'document',
-                            download: true,
-                            preview: doc.coverImg,
-                            fileType: doc.fileType,
-                        });
-                    });
-                    break;
             }
         }
     }
@@ -588,6 +581,9 @@ function processInlineElements(children, body) {
             // `{role=video}` belongs in videos[] for the same reason a hoisted
             // one does.
             body.videos.push(item.attrs);
+        } else if (item.type === "document") {
+            // And an inline `{role=pdf}` in documents[].
+            body.documents.push(item.attrs);
         } else if (item.type === "link") {
             // Handle inline links extracted from paragraph text nodes
             body.links.push(item.attrs);

package/src/processors/sequence.js

@@ -210,7 +210,19 @@ function createSequenceElement(node, options = {}) {
         case "image":
             // `role` decides which content array this lands in — the contract
             // published in docs/reference/content-structure.md ("Media Assets"):
-            // icon → icons, video → videos, everything else → images.
+            // icon → icons, video → videos, pdf → documents, everything else →
+            // images.
+            //
+            // A document is stored as this node, as a video is, and delivered as
+            // a kind of its own (`parseMarkdownDocument`). Until 2026-09-29 it
+            // landed in `images`, drawn as an image of the file, while the
+            // runtime had guaranteed an empty `content.documents` since January.
+            if (attrs?.role === "pdf") {
+                return {
+                    type: "document",
+                    attrs: parseMarkdownDocument(attrs, options),
+                };
+            }
             //
             // The video branch was missing until 2026-07-30: every
             // `{role=video}` node ran parseImgBlock, which does not carry
@@ -315,13 +327,16 @@ function createSequenceElement(node, options = {}) {
             };
 
         case "document-group":
-            return {
-                type: "document-group",
-                documents:
-                    node.content
-                        ?.filter((c) => c.type === "document")
-                        .map((doc) => parseDocumentBlock(doc.attrs)) || [],
-            };
+            // The editor's older container: one `document` element per document
+            // it holds, so a document is one kind of content whichever node
+            // wrote it (`documentFromEditorNode`). Until 2026-09-29 this was a
+            // `document-group` element whose documents became `content.links`.
+            return (node.content || [])
+                .filter((c) => c.type === "document")
+                .map((doc) => ({
+                    type: "document",
+                    attrs: documentFromEditorNode(doc.attrs),
+                }));
 
         // The editor renamed this node to `StructuredContent` (2026-07-31,
         // editor-internal). Both are accepted: the new name is what the editor
@@ -613,13 +628,15 @@ function processInlineElements(content, options = {}) {
             // image silently deleted. groups.js has carried the receiving
             // branch (and a comment describing exactly this failure) since it
             // was written, but nothing ever emitted the item it reads.
-            const isVideo = item.attrs?.role === "video";
-            items.push({
-                type: isVideo ? "video" : "image",
-                attrs: isVideo
-                    ? parseMarkdownVideo(item.attrs, options)
-                    : parseImgBlock(item.attrs || {}, options),
-            });
+            // The role partitions it as the block case does.
+            const role = item.attrs?.role;
+            if (role === "video") {
+                items.push({ type: "video", attrs: parseMarkdownVideo(item.attrs, options) });
+            } else if (role === "pdf") {
+                items.push({ type: "document", attrs: parseMarkdownDocument(item.attrs, options) });
+            } else {
+                items.push({ type: "image", attrs: parseImgBlock(item.attrs || {}, options) });
+            }
         } else if (item.type === "math-inline" || item.type === "math_inline") {
             items.push(item);
         } else if (item.type === "inset_placeholder") {
@@ -802,6 +819,54 @@ function parseCardBlock(itemAttrs) {
     };
 }
 
+// A document authored in markdown: `![Annual report](./report.pdf){role=pdf …}`.
+//
+// Read through `parseImgBlock`, which resolves the address and the preview's the
+// way every image's are, and then narrowed to what a document IS: where the file
+// is, what it is called, and the resource's own metadata. The image layout keys
+// (direction, size, credit) describe a picture, not a file, and are left out.
+function parseMarkdownDocument(itemAttrs, options) {
+    const { url, alt, caption, role, href, target, id, preview, author, description } =
+        parseImgBlock(itemAttrs || {}, options);
+    return {
+        url,
+        alt,
+        caption,
+        role,
+        ...(href && { href }),
+        ...(target && { target }),
+        ...(id && { id }),
+        ...(preview && { preview }),
+        ...(author && { author }),
+        ...(description && { description }),
+    };
+}
+
+// The editor's older `document` node — its `title`, `coverImg` and `src` (or
+// asset identifier) — read into the same shape a markdown document has.
+function documentFromEditorNode(itemAttrs) {
+    const {
+        href,
+        downloadUrl,
+        title = "",
+        alt = "",
+        coverImg,
+        author,
+        description,
+        fileType,
+    } = parseDocumentBlock(itemAttrs || {});
+    return {
+        url: href || downloadUrl || "",
+        alt: alt || title,
+        caption: title,
+        role: "pdf",
+        ...(coverImg && { preview: coverImg }),
+        ...(author && { author }),
+        ...(description && { description }),
+        ...(fileType && { fileType }),
+    };
+}
+
 function parseDocumentBlock(itemAttrs) {
     const { src, info = {}, coverImg = null, ...others } = itemAttrs;