mdld-parse 0.9.0 → 0.9.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -62,6 +62,39 @@ console.log(result.quads);
62
62
 
63
63
  MD-LD allows you to author RDF graphs directly in Markdown using explicit `{...}` annotations:
64
64
 
65
+ ## ⚡ Performance & Architecture
66
+
67
+ MD-LD v0.10.0 features a **character-based tokenization system** for optimal performance:
68
+
69
+ - **20-28% faster parsing** than regex-based approaches
70
+ - **Memory-efficient** with ~640 bytes per quad retained
71
+ - **Streaming-friendly** with O(n) linear time complexity
72
+ - **Character-based detection** replaces complex regex patterns
73
+ - **Unified tokenizer architecture** in `src/tokenizers.js`
74
+
75
+ ### Tokenizer Architecture
76
+
77
+ The parser uses specialized character-based tokenizers:
78
+
79
+ ```javascript
80
+ // Block-level tokenizers
81
+ detectFence() // ```code blocks
82
+ detectPrefix() // [prefix] <iri>
83
+ detectHeading() // # Headings
84
+ detectList() // - List items
85
+ detectBlockquote() // > Blockquotes
86
+ detectStandaloneSubject() // {=subject}
87
+
88
+ // Inline carrier scanner
89
+ scanInlineCarriers() // [text], **bold**, `code`, <URL>
90
+ ```
91
+
92
+ This design provides:
93
+ - **Better maintainability** - Easier to debug and extend
94
+ - **Improved error handling** - More precise edge case detection
95
+ - **Cleaner code structure** - No complex regex patterns
96
+ - **Full backward compatibility** - All 127 tests passing
97
+
65
98
  ```markdown
66
99
  [my] <tag:alice@example.com,2026:>
67
100
  # 2024-07-18 {=my:journal-2024-07-18 .my:Event my:date ^^xsd:date}
@@ -199,7 +232,7 @@ Parse MD-LD markdown and return RDF quads with lean origin tracking.
199
232
  - `dataFactory` (object, optional) — Custom RDF/JS DataFactory
200
233
  - `graph` (string, optional) — Named graph IRI
201
234
 
202
- **Returns:** `{ quads, remove, statements, origin, context, primarySubject }`
235
+ **Returns:** `{ quads, remove, statements, origin, context, primarySubject, md }`
203
236
 
204
237
  > **Legacy:** `parse(text, options)` still works for backward compatibility
205
238
 
@@ -209,6 +242,7 @@ Parse MD-LD markdown and return RDF quads with lean origin tracking.
209
242
  - `origin` — Lean origin tracking object with quadIndex for UI navigation
210
243
  - `context` — Final context used (includes prefixes)
211
244
  - `primarySubject` — String IRI or null (first non-fragment subject declaration)
245
+ - `md` — Clean Markdown with all MD-LD annotations stripped (round-trip safe)
212
246
 
213
247
  ### `merge(docs, options)`
214
248
 
@@ -286,11 +320,43 @@ const nodeView = generateNode({ ...parse({ text }), focusIRI: 'http://example.or
286
320
  ```
287
321
 
288
322
  **Why this works:**
289
- - `parse()` returns `{ quads, context, primarySubject, ... }`
323
+ - `parse()` returns `{ quads, context, primarySubject, md, ... }`
290
324
  - `generate()` accepts `{ quads, context, primarySubject }`
291
325
  - `generateNode()` accepts `{ quads, context, focusIRI }` (with focusIRI override)
292
326
  - Perfect shape alignment enables elegant `{ ...spread }` composition
293
327
 
328
+ ### The `md` Field — Clean Markdown Extraction
329
+
330
+ Every `parse()` result includes a `md` field containing the original Markdown with all MD-LD annotations stripped:
331
+
332
+ ```javascript
333
+ const result = parse({
334
+ text: `# Document {=ex:doc .Article}\n[Content] {ex:content}`,
335
+ context: { ex: 'http://example.org/' }
336
+ });
337
+
338
+ console.log(result.md);
339
+ // # Document\nContent
340
+
341
+ // Round-trip safety: re-parsing clean MD produces zero quads
342
+ const reparsed = parse({ text: result.md });
343
+ console.log(reparsed.quads.length); // 0
344
+ ```
345
+
346
+ **Behavior:**
347
+ - Valid MD-LD annotations (`{=...}`, `{+...}`, `{...}`) are completely removed
348
+ - Content from value carriers (`[text]`, `**bold**`, `` `code` ``) is preserved
349
+ - Invalid syntax (annotations not at end-of-line) is preserved as visible markers
350
+ - Headings, lists, blockquotes, code blocks maintain their structure
351
+ - Prefix declarations at start of line are stripped
352
+ - Standalone subject declarations (`{=ex:subject}`) are stripped
353
+
354
+ **Use cases:**
355
+ - **Content extraction** — Get readable Markdown without semantic markup
356
+ - **Syntax validation** — Remaining `{...}` patterns indicate invalid MD-LD syntax
357
+ - **Round-trip testing** — `parse(md).md` should parse to zero quads
358
+ - **Preview generation** — Show clean document before publishing
359
+
294
360
  ### `locate(quad, origin)`
295
361
 
296
362
  Locate the origin entry for a quad using the lean origin system.
@@ -347,7 +413,7 @@ Quads are compatible with:
347
413
  The parser includes comprehensive tests covering all spec requirements:
348
414
 
349
415
  ```bash
350
- npm test
416
+ pnpm test
351
417
  ```
352
418
 
353
419
  Tests validate:
package/package.json CHANGED
@@ -1,8 +1,9 @@
1
1
  {
2
2
  "name": "mdld-parse",
3
- "version": "0.9.0",
3
+ "version": "0.9.2",
4
4
  "description": "A standards-compliant parser for **MD-LD (Markdown-Linked Data)** — a human-friendly RDF authoring format that extends Markdown with semantic annotations.",
5
5
  "type": "module",
6
+ "packageManager": "pnpm@10.6.5",
6
7
  "main": "index.js",
7
8
  "exports": {
8
9
  ".": "./src/index.js"
@@ -13,7 +14,8 @@
13
14
  ],
14
15
  "scripts": {
15
16
  "test": "node tests/index.js",
16
- "dev": "pnpx live-server"
17
+ "dev": "vite dev",
18
+ "build": "vite build && vite build --config vite.config.lib.js && cp -r docs/ dist/docs/ && cp -r examples/ dist/examples/ && cp -r spec/ dist/spec/ && cp -r README.md dist/README.md"
17
19
  },
18
20
  "keywords": [
19
21
  "mdld",
@@ -38,5 +40,19 @@
38
40
  "homepage": "https://mdld.js.org",
39
41
  "bugs": {
40
42
  "url": "https://github.com/davay42/mdld-parse/issues"
43
+ },
44
+ "devDependencies": {
45
+ "@unocss/extractor-pug": "^66.6.8",
46
+ "@vitejs/plugin-vue": "^6.0.6",
47
+ "@vueuse/core": "^14.3.0",
48
+ "markdown-it": "^14.1.1",
49
+ "n3": "^2.0.3",
50
+ "shiki": "^4.0.2",
51
+ "unocss": "^66.6.8",
52
+ "vite": "^8.0.10",
53
+ "vite-plugin-singlefile": "^2.3.3",
54
+ "vscode-textmate": "^9.3.2",
55
+ "vue": "^3.6.0-beta.10",
56
+ "codejar": "^4.3.0"
41
57
  }
42
58
  }
@@ -0,0 +1,362 @@
1
+ /**
2
+ * MD-LD Syntax Highlighter
3
+ *
4
+ * A performant, token-based syntax highlighter for MD-LD (Markdown Linked Data).
5
+ * Uses simple string operations rather than complex grammars for reliability
6
+ * and maintainability.
7
+ *
8
+ * Features:
9
+ * - Token-based parsing with string operations
10
+ * - Markdown formatting support with semantic distinction
11
+ * - Lookahead detection for legal value carriers
12
+ * - Orange semantic markers vs default markdown spans
13
+ * - Annotation, value carrier, and heading highlighting
14
+ */
15
+
16
+ // HTML escaping function
17
+ export function escapeHtml(s) {
18
+ return String(s)
19
+ .replaceAll('&', '&amp;')
20
+ .replaceAll('<', '&lt;')
21
+ .replaceAll('>', '&gt;')
22
+ .replaceAll('"', '&quot;')
23
+ .replaceAll("'", '&#39;');
24
+ }
25
+
26
+ // Token color scheme
27
+ export const TOKEN_COLORS = {
28
+ text: '#6b7280', // Grayscale - main text
29
+ marker: '#f97316', // Orange - .?!^^@
30
+ retraction: '#dc2626', // Red - whole predicate with -
31
+ value: '#f97316', // Yellow - values in []
32
+ annotation: '#4b5563' // Darker gray - annotation braces
33
+ };
34
+
35
+ // Check if markdown formatting has following annotation
36
+ function hasFollowingAnnotation(code, endPos) {
37
+ let i = endPos;
38
+ while (i < code.length && /\s/.test(code[i])) {
39
+ i++;
40
+ }
41
+ return i < code.length && code[i] === '{';
42
+ }
43
+
44
+ // Process markdown formatting in text while preserving original syntax
45
+ export function processMarkdownFormatting(text, hasFollowingAnnotation = false) {
46
+ let result = escapeHtml(text);
47
+
48
+ if (hasFollowingAnnotation) {
49
+ // Orange for legal value carriers (with following annotation)
50
+ result = result.replace(/\*\*(.*?)\*\*/g, `<strong style="color: ${TOKEN_COLORS.marker}">$1</strong>`);
51
+ result = result.replace(/__(.*?)__/g, `<strong style="color: ${TOKEN_COLORS.marker}">$1</strong>`);
52
+ result = result.replace(/\*(.*?)\*/g, `<em style="color: ${TOKEN_COLORS.marker}">$1</em>`);
53
+ result = result.replace(/_(.*?)_/g, `<em style="color: ${TOKEN_COLORS.marker}">$1</em>`);
54
+ result = result.replace(/`(.*?)`/g, `<code style="color: ${TOKEN_COLORS.marker}">$1</code>`);
55
+ } else {
56
+ // Default browser styles for regular markdown spans
57
+ result = result.replace(/\*\*(.*?)\*\*/g, '<strong>$1</strong>');
58
+ result = result.replace(/__(.*?)__/g, '<strong>$1</strong>');
59
+ result = result.replace(/\*(.*?)\*/g, '<em>$1</em>');
60
+ result = result.replace(/_(.*?)_/g, '<em>$1</em>');
61
+ result = result.replace(/`(.*?)`/g, '<code>$1</code>');
62
+ }
63
+
64
+ return result;
65
+ }
66
+
67
+ // Process markdown formatting at specific position for non-bracket text
68
+ function processMarkdownFormattingAtPosition(code, startPos) {
69
+ const char = code[startPos];
70
+
71
+ // Check for **bold**
72
+ if (char === '*' && startPos + 1 < code.length && code[startPos + 1] === '*') {
73
+ const endPos = code.indexOf('**', startPos + 2);
74
+ if (endPos !== -1) {
75
+ const content = code.slice(startPos + 2, endPos);
76
+ const hasAnnotation = hasFollowingAnnotation(code, endPos + 2);
77
+ const color = hasAnnotation ? ` style="color: ${TOKEN_COLORS.marker}"` : '';
78
+ return {
79
+ html: `<strong${color}>**${escapeHtml(content)}**</strong>`,
80
+ nextIndex: endPos + 2
81
+ };
82
+ }
83
+ }
84
+
85
+ // Check for __bold__
86
+ if (char === '_' && startPos + 1 < code.length && code[startPos + 1] === '_') {
87
+ const endPos = code.indexOf('__', startPos + 2);
88
+ if (endPos !== -1) {
89
+ const content = code.slice(startPos + 2, endPos);
90
+ const hasAnnotation = hasFollowingAnnotation(code, endPos + 2);
91
+ const color = hasAnnotation ? ` style="color: ${TOKEN_COLORS.marker}"` : '';
92
+ return {
93
+ html: `<strong${color}>__${escapeHtml(content)}__</strong>`,
94
+ nextIndex: endPos + 2
95
+ };
96
+ }
97
+ }
98
+
99
+ // Check for *italic*
100
+ if (char === '*') {
101
+ const endPos = code.indexOf('*', startPos + 1);
102
+ if (endPos !== -1) {
103
+ const content = code.slice(startPos + 1, endPos);
104
+ const hasAnnotation = hasFollowingAnnotation(code, endPos + 1);
105
+ const color = hasAnnotation ? ` style="color: ${TOKEN_COLORS.marker}"` : '';
106
+ return {
107
+ html: `<em${color}>*${escapeHtml(content)}*</em>`,
108
+ nextIndex: endPos + 1
109
+ };
110
+ }
111
+ }
112
+
113
+ // Check for _italic_
114
+ if (char === '_') {
115
+ const endPos = code.indexOf('_', startPos + 1);
116
+ if (endPos !== -1) {
117
+ const content = code.slice(startPos + 1, endPos);
118
+ const hasAnnotation = hasFollowingAnnotation(code, endPos + 1);
119
+ const color = hasAnnotation ? ` style="color: ${TOKEN_COLORS.marker}"` : '';
120
+ return {
121
+ html: `<em${color}>_${escapeHtml(content)}_</em>`,
122
+ nextIndex: endPos + 1
123
+ };
124
+ }
125
+ }
126
+
127
+ // Check for `code`
128
+ if (char === '`') {
129
+ const endPos = code.indexOf('`', startPos + 1);
130
+ if (endPos !== -1) {
131
+ const content = code.slice(startPos + 1, endPos);
132
+ const hasAnnotation = hasFollowingAnnotation(code, endPos + 1);
133
+ const color = hasAnnotation ? ` style="color: ${TOKEN_COLORS.marker}"` : '';
134
+ return {
135
+ html: `<code${color}>\`${escapeHtml(content)}\`</code>`,
136
+ nextIndex: endPos + 1
137
+ };
138
+ }
139
+ }
140
+
141
+ return null;
142
+ }
143
+
144
+ // Parse an annotation token { ... }
145
+ function parseAnnotation(content) {
146
+ const parts = content.trim().split(/\s+/);
147
+ let result = '';
148
+ let nextTermIsRetracted = false;
149
+
150
+ for (let i = 0; i < parts.length; i++) {
151
+ const part = parts[i];
152
+ if (!part) continue;
153
+
154
+ // Check for subject markers (= or +)
155
+ if (part.startsWith('=') || part.startsWith('+')) {
156
+ const marker = part[0]; // = or +
157
+ const rest = part.slice(1); // the rest of the subject
158
+ result += `<span style="color: ${TOKEN_COLORS.marker}">${escapeHtml(marker)}</span><span style="color: ${TOKEN_COLORS.text}">${escapeHtml(rest)}</span> `;
159
+ continue;
160
+ }
161
+
162
+ // Check for retraction marker (-)
163
+ if (part === '-') {
164
+ nextTermIsRetracted = true;
165
+ result += `<span style="color: ${TOKEN_COLORS.retraction}">${escapeHtml(part)}</span> `;
166
+ continue;
167
+ }
168
+
169
+ // Handle case where - and predicate are combined (e.g., -predicate)
170
+ if (part.startsWith('-') && part.length > 1) {
171
+ result += `<span style="color: ${TOKEN_COLORS.retraction}">${escapeHtml(part)}</span> `;
172
+ continue;
173
+ }
174
+
175
+ // Apply retraction color to the immediate next term only
176
+ const isRetracted = nextTermIsRetracted;
177
+ nextTermIsRetracted = false; // Reset after one term
178
+
179
+ // Check for object markers (? or !)
180
+ if (part.startsWith('?') || part.startsWith('!')) {
181
+ const marker = part[0]; // ? or !
182
+ const rest = part.slice(1); // the CURIE or rest of the term
183
+ const color = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.marker;
184
+ const textColor = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.text;
185
+ result += `<span style="color: ${color}">${escapeHtml(marker)}</span><span style="color: ${textColor}">${escapeHtml(rest)}</span> `;
186
+ continue;
187
+ }
188
+
189
+ // Check for datatype markers (^^, @)
190
+ if (part.startsWith('^^')) {
191
+ const marker = '^^'; // the datatype marker
192
+ const rest = part.slice(2); // the datatype
193
+ const color = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.marker;
194
+ const textColor = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.text;
195
+ result += `<span style="color: ${color}">${escapeHtml(marker)}</span><span style="color: ${textColor}">${escapeHtml(rest)}</span> `;
196
+ continue;
197
+ }
198
+
199
+ if (part.startsWith('@')) {
200
+ const marker = '@'; // the language marker
201
+ const rest = part.slice(1); // the language code
202
+ const color = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.marker;
203
+ const textColor = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.text;
204
+ result += `<span style="color: ${color}">${escapeHtml(marker)}</span><span style="color: ${textColor}">${escapeHtml(rest)}</span> `;
205
+ continue;
206
+ }
207
+
208
+ // Check for type marker (.)
209
+ if (part.startsWith('.')) {
210
+ const marker = '.'; // the dot
211
+ const rest = part.slice(1); // the class name
212
+ const color = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.marker;
213
+ const textColor = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.text;
214
+ result += `<span style="color: ${color}">${escapeHtml(marker)}</span><span style="color: ${textColor}">${escapeHtml(rest)}</span> `;
215
+ continue;
216
+ }
217
+
218
+ // Regular CURIE or IRI
219
+ const color = isRetracted ? TOKEN_COLORS.retraction : TOKEN_COLORS.text;
220
+ result += `<span style="color: ${color}">${escapeHtml(part)}</span> `;
221
+ }
222
+
223
+ return result.trim();
224
+ }
225
+
226
+ // Main MDLD syntax highlighting function
227
+ export function highlightMDLD(code) {
228
+ let result = '';
229
+ let i = 0;
230
+
231
+ while (i < code.length) {
232
+ const char = code[i];
233
+
234
+ // Handle annotation blocks { ... }
235
+ if (char === '{') {
236
+ const endIdx = code.indexOf('}', i);
237
+ if (endIdx === -1) {
238
+ // No closing brace, treat as regular text
239
+ result += escapeHtml(char);
240
+ i++;
241
+ continue;
242
+ }
243
+
244
+ const content = code.slice(i + 1, endIdx);
245
+ const parsed = parseAnnotation(content);
246
+
247
+ result += `<span style="color: ${TOKEN_COLORS.annotation}; opacity: 0.75">{</span>`;
248
+ result += parsed;
249
+ result += `<span style="color: ${TOKEN_COLORS.annotation}; opacity: 0.75">}</span>`;
250
+
251
+ i = endIdx + 1;
252
+ continue;
253
+ }
254
+
255
+ // Handle value carriers [ ... ] and markdown formatting
256
+ if (char === '[') {
257
+ const endIdx = code.indexOf(']', i);
258
+ if (endIdx === -1) {
259
+ // No closing bracket, treat as regular text
260
+ result += escapeHtml(char);
261
+ i++;
262
+ continue;
263
+ }
264
+
265
+ const content = code.slice(i + 1, endIdx);
266
+ const hasAnnotation = hasFollowingAnnotation(code, endIdx + 1);
267
+ const processedContent = processMarkdownFormatting(content, hasAnnotation);
268
+ result += `<span style="color: ${TOKEN_COLORS.value}; opacity: 0.85">[${processedContent}]</span>`;
269
+
270
+ i = endIdx + 1;
271
+ continue;
272
+ }
273
+
274
+ // Handle markdown formatting outside brackets
275
+ if (char === '*' || char === '_' || char === '`') {
276
+ const processed = processMarkdownFormattingAtPosition(code, i);
277
+ if (processed) {
278
+ result += processed.html;
279
+ i = processed.nextIndex;
280
+ continue;
281
+ }
282
+ }
283
+
284
+ // Handle prefix declarations < ... >
285
+ if (char === '<') {
286
+ const endIdx = code.indexOf('>', i);
287
+ if (endIdx === -1) {
288
+ result += escapeHtml(char);
289
+ i++;
290
+ continue;
291
+ }
292
+
293
+ const content = code.slice(i + 1, endIdx);
294
+ result += `<span style="color: ${TOKEN_COLORS.text}; opacity: 0.6">&lt;${escapeHtml(content)}&gt;</span>`;
295
+
296
+ i = endIdx + 1;
297
+ continue;
298
+ }
299
+
300
+ // Handle markdown headers
301
+ if (char === '#') {
302
+ let hashCount = 0;
303
+ let j = i;
304
+ while (j < code.length && code[j] === '#') {
305
+ hashCount++;
306
+ j++;
307
+ }
308
+
309
+ if (hashCount <= 6 && (j >= code.length || code[j] === ' ' || code[j] === '\t')) {
310
+ // Find end of line
311
+ const lineEnd = code.indexOf('\n', j);
312
+ const endIdx = lineEnd === -1 ? code.length : lineEnd;
313
+ const headerText = code.slice(j, endIdx).trim();
314
+
315
+ // Process heading text for annotations
316
+ let processedHeaderText = '';
317
+ let lastIndex = 0;
318
+
319
+ // Find and process annotations in heading text
320
+ let annotationStart = headerText.indexOf('{');
321
+ while (annotationStart !== -1) {
322
+ const annotationEnd = headerText.indexOf('}', annotationStart);
323
+ if (annotationEnd === -1) break;
324
+
325
+ // Add text before annotation (escaped)
326
+ const beforeText = headerText.slice(lastIndex, annotationStart);
327
+ processedHeaderText += escapeHtml(beforeText);
328
+
329
+ // Process annotation content
330
+ const annotationContent = headerText.slice(annotationStart + 1, annotationEnd);
331
+ const parsedAnnotation = parseAnnotation(annotationContent);
332
+
333
+ // Add annotation HTML
334
+ processedHeaderText += `<span style="color: ${TOKEN_COLORS.annotation}; opacity: 0.75">{</span>` +
335
+ parsedAnnotation +
336
+ `<span style="color: ${TOKEN_COLORS.annotation}; opacity: 0.75">}</span>`;
337
+
338
+ lastIndex = annotationEnd + 1;
339
+ annotationStart = headerText.indexOf('{', lastIndex);
340
+ }
341
+
342
+ // Add remaining text after last annotation (escaped)
343
+ processedHeaderText += escapeHtml(headerText.slice(lastIndex));
344
+
345
+ // Create HTML heading that preserves the original markdown syntax
346
+ const headingTag = `h${hashCount}`;
347
+ const headingStyle = `margin: 0.5em 0; font-weight: 600;`;
348
+ const hashes = '#'.repeat(hashCount);
349
+ result += `<${headingTag} style="${headingStyle}"><span style="color: ${TOKEN_COLORS.marker}; opacity: 0.8">${hashes}</span> ${processedHeaderText}</${headingTag}>`;
350
+
351
+ i = endIdx;
352
+ continue;
353
+ }
354
+ }
355
+
356
+ // Regular text
357
+ result += escapeHtml(char);
358
+ i++;
359
+ }
360
+
361
+ return result;
362
+ }