mdld-parse 0.9.9 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. package/README.md +263 -326
  2. package/dist/index.js +1860 -1337
  3. package/package.json +4 -4
  4. package/dist/.nojekyll +0 -0
  5. package/dist/CNAME +0 -1
  6. package/dist/README.md +0 -500
  7. package/dist/docs/API.md +0 -435
  8. package/dist/docs/Architecture.md +0 -190
  9. package/dist/docs/Guide.md +0 -816
  10. package/dist/docs/LLM-MDLD-Inducer.md +0 -550
  11. package/dist/docs/LiterateRDF.md +0 -472
  12. package/dist/docs/Parser.md +0 -720
  13. package/dist/docs/Performance.md +0 -238
  14. package/dist/docs/Spec-render.md +0 -1439
  15. package/dist/docs/Subject.md +0 -180
  16. package/dist/docs/Syntax.md +0 -463
  17. package/dist/docs/TodoList.md +0 -215
  18. package/dist/docs/Use-Cases.md +0 -354
  19. package/dist/docs/guide-m.md +0 -56
  20. package/dist/docs/index.html +0 -108
  21. package/dist/docs/index.md +0 -56
  22. package/dist/docs/one-page.md +0 -322
  23. package/dist/docs/origin.md +0 -460
  24. package/dist/docs/polarity.md +0 -586
  25. package/dist/docs/primary-subject.md +0 -326
  26. package/dist/docs/statements.md +0 -226
  27. package/dist/docs/tokenization-performance-report.md +0 -164
  28. package/dist/examples/Cookbook.md +0 -32
  29. package/dist/examples/Status-SHACL.md +0 -31
  30. package/dist/examples/demo/index.md +0 -35
  31. package/dist/examples/demo/journal.md +0 -58
  32. package/dist/examples/demo/minimal.md +0 -23
  33. package/dist/examples/demo/project.md +0 -84
  34. package/dist/examples/demo/prov.md +0 -107
  35. package/dist/examples/demo/rdf.md +0 -69
  36. package/dist/examples/demo/recipe.md +0 -50
  37. package/dist/examples/demo/research.md +0 -84
  38. package/dist/examples/demo/shacl.md +0 -161
  39. package/dist/examples/demo/space-mission.md +0 -58
  40. package/dist/examples/demo/tasks.md +0 -175
  41. package/dist/examples/demo/xsd.md +0 -91
  42. package/dist/examples/dogfood.md +0 -288
  43. package/dist/examples/few-shot.md +0 -309
  44. package/dist/examples/graph/rdf-graph.html +0 -473
  45. package/dist/examples/graph/rdf_force_graph_web_component.html +0 -506
  46. package/dist/examples/index.md +0 -45
  47. package/dist/examples/llm-time-workflow.md +0 -173
  48. package/dist/examples/statements.md +0 -43
  49. package/dist/examples/workflow.md +0 -150
  50. package/dist/index.html +0 -3049
  51. package/dist/logo.svg +0 -7
  52. package/dist/spec/Spec-compact.md +0 -368
  53. package/dist/spec/Spec-ultra.md +0 -232
  54. package/dist/spec/Spec.md +0 -1116
  55. package/dist/spec/index.md +0 -46
@@ -1,720 +0,0 @@
1
- # MD-LD Parser Architecture
2
-
3
- ## Overview
4
-
5
- The MD-LD parser implements a **single-pass, streaming-friendly** architecture that transforms Markdown with semantic annotations into RDF/JS quads while maintaining complete provenance and supporting elevated statements extraction.
6
-
7
- ## Character-Based Tokenization (v0.10.0)
8
-
9
- The parser uses a unified character-based tokenization system for optimal performance:
10
-
11
- ### Tokenizer Architecture
12
-
13
- ```javascript
14
- // src/tokenizers.js - Centralized tokenization logic
15
- export function detectFence(line) // ```code blocks
16
- export function detectPrefix(line) // [prefix] <uri>
17
- export function detectHeading(line) // # Headings
18
- export function detectList(line) // - List items
19
- export function detectBlockquote(line) // > Blockquotes
20
- export function detectStandaloneSubject(line) // {=subject}
21
- export function scanInlineCarriers(text) // [text], **bold**, `code`, <URL>
22
- ```
23
-
24
- ### Performance Benefits
25
-
26
- - **20-28% faster parsing** than regex-based approaches
27
- - **Memory-efficient** with ~640 bytes per quad retained
28
- - **O(n) linear time complexity** maintained
29
- - **Better error handling** with precise edge case detection
30
- - **Cleaner code structure** - No complex regex patterns
31
-
32
- ### Token Detection Flow
33
-
34
- ```javascript
35
- // Block-level detection in scanTokens()
36
- const PROCESSORS = [
37
- { type: 'fence', test: line => detectHeading(line), process: handleFence },
38
- { type: 'prefix', test: line => detectPrefix(line), process: handlePrefix },
39
- { type: 'heading', test: line => detectHeading(line), process: handleHeading },
40
- { type: 'list', test: line => detectList(line), process: handleList },
41
- { type: 'blockquote', test: line => detectBlockquote(line), process: handleBlockquote },
42
- { type: 'para', test: line => line.trim(), process: handlePara }
43
- ];
44
-
45
- // Inline carrier detection in getCarriers()
46
- function getCarriers(token) {
47
- if (token.type === 'code') return [];
48
- return token._carriers || (token._carriers = scanInlineCarriers(token.text, token.range[0]));
49
- }
50
- ```
51
-
52
- ## Core Parser Structures
53
-
54
- ### 1. Parser State Object
55
-
56
- ```javascript
57
- const state = {
58
- ctx: {}, // Prefix context for IRI expansion
59
- df: DataFactory, // RDF/JS DataFactory instance
60
- quads: [], // Final resolved graph state
61
- quadBuffer: new Map(), // Current parsing buffer
62
- removeSet: new Set(), // External retractions
63
- origin: { quadIndex: new Map() }, // Lean origin tracking
64
- currentSubject: null, // Current subject context
65
- primarySubject: null, // First non-fragment subject declaration
66
- primaryType: null, // First rdf:type declaration
67
- primaryLabel: null, // First rdfs:label literal
68
- tokens: null, // Tokenized input
69
- currentTokenIndex: -1, // Current processing position
70
- statements: [], // Elevated statements array
71
- statementCandidates: new Map() // Incomplete rdf:Statement patterns
72
- };
73
- ```
74
-
75
- #### State Components
76
-
77
- | Component | Purpose | Lifecycle |
78
- |-----------|---------|-----------|
79
- | **ctx** | Prefix mappings for IRI expansion | Persistent, updated during prefix processing |
80
- | **quads** | Final graph output | Accumulated throughout parsing |
81
- | **quadBuffer** | Intra-document cancel tracking | Modified during quad emission |
82
- | **removeSet** | External retractions | Accumulated for removal operations |
83
- | **origin** | Provenance tracking | Updated with each quad |
84
- | **currentSubject** | Subject context management | Reset and updated during parsing |
85
- | **primarySubject** | First non-fragment subject | Set once, never changed |
86
- | **primaryType** | First rdf:type declaration | Set once, never changed |
87
- | **primaryLabel** | First rdfs:label literal | Set once, never changed |
88
- | **statements** | Elevated statements | Accumulated during pattern detection |
89
- | **statementCandidates** | Pattern completion tracking | Modified during rdf:Statement detection |
90
-
91
- ### 2. Token Processing Pipeline
92
-
93
- #### Token Types
94
-
95
- ```javascript
96
- const TOKEN_TYPES = {
97
- heading: 'heading', // ## Heading {=ex:subject .type}
98
- para: 'para', // Paragraph with annotations
99
- list: 'list', // - List item {+ex:predicate}
100
- blockquote: 'blockquote', // > Quote {ex:property}
101
- code: 'code', // ```code {=ex:codeblock}
102
- prefix: 'prefix', // [prefix] <uri>
103
- fence: 'fence' // Code block delimiters
104
- };
105
- ```
106
-
107
- #### Token Structure
108
-
109
- ```javascript
110
- const token = {
111
- type: 'para', // Token type
112
- range: [start, end], // Character positions in source
113
- text: 'content', // Raw text content
114
- attrs: '{+ex:predicate}', // Annotation block
115
- attrsRange: [start, end], // Annotation positions
116
- valueRange: [start, end], // Value positions
117
- _carriers: [...] // Lazy-loaded inline carriers
118
- };
119
- ```
120
-
121
- #### Token Processors
122
-
123
- ```javascript
124
- const TOKEN_PROCESSORS = {
125
- heading: (token, state) => processTokenAnnotations(token, state, 'heading'),
126
- para: (token, state) => {
127
- processStandaloneSubject(token, state);
128
- processTokenAnnotations(token, state, 'para');
129
- },
130
- blockquote: (token, state) => processTokenAnnotations(token, state, 'blockquote'),
131
- code: (token, state) => processTokenAnnotations(token, state, 'code'),
132
- list: (token, state) => processTokenAnnotations(token, state, 'list')
133
- };
134
- ```
135
-
136
- ### 3. Semantic Block Processing
137
-
138
- #### Semantic Block Structure
139
-
140
- ```javascript
141
- const semanticBlock = {
142
- subjects: ['ex:subject'], // Zero or more subjects
143
- predicates: ['ex:predicate'], // Zero or more predicates
144
- types: ['ex:Type'], // Zero or more types
145
- object: 'literal', // Optional object value
146
- remove: false, // Polarity flag
147
- meta: { ... } // Additional metadata
148
- };
149
- ```
150
-
151
- #### Parsing Pipeline
152
-
153
- 1. **Annotation Extraction** - Identify annotation blocks in tokens
154
- 2. **Semantic Parsing** - Parse annotation blocks into semantic structures
155
- 3. **Subject Resolution** - Resolve subjects using current context
156
- 4. **Quad Generation** - Create RDF/JS quads from semantic structures
157
- 5. **Origin Tracking** - Record provenance for each quad
158
-
159
- ### 4. Retraction System
160
-
161
- Retractions are first-class citizens in MD-LD, enabling sophisticated diff authoring and document evolution through explicit polarity handling.
162
-
163
- #### Retraction Types
164
-
165
- ```javascript
166
- const RETRACTION_TYPES = {
167
- INTRA_DOCUMENT: 'intra-document', // Cancel within same document
168
- EXTERNAL_RETRACT: 'external', // Remove from prior state
169
- TYPE_MIGRATION: 'migration' // Change type assertions
170
- };
171
- ```
172
-
173
- #### Retraction State Management
174
-
175
- ```javascript
176
- const retractionState = {
177
- quadBuffer: new Map(), // Current document state
178
- removeSet: new Set(), // External retractions
179
- quads: [], // Final resolved state
180
- };
181
- ```
182
-
183
- #### Retraction Processing Pipeline
184
-
185
- ```javascript
186
- function processRetraction(quad, quadBuffer, removeSet, quads) {
187
- const quadKey = quadIndexKey(quad.subject, quad.predicate, quad.object);
188
-
189
- if (quadBuffer.has(quadKey)) {
190
- // Intra-document cancel: remove from current state
191
- quadBuffer.delete(quadKey);
192
- removeFromQuadsArray(quads, quad);
193
- } else {
194
- // External retract: target prior document state
195
- removeSet.add(quad);
196
- }
197
- }
198
- ```
199
-
200
- #### Polarity Forms
201
-
202
- | Form | Positive | Negative | Use Case |
203
- |------|-----------|-----------|-----------|
204
- | `p` | `{p}` | `{-p}` | Literal properties |
205
- | `?p` | `{?p}` | `{-?p}` | Object properties |
206
- | `!p` | `{!p}` | `{-!p}` | Reverse properties |
207
- | `.Class` | `{.Class}` | `-.Class}` | Type declarations |
208
-
209
- #### Retraction Examples
210
-
211
- ```javascript
212
- // Intra-document cancel
213
- "Alice has name 'Alice'" // {+ex:name "Alice"}
214
- "Alice {-ex:name 'Alice'}" // Cancels above, removed from final state
215
-
216
- // External retract
217
- "Bob {-ex:name 'Robert'}" // Targets Bob's name from prior documents
218
-
219
- // Type migration
220
- "Alice .foaf:Person" // Initial type
221
- "Alice {-foaf:Person}" // Remove old type
222
- "Alice .foaf:Agent" // Add new type
223
- ```
224
-
225
- #### Hard Invariant Enforcement
226
-
227
- ```javascript
228
- function enforceHardInvariant(quads, removeSet) {
229
- // Ensure: quads ∩ remove = ∅ (no overlap)
230
- const quadKeys = new Set();
231
- for (const quad of quads) {
232
- quadKeys.add(quadIndexKey(quad.subject, quad.predicate, quad.object));
233
- }
234
-
235
- // Filter removeSet to ensure no overlap
236
- return Array.from(removeSet).filter(quad => {
237
- const key = quadIndexKey(quad.subject, quad.predicate, quad.object);
238
- return !quadKeys.has(key);
239
- });
240
- }
241
- ```
242
-
243
- ### 5. Carrier Extraction System
244
-
245
- #### Carrier Types
246
-
247
- ```javascript
248
- const CARRIER_TYPES = {
249
- link: 'link', // [text] {+ex:predicate}
250
- emphasis: 'emphasis', // **text** {ex:property}
251
- code: 'code', // ``code`` {ex:property}
252
- bracket: 'bracket' // <url> {ex:reference}
253
- };
254
- ```
255
-
256
- #### Carrier Structure
257
-
258
- ```javascript
259
- const carrier = {
260
- type: 'link', // Carrier type
261
- text: 'display text', // Display content
262
- attrs: '{+ex:predicate}', // Annotation block
263
- attrsRange: [start, end], // Annotation positions
264
- valueRange: [start, end], // Value positions
265
- range: [start, end], // Full span in source
266
- pos: endPosition, // Next processing position
267
- url: 'resolved-iri' // Resolved IRI (for links)
268
- };
269
- ```
270
-
271
- #### Extraction Patterns
272
-
273
- ```javascript
274
- const CARRIER_EXTRACTORS = {
275
- '<': extractLinkCarrier, // Angle bracket links
276
- '[': extractBracketCarrier, // Square bracket carriers
277
- emphasis: extractEmphasis, // **emphasis** patterns
278
- code: extractCodeSpan // ``code`` patterns
279
- };
280
- ```
281
-
282
- ### 5. Quad Emission System
283
-
284
- #### Emit Quad Function
285
-
286
- ```javascript
287
- function emitQuad(quads, quadBuffer, removeSet, quadIndex, block,
288
- subject, predicate, object, dataFactory, meta,
289
- statements, statementCandidates) {
290
-
291
- // 1. Create RDF/JS quad
292
- const quad = dataFactory.quad(subject, predicate, object);
293
- const remove = meta?.remove || false;
294
-
295
- // 2. Handle polarity (add/remove)
296
- if (remove) {
297
- // Retraction processing
298
- const quadKey = quadIndexKey(quad.subject, quad.predicate, quad.object);
299
-
300
- if (quadBuffer.has(quadKey)) {
301
- // Intra-document cancel: remove from current state
302
- quadBuffer.delete(quadKey);
303
- removeFromQuadsArray(quads, quad);
304
- quadIndex.delete(quadKey);
305
- } else {
306
- // External retract: target prior document state
307
- removeSet.add(quad);
308
- }
309
- } else {
310
- // Add to buffer and quads array
311
- const quadKey = quadIndexKey(quad.subject, quad.predicate, quad.object);
312
- quadBuffer.set(quadKey, quad);
313
- quads.push(quad);
314
-
315
- // Detect elevated statements
316
- detectStatementPatternSinglePass(quad, dataFactory, meta, statements, statementCandidates);
317
-
318
- // Create origin entry
319
- const originEntry = createOriginEntry(block, subject, predicate, meta);
320
- quadIndex.set(quadKey, originEntry);
321
- }
322
- }
323
- ```
324
-
325
- #### Retraction-Aware Quad Processing
326
-
327
- ```javascript
328
- function removeFromQuadsArray(quads, targetQuad) {
329
- const index = quads.findIndex(q =>
330
- q.subject.value === targetQuad.subject.value &&
331
- q.predicate.value === targetQuad.predicate.value &&
332
- q.object.value === targetQuad.object.value
333
- );
334
-
335
- if (index !== -1) {
336
- quads.splice(index, 1);
337
- }
338
- }
339
- ```
340
-
341
- #### Origin Entry Creation with Polarity
342
-
343
- ```javascript
344
- function createOriginEntry(block, subject, predicate, meta) {
345
- return {
346
- blockId: block.id,
347
- range: block.range,
348
- carrierType: block.carrierType,
349
- subject: subject.value,
350
- predicate: predicate.value,
351
- context: block.context,
352
- polarity: meta?.remove ? '-' : '+', // Polarity tracking
353
- value: block.text || ''
354
- };
355
- }
356
- ```
357
-
358
- #### Quad Index Key Generation
359
-
360
- ```javascript
361
- function quadIndexKey(subject, predicate, object) {
362
- return `${subject.value}|${predicate.value}|${object.value}`;
363
- }
364
- ```
365
-
366
- ### 6. Primary Metadata Extraction
367
-
368
- #### First-Occurrence Tracking
369
-
370
- The parser tracks primary metadata during single-pass parsing to provide immediate access to commonly needed document identity:
371
-
372
- ```javascript
373
- // Track primary subject: first non-fragment subject declaration
374
- if (newSubject && !state.primarySubject && !sem.subject.startsWith('=#')) {
375
- state.primarySubject = newSubject.value; // Store as string IRI
376
- }
377
-
378
- // Track primary type and label during quad emission
379
- if (state) {
380
- if (!state.primaryType && predicate.value === 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type') {
381
- state.primaryType = object.value;
382
- }
383
- if (!state.primaryLabel && predicate.value === 'http://www.w3.org/2000/01/rdf-schema#label' && object.termType === 'Literal') {
384
- state.primaryLabel = object.value;
385
- }
386
- }
387
- ```
388
-
389
- #### Primary Metadata Trio
390
-
391
- | Field | Source | Purpose | Use Case |
392
- |-------|--------|---------|-----------|
393
- | **primarySubject** | First non-fragment `{=subject}` | Document identity | Nostr addressing tags, document routing |
394
- | **primaryType** | First `.Class` declaration | Document category | Stream filtering, UI categorization |
395
- | **primaryLabel** | First `{label}` literal | Human-readable name | Display titles, search indexing |
396
-
397
- #### Document Identity Applications
398
-
399
- The primary metadata trio provides sufficient document/append stream identity for most scenarios:
400
-
401
- ```javascript
402
- // Nostr integration example
403
- const nostrTags = [
404
- ['s', result.primarySubject], // subject addressing
405
- ['t', result.primaryType], // type filtering
406
- ['d', result.primaryLabel] // display identifier
407
- ];
408
- ```
409
-
410
- **Benefits:**
411
- - **Immediate access** - No post-processing required
412
- - **Deterministic** - First-occurrence semantics
413
- - **Lightweight** - Minimal parsing overhead
414
- - **Sufficient** - Covers 80% of use cases
415
-
416
- ### 7. Elevated Statements Detection
417
-
418
- #### Pattern Detection System
419
-
420
- ```javascript
421
- function detectStatementPatternSinglePass(quad, dataFactory, meta,
422
- statements, statementCandidates) {
423
- // Only process rdf:Statement related predicates (optimized)
424
- if (!isRDFStatementPredicate(quad.predicate.value)) return;
425
-
426
- // Start new pattern: rdf:type rdf:Statement
427
- if (isStatementType(quad)) {
428
- statementCandidates.set(quad.subject.value, { spo: {} });
429
- return;
430
- }
431
-
432
- // Complete pattern parts: rdf:subject, rdf:predicate, rdf:object
433
- const candidate = statementCandidates.get(quad.subject.value);
434
- if (candidate) {
435
- updatePattern(candidate, quad);
436
-
437
- // Create elevated SPO when pattern complete
438
- if (isPatternComplete(candidate)) {
439
- const elevatedQuad = createElevatedQuad(candidate, dataFactory);
440
- statements.push(elevatedQuad);
441
- statementCandidates.delete(quad.subject.value);
442
- }
443
- }
444
- }
445
- ```
446
-
447
- #### Pattern Completion Tracking
448
-
449
- ```javascript
450
- const patternCandidate = {
451
- spo: {
452
- subject: NamedNode, // rdf:subject value
453
- predicate: NamedNode, // rdf:predicate value
454
- object: NamedNode|Literal // rdf:object value
455
- },
456
- objectQuad: Quad // Original quad for literal extraction
457
- };
458
- ```
459
-
460
- ### 7. Origin Tracking System
461
-
462
- #### Origin Entry Structure
463
-
464
- ```javascript
465
- const originEntry = {
466
- blockId: 'block-identifier', // Unique block identifier
467
- range: [start, end], // Source location
468
- carrierType: 'para', // Token type
469
- subject: 'ex:subject', // Subject IRI
470
- predicate: 'ex:predicate', // Predicate IRI
471
- context: { ...block.context }, // Prefix context
472
- polarity: '+', // Add/remove indicator
473
- value: 'source text' // Original content
474
- };
475
- ```
476
-
477
- #### Quad Index Mapping
478
-
479
- ```javascript
480
- const quadIndex = new Map(); // quadIndexKey -> originEntry
481
- ```
482
-
483
- ## Processing Pipeline
484
-
485
- ### Phase 1: Tokenization
486
-
487
- ```javascript
488
- function scanTokens(text) {
489
- const tokens = [];
490
- const lines = text.split('\n');
491
-
492
- for (const line of lines) {
493
- // Identify token types using regex patterns
494
- // Extract annotations and ranges
495
- // Create token objects with metadata
496
- tokens.push(token);
497
- }
498
-
499
- return tokens;
500
- }
501
- ```
502
-
503
- ### Phase 2: Prefix Processing
504
-
505
- ```javascript
506
- // Single loop optimization for better performance
507
- for (const token of tokens) {
508
- if (token.type === 'prefix') {
509
- // Resolve CURIEs and update context
510
- state.ctx[token.prefix] = resolveIRI(token.iri, state.ctx);
511
- }
512
- }
513
- ```
514
-
515
- ### Phase 3: Semantic Processing
516
-
517
- ```javascript
518
- for (let i = 0; i < tokens.length; i++) {
519
- const token = tokens[i];
520
- state.currentTokenIndex = i;
521
-
522
- // Process token using appropriate processor
523
- TOKEN_PROCESSORS[token.type]?.(token, state);
524
- }
525
- ```
526
-
527
- ### Phase 4: Final Resolution
528
-
529
- ```javascript
530
- // Process retractions and ensure hard invariant
531
- const filteredRemove = processRetractions(state.quads, state.removeSet);
532
-
533
- // Create structured primary object for semantic surface
534
- const primary = {
535
- subject: state.primarySubject,
536
- type: state.primaryType,
537
- label: state.primaryLabel,
538
- comment: state.primaryComment
539
- };
540
-
541
- return {
542
- quads: state.quads, // Final resolved graph state
543
- remove: filteredRemove, // External retractions only
544
- statements: state.statements, // Elevated statements
545
- origin: state.origin, // Complete provenance
546
- context: state.ctx, // Final prefix context
547
- primarySubject: state.primarySubject, // Canonical append identity
548
- primary // Semantic surface descriptor
549
- };
550
- ```
551
-
552
- #### Retraction Processing
553
-
554
- ```javascript
555
- function processRetractions(quads, removeSet) {
556
- // Create quad key index for fast lookup
557
- const quadKeys = new Set();
558
- for (const quad of quads) {
559
- quadKeys.add(quadIndexKey(quad.subject, quad.predicate, quad.object));
560
- }
561
-
562
- // Filter removeSet to enforce hard invariant: quads ∩ remove = ∅
563
- const filteredRemove = [];
564
- for (const quad of removeSet) {
565
- const key = quadIndexKey(quad.subject, quad.predicate, quad.object);
566
- if (!quadKeys.has(key)) {
567
- filteredRemove.push(quad);
568
- }
569
- }
570
-
571
- return filteredRemove;
572
- }
573
- ```
574
-
575
- ## Performance Optimizations
576
-
577
- ### Applied Optimizations
578
-
579
- 1. **Token Processing**: Single loop instead of filter+forEach
580
- 2. **Regex Pre-compilation**: Pre-compiled carrier patterns
581
- 3. **String Operations**: Optimized fragment resolution
582
- 4. **Array Operations**: Direct iteration instead of Array.from()+filter()
583
- 5. **Object Creation**: Reduced spread operator usage
584
- 6. **Early Filtering**: Only process rdf:Statement related predicates
585
- 7. **Module Constants**: RDF constants defined once at module level
586
-
587
- ### Performance Characteristics
588
-
589
- - **Single-pass parsing**: No backtracking or multiple iterations
590
- - **O(1) additional memory**: Constant overhead for elevated statements
591
- - **Linear scalability**: Performance scales linearly with input size
592
- - **Streaming compatible**: Can process large files incrementally
593
-
594
- ## Memory Management
595
-
596
- ### Memory Usage Patterns
597
-
598
- ```javascript
599
- // Temporary objects (cleaned up automatically)
600
- const statementCandidates = new Map(); // Cleared on pattern completion
601
- const quadBuffer = new Map(); // Cleared on intra-document cancel
602
-
603
- // Persistent objects (grow with input)
604
- state.quads.push(quad); // Final output
605
- state.origin.quadIndex.set(key, entry); // Provenance tracking
606
- ```
607
-
608
- ### Cleanup Strategies
609
-
610
- 1. **Automatic Cleanup**: Statement candidates deleted on completion
611
- 2. **Buffer Management**: Quad buffer cleared on intra-document cancel
612
- 3. **Set Operations**: RemoveSet converted to array at end
613
- 4. **Lazy Loading**: Inline carriers loaded on-demand
614
-
615
- ## Error Handling
616
-
617
- ### Error Recovery Strategies
618
-
619
- ```javascript
620
- // Graceful degradation for malformed input
621
- if (!isValidAnnotation(attrs)) {
622
- // Log warning and continue processing
623
- return;
624
- }
625
-
626
- // Safe navigation for optional properties
627
- const subject = resolveSubject(sem.subject, state) || skipProcessing();
628
-
629
- // Validation with meaningful errors
630
- if (!state.currentSubject && needsSubject(sem)) {
631
- throw new ParseError('Subject required but not set', token.range);
632
- }
633
- ```
634
-
635
- ## Extensibility Points
636
-
637
- ### Custom Token Processors
638
-
639
- ```javascript
640
- // Add new token type
641
- TOKEN_PROCESSORS.custom = (token, state) => {
642
- // Custom processing logic
643
- processTokenAnnotations(token, state, 'custom');
644
- };
645
- ```
646
-
647
- ### Custom Carrier Extractors
648
-
649
- ```javascript
650
- // Add new carrier pattern
651
- CARRIER_EXTRACTORS['@'] = (text, pos, baseOffset) => {
652
- // Custom carrier extraction
653
- return createCarrier('mention', text, attrs, ranges);
654
- };
655
- ```
656
-
657
- ### Custom Semantic Parsers
658
-
659
- ```javascript
660
- // Extend semantic block parsing
661
- function parseCustomSemantic(attrs) {
662
- // Custom annotation parsing
663
- return { customProperty: value };
664
- }
665
- ```
666
-
667
- ## Integration Points
668
-
669
- ### External Libraries
670
-
671
- - **RDF/JS DataFactory**: Quad creation and term management
672
- - **Context Expansion**: IRI resolution and prefix handling
673
- - **Validation**: Optional SHACL or schema validation
674
-
675
- ### Output Formats
676
-
677
- ```javascript
678
- // Standard RDF/JS output with first-class retractions
679
- return {
680
- quads: RDFQuad[], // Primary graph (final resolved state)
681
- remove: RDFQuad[], // External retractions (targets prior state)
682
- statements: RDFQuad[], // Elevated statements from rdf:Statement patterns
683
- origin: OriginObject, // Complete provenance tracking
684
- context: ContextObject // Final prefix context
685
- };
686
- ```
687
-
688
- #### Retraction Output Characteristics
689
-
690
- ```javascript
691
- // External retractions only (intra-document cancels resolved internally)
692
- const remove = [
693
- // Quads that target prior document state
694
- // Never overlap with quads array (hard invariant enforced)
695
- ];
696
-
697
- // Usage for document evolution
698
- const priorResult = parse(previousDocument);
699
- const currentResult = parse(currentDocument);
700
-
701
- // Apply retractions to prior state
702
- const evolvedGraph = applyRetractions(priorResult.quads, currentResult.remove);
703
- ```
704
-
705
- ### Streaming Interface
706
-
707
- ```javascript
708
- // Future streaming capability
709
- function parseStream(stream, options) {
710
- return new TransformStream({
711
- transform(chunk, controller) {
712
- // Process chunk incrementally
713
- const result = parse(chunk.toString(), options);
714
- controller.enqueue(result);
715
- }
716
- });
717
- }
718
- ```
719
-
720
- This architecture provides a solid foundation for MD-LD parsing with excellent performance characteristics, maintainability, and extensibility for future enhancements.