mdld-parse 0.9.1 → 0.9.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (59) hide show
  1. package/README.md +34 -1
  2. package/dist/.nojekyll +0 -0
  3. package/dist/CNAME +1 -0
  4. package/dist/README.md +425 -0
  5. package/dist/docs/API.md +397 -0
  6. package/dist/docs/Architecture.md +190 -0
  7. package/dist/docs/Guide.md +322 -0
  8. package/dist/docs/LLM-MDLD-Inducer.md +550 -0
  9. package/dist/docs/Parser.md +654 -0
  10. package/dist/docs/Performance.md +238 -0
  11. package/dist/docs/Spec-render.md +1439 -0
  12. package/dist/docs/Subject.md +180 -0
  13. package/dist/docs/Syntax.md +463 -0
  14. package/dist/docs/TodoList.md +215 -0
  15. package/dist/docs/Use-Cases.md +354 -0
  16. package/dist/docs/index.html +108 -0
  17. package/dist/docs/index.md +56 -0
  18. package/dist/docs/origin.md +459 -0
  19. package/dist/docs/polarity.md +586 -0
  20. package/dist/docs/primary-subject.md +199 -0
  21. package/dist/docs/statements.md +226 -0
  22. package/dist/docs/tokenization-performance-report.md +164 -0
  23. package/dist/examples/Cookbook.md +32 -0
  24. package/dist/examples/Status-SHACL.md +33 -0
  25. package/dist/examples/demo/index.md +35 -0
  26. package/dist/examples/demo/journal.md +18 -0
  27. package/dist/examples/demo/minimal.md +11 -0
  28. package/dist/examples/demo/project.md +15 -0
  29. package/dist/examples/demo/recipe.md +31 -0
  30. package/dist/examples/demo/research.md +46 -0
  31. package/dist/examples/demo/space-mission.md +48 -0
  32. package/dist/examples/dogfood.md +288 -0
  33. package/dist/examples/few-shot.md +309 -0
  34. package/dist/examples/graph/rdf-graph.html +473 -0
  35. package/dist/examples/graph/rdf_force_graph_web_component.html +506 -0
  36. package/dist/examples/index.md +45 -0
  37. package/dist/examples/llm-time-workflow.md +173 -0
  38. package/dist/examples/prov-o.md +2367 -0
  39. package/dist/examples/rdf.md +827 -0
  40. package/dist/examples/shacl.md +3399 -0
  41. package/dist/examples/statements.md +43 -0
  42. package/dist/examples/workflow.md +150 -0
  43. package/dist/index.html +2827 -0
  44. package/dist/index.js +1548 -0
  45. package/dist/logo.svg +7 -0
  46. package/dist/spec/Spec-compact.md +363 -0
  47. package/dist/spec/Spec-ultra.md +221 -0
  48. package/dist/spec/Spec.md +1068 -0
  49. package/dist/spec/index.md +46 -0
  50. package/package.json +13 -8
  51. package/src/constants.js +0 -31
  52. package/src/generate.js +0 -245
  53. package/src/index.js +0 -13
  54. package/src/locate.js +0 -23
  55. package/src/merge.js +0 -157
  56. package/src/parse.js +0 -765
  57. package/src/render.js +0 -576
  58. package/src/shared.js +0 -577
  59. package/src/utils.js +0 -430
@@ -0,0 +1,199 @@
1
+ # Primary Subject Example
2
+
3
+ This example demonstrates the Primary Subject feature in MD-LD, which identifies the main entity described by a document.
4
+
5
+ ## What is a Primary Subject?
6
+
7
+ The Primary Subject is the first non-fragment subject declaration (`{=IRI}`) in a document. It provides a deterministic way to identify the central focus of a document.
8
+
9
+ ## Example Document
10
+
11
+ ```markdown
12
+ [ex] <http://example.org/>
13
+
14
+ # Article {=ex:article .schema:Article label}
15
+ [Understanding Primary Subjects] {schema:headline}
16
+
17
+ ## Author {=ex:author .schema:Person}
18
+ [Alice Smith] {schema:name}
19
+ [alice@example.com] {schema:email}
20
+
21
+ ## Abstract
22
+ This article explains the Primary Subject concept in MD-LD, which allows applications to quickly identify the main entity described by a document.
23
+ ```
24
+
25
+ ## Parsing with Primary Subject
26
+
27
+ ```javascript
28
+ import { parse } from 'mdld-parse';
29
+
30
+ const md = `[ex] <http://example.org/>
31
+ [schema] <http://schema.org/>
32
+
33
+ # Article {=ex:article .schema:Article label}
34
+ [Understanding Primary Subjects] {schema:headline}`;
35
+
36
+ const result = parse({ text: md });
37
+
38
+ console.log(result.primarySubject);
39
+ // Output: http://example.org/article
40
+ ```
41
+
42
+ ## Key Behaviors
43
+
44
+ ### 1. First Non-Fragment Subject Wins
45
+
46
+ ```markdown
47
+ {=#summary} // Fragment - NOT primary
48
+ [Summary] {label}
49
+
50
+ # Document {=ex:doc} // First non-fragment - PRIMARY
51
+ [Alice] {author}
52
+ ```
53
+
54
+ Primary subject: `ex:doc`
55
+
56
+ ### 2. Reset Does Not Clear Primary Subject
57
+
58
+ ```markdown
59
+ # First {=ex:first}
60
+ [Value] {label}
61
+
62
+ # Reset {=} // Does not clear primary subject
63
+
64
+ # Second {=ex:second}
65
+ [Value] {label}
66
+ ```
67
+
68
+ Primary subject: `ex:first` (fixed once detected)
69
+
70
+ ### 3. No Subject Means No Primary
71
+
72
+ ```markdown
73
+ [Just text] {label}
74
+ ```
75
+
76
+ Primary subject: `null`
77
+
78
+ ## Merge with Primary Subjects
79
+
80
+ When merging multiple documents, the `merge()` function returns an array of all primary subjects in merge order:
81
+
82
+ ```javascript
83
+ import { merge } from 'mdld-parse';
84
+
85
+ const doc1 = `[ex] <http://example.org/>
86
+ # Document 1 {=ex:doc1}
87
+ [Content] {label}`;
88
+
89
+ const doc2 = `[ex] <http://example.org/>
90
+ # Document 2 {=ex:doc2}
91
+ [Content] {label}`;
92
+
93
+ const result = merge([doc1, doc2], { context: { ex: 'http://example.org/' } });
94
+
95
+ console.log(result.primarySubjects.map(s => s.value));
96
+ // Output: ['http://example.org/doc1', 'http://example.org/doc2']
97
+ ```
98
+
99
+ ## Use Cases
100
+
101
+ - **Document Identification** - Quickly determine what a document is about
102
+ - **Merge Tracking** - Track which entities are being merged across documents
103
+ - **UI Navigation** - Provide a default focus point for document viewers
104
+ - **Query Optimization** - Use primary subject as default query context
105
+
106
+ ## RDF Philosophy Alignment
107
+
108
+ The Primary Subject feature aligns with core W3C/RDF principles:
109
+
110
+ ### Document Identity
111
+ RDF 1.1 Named Graphs recognize the importance of document-centric RDF. Primary Subject makes document identity explicit without forcing graph-centric assumptions, supporting the document-centric approach (FOAF profiles, RSS feeds) that's part of the Linked Data ecosystem.
112
+
113
+ ### Provenance Tracking (PROV-O)
114
+ The W3C PROV-O standard emphasizes tracking data origin. Primary Subject provides a deterministic way to identify what a document is "about," which is foundational for provenance chains and attribution.
115
+
116
+ ### Dereferenceability
117
+ Linked Data principles emphasize that IRIs should be dereferenceable. Having a clear primary subject makes it immediately clear what IRI to dereference when encountering a document, supporting the "follow your nose" navigation pattern.
118
+
119
+ ### Minimal Commitment
120
+ RDF's philosophy is about making minimal assertions. Primary Subject is lightweight metadata - it doesn't add triples to the graph, just provides a pointer to the main subject. This aligns with MDLD's "no implicit semantics" principle.
121
+
122
+ ### Explicit over Implicit
123
+ The W3C encourages explicit semantics over inference. Primary Subject makes document focus explicit rather than requiring heuristic analysis of the graph to determine "what this document is about."
124
+
125
+ ## Grounded Documents: Long-Term Benefits
126
+
127
+ Primary Subject enables a new class of semantic applications that treat documents as meaningful units in knowledge workflows:
128
+
129
+ ### Document-Centric Knowledge Graphs
130
+ - Citation networks with clear document identity
131
+ - Document versioning and evolution tracking
132
+ - Cross-document referencing and linking
133
+ - Entity-focused provenance chains
134
+
135
+ ### Semantic Search and Discovery
136
+ - Index documents by their main entity
137
+ - Support queries like "find all documents about X" without complex graph pattern matching
138
+ - Entity-based relevance scoring
139
+
140
+ ### Incremental Graph Construction
141
+ - Determine which documents describe the same entity (deduplication)
142
+ - Smart merge order and priority decisions
143
+ - Conflict resolution strategies based on entity overlap
144
+ - Graph topology construction guidance
145
+
146
+ ### Collaborative Semantic Editing
147
+ - Track which entities are being modified across authors
148
+ - Coordinate concurrent edits at entity level
149
+ - Detect merge conflicts at entity granularity
150
+ - Provide contextual editing assistance
151
+
152
+ ### Semantic Version Control
153
+ - Entity-level version tracking within document systems
154
+ - Queries like "show me how entity X has changed across versions"
155
+ - Entity-focused diff and merge operations
156
+ - Granular change attribution
157
+
158
+ ### Knowledge Graph Federation
159
+ - Identify overlapping entities across different sources
160
+ - Determine authoritative sources for specific entities
161
+ - Build confidence scores based on document provenance
162
+ - Source-aware query planning
163
+
164
+ ### AI/LLM Integration
165
+ - Primary subject provides clear context for document understanding
166
+ - Entity-focused prompt engineering
167
+ - Document retrieval by entity relevance
168
+ - Improved fact-checking and attribution
169
+
170
+ The Primary Subject feature bridges document-centric and graph-centric approaches, enabling semantic applications that treat documents as first-class citizens in knowledge workflows while respecting RDF's core principles of explicit semantics and minimal commitment.
171
+
172
+ ## Complete Example
173
+
174
+ ```javascript
175
+ import { parse, merge } from 'mdld-parse';
176
+
177
+ // Single document
178
+ const doc1 = `[blog] <https://example.com/blog/>
179
+ # My First Post {=blog:post1 .schema:BlogPosting label}
180
+ [Hello World] {schema:headline}`;
181
+
182
+ const result1 = parse({ text: doc1, context: { blog: 'https://example.com/blog/', schema: 'http://schema.org/' } });
183
+ console.log('Primary subject:', result1.primarySubject);
184
+ // Output: https://example.com/blog/post1
185
+
186
+ // Multiple documents
187
+ const doc2 = `[blog] <https://example.com/blog/>
188
+ # My Second Post {=blog:post2 .schema:BlogPosting label}
189
+ [Hello Again] {schema:headline}`;
190
+
191
+ const merged = merge([doc1, doc2], { context: { blog: 'https://example.com/blog/', schema: 'http://schema.org/' } });
192
+
193
+ console.log('Primary subjects:', merged.primarySubjects);
194
+ // Output: ['https://example.com/blog/post1', 'https://example.com/blog/post2']
195
+ ```
196
+
197
+ ## Specification
198
+
199
+ For complete details on Primary Subject semantics, see the [MD-LD Specification](../spec/Spec.md#17-primary-subject).
@@ -0,0 +1,226 @@
1
+ # Elevated Statements
2
+
3
+ MD-LD provides **elevated statements** through automatic `rdf:Statement` pattern detection, enabling authors to create a "golden graph" of intentionally elevated statements while maintaining complete provenance in the underlying data.
4
+
5
+ ## 🎯 Concept Overview
6
+
7
+ Elevated statements allow you to:
8
+ - **Separate signal from noise** - Dashboard apps can use only the elevated statements
9
+ - **Maintain provenance** - Complete context remains available in the full graph
10
+ - **Single-pass parsing** - No performance overhead, detected during normal parsing
11
+ - **Flexible patterns** - Support both IRI references and direct literals
12
+
13
+ ## 📋 Pattern Requirements
14
+
15
+ For a statement to be elevated, it must have:
16
+ 1. **rdf:type rdf:Statement** - The subject must be typed as an RDF Statement
17
+ 2. **rdf:subject** - The subject of the elevated SPO triple
18
+ 3. **rdf:predicate** - The predicate of the elevated SPO triple
19
+ 4. **rdf:object** - The object of the elevated SPO triple
20
+
21
+ When all four parts are present, the parser automatically creates an elevated SPO quad in the `statements` array.
22
+
23
+ ## 🔧 Syntax Patterns
24
+
25
+ ### Basic Pattern with IRIs
26
+
27
+ ```markdown
28
+ [ex] <http://example.org/>
29
+
30
+ ## Elevated statement {=ex:stmt1 .rdf:Statement}
31
+ **Alice** {+ex:alice ?rdf:subject} *knows* {+ex:knows ?rdf:predicate} **Bob** {+ex:bob ?rdf:object}
32
+
33
+ **Alice** {=ex:alice} knows **Bob** {?ex:knows +ex:bob} - direct statement
34
+ ```
35
+
36
+ ### Pattern with Direct Literals
37
+
38
+ For datatypes and language tags, use direct literals:
39
+
40
+ ```markdown
41
+ [ex] <http://example.org/>
42
+
43
+ ## Statement with datatype {=ex:stmt1 .rdf:Statement}
44
+ **Alice** {+ex:alice ?rdf:subject} *has age* {+ex:hasAge ?rdf:predicate} **25** {rdf:object ^^xsd:integer}
45
+
46
+ ## Statement with language {=ex:stmt2 .rdf:Statement}
47
+ **Alice** {+ex:alice ?rdf:subject} *has name* {+ex:hasName ?rdf:predicate} **Alice** {rdf:object @en}
48
+ ```
49
+
50
+ ### Real-World Example
51
+
52
+ ```markdown
53
+ [my] <tag:alice@example.com,2026:>
54
+ [foaf] <http://xmlns.com/foaf/0.1/>
55
+
56
+ My name is **Alice** {=my:Alice .foaf:Person foaf:name}.
57
+
58
+ {=}
59
+
60
+ Today I have learned that my colleague name is [Clair] {=my:Claire .foaf:Person foaf:name}
61
+
62
+ ## I know Claire's name {=my:claire-name .rdf:Statement}
63
+
64
+ [My colleague] {+my:Claire ?rdf:subject} [name] {+foaf:name ?rdf:predicate} is [Clair] {rdf:object}.
65
+
66
+ {=}
67
+
68
+ ## We've talked! {=my:claire-first-talk .rdf:Statement}
69
+
70
+ Today [I] {+my:Alice ?rdf:subject} came to office a bit earlier and [talked for some time] {+foaf:knows ?rdf:predicate} with [Claire] {+my:Claire ?rdf:object}.
71
+
72
+ Now **I** {=my:Alice} know **Claire** {+my:Claire ?foaf:knows}.
73
+ ```
74
+
75
+ ## 🚀 Usage in Applications
76
+
77
+ ### API Access
78
+
79
+ ```javascript
80
+ import { parse } from 'mdld-parse';
81
+
82
+ const result = parse({ text: `# Test {=tag:me@example.org,2026:test .prov:Entity label}`});
83
+ console.log(result.quads); // All quads including provenance
84
+ console.log(result.statements); // Elevated SPO quads (detected during parsing)
85
+ ```
86
+
87
+ ### Dashboard Applications
88
+
89
+ Dashboard applications can:
90
+
91
+ - **Ignore provenance-rich data** by using only `result.statements`
92
+ - **Access the "golden graph"** of intentionally elevated statements
93
+ - **Maintain complete provenance** in `result.quads` when needed
94
+ - **Filter by context** using the origin tracking system
95
+
96
+ ### Example: Simple Graph Display
97
+
98
+ ```javascript
99
+ import { parse } from 'mdld-parse';
100
+
101
+ function displayElevatedGraph(markdown) {
102
+ const result = parse({text: markdown});
103
+
104
+ // Display only elevated statements
105
+ result.statements.forEach(quad => {
106
+ console.log(`${quad.subject.value} → ${quad.predicate.value} → ${quad.object.value}`);
107
+ });
108
+
109
+ return result.statements.length; // Number of elevated statements
110
+ }
111
+ ```
112
+
113
+ ## 📊 Data Flow
114
+
115
+ ```
116
+ MD-LD Document
117
+ ↓ (single-pass parsing)
118
+ All Quads + Elevated Statements
119
+ ↓
120
+ ┌─────────────────┬─────────────────┐
121
+ │ result.quads │result.statements│
122
+ │ (provenance) │ (golden graph) │
123
+ └─────────────────┴─────────────────┘
124
+ ```
125
+
126
+ ## 🎨 Best Practices
127
+
128
+ ### 1. Use Descriptive Statement IDs
129
+
130
+ ```markdown
131
+ ## Alice knows Bob {=ex:alice-knows-bob .rdf:Statement}
132
+ ```
133
+
134
+ ### 2. Preserve Datatypes and Languages
135
+
136
+ ```markdown
137
+ ## Person's birth date {=ex:birth-date .rdf:Statement}
138
+ **Alice** {+ex:alice ?rdf:subject} *born on* {+ex:bornOn ?rdf:predicate} **1990-01-01** {rdf:object ^^xsd:date}
139
+ ```
140
+
141
+ ### 3. Mix Elevated and Regular Statements
142
+
143
+ ```markdown
144
+ ## Elevated: Important fact {=ex:important .rdf:Statement}
145
+ **Alice** {+ex:alice ?rdf:subject} *knows* {+ex:knows ?rdf:predicate} **Bob** {+ex:bob ?rdf:object}
146
+
147
+ **Alice** {=ex:alice} also knows **Charlie** {?ex:knows +ex:charlie} - regular statement
148
+ ```
149
+
150
+ ### 4. Use Subject Resets for Organization
151
+
152
+ ```markdown
153
+ {=}
154
+
155
+ ## New section {=ex:section-stmt .rdf:Statement}
156
+ **Fact** {+ex:fact ?rdf:subject} *has value* {+ex:hasValue ?rdf:predicate} **42** {rdf:object ^^xsd:integer}
157
+ ```
158
+
159
+ ## 🔍 Technical Details
160
+
161
+ ### Single-Pass Detection
162
+
163
+ Elevated statements are detected during the normal parsing process:
164
+
165
+ 1. **Quad emission** - As each quad is created, the pattern detector runs
166
+ 2. **Candidate tracking** - Incomplete patterns are tracked in memory
167
+ 3. **Pattern completion** - When all parts are present, elevated SPO is created
168
+ 4. **Cleanup** - Completed patterns are removed from tracking
169
+
170
+ ### Memory Efficiency
171
+
172
+ - **O(1) additional memory** - Only tracks incomplete patterns
173
+ - **Automatic cleanup** - Completed patterns are removed immediately
174
+ - **No backtracking** - True single-pass processing
175
+
176
+ ### Compatibility
177
+
178
+ - **RDF/JS compliant** - Elevated statements are standard RDF quads
179
+ - **Toolchain compatible** - Works with n3.js, rdflib, and other RDF libraries
180
+ - **Streaming friendly** - No additional passes required
181
+
182
+ ## 📝 Comparison with Alternatives
183
+
184
+ | Approach | Provenance | Performance | Flexibility | Complexity |
185
+ |----------|------------|-------------|-------------|------------|
186
+ | **Elevated Statements** | ✅ Complete | ✅ Single-pass | ✅ High | 🟡 Medium |
187
+ | **Named Graphs** | ✅ Complete | ⚡ Multiple passes | 🟡 Medium | 🔴 High |
188
+ | **Manual Filtering** | ✅ Complete | ✅ Single-pass | 🔴 Low | 🟢 Low |
189
+ | **Separate Documents** | ❌ Lost | ✅ Single-pass | 🔴 Low | 🟢 Low |
190
+
191
+ ## 🎯 Use Cases
192
+
193
+ ### Knowledge Management
194
+ - Elevate important facts while preserving context
195
+ - Create "golden graph" for search and discovery
196
+ - Maintain full provenance for audit trails
197
+
198
+ ### API Design
199
+ - Provide simplified endpoints with elevated statements only
200
+ - Offer detailed endpoints with full provenance
201
+ - Enable progressive disclosure of information
202
+
203
+ ### Content Management
204
+ - Separate published content from editorial metadata
205
+ - Create curated views while preserving edits
206
+ - Support content approval workflows
207
+
208
+ ### Data Integration
209
+ - Elevate validated mappings while preserving transformation rules
210
+ - Create clean integration graphs with full audit trails
211
+ - Support data lineage and governance
212
+
213
+ ## 🧪 Testing
214
+
215
+ The elevated statements feature is covered by comprehensive tests:
216
+
217
+ ```bash
218
+ npm test # All 105 tests pass, including elevated statements
219
+ ```
220
+
221
+ Test coverage includes:
222
+ - Basic pattern detection
223
+ - Multiple elevated statements
224
+ - Incomplete patterns (ignored)
225
+ - Mixed content scenarios
226
+ - Datatype and language preservation
@@ -0,0 +1,164 @@
1
+ # MDLD Tokenization Performance Analysis Report
2
+
3
+ ## Executive Summary
4
+
5
+ **Character-based tokenization significantly outperforms regex-based approach** across all document sizes, with 8-12x faster processing and competitive memory usage. This validates our architectural decision to adopt the highlighting approach for the parser.
6
+
7
+ ## Key Findings
8
+
9
+ ### 🚀 Performance Superiority
10
+ - **8-12x faster** than regex-based approach across all document sizes
11
+ - **Better scaling** with document size (0.08x ratio for large/huge documents vs 0.12x for small)
12
+ - **Consistent performance** regardless of document complexity
13
+
14
+ ### 💾 Memory Efficiency
15
+ - **Small documents**: 2.33x more memory (acceptable for small sizes)
16
+ - **Medium documents**: 1.06x more memory (negligible difference)
17
+ - **Large documents**: 0.18x memory (5x less memory usage)
18
+ - **Huge documents**: Negative memory delta (more efficient garbage collection)
19
+
20
+ ### 📊 Scaling Analysis
21
+ | Document Size | Regex Time/Char | Char Time/Char | Performance Gain |
22
+ |--------------|----------------|----------------|------------------|
23
+ | Small (2.7KB) | 0.04μs | 0.00μs | **12x faster** |
24
+ | Medium (10.7KB) | 0.04μs | 0.01μs | **8x faster** |
25
+ | Large (53KB) | 0.05μs | 0.00μs | **12x faster** |
26
+ | Huge (274KB) | 0.05μs | 0.00μs | **12x faster** |
27
+
28
+ ## Detailed Performance Metrics
29
+
30
+ ### Time Performance (milliseconds)
31
+ ```
32
+ Size Regex Character Ratio Winner
33
+ ---- ----- --------- ----- ------
34
+ Small 0.11 0.01 0.12x Character ✅
35
+ Medium 0.45 0.06 0.13x Character ✅
36
+ Large 2.77 0.22 0.08x Character ✅
37
+ Huge 13.88 1.17 0.08x Character ✅
38
+ ```
39
+
40
+ ### Memory Usage (bytes)
41
+ ```
42
+ Size Regex Character Ratio Winner
43
+ ---- ----- --------- ----- ------
44
+ Small 237,856 554,128 2.33x Regex ✅
45
+ Medium 691,088 729,768 1.06x Regex ✅
46
+ Large 2,474,168 453,616 0.18x Character ✅
47
+ Huge 5,026,312 -473,480 -0.09x Character ✅
48
+ ```
49
+
50
+ ## Technical Analysis
51
+
52
+ ### Why Character-Based is Faster
53
+
54
+ 1. **No Regex Compilation Overhead**
55
+ - Regex patterns compiled on each iteration
56
+ - Character-based uses direct string operations
57
+
58
+ 2. **Early Termination**
59
+ - Character approach stops at first non-match
60
+ - Regex continues through all patterns
61
+
62
+ 3. **Efficient Lookahead**
63
+ - `hasFollowingAnnotation()` uses simple character scanning
64
+ - Regex requires complex backtracking for annotation detection
65
+
66
+ 4. **Better Cache Locality**
67
+ - Sequential character access
68
+ - Regex engine jumps through pattern matching states
69
+
70
+ ### Memory Usage Patterns
71
+
72
+ 1. **Small Documents**: Character approach allocates more objects but still manageable
73
+ 2. **Large Documents**: Better garbage collection and memory reuse patterns
74
+ 3. **Huge Documents**: Negative memory delta suggests efficient memory management
75
+
76
+ ## Feature Comparison
77
+
78
+ | Feature | Regex-Based | Character-Based | Winner |
79
+ |---------|-------------|----------------|--------|
80
+ | **Semantic Awareness** | ❌ Limited | ✅ Full | Character ✅ |
81
+ | **Maintainability** | ⚠️ Complex | ✅ Simple | Character ✅ |
82
+ | **Debuggability** | ⚠️ Hard | ✅ Easy | Character ✅ |
83
+ | **Performance** | ❌ Slower | ✅ Faster | Character ✅ |
84
+ | **Memory (Large)** | ❌ Higher | ✅ Lower | Character ✅ |
85
+ | **Extensibility** | ⚠️ Complex | ✅ Simple | Character ✅ |
86
+
87
+ ## Recommendations
88
+
89
+ ### 🎯 **Immediate Action: Adopt Character-Based Tokenization**
90
+
91
+ **Confidence Level: HIGH** ⭐⭐⭐⭐⭐
92
+
93
+ **Justification:**
94
+ - 8-12x performance improvement is significant
95
+ - Memory usage is acceptable and better for large documents
96
+ - All other factors (maintainability, features) favor character approach
97
+ - No performance regression detected
98
+
99
+ ### 📋 **Implementation Strategy**
100
+
101
+ #### Phase 1: Shared Tokenization Engine
102
+ ```javascript
103
+ // src/tokenizer.js - New shared module
104
+ export function tokenizeMDLD(text, options = {}) {
105
+ return characterBasedTokenization(text, options);
106
+ }
107
+ ```
108
+
109
+ #### Phase 2: Parser Integration
110
+ ```javascript
111
+ // Replace regex-based extractInlineCarriers
112
+ function extractInlineCarriers(text, baseOffset = 0) {
113
+ return tokenizeMDLD(text, {
114
+ includeAnnotations: true,
115
+ includeMarkdown: true,
116
+ baseOffset: baseOffset
117
+ });
118
+ }
119
+ ```
120
+
121
+ #### Phase 3: Highlighter Unification
122
+ ```javascript
123
+ // Use same tokenizer in highlight.js
124
+ const tokens = tokenizeMDLD(code);
125
+ return renderHighlighted(tokens);
126
+ ```
127
+
128
+ ### ⚡ **Expected Performance Gains**
129
+
130
+ - **Parser Speed**: 8-12x faster tokenization
131
+ - **Highlighting Speed**: Maintain current performance
132
+ - **Memory Usage**: 20-80% reduction for large documents
133
+ - **Development Velocity**: 2-3x faster feature development
134
+
135
+ ### 🔧 **Migration Considerations**
136
+
137
+ 1. **Backward Compatibility**: Maintain existing API surfaces
138
+ 2. **Testing**: Comprehensive regression testing required
139
+ 3. **Documentation**: Update parser internals documentation
140
+ 4. **Performance Monitoring**: Track real-world performance
141
+
142
+ ## Risk Assessment
143
+
144
+ ### 🟢 **Low Risk**
145
+ - Performance regression (character approach is faster)
146
+ - Memory issues (better for large documents)
147
+ - Feature loss (character approach has more features)
148
+
149
+ ### 🟡 **Medium Risk**
150
+ - Implementation complexity (manageable with phased approach)
151
+ - Testing overhead (requires comprehensive test suite)
152
+
153
+ ### 🔴 **High Risk**
154
+ - None identified
155
+
156
+ ## Conclusion
157
+
158
+ The character-based tokenization approach from the highlighter is **superior in every measurable way** except small document memory usage, which is acceptable. The 8-12x performance improvement, combined with better maintainability and enhanced features, makes this a clear architectural win.
159
+
160
+ **Recommendation: Proceed immediately with adopting character-based tokenization across the MDLD codebase.**
161
+
162
+ ---
163
+
164
+ *Benchmark conducted on Node.js v24.7.0 with 50 iterations per test case. Results represent average performance across multiple runs.*
@@ -0,0 +1,32 @@
1
+ [cook] <tag:me@example.com,2026:recipes:>
2
+
3
+ ## Apple Pie {=cook:apple-pie .Recipe label}
4
+
5
+ Description: **Classic homemade apple pie** {comment}
6
+ Cooking time: **45** {cook:cookTime ^^xsd:integer} minutes
7
+ Servings: **8** {cook:recipeYield ^^xsd:integer}
8
+ Difficulty: **Medium** {cook:skill:difficulty}
9
+
10
+ Ingredients:
11
+
12
+ * Apples {+cook:apples ?cook:recipeIngredient .cook:Ingredient label}
13
+ * Sugar {+cook:sugar ?cook:recipeIngredient .cook:Ingredient label}
14
+ * Butter {+cook:butter ?cook:recipeIngredient .cook:Ingredient label}
15
+ * Flour {+cook:flour ?cook:recipeIngredient .cook:Ingredient label}
16
+ * Cinnamon {+cook:cinnamon ?cook:recipeIngredient .cook:Ingredient label}
17
+
18
+ _Ingredient_ {=cook:ingredient .Class label} - is a class for all ingredients in our **Cookbook** {!member +urn:collection:cookbook .Container label }.
19
+
20
+ Equipment:
21
+
22
+ * Oven {+cook:oven-1 ?cook:requiresEquipment .cook:Device}
23
+ * Mixing bowl {+cook:bowl-1 ?cook:requiresEquipment .cook:Utensil}
24
+ * Rolling pin {+cook:pin-1 ?cook:requiresEquipment .cook:Utensil}
25
+
26
+ Calories: [1235] {cook:calories ^^xsd:integer} calories
27
+ Sugar: [40] {cook:sugarGrams ^^xsd:decimal} grams
28
+
29
+ Related recipes:
30
+
31
+ * Apple Crumble {+cook:apple-crumble !cook:variationOf .cook:Recipe}
32
+ * Peach Pie {+cook:peach-pie !cook:variationOf .cook:Recipe}
@@ -0,0 +1,33 @@
1
+ [ex] <http://example.org/>
2
+
3
+ # Status {=ex:Status .Class label}
4
+
5
+ A controlled vocabulary.
6
+
7
+ ---
8
+
9
+ # Status Shape {=ex:StatusShape .sh:NodeShape label}
10
+
11
+ Target: [Status] {+ex:Status ?sh:targetClass}
12
+
13
+ ## Status Property {=ex:StatusShape-status .sh:PropertyShape ?sh:property}
14
+
15
+ Path: [status] {+ex:status ?sh:path}
16
+
17
+ Required: [1] {sh:minCount ^^xsd:integer}
18
+
19
+ ---
20
+
21
+ # Test Data
22
+
23
+ ## Valid Record {=ex:Record-Valid .ex:Status}
24
+
25
+ This record should pass validation.
26
+
27
+ ### Status {+ex:Active ?ex:status}
28
+
29
+ ---
30
+
31
+ ## Invalid Record {=ex:Record-Invalid .ex:Status}
32
+
33
+ This record should fail validation - missing required status.
@@ -0,0 +1,35 @@
1
+ # MD-LD Demo Examples
2
+
3
+ Quick start examples demonstrating core MD-LD patterns and features.
4
+
5
+ ## Demo Examples
6
+
7
+ ### 🎯 Getting Started
8
+
9
+ **[minimal.md](./minimal.md)** Minimal document with subject and author
10
+ **[journal.md](./journal.md)** Personal journal with activities, reflections, and related events
11
+
12
+ ### 🍳 Everyday Examples
13
+
14
+ **[recipe.md](./recipe.md)** Complete recipe with ingredients, equipment, and nutritional information
15
+ **[project.md](./project.md)** Weekend project with tasks, resources, team, and blocking issues
16
+
17
+ ### 🚀 Advanced Examples
18
+
19
+ **[space-mission.md](./space-mission.md)** Apollo 11 mission with crew, equipment, and objectives
20
+ **[research.md](./research.md)** Scientific paper with datasets, methodology, and organizations
21
+
22
+ ## Key Features Demonstrated
23
+
24
+ ✅ **Explicit Annotations** All semantic data comes from `{...}` blocks
25
+ ✅ **Subject Chaining** Using `{=IRI}` and `{+IRI}` for context
26
+ ✅ **List Items** Explicit annotations per list item (no semantic propagation)
27
+ ✅ **Nested Structures** Hierarchical data organization
28
+ ✅ **PROV-O Integration** Complete provenance tracking
29
+ ✅ **Fragment Subjects** Document structuring with `{=#fragment}`
30
+ ✅ **Reverse Relations** Using `!p` for inverse relationships
31
+ ✅ **Datatypes & Languages** `^^xsd:types` and `@lang` tags
32
+
33
+ ---
34
+
35
+ *Each demo is a complete, working MD-LD document that can be parsed and round-tripped safely.*