mdld-parse 0.9.2 → 0.9.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/.nojekyll +0 -0
- package/dist/CNAME +1 -0
- package/dist/README.md +425 -0
- package/dist/docs/API.md +397 -0
- package/dist/docs/Architecture.md +190 -0
- package/dist/docs/Guide.md +322 -0
- package/dist/docs/LLM-MDLD-Inducer.md +550 -0
- package/dist/docs/Parser.md +654 -0
- package/dist/docs/Performance.md +238 -0
- package/dist/docs/Spec-render.md +1439 -0
- package/dist/docs/Subject.md +180 -0
- package/dist/docs/Syntax.md +463 -0
- package/dist/docs/TodoList.md +215 -0
- package/dist/docs/Use-Cases.md +354 -0
- package/dist/docs/index.html +108 -0
- package/dist/docs/index.md +56 -0
- package/dist/docs/origin.md +459 -0
- package/dist/docs/polarity.md +586 -0
- package/dist/docs/primary-subject.md +199 -0
- package/dist/docs/statements.md +226 -0
- package/dist/docs/tokenization-performance-report.md +164 -0
- package/dist/examples/Cookbook.md +32 -0
- package/dist/examples/Status-SHACL.md +33 -0
- package/dist/examples/demo/index.md +35 -0
- package/dist/examples/demo/journal.md +18 -0
- package/dist/examples/demo/minimal.md +11 -0
- package/dist/examples/demo/project.md +15 -0
- package/dist/examples/demo/recipe.md +31 -0
- package/dist/examples/demo/research.md +46 -0
- package/dist/examples/demo/space-mission.md +48 -0
- package/dist/examples/dogfood.md +288 -0
- package/dist/examples/few-shot.md +309 -0
- package/dist/examples/graph/rdf-graph.html +473 -0
- package/dist/examples/graph/rdf_force_graph_web_component.html +506 -0
- package/dist/examples/index.md +45 -0
- package/dist/examples/llm-time-workflow.md +173 -0
- package/dist/examples/prov-o.md +2367 -0
- package/dist/examples/rdf.md +827 -0
- package/dist/examples/shacl.md +3399 -0
- package/dist/examples/statements.md +43 -0
- package/dist/examples/workflow.md +150 -0
- package/dist/index.html +2827 -0
- package/dist/index.js +1548 -0
- package/dist/logo.svg +7 -0
- package/dist/spec/Spec-compact.md +363 -0
- package/dist/spec/Spec-ultra.md +221 -0
- package/dist/spec/Spec.md +1068 -0
- package/dist/spec/index.md +46 -0
- package/package.json +7 -8
- package/src/constants.js +0 -31
- package/src/generate.js +0 -245
- package/src/highlight.js +0 -362
- package/src/index.js +0 -13
- package/src/locate.js +0 -23
- package/src/merge.js +0 -157
- package/src/parse.js +0 -711
- package/src/render.js +0 -576
- package/src/shared.js +0 -577
- package/src/tokenizers.js +0 -594
- package/src/utils.js +0 -430
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# Primary Subject Example
|
|
2
|
+
|
|
3
|
+
This example demonstrates the Primary Subject feature in MD-LD, which identifies the main entity described by a document.
|
|
4
|
+
|
|
5
|
+
## What is a Primary Subject?
|
|
6
|
+
|
|
7
|
+
The Primary Subject is the first non-fragment subject declaration (`{=IRI}`) in a document. It provides a deterministic way to identify the central focus of a document.
|
|
8
|
+
|
|
9
|
+
## Example Document
|
|
10
|
+
|
|
11
|
+
```markdown
|
|
12
|
+
[ex] <http://example.org/>
|
|
13
|
+
|
|
14
|
+
# Article {=ex:article .schema:Article label}
|
|
15
|
+
[Understanding Primary Subjects] {schema:headline}
|
|
16
|
+
|
|
17
|
+
## Author {=ex:author .schema:Person}
|
|
18
|
+
[Alice Smith] {schema:name}
|
|
19
|
+
[alice@example.com] {schema:email}
|
|
20
|
+
|
|
21
|
+
## Abstract
|
|
22
|
+
This article explains the Primary Subject concept in MD-LD, which allows applications to quickly identify the main entity described by a document.
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Parsing with Primary Subject
|
|
26
|
+
|
|
27
|
+
```javascript
|
|
28
|
+
import { parse } from 'mdld-parse';
|
|
29
|
+
|
|
30
|
+
const md = `[ex] <http://example.org/>
|
|
31
|
+
[schema] <http://schema.org/>
|
|
32
|
+
|
|
33
|
+
# Article {=ex:article .schema:Article label}
|
|
34
|
+
[Understanding Primary Subjects] {schema:headline}`;
|
|
35
|
+
|
|
36
|
+
const result = parse({ text: md });
|
|
37
|
+
|
|
38
|
+
console.log(result.primarySubject);
|
|
39
|
+
// Output: http://example.org/article
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Key Behaviors
|
|
43
|
+
|
|
44
|
+
### 1. First Non-Fragment Subject Wins
|
|
45
|
+
|
|
46
|
+
```markdown
|
|
47
|
+
{=#summary} // Fragment - NOT primary
|
|
48
|
+
[Summary] {label}
|
|
49
|
+
|
|
50
|
+
# Document {=ex:doc} // First non-fragment - PRIMARY
|
|
51
|
+
[Alice] {author}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Primary subject: `ex:doc`
|
|
55
|
+
|
|
56
|
+
### 2. Reset Does Not Clear Primary Subject
|
|
57
|
+
|
|
58
|
+
```markdown
|
|
59
|
+
# First {=ex:first}
|
|
60
|
+
[Value] {label}
|
|
61
|
+
|
|
62
|
+
# Reset {=} // Does not clear primary subject
|
|
63
|
+
|
|
64
|
+
# Second {=ex:second}
|
|
65
|
+
[Value] {label}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Primary subject: `ex:first` (fixed once detected)
|
|
69
|
+
|
|
70
|
+
### 3. No Subject Means No Primary
|
|
71
|
+
|
|
72
|
+
```markdown
|
|
73
|
+
[Just text] {label}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Primary subject: `null`
|
|
77
|
+
|
|
78
|
+
## Merge with Primary Subjects
|
|
79
|
+
|
|
80
|
+
When merging multiple documents, the `merge()` function returns an array of all primary subjects in merge order:
|
|
81
|
+
|
|
82
|
+
```javascript
|
|
83
|
+
import { merge } from 'mdld-parse';
|
|
84
|
+
|
|
85
|
+
const doc1 = `[ex] <http://example.org/>
|
|
86
|
+
# Document 1 {=ex:doc1}
|
|
87
|
+
[Content] {label}`;
|
|
88
|
+
|
|
89
|
+
const doc2 = `[ex] <http://example.org/>
|
|
90
|
+
# Document 2 {=ex:doc2}
|
|
91
|
+
[Content] {label}`;
|
|
92
|
+
|
|
93
|
+
const result = merge([doc1, doc2], { context: { ex: 'http://example.org/' } });
|
|
94
|
+
|
|
95
|
+
console.log(result.primarySubjects.map(s => s.value));
|
|
96
|
+
// Output: ['http://example.org/doc1', 'http://example.org/doc2']
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Use Cases
|
|
100
|
+
|
|
101
|
+
- **Document Identification** - Quickly determine what a document is about
|
|
102
|
+
- **Merge Tracking** - Track which entities are being merged across documents
|
|
103
|
+
- **UI Navigation** - Provide a default focus point for document viewers
|
|
104
|
+
- **Query Optimization** - Use primary subject as default query context
|
|
105
|
+
|
|
106
|
+
## RDF Philosophy Alignment
|
|
107
|
+
|
|
108
|
+
The Primary Subject feature aligns with core W3C/RDF principles:
|
|
109
|
+
|
|
110
|
+
### Document Identity
|
|
111
|
+
RDF 1.1 Named Graphs recognize the importance of document-centric RDF. Primary Subject makes document identity explicit without forcing graph-centric assumptions, supporting the document-centric approach (FOAF profiles, RSS feeds) that's part of the Linked Data ecosystem.
|
|
112
|
+
|
|
113
|
+
### Provenance Tracking (PROV-O)
|
|
114
|
+
The W3C PROV-O standard emphasizes tracking data origin. Primary Subject provides a deterministic way to identify what a document is "about," which is foundational for provenance chains and attribution.
|
|
115
|
+
|
|
116
|
+
### Dereferenceability
|
|
117
|
+
Linked Data principles emphasize that IRIs should be dereferenceable. Having a clear primary subject makes it immediately clear what IRI to dereference when encountering a document, supporting the "follow your nose" navigation pattern.
|
|
118
|
+
|
|
119
|
+
### Minimal Commitment
|
|
120
|
+
RDF's philosophy is about making minimal assertions. Primary Subject is lightweight metadata - it doesn't add triples to the graph, just provides a pointer to the main subject. This aligns with MDLD's "no implicit semantics" principle.
|
|
121
|
+
|
|
122
|
+
### Explicit over Implicit
|
|
123
|
+
The W3C encourages explicit semantics over inference. Primary Subject makes document focus explicit rather than requiring heuristic analysis of the graph to determine "what this document is about."
|
|
124
|
+
|
|
125
|
+
## Grounded Documents: Long-Term Benefits
|
|
126
|
+
|
|
127
|
+
Primary Subject enables a new class of semantic applications that treat documents as meaningful units in knowledge workflows:
|
|
128
|
+
|
|
129
|
+
### Document-Centric Knowledge Graphs
|
|
130
|
+
- Citation networks with clear document identity
|
|
131
|
+
- Document versioning and evolution tracking
|
|
132
|
+
- Cross-document referencing and linking
|
|
133
|
+
- Entity-focused provenance chains
|
|
134
|
+
|
|
135
|
+
### Semantic Search and Discovery
|
|
136
|
+
- Index documents by their main entity
|
|
137
|
+
- Support queries like "find all documents about X" without complex graph pattern matching
|
|
138
|
+
- Entity-based relevance scoring
|
|
139
|
+
|
|
140
|
+
### Incremental Graph Construction
|
|
141
|
+
- Determine which documents describe the same entity (deduplication)
|
|
142
|
+
- Smart merge order and priority decisions
|
|
143
|
+
- Conflict resolution strategies based on entity overlap
|
|
144
|
+
- Graph topology construction guidance
|
|
145
|
+
|
|
146
|
+
### Collaborative Semantic Editing
|
|
147
|
+
- Track which entities are being modified across authors
|
|
148
|
+
- Coordinate concurrent edits at entity level
|
|
149
|
+
- Detect merge conflicts at entity granularity
|
|
150
|
+
- Provide contextual editing assistance
|
|
151
|
+
|
|
152
|
+
### Semantic Version Control
|
|
153
|
+
- Entity-level version tracking within document systems
|
|
154
|
+
- Queries like "show me how entity X has changed across versions"
|
|
155
|
+
- Entity-focused diff and merge operations
|
|
156
|
+
- Granular change attribution
|
|
157
|
+
|
|
158
|
+
### Knowledge Graph Federation
|
|
159
|
+
- Identify overlapping entities across different sources
|
|
160
|
+
- Determine authoritative sources for specific entities
|
|
161
|
+
- Build confidence scores based on document provenance
|
|
162
|
+
- Source-aware query planning
|
|
163
|
+
|
|
164
|
+
### AI/LLM Integration
|
|
165
|
+
- Primary subject provides clear context for document understanding
|
|
166
|
+
- Entity-focused prompt engineering
|
|
167
|
+
- Document retrieval by entity relevance
|
|
168
|
+
- Improved fact-checking and attribution
|
|
169
|
+
|
|
170
|
+
The Primary Subject feature bridges document-centric and graph-centric approaches, enabling semantic applications that treat documents as first-class citizens in knowledge workflows while respecting RDF's core principles of explicit semantics and minimal commitment.
|
|
171
|
+
|
|
172
|
+
## Complete Example
|
|
173
|
+
|
|
174
|
+
```javascript
|
|
175
|
+
import { parse, merge } from 'mdld-parse';
|
|
176
|
+
|
|
177
|
+
// Single document
|
|
178
|
+
const doc1 = `[blog] <https://example.com/blog/>
|
|
179
|
+
# My First Post {=blog:post1 .schema:BlogPosting label}
|
|
180
|
+
[Hello World] {schema:headline}`;
|
|
181
|
+
|
|
182
|
+
const result1 = parse({ text: doc1, context: { blog: 'https://example.com/blog/', schema: 'http://schema.org/' } });
|
|
183
|
+
console.log('Primary subject:', result1.primarySubject);
|
|
184
|
+
// Output: https://example.com/blog/post1
|
|
185
|
+
|
|
186
|
+
// Multiple documents
|
|
187
|
+
const doc2 = `[blog] <https://example.com/blog/>
|
|
188
|
+
# My Second Post {=blog:post2 .schema:BlogPosting label}
|
|
189
|
+
[Hello Again] {schema:headline}`;
|
|
190
|
+
|
|
191
|
+
const merged = merge([doc1, doc2], { context: { blog: 'https://example.com/blog/', schema: 'http://schema.org/' } });
|
|
192
|
+
|
|
193
|
+
console.log('Primary subjects:', merged.primarySubjects);
|
|
194
|
+
// Output: ['https://example.com/blog/post1', 'https://example.com/blog/post2']
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
## Specification
|
|
198
|
+
|
|
199
|
+
For complete details on Primary Subject semantics, see the [MD-LD Specification](../spec/Spec.md#17-primary-subject).
|
|
@@ -0,0 +1,226 @@
|
|
|
1
|
+
# Elevated Statements
|
|
2
|
+
|
|
3
|
+
MD-LD provides **elevated statements** through automatic `rdf:Statement` pattern detection, enabling authors to create a "golden graph" of intentionally elevated statements while maintaining complete provenance in the underlying data.
|
|
4
|
+
|
|
5
|
+
## 🎯 Concept Overview
|
|
6
|
+
|
|
7
|
+
Elevated statements allow you to:
|
|
8
|
+
- **Separate signal from noise** - Dashboard apps can use only the elevated statements
|
|
9
|
+
- **Maintain provenance** - Complete context remains available in the full graph
|
|
10
|
+
- **Single-pass parsing** - No performance overhead, detected during normal parsing
|
|
11
|
+
- **Flexible patterns** - Support both IRI references and direct literals
|
|
12
|
+
|
|
13
|
+
## 📋 Pattern Requirements
|
|
14
|
+
|
|
15
|
+
For a statement to be elevated, it must have:
|
|
16
|
+
1. **rdf:type rdf:Statement** - The subject must be typed as an RDF Statement
|
|
17
|
+
2. **rdf:subject** - The subject of the elevated SPO triple
|
|
18
|
+
3. **rdf:predicate** - The predicate of the elevated SPO triple
|
|
19
|
+
4. **rdf:object** - The object of the elevated SPO triple
|
|
20
|
+
|
|
21
|
+
When all four parts are present, the parser automatically creates an elevated SPO quad in the `statements` array.
|
|
22
|
+
|
|
23
|
+
## 🔧 Syntax Patterns
|
|
24
|
+
|
|
25
|
+
### Basic Pattern with IRIs
|
|
26
|
+
|
|
27
|
+
```markdown
|
|
28
|
+
[ex] <http://example.org/>
|
|
29
|
+
|
|
30
|
+
## Elevated statement {=ex:stmt1 .rdf:Statement}
|
|
31
|
+
**Alice** {+ex:alice ?rdf:subject} *knows* {+ex:knows ?rdf:predicate} **Bob** {+ex:bob ?rdf:object}
|
|
32
|
+
|
|
33
|
+
**Alice** {=ex:alice} knows **Bob** {?ex:knows +ex:bob} - direct statement
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### Pattern with Direct Literals
|
|
37
|
+
|
|
38
|
+
For datatypes and language tags, use direct literals:
|
|
39
|
+
|
|
40
|
+
```markdown
|
|
41
|
+
[ex] <http://example.org/>
|
|
42
|
+
|
|
43
|
+
## Statement with datatype {=ex:stmt1 .rdf:Statement}
|
|
44
|
+
**Alice** {+ex:alice ?rdf:subject} *has age* {+ex:hasAge ?rdf:predicate} **25** {rdf:object ^^xsd:integer}
|
|
45
|
+
|
|
46
|
+
## Statement with language {=ex:stmt2 .rdf:Statement}
|
|
47
|
+
**Alice** {+ex:alice ?rdf:subject} *has name* {+ex:hasName ?rdf:predicate} **Alice** {rdf:object @en}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
### Real-World Example
|
|
51
|
+
|
|
52
|
+
```markdown
|
|
53
|
+
[my] <tag:alice@example.com,2026:>
|
|
54
|
+
[foaf] <http://xmlns.com/foaf/0.1/>
|
|
55
|
+
|
|
56
|
+
My name is **Alice** {=my:Alice .foaf:Person foaf:name}.
|
|
57
|
+
|
|
58
|
+
{=}
|
|
59
|
+
|
|
60
|
+
Today I have learned that my colleague name is [Clair] {=my:Claire .foaf:Person foaf:name}
|
|
61
|
+
|
|
62
|
+
## I know Claire's name {=my:claire-name .rdf:Statement}
|
|
63
|
+
|
|
64
|
+
[My colleague] {+my:Claire ?rdf:subject} [name] {+foaf:name ?rdf:predicate} is [Clair] {rdf:object}.
|
|
65
|
+
|
|
66
|
+
{=}
|
|
67
|
+
|
|
68
|
+
## We've talked! {=my:claire-first-talk .rdf:Statement}
|
|
69
|
+
|
|
70
|
+
Today [I] {+my:Alice ?rdf:subject} came to office a bit earlier and [talked for some time] {+foaf:knows ?rdf:predicate} with [Claire] {+my:Claire ?rdf:object}.
|
|
71
|
+
|
|
72
|
+
Now **I** {=my:Alice} know **Claire** {+my:Claire ?foaf:knows}.
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## 🚀 Usage in Applications
|
|
76
|
+
|
|
77
|
+
### API Access
|
|
78
|
+
|
|
79
|
+
```javascript
|
|
80
|
+
import { parse } from 'mdld-parse';
|
|
81
|
+
|
|
82
|
+
const result = parse({ text: `# Test {=tag:me@example.org,2026:test .prov:Entity label}`});
|
|
83
|
+
console.log(result.quads); // All quads including provenance
|
|
84
|
+
console.log(result.statements); // Elevated SPO quads (detected during parsing)
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
### Dashboard Applications
|
|
88
|
+
|
|
89
|
+
Dashboard applications can:
|
|
90
|
+
|
|
91
|
+
- **Ignore provenance-rich data** by using only `result.statements`
|
|
92
|
+
- **Access the "golden graph"** of intentionally elevated statements
|
|
93
|
+
- **Maintain complete provenance** in `result.quads` when needed
|
|
94
|
+
- **Filter by context** using the origin tracking system
|
|
95
|
+
|
|
96
|
+
### Example: Simple Graph Display
|
|
97
|
+
|
|
98
|
+
```javascript
|
|
99
|
+
import { parse } from 'mdld-parse';
|
|
100
|
+
|
|
101
|
+
function displayElevatedGraph(markdown) {
|
|
102
|
+
const result = parse({text: markdown});
|
|
103
|
+
|
|
104
|
+
// Display only elevated statements
|
|
105
|
+
result.statements.forEach(quad => {
|
|
106
|
+
console.log(`${quad.subject.value} → ${quad.predicate.value} → ${quad.object.value}`);
|
|
107
|
+
});
|
|
108
|
+
|
|
109
|
+
return result.statements.length; // Number of elevated statements
|
|
110
|
+
}
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
## 📊 Data Flow
|
|
114
|
+
|
|
115
|
+
```
|
|
116
|
+
MD-LD Document
|
|
117
|
+
↓ (single-pass parsing)
|
|
118
|
+
All Quads + Elevated Statements
|
|
119
|
+
↓
|
|
120
|
+
┌─────────────────┬─────────────────┐
|
|
121
|
+
│ result.quads │result.statements│
|
|
122
|
+
│ (provenance) │ (golden graph) │
|
|
123
|
+
└─────────────────┴─────────────────┘
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
## 🎨 Best Practices
|
|
127
|
+
|
|
128
|
+
### 1. Use Descriptive Statement IDs
|
|
129
|
+
|
|
130
|
+
```markdown
|
|
131
|
+
## Alice knows Bob {=ex:alice-knows-bob .rdf:Statement}
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### 2. Preserve Datatypes and Languages
|
|
135
|
+
|
|
136
|
+
```markdown
|
|
137
|
+
## Person's birth date {=ex:birth-date .rdf:Statement}
|
|
138
|
+
**Alice** {+ex:alice ?rdf:subject} *born on* {+ex:bornOn ?rdf:predicate} **1990-01-01** {rdf:object ^^xsd:date}
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### 3. Mix Elevated and Regular Statements
|
|
142
|
+
|
|
143
|
+
```markdown
|
|
144
|
+
## Elevated: Important fact {=ex:important .rdf:Statement}
|
|
145
|
+
**Alice** {+ex:alice ?rdf:subject} *knows* {+ex:knows ?rdf:predicate} **Bob** {+ex:bob ?rdf:object}
|
|
146
|
+
|
|
147
|
+
**Alice** {=ex:alice} also knows **Charlie** {?ex:knows +ex:charlie} - regular statement
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### 4. Use Subject Resets for Organization
|
|
151
|
+
|
|
152
|
+
```markdown
|
|
153
|
+
{=}
|
|
154
|
+
|
|
155
|
+
## New section {=ex:section-stmt .rdf:Statement}
|
|
156
|
+
**Fact** {+ex:fact ?rdf:subject} *has value* {+ex:hasValue ?rdf:predicate} **42** {rdf:object ^^xsd:integer}
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
## 🔍 Technical Details
|
|
160
|
+
|
|
161
|
+
### Single-Pass Detection
|
|
162
|
+
|
|
163
|
+
Elevated statements are detected during the normal parsing process:
|
|
164
|
+
|
|
165
|
+
1. **Quad emission** - As each quad is created, the pattern detector runs
|
|
166
|
+
2. **Candidate tracking** - Incomplete patterns are tracked in memory
|
|
167
|
+
3. **Pattern completion** - When all parts are present, elevated SPO is created
|
|
168
|
+
4. **Cleanup** - Completed patterns are removed from tracking
|
|
169
|
+
|
|
170
|
+
### Memory Efficiency
|
|
171
|
+
|
|
172
|
+
- **O(1) additional memory** - Only tracks incomplete patterns
|
|
173
|
+
- **Automatic cleanup** - Completed patterns are removed immediately
|
|
174
|
+
- **No backtracking** - True single-pass processing
|
|
175
|
+
|
|
176
|
+
### Compatibility
|
|
177
|
+
|
|
178
|
+
- **RDF/JS compliant** - Elevated statements are standard RDF quads
|
|
179
|
+
- **Toolchain compatible** - Works with n3.js, rdflib, and other RDF libraries
|
|
180
|
+
- **Streaming friendly** - No additional passes required
|
|
181
|
+
|
|
182
|
+
## 📝 Comparison with Alternatives
|
|
183
|
+
|
|
184
|
+
| Approach | Provenance | Performance | Flexibility | Complexity |
|
|
185
|
+
|----------|------------|-------------|-------------|------------|
|
|
186
|
+
| **Elevated Statements** | ✅ Complete | ✅ Single-pass | ✅ High | 🟡 Medium |
|
|
187
|
+
| **Named Graphs** | ✅ Complete | ⚡ Multiple passes | 🟡 Medium | 🔴 High |
|
|
188
|
+
| **Manual Filtering** | ✅ Complete | ✅ Single-pass | 🔴 Low | 🟢 Low |
|
|
189
|
+
| **Separate Documents** | ❌ Lost | ✅ Single-pass | 🔴 Low | 🟢 Low |
|
|
190
|
+
|
|
191
|
+
## 🎯 Use Cases
|
|
192
|
+
|
|
193
|
+
### Knowledge Management
|
|
194
|
+
- Elevate important facts while preserving context
|
|
195
|
+
- Create "golden graph" for search and discovery
|
|
196
|
+
- Maintain full provenance for audit trails
|
|
197
|
+
|
|
198
|
+
### API Design
|
|
199
|
+
- Provide simplified endpoints with elevated statements only
|
|
200
|
+
- Offer detailed endpoints with full provenance
|
|
201
|
+
- Enable progressive disclosure of information
|
|
202
|
+
|
|
203
|
+
### Content Management
|
|
204
|
+
- Separate published content from editorial metadata
|
|
205
|
+
- Create curated views while preserving edits
|
|
206
|
+
- Support content approval workflows
|
|
207
|
+
|
|
208
|
+
### Data Integration
|
|
209
|
+
- Elevate validated mappings while preserving transformation rules
|
|
210
|
+
- Create clean integration graphs with full audit trails
|
|
211
|
+
- Support data lineage and governance
|
|
212
|
+
|
|
213
|
+
## 🧪 Testing
|
|
214
|
+
|
|
215
|
+
The elevated statements feature is covered by comprehensive tests:
|
|
216
|
+
|
|
217
|
+
```bash
|
|
218
|
+
npm test # All 105 tests pass, including elevated statements
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
Test coverage includes:
|
|
222
|
+
- Basic pattern detection
|
|
223
|
+
- Multiple elevated statements
|
|
224
|
+
- Incomplete patterns (ignored)
|
|
225
|
+
- Mixed content scenarios
|
|
226
|
+
- Datatype and language preservation
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# MDLD Tokenization Performance Analysis Report
|
|
2
|
+
|
|
3
|
+
## Executive Summary
|
|
4
|
+
|
|
5
|
+
**Character-based tokenization significantly outperforms regex-based approach** across all document sizes, with 8-12x faster processing and competitive memory usage. This validates our architectural decision to adopt the highlighting approach for the parser.
|
|
6
|
+
|
|
7
|
+
## Key Findings
|
|
8
|
+
|
|
9
|
+
### 🚀 Performance Superiority
|
|
10
|
+
- **8-12x faster** than regex-based approach across all document sizes
|
|
11
|
+
- **Better scaling** with document size (0.08x ratio for large/huge documents vs 0.12x for small)
|
|
12
|
+
- **Consistent performance** regardless of document complexity
|
|
13
|
+
|
|
14
|
+
### 💾 Memory Efficiency
|
|
15
|
+
- **Small documents**: 2.33x more memory (acceptable for small sizes)
|
|
16
|
+
- **Medium documents**: 1.06x more memory (negligible difference)
|
|
17
|
+
- **Large documents**: 0.18x memory (5x less memory usage)
|
|
18
|
+
- **Huge documents**: Negative memory delta (more efficient garbage collection)
|
|
19
|
+
|
|
20
|
+
### 📊 Scaling Analysis
|
|
21
|
+
| Document Size | Regex Time/Char | Char Time/Char | Performance Gain |
|
|
22
|
+
|--------------|----------------|----------------|------------------|
|
|
23
|
+
| Small (2.7KB) | 0.04μs | 0.00μs | **12x faster** |
|
|
24
|
+
| Medium (10.7KB) | 0.04μs | 0.01μs | **8x faster** |
|
|
25
|
+
| Large (53KB) | 0.05μs | 0.00μs | **12x faster** |
|
|
26
|
+
| Huge (274KB) | 0.05μs | 0.00μs | **12x faster** |
|
|
27
|
+
|
|
28
|
+
## Detailed Performance Metrics
|
|
29
|
+
|
|
30
|
+
### Time Performance (milliseconds)
|
|
31
|
+
```
|
|
32
|
+
Size Regex Character Ratio Winner
|
|
33
|
+
---- ----- --------- ----- ------
|
|
34
|
+
Small 0.11 0.01 0.12x Character ✅
|
|
35
|
+
Medium 0.45 0.06 0.13x Character ✅
|
|
36
|
+
Large 2.77 0.22 0.08x Character ✅
|
|
37
|
+
Huge 13.88 1.17 0.08x Character ✅
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Memory Usage (bytes)
|
|
41
|
+
```
|
|
42
|
+
Size Regex Character Ratio Winner
|
|
43
|
+
---- ----- --------- ----- ------
|
|
44
|
+
Small 237,856 554,128 2.33x Regex ✅
|
|
45
|
+
Medium 691,088 729,768 1.06x Regex ✅
|
|
46
|
+
Large 2,474,168 453,616 0.18x Character ✅
|
|
47
|
+
Huge 5,026,312 -473,480 -0.09x Character ✅
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Technical Analysis
|
|
51
|
+
|
|
52
|
+
### Why Character-Based is Faster
|
|
53
|
+
|
|
54
|
+
1. **No Regex Compilation Overhead**
|
|
55
|
+
- Regex patterns compiled on each iteration
|
|
56
|
+
- Character-based uses direct string operations
|
|
57
|
+
|
|
58
|
+
2. **Early Termination**
|
|
59
|
+
- Character approach stops at first non-match
|
|
60
|
+
- Regex continues through all patterns
|
|
61
|
+
|
|
62
|
+
3. **Efficient Lookahead**
|
|
63
|
+
- `hasFollowingAnnotation()` uses simple character scanning
|
|
64
|
+
- Regex requires complex backtracking for annotation detection
|
|
65
|
+
|
|
66
|
+
4. **Better Cache Locality**
|
|
67
|
+
- Sequential character access
|
|
68
|
+
- Regex engine jumps through pattern matching states
|
|
69
|
+
|
|
70
|
+
### Memory Usage Patterns
|
|
71
|
+
|
|
72
|
+
1. **Small Documents**: Character approach allocates more objects but still manageable
|
|
73
|
+
2. **Large Documents**: Better garbage collection and memory reuse patterns
|
|
74
|
+
3. **Huge Documents**: Negative memory delta suggests efficient memory management
|
|
75
|
+
|
|
76
|
+
## Feature Comparison
|
|
77
|
+
|
|
78
|
+
| Feature | Regex-Based | Character-Based | Winner |
|
|
79
|
+
|---------|-------------|----------------|--------|
|
|
80
|
+
| **Semantic Awareness** | ❌ Limited | ✅ Full | Character ✅ |
|
|
81
|
+
| **Maintainability** | ⚠️ Complex | ✅ Simple | Character ✅ |
|
|
82
|
+
| **Debuggability** | ⚠️ Hard | ✅ Easy | Character ✅ |
|
|
83
|
+
| **Performance** | ❌ Slower | ✅ Faster | Character ✅ |
|
|
84
|
+
| **Memory (Large)** | ❌ Higher | ✅ Lower | Character ✅ |
|
|
85
|
+
| **Extensibility** | ⚠️ Complex | ✅ Simple | Character ✅ |
|
|
86
|
+
|
|
87
|
+
## Recommendations
|
|
88
|
+
|
|
89
|
+
### 🎯 **Immediate Action: Adopt Character-Based Tokenization**
|
|
90
|
+
|
|
91
|
+
**Confidence Level: HIGH** ⭐⭐⭐⭐⭐
|
|
92
|
+
|
|
93
|
+
**Justification:**
|
|
94
|
+
- 8-12x performance improvement is significant
|
|
95
|
+
- Memory usage is acceptable and better for large documents
|
|
96
|
+
- All other factors (maintainability, features) favor character approach
|
|
97
|
+
- No performance regression detected
|
|
98
|
+
|
|
99
|
+
### 📋 **Implementation Strategy**
|
|
100
|
+
|
|
101
|
+
#### Phase 1: Shared Tokenization Engine
|
|
102
|
+
```javascript
|
|
103
|
+
// src/tokenizer.js - New shared module
|
|
104
|
+
export function tokenizeMDLD(text, options = {}) {
|
|
105
|
+
return characterBasedTokenization(text, options);
|
|
106
|
+
}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
#### Phase 2: Parser Integration
|
|
110
|
+
```javascript
|
|
111
|
+
// Replace regex-based extractInlineCarriers
|
|
112
|
+
function extractInlineCarriers(text, baseOffset = 0) {
|
|
113
|
+
return tokenizeMDLD(text, {
|
|
114
|
+
includeAnnotations: true,
|
|
115
|
+
includeMarkdown: true,
|
|
116
|
+
baseOffset: baseOffset
|
|
117
|
+
});
|
|
118
|
+
}
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
#### Phase 3: Highlighter Unification
|
|
122
|
+
```javascript
|
|
123
|
+
// Use same tokenizer in highlight.js
|
|
124
|
+
const tokens = tokenizeMDLD(code);
|
|
125
|
+
return renderHighlighted(tokens);
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
### ⚡ **Expected Performance Gains**
|
|
129
|
+
|
|
130
|
+
- **Parser Speed**: 8-12x faster tokenization
|
|
131
|
+
- **Highlighting Speed**: Maintain current performance
|
|
132
|
+
- **Memory Usage**: 20-80% reduction for large documents
|
|
133
|
+
- **Development Velocity**: 2-3x faster feature development
|
|
134
|
+
|
|
135
|
+
### 🔧 **Migration Considerations**
|
|
136
|
+
|
|
137
|
+
1. **Backward Compatibility**: Maintain existing API surfaces
|
|
138
|
+
2. **Testing**: Comprehensive regression testing required
|
|
139
|
+
3. **Documentation**: Update parser internals documentation
|
|
140
|
+
4. **Performance Monitoring**: Track real-world performance
|
|
141
|
+
|
|
142
|
+
## Risk Assessment
|
|
143
|
+
|
|
144
|
+
### 🟢 **Low Risk**
|
|
145
|
+
- Performance regression (character approach is faster)
|
|
146
|
+
- Memory issues (better for large documents)
|
|
147
|
+
- Feature loss (character approach has more features)
|
|
148
|
+
|
|
149
|
+
### 🟡 **Medium Risk**
|
|
150
|
+
- Implementation complexity (manageable with phased approach)
|
|
151
|
+
- Testing overhead (requires comprehensive test suite)
|
|
152
|
+
|
|
153
|
+
### 🔴 **High Risk**
|
|
154
|
+
- None identified
|
|
155
|
+
|
|
156
|
+
## Conclusion
|
|
157
|
+
|
|
158
|
+
The character-based tokenization approach from the highlighter is **superior in every measurable way** except small document memory usage, which is acceptable. The 8-12x performance improvement, combined with better maintainability and enhanced features, makes this a clear architectural win.
|
|
159
|
+
|
|
160
|
+
**Recommendation: Proceed immediately with adopting character-based tokenization across the MDLD codebase.**
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
164
|
+
*Benchmark conducted on Node.js v24.7.0 with 50 iterations per test case. Results represent average performance across multiple runs.*
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
[cook] <tag:me@example.com,2026:recipes:>
|
|
2
|
+
|
|
3
|
+
## Apple Pie {=cook:apple-pie .Recipe label}
|
|
4
|
+
|
|
5
|
+
Description: **Classic homemade apple pie** {comment}
|
|
6
|
+
Cooking time: **45** {cook:cookTime ^^xsd:integer} minutes
|
|
7
|
+
Servings: **8** {cook:recipeYield ^^xsd:integer}
|
|
8
|
+
Difficulty: **Medium** {cook:skill:difficulty}
|
|
9
|
+
|
|
10
|
+
Ingredients:
|
|
11
|
+
|
|
12
|
+
* Apples {+cook:apples ?cook:recipeIngredient .cook:Ingredient label}
|
|
13
|
+
* Sugar {+cook:sugar ?cook:recipeIngredient .cook:Ingredient label}
|
|
14
|
+
* Butter {+cook:butter ?cook:recipeIngredient .cook:Ingredient label}
|
|
15
|
+
* Flour {+cook:flour ?cook:recipeIngredient .cook:Ingredient label}
|
|
16
|
+
* Cinnamon {+cook:cinnamon ?cook:recipeIngredient .cook:Ingredient label}
|
|
17
|
+
|
|
18
|
+
_Ingredient_ {=cook:ingredient .Class label} - is a class for all ingredients in our **Cookbook** {!member +urn:collection:cookbook .Container label }.
|
|
19
|
+
|
|
20
|
+
Equipment:
|
|
21
|
+
|
|
22
|
+
* Oven {+cook:oven-1 ?cook:requiresEquipment .cook:Device}
|
|
23
|
+
* Mixing bowl {+cook:bowl-1 ?cook:requiresEquipment .cook:Utensil}
|
|
24
|
+
* Rolling pin {+cook:pin-1 ?cook:requiresEquipment .cook:Utensil}
|
|
25
|
+
|
|
26
|
+
Calories: [1235] {cook:calories ^^xsd:integer} calories
|
|
27
|
+
Sugar: [40] {cook:sugarGrams ^^xsd:decimal} grams
|
|
28
|
+
|
|
29
|
+
Related recipes:
|
|
30
|
+
|
|
31
|
+
* Apple Crumble {+cook:apple-crumble !cook:variationOf .cook:Recipe}
|
|
32
|
+
* Peach Pie {+cook:peach-pie !cook:variationOf .cook:Recipe}
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
[ex] <http://example.org/>
|
|
2
|
+
|
|
3
|
+
# Status {=ex:Status .Class label}
|
|
4
|
+
|
|
5
|
+
A controlled vocabulary.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Status Shape {=ex:StatusShape .sh:NodeShape label}
|
|
10
|
+
|
|
11
|
+
Target: [Status] {+ex:Status ?sh:targetClass}
|
|
12
|
+
|
|
13
|
+
## Status Property {=ex:StatusShape-status .sh:PropertyShape ?sh:property}
|
|
14
|
+
|
|
15
|
+
Path: [status] {+ex:status ?sh:path}
|
|
16
|
+
|
|
17
|
+
Required: [1] {sh:minCount ^^xsd:integer}
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
# Test Data
|
|
22
|
+
|
|
23
|
+
## Valid Record {=ex:Record-Valid .ex:Status}
|
|
24
|
+
|
|
25
|
+
This record should pass validation.
|
|
26
|
+
|
|
27
|
+
### Status {+ex:Active ?ex:status}
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## Invalid Record {=ex:Record-Invalid .ex:Status}
|
|
32
|
+
|
|
33
|
+
This record should fail validation - missing required status.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# MD-LD Demo Examples
|
|
2
|
+
|
|
3
|
+
Quick start examples demonstrating core MD-LD patterns and features.
|
|
4
|
+
|
|
5
|
+
## Demo Examples
|
|
6
|
+
|
|
7
|
+
### 🎯 Getting Started
|
|
8
|
+
|
|
9
|
+
**[minimal.md](./minimal.md)** Minimal document with subject and author
|
|
10
|
+
**[journal.md](./journal.md)** Personal journal with activities, reflections, and related events
|
|
11
|
+
|
|
12
|
+
### 🍳 Everyday Examples
|
|
13
|
+
|
|
14
|
+
**[recipe.md](./recipe.md)** Complete recipe with ingredients, equipment, and nutritional information
|
|
15
|
+
**[project.md](./project.md)** Weekend project with tasks, resources, team, and blocking issues
|
|
16
|
+
|
|
17
|
+
### 🚀 Advanced Examples
|
|
18
|
+
|
|
19
|
+
**[space-mission.md](./space-mission.md)** Apollo 11 mission with crew, equipment, and objectives
|
|
20
|
+
**[research.md](./research.md)** Scientific paper with datasets, methodology, and organizations
|
|
21
|
+
|
|
22
|
+
## Key Features Demonstrated
|
|
23
|
+
|
|
24
|
+
✅ **Explicit Annotations** All semantic data comes from `{...}` blocks
|
|
25
|
+
✅ **Subject Chaining** Using `{=IRI}` and `{+IRI}` for context
|
|
26
|
+
✅ **List Items** Explicit annotations per list item (no semantic propagation)
|
|
27
|
+
✅ **Nested Structures** Hierarchical data organization
|
|
28
|
+
✅ **PROV-O Integration** Complete provenance tracking
|
|
29
|
+
✅ **Fragment Subjects** Document structuring with `{=#fragment}`
|
|
30
|
+
✅ **Reverse Relations** Using `!p` for inverse relationships
|
|
31
|
+
✅ **Datatypes & Languages** `^^xsd:types` and `@lang` tags
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
*Each demo is a complete, working MD-LD document that can be parsed and round-tripped safely.*
|