mdld-parse 0.9.9 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +263 -326
- package/dist/index.js +1860 -1337
- package/package.json +4 -4
- package/dist/.nojekyll +0 -0
- package/dist/CNAME +0 -1
- package/dist/README.md +0 -500
- package/dist/docs/API.md +0 -435
- package/dist/docs/Architecture.md +0 -190
- package/dist/docs/Guide.md +0 -816
- package/dist/docs/LLM-MDLD-Inducer.md +0 -550
- package/dist/docs/LiterateRDF.md +0 -472
- package/dist/docs/Parser.md +0 -720
- package/dist/docs/Performance.md +0 -238
- package/dist/docs/Spec-render.md +0 -1439
- package/dist/docs/Subject.md +0 -180
- package/dist/docs/Syntax.md +0 -463
- package/dist/docs/TodoList.md +0 -215
- package/dist/docs/Use-Cases.md +0 -354
- package/dist/docs/guide-m.md +0 -56
- package/dist/docs/index.html +0 -108
- package/dist/docs/index.md +0 -56
- package/dist/docs/one-page.md +0 -322
- package/dist/docs/origin.md +0 -460
- package/dist/docs/polarity.md +0 -586
- package/dist/docs/primary-subject.md +0 -326
- package/dist/docs/statements.md +0 -226
- package/dist/docs/tokenization-performance-report.md +0 -164
- package/dist/examples/Cookbook.md +0 -32
- package/dist/examples/Status-SHACL.md +0 -31
- package/dist/examples/demo/index.md +0 -35
- package/dist/examples/demo/journal.md +0 -58
- package/dist/examples/demo/minimal.md +0 -23
- package/dist/examples/demo/project.md +0 -84
- package/dist/examples/demo/prov.md +0 -107
- package/dist/examples/demo/rdf.md +0 -69
- package/dist/examples/demo/recipe.md +0 -50
- package/dist/examples/demo/research.md +0 -84
- package/dist/examples/demo/shacl.md +0 -161
- package/dist/examples/demo/space-mission.md +0 -58
- package/dist/examples/demo/tasks.md +0 -175
- package/dist/examples/demo/xsd.md +0 -91
- package/dist/examples/dogfood.md +0 -288
- package/dist/examples/few-shot.md +0 -309
- package/dist/examples/graph/rdf-graph.html +0 -473
- package/dist/examples/graph/rdf_force_graph_web_component.html +0 -506
- package/dist/examples/index.md +0 -45
- package/dist/examples/llm-time-workflow.md +0 -173
- package/dist/examples/statements.md +0 -43
- package/dist/examples/workflow.md +0 -150
- package/dist/index.html +0 -3049
- package/dist/logo.svg +0 -7
- package/dist/spec/Spec-compact.md +0 -368
- package/dist/spec/Spec-ultra.md +0 -232
- package/dist/spec/Spec.md +0 -1116
- package/dist/spec/index.md +0 -46
package/dist/docs/Performance.md
DELETED
|
@@ -1,238 +0,0 @@
|
|
|
1
|
-
# MDLD Performance
|
|
2
|
-
|
|
3
|
-
## Testing Methodology
|
|
4
|
-
|
|
5
|
-
### Environment
|
|
6
|
-
- **Hardware**: MacBook Pro M1
|
|
7
|
-
- **Node.js**: v24.7.0
|
|
8
|
-
- **Method**: Median of multiple runs, warmed-up parser
|
|
9
|
-
- **Data**: Synthetic and real-world ontologies
|
|
10
|
-
|
|
11
|
-
### Test Types
|
|
12
|
-
1. **Synthetic Scaling** - Precise quad count testing
|
|
13
|
-
2. **Real-World Ontologies** - PROV-O, RDF+RDFS, SHACL
|
|
14
|
-
3. **Binary Search** - Exact limit determination
|
|
15
|
-
4. **Conservative Limits** - 95% confidence with 10% safety margin
|
|
16
|
-
|
|
17
|
-
## Grounded Performance Numbers
|
|
18
|
-
|
|
19
|
-
### Key Metrics
|
|
20
|
-
|
|
21
|
-
| Metric | Value | Use Case |
|
|
22
|
-
|--------|-------|----------|
|
|
23
|
-
| **60fps Frame** | 4,527 quads | Interactive knowledge graphs |
|
|
24
|
-
| **1-Second Batch** | 225,059 quads | Background reindexing, imports |
|
|
25
|
-
| **Sustained Rate** | 252K quads/sec | Continuous processing |
|
|
26
|
-
|
|
27
|
-
### Real-World Efficiency
|
|
28
|
-
|
|
29
|
-
| Document Type | Quads | Size (KB) | Quads/sec |
|
|
30
|
-
|---------------|-------|-----------|-----------|
|
|
31
|
-
| PROV-O | 944 | 141 | 27,505 |
|
|
32
|
-
| RDF+RDFS | 231 | 39 | 36,903 |
|
|
33
|
-
| SHACL | 286 | 109 | 23,692 |
|
|
34
|
-
| Mixed Set | 1,461 | 288 | 51,684 |
|
|
35
|
-
|
|
36
|
-
## Scaling Characteristics
|
|
37
|
-
|
|
38
|
-
### 60fps Performance
|
|
39
|
-
|
|
40
|
-
| Scale | Quads | Parse Time | Status |
|
|
41
|
-
|-------|-------|-------------|--------|
|
|
42
|
-
| **Conservative Limit** | 4,527 | 16.12ms | ✅ 96.7% budget |
|
|
43
|
-
| **Maximum Limit** | 5,031 | 19.15ms | ❌ Exceeds budget |
|
|
44
|
-
| **Real-World Sets** | 4,383 | 115.28ms | ❌ 87.1% efficiency |
|
|
45
|
-
|
|
46
|
-
### 1-Second Performance
|
|
47
|
-
|
|
48
|
-
| Scale | Quads | Parse Time | Status |
|
|
49
|
-
|-------|-------|-------------|--------|
|
|
50
|
-
| **Conservative Limit** | 225,059 | 890ms | ✅ 89.0% budget |
|
|
51
|
-
| **Maximum Limit** | 250,066 | 990ms | ✅ Within budget |
|
|
52
|
-
| **Real-World Sets** | 249,831 | 6088ms | ✅ 99.9% efficiency |
|
|
53
|
-
|
|
54
|
-
## Enterprise Scaling
|
|
55
|
-
|
|
56
|
-
### Document Collection Limits
|
|
57
|
-
|
|
58
|
-
| Enterprise Size | Total Quads | Ontology Sets | Parse Time | Architecture |
|
|
59
|
-
|----------------|-------------|---------------|------------|--------------|
|
|
60
|
-
| **Small** | 7,602 | 3 | 147ms | Full reparse OK |
|
|
61
|
-
| **Medium** | 28,670 | 19 | 647ms | Background reparse |
|
|
62
|
-
| **Large** | 72,830 | 49 | 1,461ms | Incremental only |
|
|
63
|
-
|
|
64
|
-
### Documents Per Second
|
|
65
|
-
|
|
66
|
-
| Document Type | Parses/sec | Quads/sec |
|
|
67
|
-
|---------------|-------------|-----------|
|
|
68
|
-
| PROV-O | 60 | 27,505 |
|
|
69
|
-
| RDF+RDFS | 349 | 36,903 |
|
|
70
|
-
| SHACL | 153 | 23,692 |
|
|
71
|
-
| Mixed Set | 35 | 51,684 |
|
|
72
|
-
|
|
73
|
-
## Performance Optimization
|
|
74
|
-
|
|
75
|
-
### Character-Based Tokenization
|
|
76
|
-
|
|
77
|
-
**Improvement**: 20-28% faster than regex-based approaches
|
|
78
|
-
|
|
79
|
-
**Techniques**:
|
|
80
|
-
- Direct character inspection
|
|
81
|
-
- No regex engine overhead
|
|
82
|
-
- Predictable O(n) complexity
|
|
83
|
-
- Memory-efficient state tracking
|
|
84
|
-
|
|
85
|
-
### Memory Management
|
|
86
|
-
|
|
87
|
-
**Characteristics**:
|
|
88
|
-
- ~640 bytes per quad after GC
|
|
89
|
-
- Streaming-friendly single-pass
|
|
90
|
-
- No full document copies
|
|
91
|
-
- Efficient indexing structures
|
|
92
|
-
|
|
93
|
-
### Incremental Updates
|
|
94
|
-
|
|
95
|
-
**Performance**:
|
|
96
|
-
- <5ms per ontology addition
|
|
97
|
-
- O(new) complexity
|
|
98
|
-
- Real-time capable for <4K quads
|
|
99
|
-
- Background processing for larger sets
|
|
100
|
-
|
|
101
|
-
## Test Results
|
|
102
|
-
|
|
103
|
-
### Binary Search Results
|
|
104
|
-
|
|
105
|
-
```
|
|
106
|
-
60fps Target: 16.67ms
|
|
107
|
-
├── Conservative: 4,527 quads @ 16.12ms (96.7% budget)
|
|
108
|
-
└── Maximum: 5,031 quads @ 19.15ms (exceeds budget)
|
|
109
|
-
|
|
110
|
-
1-Second Target: 1000ms
|
|
111
|
-
├── Conservative: 225,059 quads @ 890ms (89.0% budget)
|
|
112
|
-
└── Maximum: 250,066 quads @ 990ms (within budget)
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
### Real-World vs Synthetic
|
|
116
|
-
|
|
117
|
-
| Metric | Synthetic | Real-World | Efficiency |
|
|
118
|
-
|--------|-----------|------------|------------|
|
|
119
|
-
| 60fps Quads | 5,031 | 4,383 | 87.1% |
|
|
120
|
-
| 1-Sec Quads | 250,066 | 249,831 | 99.9% |
|
|
121
|
-
| Parse Rate | 252K/sec | 51K/sec | 20.2% |
|
|
122
|
-
|
|
123
|
-
## Usage Recommendations
|
|
124
|
-
|
|
125
|
-
### Real-Time Applications (60fps)
|
|
126
|
-
|
|
127
|
-
**Requirements**:
|
|
128
|
-
- Document size: ≤4K quads
|
|
129
|
-
- Update pattern: Incremental only
|
|
130
|
-
- Use case: Interactive knowledge graphs
|
|
131
|
-
|
|
132
|
-
**Architecture**:
|
|
133
|
-
- Incremental updates
|
|
134
|
-
- Pre-built indexes
|
|
135
|
-
- Background indexing for larger sets
|
|
136
|
-
|
|
137
|
-
### Batch Processing (1-Second)
|
|
138
|
-
|
|
139
|
-
**Requirements**:
|
|
140
|
-
- Document size: ≤225K quads
|
|
141
|
-
- Update pattern: Full reparse
|
|
142
|
-
- Use case: Background reindexing, imports
|
|
143
|
-
|
|
144
|
-
**Architecture**:
|
|
145
|
-
- Worker thread processing
|
|
146
|
-
- Chunked processing for >50MB
|
|
147
|
-
- Progress reporting
|
|
148
|
-
|
|
149
|
-
### Enterprise Knowledge Bases
|
|
150
|
-
|
|
151
|
-
**Scale Guidelines**:
|
|
152
|
-
- **Small** (<4K quads): Real-time updates
|
|
153
|
-
- **Medium** (4-225K quads): Batch reindexing
|
|
154
|
-
- **Large** (>225K quads): Streaming architecture
|
|
155
|
-
|
|
156
|
-
**Best Practices**:
|
|
157
|
-
- Use incremental updates for real-time
|
|
158
|
-
- Batch process during maintenance windows
|
|
159
|
-
- Implement streaming for large corpora
|
|
160
|
-
|
|
161
|
-
## Performance Monitoring
|
|
162
|
-
|
|
163
|
-
### Built-in Metrics
|
|
164
|
-
|
|
165
|
-
The parser includes performance monitoring:
|
|
166
|
-
- Parse time measurement
|
|
167
|
-
- Memory usage tracking
|
|
168
|
-
- Quad count statistics
|
|
169
|
-
- Error rate monitoring
|
|
170
|
-
|
|
171
|
-
### Benchmarking Tools
|
|
172
|
-
|
|
173
|
-
**Available Tests**:
|
|
174
|
-
- `load-test-grounded-numbers.js` - Exact limit determination
|
|
175
|
-
- `load-test-real-ontologies.js` - Real-world performance
|
|
176
|
-
- `load-test-60fps-scale.js` - Interactive performance
|
|
177
|
-
- `load-test-knowledge-workbench.js` - Enterprise simulation
|
|
178
|
-
|
|
179
|
-
### Performance Regression Testing
|
|
180
|
-
|
|
181
|
-
**Method**:
|
|
182
|
-
- Automated performance benchmarks
|
|
183
|
-
- CI/CD integration
|
|
184
|
-
- Alert on >10% performance degradation
|
|
185
|
-
- Historical trend tracking
|
|
186
|
-
|
|
187
|
-
## Architectural Implications
|
|
188
|
-
|
|
189
|
-
### Real-Time Capabilities
|
|
190
|
-
|
|
191
|
-
✅ **Interactive UI**: Up to 4K quads with 60fps updates
|
|
192
|
-
✅ **Background Processing**: Up to 225K quads per second
|
|
193
|
-
✅ **Enterprise Scale**: 154 ontology sets per reparse
|
|
194
|
-
❌ **Full UI Reparse**: Not viable beyond 4K quads
|
|
195
|
-
|
|
196
|
-
### Design Patterns
|
|
197
|
-
|
|
198
|
-
**Incremental Architecture**:
|
|
199
|
-
- Stream updates as they arrive
|
|
200
|
-
- Maintain pre-built indexes
|
|
201
|
-
- Use background workers for bulk operations
|
|
202
|
-
|
|
203
|
-
**Batch Architecture**:
|
|
204
|
-
- Process documents in chunks
|
|
205
|
-
- Use worker threads
|
|
206
|
-
- Implement progress reporting
|
|
207
|
-
|
|
208
|
-
**Streaming Architecture**:
|
|
209
|
-
- Process large corpora incrementally
|
|
210
|
-
- Use memory-efficient streaming
|
|
211
|
-
- Implement checkpointing
|
|
212
|
-
|
|
213
|
-
## Future Optimizations
|
|
214
|
-
|
|
215
|
-
### Potential Improvements
|
|
216
|
-
|
|
217
|
-
1. **Parallel Processing** - Multi-threaded tokenization
|
|
218
|
-
2. **Memory Pooling** - Reduced GC pressure
|
|
219
|
-
3. **SIMD Optimization** - Vectorized character processing
|
|
220
|
-
4. **WebAssembly** - Performance-critical paths
|
|
221
|
-
|
|
222
|
-
### Scaling Targets
|
|
223
|
-
|
|
224
|
-
**Short-term** (6 months):
|
|
225
|
-
- 10% improvement in character tokenization
|
|
226
|
-
- Better memory management
|
|
227
|
-
|
|
228
|
-
**Long-term** (12 months):
|
|
229
|
-
- 25% overall performance improvement
|
|
230
|
-
- Parallel processing support
|
|
231
|
-
|
|
232
|
-
---
|
|
233
|
-
|
|
234
|
-
## Bottom Line
|
|
235
|
-
|
|
236
|
-
MDLD delivers **enterprise-scale performance** with **252K quads/sec sustained throughput** and **real-time capabilities** for interactive applications. The character-based tokenizer provides **significant performance advantages** while maintaining clean, maintainable code architecture.
|
|
237
|
-
|
|
238
|
-
**Key Takeaway**: MDLD can reliably process **4K quads at 60fps** and **225K quads per second**, making it suitable for both real-time interactive applications and large-scale knowledge management systems.
|