mdld-parse 0.9.9 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. package/README.md +263 -326
  2. package/dist/index.js +1860 -1337
  3. package/package.json +4 -4
  4. package/dist/.nojekyll +0 -0
  5. package/dist/CNAME +0 -1
  6. package/dist/README.md +0 -500
  7. package/dist/docs/API.md +0 -435
  8. package/dist/docs/Architecture.md +0 -190
  9. package/dist/docs/Guide.md +0 -816
  10. package/dist/docs/LLM-MDLD-Inducer.md +0 -550
  11. package/dist/docs/LiterateRDF.md +0 -472
  12. package/dist/docs/Parser.md +0 -720
  13. package/dist/docs/Performance.md +0 -238
  14. package/dist/docs/Spec-render.md +0 -1439
  15. package/dist/docs/Subject.md +0 -180
  16. package/dist/docs/Syntax.md +0 -463
  17. package/dist/docs/TodoList.md +0 -215
  18. package/dist/docs/Use-Cases.md +0 -354
  19. package/dist/docs/guide-m.md +0 -56
  20. package/dist/docs/index.html +0 -108
  21. package/dist/docs/index.md +0 -56
  22. package/dist/docs/one-page.md +0 -322
  23. package/dist/docs/origin.md +0 -460
  24. package/dist/docs/polarity.md +0 -586
  25. package/dist/docs/primary-subject.md +0 -326
  26. package/dist/docs/statements.md +0 -226
  27. package/dist/docs/tokenization-performance-report.md +0 -164
  28. package/dist/examples/Cookbook.md +0 -32
  29. package/dist/examples/Status-SHACL.md +0 -31
  30. package/dist/examples/demo/index.md +0 -35
  31. package/dist/examples/demo/journal.md +0 -58
  32. package/dist/examples/demo/minimal.md +0 -23
  33. package/dist/examples/demo/project.md +0 -84
  34. package/dist/examples/demo/prov.md +0 -107
  35. package/dist/examples/demo/rdf.md +0 -69
  36. package/dist/examples/demo/recipe.md +0 -50
  37. package/dist/examples/demo/research.md +0 -84
  38. package/dist/examples/demo/shacl.md +0 -161
  39. package/dist/examples/demo/space-mission.md +0 -58
  40. package/dist/examples/demo/tasks.md +0 -175
  41. package/dist/examples/demo/xsd.md +0 -91
  42. package/dist/examples/dogfood.md +0 -288
  43. package/dist/examples/few-shot.md +0 -309
  44. package/dist/examples/graph/rdf-graph.html +0 -473
  45. package/dist/examples/graph/rdf_force_graph_web_component.html +0 -506
  46. package/dist/examples/index.md +0 -45
  47. package/dist/examples/llm-time-workflow.md +0 -173
  48. package/dist/examples/statements.md +0 -43
  49. package/dist/examples/workflow.md +0 -150
  50. package/dist/index.html +0 -3049
  51. package/dist/logo.svg +0 -7
  52. package/dist/spec/Spec-compact.md +0 -368
  53. package/dist/spec/Spec-ultra.md +0 -232
  54. package/dist/spec/Spec.md +0 -1116
  55. package/dist/spec/index.md +0 -46
@@ -1,238 +0,0 @@
1
- # MDLD Performance
2
-
3
- ## Testing Methodology
4
-
5
- ### Environment
6
- - **Hardware**: MacBook Pro M1
7
- - **Node.js**: v24.7.0
8
- - **Method**: Median of multiple runs, warmed-up parser
9
- - **Data**: Synthetic and real-world ontologies
10
-
11
- ### Test Types
12
- 1. **Synthetic Scaling** - Precise quad count testing
13
- 2. **Real-World Ontologies** - PROV-O, RDF+RDFS, SHACL
14
- 3. **Binary Search** - Exact limit determination
15
- 4. **Conservative Limits** - 95% confidence with 10% safety margin
16
-
17
- ## Grounded Performance Numbers
18
-
19
- ### Key Metrics
20
-
21
- | Metric | Value | Use Case |
22
- |--------|-------|----------|
23
- | **60fps Frame** | 4,527 quads | Interactive knowledge graphs |
24
- | **1-Second Batch** | 225,059 quads | Background reindexing, imports |
25
- | **Sustained Rate** | 252K quads/sec | Continuous processing |
26
-
27
- ### Real-World Efficiency
28
-
29
- | Document Type | Quads | Size (KB) | Quads/sec |
30
- |---------------|-------|-----------|-----------|
31
- | PROV-O | 944 | 141 | 27,505 |
32
- | RDF+RDFS | 231 | 39 | 36,903 |
33
- | SHACL | 286 | 109 | 23,692 |
34
- | Mixed Set | 1,461 | 288 | 51,684 |
35
-
36
- ## Scaling Characteristics
37
-
38
- ### 60fps Performance
39
-
40
- | Scale | Quads | Parse Time | Status |
41
- |-------|-------|-------------|--------|
42
- | **Conservative Limit** | 4,527 | 16.12ms | ✅ 96.7% budget |
43
- | **Maximum Limit** | 5,031 | 19.15ms | ❌ Exceeds budget |
44
- | **Real-World Sets** | 4,383 | 115.28ms | ❌ 87.1% efficiency |
45
-
46
- ### 1-Second Performance
47
-
48
- | Scale | Quads | Parse Time | Status |
49
- |-------|-------|-------------|--------|
50
- | **Conservative Limit** | 225,059 | 890ms | ✅ 89.0% budget |
51
- | **Maximum Limit** | 250,066 | 990ms | ✅ Within budget |
52
- | **Real-World Sets** | 249,831 | 6088ms | ✅ 99.9% efficiency |
53
-
54
- ## Enterprise Scaling
55
-
56
- ### Document Collection Limits
57
-
58
- | Enterprise Size | Total Quads | Ontology Sets | Parse Time | Architecture |
59
- |----------------|-------------|---------------|------------|--------------|
60
- | **Small** | 7,602 | 3 | 147ms | Full reparse OK |
61
- | **Medium** | 28,670 | 19 | 647ms | Background reparse |
62
- | **Large** | 72,830 | 49 | 1,461ms | Incremental only |
63
-
64
- ### Documents Per Second
65
-
66
- | Document Type | Parses/sec | Quads/sec |
67
- |---------------|-------------|-----------|
68
- | PROV-O | 60 | 27,505 |
69
- | RDF+RDFS | 349 | 36,903 |
70
- | SHACL | 153 | 23,692 |
71
- | Mixed Set | 35 | 51,684 |
72
-
73
- ## Performance Optimization
74
-
75
- ### Character-Based Tokenization
76
-
77
- **Improvement**: 20-28% faster than regex-based approaches
78
-
79
- **Techniques**:
80
- - Direct character inspection
81
- - No regex engine overhead
82
- - Predictable O(n) complexity
83
- - Memory-efficient state tracking
84
-
85
- ### Memory Management
86
-
87
- **Characteristics**:
88
- - ~640 bytes per quad after GC
89
- - Streaming-friendly single-pass
90
- - No full document copies
91
- - Efficient indexing structures
92
-
93
- ### Incremental Updates
94
-
95
- **Performance**:
96
- - <5ms per ontology addition
97
- - O(new) complexity
98
- - Real-time capable for <4K quads
99
- - Background processing for larger sets
100
-
101
- ## Test Results
102
-
103
- ### Binary Search Results
104
-
105
- ```
106
- 60fps Target: 16.67ms
107
- ├── Conservative: 4,527 quads @ 16.12ms (96.7% budget)
108
- └── Maximum: 5,031 quads @ 19.15ms (exceeds budget)
109
-
110
- 1-Second Target: 1000ms
111
- ├── Conservative: 225,059 quads @ 890ms (89.0% budget)
112
- └── Maximum: 250,066 quads @ 990ms (within budget)
113
- ```
114
-
115
- ### Real-World vs Synthetic
116
-
117
- | Metric | Synthetic | Real-World | Efficiency |
118
- |--------|-----------|------------|------------|
119
- | 60fps Quads | 5,031 | 4,383 | 87.1% |
120
- | 1-Sec Quads | 250,066 | 249,831 | 99.9% |
121
- | Parse Rate | 252K/sec | 51K/sec | 20.2% |
122
-
123
- ## Usage Recommendations
124
-
125
- ### Real-Time Applications (60fps)
126
-
127
- **Requirements**:
128
- - Document size: ≤4K quads
129
- - Update pattern: Incremental only
130
- - Use case: Interactive knowledge graphs
131
-
132
- **Architecture**:
133
- - Incremental updates
134
- - Pre-built indexes
135
- - Background indexing for larger sets
136
-
137
- ### Batch Processing (1-Second)
138
-
139
- **Requirements**:
140
- - Document size: ≤225K quads
141
- - Update pattern: Full reparse
142
- - Use case: Background reindexing, imports
143
-
144
- **Architecture**:
145
- - Worker thread processing
146
- - Chunked processing for >50MB
147
- - Progress reporting
148
-
149
- ### Enterprise Knowledge Bases
150
-
151
- **Scale Guidelines**:
152
- - **Small** (<4K quads): Real-time updates
153
- - **Medium** (4-225K quads): Batch reindexing
154
- - **Large** (>225K quads): Streaming architecture
155
-
156
- **Best Practices**:
157
- - Use incremental updates for real-time
158
- - Batch process during maintenance windows
159
- - Implement streaming for large corpora
160
-
161
- ## Performance Monitoring
162
-
163
- ### Built-in Metrics
164
-
165
- The parser includes performance monitoring:
166
- - Parse time measurement
167
- - Memory usage tracking
168
- - Quad count statistics
169
- - Error rate monitoring
170
-
171
- ### Benchmarking Tools
172
-
173
- **Available Tests**:
174
- - `load-test-grounded-numbers.js` - Exact limit determination
175
- - `load-test-real-ontologies.js` - Real-world performance
176
- - `load-test-60fps-scale.js` - Interactive performance
177
- - `load-test-knowledge-workbench.js` - Enterprise simulation
178
-
179
- ### Performance Regression Testing
180
-
181
- **Method**:
182
- - Automated performance benchmarks
183
- - CI/CD integration
184
- - Alert on >10% performance degradation
185
- - Historical trend tracking
186
-
187
- ## Architectural Implications
188
-
189
- ### Real-Time Capabilities
190
-
191
- ✅ **Interactive UI**: Up to 4K quads with 60fps updates
192
- ✅ **Background Processing**: Up to 225K quads per second
193
- ✅ **Enterprise Scale**: 154 ontology sets per reparse
194
- ❌ **Full UI Reparse**: Not viable beyond 4K quads
195
-
196
- ### Design Patterns
197
-
198
- **Incremental Architecture**:
199
- - Stream updates as they arrive
200
- - Maintain pre-built indexes
201
- - Use background workers for bulk operations
202
-
203
- **Batch Architecture**:
204
- - Process documents in chunks
205
- - Use worker threads
206
- - Implement progress reporting
207
-
208
- **Streaming Architecture**:
209
- - Process large corpora incrementally
210
- - Use memory-efficient streaming
211
- - Implement checkpointing
212
-
213
- ## Future Optimizations
214
-
215
- ### Potential Improvements
216
-
217
- 1. **Parallel Processing** - Multi-threaded tokenization
218
- 2. **Memory Pooling** - Reduced GC pressure
219
- 3. **SIMD Optimization** - Vectorized character processing
220
- 4. **WebAssembly** - Performance-critical paths
221
-
222
- ### Scaling Targets
223
-
224
- **Short-term** (6 months):
225
- - 10% improvement in character tokenization
226
- - Better memory management
227
-
228
- **Long-term** (12 months):
229
- - 25% overall performance improvement
230
- - Parallel processing support
231
-
232
- ---
233
-
234
- ## Bottom Line
235
-
236
- MDLD delivers **enterprise-scale performance** with **252K quads/sec sustained throughput** and **real-time capabilities** for interactive applications. The character-based tokenizer provides **significant performance advantages** while maintaining clean, maintainable code architecture.
237
-
238
- **Key Takeaway**: MDLD can reliably process **4K quads at 60fps** and **225K quads per second**, making it suitable for both real-time interactive applications and large-scale knowledge management systems.