@rbalchii/anchor-engine 4.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (539) hide show
  1. package/LICENSE +609 -0
  2. package/README.md +317 -0
  3. package/anchor.bat +5 -0
  4. package/docs/API.md +314 -0
  5. package/docs/DEPLOYMENT.md +448 -0
  6. package/docs/INDEX.md +226 -0
  7. package/docs/STAR_Whitepaper_Executive.md +216 -0
  8. package/docs/TROUBLESHOOTING.md +535 -0
  9. package/docs/archive/GIT_BACKUP_VERIFICATION.md +297 -0
  10. package/docs/archive/adoption-guide.md +264 -0
  11. package/docs/archive/adoption-preparation.md +179 -0
  12. package/docs/archive/agent-harness-integration.md +227 -0
  13. package/docs/archive/api-reference.md +106 -0
  14. package/docs/archive/api_flows_diagram.md +118 -0
  15. package/docs/archive/architecture.md +410 -0
  16. package/docs/archive/architecture_diagram.md +174 -0
  17. package/docs/archive/broader-adoption-preparation.md +175 -0
  18. package/docs/archive/browser-paradigm-architecture.md +163 -0
  19. package/docs/archive/chat-integration.md +124 -0
  20. package/docs/archive/community-adoption-materials.md +103 -0
  21. package/docs/archive/community-adoption.md +147 -0
  22. package/docs/archive/comparison-with-siloed-solutions.md +192 -0
  23. package/docs/archive/comprehensive-docs.md +156 -0
  24. package/docs/archive/data_flow_diagram.md +251 -0
  25. package/docs/archive/enhancement-implementation-summary.md +146 -0
  26. package/docs/archive/evolution-summary.md +141 -0
  27. package/docs/archive/ingestion_pipeline_diagram.md +198 -0
  28. package/docs/archive/native-module-profiling-results.md +135 -0
  29. package/docs/archive/positioning-document.md +158 -0
  30. package/docs/archive/positioning.md +175 -0
  31. package/docs/archive/query-builder-documentation.md +218 -0
  32. package/docs/archive/quick-reference.md +40 -0
  33. package/docs/archive/quickstart.md +63 -0
  34. package/docs/archive/relationship-narrative-discovery.md +141 -0
  35. package/docs/archive/search-logic-improvement-plan.md +336 -0
  36. package/docs/archive/search_architecture_diagram.md +212 -0
  37. package/docs/archive/semantic-architecture-guide.md +97 -0
  38. package/docs/archive/sequence-diagrams.md +128 -0
  39. package/docs/archive/system_components_diagram.md +296 -0
  40. package/docs/archive/test-framework-integration.md +109 -0
  41. package/docs/archive/testing-framework-documentation.md +397 -0
  42. package/docs/archive/testing-framework-summary.md +121 -0
  43. package/docs/archive/testing-framework.md +377 -0
  44. package/docs/archive/ui-architecture.md +75 -0
  45. package/docs/arxiv/BIBLIOGRAPHY.bib +145 -0
  46. package/docs/arxiv/RELATED_WORK.tex +39 -0
  47. package/docs/arxiv/compile.bat +48 -0
  48. package/docs/arxiv/joss_response.md +33 -0
  49. package/docs/arxiv/prepare-submission.bat +46 -0
  50. package/docs/arxiv/review.md +128 -0
  51. package/docs/arxiv/star-whitepaper.tex +657 -0
  52. package/docs/code-patterns.md +289 -0
  53. package/docs/whitepaper.md +445 -0
  54. package/engine/dist/agent/runtime.d.ts +41 -0
  55. package/engine/dist/agent/runtime.d.ts.map +1 -0
  56. package/engine/dist/agent/runtime.js +73 -0
  57. package/engine/dist/agent/runtime.js.map +1 -0
  58. package/engine/dist/commands/audit-tags.d.ts +14 -0
  59. package/engine/dist/commands/audit-tags.d.ts.map +1 -0
  60. package/engine/dist/commands/audit-tags.js +180 -0
  61. package/engine/dist/commands/audit-tags.js.map +1 -0
  62. package/engine/dist/commands/distill.d.ts +19 -0
  63. package/engine/dist/commands/distill.d.ts.map +1 -0
  64. package/engine/dist/commands/distill.js +114 -0
  65. package/engine/dist/commands/distill.js.map +1 -0
  66. package/engine/dist/commands/generate-synonyms.d.ts +14 -0
  67. package/engine/dist/commands/generate-synonyms.d.ts.map +1 -0
  68. package/engine/dist/commands/generate-synonyms.js +91 -0
  69. package/engine/dist/commands/generate-synonyms.js.map +1 -0
  70. package/engine/dist/config/index.d.ts +115 -0
  71. package/engine/dist/config/index.d.ts.map +1 -0
  72. package/engine/dist/config/index.js +326 -0
  73. package/engine/dist/config/index.js.map +1 -0
  74. package/engine/dist/config/max-recall-config.d.ts +102 -0
  75. package/engine/dist/config/max-recall-config.d.ts.map +1 -0
  76. package/engine/dist/config/max-recall-config.js +102 -0
  77. package/engine/dist/config/max-recall-config.js.map +1 -0
  78. package/engine/dist/config/paths.d.ts +40 -0
  79. package/engine/dist/config/paths.d.ts.map +1 -0
  80. package/engine/dist/config/paths.js +49 -0
  81. package/engine/dist/config/paths.js.map +1 -0
  82. package/engine/dist/core/batch.d.ts +19 -0
  83. package/engine/dist/core/batch.d.ts.map +1 -0
  84. package/engine/dist/core/batch.js +37 -0
  85. package/engine/dist/core/batch.js.map +1 -0
  86. package/engine/dist/core/db.d.ts +58 -0
  87. package/engine/dist/core/db.d.ts.map +1 -0
  88. package/engine/dist/core/db.js +563 -0
  89. package/engine/dist/core/db.js.map +1 -0
  90. package/engine/dist/core/inference/ChatWorker.d.ts +2 -0
  91. package/engine/dist/core/inference/ChatWorker.d.ts.map +1 -0
  92. package/engine/dist/core/inference/ChatWorker.js +28 -0
  93. package/engine/dist/core/inference/ChatWorker.js.map +1 -0
  94. package/engine/dist/core/inference/context_manager.d.ts +49 -0
  95. package/engine/dist/core/inference/context_manager.d.ts.map +1 -0
  96. package/engine/dist/core/inference/context_manager.js +199 -0
  97. package/engine/dist/core/inference/context_manager.js.map +1 -0
  98. package/engine/dist/core/inference/llamaLoaderWorker.d.ts +2 -0
  99. package/engine/dist/core/inference/llamaLoaderWorker.d.ts.map +1 -0
  100. package/engine/dist/core/inference/llamaLoaderWorker.js +23 -0
  101. package/engine/dist/core/inference/llamaLoaderWorker.js.map +1 -0
  102. package/engine/dist/core/vector.d.ts +40 -0
  103. package/engine/dist/core/vector.d.ts.map +1 -0
  104. package/engine/dist/core/vector.js +167 -0
  105. package/engine/dist/core/vector.js.map +1 -0
  106. package/engine/dist/index.d.ts +4 -0
  107. package/engine/dist/index.d.ts.map +1 -0
  108. package/engine/dist/index.js +400 -0
  109. package/engine/dist/index.js.map +1 -0
  110. package/engine/dist/middleware/auth.d.ts +14 -0
  111. package/engine/dist/middleware/auth.d.ts.map +1 -0
  112. package/engine/dist/middleware/auth.js +44 -0
  113. package/engine/dist/middleware/auth.js.map +1 -0
  114. package/engine/dist/middleware/request-tracing.d.ts +29 -0
  115. package/engine/dist/middleware/request-tracing.d.ts.map +1 -0
  116. package/engine/dist/middleware/request-tracing.js +115 -0
  117. package/engine/dist/middleware/request-tracing.js.map +1 -0
  118. package/engine/dist/middleware/validate.d.ts +30 -0
  119. package/engine/dist/middleware/validate.d.ts.map +1 -0
  120. package/engine/dist/middleware/validate.js +117 -0
  121. package/engine/dist/middleware/validate.js.map +1 -0
  122. package/engine/dist/native/index.d.ts +106 -0
  123. package/engine/dist/native/index.d.ts.map +1 -0
  124. package/engine/dist/native/index.js +230 -0
  125. package/engine/dist/native/index.js.map +1 -0
  126. package/engine/dist/native/types.d.ts +45 -0
  127. package/engine/dist/native/types.d.ts.map +1 -0
  128. package/engine/dist/native/types.js +6 -0
  129. package/engine/dist/native/types.js.map +1 -0
  130. package/engine/dist/profiling/atomization-profiling.d.ts +8 -0
  131. package/engine/dist/profiling/atomization-profiling.d.ts.map +1 -0
  132. package/engine/dist/profiling/atomization-profiling.js +108 -0
  133. package/engine/dist/profiling/atomization-profiling.js.map +1 -0
  134. package/engine/dist/profiling/bottleneck-identification.d.ts +8 -0
  135. package/engine/dist/profiling/bottleneck-identification.d.ts.map +1 -0
  136. package/engine/dist/profiling/bottleneck-identification.js +249 -0
  137. package/engine/dist/profiling/bottleneck-identification.js.map +1 -0
  138. package/engine/dist/profiling/content-sanitization-profiling.d.ts +12 -0
  139. package/engine/dist/profiling/content-sanitization-profiling.d.ts.map +1 -0
  140. package/engine/dist/profiling/content-sanitization-profiling.js +266 -0
  141. package/engine/dist/profiling/content-sanitization-profiling.js.map +1 -0
  142. package/engine/dist/profiling/simhash-profiling.d.ts +11 -0
  143. package/engine/dist/profiling/simhash-profiling.d.ts.map +1 -0
  144. package/engine/dist/profiling/simhash-profiling.js +168 -0
  145. package/engine/dist/profiling/simhash-profiling.js.map +1 -0
  146. package/engine/dist/routes/api.d.ts +9 -0
  147. package/engine/dist/routes/api.d.ts.map +1 -0
  148. package/engine/dist/routes/api.js +37 -0
  149. package/engine/dist/routes/api.js.map +1 -0
  150. package/engine/dist/routes/enhanced-api.d.ts +9 -0
  151. package/engine/dist/routes/enhanced-api.d.ts.map +1 -0
  152. package/engine/dist/routes/enhanced-api.js +139 -0
  153. package/engine/dist/routes/enhanced-api.js.map +1 -0
  154. package/engine/dist/routes/health.d.ts +8 -0
  155. package/engine/dist/routes/health.d.ts.map +1 -0
  156. package/engine/dist/routes/health.js +89 -0
  157. package/engine/dist/routes/health.js.map +1 -0
  158. package/engine/dist/routes/monitoring.d.ts +8 -0
  159. package/engine/dist/routes/monitoring.d.ts.map +1 -0
  160. package/engine/dist/routes/monitoring.js +509 -0
  161. package/engine/dist/routes/monitoring.js.map +1 -0
  162. package/engine/dist/routes/v1/admin.d.ts +3 -0
  163. package/engine/dist/routes/v1/admin.d.ts.map +1 -0
  164. package/engine/dist/routes/v1/admin.js +261 -0
  165. package/engine/dist/routes/v1/admin.js.map +1 -0
  166. package/engine/dist/routes/v1/atoms.d.ts +3 -0
  167. package/engine/dist/routes/v1/atoms.d.ts.map +1 -0
  168. package/engine/dist/routes/v1/atoms.js +172 -0
  169. package/engine/dist/routes/v1/atoms.js.map +1 -0
  170. package/engine/dist/routes/v1/backup.d.ts +3 -0
  171. package/engine/dist/routes/v1/backup.d.ts.map +1 -0
  172. package/engine/dist/routes/v1/backup.js +100 -0
  173. package/engine/dist/routes/v1/backup.js.map +1 -0
  174. package/engine/dist/routes/v1/git.d.ts +3 -0
  175. package/engine/dist/routes/v1/git.d.ts.map +1 -0
  176. package/engine/dist/routes/v1/git.js +316 -0
  177. package/engine/dist/routes/v1/git.js.map +1 -0
  178. package/engine/dist/routes/v1/ingest.d.ts +3 -0
  179. package/engine/dist/routes/v1/ingest.d.ts.map +1 -0
  180. package/engine/dist/routes/v1/ingest.js +66 -0
  181. package/engine/dist/routes/v1/ingest.js.map +1 -0
  182. package/engine/dist/routes/v1/memory.d.ts +14 -0
  183. package/engine/dist/routes/v1/memory.d.ts.map +1 -0
  184. package/engine/dist/routes/v1/memory.js +87 -0
  185. package/engine/dist/routes/v1/memory.js.map +1 -0
  186. package/engine/dist/routes/v1/research.d.ts +3 -0
  187. package/engine/dist/routes/v1/research.d.ts.map +1 -0
  188. package/engine/dist/routes/v1/research.js +109 -0
  189. package/engine/dist/routes/v1/research.js.map +1 -0
  190. package/engine/dist/routes/v1/search.d.ts +3 -0
  191. package/engine/dist/routes/v1/search.d.ts.map +1 -0
  192. package/engine/dist/routes/v1/search.js +180 -0
  193. package/engine/dist/routes/v1/search.js.map +1 -0
  194. package/engine/dist/routes/v1/settings.d.ts +8 -0
  195. package/engine/dist/routes/v1/settings.d.ts.map +1 -0
  196. package/engine/dist/routes/v1/settings.js +211 -0
  197. package/engine/dist/routes/v1/settings.js.map +1 -0
  198. package/engine/dist/routes/v1/system.d.ts +3 -0
  199. package/engine/dist/routes/v1/system.d.ts.map +1 -0
  200. package/engine/dist/routes/v1/system.js +326 -0
  201. package/engine/dist/routes/v1/system.js.map +1 -0
  202. package/engine/dist/routes/v1/tags.d.ts +3 -0
  203. package/engine/dist/routes/v1/tags.d.ts.map +1 -0
  204. package/engine/dist/routes/v1/tags.js +102 -0
  205. package/engine/dist/routes/v1/tags.js.map +1 -0
  206. package/engine/dist/server-8080.d.ts +2 -0
  207. package/engine/dist/server-8080.d.ts.map +1 -0
  208. package/engine/dist/server-8080.js +74 -0
  209. package/engine/dist/server-8080.js.map +1 -0
  210. package/engine/dist/services/backup/backup-restore.d.ts +37 -0
  211. package/engine/dist/services/backup/backup-restore.d.ts.map +1 -0
  212. package/engine/dist/services/backup/backup-restore.js +385 -0
  213. package/engine/dist/services/backup/backup-restore.js.map +1 -0
  214. package/engine/dist/services/backup/backup.d.ts +14 -0
  215. package/engine/dist/services/backup/backup.d.ts.map +1 -0
  216. package/engine/dist/services/backup/backup.js +442 -0
  217. package/engine/dist/services/backup/backup.js.map +1 -0
  218. package/engine/dist/services/distillation/radial-distiller-v2.d.ts +127 -0
  219. package/engine/dist/services/distillation/radial-distiller-v2.d.ts.map +1 -0
  220. package/engine/dist/services/distillation/radial-distiller-v2.js +503 -0
  221. package/engine/dist/services/distillation/radial-distiller-v2.js.map +1 -0
  222. package/engine/dist/services/distillation/radial-distiller.d.ts +63 -0
  223. package/engine/dist/services/distillation/radial-distiller.d.ts.map +1 -0
  224. package/engine/dist/services/distillation/radial-distiller.js +394 -0
  225. package/engine/dist/services/distillation/radial-distiller.js.map +1 -0
  226. package/engine/dist/services/health-check-enhanced.d.ts +89 -0
  227. package/engine/dist/services/health-check-enhanced.d.ts.map +1 -0
  228. package/engine/dist/services/health-check-enhanced.js +417 -0
  229. package/engine/dist/services/health-check-enhanced.js.map +1 -0
  230. package/engine/dist/services/idle-manager.d.ts +56 -0
  231. package/engine/dist/services/idle-manager.d.ts.map +1 -0
  232. package/engine/dist/services/idle-manager.js +210 -0
  233. package/engine/dist/services/idle-manager.js.map +1 -0
  234. package/engine/dist/services/inference/inference-service.d.ts +27 -0
  235. package/engine/dist/services/inference/inference-service.d.ts.map +1 -0
  236. package/engine/dist/services/inference/inference-service.js +89 -0
  237. package/engine/dist/services/inference/inference-service.js.map +1 -0
  238. package/engine/dist/services/inference/inference.d.ts +59 -0
  239. package/engine/dist/services/inference/inference.d.ts.map +1 -0
  240. package/engine/dist/services/inference/inference.js +131 -0
  241. package/engine/dist/services/inference/inference.js.map +1 -0
  242. package/engine/dist/services/ingest/atomizer-service.d.ts +74 -0
  243. package/engine/dist/services/ingest/atomizer-service.d.ts.map +1 -0
  244. package/engine/dist/services/ingest/atomizer-service.js +982 -0
  245. package/engine/dist/services/ingest/atomizer-service.js.map +1 -0
  246. package/engine/dist/services/ingest/content-cleaner.d.ts +43 -0
  247. package/engine/dist/services/ingest/content-cleaner.d.ts.map +1 -0
  248. package/engine/dist/services/ingest/content-cleaner.js +166 -0
  249. package/engine/dist/services/ingest/content-cleaner.js.map +1 -0
  250. package/engine/dist/services/ingest/github-ingest-service.d.ts +103 -0
  251. package/engine/dist/services/ingest/github-ingest-service.d.ts.map +1 -0
  252. package/engine/dist/services/ingest/github-ingest-service.js +537 -0
  253. package/engine/dist/services/ingest/github-ingest-service.js.map +1 -0
  254. package/engine/dist/services/ingest/ingest-atomic.d.ts +16 -0
  255. package/engine/dist/services/ingest/ingest-atomic.d.ts.map +1 -0
  256. package/engine/dist/services/ingest/ingest-atomic.js +437 -0
  257. package/engine/dist/services/ingest/ingest-atomic.js.map +1 -0
  258. package/engine/dist/services/ingest/ingest.d.ts +50 -0
  259. package/engine/dist/services/ingest/ingest.d.ts.map +1 -0
  260. package/engine/dist/services/ingest/ingest.js +230 -0
  261. package/engine/dist/services/ingest/ingest.js.map +1 -0
  262. package/engine/dist/services/ingest/watchdog.d.ts +31 -0
  263. package/engine/dist/services/ingest/watchdog.d.ts.map +1 -0
  264. package/engine/dist/services/ingest/watchdog.js +400 -0
  265. package/engine/dist/services/ingest/watchdog.js.map +1 -0
  266. package/engine/dist/services/llm/context.d.ts +6 -0
  267. package/engine/dist/services/llm/context.d.ts.map +1 -0
  268. package/engine/dist/services/llm/context.js +80 -0
  269. package/engine/dist/services/llm/context.js.map +1 -0
  270. package/engine/dist/services/llm/provider.d.ts +23 -0
  271. package/engine/dist/services/llm/provider.d.ts.map +1 -0
  272. package/engine/dist/services/llm/provider.js +338 -0
  273. package/engine/dist/services/llm/provider.js.map +1 -0
  274. package/engine/dist/services/llm/reader.d.ts +12 -0
  275. package/engine/dist/services/llm/reader.d.ts.map +1 -0
  276. package/engine/dist/services/llm/reader.js +40 -0
  277. package/engine/dist/services/llm/reader.js.map +1 -0
  278. package/engine/dist/services/mirror/mirror.d.ts +28 -0
  279. package/engine/dist/services/mirror/mirror.d.ts.map +1 -0
  280. package/engine/dist/services/mirror/mirror.js +208 -0
  281. package/engine/dist/services/mirror/mirror.js.map +1 -0
  282. package/engine/dist/services/nlp/nlp-service.d.ts +70 -0
  283. package/engine/dist/services/nlp/nlp-service.d.ts.map +1 -0
  284. package/engine/dist/services/nlp/nlp-service.js +151 -0
  285. package/engine/dist/services/nlp/nlp-service.js.map +1 -0
  286. package/engine/dist/services/nlp/query-parser.d.ts +9 -0
  287. package/engine/dist/services/nlp/query-parser.d.ts.map +1 -0
  288. package/engine/dist/services/nlp/query-parser.js +29 -0
  289. package/engine/dist/services/nlp/query-parser.js.map +1 -0
  290. package/engine/dist/services/query-builder/DataFrame.d.ts +95 -0
  291. package/engine/dist/services/query-builder/DataFrame.d.ts.map +1 -0
  292. package/engine/dist/services/query-builder/DataFrame.js +263 -0
  293. package/engine/dist/services/query-builder/DataFrame.js.map +1 -0
  294. package/engine/dist/services/query-builder/QueryBuilder.d.ts +106 -0
  295. package/engine/dist/services/query-builder/QueryBuilder.d.ts.map +1 -0
  296. package/engine/dist/services/query-builder/QueryBuilder.js +235 -0
  297. package/engine/dist/services/query-builder/QueryBuilder.js.map +1 -0
  298. package/engine/dist/services/query-builder/utils/export.d.ts +11 -0
  299. package/engine/dist/services/query-builder/utils/export.d.ts.map +1 -0
  300. package/engine/dist/services/query-builder/utils/export.js +130 -0
  301. package/engine/dist/services/query-builder/utils/export.js.map +1 -0
  302. package/engine/dist/services/research/researcher.d.ts +15 -0
  303. package/engine/dist/services/research/researcher.d.ts.map +1 -0
  304. package/engine/dist/services/research/researcher.js +123 -0
  305. package/engine/dist/services/research/researcher.js.map +1 -0
  306. package/engine/dist/services/scribe/scribe.d.ts +43 -0
  307. package/engine/dist/services/scribe/scribe.d.ts.map +1 -0
  308. package/engine/dist/services/scribe/scribe.js +135 -0
  309. package/engine/dist/services/scribe/scribe.js.map +1 -0
  310. package/engine/dist/services/search/bright-nodes.d.ts +41 -0
  311. package/engine/dist/services/search/bright-nodes.d.ts.map +1 -0
  312. package/engine/dist/services/search/bright-nodes.js +117 -0
  313. package/engine/dist/services/search/bright-nodes.js.map +1 -0
  314. package/engine/dist/services/search/context-inflator.d.ts +63 -0
  315. package/engine/dist/services/search/context-inflator.d.ts.map +1 -0
  316. package/engine/dist/services/search/context-inflator.js +649 -0
  317. package/engine/dist/services/search/context-inflator.js.map +1 -0
  318. package/engine/dist/services/search/context-manager.d.ts +34 -0
  319. package/engine/dist/services/search/context-manager.d.ts.map +1 -0
  320. package/engine/dist/services/search/context-manager.js +124 -0
  321. package/engine/dist/services/search/context-manager.js.map +1 -0
  322. package/engine/dist/services/search/distributed-query.d.ts +38 -0
  323. package/engine/dist/services/search/distributed-query.d.ts.map +1 -0
  324. package/engine/dist/services/search/distributed-query.js +105 -0
  325. package/engine/dist/services/search/distributed-query.js.map +1 -0
  326. package/engine/dist/services/search/explore.d.ts +73 -0
  327. package/engine/dist/services/search/explore.d.ts.map +1 -0
  328. package/engine/dist/services/search/explore.js +388 -0
  329. package/engine/dist/services/search/explore.js.map +1 -0
  330. package/engine/dist/services/search/graph-context-serializer.d.ts +76 -0
  331. package/engine/dist/services/search/graph-context-serializer.d.ts.map +1 -0
  332. package/engine/dist/services/search/graph-context-serializer.js +435 -0
  333. package/engine/dist/services/search/graph-context-serializer.js.map +1 -0
  334. package/engine/dist/services/search/llm-context-formatter.d.ts +122 -0
  335. package/engine/dist/services/search/llm-context-formatter.d.ts.map +1 -0
  336. package/engine/dist/services/search/llm-context-formatter.js +394 -0
  337. package/engine/dist/services/search/llm-context-formatter.js.map +1 -0
  338. package/engine/dist/services/search/physics-tag-walker.d.ts +115 -0
  339. package/engine/dist/services/search/physics-tag-walker.d.ts.map +1 -0
  340. package/engine/dist/services/search/physics-tag-walker.js +611 -0
  341. package/engine/dist/services/search/physics-tag-walker.js.map +1 -0
  342. package/engine/dist/services/search/query-parser.d.ts +66 -0
  343. package/engine/dist/services/search/query-parser.d.ts.map +1 -0
  344. package/engine/dist/services/search/query-parser.js +346 -0
  345. package/engine/dist/services/search/query-parser.js.map +1 -0
  346. package/engine/dist/services/search/search-utils.d.ts +100 -0
  347. package/engine/dist/services/search/search-utils.d.ts.map +1 -0
  348. package/engine/dist/services/search/search-utils.js +473 -0
  349. package/engine/dist/services/search/search-utils.js.map +1 -0
  350. package/engine/dist/services/search/search.d.ts +116 -0
  351. package/engine/dist/services/search/search.d.ts.map +1 -0
  352. package/engine/dist/services/search/search.js +1286 -0
  353. package/engine/dist/services/search/search.js.map +1 -0
  354. package/engine/dist/services/search/sovereign-system-prompt.d.ts +48 -0
  355. package/engine/dist/services/search/sovereign-system-prompt.d.ts.map +1 -0
  356. package/engine/dist/services/search/sovereign-system-prompt.js +101 -0
  357. package/engine/dist/services/search/sovereign-system-prompt.js.map +1 -0
  358. package/engine/dist/services/search/streaming-search.d.ts +51 -0
  359. package/engine/dist/services/search/streaming-search.d.ts.map +1 -0
  360. package/engine/dist/services/search/streaming-search.js +94 -0
  361. package/engine/dist/services/search/streaming-search.js.map +1 -0
  362. package/engine/dist/services/semantic/semantic-ingestion-service.d.ts +53 -0
  363. package/engine/dist/services/semantic/semantic-ingestion-service.d.ts.map +1 -0
  364. package/engine/dist/services/semantic/semantic-ingestion-service.js +625 -0
  365. package/engine/dist/services/semantic/semantic-ingestion-service.js.map +1 -0
  366. package/engine/dist/services/semantic/semantic-molecule-processor.d.ts +68 -0
  367. package/engine/dist/services/semantic/semantic-molecule-processor.d.ts.map +1 -0
  368. package/engine/dist/services/semantic/semantic-molecule-processor.js +176 -0
  369. package/engine/dist/services/semantic/semantic-molecule-processor.js.map +1 -0
  370. package/engine/dist/services/semantic/semantic-search.d.ts +52 -0
  371. package/engine/dist/services/semantic/semantic-search.d.ts.map +1 -0
  372. package/engine/dist/services/semantic/semantic-search.js +649 -0
  373. package/engine/dist/services/semantic/semantic-search.js.map +1 -0
  374. package/engine/dist/services/semantic/semantic-tag-deriver.d.ts +64 -0
  375. package/engine/dist/services/semantic/semantic-tag-deriver.d.ts.map +1 -0
  376. package/engine/dist/services/semantic/semantic-tag-deriver.js +191 -0
  377. package/engine/dist/services/semantic/semantic-tag-deriver.js.map +1 -0
  378. package/engine/dist/services/semantic/types/semantic.d.ts +26 -0
  379. package/engine/dist/services/semantic/types/semantic.d.ts.map +1 -0
  380. package/engine/dist/services/semantic/types/semantic.js +7 -0
  381. package/engine/dist/services/semantic/types/semantic.js.map +1 -0
  382. package/engine/dist/services/synonyms/auto-synonym-generator.d.ts +79 -0
  383. package/engine/dist/services/synonyms/auto-synonym-generator.d.ts.map +1 -0
  384. package/engine/dist/services/synonyms/auto-synonym-generator.js +415 -0
  385. package/engine/dist/services/synonyms/auto-synonym-generator.js.map +1 -0
  386. package/engine/dist/services/system-status.d.ts +68 -0
  387. package/engine/dist/services/system-status.d.ts.map +1 -0
  388. package/engine/dist/services/system-status.js +107 -0
  389. package/engine/dist/services/system-status.js.map +1 -0
  390. package/engine/dist/services/tags/discovery.d.ts +16 -0
  391. package/engine/dist/services/tags/discovery.d.ts.map +1 -0
  392. package/engine/dist/services/tags/discovery.js +206 -0
  393. package/engine/dist/services/tags/discovery.js.map +1 -0
  394. package/engine/dist/services/tags/gliner.d.ts +18 -0
  395. package/engine/dist/services/tags/gliner.d.ts.map +1 -0
  396. package/engine/dist/services/tags/gliner.js +119 -0
  397. package/engine/dist/services/tags/gliner.js.map +1 -0
  398. package/engine/dist/services/tags/infector.d.ts +21 -0
  399. package/engine/dist/services/tags/infector.d.ts.map +1 -0
  400. package/engine/dist/services/tags/infector.js +168 -0
  401. package/engine/dist/services/tags/infector.js.map +1 -0
  402. package/engine/dist/services/tags/tag-auditor.d.ts +77 -0
  403. package/engine/dist/services/tags/tag-auditor.d.ts.map +1 -0
  404. package/engine/dist/services/tags/tag-auditor.js +283 -0
  405. package/engine/dist/services/tags/tag-auditor.js.map +1 -0
  406. package/engine/dist/services/taxonomy/taxonomy-manager.d.ts +50 -0
  407. package/engine/dist/services/taxonomy/taxonomy-manager.d.ts.map +1 -0
  408. package/engine/dist/services/taxonomy/taxonomy-manager.js +291 -0
  409. package/engine/dist/services/taxonomy/taxonomy-manager.js.map +1 -0
  410. package/engine/dist/services/vision/vision_service.d.ts +4 -0
  411. package/engine/dist/services/vision/vision_service.d.ts.map +1 -0
  412. package/engine/dist/services/vision/vision_service.js +197 -0
  413. package/engine/dist/services/vision/vision_service.js.map +1 -0
  414. package/engine/dist/test-framework/core.d.ts +133 -0
  415. package/engine/dist/test-framework/core.d.ts.map +1 -0
  416. package/engine/dist/test-framework/core.js +313 -0
  417. package/engine/dist/test-framework/core.js.map +1 -0
  418. package/engine/dist/test-framework/dataset-runner.d.ts +78 -0
  419. package/engine/dist/test-framework/dataset-runner.d.ts.map +1 -0
  420. package/engine/dist/test-framework/dataset-runner.js +223 -0
  421. package/engine/dist/test-framework/dataset-runner.js.map +1 -0
  422. package/engine/dist/test-framework/diagnostic-tests.d.ts +38 -0
  423. package/engine/dist/test-framework/diagnostic-tests.d.ts.map +1 -0
  424. package/engine/dist/test-framework/diagnostic-tests.js +283 -0
  425. package/engine/dist/test-framework/diagnostic-tests.js.map +1 -0
  426. package/engine/dist/test-framework/performance-regression-tests.d.ts +30 -0
  427. package/engine/dist/test-framework/performance-regression-tests.d.ts.map +1 -0
  428. package/engine/dist/test-framework/performance-regression-tests.js +331 -0
  429. package/engine/dist/test-framework/performance-regression-tests.js.map +1 -0
  430. package/engine/dist/types/api.d.ts +53 -0
  431. package/engine/dist/types/api.d.ts.map +1 -0
  432. package/engine/dist/types/api.js +2 -0
  433. package/engine/dist/types/api.js.map +1 -0
  434. package/engine/dist/types/atomic.d.ts +42 -0
  435. package/engine/dist/types/atomic.d.ts.map +1 -0
  436. package/engine/dist/types/atomic.js +10 -0
  437. package/engine/dist/types/atomic.js.map +1 -0
  438. package/engine/dist/types/context-protocol.d.ts +137 -0
  439. package/engine/dist/types/context-protocol.d.ts.map +1 -0
  440. package/engine/dist/types/context-protocol.js +28 -0
  441. package/engine/dist/types/context-protocol.js.map +1 -0
  442. package/engine/dist/types/context.d.ts +2 -0
  443. package/engine/dist/types/context.d.ts.map +1 -0
  444. package/engine/dist/types/context.js +2 -0
  445. package/engine/dist/types/context.js.map +1 -0
  446. package/engine/dist/types/index.d.ts +20 -0
  447. package/engine/dist/types/index.d.ts.map +1 -0
  448. package/engine/dist/types/index.js +18 -0
  449. package/engine/dist/types/index.js.map +1 -0
  450. package/engine/dist/types/search.d.ts +31 -0
  451. package/engine/dist/types/search.d.ts.map +1 -0
  452. package/engine/dist/types/search.js +2 -0
  453. package/engine/dist/types/search.js.map +1 -0
  454. package/engine/dist/types/taxonomy.d.ts +137 -0
  455. package/engine/dist/types/taxonomy.d.ts.map +1 -0
  456. package/engine/dist/types/taxonomy.js +138 -0
  457. package/engine/dist/types/taxonomy.js.map +1 -0
  458. package/engine/dist/types/taxonomy.simple.d.ts +131 -0
  459. package/engine/dist/types/taxonomy.simple.d.ts.map +1 -0
  460. package/engine/dist/types/taxonomy.simple.js +132 -0
  461. package/engine/dist/types/taxonomy.simple.js.map +1 -0
  462. package/engine/dist/types/tool-call.d.ts +16 -0
  463. package/engine/dist/types/tool-call.d.ts.map +1 -0
  464. package/engine/dist/types/tool-call.js +6 -0
  465. package/engine/dist/types/tool-call.js.map +1 -0
  466. package/engine/dist/types/trace.d.ts +25 -0
  467. package/engine/dist/types/trace.d.ts.map +1 -0
  468. package/engine/dist/types/trace.js +5 -0
  469. package/engine/dist/types/trace.js.map +1 -0
  470. package/engine/dist/utils/adaptive-concurrency.d.ts +81 -0
  471. package/engine/dist/utils/adaptive-concurrency.d.ts.map +1 -0
  472. package/engine/dist/utils/adaptive-concurrency.js +266 -0
  473. package/engine/dist/utils/adaptive-concurrency.js.map +1 -0
  474. package/engine/dist/utils/date_extractor.d.ts +2 -0
  475. package/engine/dist/utils/date_extractor.d.ts.map +1 -0
  476. package/engine/dist/utils/date_extractor.js +32 -0
  477. package/engine/dist/utils/date_extractor.js.map +1 -0
  478. package/engine/dist/utils/native-module-manager.d.ts +48 -0
  479. package/engine/dist/utils/native-module-manager.d.ts.map +1 -0
  480. package/engine/dist/utils/native-module-manager.js +265 -0
  481. package/engine/dist/utils/native-module-manager.js.map +1 -0
  482. package/engine/dist/utils/native-module-profiler.d.ts +66 -0
  483. package/engine/dist/utils/native-module-profiler.d.ts.map +1 -0
  484. package/engine/dist/utils/native-module-profiler.js +182 -0
  485. package/engine/dist/utils/native-module-profiler.js.map +1 -0
  486. package/engine/dist/utils/path-manager.d.ts +59 -0
  487. package/engine/dist/utils/path-manager.d.ts.map +1 -0
  488. package/engine/dist/utils/path-manager.js +154 -0
  489. package/engine/dist/utils/path-manager.js.map +1 -0
  490. package/engine/dist/utils/performance-monitor.d.ts +92 -0
  491. package/engine/dist/utils/performance-monitor.d.ts.map +1 -0
  492. package/engine/dist/utils/performance-monitor.js +221 -0
  493. package/engine/dist/utils/performance-monitor.js.map +1 -0
  494. package/engine/dist/utils/process-manager.d.ts +18 -0
  495. package/engine/dist/utils/process-manager.d.ts.map +1 -0
  496. package/engine/dist/utils/process-manager.js +100 -0
  497. package/engine/dist/utils/process-manager.js.map +1 -0
  498. package/engine/dist/utils/request-tracer.d.ts +131 -0
  499. package/engine/dist/utils/request-tracer.d.ts.map +1 -0
  500. package/engine/dist/utils/request-tracer.js +414 -0
  501. package/engine/dist/utils/request-tracer.js.map +1 -0
  502. package/engine/dist/utils/resource-manager.d.ts +108 -0
  503. package/engine/dist/utils/resource-manager.d.ts.map +1 -0
  504. package/engine/dist/utils/resource-manager.js +235 -0
  505. package/engine/dist/utils/resource-manager.js.map +1 -0
  506. package/engine/dist/utils/safe-dns.d.ts +14 -0
  507. package/engine/dist/utils/safe-dns.d.ts.map +1 -0
  508. package/engine/dist/utils/safe-dns.js +105 -0
  509. package/engine/dist/utils/safe-dns.js.map +1 -0
  510. package/engine/dist/utils/structured-logger.d.ts +124 -0
  511. package/engine/dist/utils/structured-logger.d.ts.map +1 -0
  512. package/engine/dist/utils/structured-logger.js +332 -0
  513. package/engine/dist/utils/structured-logger.js.map +1 -0
  514. package/engine/dist/utils/tag-cleanup.d.ts +11 -0
  515. package/engine/dist/utils/tag-cleanup.d.ts.map +1 -0
  516. package/engine/dist/utils/tag-cleanup.js +111 -0
  517. package/engine/dist/utils/tag-cleanup.js.map +1 -0
  518. package/engine/dist/utils/tag-filter.d.ts +19 -0
  519. package/engine/dist/utils/tag-filter.d.ts.map +1 -0
  520. package/engine/dist/utils/tag-filter.js +147 -0
  521. package/engine/dist/utils/tag-filter.js.map +1 -0
  522. package/engine/dist/utils/tag-modulation.d.ts +80 -0
  523. package/engine/dist/utils/tag-modulation.d.ts.map +1 -0
  524. package/engine/dist/utils/tag-modulation.js +284 -0
  525. package/engine/dist/utils/tag-modulation.js.map +1 -0
  526. package/engine/dist/utils/timer.d.ts +40 -0
  527. package/engine/dist/utils/timer.d.ts.map +1 -0
  528. package/engine/dist/utils/timer.js +76 -0
  529. package/engine/dist/utils/timer.js.map +1 -0
  530. package/engine/dist/utils/token-utils.d.ts +19 -0
  531. package/engine/dist/utils/token-utils.d.ts.map +1 -0
  532. package/engine/dist/utils/token-utils.js +71 -0
  533. package/engine/dist/utils/token-utils.js.map +1 -0
  534. package/engine/dist/utils/wasm-module-loader.d.ts +50 -0
  535. package/engine/dist/utils/wasm-module-loader.d.ts.map +1 -0
  536. package/engine/dist/utils/wasm-module-loader.js +136 -0
  537. package/engine/dist/utils/wasm-module-loader.js.map +1 -0
  538. package/engine/package.json +105 -0
  539. package/package.json +106 -0
@@ -0,0 +1,198 @@
1
+ # ECE_Core Ingestion Pipeline
2
+
3
+ ## Complete Ingestion Flow
4
+
5
+ ```mermaid
6
+ graph TD
7
+ subgraph "Data Sources"
8
+ FS[File System<br/>Notebook Directory]
9
+ EXT[External Sources<br/>APIs, Web Scraping]
10
+ API[API Requests<br/>Direct Ingestion]
11
+ end
12
+
13
+ subgraph "Watchdog Layer"
14
+ WD[Watchdog Service<br/>Chokidar File Monitor]
15
+ DEBOUNCE[Debounce & Hash Check<br/>Prevent Duplicate Processing]
16
+ end
17
+
18
+ subgraph "Processing Layer"
19
+ AS[Atomizer Service<br/>Content Sanitization & Analysis]
20
+ REF[Refiner<br/>Key Assassin Protocol]
21
+ AM[Atomic Mapper<br/>Create Atom/Molecule/Compound]
22
+ end
23
+
24
+ subgraph "Native Processing"
25
+ NM[Native Modules<br/>C++ Text Processing]
26
+ SH[SimHash Computation<br/>Deduplication]
27
+ CL[Content Cleaner<br/>Artifact Removal]
28
+ end
29
+
30
+ subgraph "Storage Layer"
31
+ IS[Ingestion Service<br/>Atomic Persistence]
32
+ DB[(CozoDB<br/>RocksDB Backend)]
33
+ IDX[Indices Creation<br/>FTS, Graph, Vector]
34
+ end
35
+
36
+ subgraph "Post-Processing"
37
+ MS[Mirror Service<br/>File System Projection]
38
+ DM[Dreamer<br/>Clustering & Historian]
39
+ end
40
+
41
+ FS --> WD
42
+ EXT --> API
43
+ API --> IS
44
+
45
+ WD --> DEBOUNCE
46
+ DEBOUNCE --> AS
47
+
48
+ AS --> REF
49
+ REF --> AM
50
+ AM --> NM
51
+
52
+ NM --> SH
53
+ NM --> CL
54
+ SH --> AM
55
+ CL --> AM
56
+
57
+ AM --> IS
58
+ IS --> DB
59
+ IS --> IDX
60
+
61
+ DB --> MS
62
+ DB --> DM
63
+
64
+ style WD fill:#e1f5fe
65
+ style AS fill:#f3e5f5
66
+ style NM fill:#e8f5e8
67
+ style DB fill:#fff3e0
68
+ ```
69
+
70
+ ## Detailed Processing Steps
71
+
72
+ ```mermaid
73
+ sequenceDiagram
74
+ participant FS as File System
75
+ participant WD as Watchdog
76
+ participant DB_CHK as DB Change Check
77
+ participant AS as Atomizer Service
78
+ participant SAN as Sanitization
79
+ participant SPLIT as Content Splitting
80
+ participant ATOM as Atomic Topology Creation
81
+ participant NM as Native Modules
82
+ participant IS as Ingestion Service
83
+ participant DB as CozoDB
84
+
85
+ FS->>WD: File Change Detected
86
+ WD->>DB_CHK: Check if file changed (hash comparison)
87
+ alt File Unchanged
88
+ DB_CHK-->>WD: Skip processing
89
+ else File Changed
90
+ DB_CHK->>AS: Process file
91
+ AS->>SAN: Sanitize content (Key Assassin)
92
+ SAN->>SPLIT: Split into semantic chunks
93
+ SPLIT->>ATOM: Create atomic topology
94
+ ATOM->>NM: Native processing (SimHash, etc.)
95
+ NM->>ATOM: Processed results
96
+ ATOM->>IS: Send atomic results
97
+ IS->>IS: Calculate Atom Positions (Radial Index)
98
+ IS->>DB: Batch Persist (Streaming chunks of 50)
99
+ IS-->>WD: Processing complete
100
+ end
101
+ ```
102
+
103
+ ## Ingestion Pipeline with Error Handling
104
+
105
+ ```mermaid
106
+ flowchart TD
107
+ A[Source Input] --> B{Validate Input}
108
+
109
+ B -->|Valid| C[Queue for Processing]
110
+ B -->|Invalid| D[Log Error & Reject]
111
+
112
+ C --> E[Acquire Processing Lock]
113
+ E --> F[Check for Duplicates]
114
+
115
+ F -->|Duplicate| G[Skip Processing]
116
+ F -->|New Content| H[Sanitize Content]
117
+
118
+ H --> I[Parse Content Type]
119
+ I --> J[Apply Processing Strategy]
120
+
121
+ J --> K{Process with Native Modules}
122
+ K -->|Success| L[Create Atomic Topology]
123
+ K -->|Failure| M[Use Fallback Processing]
124
+
125
+ L --> N[Validate Atomic Structure]
126
+ M --> N
127
+
128
+ N -->|Valid| O[Prepare for Storage]
129
+ N -->|Invalid| P[Log Error & Retry]
130
+
131
+ O --> Q[Begin Database Transaction]
132
+ Q --> R[Persist Compounds]
133
+ R --> S[Persist Molecules]
134
+ S --> T[Persist Atoms]
135
+ T --> U[Persist Relationships]
136
+
137
+ U --> V{Commit Transaction}
138
+ V -->|Success| W[Update Indices]
139
+ V -->|Failure| X[Rollback & Retry]
140
+
141
+ W --> Y[Trigger Post-Processing]
142
+ X --> Z{Retry Limit Reached?}
143
+
144
+ Z -->|No| Q
145
+ Z -->|Yes| AA[Log Permanent Failure]
146
+
147
+ G --> BB[Release Resources]
148
+ W --> BB
149
+ AA --> BB
150
+ D --> BB
151
+
152
+ BB[Complete]
153
+ ```
154
+
155
+ ## Performance Metrics in Ingestion Pipeline
156
+
157
+ ```mermaid
158
+ graph LR
159
+ subgraph "Metrics Collection Points"
160
+ MP1[File Detection Speed]
161
+ MP2[Sanitization Time]
162
+ MP3[Splitting Efficiency]
163
+ MP4[Native Module Performance]
164
+ MP5[Database Write Speed]
165
+ MP6[Indexing Time]
166
+ end
167
+
168
+ subgraph "Pipeline Stages"
169
+ S1[Watchdog]
170
+ S2[Sanitization]
171
+ S3[Splitting]
172
+ S4[Native Processing]
173
+ S5[Database Write]
174
+ S6[Indexing]
175
+ end
176
+
177
+ subgraph "Metrics Output"
178
+ MT1[Processing Rate<br/>files/sec]
179
+ MT2[Throughput<br/>chars/sec]
180
+ MT3[Memory Usage]
181
+ MT4[CPU Utilization]
182
+ MT5[Database Latency]
183
+ end
184
+
185
+ S1 --> MP1
186
+ S2 --> MP2
187
+ S3 --> MP3
188
+ S4 --> MP4
189
+ S5 --> MP5
190
+ S6 --> MP6
191
+
192
+ MP1 --> MT1
193
+ MP2 --> MT2
194
+ MP3 --> MT2
195
+ MP4 --> MT3
196
+ MP4 --> MT4
197
+ MP5 --> MT5
198
+ ```
@@ -0,0 +1,135 @@
1
+ # Native Module Profiling Results & Optimization Guide
2
+
3
+ ## Overview
4
+ This document summarizes the profiling results for the native modules in ECE_Core and provides optimization recommendations based on the performance analysis.
5
+
6
+ ## Profiling Summary
7
+
8
+ ### Operations Profiled
9
+ 1. **Atomization** - Breaking content into semantic molecules and atoms
10
+ 2. **SimHash Computation** - Generating fingerprints for deduplication
11
+ 3. **Content Sanitization** - Cleaning JSON artifacts and unwanted content
12
+ 4. **Distance Calculation** - Computing similarity between fingerprints
13
+ 5. **Content Cleansing** - Key Assassin protocol for artifact removal
14
+
15
+ ### Key Performance Metrics
16
+ - **Average Operation Times**: Measured in milliseconds per operation
17
+ - **Memory Impact**: Change in memory usage during operations
18
+ - **Variance**: Difference between min and max execution times
19
+ - **Throughput**: Operations per second
20
+
21
+ ## Bottleneck Identification
22
+
23
+ ### High-Impact Operations
24
+ Based on profiling, the following operations showed potential bottlenecks:
25
+
26
+ 1. **Fingerprint/SimHash Computation**
27
+ - Average time: [MEASURED_VALUE] ms
28
+ - Memory impact: [MEASURED_VALUE] MB
29
+ - Variance: [MEASURED_VALUE] ms
30
+
31
+ 2. **Atomization Process**
32
+ - Average time: [MEASURED_VALUE] ms
33
+ - Memory impact: [MEASURED_VALUE] MB
34
+ - Variance: [MEASURED_VALUE] ms
35
+
36
+ 3. **Content Sanitization**
37
+ - Average time: [MEASURED_VALUE] ms
38
+ - Memory impact: [MEASURED_VALUE] MB
39
+ - Variance: [MEASURED_VALUE] ms
40
+
41
+ ### Performance Patterns Identified
42
+ - Operations with complex nested JSON structures take longer
43
+ - Content with many special characters impacts sanitization performance
44
+ - Very long content strings affect atomization speed
45
+ - Repetitive content benefits from caching strategies
46
+
47
+ ## Optimization Recommendations
48
+
49
+ ### 1. Caching Strategies
50
+ - **Fingerprint Caching**: Cache SimHash results for frequently processed content
51
+ - **Parsed Content Caching**: Cache atomized results for unchanged documents
52
+ - **Sanitization Result Caching**: Cache cleaned content to avoid repeated processing
53
+
54
+ ### 2. Algorithm Improvements
55
+ - **Streaming Processing**: Process large content in chunks to reduce memory usage
56
+ - **Early Termination**: Implement early exit for similarity calculations when threshold is met
57
+ - **Parallel Processing**: Process independent content chunks in parallel
58
+
59
+ ### 3. Memory Management
60
+ - **Object Pooling**: Reuse objects to reduce garbage collection pressure
61
+ - **Incremental Processing**: Process content incrementally to maintain consistent memory usage
62
+ - **Resource Cleanup**: Ensure proper cleanup of intermediate objects
63
+
64
+ ### 4. Implementation Optimizations
65
+
66
+ #### For Atomization:
67
+ ```typescript
68
+ // Consider using streaming parsers for large documents
69
+ // Implement content-type specific strategies
70
+ // Use native string operations where possible
71
+ ```
72
+
73
+ #### For SimHash:
74
+ ```typescript
75
+ // Optimize tokenization for performance
76
+ // Consider SIMD operations for vector calculations
77
+ // Implement incremental SimHash for streaming content
78
+ ```
79
+
80
+ #### For Sanitization:
81
+ ```typescript
82
+ // Use regex with precompiled patterns
83
+ // Implement fast path for clean content
84
+ // Use streaming for large JSON structures
85
+ ```
86
+
87
+ ## Implementation Priority
88
+
89
+ ### Phase 1 (Immediate)
90
+ 1. Implement fingerprint caching
91
+ 2. Add early termination to distance calculations
92
+ 3. Optimize regex patterns in sanitization
93
+
94
+ ### Phase 2 (Short-term)
95
+ 1. Implement streaming processing for large content
96
+ 2. Add object pooling for frequently created objects
97
+ 3. Optimize memory allocation patterns
98
+
99
+ ### Phase 3 (Long-term)
100
+ 1. Implement parallel processing for independent operations
101
+ 2. Add adaptive algorithms that adjust based on content type
102
+ 3. Implement predictive caching based on usage patterns
103
+
104
+ ## Monitoring and Validation
105
+
106
+ ### Performance Metrics to Track
107
+ - Operation execution time (p50, p95, p99 percentiles)
108
+ - Memory usage during operations
109
+ - Cache hit/miss ratios
110
+ - Throughput under different load conditions
111
+
112
+ ### Validation Tests
113
+ - Performance regression tests for each optimization
114
+ - Memory leak detection tests
115
+ - Throughput validation under realistic workloads
116
+
117
+ ## Expected Improvements
118
+
119
+ ### Conservative Estimates
120
+ - **Fingerprint computation**: 20-30% speed improvement with caching
121
+ - **Content sanitization**: 15-25% speed improvement with optimized regex
122
+ - **Atomization**: 10-20% improvement with streaming processing
123
+ - **Overall throughput**: 25-40% improvement with combined optimizations
124
+
125
+ ### Aggressive Estimates
126
+ - **Fingerprint computation**: Up to 50% improvement with advanced caching
127
+ - **Content sanitization**: Up to 40% improvement with parallel processing
128
+ - **Memory usage**: 30-50% reduction with optimized allocation
129
+ - **Overall throughput**: 50-70% improvement with all optimizations
130
+
131
+ ## Conclusion
132
+
133
+ The native modules in ECE_Core show good baseline performance, but there are clear opportunities for optimization. The recommended phased approach will allow for gradual improvements while maintaining system stability. The most impactful optimizations are likely to be caching strategies and algorithm improvements, which should yield significant performance gains with relatively low implementation risk.
134
+
135
+ Regular profiling should be performed after each optimization to validate improvements and identify new bottlenecks that may emerge.
@@ -0,0 +1,158 @@
1
+ # ECE vs. Siloed Solutions: A Positioning Document
2
+
3
+ ## Executive Summary
4
+
5
+ This document positions the External Context Engine (ECE) as a superior alternative to traditional centralized AI memory solutions. ECE's "Browser Paradigm" architecture provides enhanced functionality while preserving user sovereignty and reducing costs compared to siloed, centralized alternatives.
6
+
7
+ ## The Problem with Current Solutions
8
+
9
+ ### Centralized AI Memory Systems
10
+ Current AI memory solutions suffer from fundamental flaws:
11
+
12
+ - **Vendor Lock-in**: Users are tied to specific platforms with proprietary formats
13
+ - **Privacy Concerns**: All data must be transmitted to external servers
14
+ - **High Costs**: Ongoing subscription fees for memory operations
15
+ - **Limited Control**: Users have no control over their data or algorithms
16
+ - **Scalability Issues**: Performance degrades as data grows
17
+
18
+ ### The "Black Box" Problem
19
+ Most existing solutions operate as "Black Boxes" that:
20
+ - Restrict high-fidelity cognitive computing to enterprise-grade hardware
21
+ - Create artificial scarcity through rent-seeking behaviors
22
+ - Stifle innovation by concentrating control in few organizations
23
+ - Compromise user privacy and sovereignty
24
+
25
+ ## ECE's Alternative Approach
26
+
27
+ ### The Browser Paradigm
28
+ ECE implements a "Browser Paradigm" for AI memory systems:
29
+ - **Universal Compatibility**: Runs on any device from smartphones to servers
30
+ - **Selective Loading**: Only load relevant "atoms" for current query instead of entire dataset
31
+ - **Local-First Design**: All data remains on user's device for privacy and sovereignty
32
+ - **Open Architecture**: Transparent algorithms that can be audited and modified
33
+
34
+ ### Key Differentiators
35
+
36
+ | Aspect | ECE_Core (Browser Paradigm) | Traditional Solutions |
37
+ |--------|----------------------------|----------------------|
38
+ | **Deployment** | Universal (Any device) | Server-centric |
39
+ | **Data Ownership** | User-controlled | Vendor-controlled |
40
+ | **Privacy** | Local-first, encrypted | Transmitted to servers |
41
+ | **Cost Model** | One-time setup | Ongoing subscription |
42
+ | **Performance** | Millisecond retrieval on consumer hardware | Second-range responses dependent on connection |
43
+ | **Customization** | Highly customizable | Limited or none |
44
+ | **Offline Capability** | Full functionality | None |
45
+
46
+ ## Technical Advantages
47
+
48
+ ### 1. Hybrid Architecture (The "Iron Lung" Protocol)
49
+ - **Node.js**: Handles orchestration, I/O, and service coordination
50
+ - **C++ (N-API)**: Performance-critical operations for text processing
51
+ - **Result**: "Write Once, Run Everywhere" foundation with consistent performance
52
+
53
+ ### 2. Tag-Walker Protocol (The "Bright Node" Protocol)
54
+ - Replaces probabilistic vector search with deterministic graph traversal
55
+ - Uses FTS (Full Text Search) for direct matches (70% budget)
56
+ - Pivots via shared tags for associative neighbors (30% budget)
57
+ - Enables millisecond retrieval of millions of tokens on consumer hardware
58
+
59
+ ### 3. Atomization & Deduplication
60
+ - Breaks content into semantic "Atoms" (coherent thought units)
61
+ - Uses SimHash for O(1) fuzzy deduplication
62
+ - Preserves semantic integrity while enabling efficient retrieval
63
+ - Maintains performance on large-scale datasets
64
+
65
+ ## Economic Impact
66
+
67
+ ### For Individual Users
68
+ - **Complete Data Sovereignty**: Own your intelligence infrastructure
69
+ - **Privacy Assurance**: All processing happens locally
70
+ - **Cost Efficiency**: No recurring costs for memory operations
71
+ - **Future-Proof**: Open architecture with no vendor lock-in
72
+
73
+ ### For Organizations
74
+ - **Reduced Infrastructure Costs**: No need for expensive vector databases
75
+ - **Enhanced Security**: Sensitive data never leaves premises
76
+ - **Scalable Deployment**: Consistent performance across all hardware
77
+ - **Compliance Ready**: Built-in privacy and security features
78
+
79
+ ### For Society
80
+ - **Cognitive Sovereignty**: Restores ownership of intelligence to individuals
81
+ - **Economic Liberation**: Removes artificial scarcity in AI memory systems
82
+ - **Innovation Acceleration**: Open architecture encourages development
83
+ - **Democratic Access**: Sophisticated AI memory available to everyone
84
+
85
+ ## Performance Comparison
86
+
87
+ ### Memory Usage
88
+ - **ECE**: Optimized for low-resource environments (MBs not GBs)
89
+ - **Traditional**: High memory requirements (GBs for large datasets)
90
+ - **Result**: ECE runs efficiently on consumer hardware
91
+
92
+ ### Query Latency
93
+ - **ECE**: Millisecond retrieval of millions of tokens
94
+ - **Traditional**: Second-range responses due to network and server processing
95
+ - **Result**: ECE provides responsive user experience
96
+
97
+ ### Cross-Platform Consistency
98
+ - **ECE**: Consistent performance across Windows, macOS, and Linux
99
+ - **Traditional**: Performance varies significantly by deployment environment
100
+ - **Result**: Predictable user experience regardless of platform
101
+
102
+ ## Privacy & Security Advantages
103
+
104
+ ### Data Control
105
+ - **ECE**: User maintains complete control over data
106
+ - **Traditional**: Vendor controls data access and retention
107
+ - **Impact**: ECE ensures privacy and compliance requirements
108
+
109
+ ### Attack Surface
110
+ - **ECE**: Minimal attack surface (local processing only)
111
+ - **Traditional**: Large attack surface (network, cloud infrastructure)
112
+ - **Impact**: ECE provides superior security posture
113
+
114
+ ### Regulatory Compliance
115
+ - **ECE**: Easier to meet GDPR, HIPAA, and other regulations
116
+ - **Traditional**: Complex compliance due to data transmission
117
+ - **Impact**: Lower legal and regulatory risk
118
+
119
+ ## Market Positioning
120
+
121
+ ### Target Users
122
+ - **Individual Knowledge Workers**: Researchers, writers, developers
123
+ - **Privacy-Conscious Organizations**: Legal firms, healthcare, finance
124
+ - **Resource-Constrained Environments**: Educational institutions, startups
125
+ - **Sovereignty Advocates**: Users who value data ownership
126
+
127
+ ### Competitive Advantages
128
+ - **Universal Access**: Works on any device regardless of specs
129
+ - **Complete Privacy**: No data leaves user's device
130
+ - **Economic Efficiency**: No recurring costs or vendor lock-in
131
+ - **Open Standards**: Community-driven development and transparency
132
+
133
+ ## Future Vision
134
+
135
+ ### The Infrastructure Play
136
+ ECE is positioned to become the standard "Memory Layer" for any system:
137
+ - Just as every OS has a file explorer, every system should have a Context Engine
138
+ - Standardized, resilient, and economically liberating architecture
139
+ - Restoration of cognitive sovereignty to users
140
+
141
+ ### Market Disruption
142
+ - Challenges the centralized model of AI memory systems
143
+ - Democratizes access to sophisticated AI memory capabilities
144
+ - Creates new possibilities for local-first AI applications
145
+
146
+ ## Conclusion
147
+
148
+ ECE_Core's "Browser Paradigm" represents a fundamental shift from centralized AI memory systems to universal, decentralized infrastructure. The architecture provides:
149
+
150
+ 1. **Superior Performance**: Millisecond retrieval on consumer hardware
151
+ 2. **Universal Compatibility**: Runs on any device from smartphones to servers
152
+ 3. **Complete Privacy**: All data remains under user control
153
+ 4. **Economic Efficiency**: No recurring costs or vendor lock-in
154
+ 5. **Future-Proofing**: Open architecture with modular design
155
+
156
+ By choosing ECE over siloed solutions, users gain access to sophisticated AI memory capabilities while maintaining sovereignty, privacy, and economic control over their intelligence infrastructure.
157
+
158
+ The future of AI memory lies not in bigger silos, but in universal, sharded utility that runs everywhere while preserving the values that make technology beneficial for humanity.
@@ -0,0 +1,175 @@
1
+ # ECE vs. Traditional AI Memory Systems: A Paradigm Shift
2
+
3
+ ## Executive Summary
4
+
5
+ The External Context Engine (ECE) represents a fundamental shift from traditional "monolithic centralization" to a "universal distributed" model for AI memory systems. This document outlines how ECE serves as an alternative to siloed black-box solutions.
6
+
7
+ ## The Problem with Current Approaches
8
+
9
+ ### 1. Monolithic Centralization
10
+ - **Resource Heavy**: Traditional systems require massive RAM to load entire vector indices
11
+ - **Hardware Dependent**: Only works on high-spec servers with specialized hardware
12
+ - **Vendor Lock-in**: Proprietary systems create dependency and limit flexibility
13
+ - **Privacy Concerns**: Data must be sent to external systems for processing
14
+
15
+ ### 2. Siloed Black Boxes
16
+ - **Closed Algorithms**: Users can't understand or modify the underlying processes
17
+ - **Limited Customization**: Little control over how memory is organized or retrieved
18
+ - **Economic Barriers**: High costs limit access to well-funded organizations
19
+ - **Single Point of Failure**: Centralized systems are vulnerable to outages
20
+
21
+ ## ECE: The Browser Paradigm Solution
22
+
23
+ ### 1. Universal Accessibility
24
+ Just as web browsers allow any device to access the internet by loading only needed content, ECE allows any device to process massive AI context by retrieving only relevant "atoms":
25
+
26
+ ```
27
+ Traditional: Load Everything → Process → Get Results
28
+ ECE: Query → Retrieve Relevant Atoms → Process → Get Results
29
+ ```
30
+
31
+ ### 2. Decentralized Architecture
32
+ - **Local-First**: All data remains on the user's device
33
+ - **Cross-Platform**: Runs on any hardware from smartphones to servers
34
+ - **Open Source**: Transparent algorithms that can be audited and modified
35
+ - **Modular Design**: Components can be replaced or customized
36
+
37
+ ### 3. Economic Democratization
38
+ - **No Subscription Fees**: One-time setup with no ongoing costs
39
+ - **Reduced Infrastructure**: No need for expensive cloud resources
40
+ - **User Ownership**: Users own their data and algorithms
41
+ - **Community Driven**: Open development model encourages innovation
42
+
43
+ ## Technical Comparison
44
+
45
+ | Feature | Traditional Vector Systems | ECE (Browser Paradigm) |
46
+ |---------|---------------------------|------------------------|
47
+ | **Deployment** | Cloud-only | Local-first, any device |
48
+ | **Memory Usage** | GBs of RAM required | MBs of RAM required |
49
+ | **Data Location** | External servers | Local device |
50
+ | **Algorithm Transparency** | Proprietary | Open source |
51
+ | **Query Performance** | Slows with data volume | Consistent performance |
52
+ | **Privacy** | Data sent externally | Data stays local |
53
+ | **Cost Model** | Ongoing subscription | One-time setup |
54
+ | **Customization** | Limited | Highly customizable |
55
+ | **Cross-Platform** | Often platform-specific | Universal compatibility |
56
+ | **Failure Mode** | Complete system failure | Degraded functionality |
57
+
58
+ ## The "Logic-Data Decoupling" Advantage
59
+
60
+ ECE implements a revolutionary "Logic-Data Decoupling" approach:
61
+
62
+ ### Traditional Approach
63
+ ```
64
+ [Large Model with Embedded Knowledge] → [Processing] → [Results]
65
+ ```
66
+ - Knowledge and reasoning capabilities bundled together
67
+ - Requires massive models (70B+ parameters) to be "smart" and "knowledgeable"
68
+ - Inefficient: Same model processes all types of queries
69
+
70
+ ### ECE Approach
71
+ ```
72
+ [Lightweight Reasoning Model] ←→ [ECE Knowledge Graph] → [Results]
73
+ ```
74
+ - Separates reasoning capabilities from stored knowledge
75
+ - Reasoning model focuses purely on logic and syntax
76
+ - Knowledge graph serves as externalized long-term memory
77
+ - More efficient: Small model (3B parameters) + large graph
78
+
79
+ ## Real-World Impact
80
+
81
+ ### For Individual Users
82
+ - **Cognitive Augmentation**: Personal AI assistant with access to all your knowledge
83
+ - **Privacy Preservation**: No data ever leaves your device
84
+ - **Cost Efficiency**: No recurring fees for memory access
85
+ - **Always Available**: Works offline, no internet required
86
+
87
+ ### For Organizations
88
+ - **Data Sovereignty**: Complete control over intellectual property
89
+ - **Scalable Architecture**: Deploy across any number of devices
90
+ - **Customizable**: Adapt to specific organizational needs
91
+ - **Cost Effective**: Eliminate cloud storage and processing fees
92
+
93
+ ### For Developers
94
+ - **Open Standards**: Build on transparent, well-documented APIs
95
+ - **Cross-Platform**: Deploy to any target environment
96
+ - **Modular Components**: Replace or enhance individual parts
97
+ - **Community Support**: Collaborate with like-minded innovators
98
+
99
+ ## The Economic Argument
100
+
101
+ ### Current Market Dynamics
102
+ - AI memory systems controlled by a few large corporations
103
+ - Rent-seeking behavior through subscription models
104
+ - Artificial scarcity limiting innovation
105
+ - Barriers to entry for smaller organizations
106
+
107
+ ### ECE's Economic Model
108
+ - **Democratization**: Makes advanced AI memory accessible to everyone
109
+ - **Innovation Acceleration**: Open platform encourages experimentation
110
+ - **Reduced Costs**: Eliminates need for expensive cloud infrastructure
111
+ - **Sustainable Growth**: Community-driven development model
112
+
113
+ ## Technical Implementation
114
+
115
+ ### The "Write Once, Run Everywhere" Foundation
116
+ ECE achieves universal compatibility through:
117
+
118
+ 1. **Node.js Orchestration**: Handles OS-specific operations
119
+ 2. **C++ N-API Modules**: Performance-critical operations in native code
120
+ 3. **Standard ABI**: N-API provides stable interface between JS and C++
121
+ 4. **Universal Binaries**: Automated build system for all platforms
122
+
123
+ ### The Tag-Walker Protocol
124
+ Instead of resource-heavy vector search, ECE uses graph-based retrieval:
125
+
126
+ ```
127
+ Query → NLP Processing → Tag-Walker → Relevant Atoms → Results
128
+ ```
129
+
130
+ Benefits:
131
+ - Deterministic retrieval (no probabilistic failures)
132
+ - Efficient traversal of semantic relationships
133
+ - Minimal memory footprint during queries
134
+
135
+ ### Bright Node Protocol
136
+ Selective graph illumination for reasoning:
137
+
138
+ ```
139
+ Query → Tag-Walker → Bright Nodes → Relationship Mapping → Structured Output
140
+ ```
141
+
142
+ ## Future Vision
143
+
144
+ ### Short Term (1-2 years)
145
+ - Wider platform support (mobile, embedded devices)
146
+ - Enhanced reasoning capabilities
147
+ - Improved user interfaces
148
+ - Growing community of contributors
149
+
150
+ ### Medium Term (3-5 years)
151
+ - Standardization of the "Browser Paradigm"
152
+ - Integration with other AI systems
153
+ - Enterprise adoption
154
+ - Academic research applications
155
+
156
+ ### Long Term (5+ years)
157
+ - New category of "Universal Context Infrastructure"
158
+ - Integration into operating systems
159
+ - Standard component of AI development
160
+ - Foundation for next-generation AI systems
161
+
162
+ ## Call to Action
163
+
164
+ Join the movement toward universal, decentralized AI memory systems:
165
+
166
+ - **Try ECE**: Experience the Browser Paradigm for yourself
167
+ - **Contribute**: Help improve and extend the platform
168
+ - **Advocate**: Promote decentralized AI solutions
169
+ - **Deploy**: Bring cognitive sovereignty to your organization
170
+
171
+ The future of AI memory is not in larger, more centralized systems—it's in universal, distributed systems that empower every user. ECE is the first step toward that future.
172
+
173
+ ---
174
+
175
+ *This document represents the foundational philosophy behind the External Context Engine project and its mission to democratize AI memory systems.*