omnichunk 2.1.2__tar.gz → 2.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (355) hide show
  1. {omnichunk-2.1.2 → omnichunk-2.2.0}/CHANGELOG.md +96 -1
  2. {omnichunk-2.1.2 → omnichunk-2.2.0}/CONTRIBUTING.md +8 -4
  3. {omnichunk-2.1.2 → omnichunk-2.2.0}/PKG-INFO +64 -21
  4. {omnichunk-2.1.2 → omnichunk-2.2.0}/README.md +51 -15
  5. omnichunk-2.2.0/ROADMAP.md +324 -0
  6. omnichunk-2.2.0/benchmarks/README.md +245 -0
  7. omnichunk-2.2.0/benchmarks/import_squad.py +239 -0
  8. omnichunk-2.2.0/benchmarks/prepare_dense.py +227 -0
  9. omnichunk-2.2.0/benchmarks/retrieval/__init__.py +1 -0
  10. omnichunk-2.2.0/benchmarks/retrieval/adapters.py +240 -0
  11. omnichunk-2.2.0/benchmarks/retrieval/corpus.py +226 -0
  12. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/LICENSE +15 -0
  13. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/README.md +43 -0
  14. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-02f22d31b7fdcea71d773bf551a8f2724838d405738fa2ef0af442c8070ac54d.txt +1 -0
  15. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-15927f44d6eeb6a909c22b635c110cf42d38e3caaeed36d855346fdd26c852d2.txt +1 -0
  16. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-17aed191342fcd8b9267f3e7c2b32bd668a14b79b8326e6bb6130b3a91939bc0.txt +1 -0
  17. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-200d518cfff7ce66d0ceec1d78cdd6008b99b91762a84f5ddcbb14347158f609.txt +1 -0
  18. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-2194333995a547070579bfbd1a2b852186c713dc0ac5cdc8e3de93e21649664d.txt +1 -0
  19. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-275a7a0d1b92cd14f6bba185f1649e64b6a08fdff921841fc5a8ecf844a5a2c0.txt +1 -0
  20. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-28db5136c5154a6f352a4c7c65d9faa91bb4d54e805e3291c6a3a7e087dd3716.txt +1 -0
  21. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-3e35c21b4b52dc7791816ad4de620c8e4121981bf1467b82cc0b7d11216691a6.txt +1 -0
  22. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-3e4ffbe4504e08cde32d72e8d368130e3e7c2900dd70566b582d1930cb1db1dc.txt +1 -0
  23. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-4b04ecdfee380a0ccddc69abe0580399f947287f1effb51eb70832241adce75c.txt +1 -0
  24. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-563d5d191dbbfc2d09e51931c3bfa8a9fabfad63f29c0a39d5b0883f4a2cf926.txt +1 -0
  25. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-5dc06dd0ecad8480cc5778141bfcc449500ef4e91071ca4803e2a58657855907.txt +1 -0
  26. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-5ff5f51a1cc51906b646d6d26839d7263ae36b42e13455ce7fa04282a3e318d1.txt +1 -0
  27. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-626b284e073168611eefd03ef99ce9ea6f6edc26edd9a40065fa4db1206835ef.txt +1 -0
  28. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-644406a38ef9c2f1ad0865c00a9f41b3726e5cd3fac0c3ec11a741ecab0c8ee8.txt +1 -0
  29. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-6f38f5caeb0cebf7d8f274962769c96d26b3f48567707ed996d9354db55e6ac4.txt +1 -0
  30. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-7ab1e56997bdd8a6218e2b1259d60b97ba5b8a0977a2f8f0630ae3b51d485f28.txt +1 -0
  31. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-84b2a4f870a44fb79b3450644f6034d71c05de0218d873de74fdaf521dcfa7aa.txt +1 -0
  32. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-915fc20f4277ef271d0fc5bc849551ac80528b0d711e32f251c5afb383ae35cd.txt +1 -0
  33. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-968bf3f82d0e9893fda7d11a74bd3d00f0497660d75d27bdab7adabf75acf841.txt +1 -0
  34. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-9ca5564ca4da3a5d82f2cd2597f1062f59fd271bf2af7f250c7c68372d81dedd.txt +1 -0
  35. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-a3592c30503b44fe1ac02896051d241b45383834c049d7a7f08bb70017a46950.txt +1 -0
  36. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-bf1a1842a4b1b4b2a324a95148e1f27dfa80ea6325d6cebc82c740c1d7b29ecc.txt +1 -0
  37. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-cb98c8b99d635a90d3229490c358ae6d2ba74a1b43666a26a3d875079e4d4ee3.txt +1 -0
  38. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-cd69bab5cf9e51f661806efda31aa9a54094040ac96af552a1c9830a5c660176.txt +1 -0
  39. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-d07c0b822954c98d025317dbff67421761ae2f34f146912fdb2ba5f783783d0e.txt +1 -0
  40. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-d09a40b18e6407b0f3068ada78d11255bdf7e428ae5a6f68f83ed71321fd030d.txt +1 -0
  41. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-d09d4a86bab66348f1de715bb7744b60c3f588403f39f888c3e708fc0962b82b.txt +1 -0
  42. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-e937f6a874321ceb9be652103e9680d4638bfe07618bda38c0cd5968d326616f.txt +1 -0
  43. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-f207ff5d9eac22d799bcbd808fd7445a414ad77c3e82fcab791f3c09647f1b50.txt +1 -0
  44. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-ff9356935c34fecf2330ca1ba8281801604325274930ea7b72b847e19c86e6d9.txt +1 -0
  45. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/documents/squad-ffe9cd82cb387d87ebd63d703bd85c84a9a7b32dc7ead0612ff3330f5cb39d26.txt +1 -0
  46. omnichunk-2.2.0/benchmarks/retrieval/corpus_squad_v1/manifest.json +1647 -0
  47. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/LICENSE +21 -0
  48. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/connector_matrix.md +14 -0
  49. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/file_reader.py +32 -0
  50. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/incident_playbook.md +22 -0
  51. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/log_filter.py +32 -0
  52. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/model_rollout.md +22 -0
  53. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/plan_entitlements.md +14 -0
  54. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/retention_matrix.md +14 -0
  55. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/retry_client.py +32 -0
  56. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/signed_release.md +22 -0
  57. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/en/task_lease.py +32 -0
  58. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/accessibility-release-review.md +22 -0
  59. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/edge-cache-purge.md +14 -0
  60. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/glossary-governance.md +22 -0
  61. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/image-tile-assembly.py +32 -0
  62. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/media-transform.md +14 -0
  63. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/search-visibility.md +14 -0
  64. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/snapshot-pinning.py +32 -0
  65. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/staged-schema-migration.py +32 -0
  66. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/tenant-export-appeal.md +22 -0
  67. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/ja/webhook-receipts.py +32 -0
  68. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/accessibility_matrix.md +14 -0
  69. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/approval_matrix.md +14 -0
  70. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/batch_status_matrix.md +14 -0
  71. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/catalog_alias.py +32 -0
  72. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/import_workflow.md +22 -0
  73. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/invoice_rounding.py +32 -0
  74. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/knowledge_approval.md +22 -0
  75. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/org_invitation.md +22 -0
  76. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/report_filter.py +32 -0
  77. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/documents/tr/stock_reservation.py +32 -0
  78. omnichunk-2.2.0/benchmarks/retrieval/corpus_v1/manifest.json +4162 -0
  79. omnichunk-2.2.0/benchmarks/retrieval/dense.py +265 -0
  80. omnichunk-2.2.0/benchmarks/retrieval/ranking.py +88 -0
  81. omnichunk-2.2.0/benchmarks/retrieval/requirements.txt +7 -0
  82. omnichunk-2.2.0/benchmarks/run_retrieval.py +378 -0
  83. {omnichunk-2.1.2 → omnichunk-2.2.0}/docs/api-stability.md +8 -0
  84. omnichunk-2.2.0/docs/index.md +78 -0
  85. omnichunk-2.2.0/docs/indexing.md +323 -0
  86. {omnichunk-2.1.2 → omnichunk-2.2.0}/docs/install.md +21 -1
  87. omnichunk-2.2.0/docs/performance/sla.md +211 -0
  88. omnichunk-2.2.0/docs/reference/chunker.md +55 -0
  89. {omnichunk-2.1.2 → omnichunk-2.2.0}/docs/reference/contracts.md +15 -0
  90. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/01_quickstart.py +5 -1
  91. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/02_code_chunking.py +2 -2
  92. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/04_markup_chunking.py +4 -1
  93. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/05_async_and_streaming.py +6 -0
  94. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/06_hierarchical_chunking.py +2 -9
  95. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/07_incremental_diff.py +5 -9
  96. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/08_token_budget_optimizer.py +12 -11
  97. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/09_semantic_chunking.py +22 -13
  98. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/10_graphrag.py +9 -15
  99. omnichunk-2.2.0/examples/11_vector_db_export.py +54 -0
  100. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/12_plugin_api.py +0 -7
  101. omnichunk-2.2.0/examples/13_local_rag_lifecycle.py +130 -0
  102. omnichunk-2.2.0/examples/14_recoverable_indexing.py +161 -0
  103. omnichunk-2.2.0/examples/README.md +67 -0
  104. omnichunk-2.2.0/examples/integrations/pgvector_indexing.py +94 -0
  105. omnichunk-2.2.0/examples/integrations/qdrant_indexing.py +84 -0
  106. omnichunk-2.2.0/examples/notebooks/06_hierarchical_chunking.ipynb +77 -0
  107. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/notebooks/semantic_boundary_analysis.ipynb +4 -1
  108. {omnichunk-2.1.2 → omnichunk-2.2.0}/mkdocs.yml +5 -7
  109. {omnichunk-2.1.2 → omnichunk-2.2.0}/pyproject.toml +13 -7
  110. omnichunk-2.2.0/scripts/publish_github_release.py +88 -0
  111. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/__init__.py +48 -0
  112. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/_version.py +1 -1
  113. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/budget/optimizer.py +23 -3
  114. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/chunker.py +17 -2
  115. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/code_engine.py +17 -16
  116. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/prose_engine.py +105 -62
  117. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/finalize.py +3 -22
  118. omnichunk-2.2.0/src/omnichunk/indexing/__init__.py +37 -0
  119. omnichunk-2.2.0/src/omnichunk/indexing/journal.py +790 -0
  120. omnichunk-2.2.0/src/omnichunk/indexing/models.py +305 -0
  121. omnichunk-2.2.0/src/omnichunk/indexing/sinks.py +276 -0
  122. omnichunk-2.2.0/src/omnichunk/integrations/__init__.py +1 -0
  123. omnichunk-2.2.0/src/omnichunk/integrations/_common.py +95 -0
  124. omnichunk-2.2.0/src/omnichunk/integrations/langchain.py +72 -0
  125. omnichunk-2.2.0/src/omnichunk/integrations/llamaindex.py +212 -0
  126. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/mcp/server.py +22 -4
  127. omnichunk-2.2.0/src/omnichunk/retrieval_eval.py +180 -0
  128. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/sentences.py +49 -24
  129. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/serialization.py +166 -16
  130. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/sizing/counter.py +27 -0
  131. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/sizing/tokenizers.py +4 -3
  132. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/store/chunk_store.py +1 -1
  133. omnichunk-2.2.0/src/omnichunk/windowing/greedy.py +187 -0
  134. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/windowing/merge.py +11 -3
  135. omnichunk-2.2.0/src/omnichunk/windowing/models.py +65 -0
  136. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/windowing/split.py +26 -1
  137. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_budget.py +57 -1
  138. omnichunk-2.2.0/tests/test_code_budget_units.py +233 -0
  139. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_diff.py +18 -2
  140. omnichunk-2.2.0/tests/test_examples.py +26 -0
  141. omnichunk-2.2.0/tests/test_export_integrity.py +113 -0
  142. omnichunk-2.2.0/tests/test_framework_adapters.py +280 -0
  143. omnichunk-2.2.0/tests/test_index_recovery.py +415 -0
  144. omnichunk-2.2.0/tests/test_indexing.py +462 -0
  145. omnichunk-2.2.0/tests/test_indexing_sinks.py +297 -0
  146. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_mcp_server.py +37 -3
  147. omnichunk-2.2.0/tests/test_pinecone_request.py +117 -0
  148. omnichunk-2.2.0/tests/test_prepare_dense.py +207 -0
  149. omnichunk-2.2.0/tests/test_prose_boundaries.py +265 -0
  150. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_prose_chunking.py +41 -0
  151. omnichunk-2.2.0/tests/test_retrieval_adapters.py +174 -0
  152. omnichunk-2.2.0/tests/test_retrieval_corpus.py +288 -0
  153. omnichunk-2.2.0/tests/test_retrieval_dense.py +203 -0
  154. omnichunk-2.2.0/tests/test_retrieval_eval.py +161 -0
  155. omnichunk-2.2.0/tests/test_retrieval_runner.py +331 -0
  156. omnichunk-2.2.0/tests/test_sentence_boundaries.py +85 -0
  157. omnichunk-2.2.0/tests/test_squad_import.py +159 -0
  158. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_execution.py +33 -0
  159. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_source_code.py +19 -0
  160. omnichunk-2.1.2/ARCHITECTURE.md +0 -129
  161. omnichunk-2.1.2/ROADMAP.md +0 -463
  162. omnichunk-2.1.2/benchmarks/README.md +0 -121
  163. omnichunk-2.1.2/docs/index.md +0 -42
  164. omnichunk-2.1.2/docs/migrations/index.md +0 -99
  165. omnichunk-2.1.2/docs/performance/sla.md +0 -69
  166. omnichunk-2.1.2/docs/reference/chunker.md +0 -3
  167. omnichunk-2.1.2/examples/11_vector_db_export.py +0 -46
  168. omnichunk-2.1.2/examples/README.md +0 -52
  169. omnichunk-2.1.2/examples/notebooks/06_hierarchical_chunking.ipynb +0 -114
  170. omnichunk-2.1.2/scripts/check_ai_rules_sync.py +0 -54
  171. omnichunk-2.1.2/src/omnichunk/windowing/greedy.py +0 -100
  172. omnichunk-2.1.2/src/omnichunk/windowing/models.py +0 -12
  173. {omnichunk-2.1.2 → omnichunk-2.2.0}/.gitignore +0 -0
  174. {omnichunk-2.1.2 → omnichunk-2.2.0}/CODE_OF_CONDUCT.md +0 -0
  175. {omnichunk-2.1.2 → omnichunk-2.2.0}/LICENSE +0 -0
  176. {omnichunk-2.1.2 → omnichunk-2.2.0}/SECURITY.md +0 -0
  177. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/bench_tfidf_memory.py +0 -0
  178. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/comparison_format.py +0 -0
  179. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/corpus.json +0 -0
  180. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_benchmarks.py +0 -0
  181. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_comparisons.py +0 -0
  182. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_gutenberg.py +0 -0
  183. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_hotspot_profile.py +0 -0
  184. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_html_report.py +0 -0
  185. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_large_corpus.py +0 -0
  186. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_profiler_deep.py +0 -0
  187. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_quality_report.py +0 -0
  188. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/run_v09_stress.py +0 -0
  189. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/transition_baseline.json +0 -0
  190. {omnichunk-2.1.2 → omnichunk-2.2.0}/benchmarks/workloads.py +0 -0
  191. {omnichunk-2.1.2 → omnichunk-2.2.0}/docs/contributing/mutation-testing.md +0 -0
  192. {omnichunk-2.1.2 → omnichunk-2.2.0}/docs/migrations/v2.md +0 -0
  193. {omnichunk-2.1.2 → omnichunk-2.2.0}/examples/03_prose_and_markdown.py +0 -0
  194. {omnichunk-2.1.2 → omnichunk-2.2.0}/rust/omnichunk_rust/Cargo.toml +0 -0
  195. {omnichunk-2.1.2 → omnichunk-2.2.0}/rust/omnichunk_rust/src/lib.rs +0 -0
  196. {omnichunk-2.1.2 → omnichunk-2.2.0}/scripts/build_rust_extension.py +0 -0
  197. {omnichunk-2.1.2 → omnichunk-2.2.0}/scripts/check_benchmarks.py +0 -0
  198. {omnichunk-2.1.2 → omnichunk-2.2.0}/scripts/check_release.py +0 -0
  199. {omnichunk-2.1.2 → omnichunk-2.2.0}/scripts/run_mutmut.sh +0 -0
  200. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/budget/__init__.py +0 -0
  201. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/cli.py +0 -0
  202. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/config.py +0 -0
  203. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/__init__.py +0 -0
  204. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/entities.py +0 -0
  205. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/format.py +0 -0
  206. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/imports.py +0 -0
  207. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/range_index.py +0 -0
  208. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/rebase.py +0 -0
  209. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/scope.py +0 -0
  210. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/context/siblings.py +0 -0
  211. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/dedup.py +0 -0
  212. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/diff/__init__.py +0 -0
  213. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/diff/engine.py +0 -0
  214. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/__init__.py +0 -0
  215. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/hybrid_engine.py +0 -0
  216. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/markup_engine.py +0 -0
  217. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/router.py +0 -0
  218. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/engine/semantic_engine.py +0 -0
  219. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/eval.py +0 -0
  220. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/__init__.py +0 -0
  221. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/chunk.py +0 -0
  222. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/docx_loader.py +0 -0
  223. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/ipynb.py +0 -0
  224. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/pdf.py +0 -0
  225. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/rst.py +0 -0
  226. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/tex.py +0 -0
  227. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/formats/types.py +0 -0
  228. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/graph/__init__.py +0 -0
  229. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/graph/analysis.py +0 -0
  230. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/graph/builder.py +0 -0
  231. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/graph/types.py +0 -0
  232. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/hierarchy/__init__.py +0 -0
  233. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/hierarchy/builder.py +0 -0
  234. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/mcp/__init__.py +0 -0
  235. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/otel/__init__.py +0 -0
  236. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/otel/util.py +0 -0
  237. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/parser/__init__.py +0 -0
  238. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/parser/html_parser.py +0 -0
  239. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/parser/languages.py +0 -0
  240. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/parser/markdown_parser.py +0 -0
  241. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/parser/query_patterns.py +0 -0
  242. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/parser/tree_sitter.py +0 -0
  243. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/plugins.py +0 -0
  244. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/propositions/__init__.py +0 -0
  245. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/propositions/heuristic.py +0 -0
  246. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/propositions/llm_extract.py +0 -0
  247. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/propositions/types.py +0 -0
  248. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/protocols.py +0 -0
  249. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/py.typed +0 -0
  250. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/quality.py +0 -0
  251. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/__init__.py +0 -0
  252. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/boundaries.py +0 -0
  253. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/cache.py +0 -0
  254. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/embedders.py +0 -0
  255. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/rerank.py +0 -0
  256. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/splitter.py +0 -0
  257. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/semantic/tfidf.py +0 -0
  258. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/sizing/__init__.py +0 -0
  259. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/sizing/nws.py +0 -0
  260. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/sizing/rust_accel.py +0 -0
  261. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/store/__init__.py +0 -0
  262. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/types.py +0 -0
  263. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/util/__init__.py +0 -0
  264. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/util/detect.py +0 -0
  265. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/util/text_index.py +0 -0
  266. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/windowing/__init__.py +0 -0
  267. {omnichunk-2.1.2 → omnichunk-2.2.0}/src/omnichunk/windowing/overlap.py +0 -0
  268. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/conftest.py +0 -0
  269. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/bash_complex.sh +0 -0
  270. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/c_complex.c +0 -0
  271. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/cpp_complex.cpp +0 -0
  272. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/csharp_complex.cs +0 -0
  273. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/elixir_complex.ex +0 -0
  274. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/go_modern.go +0 -0
  275. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/html_page.html +0 -0
  276. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/java_modern.java +0 -0
  277. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/kotlin_complex.kt +0 -0
  278. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/markdown_doc.md +0 -0
  279. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/mega_python_50x.py +0 -0
  280. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/mixed_notebook.py +0 -0
  281. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/notebook_with_outputs.ipynb +0 -0
  282. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/php_complex.php +0 -0
  283. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/python_complex.py +0 -0
  284. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/ruby_complex.rb +0 -0
  285. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/rust_complex.rs +0 -0
  286. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/rust_modern.rs +0 -0
  287. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample.json +0 -0
  288. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample.rst +0 -0
  289. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample.toml +0 -0
  290. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample.yaml +0 -0
  291. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample_math.tex +0 -0
  292. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample_v09.ipynb +0 -0
  293. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sample_v09.tex +0 -0
  294. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/scala_complex.scala +0 -0
  295. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/sql_complex.sql +0 -0
  296. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/typescript_complex.ts +0 -0
  297. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/typescript_modern.ts +0 -0
  298. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/fixtures/v2_multilingual.txt +0 -0
  299. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_adaptive_boundaries.py +0 -0
  300. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_api.py +0 -0
  301. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_async.py +0 -0
  302. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_benchmark_fairness.py +0 -0
  303. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_benchmark_gate.py +0 -0
  304. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_benchmarks.py +0 -0
  305. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_chunk_store.py +0 -0
  306. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_cli.py +0 -0
  307. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_code_chunking.py +0 -0
  308. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_conformance_corpus.py +0 -0
  309. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_context.py +0 -0
  310. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_coverage_branch_gaps.py +0 -0
  311. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_coverage_focus.py +0 -0
  312. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_edge_cases.py +0 -0
  313. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_embed_validation.py +0 -0
  314. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_embedders.py +0 -0
  315. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_entity_extraction.py +0 -0
  316. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_extended_languages.py +0 -0
  317. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_graph.py +0 -0
  318. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_graph_centrality.py +0 -0
  319. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_hierarchy.py +0 -0
  320. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_html_modern.py +0 -0
  321. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_html_report.py +0 -0
  322. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_hybrid_chunking.py +0 -0
  323. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_invariants_hypothesis.py +0 -0
  324. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_ipynb_outputs.py +0 -0
  325. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_latex_math.py +0 -0
  326. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_llm_extract_batch.py +0 -0
  327. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_markup_chunking.py +0 -0
  328. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_name_capture_index.py +0 -0
  329. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_optional_backend_recovery.py +0 -0
  330. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_otel.py +0 -0
  331. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_overlap.py +0 -0
  332. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_plugins.py +0 -0
  333. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_propositions.py +0 -0
  334. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_quality.py +0 -0
  335. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_query_patterns.py +0 -0
  336. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_release_checks.py +0 -0
  337. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_rerank.py +0 -0
  338. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_rst_format.py +0 -0
  339. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_semantic.py +0 -0
  340. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_semantic_cache.py +0 -0
  341. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_semantic_rust.py +0 -0
  342. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_sizing.py +0 -0
  343. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_statement_split.py +0 -0
  344. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_stream_upsert.py +0 -0
  345. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_text_index.py +0 -0
  346. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_text_index_slice.py +0 -0
  347. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_tfidf_sparse.py +0 -0
  348. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_topic_shift.py +0 -0
  349. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v09_multiformat_dedup_eval.py +0 -0
  350. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_contracts.py +0 -0
  351. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_entity_fallbacks.py +0 -0
  352. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_finalization_review.py +0 -0
  353. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_parser_formats.py +0 -0
  354. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_v2_persistence.py +0 -0
  355. {omnichunk-2.1.2 → omnichunk-2.2.0}/tests/test_vectordb_export.py +0 -0
@@ -5,7 +5,102 @@ This project follows [Keep a Changelog](https://keepachangelog.com/) and [Semant
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
- No unreleased changes are staged yet for a new release.
8
+ ## [2.2.0] - 2026-10-25
9
+
10
+ ### Added
11
+ - A separately licensed, checksum-pinned SQuAD 1.1 retrieval corpus with 96
12
+ human-annotated queries, article-disjoint splits and offline import tooling.
13
+ - Frozen dense-vector retrieval and optional local-model preparation with exact
14
+ payload/model provenance and fail-closed token overflow checks.
15
+ - Identity-safe `OmniChunkIngestionPipeline` for cached native LlamaIndex ingestion.
16
+ - Recoverable indexing with `IndexJournal`, versioned dry-run `IndexPlan` JSON,
17
+ source/scope ownership, generation checks, persistent embedding reuse and
18
+ acknowledged bounded delivery. Pending operations survive process restarts;
19
+ unknown/permanent failures require explicit retry after intervention.
20
+ - Optional native LangChain `OmniChunkTextSplitter` and LlamaIndex
21
+ `OmniChunkNodeParser`, preserving stable identity, complete provenance, caller
22
+ metadata and hierarchy relationships with cacheable transform configuration.
23
+ - Optional Qdrant and pgvector sinks, executable integration recipes and a
24
+ minimum/current compatibility matrix with real local clients/PostgreSQL tests.
25
+ - `chunks_to_pinecone_upsert_request()` places namespaces on the request envelope
26
+ and preserves complete chunk metadata within the vendor's flat metadata shape.
27
+ - Dependency-free public retrieval ranking metrics (Recall@k, MRR@k and nDCG@k)
28
+ and canonical source-byte evidence precision, recall and IoU.
29
+ - A versioned MIT-licensed synthetic retrieval corpus: 30 documents and 300
30
+ exact-span questions across English, Turkish and Japanese, with document-level
31
+ development/test separation and integrity validation.
32
+ - A reproducible retrieval comparison runner with seven optional/built-in
33
+ chunkers, shared budgets, fixed lexical retrieval, per-query/slice diagnostics
34
+ and source/runtime fingerprints. CI preserves the full development report.
35
+
36
+ ### Fixed
37
+ - Imported index plans cannot retain stale payloads when changing context policy.
38
+ - Qdrant/pgvector/vector exports reject float32 overflow before remote mutation;
39
+ transient Qdrant gRPC failures remain retryable. Pinecone filter metadata rejects
40
+ reserved keys and numeric identifiers that would lose precision.
41
+ - Retrieval corpus validation rejects ambiguous JSON and normalized cross-split
42
+ duplicates; comparator source-offset mapping is linear in document size.
43
+ - Code windows now measure the configured character, non-whitespace-character or
44
+ tokenizer budget throughout AST grouping and splitting, preserving fitting
45
+ Unicode declarations. Complete source gaps count toward grouping, minimum-size
46
+ merges stay bounded, and transient token probes are not cached.
47
+ - Prose splitting now respects paragraph, line/table-row and Unicode sentence
48
+ boundaries before word-level fallbacks. Node grouping counts the actual source
49
+ slice, including whitespace and nonadditive token costs; minimum-size merging
50
+ cannot exceed the configured maximum. Explicit prose overflow policies also
51
+ apply when the minimum size is zero. Transient token-budget probes are uncached
52
+ so large windows do not retain every growing source prefix.
53
+ - Default sentence segmentation recognizes Turkish and other Unicode starters,
54
+ Japanese/Chinese punctuation without spaces, closing quotes and CRLF paragraphs
55
+ while preserving exact source coordinates.
56
+ - Batch inputs now apply per-document options, matching asynchronous batch behavior.
57
+ - Unbroken prose uses measured hard splits instead of producing hundreds of tiny
58
+ chunks; code hard splits preserve UTF-8 character boundaries.
59
+ - Retrieval budget selection counts rendered context, rejects unknown size units
60
+ and refuses token budgeting for contextualized chunks without rendered counts.
61
+ - Vector exports reject empty, inconsistent or non-finite embeddings. Streaming
62
+ exports detect dimension changes between batches.
63
+ - LangChain/LlamaIndex exports carry native document IDs. Weaviate exports use
64
+ deterministic UUIDs; `weaviate_object_id()` maps chunk IDs for deletions.
65
+ - Stores reject unknown SQLite schemas without overwriting their version marker.
66
+ - JSON-RPC validates finite numbers/timeouts, safely serializes malformed Unicode
67
+ request IDs and refuses nonregular file inputs.
68
+ - Release type checking now targets each interpreter in the validation matrix.
69
+ - Python 3.10 documentation dependencies retain the highlighter API required by
70
+ its supported mkdocstrings release; newer Python versions remain unconstrained.
71
+
72
+ ### Documentation
73
+ - Replaced the historical roadmap with an evidence-based enterprise plan,
74
+ competitor gaps, ordered milestones and measurable release criteria.
75
+ - Kept the generated MkDocs API site, simplified navigation and improved runnable
76
+ examples for retrieval and the incremental indexing lifecycle.
77
+ - Removed duplicate agent rule files and their synchronization tooling;
78
+ `AGENTS.md` is the sole agent instruction file. Removed `ARCHITECTURE.md`.
79
+
80
+ ### Compatibility
81
+ - The delivery journal uses a separate versioned database from `ChunkStore`.
82
+ Rebuild into a new target when adopting its vendor identity mapping; keep the
83
+ old target for rollback. Delivery is at least once, and does not provide atomic
84
+ visibility across remote batches or enforce application authorization.
85
+ - Native framework extras now include their verified splitter/parser dependencies;
86
+ minimum and current compatible versions are listed in the installation guide.
87
+ - Exported native IDs change record identity for integrations that previously
88
+ generated random IDs. Rebuild or reconcile those external indexes before reuse.
89
+ - Corrected splitting can change chunk boundaries and IDs; reindex affected
90
+ documents and apply removals. Exact chunk layouts are not a compatibility promise.
91
+ - Invalid embeddings and incomplete rendered token metadata now fail explicitly.
92
+ - No package version bump or release publication is included in these changes.
93
+
94
+ ## [2.1.4] - 2026-09-08
95
+
96
+ ### Fixed
97
+ - Bumped package runtime version to `2.1.4`.
98
+ - Publication metadata and changelog maintenance wording were aligned for post-
99
+ 2.1.x release tracking.
100
+
101
+ ### Compatibility
102
+ - No API/runtime behavior changes.
103
+ - Backward-compatible maintenance release.
9
104
 
10
105
  ## [2.1.1] - 2026-09-08
11
106
 
@@ -7,7 +7,7 @@ Thanks for contributing.
7
7
  1. Create a virtualenv and install dev dependencies:
8
8
 
9
9
  ```bash
10
- pip install -e ".[dev]"
10
+ pip install -e ".[dev,docs,langchain,llamaindex]"
11
11
  ```
12
12
 
13
13
  2. Run tests:
@@ -28,9 +28,13 @@ plus a follow-up `pytest -m no_xdist` in series.
28
28
  ```bash
29
29
  ruff check src tests
30
30
  ruff format --check src tests
31
- mypy src/omnichunk
31
+ mypy --python-version 3.12 src/omnichunk
32
32
  ```
33
33
 
34
+ Use your installed interpreter's version as the mypy target. Optional dependency
35
+ stubs can require newer syntax than the package's minimum Python version; CI runs
36
+ the supported Python 3.10–3.13 matrix in matching environments.
37
+
34
38
  ## Development rules
35
39
 
36
40
  - Keep changes deterministic.
@@ -62,7 +66,7 @@ reconstruct the original binary container or notebook JSON.
62
66
  - [ ] Verified lint, formatting, and type checks
63
67
  - [ ] Updated docs/README if behavior changed
64
68
  - [ ] Preserved chunk invariants
65
- - [ ] Kept rule files synchronized (`AI_RULES.md`, `.cursorrules`, `.windsurfrules`, `CLAUDE.md`, `.github/copilot-instructions.md`)
69
+ - [ ] Verified `mkdocs build --strict` if documentation or public APIs changed
66
70
 
67
71
  ## Commit style
68
72
 
@@ -88,7 +92,7 @@ release quality and roadmap priorities.
88
92
  - Set the version only in `src/omnichunk/_version.py`; Hatch derives package
89
93
  metadata from it. The release tag must match that version.
90
94
  - Run the full line-and-branch coverage gate (90%, no core exclusions), strict
91
- MkDocs build, rule sync and benchmark checks. Validate the built wheel/sdist
95
+ MkDocs build and benchmark checks. Validate the built wheel/sdist
92
96
  with `python scripts/check_release.py --help` for the supported arguments.
93
97
  - Build and validate separate indexes before switching consumers across the
94
98
  [2.x migration](docs/migrations/v2.md); retain the old index for rollback.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: omnichunk
3
- Version: 2.1.2
3
+ Version: 2.2.0
4
4
  Summary: Structure-aware deterministic chunking for code, prose, and markup.
5
5
  Project-URL: Homepage, https://github.com/oguzhankir/omnichunk
6
6
  Project-URL: Documentation, https://oguzhankir.github.io/omnichunk/
@@ -24,9 +24,10 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
24
24
  Classifier: Typing :: Typed
25
25
  Requires-Python: >=3.10
26
26
  Provides-Extra: all
27
- Requires-Dist: langchain-core>=0.2.0; extra == 'all'
27
+ Requires-Dist: langchain-core<2,>=0.2.11; extra == 'all'
28
+ Requires-Dist: langchain-text-splitters<2,>=0.2.2; extra == 'all'
28
29
  Requires-Dist: line-profiler>=4.1.0; extra == 'all'
29
- Requires-Dist: llama-index-core>=0.10.0; extra == 'all'
30
+ Requires-Dist: llama-index-core<0.15,>=0.10.68.post1; extra == 'all'
30
31
  Requires-Dist: numpy>=1.24.0; extra == 'all'
31
32
  Requires-Dist: opentelemetry-api>=1.21.0; extra == 'all'
32
33
  Requires-Dist: py-spy>=0.3.14; extra == 'all'
@@ -114,7 +115,8 @@ Provides-Extra: docs
114
115
  Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
115
116
  Requires-Dist: mkdocs>=1.6.0; extra == 'docs'
116
117
  Requires-Dist: mkdocstrings[python]>=0.25.0; extra == 'docs'
117
- Requires-Dist: pymdown-extensions>=10.0; extra == 'docs'
118
+ Requires-Dist: pymdown-extensions<12.2,>=10.0; (python_version < '3.11') and extra == 'docs'
119
+ Requires-Dist: pymdown-extensions>=10.0; (python_version >= '3.11') and extra == 'docs'
118
120
  Provides-Extra: docx
119
121
  Requires-Dist: python-docx>=1.0; extra == 'docx'
120
122
  Provides-Extra: formats
@@ -122,18 +124,23 @@ Requires-Dist: pypdf>=3.0; extra == 'formats'
122
124
  Requires-Dist: python-docx>=1.0; extra == 'formats'
123
125
  Provides-Extra: graph
124
126
  Provides-Extra: langchain
125
- Requires-Dist: langchain-core>=0.2.0; extra == 'langchain'
127
+ Requires-Dist: langchain-core<2,>=0.2.11; extra == 'langchain'
128
+ Requires-Dist: langchain-text-splitters<2,>=0.2.2; extra == 'langchain'
126
129
  Provides-Extra: llamaindex
127
- Requires-Dist: llama-index-core>=0.10.0; extra == 'llamaindex'
130
+ Requires-Dist: llama-index-core<0.15,>=0.10.68.post1; extra == 'llamaindex'
128
131
  Provides-Extra: mcp
129
132
  Provides-Extra: otel
130
133
  Requires-Dist: opentelemetry-api>=1.21.0; extra == 'otel'
131
134
  Provides-Extra: pdf
132
135
  Requires-Dist: pypdf>=3.0; extra == 'pdf'
136
+ Provides-Extra: pgvector
137
+ Requires-Dist: psycopg[binary]<4,>=3.2.1; extra == 'pgvector'
133
138
  Provides-Extra: pinecone
134
139
  Provides-Extra: profiling
135
140
  Requires-Dist: line-profiler>=4.1.0; extra == 'profiling'
136
141
  Requires-Dist: py-spy>=0.3.14; extra == 'profiling'
142
+ Provides-Extra: qdrant
143
+ Requires-Dist: qdrant-client<2,>=1.12; extra == 'qdrant'
137
144
  Provides-Extra: rust
138
145
  Requires-Dist: maturin>=1.7.0; extra == 'rust'
139
146
  Provides-Extra: scipy
@@ -163,19 +170,23 @@ Description-Content-Type: text/markdown
163
170
 
164
171
  # Omnichunk
165
172
 
166
- Structure-aware chunking for code, prose and mixed documents. Chunks carry source
167
- ranges and context such as scopes, imports, headings and notebook cells, with
168
- helpers for indexing and retrieval workflows.
173
+ Structure-aware chunking and index preparation for RAG. Turn code, prose and
174
+ mixed documents into traceable chunks with explicit size budgets, source ranges
175
+ and context such as scopes, imports, headings and notebook cells.
169
176
 
170
- **Development status:** the 2.0 foundation changes and 2.1 code-quality milestone
171
- are implemented in this checkout for the next **2.1.0** release. Read the
172
- [migration guide](docs/migrations/v2.md) before upgrading existing indexes.
173
- The [roadmap](ROADMAP.md) tracks verification, compatible additions through 3.0,
174
- and the 4.0 vision. This is unreleased work; the maintainer sets the release version
175
- in `src/omnichunk/_version.py` after validation.
177
+ **Our goal:** make reliable chunking the simplest part of building a retrieval
178
+ system. Preserve useful structure, explain every source span, update indexes
179
+ without stale records, and demonstrate quality on reproducible retrieval tasks.
180
+ The [roadmap](ROADMAP.md) defines the evidence required to get there.
181
+
182
+ **Status:** the current source version is **2.1.4**, with hardening changes tracked
183
+ in the [changelog](CHANGELOG.md). The package is classified as beta. Its enterprise
184
+ readiness and superiority to other chunkers are goals to validate, not current
185
+ certifications. Read the [migration guide](docs/migrations/v2.md) before upgrading
186
+ existing indexes.
176
187
 
177
188
  [Documentation](https://oguzhankir.github.io/omnichunk/) ·
178
- [Architecture](ARCHITECTURE.md) · [Examples](examples/README.md) ·
189
+ [Roadmap](ROADMAP.md) · [Examples](examples/README.md) ·
179
190
  [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md)
180
191
 
181
192
  ## Installation
@@ -196,7 +207,7 @@ code uses a diagnosed fallback. Structural chunking requires no external service
196
207
  optional model/tokenizer integrations may download assets or use your callbacks.
197
208
 
198
209
  Other extras include `transformers`, `sentence-transformers`, `scipy`, `langchain`,
199
- `llamaindex`, `otel`, `docs`, `dev` and optional Rust build tooling (`rust`).
210
+ `llamaindex`, `qdrant`, `pgvector`, `otel`, `docs`, `dev` and optional Rust build tooling (`rust`).
200
211
  See [installation details](docs/install.md) for the full grouping and limitations.
201
212
 
202
213
  ## Quick start
@@ -247,8 +258,12 @@ Context that will not fit is omitted by default, or rejected with
247
258
  storage, token-budget selection, exact/near deduplication, entity–chunk graphs,
248
259
  centrality/community analysis and heuristic/callback-based propositions.
249
260
  - **Integration:** synchronous/async/batch APIs, iterators, CLI, JSONL/CSV,
250
- LangChain/LlamaIndex document conversion, vendor-shaped vector payloads,
261
+ native LangChain splitter and LlamaIndex node parser, document conversion,
262
+ vendor-shaped vector payloads,
251
263
  parser/formatter registration and optional OpenTelemetry.
264
+ - **Recoverable indexing:** serializable change plans, a SQLite delivery journal,
265
+ acknowledged batches, persistent embedding reuse and scoped deletion. Optional
266
+ Qdrant and pgvector sinks resume pending writes after a process restart.
252
267
 
253
268
  Use `language_capabilities()` to distinguish detected languages, installed
254
269
  grammars, extraction queries and fallback behavior. The
@@ -256,6 +271,13 @@ grammars, extraction queries and fallback behavior. The
256
271
  Graph helpers provide entity relationships; they are not a complete GraphRAG
257
272
  retriever. PDF/DOCX support extracts text; it does not provide OCR or full layout.
258
273
 
274
+ For recoverable index updates, follow the [indexing guide](docs/indexing.md) and run
275
+ `python examples/14_recoverable_indexing.py`. It demonstrates a failed delivery,
276
+ restart/replay, metadata updates without re-embedding and deletion. Delivery is
277
+ at least once: applications own authorization, and sinks must make repeated
278
+ upserts/deletes safe. Acknowledged delivery does not make a multi-batch remote
279
+ update atomically visible to readers.
280
+
259
281
  ## Files, directories and CLI
260
282
 
261
283
  ```python
@@ -285,6 +307,13 @@ For reconstruction evaluation, `--source` must name the same canonical source
285
307
  that produced the chunks; combining unrelated files with one source is invalid.
286
308
  Current offline coherence/coverage metrics are heuristics, not retrieval scores.
287
309
 
310
+ For source-labeled retrieval evaluation, use the
311
+ [retrieval benchmark](benchmarks/README.md#retrieval-evaluation-and-tuning). It
312
+ compares chunking settings under shared budgets and reports ranking/evidence
313
+ metrics on a versioned, multilingual synthetic corpus. Tune on the development
314
+ split before evaluating held-out documents; the initial offline lexical baseline
315
+ does not establish production retrieval quality.
316
+
288
317
  ## Source ranges and context
289
318
 
290
319
  Each `Chunk` contains `text`, `contextualized_text`, `byte_range`, `line_range`,
@@ -342,6 +371,18 @@ Versioned JSON round-trips full chunk metadata. The
342
371
  [migration guide](docs/migrations/v2.md) includes legacy conversion and separate
343
372
  index rebuild/rollback examples.
344
373
 
374
+ The local store transaction does not include a remote vector database. Persist
375
+ pending deletions/upserts and retry them in your application; `stream_upsert()`
376
+ yields payloads and does not acknowledge database writes. Framework exports carry
377
+ native document IDs; Weaviate objects use deterministic UUIDs derived from chunk
378
+ IDs. Use `weaviate_object_id(chunk_id)` when deleting those objects. Existing
379
+ indexes created with autogenerated IDs need a rebuild or explicit reconciliation.
380
+
381
+ `TokenBudgetOptimizer` budgets `contextualized_text`, including derived context.
382
+ Reserve additional space for prompt instructions, citation labels and separators.
383
+ Token counts follow the configured chunking tokenizer; estimated counts remain
384
+ estimates and should not be treated as exact model limits.
385
+
345
386
  Semantic caches isolate provider, model revision and preprocessing. Give opaque
346
387
  callbacks an explicit stable fingerprint/revision for durable reuse. Parser and
347
388
  formatter plugins can be scoped through `PluginRegistry`.
@@ -354,12 +395,11 @@ standard MCP lifecycle support remains a later optional integration.
354
395
  ## Development and evaluation
355
396
 
356
397
  ```bash
357
- pip install -e ".[dev,docs]"
398
+ pip install -e ".[dev,docs,langchain,llamaindex]"
358
399
  pytest -q
359
400
  ruff check src tests
360
401
  ruff format --check src tests
361
- mypy src/omnichunk
362
- python scripts/check_ai_rules_sync.py
402
+ mypy --python-version 3.12 src/omnichunk
363
403
  python scripts/check_benchmarks.py --run-quality
364
404
  mkdocs build --strict
365
405
  ```
@@ -369,6 +409,9 @@ tests marked `slow`. Coverage measures all Python source with a 90% aggregate
369
409
  line-and-branch gate. The [roadmap](ROADMAP.md) separates local verification from
370
410
  the release matrix and publication checks.
371
411
 
412
+ Match mypy's target to your Python interpreter; CI checks Python 3.10–3.13 with
413
+ the optional dependency versions installable on each interpreter.
414
+
372
415
  [Benchmark instructions](benchmarks/README.md) explain reproducibility and the
373
416
  current comparison limitations. No universal speed or retrieval-quality leadership
374
417
  claim has been established.
@@ -10,19 +10,23 @@
10
10
 
11
11
  # Omnichunk
12
12
 
13
- Structure-aware chunking for code, prose and mixed documents. Chunks carry source
14
- ranges and context such as scopes, imports, headings and notebook cells, with
15
- helpers for indexing and retrieval workflows.
13
+ Structure-aware chunking and index preparation for RAG. Turn code, prose and
14
+ mixed documents into traceable chunks with explicit size budgets, source ranges
15
+ and context such as scopes, imports, headings and notebook cells.
16
16
 
17
- **Development status:** the 2.0 foundation changes and 2.1 code-quality milestone
18
- are implemented in this checkout for the next **2.1.0** release. Read the
19
- [migration guide](docs/migrations/v2.md) before upgrading existing indexes.
20
- The [roadmap](ROADMAP.md) tracks verification, compatible additions through 3.0,
21
- and the 4.0 vision. This is unreleased work; the maintainer sets the release version
22
- in `src/omnichunk/_version.py` after validation.
17
+ **Our goal:** make reliable chunking the simplest part of building a retrieval
18
+ system. Preserve useful structure, explain every source span, update indexes
19
+ without stale records, and demonstrate quality on reproducible retrieval tasks.
20
+ The [roadmap](ROADMAP.md) defines the evidence required to get there.
21
+
22
+ **Status:** the current source version is **2.1.4**, with hardening changes tracked
23
+ in the [changelog](CHANGELOG.md). The package is classified as beta. Its enterprise
24
+ readiness and superiority to other chunkers are goals to validate, not current
25
+ certifications. Read the [migration guide](docs/migrations/v2.md) before upgrading
26
+ existing indexes.
23
27
 
24
28
  [Documentation](https://oguzhankir.github.io/omnichunk/) ·
25
- [Architecture](ARCHITECTURE.md) · [Examples](examples/README.md) ·
29
+ [Roadmap](ROADMAP.md) · [Examples](examples/README.md) ·
26
30
  [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md)
27
31
 
28
32
  ## Installation
@@ -43,7 +47,7 @@ code uses a diagnosed fallback. Structural chunking requires no external service
43
47
  optional model/tokenizer integrations may download assets or use your callbacks.
44
48
 
45
49
  Other extras include `transformers`, `sentence-transformers`, `scipy`, `langchain`,
46
- `llamaindex`, `otel`, `docs`, `dev` and optional Rust build tooling (`rust`).
50
+ `llamaindex`, `qdrant`, `pgvector`, `otel`, `docs`, `dev` and optional Rust build tooling (`rust`).
47
51
  See [installation details](docs/install.md) for the full grouping and limitations.
48
52
 
49
53
  ## Quick start
@@ -94,8 +98,12 @@ Context that will not fit is omitted by default, or rejected with
94
98
  storage, token-budget selection, exact/near deduplication, entity–chunk graphs,
95
99
  centrality/community analysis and heuristic/callback-based propositions.
96
100
  - **Integration:** synchronous/async/batch APIs, iterators, CLI, JSONL/CSV,
97
- LangChain/LlamaIndex document conversion, vendor-shaped vector payloads,
101
+ native LangChain splitter and LlamaIndex node parser, document conversion,
102
+ vendor-shaped vector payloads,
98
103
  parser/formatter registration and optional OpenTelemetry.
104
+ - **Recoverable indexing:** serializable change plans, a SQLite delivery journal,
105
+ acknowledged batches, persistent embedding reuse and scoped deletion. Optional
106
+ Qdrant and pgvector sinks resume pending writes after a process restart.
99
107
 
100
108
  Use `language_capabilities()` to distinguish detected languages, installed
101
109
  grammars, extraction queries and fallback behavior. The
@@ -103,6 +111,13 @@ grammars, extraction queries and fallback behavior. The
103
111
  Graph helpers provide entity relationships; they are not a complete GraphRAG
104
112
  retriever. PDF/DOCX support extracts text; it does not provide OCR or full layout.
105
113
 
114
+ For recoverable index updates, follow the [indexing guide](docs/indexing.md) and run
115
+ `python examples/14_recoverable_indexing.py`. It demonstrates a failed delivery,
116
+ restart/replay, metadata updates without re-embedding and deletion. Delivery is
117
+ at least once: applications own authorization, and sinks must make repeated
118
+ upserts/deletes safe. Acknowledged delivery does not make a multi-batch remote
119
+ update atomically visible to readers.
120
+
106
121
  ## Files, directories and CLI
107
122
 
108
123
  ```python
@@ -132,6 +147,13 @@ For reconstruction evaluation, `--source` must name the same canonical source
132
147
  that produced the chunks; combining unrelated files with one source is invalid.
133
148
  Current offline coherence/coverage metrics are heuristics, not retrieval scores.
134
149
 
150
+ For source-labeled retrieval evaluation, use the
151
+ [retrieval benchmark](benchmarks/README.md#retrieval-evaluation-and-tuning). It
152
+ compares chunking settings under shared budgets and reports ranking/evidence
153
+ metrics on a versioned, multilingual synthetic corpus. Tune on the development
154
+ split before evaluating held-out documents; the initial offline lexical baseline
155
+ does not establish production retrieval quality.
156
+
135
157
  ## Source ranges and context
136
158
 
137
159
  Each `Chunk` contains `text`, `contextualized_text`, `byte_range`, `line_range`,
@@ -189,6 +211,18 @@ Versioned JSON round-trips full chunk metadata. The
189
211
  [migration guide](docs/migrations/v2.md) includes legacy conversion and separate
190
212
  index rebuild/rollback examples.
191
213
 
214
+ The local store transaction does not include a remote vector database. Persist
215
+ pending deletions/upserts and retry them in your application; `stream_upsert()`
216
+ yields payloads and does not acknowledge database writes. Framework exports carry
217
+ native document IDs; Weaviate objects use deterministic UUIDs derived from chunk
218
+ IDs. Use `weaviate_object_id(chunk_id)` when deleting those objects. Existing
219
+ indexes created with autogenerated IDs need a rebuild or explicit reconciliation.
220
+
221
+ `TokenBudgetOptimizer` budgets `contextualized_text`, including derived context.
222
+ Reserve additional space for prompt instructions, citation labels and separators.
223
+ Token counts follow the configured chunking tokenizer; estimated counts remain
224
+ estimates and should not be treated as exact model limits.
225
+
192
226
  Semantic caches isolate provider, model revision and preprocessing. Give opaque
193
227
  callbacks an explicit stable fingerprint/revision for durable reuse. Parser and
194
228
  formatter plugins can be scoped through `PluginRegistry`.
@@ -201,12 +235,11 @@ standard MCP lifecycle support remains a later optional integration.
201
235
  ## Development and evaluation
202
236
 
203
237
  ```bash
204
- pip install -e ".[dev,docs]"
238
+ pip install -e ".[dev,docs,langchain,llamaindex]"
205
239
  pytest -q
206
240
  ruff check src tests
207
241
  ruff format --check src tests
208
- mypy src/omnichunk
209
- python scripts/check_ai_rules_sync.py
242
+ mypy --python-version 3.12 src/omnichunk
210
243
  python scripts/check_benchmarks.py --run-quality
211
244
  mkdocs build --strict
212
245
  ```
@@ -216,6 +249,9 @@ tests marked `slow`. Coverage measures all Python source with a 90% aggregate
216
249
  line-and-branch gate. The [roadmap](ROADMAP.md) separates local verification from
217
250
  the release matrix and publication checks.
218
251
 
252
+ Match mypy's target to your Python interpreter; CI checks Python 3.10–3.13 with
253
+ the optional dependency versions installable on each interpreter.
254
+
219
255
  [Benchmark instructions](benchmarks/README.md) explain reproducibility and the
220
256
  current comparison limitations. No universal speed or retrieval-quality leadership
221
257
  claim has been established.