spectraltm-db 0.1.0__tar.gz

spectraltm_db-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of tracking, enhancing, or otherwise modifying
+      the Work, but excludes communication that is conspicuously marked or
+      otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have signed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for describing the origin of the Work and
+      reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Support. While redistributing the Work or
+      Derivative Works thereof, You may choose to offer, and charge a
+      fee for, acceptance of support, warranty, indemnity, or other
+      liability obligations and/or rights consistent with this License.
+      However, in accepting such obligations, You may act only on Your
+      own behalf and on Your sole responsibility, not on behalf of any
+      other Contributor, and only if You agree to indemnify, defend,
+      and hold each Contributor harmless for any liability incurred by,
+      or claims asserted against, such Contributor by reason of your
+      accepting any such warranty or support.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Gerald Enrique Nelson Mc Kenzie
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

spectraltm_db-0.1.0/PKG-INFO

@@ -0,0 +1,294 @@
+Metadata-Version: 2.4
+Name: spectraltm-db
+Version: 0.1.0
+Summary: Pinecone-shaped local vector database on top of the SpectralTM Rust crate (Sparse Spectral Encoding cold-tier compression).
+Author-email: Gerald Enrique Nelson Mc Kenzie <lordxmen2k@gmail.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/lordxmen2k/sparse-spectral-encoding
+Keywords: vector-database,semantic-search,rag,pinecone,spectral-encoding,cold-tier,memory
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: spectraltm>=0.1.1
+Requires-Dist: numpy>=1.24
+Requires-Dist: msgpack>=1.0.7
+Provides-Extra: encoder
+Requires-Dist: sentence-transformers>=2.2; extra == "encoder"
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.1; extra == "langchain"
+Provides-Extra: llamaindex
+Requires-Dist: llama-index-core>=0.10; extra == "llamaindex"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110; extra == "server"
+Requires-Dist: uvicorn>=0.27; extra == "server"
+Provides-Extra: all
+Requires-Dist: sentence-transformers>=2.2; extra == "all"
+Requires-Dist: langchain-core>=0.1; extra == "all"
+Requires-Dist: llama-index-core>=0.10; extra == "all"
+Requires-Dist: fastapi>=0.110; extra == "all"
+Requires-Dist: uvicorn>=0.27; extra == "all"
+Dynamic: license-file
+
+# spectraltm-db
+
+[![PyPI version](https://img.shields.io/badge/pypi-v0.1.0.dev0-blue)](https://pypi.org/project/spectraltm-db/)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+> **Pinecone-shape local vector database built on the Spectral encoding Rust engine.**
+
+`spectraltm-db` wraps the AVX2-accelerated [`spectraltm`](https://pypi.org/project/spectraltm/) crate with a Pinecone-compatible Python API: same `Index.create`, `Index.upsert`, `Index.query`, metadata filters, namespaces, REST server — running entirely on disk, single-process, no cloud dependency.
+
+## Features
+
+- **Pinecone-shape surface**: `Index.create`, `open`, `upsert`, `query`, `fetch`, `delete`, `update`, `describe_index_stats`, `compact` — wire-compatible with the `pinecone` Python SDK.
+- **Spectral encoding compression**: 192 B/chunk at K=64, d=384 (vs ~1.5 KB for float32 embeddings); four named levels: `spectral_k64 / k128 / k256 / float32`.
+- **11 metadata filter operators**: `$eq $ne $gt $gte $lt $lte $in $nin $exists $and $or` — compiled to `json_extract(...)` SQL via a tiny in-process SQLite store.
+- **Multi-namespace** by directory partitioning; per-namespace WAL, calibration buffer, `raw.npy` source-of-truth.
+- **Soft delete + real `compact()`**: re-encodes from `raw.npy` minus tombstones, recovers on-disk space — not just a flag-flip.
+- **LangChain + LlamaIndex adapters** (`pip install spectraltm-db[langchain]` / `[llamaindex]`).
+- **Pinecone-identical REST server** (`pip install spectraltm-db[server]`): 11 endpoints, `Api-Key` auth, multi-index registry.
+- **CLI** (`python -m spectraltm_db {create,stats,upsert,query,delete,update,compact,serve,list}`).
+- **AVX2 SIMD scoring** via the Rust crate; falls back to scalar on other platforms.
+- **Optional `SpectralIndex.score_one` PyO3 binding** for safe single-chunk scoring at n=1 (where `select_nth_unstable_by` would panic).
+
+## Installation
+
+```bash
+pip install spectraltm-db
+```
+
+Requires Python 3.10+. Pulls in `spectraltm>=0.1.1` (Rust SSE engine) and a pure-Python stack: `numpy`, `msgpack`, stdlib `sqlite3`.
+
+Optional extras:
+
+```bash
+pip install "spectraltm-db[encoder]"      # sentence-transformers auto-embedding
+pip install "spectraltm-db[langchain]"    # langchain_core VectorStore adapter
+pip install "spectraltm-db[llamaindex]"   # llama-index VectorStore adapter
+pip install "spectraltm-db[server]"       # FastAPI REST server (11 endpoints)
+pip install "spectraltm-db[all]"          # everything
+```
+
+## Quick start
+
+```python
+import spectraltm_db as stm
+
+# Calibration sample: 1,000-10,000 random vectors of dim `dimension`.
+# Calibrates the SSE encoder's magnitude/phase quantization grids.
+calibration = [random.gauss(0, 1) for _ in range(2000 * 384)]
+
+idx = stm.Index.create(
+    name="conversations",
+    path="~/.spectraltm/conversations",
+    dimension=384,
+    compression="spectral_k64",     # or spectral_k128 / spectral_k256 / float32
+    calibration_sample=calibration,
+)
+
+# Upsert some vectors (Pinecone-style dicts)
+idx.upsert(vectors=[
+    {"id": "turn_001",
+     "values": [...],                 # length-384 list of floats
+     "metadata": {"session_id": "abc", "role": "user"}},
+    {"id": "turn_002",
+     "values": [...],
+     "metadata": {"session_id": "abc", "role": "assistant"}},
+])
+
+# Pinecone-shape query
+results = idx.query(
+    vector=[...],                              # length-384 list
+    top_k=10,
+    filter={"session_id": {"$eq": "abc"}},     # metadata filter
+)
+for m in results["matches"]:
+    print(m["id"], m["score"], m["metadata"])
+
+# Reopen after process restart
+idx.flush()
+idx2 = stm.Index.open("~/.spectraltm/conversations")
+```
+
+See `examples/rag_demo.py` for a runnable end-to-end example.
+
+## Benchmarks — measured at v0.1.0.dev0 (median, top_k=10, dim=384)
+
+Run via `python -m spectraltm_db.examples.benchmark`. Ingest uses the bulk
+`add_batch` Rust path; query latency is single-threaded.
+
+| N        | K  | dim | bytes/chunk | total MB | ms/query (median) | ms/query (p95) | ms/query (p99) | ingest (s) |
+|----------|----|-----|-------------|----------|-------------------|----------------|----------------|------------|
+| 10 000   | 64 | 384 | 192         | 1.83     | **7.29 ms**       | 9.24 ms        | 14.31 ms       | ~10 s      |
+| 100 000  | 64 | 384 | 192         | 18.3     | **86.43 ms**      | 93.88 ms       | 95.50 ms       | ~451 s     |
+
+Honest framing:
+
+- **Search latency scales linearly with N at AVX2 throughput** (~12× slower
+  for 10× more data). The n=1 corner case is handled by Rust `score_one`
+  or NumPy fallback when the binding isn't present.
+- **Ingest is dominated by per-vector rFFT encoding.** Each new vector
+  pays an O(F log F) encoding pass; the bulk `add_batch` path amortizes
+  Python→Rust overhead at 1.4 ms/vector but cannot skip the encoding.
+  The fully-vectorized "load_codes" path (accepting pre-quantized codes)
+  is on the v0.2 roadmap and is what unlocks billion-vector scale.
+- **`spectraltm-db` is a lossy compressed index, not a brute-force
+  cosine replacement.** See `spectraltm`'s BEIR numbers (K=8 loses ~65%
+  nDCG, K=64 ~12%) for the quality curve. Your choice of K is a
+  storage-budget / quality-bar tradeoff.
+
+## When to use spectraltm-db
+
+Use it when:
+
+- You want a **Pinecone-shaped API** without a managed service: RAG,
+  conversation memory, semantic search — anything you'd point at a
+  Pinecone index and don't want cloud cost or egress.
+- Your corpus fits on a single machine and on the order of N ≤ 1M
+  vectors (above which encoding dominates ingest).
+- You value **single-tenant, on-disk persistence**: no cluster, no
+  replicas, just a folder on disk that you can `cp`, `rsync`, or
+  back up however you like.
+
+Don't use it when:
+
+- You need real Pinecone parity in **multi-region / serverless
+  autoscaling / RBAC / backups** — that's v0.2+.
+- You need **sparse-dense hybrid ranking** with actual BM25/Reciprocal
+  Rank Fusion — wire surface only in v0.1, scoring lands in v0.2.
+- You need **billion-vector scale** — the AVX2 inner loop scales
+  linearly with N; ingest cost dominates above ~1M without a pre-coded
+  bulk-load path.
+- You need **multi-process writers** — single-process at a time;
+  multi-reader via SQLite WAL is fine; threaded HTTP works in-process.
+
+## Project layout
+
+```
+spectraltm_db/
+├── spectraltm_db/
+│   ├── __init__.py        # Index / IndexConfig / errors
+│   ├── __main__.py        # CLI entry point
+│   ├── config.py          # IndexConfig + CompressionSpec + MetricSpec
+│   ├── errors.py          # exception hierarchy
+│   ├── index.py           # Index — Pinecone-shape public API
+│   ├── wal.py             # minimal JSONL write-ahead log
+│   ├── sparse.py          # Sparse-dense wire surface (v0.2 stub)
+│   ├── server.py          # FastAPI REST server (optional)
+│   ├── .github/workflows/test.yml  # CI
+│   ├── storage/
+│   │   ├── vector_store.py     # wraps spectraltm.SpectralIndex
+│   │   ├── id_map.py           # str-id ↔ chunk_id bijection
+│   │   ├── metadata_store.py   # SQLite metadata + dynamic indexes
+│   │   └── namespaces.py       # directory partitioning
+│   ├── retrieval/
+│   │   ├── filter_compiler.py  # dict → SQL
+│   │   └── filter_executor.py  # SQL → chunk_id set
+│   ├── integrations/
+│   │   ├── langchain.py        # VectorStore adapter
+│   │   └── llamaindex.py       # VectorStore adapter
+│   └── examples/
+│       ├── smoke_test.py
+│       ├── rag_demo.py
+│       └── benchmark.py
+├── tests/                # 107 tests (3 conditional skips for optional Rust binding)
+├── pyproject.toml        # pure-python; depends on spectraltm from PyPI
+├── README.md
+├── CHANGELOG.md
+├── PUBLISH.md            # how to publish to PyPI
+└── LICENSE               # Apache-2.0
+```
+
+## How It Works
+
+```
+embeddings ─┐
+            ├─► [encoder.calibrate]  ─► quantizer grids (mags, phases, norms)
+            │
+            ├─► [encoder.encode]      ─► SpectralCodes (idx, mag_q, phase_q, norm_q)
+            │
+            └─► [vector_store.add_batch] ─► Rust add_embeddings ─► dense (N,F) mags + phases
+
+queries  ──┬─► [filter_compiler]   ─► SQL chunk_id whitelist (when filter present)
+            │
+            ├─► [encoder.encode]      ─► query SpectralCodes
+            │
+            └─► [SpectralIndex.search]  ─► AVX2 SIMD inner loop ─► top-K cosine scores
+                                    │
+                                    └─► [metadata_store.hydrate] ─► Pinecone-shaped response
+```
+
+The hot path is the Rust `SpectralIndex.search` (AVX2 SIMD on x86_64,
+scalar fallback elsewhere). Filter pre-narrowing happens in SQLite
+before we hand the candidate set to SSE; this matches the Pinecone
+data plane, where `filter` runs before scoring. n=1 uses an optional
+Rust `score_one` binding to bypass `select_nth_unstable_by` (which would
+panic for `top_k >= n`).
+
+## Development
+
+```bash
+# Editable install (Pure-Python) — pulls spectraltm from PyPI:
+pip install -e ".[all]"
+
+# Run the test suite:
+pytest tests/ -v
+
+# Reinstall the built wheel:
+pip install --force-reinstall --no-deps dist/spectraltm_db-*-py3-none-any.whl
+
+# Smoke CLI:
+python -m spectraltm_db --help
+```
+
+For the optional `score_one` Rust binding path, you need a local build
+of `spectraltm>=0.1.3` (workspace has the Rust patch; release to
+PyPI is on the v0.2 roadmap). Until then, `VectorStore.search` falls
+back to a NumPy dot product for the n=1 corner.
+
+See [`PUBLISH.md`](PUBLISH.md) for the PyPI release flow.
+
+## Citation
+
+```
+@misc{mckenzie2026spectraldb,
+  title={spectraltm-db — Pinecone-shape local vector database on Spectral encoding},
+  author={Mc Kenzie, Gerald Enrique Nelson},
+  year={2026},
+  month={6},
+  day={29},
+  howpublished={Companion to spectraltm (Sparse Spectral Encoding)},
+  license={Apache-2.0},
+  note={Uses spectraltm https://pypi.org/project/spectraltm/ as the underlying engine.}
+}
+```
+
+**Metadata**
+
+- **Author:** Gerald Enrique Nelson Mc Kenzie
+- **Date:** 2026-06-29
+- **Repository:** [github.com/lordxmen2k/sparse-spectral-encoding](https://github.com/lordxmen2k/sparse-spectral-encoding)
+- **Engine:** [spectraltm](https://pypi.org/project/spectraltm/) on PyPI
+- **License:** Apache License 2.0
+- **Contact:** lordxmen2k@gmail.com
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).
+
+---
+
+Built as a Pinecone-shape wrapper around the [`spectraltm`](https://pypi.org/project/spectraltm/)
+crate. The two projects share zero source code per the explicit
+separation rule; the only cross-project coupling is the `spectraltm`
+PyPI dependency.