needtoknow 1.0.0__tar.gz

needtoknow-1.0.0/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+.env
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+dist/
+build/
+docs/_build/
+*.local.md

needtoknow-1.0.0/CHANGELOG.md

@@ -0,0 +1,107 @@
+# Changelog
+
+All notable changes are recorded here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/). From 1.0 on, the public API is stable.
+
+## [Unreleased]
+
+## [1.0.0] - 2026-10-16
+
+The first stable release. The public API in the
+[API reference](https://muzakkirhussain011.github.io/needtoknow/api.html) now follows semantic
+versioning. Everything in 1.0.0rc1 below is included; these are the changes since the release
+candidate.
+
+### Added
+
+- Tested on Python 3.14. Every Python version CPython still supports is supported, and a version is
+  dropped in the first minor release after its end of life (Python 3.10's is 2026-10-31).
+- A release procedure for maintainers in CONTRIBUTING.
+
+### Removed
+
+- `RetrievalEvent.extra`, a field nothing ever set. Audit lines no longer carry an empty `"extra"`.
+  Removed before 1.0 so that it is not frozen into the stable API; adding a field later is
+  compatible, removing one is not.
+
+## [1.0.0rc1] - 2026-10-16
+
+The first release candidate for 1.0. The public API in the
+[API reference](https://muzakkirhussain011.github.io/needtoknow/api.html) is what 1.0.0 will ship:
+changes before then are fixes, unless testing the release candidate shows that a breaking change is
+needed, in which case it will be announced in a second release candidate. Install it with
+`pip install --pre needtoknow`.
+
+### Added
+
+- `PermissionedRetriever` with `prefilter`, `postfilter` and `auto` strategies, sync
+  (`retrieve`) and async (`aretrieve`) entry points sharing one implementation.
+- Failing closed: any authorizer error, wrong number of decisions or non-boolean decision raises
+  `AuthorizationError` and returns nothing.
+- Re-check of pre-filtered results, with `RetrievalResult.violations` counting hits that a
+  backend returned despite the filter.
+- A backend-neutral filter language (`In`, `And`, `Or`) with reference semantics in `matches`;
+  empty `And`/`Or` are rejected because an empty conjunction would mean "allow everything".
+- `MetadataACLAuthorizer` (permissions stored on each chunk) and `FunctionAuthorizer` (your own
+  logic).
+- `InMemoryBackend` with deterministic keyword scoring, for tests and small corpora.
+- Property-based tests (Hypothesis) for the security guarantees on Python 3.10 to 3.13.
+- Contributor documentation: CONTRIBUTING, GOVERNANCE, CODE_OF_CONDUCT, SECURITY.
+- `needtoknow.backends.pgvector`: PostgreSQL + pgvector backend with the permission filter in the
+  query as JSON containment tests (GIN-indexable), iterative HNSW scans, `create_schema`, and a
+  native async search when given an async pool. Extra: `needtoknow[pgvector]`.
+- `needtoknow.backends.chroma`: Chroma backend translating filters into `where` clauses. It never
+  stores Chroma array metadata: in chromadb 1.5.x (local mode) a deleted collection's array
+  metadata surfaces on new records in other collections and is used by `where` filters
+  (chroma-core/chroma#7594). List values become
+  one boolean field each, and search results are rebuilt from an exact JSON copy of the metadata,
+  so the re-check always sees the true permissions. Extra: `needtoknow[chroma]`.
+- `needtoknow.authorizers.opa`: Open Policy Agent authorizer using only the standard library, with
+  per-chunk decisions and optional policy-provided filters; every failure fails closed. Example
+  policy in `examples/opa`.
+- `filter_to_dict` / `filter_from_dict` for a JSON form of filters.
+- `needtoknow.authorizers.openfga`: OpenFGA authorizer over the HTTP API (no extra dependency):
+  batched BatchCheck, ListObjects pre-filtering that falls back to post-filtering when the list may
+  be truncated, and the subject's groups sent as contextual tuples. Tested against a real OpenFGA
+  for random relationship graphs.
+- `needtoknow.langchain`: `NeedToKnowRetriever` (a LangChain retriever; refuses to run without a
+  subject) and `LangChainVectorStoreBackend` (any LangChain vector store, post-filtered).
+  Extra: `needtoknow[langchain]`.
+- `needtoknow.llamaindex`: `NeedToKnowRetriever`, `PermissionNodePostprocessor` and
+  `LlamaIndexBackend`. Extra: `needtoknow[llamaindex]`.
+- Runnable examples for LangChain and LlamaIndex, executed in CI.
+- `CachingAuthorizer`: caches pre-filters and per-chunk decisions (keyed by chunk id and
+  metadata) per subject for `ttl` seconds, with that as the documented revocation bound; errors and
+  malformed answers are never cached; `invalidate()`; sync and async.
+- Retrieval events: every call, including failed ones, produces a `RetrievalEvent` for
+  `observers` (subject id, strategy, returned ids, denied and violation counts, duration, error
+  type, optional HMAC of the query). Never query or chunk text. `JsonLinesAudit` writes them as
+  JSON lines; observer failures are logged and never affect retrieval.
+- `needtoknow.otel.OpenTelemetryObserver`: one span per retrieval with counts and outcomes, the
+  subject id only on request. Extra: `needtoknow[otel]`.
+- Documentation site (Sphinx, MyST, Furo) in `docs/`: getting started, concepts, a security model,
+  operations, one page per integration, extending needtoknow, benchmarks and an API reference.
+  CI builds it with warnings as errors, and every push to `main` publishes it to
+  <https://muzakkirhussain011.github.io/needtoknow/>. Extra: `needtoknow[docs]`.
+- Benchmark harness (`benchmarks/`): recall@k against exact ground truth, leaks, completeness and
+  latency for exact search, Chroma and pgvector across permission selectivity and layouts, with a
+  Kaggle notebook for real embeddings. CI runs a small version against real Chroma and PostgreSQL
+  and fails on any leak.
+- Conformance tests: every backend must return exactly the documents the reference semantics
+  select, for random metadata and nested filters, and never leak through the retriever. CI runs
+  them against real PostgreSQL, Chroma and OPA.
+
+### Changed
+
+- `RetrievalResult.complete` is documented precisely: it is `False` when needtoknow knows results
+  may be missing, and it trusts the backend's ranking, so it cannot see results an approximate index
+  failed to return. The benchmarks measure how often that happens.
+- Filter values must be strings or integers, and metadata values of other types (booleans,
+  floats) never match. This keeps every backend's answer identical to the reference.
+- The project is now `needtoknow`, a permission-aware retrieval library (ADR 0005). The earlier
+  Rampart gateway lives on as an example deployment in `examples/gateway`.
+
+[Unreleased]: https://github.com/muzakkirhussain011/needtoknow/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/muzakkirhussain011/needtoknow/compare/v1.0.0rc1...v1.0.0
+[1.0.0rc1]: https://github.com/muzakkirhussain011/needtoknow/releases/tag/v1.0.0rc1

needtoknow-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Syed Muzakkir Hussain
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

needtoknow-1.0.0/PKG-INFO

@@ -0,0 +1,297 @@
+Metadata-Version: 2.5
+Name: needtoknow
+Version: 1.0.0
+Summary: Permission-aware retrieval for RAG: every user gets only the chunks they are allowed to read.
+Project-URL: Homepage, https://github.com/muzakkirhussain011/needtoknow
+Project-URL: Repository, https://github.com/muzakkirhussain011/needtoknow
+Project-URL: Issues, https://github.com/muzakkirhussain011/needtoknow/issues
+Project-URL: Changelog, https://github.com/muzakkirhussain011/needtoknow/blob/main/CHANGELOG.md
+Project-URL: Documentation, https://muzakkirhussain011.github.io/needtoknow/
+Author: Syed Muzakkir Hussain
+License-Expression: MIT
+License-File: LICENSE
+Keywords: access-control,authorization,langchain,llamaindex,llm,openfga,rag,retrieval,security,vector-search
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Security
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: chroma
+Requires-Dist: chromadb>=1.5; extra == 'chroma'
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6.100; extra == 'dev'
+Requires-Dist: pytest>=8.2; extra == 'dev'
+Requires-Dist: ruff==0.17.1; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: furo>=2025.12.19; extra == 'docs'
+Requires-Dist: myst-parser>=5.1; extra == 'docs'
+Requires-Dist: sphinx>=9.1; extra == 'docs'
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=1.0; extra == 'langchain'
+Provides-Extra: llamaindex
+Requires-Dist: llama-index-core>=0.12; extra == 'llamaindex'
+Provides-Extra: otel
+Requires-Dist: opentelemetry-api>=1.30; extra == 'otel'
+Provides-Extra: pgvector
+Requires-Dist: psycopg[binary,pool]>=3.2; extra == 'pgvector'
+Description-Content-Type: text/markdown
+
+# needtoknow
+
+[![CI](https://github.com/muzakkirhussain011/needtoknow/actions/workflows/ci.yml/badge.svg)](https://github.com/muzakkirhussain011/needtoknow/actions/workflows/ci.yml)
+[![Docs](https://github.com/muzakkirhussain011/needtoknow/actions/workflows/docs.yml/badge.svg)](https://muzakkirhussain011.github.io/needtoknow/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/muzakkirhussain011/needtoknow/blob/main/LICENSE)
+![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)
+![Dependencies: none](https://img.shields.io/badge/core%20dependencies-none-brightgreen.svg)
+
+**Permission-aware retrieval for RAG.** Every user gets only the chunks they are allowed to read,
+whatever vector store, permission system or framework you use.
+
+```python
+from needtoknow import PermissionedRetriever, MetadataACLAuthorizer, Subject
+
+retriever = PermissionedRetriever(backend, MetadataACLAuthorizer())
+result = retriever.retrieve("What are this year's salary bands?", Subject("bob", groups=["engineering"]), k=5)
+# result.hits holds only chunks Bob may read. HR documents never reach the prompt.
+```
+
+**[Documentation](https://muzakkirhussain011.github.io/needtoknow/)** ·
+[Security model](https://muzakkirhussain011.github.io/needtoknow/security.html) ·
+[Benchmarks](https://muzakkirhussain011.github.io/needtoknow/benchmarks.html) ·
+[Changelog](https://github.com/muzakkirhussain011/needtoknow/blob/main/CHANGELOG.md)
+
+## Why
+
+A RAG assistant over company documents is only as safe as its retrieval step. If the vector search
+ignores who is asking, an engineer can ask the assistant about salaries and get HR documents
+summarised back. Every team building internal RAG has to solve this, and most solve it by hand,
+differently each time.
+
+`needtoknow` is that layer as a small, tested library:
+
+- **Works with what you have.** Wrap any search backend and any permission system. The core has
+  no dependencies.
+- **Filters in the right place.** When the permission system can describe "everything Bob may
+  read" as a filter, the filter goes *inside* the vector search, so results are the true top-k of
+  what Bob may read. When it cannot, candidates are checked after the search and the search widens
+  until enough permitted results are found.
+- **Fails closed.** If the permission service errors, times out or returns malformed answers, the
+  user gets an error, never unfiltered results.
+- **Defence in depth.** Pre-filtered results are checked again, so a backend bug that ignores the
+  filter cannot leak. Each result reports how many such violations it caught.
+
+## Install
+
+```bash
+pip install needtoknow
+```
+
+Python 3.10 or newer: every Python version CPython still supports, each dropped in the first minor
+release after its end of life. Adapters for external systems install as optional extras.
+
+## Quickstart
+
+```python
+from needtoknow import Document, InMemoryBackend, MetadataACLAuthorizer, PermissionedRetriever, Subject
+
+backend = InMemoryBackend(
+    [
+        Document("handbook", "Annual leave is 25 days.", {"allowed_groups": ["*"]}),
+        Document("salaries", "Salary bands for 2026 ...", {"allowed_groups": ["hr"]}),
+        Document("runbook", "Roll back a failed deploy ...", {"allowed_groups": ["engineering"]}),
+    ]
+)
+retriever = PermissionedRetriever(backend, MetadataACLAuthorizer())
+
+result = retriever.retrieve("leave salary bands", Subject("bob", groups=["engineering"]), k=3)
+print([hit.id for hit in result.hits])  # ['handbook']
+print(result.strategy, result.complete)  # prefilter True
+```
+
+`aretrieve` is the async equivalent, with the same arguments and guarantees.
+
+## How it works
+
+| Strategy | When | Cost | Result |
+|---|---|---|---|
+| `prefilter` | The authorizer can return a filter and the backend supports filters | One search | The exact top-k of what the subject may read |
+| `postfilter` | Any authorizer | Searches `k × overfetch` candidates, widening up to `max_candidates` | Exact when enough permitted candidates exist; `complete=False` otherwise |
+| `auto` (default) | Picks `prefilter` when both sides support it | | |
+
+Every call returns a `RetrievalResult` with the hits plus `strategy`, `examined`, `denied`,
+`violations` (pre-filtered hits the re-check rejected, which should always be 0) and `complete`.
+
+## Integrations
+
+Each integration is an optional extra, so the core stays dependency-free.
+
+**PostgreSQL + pgvector** (`pip install "needtoknow[pgvector]"`, pgvector 0.8+). The permission
+filter runs inside the SQL query as JSON containment tests, served by a GIN index, and iterative
+index scans keep HNSW returning k rows under selective filters.
+
+```python
+from psycopg_pool import ConnectionPool
+
+from needtoknow.backends.pgvector import PgVectorBackend, create_schema
+
+pool = ConnectionPool("postgresql://localhost/app")
+create_schema(pool, dim=768)
+backend = PgVectorBackend(pool, embed=my_embedding_model)  # embed: str -> list[float]
+backend.add(documents)
+```
+
+**Chroma** (`pip install "needtoknow[chroma]"`). The filter becomes Chroma's `where` clause. Each
+value in a permission list is stored as its own field rather than as a Chroma array, and an exact
+copy of the metadata is kept for the re-check: in chromadb 1.5.x, deleting a collection leaves its
+array metadata behind for new records in other collections to pick up
+([chroma-core/chroma#7594](https://github.com/chroma-core/chroma/issues/7594)), which for
+permissions would be a leak.
+
+```python
+import chromadb
+
+from needtoknow.backends.chroma import ChromaBackend
+
+collection = chromadb.PersistentClient(path="./chroma").get_or_create_collection("docs")
+backend = ChromaBackend(collection)  # uses the collection's embedding function
+```
+
+**Open Policy Agent** (no extra dependency). Your Rego policy returns one decision per chunk, and
+can optionally return a filter so the vector store does the filtering. A complete policy is in
+[`examples/opa/needtoknow.rego`](https://github.com/muzakkirhussain011/needtoknow/blob/main/examples/opa/needtoknow.rego).
+
+```python
+from needtoknow.authorizers.opa import OPAAuthorizer
+
+authorizer = OPAAuthorizer("http://localhost:8181", filter_path="needtoknow/filter")
+```
+
+**OpenFGA** (no extra dependency). Relationship-based access control in the Google Zanzibar style.
+Checks run in batches; for pre-filtering, ListObjects returns every document the user may view,
+and if that list may have been truncated the retriever post-filters instead. The subject's groups
+can be sent as contextual tuples, so memberships from your identity provider need no syncing.
+
+```python
+from needtoknow.authorizers.openfga import OpenFGAAuthorizer
+
+authorizer = OpenFGAAuthorizer("http://localhost:8080", store_id, resource_field="doc_id")
+```
+
+**LangChain** (`pip install "needtoknow[langchain]"`). `NeedToKnowRetriever` is a LangChain
+retriever, so it works in chains, agents and LangGraph tools; `LangChainVectorStoreBackend` turns
+any LangChain vector store into a needtoknow backend (post-filtered).
+
+```python
+from needtoknow.langchain import NeedToKnowRetriever
+
+retriever = NeedToKnowRetriever(retriever=PermissionedRetriever(backend, authorizer))
+docs = retriever.invoke("What are this year's salary bands?", subject=Subject("bob", ["sales"]))
+```
+
+**LlamaIndex** (`pip install "needtoknow[llamaindex]"`). `NeedToKnowRetriever` for query and chat
+engines, `PermissionNodePostprocessor` for existing pipelines, and `LlamaIndexBackend` to wrap an
+existing index. Runnable examples for both frameworks are in [`examples/`](https://github.com/muzakkirhussain011/needtoknow/tree/main/examples).
+
+Filter values are strings or integers, the types permission identifiers use. That keeps every
+backend's answer identical to the reference semantics, which the test suite checks for each one.
+
+## Caching, audit and tracing
+
+```python
+from needtoknow import CachingAuthorizer, JsonLinesAudit
+from needtoknow.otel import OpenTelemetryObserver  # pip install "needtoknow[otel]"
+
+retriever = PermissionedRetriever(
+    backend,
+    CachingAuthorizer(authorizer, ttl=30),  # a revoked permission stops granting access within 30 s
+    observers=[JsonLinesAudit(audit_file), OpenTelemetryObserver()],  # audit_file: an open text file
+    audit_key=b"...",  # optional: an HMAC of each query, so repeats can be matched without storing them
+)
+```
+
+`CachingAuthorizer` caches pre-filters and per-chunk decisions per subject for `ttl` seconds and
+never caches an error; `invalidate()` drops decisions after a change you know about. Every
+retrieval, including failed ones, produces one `RetrievalEvent` (who asked, strategy, returned ids,
+denied and violation counts, duration, error type). Events never contain query or chunk text, and
+an observer that fails is logged without affecting retrieval.
+
+## Bring your own backend or permission system
+
+A backend is any object with a `search` method:
+
+```python
+class MyBackend:
+    supports_filters = True  # set False if search() ignores `filter`
+
+    def search(self, query: str, k: int, filter=None) -> list[Hit]:
+        """Return hits best first. Translate `filter` (In / And / Or) into your store's filter."""
+```
+
+An authorizer answers two questions:
+
+```python
+class MyAuthorizer:
+    def check(self, subject, hits, action="read") -> list[bool]:
+        """One decision per hit, in order. Raise on any doubt: the retriever fails closed."""
+
+    def filter_for(self, subject, action="read"):
+        """A Filter selecting everything the subject may read, or None if you cannot express one."""
+```
+
+Either can add `async` versions (`asearch`, `acheck`, `afilter_for`); otherwise the sync methods
+run in a thread under `aretrieve`. `FunctionAuthorizer` wraps a plain function.
+
+## Testing approach
+
+The guarantees are tested as properties with [Hypothesis](https://hypothesis.readthedocs.io):
+for randomly generated corpora, permissions, users and queries, every strategy must
+
+1. never return a chunk the user may not read,
+2. return exactly the top-k of what the user may read (checked against an independent oracle),
+3. stay safe when the backend ignores the filter, and
+4. give identical results through `retrieve` and `aretrieve`.
+
+## Benchmarks
+
+On 57,600 real chunks (BEIR FiQA, Qwen3-Embedding-0.6B), across 33,000 retrievals, no needtoknow
+strategy returned a single forbidden chunk. Filtering inside the search kept recall@10 at 0.949 or
+better on pgvector for users who may read anywhere from 0.06% to 100% of the corpus, including when
+what they may read is far from what they ask about. Filtering after a top-10 search found almost
+nothing for selective permissions. Method, tables and caveats: [Benchmarks](https://muzakkirhussain011.github.io/needtoknow/benchmarks.html).
+
+| Readable (far from the query) | Naive top-k, then filter | needtoknow postfilter | needtoknow prefilter (pgvector) |
+|---|---|---|---|
+| 0.9% | 0.000 | 0.048 | 0.995 |
+| 4.8% | 0.000 | 0.314 | 0.950 |
+| 20% | 0.041 | 0.974 | 0.949 |
+
+## Stability and roadmap
+
+1.0 is the stable base: the core, the pgvector and Chroma backends, the OPA and OpenFGA authorizers,
+the LangChain and LlamaIndex integrations, caching, audit events and tracing. The public API in the
+[API reference](https://muzakkirhussain011.github.io/needtoknow/api.html) follows semantic
+versioning: no breaking change without a major version and a deprecation period first.
+
+Next, in rough order: permission sync from Google Drive, SharePoint and Confluence; Qdrant and
+Elasticsearch backends; SpiceDB and Cerbos authorizers. Contributions are welcome; see
+[CONTRIBUTING.md](https://github.com/muzakkirhussain011/needtoknow/blob/main/CONTRIBUTING.md).
+
+## Example deployment
+
+[`examples/gateway`](https://github.com/muzakkirhussain011/needtoknow/tree/main/examples/gateway) is a complete, self-hosted, OpenAI-compatible gateway built
+on the same ideas: OPA policies, pgvector hybrid search with the ACL inside the query, PII
+pseudonymisation, prompt-injection quarantine and a hash-chained audit log. It is being ported onto
+this library.
+
+## License
+
+[MIT](https://github.com/muzakkirhussain011/needtoknow/blob/main/LICENSE)