langchain-okf 0.2.0__tar.gz

langchain_okf-0.2.0/.gitignore

@@ -0,0 +1,37 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+dist/
+build/
+.eggs/
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+.DS_Store
+
+# Testing & Benchmarks
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+htmlcov/
+.coverage
+coverage.xml
+benchmarks/bundle_*/
+benchmarks/**/.chroma/
+**/.okf_cache.json
+**/.okf_index_hash
+
+# Distribution
+*.tar.gz
+*.whl

langchain_okf-0.2.0/CHANGELOG.md

@@ -0,0 +1,109 @@
+# Changelog
+
+<!-- version list -->
+
+## v0.2.0 (2026-11-15)
+
+### Features
+
+- Prune dependencies, add retriever over-fetching, and activate CI/CD workflows
+  ([#1](https://github.com/AyushDey/langchain-okf/pull/1),
+  [`be14087`](https://github.com/AyushDey/langchain-okf/commit/be140877197a61f42a24a94642d694a07ab21206))
+
+
+## Unreleased
+
+## 0.2.0 (2026-11-14)
+
+### Added
+- Dependency pruning: pruned unused dependencies (`langchain-ollama`, `langchain-text-splitters`) from core dependencies, made `langchain-chroma` optional, and kept core lean (`langchain-core`, `pypdf`, `pyyaml`).
+- Over-fetching in `OKFRetriever`: dynamically scales candidate pool when post-filters (`trust_tier`, `stale_after`, `status`, `tags`) are active to prevent result starvation.
+- Thread-safe and non-blocking agent memory tools: `OKFWriteConceptTool` and `OKFVerifyConceptTool` wrapped in reentrant write locks and native async `_arun` implementations via `asyncio.to_thread`.
+- In-memory cross-link caching: `OKFConcept.cross_links()` caches parsed markdown references to eliminate repeated regex scans across graph hops.
+- Automated CI/CD workflows: GitHub Actions for multi-python testing matrix (3.10–3.13) and OIDC Trusted Publishing to PyPI.
+- Millisecond-level reruns via caching: `OKFBundleGenerator` hashes content chunks (SHA-256) and caches LLM results in `.okf_cache.json` in the bundle directory. If chunks or images are unchanged, extraction runs in milliseconds and uses 0 LLM API calls.
+- Parallel LLM extraction: `OKFBundleGenerator` executes LLM calls concurrently using thread pools (controlled via `max_concurrency`, default `4`) to speed up first-run extraction.
+- Decorative image filtering: `OKFBundleGenerator` automatically skips analyzing decorative graphics, logos, banners, headers, and footers to save API costs and processing time. Uses a combination of size/aspect-ratio heuristics and multimodal LLM classification.
+- Asset directory cleanup: `OKFBundleGenerator` removes any unused/irrelevant extracted images from the bundle's `assets/` directory at the end of the run.
+- Model-agnostic chunking: `OKFBundleGenerator` accepts a `chunk_size` at
+  construction, and `generate(chunk_size=...)` overrides per call. When
+  unset, a conservative default is used that is reliable on any model, local
+  or hosted — no guessing from model names or context windows.
+- Output-truncation recovery: when a model hits its output-token limit
+  mid-answer (`finish_reason == "length"` and equivalents), the chunk is split
+  in half and re-extracted recursively so no concepts are lost.
+- Concept normalization: every extracted concept is guaranteed to have the
+  keys the pipeline depends on (`category`, `filename`, `type`, `title`,
+  `body`); concepts with no usable content are dropped.
+- JSON salvage: complete concept objects are recovered from truncated or
+  malformed JSON arrays (unescaped newlines, trailing prose, cut-off output).
+- Oversized-passage handling: a single paragraph/table larger than the chunk
+  budget is hard-split so no call ever exceeds the model's limits.
+- Transient-error resilience: rate limits and 5xx provider errors (capacity,
+  timeouts, gateway failures) are retried with exponential backoff + jitter,
+  and the default attempt count was raised so a brief provider outage can
+  recover instead of failing instantly. Verbose multi-line SDK/provider errors
+  (e.g. Pydantic validation dumps) are collapsed to a single readable line with
+  the error type and any HTTP status code.
+
+### Changed
+- Progress feedback: the chunk-size line notes whether the size is the default
+  or custom; per-chunk lines show a running elapsed clock; zero-concept chunks
+  and images are reported distinctly (pointing to the warnings above); image
+  extraction is announced with a note that it needs a multimodal LLM; and the
+  final "no concepts" warning suggests concrete next steps.
+
+### Fixed
+- `FileExistsError` / nested-directory bug when a model returned a `category`
+  or `filename` containing slashes (e.g. `research/r_swa`), a `.md` suffix, or
+  other path characters. Category and filename are now sanitized to a single
+  flat, lowercased path segment, and writes disambiguate genuine collisions
+  (file-vs-directory, or two concepts resolving to the same path) with a
+  numeric suffix. Index links use the same sanitized names.
+- `KeyError: 'category'` (and other missing keys) when building the
+  incremental-enrichment prompt from previously stored concepts — the prompt
+  now tolerates concepts with missing fields, and normalization prevents such
+  concepts from being stored in the first place.
+- `shutil.SameFileError` when Docling-extracted images already live in the
+  bundle's `assets/` directory — `_copy_image_asset` now skips same-path copies.
+- Message content returned as a list of content blocks (LangChain v1 chat
+  models) is now coerced to text before JSON parsing.
+
+## 0.1.0 (2026-06-27)
+
+Initial release.
+
+### Components
+- `OKFBundleLoader` — Load OKF bundles as LangChain Documents with type/tag filtering
+- `OKFKnowledgeGraph` — Directed graph from OKF cross-links with neighbor/backlink/subgraph queries
+- `OKFRetriever` — Graph-aware retriever combining vector similarity with cross-link expansion
+- `OKFToolkit` — 4 agent tools (list, read, related, search) with source citations
+- `OKFBundleGenerator` — Generate OKF bundles from PDFs, text, images, CSVs, URLs using an LLM
+  - Auto-detect incremental mode: re-running on an existing bundle merges new content
+  - Docling (MIT) as preferred extraction backend — structured text, tables, images, OCR
+  - Fallback chain: Docling → pymupdf → pypdf (choose by license needs)
+  - Domain-aware extraction with 7 presets + custom free-text domains
+  - Error recovery with automatic retry on LLM failures
+  - Progress logging with `verbose` flag and custom `logger` support
+  - Smart chunking at paragraph boundaries with tiny-chunk merging
+  - Anti-hallucination prompts strictly grounded in source material
+  - PDF image extraction with classification (diagram/table/chart/photo)
+  - Source citations embedded in every concept for traceability
+  - Works fully offline after initial model download
+- `validate_bundle()` — Check OKF v0.1 conformance of a bundle directory
+
+### Integrations
+- Embedding providers: OpenAI, Ollama, OpenRouter
+- Vector stores: Chroma, pgvector, Pinecone, Qdrant, FAISS, Milvus
+- Agent framework: LangChain v1 `create_agent` with middleware support
+- Source extraction: Docling (MIT, recommended), pymupdf (AGPL), pypdf (BSD)
+- Async support: `OKFRetriever` and all toolkit tools support async (`ainvoke`, `_arun`)
+
+### Infrastructure
+- Built with hatchling, compatible with uv and pip
+- PEP 735 `[dependency-groups]` for dev dependencies
+- `py.typed` marker for type-checking support
+- 90+ unit tests with mock LLM (no API keys needed)
+- Integration test stubs (run with `OPENAI_API_KEY` set)
+- Python 3.10+ (aligned with LangChain v1)
+- All core dependencies MIT/BSD licensed

langchain_okf-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ayush Dey
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

langchain_okf-0.2.0/PKG-INFO

@@ -0,0 +1,367 @@
+Metadata-Version: 2.5
+Name: langchain-okf
+Version: 0.2.0
+Summary: LangChain integration for Google's Open Knowledge Format (OKF)
+Project-URL: Repository, https://github.com/AyushDey/langchain-okf
+Project-URL: Documentation, https://github.com/AyushDey/langchain-okf#readme
+Project-URL: Changelog, https://github.com/AyushDey/langchain-okf/blob/main/CHANGELOG.md
+Project-URL: Bug Tracker, https://github.com/AyushDey/langchain-okf/issues
+Author: Ayush Dey
+License-Expression: MIT
+License-File: LICENSE
+Keywords: knowledge-base,knowledge-graph,langchain,okf,open-knowledge-format,rag,retrieval-augmented-generation
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: langchain-core>=0.3.0
+Requires-Dist: pypdf>=4.0.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: agent
+Requires-Dist: langchain-text-splitters>=0.3.0; extra == 'agent'
+Requires-Dist: langchain>=1.0.0; extra == 'agent'
+Provides-Extra: all
+Requires-Dist: docling>=2.100.0; extra == 'all'
+Requires-Dist: faiss-cpu>=1.7.0; extra == 'all'
+Requires-Dist: langchain-chroma>=0.2.0; extra == 'all'
+Requires-Dist: langchain-milvus>=0.2.0; extra == 'all'
+Requires-Dist: langchain-ollama>=0.3.0; extra == 'all'
+Requires-Dist: langchain-openai>=0.3.0; extra == 'all'
+Requires-Dist: langchain-openrouter>=0.2.0; extra == 'all'
+Requires-Dist: langchain-pinecone>=0.2.0; extra == 'all'
+Requires-Dist: langchain-postgres>=0.0.12; extra == 'all'
+Requires-Dist: langchain-qdrant>=0.2.0; extra == 'all'
+Requires-Dist: langchain-text-splitters>=0.3.0; extra == 'all'
+Requires-Dist: langchain>=1.0.0; extra == 'all'
+Provides-Extra: chroma
+Requires-Dist: langchain-chroma>=0.2.0; extra == 'chroma'
+Provides-Extra: dev
+Requires-Dist: langchain-chroma>=0.2.0; extra == 'dev'
+Requires-Dist: langchain-ollama>=0.3.0; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Provides-Extra: faiss
+Requires-Dist: faiss-cpu>=1.7.0; extra == 'faiss'
+Provides-Extra: loaders
+Requires-Dist: docling>=2.100.0; extra == 'loaders'
+Provides-Extra: loaders-legacy
+Requires-Dist: beautifulsoup4>=4.12; extra == 'loaders-legacy'
+Requires-Dist: docx2txt>=0.8; extra == 'loaders-legacy'
+Requires-Dist: langchain-community>=0.4.0; extra == 'loaders-legacy'
+Requires-Dist: pymupdf>=1.24.0; extra == 'loaders-legacy'
+Requires-Dist: pypdf>=4.0; extra == 'loaders-legacy'
+Provides-Extra: milvus
+Requires-Dist: langchain-milvus>=0.2.0; extra == 'milvus'
+Provides-Extra: ollama
+Requires-Dist: langchain-ollama>=0.3.0; extra == 'ollama'
+Provides-Extra: openai
+Requires-Dist: langchain-openai>=0.3.0; extra == 'openai'
+Provides-Extra: openrouter
+Requires-Dist: langchain-openrouter>=0.2.0; extra == 'openrouter'
+Provides-Extra: pgvector
+Requires-Dist: langchain-postgres>=0.0.12; extra == 'pgvector'
+Provides-Extra: pinecone
+Requires-Dist: langchain-pinecone>=0.2.0; extra == 'pinecone'
+Provides-Extra: qdrant
+Requires-Dist: langchain-qdrant>=0.2.0; extra == 'qdrant'
+Provides-Extra: vectorstore
+Requires-Dist: langchain-chroma>=0.2.0; extra == 'vectorstore'
+Requires-Dist: langchain-openai>=0.3.0; extra == 'vectorstore'
+Description-Content-Type: text/markdown
+
+# langchain-okf
+
+[![PyPI version](https://img.shields.io/pypi/v/langchain-okf.svg)](https://pypi.org/project/langchain-okf/)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![OKF v0.2 Compliant](https://img.shields.io/badge/OKF-v0.2%20Compliant-green.svg)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**Plug-and-play, self-improving memory layer and knowledge graph retriever for LangChain agents**, fully conforming to Google's [Open Knowledge Format (OKF v0.2)](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md).
+
+Delivers **sub-second cold loads, search, retrieval, real-time memory writes, and trust verification** with both local models (LM Studio, Ollama) and cloud providers (OpenAI, Anthropic, Gemini).
+
+---
+
+## Highlights
+
+- ⚡ **Sub-Second Performance**:
+  - **Cold Index Reload in 7–10 ms** (via `.okf_index_hash` content fingerprinting — bypasses re-embedding).
+  - **In-Memory Graph Search in 0.02–0.4 ms** (instant bidirectional link traversals).
+  - **Dynamic Agent Memory Writes in 0.69 ms** ([`OKFWriteConceptTool`](file:///Users/ayushdey/Desktop/langchain-okf/langchain_okf/toolkit.py#L320)).
+  - **Trust Verification in 0.78 ms** ([`OKFVerifyConceptTool`](file:///Users/ayushdey/Desktop/langchain-okf/langchain_okf/toolkit.py#L398)).
+  - **Incremental Bundle Re-generation in 5–8 ms** (via SHA-256 `.okf_cache.json`).
+- 🛡️ **OKF v0.2 Spec Conformance**:
+  - **Trust Tiers**: Automatic derivation: `human-reviewed` > `machine-confirmed` > `unverified`.
+  - **Provenance Tracking**: `sources: [{ resource, title, author }]` preserved through ingestion.
+  - **Lifecycle & Freshness**: `status: stable | draft | deprecated` and `stale_after` automatic eviction.
+  - **Audit Logging**: Root `index.md` (`okf_version: "0.2"`) and sequential `log.md` event logs.
+- 🧠 **Self-Improving Agent Memory**:
+  - Agents can read, search, synthesize, write new concepts, and verify knowledge during execution.
+- 📈 **Extreme Scalability (1 to 100s of PDFs)**:
+  - Parallel LLM chunk extraction with thread-pool concurrency (`max_concurrency`).
+  - Native PDF ingestion (`pypdf`, `docling`, or `pymupdf`) with lazy loading.
+  - Deduplication and prompt optimization to eliminate quadratic prompt bloat.
+
+---
+
+## Benchmark Results (LM Studio Live Server)
+
+Tested on local LM Studio (`http://127.0.0.1:1234/v1`) using `google/gemma-4-12b` and `text-embedding-nomic-embed-text-v1.5` (768-dim):
+
+| Operation | Latency | Sub-Second? | Mechanism |
+| :--- | :---: | :---: | :--- |
+| **LM Studio Embedding API Call** | `73.4 ms` | ✅ YES | Direct `/v1/embeddings` HTTP batch |
+| **Cached Bundle Re-generation** | `8.46 ms` | ✅ YES | SHA-256 chunk cache in `.okf_cache.json` |
+| **Persisted Index Reload** | `7.94 ms` | ✅ YES | Fingerprint caching via `.okf_index_hash` (92.9x speedup) |
+| **Vector Similarity Search** | `25.2 ms` | ✅ YES | ChromaDB cosine similarity + graph hop |
+| **Graph In-Memory Search** | `0.026 ms` | ✅ YES | In-memory `OKFKnowledgeGraph` traversal |
+| **Agent Real-Time Memory Write** | `0.70 ms` | ✅ YES | Atomic file creation + graph update |
+| **Agent Trust Tier Upgrade** | `0.83 ms` | ✅ YES | In-place frontmatter mutation + `log.md` event |
+
+---
+
+## Install
+
+```bash
+pip install langchain-okf
+```
+
+Install optional integrations as needed:
+
+```bash
+# Embedding & LLM providers
+pip install "langchain-okf[openai]"       # OpenAI / LM Studio / Local OpenAI-compatible
+pip install "langchain-okf[ollama]"       # Ollama
+pip install "langchain-okf[openrouter]"   # OpenRouter
+
+# Vector stores
+pip install "langchain-okf[chroma]"       # Chroma (file-based, default)
+pip install "langchain-okf[pgvector]"     # PostgreSQL + pgvector
+pip install "langchain-okf[pinecone]"     # Pinecone
+pip install "langchain-okf[qdrant]"       # Qdrant
+
+# PDF & Document loaders
+pip install pypdf                         # Fast, BSD-licensed PDF ingestion
+pip install "langchain-okf[loaders]"      # Docling (MIT) — structured text + tables + OCR
+```
+
+---
+
+## Quick Start
+
+### 1. Sub-Second Retriever with Local Models (LM Studio or Ollama)
+
+```python
+from langchain_openai import OpenAIEmbeddings
+from langchain_okf import OKFRetriever
+
+# Connect to LM Studio local server
+embeddings = OpenAIEmbeddings(
+    base_url="http://127.0.0.1:1234/v1",
+    api_key="lm-studio",
+    model="text-embedding-nomic-embed-text-v1.5",
+)
+
+# Loads in ~7 ms on subsequent runs (skips re-embedding unchanged bundles)
+retriever = OKFRetriever.from_bundle(
+    "./my_knowledge/",
+    embeddings=embeddings,
+    persist_directory="./my_knowledge/.chroma",
+    min_trust_tier="machine-confirmed",  # Filter out unverified hallucinations
+    exclude_stale=True,                  # Automatically skip expired concepts
+    expand_depth=1,                      # Traverse markdown cross-links
+)
+
+docs = retriever.invoke("What are the core context engineering boundaries?")
+for doc in docs:
+    print(f"[{doc.metadata.get('trust_tier')}] {doc.metadata.get('title')}")
+```
+
+---
+
+### 2. Self-Improving Agent Memory Layer
+
+Equip your agent with the complete OKF v0.2 toolkit, enabling both retrieval and active memory authoring:
+
+```python
+from langchain.agents import create_agent
+from langchain_openai import ChatOpenAI
+from langchain_okf import OKFToolkit
+
+toolkit = OKFToolkit.from_bundle(
+    "./my_knowledge/",
+    actor="agent/customer-support-v1",
+    include_write_tools=True,
+)
+tools = toolkit.get_tools()
+
+agent = create_agent(
+    model=ChatOpenAI(model="gpt-4o", temperature=0),
+    tools=tools,
+    system_prompt="You are an autonomous agent with persistent OKF memory.",
+)
+```
+
+The agent receives 6 specialized tools:
+
+| Tool | Action | Latency | Purpose |
+| :--- | :--- | :---: | :--- |
+| `okf_search` | Keyword & title search | `< 0.1 ms` | Exact lookups across titles, tags, and bodies |
+| `okf_read_concept` | Read full concept | `< 0.1 ms` | Inspect full body, metadata, trust tier, and sources |
+| `okf_related_concepts` | Follow graph edges | `< 0.1 ms` | Inspect outgoing links and incoming backlinks |
+| `okf_list_concepts` | Filtered browsing | `< 0.1 ms` | Filter by `type:`, `tag:`, `trust:`, or `status:` |
+| `okf_write_concept` | **Write new memory** | `0.69 ms` | Agent saves learned rules, edge cases, and facts |
+| `okf_verify_concept` | **Upgrade trust tier** | `0.78 ms` | Mark verified (`human-reviewed` or `machine-confirmed`) |
+
+---
+
+### 3. Generate Knowledge Bundles from PDFs and Documents
+
+Ingest PDFs, markdown, text files, or CSVs and convert them into structured OKF concepts:
+
+```python
+from langchain_openai import ChatOpenAI
+from langchain_okf import OKFBundleGenerator
+
+generator = OKFBundleGenerator(
+    llm=ChatOpenAI(model="gpt-4o", temperature=0),
+    output_dir="./my_knowledge/",
+    bundle_name="Technical Architecture",
+    domain="Microservices, distributed caching, and auth",
+    stale_after_days=90,     # OKF v0.2 lifecycle management
+    max_concurrency=8,       # Concurrently extract chunks
+)
+
+generator.add_sources([
+    "architecture_doc.pdf",
+    "deployment_runbook.md",
+    "service_api.txt",
+])
+
+# Cold run extracts concepts; subsequent runs use SHA-256 chunk cache (5-8 ms)
+bundle_path = generator.generate()
+```
+
+---
+
+### 4. Conformance Audit (OKF v0.2 Spec §11)
+
+Verify bundle health against the official OKF specification:
+
+```python
+from langchain_okf import validate_bundle
+
+# Strict mode validates ISO 8601 UTC offsets, actor conventions, and sources
+issues = validate_bundle("./my_knowledge/", strict=True)
+if not issues:
+    print("✅ 100% OKF v0.2 spec compliant")
+else:
+    for issue in issues:
+        print(f"⚠️ {issue}")
+```
+
+---
+
+## Scalability: Handling 20 to 100s of PDFs
+
+`langchain-okf` is engineered from the ground up for high-throughput multi-document ingestion:
+
+```
+[20–100+ PDFs] ──▶ Direct pypdf Stream (1–6s) ──▶ SHA-256 Chunk Hash Check
+                                                        │
+                      ┌─────────────────────────────────┴─────────────────────────────────┐
+                      ▼                                                                   ▼
+             Uncached Chunks (New)                                               Cached Chunks (Unchanged)
+          Multi-Worker Thread Pool                                              Instant In-Memory Load (< 10 ms)
+        (Cloud LLM / local vLLM / LM Studio)
+                      │                                                                   │
+                      └─────────────────────────────────┬─────────────────────────────────┘
+                                                        ▼
+                                         OKF v0.2 Markdown Knowledge Bundle
+                                                        │
+                                                        ▼
+                                   .okf_manifest.json + .okf_index_hash
+                                                        │
+                      ┌─────────────────────────────────┴─────────────────────────────────┐
+                      ▼                                                                   ▼
+             Initial Build (Cold)                                                Incremental Sync (Hot)
+        Batch Embeddings + ChromaDB                                         Embeds ONLY Delta ΔN Concepts
+           O(N log N) or O(N/P)                                             O(ΔN log N) -> O(1) for single additions
+```
+
+1. **Streaming PDF Extraction**:
+   - Uses zero-overhead `pypdf` stream processing to extract text and pages in memory.
+   - Extracts ~20 PDFs (200 pages) in **1.2s** and ~100 PDFs (1,000 pages) in **~6s**.
+2. **$O(1)$ Hash Caching (`.okf_cache.json`)**:
+   - Each text chunk is hashed with SHA-256.
+   - If 100 PDFs are ingested and only 2 are updated, only the modified chunks call the LLM. The other 98 PDFs load from cache in **milliseconds**.
+3. **Multi-Worker Concurrency (`max_concurrency`)**:
+   - Extraction runs on a configurable `ThreadPoolExecutor`.
+   - With cloud models or local vLLM / high-throughput endpoints, extraction scales linearly across available workers.
+4. **$O(\Delta N)$ Incremental Vector Indexing**:
+   - Tracks a per-concept content manifest (`.okf_manifest.json`).
+   - When a bundle is updated with new or modified documents, `OKFRetriever` detects the exact diff and embeds/indexes **only the changed concepts ($\Delta N$)** instead of rebuilding the whole index.
+   - For adding a single concept or document, time complexity drops from $O(N \log N)$ to **$O(1)$** (~15–20 ms).
+5. **Persistent Fingerprint Reload (`.okf_index_hash`)**:
+   - For unchanged bundles, `OKFRetriever` reattaches to the persisted vector store in **7–15 ms** ($O(1)$) regardless of whether the bundle contains 10 or 10,000 concepts.
+6. **Memory Footprint**:
+   - In-memory graph of 10,000 concepts uses less than 25 MB of RAM, with sub-millisecond Dijkstra and BFS hop lookups.
+
+---
+
+## Dynamic Benchmark Tool
+
+A built-in benchmark harness lets you evaluate ingestion and retrieval performance on your own files:
+
+```bash
+# Benchmark your own PDFs with LM Studio
+uv run python benchmarks/run_benchmark_lmstudio.py paper1.pdf paper2.pdf
+
+# Benchmark an entire directory
+uv run python benchmarks/run_benchmark_lmstudio.py ./my_pdf_folder/
+
+# Run with full live LLM extraction
+uv run python benchmarks/run_benchmark_lmstudio.py report.pdf --live-llm
+```
+
+---
+
+## Components
+
+| Component | Type | Description |
+| :--- | :--- | :--- |
+| `OKFBundleGenerator` | Generator | Multi-format ingestion (PDF, MD, TXT, CSV) with SHA-256 caching and concurrency |
+| `OKFBundleLoader` | Document Loader | Generator-based lazy loading (`lazy_load`) with trust and staleness filters |
+| `OKFRetriever` | Retriever | Hybrid vector similarity + cross-link expansion + persistent disk caching |
+| `OKFKnowledgeGraph` | Graph Engine | Directed graph for sub-millisecond edge navigation and backlinks |
+| `OKFToolkit` | Agent Toolkit | 6 tools for search, graph navigation, atomic writes, and trust verification |
+| `validate_bundle` | Validator | Strict validator auditing OKF v0.2 schema, actors, ISO 8601 dates, and sources |
+
+---
+
+## Development & Testing
+
+To run tests and linters locally:
+
+```bash
+uv sync
+uv run pytest tests/ -v
+uv run ruff check langchain_okf/
+uv run mypy langchain_okf/
+```
+
+---
+
+## License
+
+MIT © Ayush Dey