knolo-core 0.2.0 → 0.2.1

package/DOCS.md

@@ -0,0 +1,355 @@
+
+# DOCS.md — KnoLo Core
+
+> Deterministic, embedding-free retrieval and portable knowledge packs.
+
+## Table of Contents
+
+1. [What is KnoLo Core?](#what-is-knolo-core)
+2. [Quickstart](#quickstart)
+3. [Concepts](#concepts)
+4. [Building Packs](#building-packs)
+5. [Querying & Results](#querying--results)
+6. [LLM Context Patches](#llm-context-patches)
+7. [Advanced Retrieval Controls](#advanced-retrieval-controls)
+8. [Pack Format (Spec)](#pack-format-spec)
+9. [Performance & Tuning](#performance--tuning)
+10. [React Native / Expo Notes](#react-native--expo-notes)
+11. [Testing & QA](#testing--qa)
+12. [Migration 0.1.x → 0.2.0](#migration-01x--020)
+13. [Security & Privacy](#security--privacy)
+14. [FAQ](#faq)
+15. [Glossary](#glossary)
+16. [Versioning & Releases](#versioning--releases)
+
+---
+
+## What is KnoLo Core?
+
+KnoLo Core packages your corpus into a single `.knolo` file and performs **deterministic lexical retrieval**—no embeddings or vector DBs. It’s designed for **local-first, offline** LLM use.
+
+**Key properties**
+
+* **Deterministic**: phrase enforcement, proximity scoring, heading boosts
+* **Duplicate-free**: near-duplicate suppression + MMR diversity
+* **Portable**: single-pack file; Node, browsers, Expo
+* **LLM-ready**: outputs structured **Context Patches**
+
+---
+
+## Quickstart
+
+### Install
+
+```bash
+npm install knolo-core
+```
+
+### Minimal example
+
+```ts
+import { buildPack, mountPack, query, makeContextPatch } from "knolo-core";
+
+const docs = [
+  { id: "guide", heading: "React Native Bridge", text: "The bridge sends messages between JS and native. You can throttle events..." },
+  { id: "throttle", heading: "Throttling", text: "Throttling reduces frequency of events..." }
+];
+
+const bytes = await buildPack(docs);
+const kb = await mountPack({ src: bytes });
+const hits = query(kb, '"react native bridge" throttling', { topK: 5 });
+const patch = makeContextPatch(hits, { budget: "small" });
+```
+
+### CLI build
+
+```bash
+# input docs.json -> output knowledge.knolo
+npx knolo docs.json knowledge.knolo
+```
+
+---
+
+## Concepts
+
+* **Pack (.knolo)**: single-file container with metadata, lexicon, postings, and blocks.
+* **Block**: chunk of text (\~512 tokens recommended) with optional `heading` and `id`.
+* **Deterministic Retrieval**: lexical signals (terms, phrases, positions), not embeddings.
+* **Proximity**: bonus for smaller minimal span covering all query terms.
+* **MMR**: Maximum Marginal Relevance to promote diversity in the top-K.
+* **KNS**: tiny lexical numeric signature for stable tie-breaking.
+* **Context Patch**: structured snippets for LLM prompts (budgeted).
+
+---
+
+## Building Packs
+
+### Input format
+
+```ts
+type BuildInputDoc = {
+  id?: string;          // exposed later as hit.source
+  heading?: string;     // boosts relevance when overlapping query terms
+  text: string;         // raw markdown accepted (lightly stripped)
+};
+```
+
+### API
+
+```ts
+const bytes: Uint8Array = await buildPack(docs: BuildInputDoc[]);
+```
+
+**Tips**
+
+* Prefer multiple smaller blocks (\~512 tokens).
+* Provide `heading` for stronger field boosts.
+* Use stable `id` if you want `hit.source`.
+
+---
+
+## Querying & Results
+
+### API
+
+```ts
+type QueryOptions = {
+  topK?: number;               // default 10
+  requirePhrases?: string[];   // phrases that must appear verbatim
+};
+
+type Hit = {
+  blockId: number;
+  score: number;
+  text: string;
+  source?: string;             // docId if provided
+};
+
+const hits: Hit[] = query(pack, '“react native bridge” throttling', {
+  topK: 5,
+  requirePhrases: ["maximum rate"] // hard constraint
+});
+```
+
+**What the ranker does**
+
+1. Enforces quoted/required phrases (hard filter)
+2. BM25L with precomputed avg block length
+3. **Proximity bonus** (minimal span cover)
+4. **Heading overlap** boost
+5. **KNS** tie-breaker (small, deterministic)
+6. **De-dupe + MMR** diversity for final top-K
+
+---
+
+## LLM Context Patches
+
+### API
+
+```ts
+type ContextPatch = {
+  background: string[];
+  snippets: Array<{ text: string; source?: string }>;
+  definitions: Array<{ term: string; def: string; evidence?: number[] }>;
+  facts: Array<{ s: string; p: string; o: string; evidence?: number[] }>;
+};
+
+const patch = makeContextPatch(hits, { budget: "mini" | "small" | "full" });
+```
+
+**Budgets**
+
+* `mini` ≈ 512 tokens
+* `small` ≈ 1k tokens
+* `full` ≈ 2k tokens
+
+**Best practices**
+
+* Prefer `background` as setup lines for the system prompt.
+* Place `snippets` nearest to the user’s question in the prompt.
+
+---
+
+## Advanced Retrieval Controls
+
+### Require phrases (hard constraints)
+
+```ts
+query(pack, "throttling", { requirePhrases: ["react native bridge"] });
+```
+
+### Tight vs. scattered matches
+
+Proximity bonus favors blocks where all query terms co-occur in a small span.
+
+### Diversity
+
+Top-K results apply near-duplicate suppression (5-gram Jaccard) and MMR (λ≈0.8).
+
+**Tuning (if you fork)**
+
+* Jaccard threshold default \~0.92
+* MMR λ default \~0.8
+* Proximity multiplier default \~0.15
+
+---
+
+## Pack Format (Spec)
+
+**Binary layout**
+
+```
+[metaLen:u32][meta JSON]
+[lexLen:u32][lexicon JSON]
+[postCount:u32][postings u32[]]
+[blocksLen:u32][blocks JSON]
+```
+
+**Meta JSON**
+
+```json
+{
+  "version": 2,
+  "stats": {
+    "docs": <number>,
+    "blocks": <number>,
+    "terms": <number>,
+    "avgBlockLen": <number> // optional in older packs
+  }
+}
+```
+
+**Lexicon JSON**
+
+* Array of `[term, termId]` pairs.
+
+**Postings**
+
+* Flattened `Uint32Array`:
+
+  ```
+  termId, blockId, pos, pos, …, 0, blockId, …, 0, 0, termId, ...
+  ```
+
+  Each block section ends with `0`, each term section ends with `0`.
+
+**Blocks JSON (v1 / v2)**
+
+* **v1**: `string[]` (text only)
+* **v2**: `{ text, heading?, docId? }[]`
+
+Runtime auto-detects and exposes:
+
+```ts
+type Pack = {
+  meta, lexicon, postings, blocks: string[],
+  headings?: (string|null)[],
+  docIds?: (string|null)[]
+}
+```
+
+---
+
+## Performance & Tuning
+
+**Targets (typical)**
+
+* Query < 50 ms on mid-range laptops (packs ≤ 200 MB)
+* Memory < 10 MB for \~50k blocks
+* Pack size ≈ 6–12% of raw text
+
+**Tuning checklist**
+
+* Split large documents into \~512-token blocks
+* Provide informative `heading`s
+* Shard packs by domain if you exceed 200–500 MB
+* Cache mounted packs in memory if app does repeated queries
+
+---
+
+## React Native / Expo Notes
+
+* Built-in ponyfills for `TextEncoder`/`TextDecoder`; no extra deps needed.
+* To load `.knolo` assets, read as Base64 then convert to `Uint8Array` before `mountPack({ src })`.
+* Hermes compatible.
+
+---
+
+## Testing & QA
+
+### Smoke test
+
+We ship a no-deps smoke test that exercises phrase enforcement, proximity, de-dupe, and heading boosts.
+
+Run:
+
+```bash
+npm run build
+npm run smoke
+```
+
+### What it validates
+
+* Basic query returns results
+* Quoted phrases enforced
+* `requirePhrases` enforced (normalized)
+* Tight spans outrank scattered
+* No near-duplicates in top-K
+* `source` type is `string | undefined`
+
+---
+
+## Migration 0.1.x → 0.2.0
+
+* **No API breaks**: `buildPack`, `mountPack`, `query`, `makeContextPatch` unchanged.
+* **Compatibility**: v0.2.0 mounts v1 packs (string blocks) and v2 (object blocks).
+* **New**: phrase enforcement, proximity bonus, heading boosts, KNS tie-breaker, de-dupe + MMR, avgBlockLen in meta, RN/Expo ponyfills.
+* Provide `id` and `heading` at build time to enable `hit.source` and field boosts.
+
+---
+
+## Security & Privacy
+
+* All retrieval is **local**; no network dependency for search.
+* Packs are plain JSON sections + typed arrays—auditable and diff-able.
+* If packs contain sensitive data, treat `.knolo` files as confidential artifacts.
+
+---
+
+## FAQ
+
+**Q: Does this use embeddings or a vector DB?**
+A: No—pure lexical retrieval with positions and structural cues.
+
+**Q: Why am I still seeing similar results?**
+A: De-dup suppresses near-duplicates but allows related passages. Increase Jaccard threshold or tune λ (if forking).
+
+**Q: How do I improve recalls for synonyms?**
+A: Add domain alias tables or expand queries; we intentionally avoid opaque embeddings.
+
+**Q: Does it work offline?**
+A: Yes, end-to-end.
+
+---
+
+## Glossary
+
+* **BM25L**: length-normalized lexical ranking.
+* **Minimal span cover**: smallest token window containing all query terms.
+* **MMR**: diversity-promoting re-ranker balancing relevance & novelty.
+* **KNS**: deterministic lexical numeric signature (tie-breaker).
+
+---
+
+## Versioning & Releases
+
+* **SemVer**: feature ≥ minor, bugs = patch.
+* Tag releases: `vX.Y.Z`.
+* Recommended commit message (example):
+
+```
+feat(core): deterministic retrieval upgrades
+- phrase enforcement, proximity, heading boosts
+- de-dup + MMR, KNS tie-breaker
+- avgBlockLen, RN/Expo ponyfills
+```

package/LICENSE

@@ -1,21 +1,24 @@
-MIT License
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
 
-Copyright (c) 2025 KnoLo Contributors
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+   1. Definitions.
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+      ...
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+   (rest of standard Apache 2.0 text continues unchanged)

package/README.md

@@ -281,5 +281,5 @@ Yes—TextEncoder/TextDecoder ponyfills are included.
 
 ## 📄 License
 
-MIT — see [LICENSE](./LICENSE).
+Apache-2.0 — see [LICENSE](./LICENSE).
 

package/package.json

@@ -1,15 +1,29 @@
 {
   "name": "knolo-core",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "type": "module",
   "description": "Local-first knowledge packs for small LLMs.",
-  "keywords": ["llm", "knowledge-base", "rag", "local", "expo"],
+  "keywords": [
+    "llm",
+    "knowledge-base",
+    "rag",
+    "local",
+    "expo"
+  ],
   "author": "Sam Paniagua",
-  "license": "MIT",
+  "license": "Apache-2.0",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
-  "files": ["dist", "bin", "README.md", "LICENSE"],
-  "bin": { "knolo": "./bin/knolo.mjs" },
+  "files": [
+    "dist",
+    "bin",
+    "README.md",
+    "LICENSE",
+    "DOCS.md"
+  ],
+  "bin": {
+    "knolo": "./bin/knolo.mjs"
+  },
   "exports": {
     ".": {
       "import": "./dist/index.js",
@@ -19,7 +33,7 @@
   "scripts": {
     "build": "tsc -p tsconfig.json",
     "prepublishOnly": "npm run build",
-     "smoke": "node scripts/smoke.mjs"
+    "smoke": "node scripts/smoke.mjs"
   },
   "devDependencies": {
     "typescript": "^5.5.0",