@kedem/okdb 1.0.0

package/docs/files.md

@@ -0,0 +1,131 @@
+# Files
+
+OKDB includes content-addressable blob storage integrated with the database. Files are stored as SHA-256-hashed blobs on disk and tracked as metadata records in the `~files` OKDB type.
+
+---
+
+## Storing a file
+
+```javascript
+// From a Buffer
+const file = await okdb.files.put(buffer, {
+    path: '/uploads/docs/report.pdf',
+    mime: 'application/pdf',
+});
+// file = { id, hash, size, mime, path, createdAt }
+
+// From a stream
+const stream = fs.createReadStream('/local/file.jpg');
+const file = await okdb.files.put(stream, {
+    path: '/avatars/alice.jpg',
+    mime: 'image/jpeg',
+    id: 'avatar-alice', // optional explicit ID (UUID generated if omitted)
+});
+```
+
+The blob is written to `<okdb-root>/blobs/<first2>/<rest>` using the SHA-256 hex as the filename. Storing the same bytes twice stores them once — the second `put` returns a record pointing to the existing blob.
+
+---
+
+## Reading a file
+
+```javascript
+// Get metadata by ID
+const meta = okdb.files.get('avatar-alice');
+// { id, hash, size, mime, path, createdAt }
+
+// Get the raw Buffer
+const buf = await okdb.files.read('avatar-alice');
+
+// Stream the blob
+const stream = okdb.files.stream('avatar-alice');
+stream.pipe(response);
+```
+
+---
+
+## Deleting a file
+
+```javascript
+await okdb.files.remove('avatar-alice');
+```
+
+Deletion removes the metadata record. The blob bytes are garbage-collected when no other `~files` record references the same hash.
+
+---
+
+## Listing files
+
+```javascript
+// All files
+for (const meta of okdb.files.list()) {
+    console.log(meta.path, meta.size, meta.hash);
+}
+
+// By path prefix
+for (const meta of okdb.files.list({ prefix: '/uploads/' })) { ... }
+```
+
+---
+
+## HTTP upload and download
+
+When the HTTP server is running, files are accessible via REST:
+
+```bash
+# Upload (multipart or raw body)
+curl -X POST http://localhost:8080/api/files \
+  -H "Authorization: Bearer my-token" \
+  -F "file=@report.pdf" \
+  -F "path=/docs/report.pdf"
+
+# Download by ID
+curl http://localhost:8080/api/files/avatar-alice \
+  -H "Authorization: Bearer my-token" \
+  --output avatar.jpg
+```
+
+---
+
+## Files per environment
+
+Each OKDB environment has its own `~files` type (metadata lives in that env), but all environments share the same blob directory at the OKDB root level. This means:
+
+- A blob stored in the `default` env and the same blob in `analytics` share the same bytes on disk
+- Reference counting tracks how many metadata records point to each hash
+- The blob is only deleted when the ref count drops to zero
+
+```javascript
+// Files in a custom environment
+const analyticsEnv = okdb.env('analytics');
+const file = await analyticsEnv.files.put(buffer, { path: '/exports/data.csv', mime: 'text/csv' });
+```
+
+---
+
+## Storage layout
+
+```
+<okdb-root>/blobs/
+  ab/cdef1234...   ← blob file (path = first2/rest of SHA-256 hex)
+  ff/0123abcd...
+```
+
+In LMDB (default env):
+
+```
+~files          ← metadata records (id → { hash, size, mime, path, createdAt })
+~file:blob_status  ← ref-counting sub-db (hash → { refs, nodes })
+```
+
+---
+
+## Sync
+
+`~files` metadata **is synced** across cluster nodes. Blob bytes are **not** — they are transferred separately via a blob mesh protocol (HTTP) when a node needs a blob it doesn't have locally.
+
+This means: metadata replicates instantly; actual bytes follow on demand.
+
+:::note
+Blob sync is pull-based. A node will fetch the blob from a peer only when a consumer requests it and the local blob is missing.
+:::

package/docs/fts.md

@@ -0,0 +1,139 @@
+# Full-Text Search
+
+OKDB includes a built-in full-text search engine backed by an LMDB inverted index. Each FTS index lives in its own LMDB environment (separate directory) so it can be independently compacted or rebuilt without touching user data.
+
+---
+
+## Registering an FTS index
+
+```javascript
+await okdb.fts.register('articles', 'content', {
+    fields: ['title', 'body'], // fields to index
+    tokenizer: {
+        minTokenLength: 2,
+        maxTokenLength: 64,
+        keepNumbers: true,
+        toLower: true, // case-insensitive search
+    },
+});
+```
+
+The index builds from existing data immediately after registration. Wait for it:
+
+```javascript
+await okdb.fts.ready('articles', 'content');
+```
+
+### Multiple FTS indexes on one type
+
+You can have several FTS indexes on the same type, each covering different fields or using different tokenizer settings:
+
+```javascript
+await okdb.fts.register('products', 'name', { fields: ['name'] });
+await okdb.fts.register('products', 'description', { fields: ['description'], tokenizer: { minTokenLength: 3 } });
+```
+
+---
+
+## Querying
+
+```javascript
+const results = okdb.ftsQuery('articles', 'content', 'embedded database Node.js');
+for (const { key, value } of results) {
+    console.log(key, value.title);
+}
+```
+
+`ftsQuery` returns records that contain **any** of the query tokens (OR semantics). Results are not ranked by relevance — they are returned in primary-key order.
+
+### Combining FTS with field filters
+
+Pass a sift-compatible filter as the fourth argument to narrow results:
+
+```javascript
+// Articles matching the query that are also published
+const results = okdb.ftsQuery('articles', 'content', 'database performance', {
+    status: 'published',
+});
+```
+
+### Options
+
+```javascript
+const results = okdb.ftsQuery('articles', 'content', 'hello world', null, {
+    limit: 20,
+    offset: 0,
+});
+```
+
+---
+
+## Index management
+
+```javascript
+// Check if an index exists
+okdb.fts.has('articles', 'content'); // → boolean
+
+// Get status
+okdb.fts.status('articles', 'content'); // → 'creating' | 'ready' | null
+
+// List all FTS indexes on a type
+okdb.fts.list('articles');
+// → [{ name, status, config, created, updated, sizeBytes }]
+
+// Drop an index
+await okdb.fts.drop('articles', 'content');
+```
+
+---
+
+## How it works
+
+OKDB's FTS engine is a simple inverted posting list:
+
+1. On every write, the changed record's indexed fields are tokenized
+2. Removed tokens are deleted from the posting list (`post` sub-db)
+3. Added tokens are inserted (`token → docKey` pairs in a `dupSort` LMDB sub-db)
+4. A forward index (`docs` sub-db: `docKey → [tokens]`) tracks which tokens each doc holds, enabling correct updates
+
+Token processing (default):
+
+- Split on whitespace and punctuation
+- Lowercase
+- Discard tokens shorter than `minTokenLength` (default 2) or longer than `maxTokenLength` (default 64)
+- Numbers kept by default (`keepNumbers: true`)
+
+### Storage layout
+
+```
+~fts/<envName>/<type>/<indexName>/
+  post    dupSort=true   token → docKey   (inverted list)
+  docs                   docKey → tokens  (forward index)
+```
+
+FTS environments are `sync: false` — they are never replicated. On a new node they rebuild from synced type data automatically.
+
+---
+
+## Example: articles with FTS
+
+```javascript
+await okdb.ensureType('articles');
+await okdb.fts.register('articles', 'search', {
+    fields: ['title', 'body', 'tags'],
+});
+await okdb.fts.ready('articles', 'search');
+
+await okdb.put('articles', 'a1', {
+    title: 'Getting started with LMDB',
+    body: 'LMDB is a B+ tree key-value store with MVCC...',
+    tags: ['database', 'lmdb', 'storage'],
+});
+
+const results = okdb.ftsQuery('articles', 'search', 'LMDB key value');
+// → includes 'a1'
+```
+
+:::tip
+FTS is good for keyword search across free-form text. For semantic similarity ("find documents about the same topic"), use [Embeddings](./embeddings.md) instead.
+:::