endee 1.7.1 → 1.8.0-dev.2

package/README.md

@@ -1,545 +1,720 @@
-# Endee - TypeScript Vector Database Client
-
-Endee is a TypeScript client for a local vector database designed for maximum speed and efficiency. This package provides full type safety, modern ES module support, and optimized code for rapid Approximate Nearest Neighbor (ANN) searches on vector data.
-
-## Key Features
-
-- **TypeScript First**: Full type safety and IntelliSense support
-- **Fast ANN Searches**: Efficient similarity searches on vector data
-- **Multiple Distance Metrics**: Support for cosine, L2, and inner product distance metrics
-- **Hybrid Indexes**: Support for dense vectors, sparse vectors, and hybrid (dense + sparse) searches
-- **Metadata Support**: Attach and search with metadata and filters
-- **High Performance**: Optimized for speed and efficiency
-- **Modern ES Modules**: Native ES module support with proper tree-shaking
-
-## Requirements
-
-- Node.js >= 18.0.0
-- Endee Local server running (see [Quick Start](https://docs.endee.io/quick-start))
-
-## Installation
+# Endee — TypeScript / JavaScript Client
+
+Endee is a high-performance C++ vector database for AI search, RAG, and hybrid
+retrieval. A **collection** holds objects, and each object can carry values for
+several **named, typed fields** at once — dense vectors, sparse vectors, and
+multi-vectors — so a single object can be searched many ways.
+
+- **Multi-field objects** — one object, many vector fields (dense + sparse + multi-vector).
+- **Hybrid search** — query any subset of fields; `search()` returns per-field results, which you can fuse with **`rerank()`** (RRF).
+- **Client-side normalization** — cosine vectors are L2-normalized for you.
+- **Full control plane** — collections, objects, filters, rebuild/shrink, backups, databases, and tokens.
+- **TypeScript first** — full type definitions and IntelliSense for every call.
+- **ES modules** — native ESM with proper tree-shaking.
+
+---
+
+## Table of contents
+
+1. [Install](#install)
+2. [Quick start](#quick-start)
+3. [Connecting & tokens](#connecting--tokens)
+4. [Create a collection](#create-a-collection)
+5. [Insert objects (combining vector types)](#insert-objects-combining-vector-types)
+6. [Search](#search)
+   - [The unified field shape](#the-unified-field-shape)
+   - [Per-field `limit` and `ef_search`](#per-field-limit-and-ef_search)
+   - [Single field](#single-field-search)
+   - [Multi-field](#multi-field-search)
+   - [Fusing results with `rerank()`](#fusing-results-with-rerank)
+   - [Filters](#filtered-search)
+   - [Filter tuning](#filter-tuning)
+7. [Object operations](#object-operations)
+8. [Collection maintenance](#collection-maintenance)
+9. [Backups](#backups)
+10. [Database administration (root token)](#database-administration-root-token)
+11. [Reference](#reference)
+12. [Error handling](#error-handling)
+
+---
+
+## Install
 
 ```bash
 npm install endee
 ```
 
-## Quick Start
+Requires Node.js ≥ 18. Depends on `axios` and `@msgpack/msgpack`.
 
-### Initialize the Client
+---
 
-The Endee client connects to your local server (defaults to `http://127.0.0.1:8080/api/v1`):
+## Quick start
 
 ```typescript
-import { Endee, Precision } from 'endee';
+import { Endee } from 'endee';
+
+// A "db token" (db_name:secret) scopes you to one database.
+const client = new Endee('my_db:xxxxxxxx');
+client.setBaseUrl('http://localhost:8080/api/v2'); // include /api/v2
+
+// 1. Create a collection with one of each field type.
+await client.createCollection({
+  name: 'products',
+  fields: [
+    {
+      name: 'embedding',
+      type: 'vector',
+      params: { dimension: 8, space_type: 'cosine', precision: 'int8' },
+    },
+    { name: 'keywords', type: 'sparse', sparse_model: 'default' },
+    {
+      name: 'colbert',
+      type: 'multi_vector',
+      params: { dimension: 8, space_type: 'cosine', precision: 'int8', pooling: 'mean' },
+    },
+  ],
+});
 
-// Connect to local Endee server (defaults to localhost:8080)
-const client = new Endee();
-```
+const collection = await client.getCollection('products');
 
-**Using Authentication?** If your server has `NDD_AUTH_TOKEN` set, pass the same token when initializing:
+// 2. Insert an object carrying ALL THREE field types at once.
+await collection.upsert([
+  {
+    id: 'p1',
+    meta: { name: 'Wireless Headphones', price: 99 },
+    filter: { category: 'electronics' },
+    fields: {
+      embedding: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
+      keywords: { indices: [3, 17, 42], values: [0.9, 0.5, 0.2] },
+      colbert: [Array(8).fill(0.1), Array(8).fill(0.2)],
+    },
+  },
+]);
 
-```typescript
-const client = new Endee('your-auth-token');
+// 3. Search the dense field. `results` is keyed by field name.
+const res = await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2), limit: 5 } },
+});
+for (const h of res.results.embedding) {
+  console.log(h.id, h.similarity, h.meta);
+}
 ```
 
-### Setting a Custom Base URL
+---
 
-If your server runs on a different port, use `setBaseUrl()`:
+## Connecting & tokens
 
-```typescript
-const client = new Endee();
+**Authentication is always required.** The server has no anonymous mode — every
+request must carry a token, or it is rejected with **401 Unauthorized**.
 
-// Set custom base URL for non-default port
-client.setBaseUrl('http://0.0.0.0:8081/api/v1');
-```
-
-### Create a Dense Index
+A **db token** has the form `db_name:secret` and scopes you to all collection /
+object / search work inside one database. (Database administration and minting
+tokens require the server's **root token** — those control-plane operations are
+not yet exposed in the JS client; see [Reference](#reference).)
 
 ```typescript
-import { Precision } from 'endee';
-
-await client.createIndex({
-  name: 'my_vectors',
-  dimension: 384,
-  spaceType: 'cosine',
-  precision: Precision.INT16,
-});
+const client = new Endee('my_db:xxxxxxxx');
+client.setBaseUrl('http://localhost:8080/api/v2'); // must point at the /api/v2 root
 ```
 
-**Dense Index Parameters:**
+- A token of the form `db_name:secret:region` is parsed automatically — the
+  client targets `https://<region>.endee.io/api/v2` and strips the region from
+  the token it sends.
+- `setBaseUrl` must point at the `/api/v2` root of your server.
+- A missing/invalid token → `AuthenticationException` (401); a read-only token
+  on a write → `ForbiddenException` (403).
 
-| Parameter | Description |
-|-----------|-------------|
-| `name` | Unique name for your index |
-| `dimension` | Vector dimensionality (must match your embedding model's output) |
-| `spaceType` | Distance metric - `"cosine"`, `"l2"`, or `"ip"` (inner product) |
-| `M` | Graph connectivity - higher values increase recall but use more memory (default: 16) |
-| `efCon` | Construction-time parameter - higher values improve index quality (default: 128) |
-| `precision` | Quantization precision (default: `Precision.INT16`) |
+---
 
-### Create a Hybrid Index
+## Create a collection
 
-Hybrid indexes combine dense vector search with sparse vector search. Add the `sparseDimension` parameter:
+A collection is a set of **named, typed fields**. Each field is a plain object
+sent to the server as-is.
+
+| Field type | Required keys | `params` |
+|------------|---------------|----------|
+| `vector` | `name`, `type` | `dimension`, `space_type`, `precision` (+ optional `M`, `ef_con`) |
+| `sparse` | `name`, `type`, `sparse_model` | — (no `params`) |
+| `multi_vector` | `name`, `type` | `dimension`, `space_type`, `precision`, **`pooling`** (`"mean"`/`"max"`) (+ optional `M`, `ef_con`) |
 
 ```typescript
-await client.createIndex({
-  name: 'hybrid_index',
-  dimension: 384,           // Dense vector dimension
-  sparseDimension: 30000,   // Sparse vector dimension (vocabulary size)
-  spaceType: 'cosine',
-  precision: Precision.INT16,
+await client.createCollection({
+  name: 'products',
+  fields: [
+    // Dense embedding (single vector per object)
+    {
+      name: 'embedding',
+      type: 'vector',
+      params: { dimension: 768, space_type: 'cosine', precision: 'int8' },
+    },
+
+    // Sparse / keyword field (BM25-style)
+    { name: 'keywords', type: 'sparse', sparse_model: 'default' },
+
+    // Multi-vector field (many vectors per object)
+    {
+      name: 'colbert',
+      type: 'multi_vector',
+      params: { dimension: 768, space_type: 'cosine', precision: 'int8', pooling: 'mean' },
+    },
+  ],
 });
 ```
 
-### List and Access Indexes
+- `space_type`: `cosine` (default), `l2`, `ip`.
+- `precision`: `float32`, `float16`, `int16` (default), `int8`, `int8e`, `binary`.
+- `M` / `ef_con`: HNSW build params (optional; sensible defaults applied).
+- **Cosine vectors are L2-normalized by the client** before sending.
+
+Other client-level collection calls:
 
 ```typescript
-// List all indexes
-const indexes = await client.listIndexes();
+await client.listCollections(); // [{ name: "products", ... }, ...]
+const collection = await client.getCollection('products');
+await collection.describe(); // { name, fields, created_at, layout_version }
+await client.deleteCollection('products');
+```
+
+---
+
+## Insert objects (combining vector types)
 
-// Get reference to an existing index
-const index = await client.getIndex('my_vectors');
+`collection.upsert(objects)` takes an array of objects. Each object is:
 
-// Delete an index
-await client.deleteIndex('my_vectors');
+```typescript
+{
+  id: '<unique string id>',
+  meta?: { /* ... */ },     // arbitrary JSON, returned in search results (optional)
+  filter?: { /* ... */ },   // tag object used by filtered search (optional)
+  fields?: {                // field_name -> value, ONE entry per field you set
+    '<dense_field>':  [/* float, ... */],                       // dense  → array
+    '<sparse_field>': { indices: [/* ... */], values: [/* ... */] }, // sparse → object
+    '<multi_field>':  [[/* float, ... */], [/* float, ... */]], // multi  → array of arrays
+  },
+}
 ```
 
-## Upserting Vectors
+Per-field value formats:
 
-The `index.upsert()` method adds or updates vectors in an existing index.
+| Field type | Value format |
+|------------|--------------|
+| `vector` | `[0.1, 0.2, ...]` — a flat array of floats |
+| `sparse` | `{ indices: [3, 17], values: [0.9, 0.5] }` (also accepts `sparse_indices`/`sparse_values`) |
+| `multi_vector` | `[[...], [...], ...]` — an array of equal-length float arrays |
 
-```typescript
-const index = await client.getIndex('my_index');
+### One object, multiple field types
 
-await index.upsert([
+A single object can populate **any combination** of the collection's fields in
+one call. They're stored together under the same id and can be searched
+independently or together:
+
+```typescript
+await collection.upsert([
   {
-    id: 'vec1',
-    vector: [0.1, 0.2, 0.3 /* ... */],
-    meta: { title: 'First document' },
-    filter: { category: 'tech' },
+    id: 'p1',
+    meta: { name: 'Wireless Headphones', price: 99 },
+    filter: { category: 'electronics' },
+    fields: {
+      embedding: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], // dense
+      keywords: { indices: [3, 17, 42], values: [0.9, 0.5, 0.2] }, // sparse
+      colbert: [
+        // multi-vector
+        [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
+        [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
+      ],
+    },
   },
   {
-    id: 'vec2',
-    vector: [0.3, 0.4, 0.5 /* ... */],
-    meta: { title: 'Second document' },
-    filter: { category: 'science' },
+    id: 'p2',
+    meta: { name: 'Running Shoes' },
+    filter: { category: 'footwear' },
+    fields: {
+      // An object may set only SOME fields — here just dense + sparse.
+      embedding: [0.5, 0.4, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01],
+      keywords: { indices: [5, 17, 90], values: [0.7, 0.6, 0.1] },
+    },
   },
 ]);
+// → { upserted: 2 }
 ```
 
-**Vector Object Fields:**
+Notes:
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | Unique identifier for the vector |
-| `vector` | Yes | Array of floats representing the embedding |
-| `meta` | No | Arbitrary metadata object |
-| `filter` | No | Key-value pairs for filtering during queries |
+- You don't have to fill every field on every object — set whatever subset you have.
+- `upsert` is insert-or-replace by `id`.
+- Cosine `vector`/`multi_vector` values are normalized client-side before sending;
+  the original norm is stored so `getObjects` returns the original (pre-normalization) vectors.
+- Validation is done client-side (fails fast): batch size ≤ 10,000, no duplicate
+  ids in a batch, dense/multi dims must match the field, sparse `indices`/`values`
+  lengths must match, filter keys ≤ 128 bytes / string values ≤ 1024 bytes.
 
-## Querying the Index
+---
 
-The `index.query()` method performs a similarity search.
+## Search
 
-```typescript
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],
-  topK: 5,
-  ef: 128,
-  includeVectors: true,
-});
+`collection.search({ fields, filter })` queries one or more fields **in a
+single request** and **always returns one ranked list per field**, keyed by field
+name. To collapse those lists into a single ranked list, fuse them with
+[`rerank()`](#fusing-results-with-rerank).
 
-for (const item of results) {
-  console.log(`ID: ${item.id}, Similarity: ${item.similarity}`);
-}
-```
+### The unified field shape
 
-**Query Parameters:**
+Every field is queried with the **same object form** — `{ query, limit?, ef_search? }`.
+There is no bare-value shorthand; `query` is required. The `query` value matches
+the field's type:
 
-| Parameter | Description |
-|-----------|-------------|
-| `vector` | Query vector (must match index dimension) |
-| `topK` | Number of results to return (default: 10, max: 512) |
-| `ef` | Search quality parameter (default: 128, max: 1024) |
-| `includeVectors` | Include vector data in results (default: false) |
-| `filter` | Optional filter criteria (array of filter objects) |
-| `sparseIndices` | Sparse vector indices for hybrid search (default: null) |
-| `sparseValues` | Sparse vector values for hybrid search (default: null) |
-| `prefilterCardinalityThreshold` | Controls when search switches from HNSW filtered search to brute-force prefiltering on the matched subset (default: 10,000, range: 1,000–1,000,000). See [Filter Tuning](#filter-tuning) for details. |
-| `filterBoostPercentage` | Expands the internal HNSW candidate pool by this percentage when a filter is active, compensating for filtered-out results (default: 0, range: 0–100). See [Filter Tuning](#filter-tuning) for details. |
+| Field type | `query` shape |
+|------------|-------------|
+| `vector` | `[0.2, 0.2, ...]` |
+| `sparse` | `{ indices: [3, 17], values: [0.8, 0.4] }` |
+| `multi_vector` | `[[...], [...]]` |
 
-## Filtered Querying
-
-Use the `filter` parameter to restrict results. All filters are combined with **logical AND**.
+Each result **hit** is:
 
 ```typescript
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],
-  topK: 5,
-  filter: [
-    { category: { $eq: 'tech' } },
-    { score: { $range: [80, 100] } },
-  ],
-});
+{ id: 'p1', similarity: 0.94, meta: { /* ... */ }, filter: { /* ... */ } }
 ```
 
-### Filtering Operators
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `$eq` | Exact match | `{ status: { $eq: 'published' } }` |
-| `$in` | Match any in list | `{ tags: { $in: ['ai', 'ml'] } }` |
-| `$range` | Numeric range (inclusive) | `{ score: { $range: [70, 95] } }` |
-
-> **Note:** The `$range` operator supports values within **[0 - 999]**. Normalize larger values before upserting.
+> Search results carry `meta`/`filter` but **not** the stored vectors. To get the
+> vectors back, use [`getObjects`](#object-operations).
 
-### Filter Tuning
+### Per-field `limit` and `ef_search`
 
-When using filtered queries, two optional parameters let you tune the trade-off between search speed and recall:
+`limit` is the **max number of hits to return for that field** (defaults to
+`10`). It's set per field, so each field draws independently. `ef_search` tunes
+that field's HNSW recall/latency.
 
-#### `prefilterCardinalityThreshold`
-
-Controls when the search strategy switches from **HNSW filtered search** (fast, graph-based) to **brute-force prefiltering** (exhaustive scan on the matched subset).
+```typescript
+// embedding returns up to 35 hits, keywords up to 15 — each independent
+await collection.search({
+  fields: {
+    embedding: { query: Array(8).fill(0.2), limit: 35, ef_search: 256 },
+    keywords: { query: { indices: [3, 17], values: [0.8, 0.4] }, limit: 15 },
+  },
+});
+```
 
-| Value | Behavior |
-|-------|----------|
-| `1,000` | Prefilter only for very selective filters — minimum value |
-| `10,000` | Prefilter only when the filter matches ≤10,000 vectors **(default)** |
-| `1,000,000` | Prefilter for almost all filtered searches — maximum value |
+### Single-field search
 
-The intuition: when very few vectors match your filter, HNSW may struggle to find enough valid candidates through graph traversal. In that case, scanning the filtered subset directly (prefiltering) is faster and more accurate. Raising the threshold means prefiltering kicks in more often; lowering it favors HNSW graph search.
+A single field is just the multi-field shape with one entry. `results` is keyed
+by that field name:
 
 ```typescript
-// Only prefilter when filter matches ≤5,000 vectors
-const results = await index.query({
-  vector: [/* ... */],
-  topK: 10,
-  filter: [{ category: { $eq: 'rare' } }],
-  prefilterCardinalityThreshold: 5000,
+const res = await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2), limit: 3 } },
+});
+res.results.embedding;
+// [
+//   { id: "p1", similarity: 0.97, meta: {...}, filter: {...} },
+//   { id: "p2", similarity: 0.88, meta: {...}, filter: {...} },
+// ]
+
+// sparse
+await collection.search({
+  fields: { keywords: { query: { indices: [3, 17], values: [0.8, 0.4] }, limit: 3 } },
 });
-```
 
-#### `filterBoostPercentage`
+// multi-vector
+await collection.search({
+  fields: { colbert: { query: [Array(8).fill(0.1), Array(8).fill(0.2)], limit: 3 } },
+});
+```
 
-When using HNSW filtered search, some candidates explored during graph traversal are discarded by the filter, which can leave you with fewer results than `topK`. `filterBoostPercentage` compensates by expanding the internal candidate pool before filtering is applied.
+### Multi-field search
 
-- `0` → no boost, standard candidate pool size **(default)**
-- `20` → fetch 20% more candidates internally before applying the filter
-- Maximum: `100` (doubles the candidate pool)
+Query several fields and you get **each field's own ranked list, unfused** —
+`results` is an object keyed by field name. Merge or display them however you
+like, or hand them to [`rerank()`](#fusing-results-with-rerank).
 
 ```typescript
-// Fetch 30% more candidates to compensate for aggressive filtering
-const results = await index.query({
-  vector: [/* ... */],
-  topK: 10,
-  filter: [{ visibility: { $eq: 'public' } }],
-  filterBoostPercentage: 30,
+const res = await collection.search({
+  fields: {
+    embedding: { query: [0.2, 0.2, 0.3, 0.3], limit: 2 },
+    keywords: { query: { indices: [3, 17], values: [0.8, 0.4] }, limit: 2 },
+  },
 });
 ```
 
-#### Using Both Together
+```json
+{
+  "results": {
+    "embedding": [
+      { "id": "p1", "meta": { "name": "Wireless Headphones" },
+        "filter": { "category": "electronics" }, "similarity": 0.4938 },
+      { "id": "p2", "meta": { "name": "Running Shoes" },
+        "filter": { "category": "footwear" }, "similarity": 0.4505 }
+    ],
+    "keywords": [
+      { "id": "p1", "meta": { "name": "Wireless Headphones" },
+        "filter": { "category": "electronics" }, "similarity": 0.9200 },
+      { "id": "p2", "meta": { "name": "Running Shoes" },
+        "filter": { "category": "footwear" }, "similarity": 0.5591 }
+    ]
+  }
+}
+```
 
 ```typescript
-const results = await index.query({
-  vector: [/* ... */],
-  topK: 10,
-  filter: [{ category: { $eq: 'rare' } }],
-  prefilterCardinalityThreshold: 5000,  // switch to brute-force for small match sets
-  filterBoostPercentage: 25,             // boost candidates for HNSW filtered search
-});
+res.results.embedding; // this field's ranked hits
+res.results.keywords; // that field's ranked hits
 ```
 
-> **Tip:** Start with the defaults (`prefilterCardinalityThreshold: 10,000`, `filterBoostPercentage: 0`). If filtered queries return fewer results than expected, try increasing `filterBoostPercentage`. If filtered queries are slow on selective filters, try lowering `prefilterCardinalityThreshold`. Valid range for the threshold is `1,000–1,000,000`.
+> `similarity` is in **each field's own scale** (cosine for dense ≈ 0.49, sparse
+> dot for keywords ≈ 0.92) — scores are **not comparable across fields**. That's
+> exactly why fusion is a separate, opt-in step.
 
-## Hybrid Search
+### Fusing results with `rerank()`
 
-### Upserting Hybrid Vectors
-
-Provide both dense vectors and sparse representations:
+`rerank(searchResults, { name, limit, fieldWeights, rrfK })` fuses the per-field
+lists from `search()` into **one ranked list** using Reciprocal Rank Fusion. It's
+a standalone function — run `search()`, then pass its result to `rerank()`:
 
 ```typescript
-const index = await client.getIndex('hybrid_index');
+import { rerank } from 'endee';
 
-await index.upsert([
-  {
-    id: 'doc1',
-    vector: [0.1, 0.2 /* ... */],           // Dense vector
-    sparseIndices: [10, 50, 200],            // Non-zero term positions
-    sparseValues: [0.8, 0.5, 0.3],           // Weights for each position
-    meta: { title: 'Document 1' },
-  },
-  {
-    id: 'doc2',
-    vector: [0.3, 0.4 /* ... */],
-    sparseIndices: [15, 100, 500],
-    sparseValues: [0.9, 0.4, 0.6],
-    meta: { title: 'Document 2' },
+const res = await collection.search({
+  fields: {
+    embedding: { query: [0.2, 0.2, 0.3, 0.3], limit: 50 },
+    keywords: { query: { indices: [3, 17], values: [0.8, 0.4] }, limit: 50 },
+    colbert: { query: [Array(8).fill(0.1), Array(8).fill(0.2)], limit: 50 },
   },
-]);
+});
+
+const fused = rerank(res, {
+  name: 'rrf',
+  limit: 10,
+  // optional: weight each field's contribution (must sum to 1.0; equal by default)
+  fieldWeights: { embedding: 0.5, keywords: 0.3, colbert: 0.2 },
+  // optional: RRF rank constant k (default 60)
+  rrfK: 60,
+});
 ```
 
-**Hybrid Vector Fields:**
+```json
+{
+  "results": [
+    { "id": "p1", "similarity": 0.0164,
+      "meta": { "name": "Wireless Headphones" }, "filter": { "category": "electronics" } },
+    { "id": "p3", "similarity": 0.0160,
+      "meta": { "name": "Smart Watch" }, "filter": { "category": "electronics" } },
+    { "id": "p2", "similarity": 0.0160,
+      "meta": { "name": "Running Shoes" }, "filter": { "category": "footwear" } }
+  ]
+}
+```
+
+`name` selects the algorithm (only `"rrf"` is supported today). `rrfK` is the RRF
+rank constant (default `60`); a larger value flattens the contribution of top
+ranks, a smaller value sharpens it. Each fused hit keeps the `meta`/`filter` from
+its per-field hit; `similarity` is replaced with the RRF score (sum of
+`weight / (rrfK + rank)` across the fields it appeared in).
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | Unique identifier |
-| `vector` | Yes | Dense embedding vector |
-| `sparseIndices` | Yes (hybrid) | Non-zero term positions in sparse vector |
-| `sparseValues` | Yes (hybrid) | Weights for each sparse index |
-| `meta` | No | Metadata dictionary |
-| `filter` | No | Filter fields |
+**Summary of return shapes**
 
-> **Important:** `sparseIndices` and `sparseValues` must have the same length. Values in `sparseIndices` must be within `[0, sparseDimension)`.
+| Call | `results` shape |
+|------|-----------------|
+| `search(...)` (1 or N fields) | `Record<field_name, SearchHit[]>` |
+| `rerank(searchResults, { name: 'rrf', ... })` | `SearchHit[]` (fused) |
 
-### Querying Hybrid Index
+### Filtered search
 
-Provide both dense and sparse query vectors:
+`filter` applies to any search. It's an array of conditions:
 
 ```typescript
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],       // Dense query
-  sparseIndices: [10, 100, 300],         // Sparse query positions
-  sparseValues: [0.7, 0.5, 0.4],         // Sparse query weights
-  topK: 5,
+await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2), limit: 5 } },
+  filter: [{ category: { $eq: 'electronics' } }],
 });
-
-for (const item of results) {
-  console.log(`ID: ${item.id}, Similarity: ${item.similarity}`);
-}
 ```
 
-**Hybrid Query Parameters:**
+Operators: `$eq`, `$in`, `$range`, `$gt`, `$gte`, `$lt`, `$lte` (see [Reference](#reference)).
 
-| Parameter | Description |
-|-----------|-------------|
-| `vector` | Dense query vector |
-| `sparseIndices` | Sparse query positions |
-| `sparseValues` | Sparse query weights |
-| `topK` | Number of results to return |
-| `ef` | Search quality parameter (max 1024, default: 128) |
-| `includeVectors` | Include vector data in results (default: false) |
-| `filter` | Optional filter criteria |
-| `prefilterCardinalityThreshold` | Controls when search switches from HNSW filtered search to brute-force prefiltering on the matched subset (default: 10,000, range: 1,000–1,000,000). See [Filter Tuning](#filter-tuning) for details. |
-| `filterBoostPercentage` | Expands the internal HNSW candidate pool by this percentage when a filter is active, compensating for filtered-out results (default: 0, range: 0–100). See [Filter Tuning](#filter-tuning) for details. |
+### Filter tuning
 
-You can also query with:
-- **Dense only**: Provide only `vector`
-- **Sparse only**: Provide only `sparseIndices` and `sparseValues`
-- **Hybrid**: Provide all three for combined results
-
-## Updating Filters
-
-Use `index.updateFilters()` to update the filter fields of existing vectors without re-upserting them.
+Two optional params tune the speed/recall trade-off of **filtered** queries.
+They're sent to the server under `filter_params` and ignored on unfiltered
+searches.
 
 ```typescript
-await index.updateFilters([
-  { id: 'vec1', filter: { category: 'ml', score: 95 } },
-  { id: 'vec2', filter: { category: 'science', score: 80 } },
-]);
+await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2), limit: 10 } },
+  filter: [{ category: { $eq: 'rare' } }],
+  prefilterCardinalityThreshold: 5000, // default 10,000; range 1,000–1,000,000
+  filterBoostPercentage: 25, // default 0; max 100
+});
 ```
 
-**`UpdateFilterParams` Fields:**
+**`prefilterCardinalityThreshold`** — the cardinality below which the search
+switches from HNSW filtered search to brute-force prefiltering. When very few
+vectors match the filter, scanning the matched subset directly is faster and
+more accurate than HNSW graph traversal. **Raising** it makes prefiltering kick
+in more often (favors the exhaustive scan); **lowering** it favors HNSW graph
+search (favors speed on large datasets).
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | ID of the vector to update |
-| `filter` | Yes | New filter object to replace the existing filter |
+**`filterBoostPercentage`** — in HNSW filtered search, candidates that fail the
+filter are discarded, which can leave fewer than `limit` results. This expands
+the internal candidate pool before filtering to compensate. `0` = no boost,
+`100` = doubles the pool.
 
-> **Note:** The entire filter object is replaced, not merged.
+---
 
-## Deletion Methods
+## Object operations
 
-### Delete by ID
-Delete vector with a specifc vector id.
-```typescript
-await index.deleteVector('vec1');
-```
-### Delete by Filter
-Delete all vectors matching specific filters.
 ```typescript
-await index.deleteWithFilter([{'category': {'$eq' : 'tech'}}]);
+// Fetch full stored objects by id (meta, filter, and the stored vectors).
+const objs = await collection.getObjects(['p1', 'p2']);
+// [
+//   { id: "p1", meta: {...}, filter: {...},
+//     vectors: { embedding: [...] },               // dense fields (original, pre-normalization)
+//     sparses: { keywords: { indices: [...], values: [...] } },
+//     multi_vectors: { colbert: [[...], [...]] } },
+//   ...
+// ]
+
+// Delete a single object by id.
+await collection.deleteObject('p1'); // { deleted: "p1" }
+
+// Delete every object matching a filter.
+await collection.deleteByFilter([{ category: { $eq: 'footwear' } }]); // { deleted: <count> }
+
+// Update only the filter tags on existing objects (no re-upsert of vectors).
+await collection.updateFilters([{ id: 'p2', filter: { category: 'sale' } }]); // { updated: <count> }
 ```
 
-### Delete Index
-Delete an entire Index.
-```typescript
-await client.deleteIndex('my_index');
-```
+---
 
-> **Warning:** Deletion operations are **irreversible**.
+## Collection maintenance
 
-## Additional Operations
+```typescript
+await collection.describe(); // { name, fields, created_at, layout_version }
 
-### Get Vector by ID
+// Rebuild a field's HNSW graph with new M / ef_con (async).
+await collection.rebuild({ field: 'embedding', m: 24, efCon: 200 }); // { status: "in_progress", ... }
+await collection.rebuildStatus(); // { percent_complete, ... }
 
-```typescript
-const vector = await index.getVector('vec1');
+// Defragment storage in place.
+await collection.shrink(); // { status: "ok", reclaimed_bytes: <int> }
 ```
 
-### Describe Index
+---
+
+## Backups
+
+Backups are per-database. Creating one is per-collection and **asynchronous**.
 
 ```typescript
-const info = index.describe();
-console.log(info);
-// { name, spaceType, dimension, sparseDimension, isHybrid, count, precision, M }
-```
+// Start a backup of a collection (async).
+await collection.createBackup('nightly'); // { backup_name: "nightly", status: "in_progress" }
 
-## Precision Options
+// Poll for completion.
+while ((await client.activeBackup()).active) {
+  // ...wait, then re-check
+}
 
-Endee supports different quantization precision levels:
+await client.listBackups(); // { nightly: { ...metadata }, ... }
+await client.backupInfo('nightly'); // metadata for one backup
 
-```typescript
-import { Precision } from 'endee';
+// Restore a backup into a new collection.
+await client.restoreBackup('nightly', 'products_restored');
+
+// Move backups around.
+await client.downloadBackup('nightly', '/tmp/nightly.tar'); // writes a .tar, returns the path
+await client.uploadBackup('/tmp/nightly.tar'); // uploads a .tar into this db
 
-Precision.BINARY;   // Binary quantization (1-bit) - smallest storage, fastest search
-Precision.INT8;    // 8-bit integer quantization - balanced performance
-Precision.INT16;   // 16-bit integer quantization (default) - higher precision
-Precision.FLOAT16;  // 16-bit floating point - good balance
-Precision.FLOAT32;  // 32-bit floating point - highest precision
+await client.deleteBackup('nightly');
 ```
 
-**Choosing Precision:**
+> `downloadBackup` / `uploadBackup` use Node's filesystem (`fs`). With the **root
+> token**, pass a third `dbName` arg to `downloadBackup` to target a specific
+> database's backup.
 
-- `BINARY`: Best for very large datasets where speed and storage are critical
-- `INT8` : Recommended for most use cases - good balance of accuracy and performance
-- `INT16` (default) : When you need better accuracy than INT8 but less storage than FLOAT32
-- `FLOAT16`: Good compromise between precision and storage for embeddings
-- `FLOAT32`: When you need maximum precision and storage is not a concern
+---
 
-## Complete Example
+## Database administration (root token)
+
+Database and token administration requires the server's **root token** (its
+`NDD_ROOT_TOKEN`). Create an admin client with that token:
 
 ```typescript
-import { Endee, Precision } from 'endee';
+const admin = new Endee('roottoken');
+admin.setBaseUrl('http://localhost:8080/api/v2');
+```
 
-// Initialize client
-const client = new Endee();
+> The root token administers databases/tokens; it **cannot run data ops directly**.
+> A db token (which it mints) does the collection/object/search work.
 
-// Create a dense index
-await client.createIndex({
-  name: 'documents',
-  dimension: 384,
-  spaceType: 'cosine',
-  precision: Precision.INT16,
-});
+### Databases
 
-// Get the index
-const index = await client.getIndex('documents');
+```typescript
+// Create a database → returns the new db_token.
+const res = await admin.createDatabase('my_db', 'scale'); // db_type: starter|pro|scale|enterprise
+const dbToken = res.db_token;
 
-// Add vectors
-await index.upsert([
-  {
-    id: 'doc1',
-    vector: [0.1, 0.2 /* ... 384 dimensions */],
-    meta: { title: 'First Document' },
-    filter: { category: 'tech' },
-  },
-  {
-    id: 'doc2',
-    vector: [0.3, 0.4 /* ... 384 dimensions */],
-    meta: { title: 'Second Document' },
-    filter: { category: 'science' },
-  },
-]);
+await admin.listDatabases(); // [{ db_name, db_type, is_active, created_at }, ...]
+await admin.getDatabase('my_db'); // single db info
 
-// Query the index
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],
-  topK: 5,
-});
+await admin.setDatabaseType('my_db', 'pro'); // change tier
+await admin.deactivateDatabase('my_db'); // block its tokens (data is kept)
+await admin.activateDatabase('my_db'); // re-enable
+await admin.deleteDatabase('my_db'); // delete db + ALL its data
+```
 
-for (const item of results) {
-  console.log(`ID: ${item.id}, Similarity: ${item.similarity}`);
-}
+### Collections across databases (admin view)
+
+```typescript
+await admin.listDbCollections('my_db'); // ["products", ...] for one db
+await admin.listAllCollections(); // [{ db_name: "my_db", collections: [...] }, ...]
+await admin.deleteDbCollection('my_db', 'products');
 ```
 
-## Collections & Objects API
+### Tokens for any database
 
-Version 1.5+ ships a new, friendlier naming convention alongside the original API. Both names are fully supported — existing code using `Index` / `getVector` / `deleteVector` continues to work without any changes.
+```typescript
+// token_type: "rw" (read-write, default) or "r" (read-only).
+const t = await admin.createToken('my_db', 'ci-reader', 'r'); // { db_token, ... }
+await admin.listTokens('my_db'); // [{ name, token_type, created_at }, ...] (no secrets)
+await admin.deleteToken('my_db', 'ci-reader');
+```
 
-| Old name | New name | Notes |
-|----------|----------|-------|
-| `createIndex` | `createCollection` | Same signature |
-| `listIndexes` | `listCollections` | Same signature |
-| `deleteIndex` | `deleteCollection` | Same signature |
-| `getIndex` | `getCollection` | Returns same object |
-| `getVector` | `getObject` | Method on the collection handle |
-| `deleteVector` | `deleteObject` | Method on the collection handle |
-| `Index` (export) | `Collection` (export) | Same class, two names |
+### Self-service tokens (your own database)
 
-**Example using the new naming:**
+Using a **db token**, you can manage your own database's tokens:
 
 ```typescript
-import { Endee, Collection, Precision } from 'endee';
+await client.createMyToken('reader', 'r'); // { db_token, ... }
+await client.listMyTokens();
+await client.deleteMyToken('reader');
+```
 
-const client = new Endee();
+### Server info
 
-// Create
-await client.createCollection({ name: 'products', dimension: 384, spaceType: 'cosine' });
+```typescript
+await client.health(); // { status, timestamp }
+await client.stats(); // { version, uptime, total_requests }
+```
 
-// Get a handle
-const collection = await client.getCollection('products');
+---
 
-// Upsert objects
-await collection.upsert([
-  { id: 'p1', vector: [0.1, 0.2 /* ... */], meta: { title: 'Shoes' } },
-]);
+## Reference
 
-// Fetch an object by ID
-const obj = await collection.getObject('p1');
+**Distance / space types** (`space_type`)
 
-// Delete an object by ID
-await collection.deleteObject('p1');
+| Value | Meaning |
+|-------|---------|
+| `cosine` | Cosine similarity (vectors L2-normalized client-side). Default. |
+| `l2` | Euclidean distance. |
+| `ip` | Inner product. |
 
-// Delete the collection
-await client.deleteCollection('products');
-```
+**Precisions** (`precision`) — memory/accuracy trade-off
 
-The `Collection` export is an alias for `Index` — both refer to the same class and have identical methods.
+`float32`, `float16`, `int16` (default), `int8`, `int8e`, `binary`. The
+`Precision` enum is exported for convenience:
 
-## API Reference
+```typescript
+import { Precision } from 'endee';
+Precision.INT8; // "int8"
+```
 
-### Endee Class
+**Token types**: `rw` (read-write), `r` (read-only).
 
-| Method | Alias | Description |
-|--------|-------|-------------|
-| `createIndex(options)` | `createCollection(options)` | Create a new index / collection |
-| `listIndexes()` | `listCollections()` | List all indexes / collections |
-| `deleteIndex(name)` | `deleteCollection(name)` | Delete an index / collection |
-| `getIndex(name)` | `getCollection(name)` | Get a handle to an index / collection |
-| `setBaseUrl(url)` | — | Set a custom base URL |
+**Filter operators**
 
-### Index / Collection Class
+| Operator | Example | Meaning |
+|----------|---------|---------|
+| `$eq` | `{ category: { $eq: 'news' } }` | equals |
+| `$in` | `{ tag: { $in: ['a', 'b'] } }` | in set |
+| `$range` | `{ price: { $range: [10, 100] } }` | inclusive range |
+| `$gt` / `$gte` | `{ price: { $gt: 50 } }` | greater than / or equal |
+| `$lt` / `$lte` | `{ price: { $lte: 100 } }` | less than / or equal |
 
-| Method | Alias | Description |
-|--------|-------|-------------|
-| `upsert(vectors)` | — | Insert or update vectors / objects |
-| `query(options)` | — | Search for similar vectors / objects |
-| `updateFilters(updates)` | — | Update filter fields on existing vectors by ID |
-| `deleteVector(id)` | `deleteObject(id)` | Delete a vector / object by ID |
-| `deleteWithFilter(filter)` | — | Delete vectors / objects by filter |
-| `getVector(id)` | `getObject(id)` | Get a vector / object by ID |
-| `describe()` | — | Get index / collection info |
+**Client-side validation / limits**
 
-## TypeScript Types
+| Limit | Value |
+|-------|-------|
+| Objects per `upsert` | ≤ 10,000 |
+| `limit` (top-k) | 1 … 4096 |
+| `efSearch` | 1 … 1024 |
+| Filter key | ≤ 128 bytes |
+| Filter string value | ≤ 1024 bytes |
+| Vector dimension | must match the field's configured dimension |
 
-The package includes comprehensive TypeScript types:
+**Exported TypeScript types**
 
 ```typescript
 import type {
-  VectorItem,
-  QueryOptions,
-  QueryResult,
-  CreateIndexOptions,
-  IndexDescription,
+  FieldDefinition,
+  FieldType,
+  ObjectInput,
+  SearchOptions,
+  SearchHit,
+  SearchResponse,
+  RerankOptions,
+  RerankResponse,
+  FullObject,
+  CollectionMetadata,
+  UpdateFilterEntry,
+  RebuildOptions,
+  DatabaseInfo,
+  TokenInfo,
+  DbType,
+  TokenType,
   SpaceType,
   Precision,
-  UpdateFilterParams,
 } from 'endee';
 ```
 
-## License
+---
 
-MIT
+## Error handling
 
-## Author
+Non-2xx responses throw typed exceptions:
 
-Harshit Raj
+```typescript
+import {
+  EndeeException, // base class for all client errors
+  APIException, // generic 4xx (e.g. bad request)
+  AuthenticationException, // 401
+  ForbiddenException, // 403 (e.g. read-only token on a write)
+  NotFoundException, // 404
+  ConflictException, // 409 (e.g. duplicate collection)
+  ServerException, // 5xx
+} from 'endee';
+
+try {
+  await collection.upsert(objects);
+} catch (e) {
+  if (e instanceof NotFoundException) {
+    console.error('not found:', e.message);
+  } else if (e instanceof EndeeException) {
+    // catches all of the above
+    console.error('request failed:', e.message);
+  } else {
+    throw e;
+  }
+}
+```
+
+All exception classes are importable from `endee`: `EndeeException` (base),
+`APIException`, `AuthenticationException`, `ForbiddenException`,
+`NotFoundException`, `ConflictException`, `ServerException`,
+`SubscriptionException`.
+
+Client-side input mistakes (bad dimensions, mismatched sparse lengths, oversized
+batch, sum-of-weights ≠ 1.0, etc.) throw a plain `Error` **before** any network
+call.
+
+---
+
+## End-to-end example
+
+See `tests/integration_v2.ts` — a full walkthrough (create → multi-field upsert →
+every search mode → object ops → maintenance → cleanup). Run it against a local
+server with a db token:
+
+```bash
+npx tsx tests/integration_v2.ts --token my_db:xxxx --url http://localhost:8080/api/v2
+```
+
+## License
+
+MIT