endee 1.7.1-dev.3 → 1.8.0-dev.1

package/README.md

@@ -1,545 +1,698 @@
-# Endee - TypeScript Vector Database Client
-
-Endee is a TypeScript client for a local vector database designed for maximum speed and efficiency. This package provides full type safety, modern ES module support, and optimized code for rapid Approximate Nearest Neighbor (ANN) searches on vector data.
-
-## Key Features
-
-- **TypeScript First**: Full type safety and IntelliSense support
-- **Fast ANN Searches**: Efficient similarity searches on vector data
-- **Multiple Distance Metrics**: Support for cosine, L2, and inner product distance metrics
-- **Hybrid Indexes**: Support for dense vectors, sparse vectors, and hybrid (dense + sparse) searches
-- **Metadata Support**: Attach and search with metadata and filters
-- **High Performance**: Optimized for speed and efficiency
-- **Modern ES Modules**: Native ES module support with proper tree-shaking
-
-## Requirements
-
-- Node.js >= 18.0.0
-- Endee Local server running (see [Quick Start](https://docs.endee.io/quick-start))
-
-## Installation
+# Endee — TypeScript / JavaScript Client
+
+Endee is a high-performance C++ vector database for AI search, RAG, and hybrid
+retrieval. A **collection** holds objects, and each object can carry values for
+several **named, typed fields** at once — dense vectors, sparse vectors, and
+multi-vectors — so a single object can be searched many ways.
+
+- **Multi-field objects** — one object, many vector fields (dense + sparse + multi-vector).
+- **Hybrid search** — query any subset of fields; `search()` returns per-field results, which you can fuse with **`rerank()`** (RRF).
+- **Client-side normalization** — cosine vectors are L2-normalized for you.
+- **Full control plane** — collections, objects, filters, rebuild/shrink, backups, databases, and tokens.
+- **TypeScript first** — full type definitions and IntelliSense for every call.
+- **ES modules** — native ESM with proper tree-shaking.
+
+---
+
+## Table of contents
+
+1. [Install](#install)
+2. [Quick start](#quick-start)
+3. [Connecting & tokens](#connecting--tokens)
+4. [Create a collection](#create-a-collection)
+5. [Insert objects (combining vector types)](#insert-objects-combining-vector-types)
+6. [Search](#search)
+   - [The unified field shape](#the-unified-field-shape)
+   - [Per-field `weight` and `ef_search`](#per-field-weight-and-ef_search)
+   - [Single field](#single-field-search)
+   - [Multi-field](#multi-field-search)
+   - [Fusing results with `rerank()`](#fusing-results-with-rerank)
+   - [Filters](#filtered-search)
+7. [Object operations](#object-operations)
+8. [Collection maintenance](#collection-maintenance)
+9. [Backups](#backups)
+10. [Database administration (root token)](#database-administration-root-token)
+11. [Reference](#reference)
+12. [Error handling](#error-handling)
+
+---
+
+## Install
 
 ```bash
 npm install endee
 ```
 
-## Quick Start
+Requires Node.js ≥ 18. Depends on `axios` and `@msgpack/msgpack`.
 
-### Initialize the Client
+---
 
-The Endee client connects to your local server (defaults to `http://127.0.0.1:8080/api/v1`):
+## Quick start
 
 ```typescript
-import { Endee, Precision } from 'endee';
+import { Endee } from 'endee';
+
+// A "db token" (db_name:secret) scopes you to one database.
+const client = new Endee('my_db:xxxxxxxx');
+client.setBaseUrl('http://localhost:8080/api/v2'); // include /api/v2
+
+// 1. Create a collection with one of each field type.
+await client.createCollection({
+  name: 'products',
+  fields: [
+    {
+      name: 'embedding',
+      type: 'vector',
+      params: { dimension: 8, space_type: 'cosine', precision: 'int8' },
+    },
+    { name: 'keywords', type: 'sparse', sparse_model: 'default' },
+    {
+      name: 'colbert',
+      type: 'multi_vector',
+      params: { dimension: 8, space_type: 'cosine', precision: 'int8', pooling: 'mean' },
+    },
+  ],
+});
 
-// Connect to local Endee server (defaults to localhost:8080)
-const client = new Endee();
-```
+const collection = await client.getCollection('products');
 
-**Using Authentication?** If your server has `NDD_AUTH_TOKEN` set, pass the same token when initializing:
+// 2. Insert an object carrying ALL THREE field types at once.
+await collection.upsert([
+  {
+    id: 'p1',
+    meta: { name: 'Wireless Headphones', price: 99 },
+    filter: { category: 'electronics' },
+    fields: {
+      embedding: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
+      keywords: { indices: [3, 17, 42], values: [0.9, 0.5, 0.2] },
+      colbert: [Array(8).fill(0.1), Array(8).fill(0.2)],
+    },
+  },
+]);
 
-```typescript
-const client = new Endee('your-auth-token');
+// 3. Search the dense field. `results` is keyed by field name.
+const res = await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2) } },
+  limit: 5,
+});
+for (const h of res.results.embedding) {
+  console.log(h.id, h.similarity, h.meta);
+}
 ```
 
-### Setting a Custom Base URL
+---
 
-If your server runs on a different port, use `setBaseUrl()`:
+## Connecting & tokens
 
-```typescript
-const client = new Endee();
-
-// Set custom base URL for non-default port
-client.setBaseUrl('http://0.0.0.0:8081/api/v1');
-```
+**Authentication is always required.** The server has no anonymous mode — every
+request must carry a token, or it is rejected with **401 Unauthorized**.
 
-### Create a Dense Index
+A **db token** has the form `db_name:secret` and scopes you to all collection /
+object / search work inside one database. (Database administration and minting
+tokens require the server's **root token** — those control-plane operations are
+not yet exposed in the JS client; see [Reference](#reference).)
 
 ```typescript
-import { Precision } from 'endee';
-
-await client.createIndex({
-  name: 'my_vectors',
-  dimension: 384,
-  spaceType: 'cosine',
-  precision: Precision.INT16,
-});
+const client = new Endee('my_db:xxxxxxxx');
+client.setBaseUrl('http://localhost:8080/api/v2'); // must point at the /api/v2 root
 ```
 
-**Dense Index Parameters:**
+- A token of the form `db_name:secret:region` is parsed automatically — the
+  client targets `https://<region>.endee.io/api/v2` and strips the region from
+  the token it sends.
+- `setBaseUrl` must point at the `/api/v2` root of your server.
+- A missing/invalid token → `AuthenticationException` (401); a read-only token
+  on a write → `ForbiddenException` (403).
+
+---
 
-| Parameter | Description |
-|-----------|-------------|
-| `name` | Unique name for your index |
-| `dimension` | Vector dimensionality (must match your embedding model's output) |
-| `spaceType` | Distance metric - `"cosine"`, `"l2"`, or `"ip"` (inner product) |
-| `M` | Graph connectivity - higher values increase recall but use more memory (default: 16) |
-| `efCon` | Construction-time parameter - higher values improve index quality (default: 128) |
-| `precision` | Quantization precision (default: `Precision.INT16`) |
+## Create a collection
 
-### Create a Hybrid Index
+A collection is a set of **named, typed fields**. Each field is a plain object
+sent to the server as-is.
 
-Hybrid indexes combine dense vector search with sparse vector search. Add the `sparseDimension` parameter:
+| Field type | Required keys | `params` |
+|------------|---------------|----------|
+| `vector` | `name`, `type` | `dimension`, `space_type`, `precision` (+ optional `M`, `ef_con`) |
+| `sparse` | `name`, `type`, `sparse_model` | — (no `params`) |
+| `multi_vector` | `name`, `type` | `dimension`, `space_type`, `precision`, **`pooling`** (`"mean"`/`"max"`) (+ optional `M`, `ef_con`) |
 
 ```typescript
-await client.createIndex({
-  name: 'hybrid_index',
-  dimension: 384,           // Dense vector dimension
-  sparseDimension: 30000,   // Sparse vector dimension (vocabulary size)
-  spaceType: 'cosine',
-  precision: Precision.INT16,
+await client.createCollection({
+  name: 'products',
+  fields: [
+    // Dense embedding (single vector per object)
+    {
+      name: 'embedding',
+      type: 'vector',
+      params: { dimension: 768, space_type: 'cosine', precision: 'int8' },
+    },
+
+    // Sparse / keyword field (BM25-style)
+    { name: 'keywords', type: 'sparse', sparse_model: 'default' },
+
+    // Multi-vector field (many vectors per object)
+    {
+      name: 'colbert',
+      type: 'multi_vector',
+      params: { dimension: 768, space_type: 'cosine', precision: 'int8', pooling: 'mean' },
+    },
+  ],
 });
 ```
 
-### List and Access Indexes
+- `space_type`: `cosine` (default), `l2`, `ip`.
+- `precision`: `float32`, `float16`, `int16` (default), `int8`, `int8e`, `binary`.
+- `M` / `ef_con`: HNSW build params (optional; sensible defaults applied).
+- **Cosine vectors are L2-normalized by the client** before sending.
+
+Other client-level collection calls:
 
 ```typescript
-// List all indexes
-const indexes = await client.listIndexes();
+await client.listCollections(); // [{ name: "products", ... }, ...]
+const collection = await client.getCollection('products');
+await collection.describe(); // { name, fields, created_at, layout_version }
+await client.deleteCollection('products');
+```
+
+---
 
-// Get reference to an existing index
-const index = await client.getIndex('my_vectors');
+## Insert objects (combining vector types)
 
-// Delete an index
-await client.deleteIndex('my_vectors');
+`collection.upsert(objects)` takes an array of objects. Each object is:
+
+```typescript
+{
+  id: '<unique string id>',
+  meta?: { /* ... */ },     // arbitrary JSON, returned in search results (optional)
+  filter?: { /* ... */ },   // tag object used by filtered search (optional)
+  fields?: {                // field_name -> value, ONE entry per field you set
+    '<dense_field>':  [/* float, ... */],                       // dense  → array
+    '<sparse_field>': { indices: [/* ... */], values: [/* ... */] }, // sparse → object
+    '<multi_field>':  [[/* float, ... */], [/* float, ... */]], // multi  → array of arrays
+  },
+}
 ```
 
-## Upserting Vectors
+Per-field value formats:
 
-The `index.upsert()` method adds or updates vectors in an existing index.
+| Field type | Value format |
+|------------|--------------|
+| `vector` | `[0.1, 0.2, ...]` — a flat array of floats |
+| `sparse` | `{ indices: [3, 17], values: [0.9, 0.5] }` (also accepts `sparse_indices`/`sparse_values`) |
+| `multi_vector` | `[[...], [...], ...]` — an array of equal-length float arrays |
 
-```typescript
-const index = await client.getIndex('my_index');
+### One object, multiple field types
+
+A single object can populate **any combination** of the collection's fields in
+one call. They're stored together under the same id and can be searched
+independently or together:
 
-await index.upsert([
+```typescript
+await collection.upsert([
   {
-    id: 'vec1',
-    vector: [0.1, 0.2, 0.3 /* ... */],
-    meta: { title: 'First document' },
-    filter: { category: 'tech' },
+    id: 'p1',
+    meta: { name: 'Wireless Headphones', price: 99 },
+    filter: { category: 'electronics' },
+    fields: {
+      embedding: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], // dense
+      keywords: { indices: [3, 17, 42], values: [0.9, 0.5, 0.2] }, // sparse
+      colbert: [
+        // multi-vector
+        [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
+        [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
+      ],
+    },
   },
   {
-    id: 'vec2',
-    vector: [0.3, 0.4, 0.5 /* ... */],
-    meta: { title: 'Second document' },
-    filter: { category: 'science' },
+    id: 'p2',
+    meta: { name: 'Running Shoes' },
+    filter: { category: 'footwear' },
+    fields: {
+      // An object may set only SOME fields — here just dense + sparse.
+      embedding: [0.5, 0.4, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01],
+      keywords: { indices: [5, 17, 90], values: [0.7, 0.6, 0.1] },
+    },
   },
 ]);
+// → { upserted: 2 }
 ```
 
-**Vector Object Fields:**
+Notes:
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | Unique identifier for the vector |
-| `vector` | Yes | Array of floats representing the embedding |
-| `meta` | No | Arbitrary metadata object |
-| `filter` | No | Key-value pairs for filtering during queries |
+- You don't have to fill every field on every object — set whatever subset you have.
+- `upsert` is insert-or-replace by `id`.
+- Cosine `vector`/`multi_vector` values are normalized client-side before sending;
+  the original norm is stored so `getObjects` returns the original (pre-normalization) vectors.
+- Validation is done client-side (fails fast): batch size ≤ 10,000, no duplicate
+  ids in a batch, dense/multi dims must match the field, sparse `indices`/`values`
+  lengths must match, filter keys ≤ 128 bytes / string values ≤ 1024 bytes.
 
-## Querying the Index
+---
 
-The `index.query()` method performs a similarity search.
+## Search
 
-```typescript
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],
-  topK: 5,
-  ef: 128,
-  includeVectors: true,
-});
-
-for (const item of results) {
-  console.log(`ID: ${item.id}, Similarity: ${item.similarity}`);
-}
-```
+`collection.search({ fields, limit, filter })` queries one or more fields **in a
+single request** and **always returns one ranked list per field**, keyed by field
+name. To collapse those lists into a single ranked list, fuse them with
+[`rerank()`](#fusing-results-with-rerank).
 
-**Query Parameters:**
+### The unified field shape
 
-| Parameter | Description |
-|-----------|-------------|
-| `vector` | Query vector (must match index dimension) |
-| `topK` | Number of results to return (default: 10, max: 512) |
-| `ef` | Search quality parameter (default: 128, max: 1024) |
-| `includeVectors` | Include vector data in results (default: false) |
-| `filter` | Optional filter criteria (array of filter objects) |
-| `sparseIndices` | Sparse vector indices for hybrid search (default: null) |
-| `sparseValues` | Sparse vector values for hybrid search (default: null) |
-| `prefilterCardinalityThreshold` | Controls when search switches from HNSW filtered search to brute-force prefiltering on the matched subset (default: 10,000, range: 1,000–1,000,000). See [Filter Tuning](#filter-tuning) for details. |
-| `filterBoostPercentage` | Expands the internal HNSW candidate pool by this percentage when a filter is active, compensating for filtered-out results (default: 0, range: 0–100). See [Filter Tuning](#filter-tuning) for details. |
+Every field is queried with the **same object form** — `{ query, weight?, ef_search? }`.
+There is no bare-value shorthand; `query` is required. The `query` value matches
+the field's type:
 
-## Filtered Querying
+| Field type | `query` shape |
+|------------|-------------|
+| `vector` | `[0.2, 0.2, ...]` |
+| `sparse` | `{ indices: [3, 17], values: [0.8, 0.4] }` |
+| `multi_vector` | `[[...], [...]]` |
 
-Use the `filter` parameter to restrict results. All filters are combined with **logical AND**.
+Each result **hit** is:
 
 ```typescript
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],
-  topK: 5,
-  filter: [
-    { category: { $eq: 'tech' } },
-    { score: { $range: [80, 100] } },
-  ],
-});
+{ id: 'p1', similarity: 0.94, meta: { /* ... */ }, filter: { /* ... */ } }
 ```
 
-### Filtering Operators
-
-| Operator | Description | Example |
-|----------|-------------|---------|
-| `$eq` | Exact match | `{ status: { $eq: 'published' } }` |
-| `$in` | Match any in list | `{ tags: { $in: ['ai', 'ml'] } }` |
-| `$range` | Numeric range (inclusive) | `{ score: { $range: [70, 95] } }` |
-
-> **Note:** The `$range` operator supports values within **[0 - 999]**. Normalize larger values before upserting.
+> Search results carry `meta`/`filter` but **not** the stored vectors. To get the
+> vectors back, use [`getObjects`](#object-operations).
 
-### Filter Tuning
+### Per-field `weight` and `ef_search`
 
-When using filtered queries, two optional parameters let you tune the trade-off between search speed and recall:
+`weight` is the **fraction of the request `limit` to draw from that field**: the
+effective per-field limit is `ceil(weight * limit)`. Omit `weight` to use the full
+`limit`. `ef_search` tunes that field's HNSW recall/latency.
 
-#### `prefilterCardinalityThreshold`
-
-Controls when the search strategy switches from **HNSW filtered search** (fast, graph-based) to **brute-force prefiltering** (exhaustive scan on the matched subset).
+```typescript
+// limit 50 → embedding fetches ceil(0.7*50)=35, keywords fetches ceil(0.3*50)=15
+await collection.search({
+  fields: {
+    embedding: { query: Array(8).fill(0.2), weight: 0.7, ef_search: 256 },
+    keywords: { query: { indices: [3, 17], values: [0.8, 0.4] }, weight: 0.3 },
+  },
+  limit: 50,
+});
+```
 
-| Value | Behavior |
-|-------|----------|
-| `1,000` | Prefilter only for very selective filters — minimum value |
-| `10,000` | Prefilter only when the filter matches ≤10,000 vectors **(default)** |
-| `1,000,000` | Prefilter for almost all filtered searches — maximum value |
+### Single-field search
 
-The intuition: when very few vectors match your filter, HNSW may struggle to find enough valid candidates through graph traversal. In that case, scanning the filtered subset directly (prefiltering) is faster and more accurate. Raising the threshold means prefiltering kicks in more often; lowering it favors HNSW graph search.
+A single field is just the multi-field shape with one entry. `results` is keyed
+by that field name:
 
 ```typescript
-// Only prefilter when filter matches ≤5,000 vectors
-const results = await index.query({
-  vector: [/* ... */],
-  topK: 10,
-  filter: [{ category: { $eq: 'rare' } }],
-  prefilterCardinalityThreshold: 5000,
+const res = await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2) } },
+  limit: 3,
+});
+res.results.embedding;
+// [
+//   { id: "p1", similarity: 0.97, meta: {...}, filter: {...} },
+//   { id: "p2", similarity: 0.88, meta: {...}, filter: {...} },
+// ]
+
+// sparse
+await collection.search({
+  fields: { keywords: { query: { indices: [3, 17], values: [0.8, 0.4] } } },
+  limit: 3,
 });
-```
 
-#### `filterBoostPercentage`
+// multi-vector
+await collection.search({
+  fields: { colbert: { query: [Array(8).fill(0.1), Array(8).fill(0.2)] } },
+  limit: 3,
+});
+```
 
-When using HNSW filtered search, some candidates explored during graph traversal are discarded by the filter, which can leave you with fewer results than `topK`. `filterBoostPercentage` compensates by expanding the internal candidate pool before filtering is applied.
+### Multi-field search
 
-- `0` → no boost, standard candidate pool size **(default)**
-- `20` → fetch 20% more candidates internally before applying the filter
-- Maximum: `100` (doubles the candidate pool)
+Query several fields and you get **each field's own ranked list, unfused** —
+`results` is an object keyed by field name. Merge or display them however you
+like, or hand them to [`rerank()`](#fusing-results-with-rerank).
 
 ```typescript
-// Fetch 30% more candidates to compensate for aggressive filtering
-const results = await index.query({
-  vector: [/* ... */],
-  topK: 10,
-  filter: [{ visibility: { $eq: 'public' } }],
-  filterBoostPercentage: 30,
+const res = await collection.search({
+  fields: {
+    embedding: { query: [0.2, 0.2, 0.3, 0.3] },
+    keywords: { query: { indices: [3, 17], values: [0.8, 0.4] } },
+  },
+  limit: 2,
 });
 ```
 
-#### Using Both Together
+```json
+{
+  "results": {
+    "embedding": [
+      { "id": "p1", "meta": { "name": "Wireless Headphones" },
+        "filter": { "category": "electronics" }, "similarity": 0.4938 },
+      { "id": "p2", "meta": { "name": "Running Shoes" },
+        "filter": { "category": "footwear" }, "similarity": 0.4505 }
+    ],
+    "keywords": [
+      { "id": "p1", "meta": { "name": "Wireless Headphones" },
+        "filter": { "category": "electronics" }, "similarity": 0.9200 },
+      { "id": "p2", "meta": { "name": "Running Shoes" },
+        "filter": { "category": "footwear" }, "similarity": 0.5591 }
+    ]
+  }
+}
+```
 
 ```typescript
-const results = await index.query({
-  vector: [/* ... */],
-  topK: 10,
-  filter: [{ category: { $eq: 'rare' } }],
-  prefilterCardinalityThreshold: 5000,  // switch to brute-force for small match sets
-  filterBoostPercentage: 25,             // boost candidates for HNSW filtered search
-});
+res.results.embedding; // this field's ranked hits
+res.results.keywords; // that field's ranked hits
 ```
 
-> **Tip:** Start with the defaults (`prefilterCardinalityThreshold: 10,000`, `filterBoostPercentage: 0`). If filtered queries return fewer results than expected, try increasing `filterBoostPercentage`. If filtered queries are slow on selective filters, try lowering `prefilterCardinalityThreshold`. Valid range for the threshold is `1,000–1,000,000`.
-
-## Hybrid Search
+> `similarity` is in **each field's own scale** (cosine for dense ≈ 0.49, sparse
+> dot for keywords ≈ 0.92) — scores are **not comparable across fields**. That's
+> exactly why fusion is a separate, opt-in step.
 
-### Upserting Hybrid Vectors
+### Fusing results with `rerank()`
 
-Provide both dense vectors and sparse representations:
+`rerank(searchResults, { name, limit, fieldWeights })` fuses the per-field lists
+from `search()` into **one ranked list** using Reciprocal Rank Fusion. It's a
+standalone function — run `search()`, then pass its result to `rerank()`:
 
 ```typescript
-const index = await client.getIndex('hybrid_index');
+import { rerank } from 'endee';
 
-await index.upsert([
-  {
-    id: 'doc1',
-    vector: [0.1, 0.2 /* ... */],           // Dense vector
-    sparseIndices: [10, 50, 200],            // Non-zero term positions
-    sparseValues: [0.8, 0.5, 0.3],           // Weights for each position
-    meta: { title: 'Document 1' },
-  },
-  {
-    id: 'doc2',
-    vector: [0.3, 0.4 /* ... */],
-    sparseIndices: [15, 100, 500],
-    sparseValues: [0.9, 0.4, 0.6],
-    meta: { title: 'Document 2' },
+const res = await collection.search({
+  fields: {
+    embedding: { query: [0.2, 0.2, 0.3, 0.3] },
+    keywords: { query: { indices: [3, 17], values: [0.8, 0.4] } },
+    colbert: { query: [Array(8).fill(0.1), Array(8).fill(0.2)] },
   },
-]);
+  limit: 50,
+});
+
+const fused = rerank(res, {
+  name: 'rrf',
+  limit: 10,
+  // optional: weight each field's contribution (must sum to 1.0; equal by default)
+  fieldWeights: { embedding: 0.5, keywords: 0.3, colbert: 0.2 },
+  // optional: RRF rank constant k (default 60)
+  rrfK: 60,
+});
 ```
 
-**Hybrid Vector Fields:**
+```json
+{
+  "results": [
+    { "id": "p1", "similarity": 0.0164,
+      "meta": { "name": "Wireless Headphones" }, "filter": { "category": "electronics" } },
+    { "id": "p3", "similarity": 0.0160,
+      "meta": { "name": "Smart Watch" }, "filter": { "category": "electronics" } },
+    { "id": "p2", "similarity": 0.0160,
+      "meta": { "name": "Running Shoes" }, "filter": { "category": "footwear" } }
+  ]
+}
+```
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | Unique identifier |
-| `vector` | Yes | Dense embedding vector |
-| `sparseIndices` | Yes (hybrid) | Non-zero term positions in sparse vector |
-| `sparseValues` | Yes (hybrid) | Weights for each sparse index |
-| `meta` | No | Metadata dictionary |
-| `filter` | No | Filter fields |
+`name` selects the algorithm (only `"rrf"` is supported today). Each fused hit
+keeps the `meta`/`filter` from its per-field hit; `similarity` is replaced with
+the RRF score (sum of `weight / (rrfK + rank)` across the fields it appeared in).
 
-> **Important:** `sparseIndices` and `sparseValues` must have the same length. Values in `sparseIndices` must be within `[0, sparseDimension)`.
+**Summary of return shapes**
 
-### Querying Hybrid Index
+| Call | `results` shape |
+|------|-----------------|
+| `search(...)` (1 or N fields) | `Record<field_name, SearchHit[]>` |
+| `rerank(searchResults, { name: 'rrf', ... })` | `SearchHit[]` (fused) |
 
-Provide both dense and sparse query vectors:
+### Filtered search
+
+`filter` applies to any search. It's an array of conditions:
 
 ```typescript
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],       // Dense query
-  sparseIndices: [10, 100, 300],         // Sparse query positions
-  sparseValues: [0.7, 0.5, 0.4],         // Sparse query weights
-  topK: 5,
+await collection.search({
+  fields: { embedding: { query: Array(8).fill(0.2) } },
+  limit: 5,
+  filter: [{ category: { $eq: 'electronics' } }],
 });
-
-for (const item of results) {
-  console.log(`ID: ${item.id}, Similarity: ${item.similarity}`);
-}
 ```
 
-**Hybrid Query Parameters:**
-
-| Parameter | Description |
-|-----------|-------------|
-| `vector` | Dense query vector |
-| `sparseIndices` | Sparse query positions |
-| `sparseValues` | Sparse query weights |
-| `topK` | Number of results to return |
-| `ef` | Search quality parameter (max 1024, default: 128) |
-| `includeVectors` | Include vector data in results (default: false) |
-| `filter` | Optional filter criteria |
-| `prefilterCardinalityThreshold` | Controls when search switches from HNSW filtered search to brute-force prefiltering on the matched subset (default: 10,000, range: 1,000–1,000,000). See [Filter Tuning](#filter-tuning) for details. |
-| `filterBoostPercentage` | Expands the internal HNSW candidate pool by this percentage when a filter is active, compensating for filtered-out results (default: 0, range: 0–100). See [Filter Tuning](#filter-tuning) for details. |
+Operators: `$eq`, `$in`, `$range`, `$gt`, `$gte`, `$lt`, `$lte` (see [Reference](#reference)).
 
-You can also query with:
-- **Dense only**: Provide only `vector`
-- **Sparse only**: Provide only `sparseIndices` and `sparseValues`
-- **Hybrid**: Provide all three for combined results
+---
 
-## Updating Filters
-
-Use `index.updateFilters()` to update the filter fields of existing vectors without re-upserting them.
+## Object operations
 
 ```typescript
-await index.updateFilters([
-  { id: 'vec1', filter: { category: 'ml', score: 95 } },
-  { id: 'vec2', filter: { category: 'science', score: 80 } },
-]);
+// Fetch full stored objects by id (meta, filter, and the stored vectors).
+const objs = await collection.getObjects(['p1', 'p2']);
+// [
+//   { id: "p1", meta: {...}, filter: {...},
+//     vectors: { embedding: [...] },               // dense fields (original, pre-normalization)
+//     sparses: { keywords: { indices: [...], values: [...] } },
+//     multi_vectors: { colbert: [[...], [...]] } },
+//   ...
+// ]
+
+// Delete a single object by id.
+await collection.deleteObject('p1'); // { deleted: "p1" }
+
+// Delete every object matching a filter.
+await collection.deleteByFilter([{ category: { $eq: 'footwear' } }]); // { deleted: <count> }
+
+// Update only the filter tags on existing objects (no re-upsert of vectors).
+await collection.updateFilters([{ id: 'p2', filter: { category: 'sale' } }]); // { updated: <count> }
 ```
 
-**`UpdateFilterParams` Fields:**
+---
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `id` | Yes | ID of the vector to update |
-| `filter` | Yes | New filter object to replace the existing filter |
+## Collection maintenance
 
-> **Note:** The entire filter object is replaced, not merged.
+```typescript
+await collection.describe(); // { name, fields, created_at, layout_version }
 
-## Deletion Methods
+// Rebuild a field's HNSW graph with new M / ef_con (async).
+await collection.rebuild({ field: 'embedding', m: 24, efCon: 200 }); // { status: "in_progress", ... }
+await collection.rebuildStatus(); // { percent_complete, ... }
 
-### Delete by ID
-Delete vector with a specifc vector id.
-```typescript
-await index.deleteVector('vec1');
-```
-### Delete by Filter
-Delete all vectors matching specific filters.
-```typescript
-await index.deleteWithFilter([{'category': {'$eq' : 'tech'}}]);
+// Defragment storage in place.
+await collection.shrink(); // { status: "ok", reclaimed_bytes: <int> }
 ```
 
-### Delete Index
-Delete an entire Index.
+---
+
+## Backups
+
+Backups are per-database. Creating one is per-collection and **asynchronous**.
+
 ```typescript
-await client.deleteIndex('my_index');
-```
+// Start a backup of a collection (async).
+await collection.createBackup('nightly'); // { backup_name: "nightly", status: "in_progress" }
 
-> **Warning:** Deletion operations are **irreversible**.
+// Poll for completion.
+while ((await client.activeBackup()).active) {
+  // ...wait, then re-check
+}
 
-## Additional Operations
+await client.listBackups(); // { nightly: { ...metadata }, ... }
+await client.backupInfo('nightly'); // metadata for one backup
 
-### Get Vector by ID
+// Restore a backup into a new collection.
+await client.restoreBackup('nightly', 'products_restored');
 
-```typescript
-const vector = await index.getVector('vec1');
+// Move backups around.
+await client.downloadBackup('nightly', '/tmp/nightly.tar'); // writes a .tar, returns the path
+await client.uploadBackup('/tmp/nightly.tar'); // uploads a .tar into this db
+
+await client.deleteBackup('nightly');
 ```
 
-### Describe Index
+> `downloadBackup` / `uploadBackup` use Node's filesystem (`fs`). With the **root
+> token**, pass a third `dbName` arg to `downloadBackup` to target a specific
+> database's backup.
 
-```typescript
-const info = index.describe();
-console.log(info);
-// { name, spaceType, dimension, sparseDimension, isHybrid, count, precision, M }
-```
+---
 
-## Precision Options
+## Database administration (root token)
 
-Endee supports different quantization precision levels:
+Database and token administration requires the server's **root token** (its
+`NDD_ROOT_TOKEN`). Create an admin client with that token:
 
 ```typescript
-import { Precision } from 'endee';
-
-Precision.BINARY;   // Binary quantization (1-bit) - smallest storage, fastest search
-Precision.INT8;    // 8-bit integer quantization - balanced performance
-Precision.INT16;   // 16-bit integer quantization (default) - higher precision
-Precision.FLOAT16;  // 16-bit floating point - good balance
-Precision.FLOAT32;  // 32-bit floating point - highest precision
+const admin = new Endee('roottoken');
+admin.setBaseUrl('http://localhost:8080/api/v2');
 ```
 
-**Choosing Precision:**
+> The root token administers databases/tokens; it **cannot run data ops directly**.
+> A db token (which it mints) does the collection/object/search work.
 
-- `BINARY`: Best for very large datasets where speed and storage are critical
-- `INT8` : Recommended for most use cases - good balance of accuracy and performance
-- `INT16` (default) : When you need better accuracy than INT8 but less storage than FLOAT32
-- `FLOAT16`: Good compromise between precision and storage for embeddings
-- `FLOAT32`: When you need maximum precision and storage is not a concern
-
-## Complete Example
+### Databases
 
 ```typescript
-import { Endee, Precision } from 'endee';
+// Create a database → returns the new db_token.
+const res = await admin.createDatabase('my_db', 'scale'); // db_type: starter|pro|scale|enterprise
+const dbToken = res.db_token;
 
-// Initialize client
-const client = new Endee();
+await admin.listDatabases(); // [{ db_name, db_type, is_active, created_at }, ...]
+await admin.getDatabase('my_db'); // single db info
 
-// Create a dense index
-await client.createIndex({
-  name: 'documents',
-  dimension: 384,
-  spaceType: 'cosine',
-  precision: Precision.INT16,
-});
+await admin.setDatabaseType('my_db', 'pro'); // change tier
+await admin.deactivateDatabase('my_db'); // block its tokens (data is kept)
+await admin.activateDatabase('my_db'); // re-enable
+await admin.deleteDatabase('my_db'); // delete db + ALL its data
+```
 
-// Get the index
-const index = await client.getIndex('documents');
+### Collections across databases (admin view)
 
-// Add vectors
-await index.upsert([
-  {
-    id: 'doc1',
-    vector: [0.1, 0.2 /* ... 384 dimensions */],
-    meta: { title: 'First Document' },
-    filter: { category: 'tech' },
-  },
-  {
-    id: 'doc2',
-    vector: [0.3, 0.4 /* ... 384 dimensions */],
-    meta: { title: 'Second Document' },
-    filter: { category: 'science' },
-  },
-]);
+```typescript
+await admin.listDbCollections('my_db'); // ["products", ...] for one db
+await admin.listAllCollections(); // [{ db_name: "my_db", collections: [...] }, ...]
+await admin.deleteDbCollection('my_db', 'products');
+```
 
-// Query the index
-const results = await index.query({
-  vector: [0.15, 0.25 /* ... */],
-  topK: 5,
-});
+### Tokens for any database
 
-for (const item of results) {
-  console.log(`ID: ${item.id}, Similarity: ${item.similarity}`);
-}
+```typescript
+// token_type: "rw" (read-write, default) or "r" (read-only).
+const t = await admin.createToken('my_db', 'ci-reader', 'r'); // { db_token, ... }
+await admin.listTokens('my_db'); // [{ name, token_type, created_at }, ...] (no secrets)
+await admin.deleteToken('my_db', 'ci-reader');
 ```
 
-## Collections & Objects API
+### Self-service tokens (your own database)
 
-Version 1.5+ ships a new, friendlier naming convention alongside the original API. Both names are fully supported — existing code using `Index` / `getVector` / `deleteVector` continues to work without any changes.
+Using a **db token**, you can manage your own database's tokens:
 
-| Old name | New name | Notes |
-|----------|----------|-------|
-| `createIndex` | `createCollection` | Same signature |
-| `listIndexes` | `listCollections` | Same signature |
-| `deleteIndex` | `deleteCollection` | Same signature |
-| `getIndex` | `getCollection` | Returns same object |
-| `getVector` | `getObject` | Method on the collection handle |
-| `deleteVector` | `deleteObject` | Method on the collection handle |
-| `Index` (export) | `Collection` (export) | Same class, two names |
+```typescript
+await client.createMyToken('reader', 'r'); // { db_token, ... }
+await client.listMyTokens();
+await client.deleteMyToken('reader');
+```
 
-**Example using the new naming:**
+### Server info
 
 ```typescript
-import { Endee, Collection, Precision } from 'endee';
+await client.health(); // { status, timestamp }
+await client.stats(); // { version, uptime, total_requests }
+```
 
-const client = new Endee();
+---
 
-// Create
-await client.createCollection({ name: 'products', dimension: 384, spaceType: 'cosine' });
+## Reference
 
-// Get a handle
-const collection = await client.getCollection('products');
+**Distance / space types** (`space_type`)
 
-// Upsert objects
-await collection.upsert([
-  { id: 'p1', vector: [0.1, 0.2 /* ... */], meta: { title: 'Shoes' } },
-]);
+| Value | Meaning |
+|-------|---------|
+| `cosine` | Cosine similarity (vectors L2-normalized client-side). Default. |
+| `l2` | Euclidean distance. |
+| `ip` | Inner product. |
 
-// Fetch an object by ID
-const obj = await collection.getObject('p1');
+**Precisions** (`precision`) — memory/accuracy trade-off
 
-// Delete an object by ID
-await collection.deleteObject('p1');
+`float32`, `float16`, `int16` (default), `int8`, `int8e`, `binary`. The
+`Precision` enum is exported for convenience:
 
-// Delete the collection
-await client.deleteCollection('products');
+```typescript
+import { Precision } from 'endee';
+Precision.INT8; // "int8"
 ```
 
-The `Collection` export is an alias for `Index` — both refer to the same class and have identical methods.
-
-## API Reference
+**Token types**: `rw` (read-write), `r` (read-only).
 
-### Endee Class
+**Filter operators**
 
-| Method | Alias | Description |
-|--------|-------|-------------|
-| `createIndex(options)` | `createCollection(options)` | Create a new index / collection |
-| `listIndexes()` | `listCollections()` | List all indexes / collections |
-| `deleteIndex(name)` | `deleteCollection(name)` | Delete an index / collection |
-| `getIndex(name)` | `getCollection(name)` | Get a handle to an index / collection |
-| `setBaseUrl(url)` | — | Set a custom base URL |
+| Operator | Example | Meaning |
+|----------|---------|---------|
+| `$eq` | `{ category: { $eq: 'news' } }` | equals |
+| `$in` | `{ tag: { $in: ['a', 'b'] } }` | in set |
+| `$range` | `{ price: { $range: [10, 100] } }` | inclusive range |
+| `$gt` / `$gte` | `{ price: { $gt: 50 } }` | greater than / or equal |
+| `$lt` / `$lte` | `{ price: { $lte: 100 } }` | less than / or equal |
 
-### Index / Collection Class
+**Client-side validation / limits**
 
-| Method | Alias | Description |
-|--------|-------|-------------|
-| `upsert(vectors)` | — | Insert or update vectors / objects |
-| `query(options)` | — | Search for similar vectors / objects |
-| `updateFilters(updates)` | — | Update filter fields on existing vectors by ID |
-| `deleteVector(id)` | `deleteObject(id)` | Delete a vector / object by ID |
-| `deleteWithFilter(filter)` | — | Delete vectors / objects by filter |
-| `getVector(id)` | `getObject(id)` | Get a vector / object by ID |
-| `describe()` | — | Get index / collection info |
+| Limit | Value |
+|-------|-------|
+| Objects per `upsert` | ≤ 10,000 |
+| `limit` (top-k) | 1 … 4096 |
+| `efSearch` | 1 … 1024 |
+| Filter key | ≤ 128 bytes |
+| Filter string value | ≤ 1024 bytes |
+| Vector dimension | must match the field's configured dimension |
 
-## TypeScript Types
-
-The package includes comprehensive TypeScript types:
+**Exported TypeScript types**
 
 ```typescript
 import type {
-  VectorItem,
-  QueryOptions,
-  QueryResult,
-  CreateIndexOptions,
-  IndexDescription,
+  FieldDefinition,
+  FieldType,
+  ObjectInput,
+  SearchOptions,
+  SearchHit,
+  SearchResponse,
+  RerankOptions,
+  RerankResponse,
+  FullObject,
+  CollectionMetadata,
+  UpdateFilterEntry,
+  RebuildOptions,
+  DatabaseInfo,
+  TokenInfo,
+  DbType,
+  TokenType,
   SpaceType,
   Precision,
-  UpdateFilterParams,
 } from 'endee';
 ```
 
-## License
+---
 
-MIT
+## Error handling
 
-## Author
+Non-2xx responses throw typed exceptions:
 
-Harshit Raj
+```typescript
+import {
+  EndeeException, // base class for all client errors
+  APIException, // generic 4xx (e.g. bad request)
+  AuthenticationException, // 401
+  ForbiddenException, // 403 (e.g. read-only token on a write)
+  NotFoundException, // 404
+  ConflictException, // 409 (e.g. duplicate collection)
+  ServerException, // 5xx
+} from 'endee';
+
+try {
+  await collection.upsert(objects);
+} catch (e) {
+  if (e instanceof NotFoundException) {
+    console.error('not found:', e.message);
+  } else if (e instanceof EndeeException) {
+    // catches all of the above
+    console.error('request failed:', e.message);
+  } else {
+    throw e;
+  }
+}
+```
+
+All exception classes are importable from `endee`: `EndeeException` (base),
+`APIException`, `AuthenticationException`, `ForbiddenException`,
+`NotFoundException`, `ConflictException`, `ServerException`,
+`SubscriptionException`.
+
+Client-side input mistakes (bad dimensions, mismatched sparse lengths, oversized
+batch, sum-of-weights ≠ 1.0, etc.) throw a plain `Error` **before** any network
+call.
+
+---
+
+## End-to-end example
+
+See `tests/integration_v2.ts` — a full walkthrough (create → multi-field upsert →
+every search mode → object ops → maintenance → cleanup). Run it against a local
+server with a db token:
+
+```bash
+npx tsx tests/integration_v2.ts --token my_db:xxxx --url http://localhost:8080/api/v2
+```
+
+## License
+
+MIT