@dreamlake/dreamdb 0.3.0 → 0.4.0

package/README.md

@@ -0,0 +1,178 @@
+# @dreamlake/dreamdb
+
+The DreamDB SDK — read and write, in the browser and on the server, compiled to
+WebAssembly from the same Rust core the CLI and Python SDK use.
+
+There is no separate JavaScript implementation of the protocol, and that is the
+point. A hand-written port has to reproduce BLAKE3, canonical CBOR, spatial-key
+encoding, bucket headers and index layout *bit for bit*, forever. The previous
+TypeScript port did not, and its tests were all green anyway, because they were
+its own reader reading its own writer.
+
+```bash
+npm install @dreamlake/dreamdb
+```
+
+## Reading
+
+```js
+import { Space } from '@dreamlake/dreamdb'
+
+const space = await Space.fromUri('https://bucket.s3.amazonaws.com/refs/my-dataset', null)
+const hits = await space.queryVector('visual', queryVec, 24, 8)
+```
+
+Passing `null` for the backend uses direct `fetch` against the URI's base —
+enough for a public bucket. For anything else, supply a Backend.
+
+## Writing
+
+```js
+import { Writer, PresignedBackend } from '@dreamlake/dreamdb'
+
+const backend = new PresignedBackend({
+  readBase: 'https://bucket.s3.amazonaws.com',
+  mintPut: async (paths) => (await postJson('/api/dreamdb/sign', { paths })).urls,
+  commitRef: async (path, opts, bytes) =>
+    await postJson('/api/dreamdb/ref', {
+      path,
+      bytesBase64: base64(bytes),
+      ifMatch: opts.ifMatch,
+      ifNoneMatchStar: opts.ifNoneMatchStar,
+    }),
+})
+
+const w = await Writer.open('https://bucket.s3.amazonaws.com/refs/my-dataset', backend)
+await w.appendMany([
+  {
+    anchor: 1735689600000000000n,          // nanoseconds — pass a bigint
+    visual: { kind: 'embedding', algorithm: 'dreamdb.lsh-cosine', vector: vec },
+    caption: { kind: 'categorical', value: 'a red car' },
+  },
+])
+const manifest = await w.commit()
+```
+
+Credentials never enter the browser: your server mints short-lived presigned
+PUT URLs, and performs the ref's compare-and-swap itself.
+
+### Anchors are nanoseconds, and you should pass a `bigint`
+
+A `number` is accepted only while it is below `Number.MAX_SAFE_INTEGER`; above
+that the SDK throws rather than round it. This is not pedantry. The previous
+TypeScript SDK wrote anchors in **microseconds** specifically to stay under
+2^53, and every record it produced reads as 1970 in any conformant reader.
+
+## Server-side authoring
+
+`Authoring` adds dataset creation, layers, merge, compaction and history. It is
+**Node only** — the browser build exports a stub whose methods throw with an
+explanation.
+
+```js
+import { Authoring } from '@dreamlake/dreamdb'   // resolves to the Node build
+
+const a = await Authoring.create(
+  'my-dataset',
+  [
+    { name: 'visual', kind: 'embedding', dim: 768, algorithm: 'dreamdb.lsh-cosine' },
+    { name: 'caption', kind: 'scalar', valueType: 'categorical', required: false },
+  ],
+  'https://bucket.s3.amazonaws.com',
+  backend,
+)
+await a.writer().appendMany(samples)
+await a.compact(null, 1, 0)
+```
+
+The split is a size boundary with a mechanism behind it. `create` and the
+embedding-layer builders construct a `SpatialDispatcher`, which is an enum — so
+one reachable construction makes *every* index family's build code reachable
+(IVF, IMI, LSH, Vamana, AdaIVF, plus codebook training). Measured: 484 KB
+gzipped for the browser build versus 607 KB with authoring included.
+
+Embedding fields declaring `dreamdb.ivf-cosine` or `dreamdb.imi-cosine` cannot
+be created here at all: those index families need training data that does not
+exist at create time. Create with the default `dreamdb.lsh-cosine` and attach a
+trained index as a layer, or build it with the CLI.
+
+## Entry points
+
+| Import | Resolves to | Notes |
+| --- | --- | --- |
+| `@dreamlake/dreamdb` in a bundler | `--target bundler` | Needs `vite-plugin-wasm` under Vite |
+| `@dreamlake/dreamdb` in Node | `--target nodejs` | ESM and CJS both work; includes `Authoring` |
+| `@dreamlake/dreamdb/web` | `--target web` | No bundler plugin needed; `await ready()` first |
+
+```html
+<script type="module">
+  import ready, { Space } from 'https://esm.sh/@dreamlake/dreamdb/web'
+  await ready()                 // fetches and instantiates the wasm
+</script>
+```
+
+## Implementing a Backend
+
+Only `get` is required; a `get`-only backend is the read-only v1 shape and
+still works for reading. Write entry points check for `put` up front and refuse
+with a message naming what is missing.
+
+```ts
+interface Backend {
+  get(path: string, range?: { start: number; end: number })
+    : Promise<Uint8Array | { bytes: Uint8Array; etag?: string }>
+  head?(path: string): Promise<{ exists?: boolean; etag?: string; size?: number }>
+  put?(path: string, bytes: Uint8Array,
+       opts?: { ifMatch?: string; ifNoneMatchStar?: boolean })
+    : Promise<{ status: 'created' | 'exists' | 'casFailed'; etag?: string }>
+  delete?(path: string): Promise<void>
+  list?(prefix: string): Promise<string[]>
+}
+```
+
+### Two things that will bite you
+
+**Your bucket's CORS rule must include `ExposeHeaders: ["ETag"]`.**
+
+A ref advances by compare-and-swap. The SDK sends `If-Match: <etag>` when it
+knows the ref's current version and `If-None-Match: *` when it does not. Without
+`ExposeHeaders`, the browser can read the response body but not the ETag header,
+so the SDK never learns the current version, falls back to create-only, and the
+**second** commit to any ref fails while the first succeeded. That asymmetry is
+what makes it hard to recognise.
+
+```json
+{
+  "AllowedOrigins": ["https://your.app"],
+  "AllowedMethods": ["GET", "HEAD", "PUT"],
+  "AllowedHeaders": ["*"],
+  "ExposeHeaders": ["ETag", "Content-Range", "Content-Length"]
+}
+```
+
+**Any HTTP layer you put in front of the bucket must honour `Range`.**
+
+DreamDB reads exact vectors by byte offset. A server that ignores `Range` and
+returns 200 with the whole object makes every ranged read return offset 0, and
+search then returns plausible-looking results with cosine scores of 0.0000 —
+indistinguishable from a genuinely corrupt index. (`python3 -m http.server`
+does exactly this.)
+
+## Conformance
+
+`dreamdb-conformance/vectors/` holds 85 language-agnostic JSON vectors. This
+package runs the 48 that map to pure functions on its surface — canonical CBOR,
+BLAKE3 multihash, spatial keys, time anchors and buckets, address round-trips,
+modality parsing, HTTP range translation — and reports the other 37 by name and
+reason rather than omitting them silently.
+
+```bash
+node test/conformance.mjs
+```
+
+The first run of that suite found a real bug in this package's HTTP range
+translation, which is roughly the point.
+
+## Licence
+
+MIT OR Apache-2.0

package/browser/dreamdb.d.ts

@@ -0,0 +1,326 @@
+/* tslint:disable */
+/* eslint-disable */
+
+/**
+ * A `fetch`-backed backend rooted at a base URL.
+ */
+export class S3Backend {
+    free(): void;
+    [Symbol.dispose](): void;
+    /**
+     * Fetch `path` (relative to the base URL) and resolve to its bytes.
+     * `opts.noCache` is accepted for API compatibility (currently a no-op:
+     * content is hash-addressed; refs are read fresh each page load anyway).
+     */
+    get(path: string, _opts: any): Promise<Uint8Array>;
+    /**
+     * List object paths under `prefix`. Stub (returns empty); the current
+     * consumer read paths don't require listing.
+     */
+    list(_prefix: string): Promise<Array<any>>;
+    /**
+     * Construct rooted at `base_url` (trailing slash trimmed).
+     */
+    constructor(base_url: string);
+}
+
+export class Space {
+    private constructor();
+    free(): void;
+    [Symbol.dispose](): void;
+    /**
+     * Open a Space from a `.../refs/<name>` or `.../manifests/<hash>` URI,
+     * resolving the ref (if any) and loading the manifest. If `backend` is
+     * provided it handles IO (relative paths); otherwise a direct-fetch
+     * connector rooted at the parsed base URL is used (the `dreamdb-demo`
+     * no-backend path).
+     */
+    static fromUri(uri: string, backend: any): Promise<Space>;
+    /**
+     * Walk the manifest DAG from HEAD, up to `max_depth` entries. Each entry:
+     * `{ manifestHash, ts, writer, tracks: [{modality, address}] }`.
+     */
+    history(max_depth?: number | null): Promise<Array<any>>;
+    /**
+     * Hybrid search fusing a lexical (BM25) sub-query and a dense (vector)
+     * sub-query per spec/0015 §5/§6. Returns the fused `[{ anchor, score }]`
+     * list sorted by fused score descending — the same scored shape as
+     * `queryVector`.
+     *
+     * `vector` is the dense sub-query embedding (a `Float32Array`). `opts` is a
+     * plain JS object:
+     * ```js
+     * {
+     *   textField, textQuery,        // BM25 sub-query (required)
+     *   vectorField,                 // dense field the `vector` targets (required)
+     *   topK = 10,                   // fused results to return
+     *   fusion = 'rrf',              // 'rrf' | 'linear' | 'max' | 'pareto'
+     *   rrfK = 60,                   // RRF rank constant (fusion='rrf')
+     *   textWeight = 1, vectorWeight = 1, // Linear weights (fusion='linear')
+     *   textK = topK, vectorK = topK,     // per-sub-query candidate depths
+     *   textRequired = false, vectorRequired = false, // §5.3 boolean-AND gates
+     * }
+     * ```
+     *
+     * Delegates to the real (tested) `dreamdb-dataset::Dataset::query_hybrid`.
+     * (v0 does not expose the optional scalar pre/post-filter; it runs with no
+     * scalar gate.)
+     */
+    queryHybrid(vector: Float32Array, opts: any): Promise<Array<any>>;
+    /**
+     * Scalar-index lookup: every anchor whose `field` satisfies `op value`
+     * (spec/0011). Returns a `BigUint64Array`-style `Array` of anchors,
+     * tombstone-suppressed, sorted ascending and deduped.
+     *
+     * This is the expansion half of a compact lexical index. A BM25 index
+     * built over a scalar field's DISTINCT values returns one representative
+     * anchor per matched value; resolving that value back to *every* record
+     * carrying it is a scalar-index lookup, not a text-index concern. Keeping
+     * the split this way is what lets the text index stay small enough to
+     * load in a browser — indexing one document per record instead produced a
+     * 406 MB object for ~1,700 distinct strings.
+     *
+     * `op` is one of `==`, `!=`, `<`, `<=`, `>`, `>=`. `value` may be a
+     * string, number, or boolean; strings match both `String` and
+     * `Categorical` scalar fields.
+     */
+    queryScalar(field: string, op: string, value: any): Promise<Array<any>>;
+    /**
+     * Lexical BM25 text search over a text `field` (spec/0015 §3.6).
+     *
+     * Tokenizes `query` with the built-in `dreamdb.utf8-words` tokenizer
+     * (lowercase + split on non-alphanumeric — pure Rust, compiled into the
+     * wasm; no external model needed even though the field's embeddings were
+     * produced elsewhere), scores docs with BM25, and suppresses tombstoned
+     * anchors. Returns `[{ anchor, score }]` sorted by score descending — the
+     * same scored shape as `queryVector`.
+     *
+     * Delegates to the real (tested) `dreamdb-dataset::Dataset::query_text`.
+     */
+    queryText(field: string, query: string, top_k: number): Promise<Array<any>>;
+    /**
+     * Top-K cosine vector search over an embedding `field`. `opts` may be a
+     * number (topK) or `{ topK?, probeCount? }`. Returns `[{ anchor, score }]`
+     * sorted by score descending — the shape dreamdb-ts's `queryVector` yields.
+     *
+     * Delegates to the real (tested) `dreamdb-dataset` query path: ADC cull +
+     * two-pass rerank + HotShard merge + tombstone suppression.
+     */
+    queryVector(field: string, query: Float32Array, opts: any): Promise<Array<any>>;
+    /**
+     * Read a scalar track as a `Map<anchorString, value>`. Single-anchor
+     * entries resolve from the track index; multi-anchor entries fetch (and
+     * de-dupe) their bucket and roaring-decode the anchor set.
+     */
+    readScalarColumn(track: any, _opts: any): Promise<Map<any, any>>;
+    /**
+     * Fetch a track object and return its `object_index` (array of CBOR
+     * entries). Inline indexes only — paged indexes are not resolved (matching
+     * dreamdb-ts, which also doesn't support them on this path).
+     */
+    resolveObjectIndex(track: any): Promise<Array<any>>;
+    /**
+     * Base32 hash of the manifest's first timeline.
+     */
+    timelineId(): string;
+    /**
+     * Find a track by its resolved `key` or by `modality`.
+     */
+    trackFor(key_or_modality: string): any;
+    /**
+     * List resolved tracks from this manifest.
+     */
+    tracks(): Array<any>;
+    /**
+     * The base URL objects are fetched relative to (consumers build their own
+     * object URLs from this — e.g. `dreamdb-demo`'s track readers).
+     */
+    readonly connectorBase: string;
+    /**
+     * The manifest's base32 multihash.
+     */
+    readonly manifestHash: string;
+}
+
+/**
+ * A write handle onto a ref.
+ */
+export class Writer {
+    private constructor();
+    free(): void;
+    [Symbol.dispose](): void;
+    /**
+     * Append records and commit in one call.
+     *
+     * `samples` is an array of `{ anchor?: bigint|number, <field>: value }`.
+     * See `marshal::samples_from_js` for the accepted field shapes.
+     *
+     * Committing here rather than exposing a separate staged mode is
+     * deliberate for the browser surface: a staged write that is never
+     * committed leaves uploaded objects unreferenced, and a tab can close at
+     * any moment. `appendStaged` exists on the Node surface where the process
+     * lifetime is under the caller's control.
+     */
+    appendMany(samples: Array<any>): Promise<number>;
+    /**
+     * Flush staged entries and publish a new manifest.
+     *
+     * Fails with a CAS conflict if the ref moved underneath this writer. That
+     * is not retried automatically: an automatic retry would hide a logical
+     * conflict, and the caller is the only one who knows whether re-applying
+     * its records on top of the new head is correct.
+     */
+    commit(): Promise<string>;
+    /**
+     * Tombstone records by anchor. Returns the new manifest hash.
+     */
+    deleteRecords(anchors: BigUint64Array, reason?: string | null): Promise<string>;
+    /**
+     * Open a ref for writing.
+     *
+     * `uri` is the same `.../refs/<name>` form `Space.fromUri` takes; a
+     * `.../manifests/<hash>` URI is rejected rather than silently opening
+     * something unwritable, because a manifest hash names an immutable
+     * snapshot — there is nothing for a commit to advance.
+     *
+     * `backend` must implement `put` (Backend contract v2). Passing a
+     * read-only backend fails at the first write with a clear message rather
+     * than here, since the connector cannot know what the JS object omits
+     * until it calls it.
+     */
+    static open(uri: string, backend: any): Promise<Writer>;
+    /**
+     * Tag the current manifest with an immutable label (`refs/<ref>@<label>`).
+     */
+    snapshot(label: string): Promise<string>;
+    /**
+     * Current manifest hash, base32.
+     */
+    readonly manifestHash: string;
+    /**
+     * The ref this writer advances.
+     */
+    readonly refName: string;
+}
+
+/**
+ * Install a panic hook that logs the panic message AND its `file:line:col`
+ * to the console before the wasm trap surfaces to JS.
+ *
+ * Without this, every Rust panic reaches JavaScript as a bare
+ * `RuntimeError: unreachable` with nothing but wasm frame offsets — and it is
+ * indistinguishable from an allocation failure, since Rust's alloc-error
+ * handler also aborts. That ambiguity cost real debugging time: a browser
+ * `unreachable` from exhausting wasm32's 32-bit address space on an oversized
+ * index was initially read as an integer-overflow panic.
+ *
+ * Zero new deps — uses the already-enabled `web-sys` `console` feature.
+ * Runs once at module init.
+ */
+export function __wasm_init(): void;
+
+/**
+ * Parse an object path and re-format it. Byte-identity of the result is the
+ * actual assertion: a parser that silently drops a component still "parses".
+ */
+export function addressRoundTrip(path: string): string;
+
+/**
+ * The address variant name (`Genesis`, `Manifest`, `Ref`, …) for a path.
+ *
+ * Derived from the Debug representation rather than a hand-written match.
+ * `DreamDbAddress` has seventeen variants and gains one whenever the protocol
+ * does; a match arm per variant would be a second list to keep in sync, and
+ * the vectors only ever assert the name.
+ */
+export function addressVariant(path: string): string;
+
+/**
+ * Encode arbitrary bytes as lowercase RFC-4648 base32 (no padding).
+ *
+ * Byte-identical to `dreamdb-ts`'s `bytesToBase32` (alphabet
+ * `abcdefghijklmnopqrstuvwxyz234567`) and to `dreamdb_core`'s multihash
+ * `to_base32`, which is how object addresses are spelled on disk.
+ */
+export function bytesToBase32(bytes: Uint8Array): string;
+
+/**
+ * Decode a single CBOR value from `bytes` into its JS representation.
+ */
+export function decodeCbor(bytes: Uint8Array): any;
+
+/**
+ * Canonically encode a JS value as CBOR, returning the bytes.
+ *
+ * Canonical here means what spec/0002 §3.1 means: map keys sorted by their
+ * encoded bytes, shortest-form integers, no indefinite lengths. Two writers
+ * that disagree on this produce different content hashes for the same logical
+ * object, and every address derived from them diverges.
+ */
+export function encodeCbor(value: any): Uint8Array;
+
+/**
+ * Parse a modality tag into `{class, encoding, trackKind, objectKind, params,
+ * flags}`, or throw if it is invalid.
+ *
+ * `params` is a plain object of the `key=value` segments and `flags` an array
+ * of the bare ones (`bucketed`, `graph`). They are separate because the
+ * grammar treats them differently and merging them would make a flag
+ * indistinguishable from a parameter whose value happened to be empty.
+ */
+export function modalityParse(tag: string): any;
+
+/**
+ * BLAKE3-256 multihash of `bytes`, base32 (the form used in object paths).
+ */
+export function multihashBase32(bytes: Uint8Array): string;
+
+/**
+ * BLAKE3-256 multihash of `bytes` as lowercase hex (33 bytes: tag + digest).
+ */
+export function multihashHex(bytes: Uint8Array): string;
+
+/**
+ * DreamDB's half-open `[start, end)` → an HTTP `Range` header value.
+ *
+ * The off-by-one here is worth a vector of its own: HTTP ranges are
+ * *inclusive* of the end byte. Getting it wrong reads one byte too few from
+ * every object, which corrupts decode in ways that look like anything but an
+ * off-by-one.
+ */
+export function rangeHeader(start: bigint, end: bigint): string;
+
+/**
+ * Round-trip a spatial key through its base-2 form, per spec/0002 §6.
+ *
+ * Returns the re-encoded string, so a caller can assert byte-identity rather
+ * than merely "it parsed".
+ */
+export function spatialKeyRoundTrip(bits: string): string;
+
+/**
+ * The 16-char hex address form → TimeAnchor. Throws on a malformed input
+ * (wrong length, uppercase) rather than coercing it.
+ */
+export function timeAnchorFromHex(s: string): bigint;
+
+/**
+ * TimeAnchor → the 16-char hex used in addresses.
+ */
+export function timeAnchorHex(value: bigint): string;
+
+/**
+ * Which time bucket an anchor falls in, given a duration like `"1s"`/`"60s"`.
+ */
+export function timeBucket(t_start: bigint, duration: string): bigint;
+
+/**
+ * Package version — useful for consumers to confirm which build is loaded.
+ */
+export function version(): string;
+
+/**
+ * The all-zero spatial-key path segment used for non-spatial (fragment) object
+ * addresses. Matches `dreamdb-ts`'s `ZERO_SPATIAL_KEY` constant exactly.
+ */
+export function zeroSpatialKey(): string;

package/browser/dreamdb.js

@@ -0,0 +1,9 @@
+/* @ts-self-types="./dreamdb.d.ts" */
+import * as wasm from "./dreamdb_bg.wasm";
+import { __wbg_set_wasm } from "./dreamdb_bg.js";
+
+__wbg_set_wasm(wasm);
+wasm.__wbindgen_start();
+export {
+    S3Backend, Space, Writer, __wasm_init, addressRoundTrip, addressVariant, bytesToBase32, decodeCbor, encodeCbor, modalityParse, multihashBase32, multihashHex, rangeHeader, spatialKeyRoundTrip, timeAnchorFromHex, timeAnchorHex, timeBucket, version, zeroSpatialKey
+} from "./dreamdb_bg.js";