@dreamlake/dreamdb 0.3.1 → 0.5.0

package/README.md

@@ -0,0 +1,178 @@
+# @dreamlake/dreamdb
+
+The DreamDB SDK — read and write, in the browser and on the server, compiled to
+WebAssembly from the same Rust core the CLI and Python SDK use.
+
+There is no separate JavaScript implementation of the protocol, and that is the
+point. A hand-written port has to reproduce BLAKE3, canonical CBOR, spatial-key
+encoding, bucket headers and index layout *bit for bit*, forever. The previous
+TypeScript port did not, and its tests were all green anyway, because they were
+its own reader reading its own writer.
+
+```bash
+npm install @dreamlake/dreamdb
+```
+
+## Reading
+
+```js
+import { Space } from '@dreamlake/dreamdb'
+
+const space = await Space.fromUri('https://bucket.s3.amazonaws.com/refs/my-dataset', null)
+const hits = await space.queryVector('visual', queryVec, 24, 8)
+```
+
+Passing `null` for the backend uses direct `fetch` against the URI's base —
+enough for a public bucket. For anything else, supply a Backend.
+
+## Writing
+
+```js
+import { Writer, PresignedBackend } from '@dreamlake/dreamdb'
+
+const backend = new PresignedBackend({
+  readBase: 'https://bucket.s3.amazonaws.com',
+  mintPut: async (paths) => (await postJson('/api/dreamdb/sign', { paths })).urls,
+  commitRef: async (path, opts, bytes) =>
+    await postJson('/api/dreamdb/ref', {
+      path,
+      bytesBase64: base64(bytes),
+      ifMatch: opts.ifMatch,
+      ifNoneMatchStar: opts.ifNoneMatchStar,
+    }),
+})
+
+const w = await Writer.open('https://bucket.s3.amazonaws.com/refs/my-dataset', backend)
+await w.appendMany([
+  {
+    anchor: 1735689600000000000n,          // nanoseconds — pass a bigint
+    visual: { kind: 'embedding', algorithm: 'dreamdb.lsh-cosine', vector: vec },
+    caption: { kind: 'categorical', value: 'a red car' },
+  },
+])
+const manifest = await w.commit()
+```
+
+Credentials never enter the browser: your server mints short-lived presigned
+PUT URLs, and performs the ref's compare-and-swap itself.
+
+### Anchors are nanoseconds, and you should pass a `bigint`
+
+A `number` is accepted only while it is below `Number.MAX_SAFE_INTEGER`; above
+that the SDK throws rather than round it. This is not pedantry. The previous
+TypeScript SDK wrote anchors in **microseconds** specifically to stay under
+2^53, and every record it produced reads as 1970 in any conformant reader.
+
+## Server-side authoring
+
+`Authoring` adds dataset creation, layers, merge, compaction and history. It is
+**Node only** — the browser build exports a stub whose methods throw with an
+explanation.
+
+```js
+import { Authoring } from '@dreamlake/dreamdb'   // resolves to the Node build
+
+const a = await Authoring.create(
+  'my-dataset',
+  [
+    { name: 'visual', kind: 'embedding', dim: 768, algorithm: 'dreamdb.lsh-cosine' },
+    { name: 'caption', kind: 'scalar', valueType: 'categorical', required: false },
+  ],
+  'https://bucket.s3.amazonaws.com',
+  backend,
+)
+await a.writer().appendMany(samples)
+await a.compact(null, 1, 0)
+```
+
+The split is a size boundary with a mechanism behind it. `create` and the
+embedding-layer builders construct a `SpatialDispatcher`, which is an enum — so
+one reachable construction makes *every* index family's build code reachable
+(IVF, IMI, LSH, Vamana, AdaIVF, plus codebook training). Measured: 484 KB
+gzipped for the browser build versus 607 KB with authoring included.
+
+Embedding fields declaring `dreamdb.ivf-cosine` or `dreamdb.imi-cosine` cannot
+be created here at all: those index families need training data that does not
+exist at create time. Create with the default `dreamdb.lsh-cosine` and attach a
+trained index as a layer, or build it with the CLI.
+
+## Entry points
+
+| Import | Resolves to | Notes |
+| --- | --- | --- |
+| `@dreamlake/dreamdb` in a bundler | `--target bundler` | Needs `vite-plugin-wasm` under Vite |
+| `@dreamlake/dreamdb` in Node | `--target nodejs` | ESM and CJS both work; includes `Authoring` |
+| `@dreamlake/dreamdb/web` | `--target web` | No bundler plugin needed; `await ready()` first |
+
+```html
+<script type="module">
+  import ready, { Space } from 'https://esm.sh/@dreamlake/dreamdb/web'
+  await ready()                 // fetches and instantiates the wasm
+</script>
+```
+
+## Implementing a Backend
+
+Only `get` is required; a `get`-only backend is the read-only v1 shape and
+still works for reading. Write entry points check for `put` up front and refuse
+with a message naming what is missing.
+
+```ts
+interface Backend {
+  get(path: string, range?: { start: number; end: number })
+    : Promise<Uint8Array | { bytes: Uint8Array; etag?: string }>
+  head?(path: string): Promise<{ exists?: boolean; etag?: string; size?: number }>
+  put?(path: string, bytes: Uint8Array,
+       opts?: { ifMatch?: string; ifNoneMatchStar?: boolean })
+    : Promise<{ status: 'created' | 'exists' | 'casFailed'; etag?: string }>
+  delete?(path: string): Promise<void>
+  list?(prefix: string): Promise<string[]>
+}
+```
+
+### Two things that will bite you
+
+**Your bucket's CORS rule must include `ExposeHeaders: ["ETag"]`.**
+
+A ref advances by compare-and-swap. The SDK sends `If-Match: <etag>` when it
+knows the ref's current version and `If-None-Match: *` when it does not. Without
+`ExposeHeaders`, the browser can read the response body but not the ETag header,
+so the SDK never learns the current version, falls back to create-only, and the
+**second** commit to any ref fails while the first succeeded. That asymmetry is
+what makes it hard to recognise.
+
+```json
+{
+  "AllowedOrigins": ["https://your.app"],
+  "AllowedMethods": ["GET", "HEAD", "PUT"],
+  "AllowedHeaders": ["*"],
+  "ExposeHeaders": ["ETag", "Content-Range", "Content-Length"]
+}
+```
+
+**Any HTTP layer you put in front of the bucket must honour `Range`.**
+
+DreamDB reads exact vectors by byte offset. A server that ignores `Range` and
+returns 200 with the whole object makes every ranged read return offset 0, and
+search then returns plausible-looking results with cosine scores of 0.0000 —
+indistinguishable from a genuinely corrupt index. (`python3 -m http.server`
+does exactly this.)
+
+## Conformance
+
+`dreamdb-conformance/vectors/` holds 85 language-agnostic JSON vectors. This
+package runs the 48 that map to pure functions on its surface — canonical CBOR,
+BLAKE3 multihash, spatial keys, time anchors and buckets, address round-trips,
+modality parsing, HTTP range translation — and reports the other 37 by name and
+reason rather than omitting them silently.
+
+```bash
+node test/conformance.mjs
+```
+
+The first run of that suite found a real bug in this package's HTTP range
+translation, which is roughly the point.
+
+## Licence
+
+MIT OR Apache-2.0

package/{dreamdb.d.ts → browser/dreamdb.d.ts}

@@ -142,6 +142,67 @@ export class Space {
     readonly manifestHash: string;
 }
 
+/**
+ * A write handle onto a ref.
+ */
+export class Writer {
+    private constructor();
+    free(): void;
+    [Symbol.dispose](): void;
+    /**
+     * Append records and commit in one call.
+     *
+     * `samples` is an array of `{ anchor?: bigint|number, <field>: value }`.
+     * See `marshal::samples_from_js` for the accepted field shapes.
+     *
+     * Committing here rather than exposing a separate staged mode is
+     * deliberate for the browser surface: a staged write that is never
+     * committed leaves uploaded objects unreferenced, and a tab can close at
+     * any moment. `appendStaged` exists on the Node surface where the process
+     * lifetime is under the caller's control.
+     */
+    appendMany(samples: Array<any>): Promise<number>;
+    /**
+     * Flush staged entries and publish a new manifest.
+     *
+     * Fails with a CAS conflict if the ref moved underneath this writer. That
+     * is not retried automatically: an automatic retry would hide a logical
+     * conflict, and the caller is the only one who knows whether re-applying
+     * its records on top of the new head is correct.
+     */
+    commit(): Promise<string>;
+    /**
+     * Tombstone records by anchor. Returns the new manifest hash.
+     */
+    deleteRecords(anchors: BigUint64Array, reason?: string | null): Promise<string>;
+    /**
+     * Open a ref for writing.
+     *
+     * `uri` is the same `.../refs/<name>` form `Space.fromUri` takes; a
+     * `.../manifests/<hash>` URI is rejected rather than silently opening
+     * something unwritable, because a manifest hash names an immutable
+     * snapshot — there is nothing for a commit to advance.
+     *
+     * `backend` must implement `put` (Backend contract v2). Passing a
+     * read-only backend fails at the first write with a clear message rather
+     * than here, since the connector cannot know what the JS object omits
+     * until it calls it.
+     */
+    static open(uri: string, backend: any): Promise<Writer>;
+    /**
+     * Tag the current manifest with an immutable label (`refs/<ref>@<label>`).
+     */
+    snapshot(label: string): Promise<string>;
+    /**
+     * Current manifest hash, base32.
+     */
+    readonly manifestHash: string;
+    /**
+     * The ref this writer advances.
+     */
+    readonly refName: string;
+}
+
 /**
  * Install a panic hook that logs the panic message AND its `file:line:col`
  * to the console before the wasm trap surfaces to JS.
@@ -158,6 +219,22 @@ export class Space {
  */
 export function __wasm_init(): void;
 
+/**
+ * Parse an object path and re-format it. Byte-identity of the result is the
+ * actual assertion: a parser that silently drops a component still "parses".
+ */
+export function addressRoundTrip(path: string): string;
+
+/**
+ * The address variant name (`Genesis`, `Manifest`, `Ref`, …) for a path.
+ *
+ * Derived from the Debug representation rather than a hand-written match.
+ * `DreamDbAddress` has seventeen variants and gains one whenever the protocol
+ * does; a match arm per variant would be a second list to keep in sync, and
+ * the vectors only ever assert the name.
+ */
+export function addressVariant(path: string): string;
+
 /**
  * Encode arbitrary bytes as lowercase RFC-4648 base32 (no padding).
  *
@@ -172,6 +249,71 @@ export function bytesToBase32(bytes: Uint8Array): string;
  */
 export function decodeCbor(bytes: Uint8Array): any;
 
+/**
+ * Canonically encode a JS value as CBOR, returning the bytes.
+ *
+ * Canonical here means what spec/0002 §3.1 means: map keys sorted by their
+ * encoded bytes, shortest-form integers, no indefinite lengths. Two writers
+ * that disagree on this produce different content hashes for the same logical
+ * object, and every address derived from them diverges.
+ */
+export function encodeCbor(value: any): Uint8Array;
+
+/**
+ * Parse a modality tag into `{class, encoding, trackKind, objectKind, params,
+ * flags}`, or throw if it is invalid.
+ *
+ * `params` is a plain object of the `key=value` segments and `flags` an array
+ * of the bare ones (`bucketed`, `graph`). They are separate because the
+ * grammar treats them differently and merging them would make a flag
+ * indistinguishable from a parameter whose value happened to be empty.
+ */
+export function modalityParse(tag: string): any;
+
+/**
+ * BLAKE3-256 multihash of `bytes`, base32 (the form used in object paths).
+ */
+export function multihashBase32(bytes: Uint8Array): string;
+
+/**
+ * BLAKE3-256 multihash of `bytes` as lowercase hex (33 bytes: tag + digest).
+ */
+export function multihashHex(bytes: Uint8Array): string;
+
+/**
+ * DreamDB's half-open `[start, end)` → an HTTP `Range` header value.
+ *
+ * The off-by-one here is worth a vector of its own: HTTP ranges are
+ * *inclusive* of the end byte. Getting it wrong reads one byte too few from
+ * every object, which corrupts decode in ways that look like anything but an
+ * off-by-one.
+ */
+export function rangeHeader(start: bigint, end: bigint): string;
+
+/**
+ * Round-trip a spatial key through its base-2 form, per spec/0002 §6.
+ *
+ * Returns the re-encoded string, so a caller can assert byte-identity rather
+ * than merely "it parsed".
+ */
+export function spatialKeyRoundTrip(bits: string): string;
+
+/**
+ * The 16-char hex address form → TimeAnchor. Throws on a malformed input
+ * (wrong length, uppercase) rather than coercing it.
+ */
+export function timeAnchorFromHex(s: string): bigint;
+
+/**
+ * TimeAnchor → the 16-char hex used in addresses.
+ */
+export function timeAnchorHex(value: bigint): string;
+
+/**
+ * Which time bucket an anchor falls in, given a duration like `"1s"`/`"60s"`.
+ */
+export function timeBucket(t_start: bigint, duration: string): bigint;
+
 /**
  * Package version — useful for consumers to confirm which build is loaded.
  */

package/browser/dreamdb.js

@@ -0,0 +1,9 @@
+/* @ts-self-types="./dreamdb.d.ts" */
+import * as wasm from "./dreamdb_bg.wasm";
+import { __wbg_set_wasm } from "./dreamdb_bg.js";
+
+__wbg_set_wasm(wasm);
+wasm.__wbindgen_start();
+export {
+    S3Backend, Space, Writer, __wasm_init, addressRoundTrip, addressVariant, bytesToBase32, decodeCbor, encodeCbor, modalityParse, multihashBase32, multihashHex, rangeHeader, spatialKeyRoundTrip, timeAnchorFromHex, timeAnchorHex, timeBucket, version, zeroSpatialKey
+} from "./dreamdb_bg.js";