@hraness/oh 0.2.7 → 0.3.1

package/spec/v1/memory.md

@@ -169,6 +169,24 @@ canonical store, import a derived tuple, grant rights, record a review, or turn
 a proposed assertion into reviewed knowledge. Destination-owned application
 code must perform those steps under its own policy and compare-and-swap head.
 
+## Memory pages and retrieval
+
+A memory page is an application profile for an ordinary `edition` record. Its
+bounded title, summary, Markdown body, exact source observations, and
+host-attested provenance are authoritative record content. The optional
+canonical `.oh.md` rendering is a self-contained one-record interchange file;
+it does not introduce another database or operation format. This gives agents
+a readable, diffable artifact without making Markdown syntax the hidden source
+of graph authority.
+
+Memory pages deliberately contain no model, vector, score, index generation,
+or provider field. The local embedding backend and the hosted semantic cache
+derive disposable indexes from the same current record digests under different
+profiles. A host may expose semantic recall as an additional convenience, but
+it MUST preserve exact `remember`, Datalog `query`, `explain`, and `nominate`
+semantics when retrieval is unavailable. Search hits are rejoined to current
+authority digests and remain retrieval evidence, not accepted facts.
+
 ## Lifecycle and custody boundary
 
 Oh deliberately does not choose a tenant, session, retention deadline,

package/spec/v1/migration.md

@@ -67,6 +67,19 @@ Semantic state MAY be rebuilt with the exact profile in
 only after it rejoins a record with the same record digest. Copying a semantic
 directory does not establish migration parity.
 
+The optional hosted cache is rebuilt from authoritative records with both the
+exact [`cloudflare-embedding-profile.json`](cloudflare-embedding-profile.json)
+and [`cloudflare-embedding-renderer.json`](cloudflare-embedding-renderer.json)
+identities. Do not copy or mix vectors across local, hosted, profile, or
+renderer identities. A permanently purged hosted authority ID cannot be reused;
+allocate a new host-bound authority epoch and stage a complete generation from
+the current source records.
+
+A canonical `.oh.md` file transports one complete memory-page record and
+recomputes its record digest. It does not carry an operation chain, store head,
+realm binding, review decision, or proof of full authority migration. Use an Oh
+operation bundle or destination-owned adoption workflow for those purposes.
+
 ## Rollback
 
 Keep the verified source database read-only until the target head has matched

package/spec/v1/semantic-cloud.md

@@ -0,0 +1,96 @@
+# Hosted semantic cache V1
+
+`@hraness/oh/semantic-cloud` is a rebuildable retrieval adapter. It combines
+one fixed Cloudflare Workers AI EmbeddingGemma profile with a separate direct
+libSQL cache. It is not an Oh graph authority, a third memory lane, or a vector
+field in an Oh record.
+
+## Profile and portability
+
+The hosted profile is `oh.cloudflare.embeddinggemma.v1`. It uses model
+`@cf/google/embeddinggemma-300m`, 768 dimensions, L2 normalization, and cosine
+similarity. Queries use `task: search result | query: {query}`. Document chunks
+use `title: {title} | text: {content}`.
+
+The renderer greedily partitions content on Unicode-scalar boundaries. Every
+complete formatted input is at most 448 UTF-8 bytes. It exposes partial or
+overlarge-prefix diagnostics instead of silently dropping text. The cache
+stager accepts only complete documents, at most 64 chunks per document, 512
+documents, and 4,096 chunks in one authority generation.
+
+The profile digest and renderer digest identify the exact derived space.
+Their complete canonical payloads and digests are published in
+[`cloudflare-embedding-profile.json`](cloudflare-embedding-profile.json) and
+[`cloudflare-embedding-renderer.json`](cloudflare-embedding-renderer.json).
+Implementations MUST NOT mix vectors across either digest. A local quantized
+implementation and a hosted implementation may use the same model family but
+are not assumed to emit byte-identical or interchangeable vectors. Portable
+authority comes from the source records: deleting and rebuilding an index is
+the model-migration path.
+
+## Provider boundary
+
+`OhCloudflareEmbeddingClientV1` sends one bounded POST to the fixed Workers AI
+account/model route, rejects redirects, applies a caller-cancellable deadline,
+and accepts only the exact successful shape. It validates and normalizes every
+finite nonzero vector. Errors retain only a fixed classification and optional
+HTTP status; credentials and response bodies do not enter an error.
+
+The host owns the account, token, spend controls, privacy decision, and network
+policy. Agent input MUST NOT select an account, endpoint, token, model, batch
+limit, or profile. Source text necessarily crosses the provider boundary when
+it is embedded, even though it is not persisted in the semantic cache.
+
+## Derived libSQL cache
+
+Schema creation is an explicit deployment operation through
+`bootstrapOhLibSqlSemanticCacheV1`. `openOhLibSqlSemanticCacheV1` verifies the
+complete schema inventory and performs no DDL. Applications SHOULD use a
+short-lived schema credential and separate runtime/purge credentials.
+
+The cache stores:
+
+- normalized vectors as exact 3,072-byte little-endian float32 blobs;
+- vector digests keyed by profile, renderer, and formatted-input digest;
+- immutable record/chunk membership for staged authority generations;
+- one compare-and-swap published generation per authority; and
+- permanent authority purge tombstones.
+
+It stores no title, source content, query, page body, record JSON, account ID,
+or provider token. A formatted-input digest permits vector reuse without
+retaining its plaintext. Cache database access still deserves protection: its
+keys, record digests, generation timing, and vector geometry are metadata.
+
+Publishing never mutates a staged generation. Search pins one published head,
+scans at most its bounded 4,096 chunks in fixed pages, computes exact cosine in
+the application, keeps the best chunk per record, and rereads the head. Every
+hit is rejoined to a caller-supplied current authority record digest. A stale
+head, changed authority digest, changed record digest, concurrent publish, or
+purge returns no stale authority.
+
+`publishedHead` returns the current compare-and-swap base or `null` for an
+absent or purged authority. Its bounded projection contains only the authority,
+generation, profile, renderer, membership, generation and publication
+identities already held by the cache; it exposes no vector, record body or
+formatted input. The read verifies that the pointer still matches its immutable
+generation and fails with an integrity error on divergence. A host can read the
+base, stage a later authoritative generation, and pass the returned generation
+to `publish` without keeping a second cache pointer elsewhere.
+
+`purgeAuthority` writes a permanent tombstone, removes the head, memberships,
+and staged generations, then deletes vectors no remaining authority uses. The
+tombstone prevents the same authority ID from being staged or published again.
+Hosts SHOULD include a session epoch in the authority ID and allocate a new ID
+for a genuinely new lifetime.
+
+## Failure and lifecycle rules
+
+Hosted semantic failure MUST NOT weaken exact graph or Datalog operations. A
+consumer may return semantic-unavailable or omit the recall lane while keeping
+authoritative remember, exact query, read, and explanation operations intact.
+
+For a purgeable working authority, the lifecycle worker MUST stop new writes,
+purge the derived semantic authority, then purge the authoritative working
+store. It acknowledges completion only after both operations have converged.
+Both purges are idempotent. A failed cache purge remains retryable and MUST NOT
+be reported as complete merely because the authority purge succeeded.

package/src/canonical.ts

@@ -69,18 +69,31 @@ function encodeCanonical(value: unknown, path: string, ancestors: Set<object>):
   ancestors.add(value);
   try {
     if (Array.isArray(value)) {
-      const encoded: string[] = [];
-      for (let index = 0; index < value.length; index += 1) {
-        if (!Object.hasOwn(value, index)) {
-          throw new OhValidationError("sparse-array", `${path}[${index}]`, "must not contain holes");
-        }
-        encoded.push(encodeCanonical(value[index], `${path}[${index}]`, ancestors));
+      const lengthDescriptor = Object.getOwnPropertyDescriptor(value, "length");
+      const length = lengthDescriptor?.value;
+      if (typeof length !== "number" || !Number.isSafeInteger(length) || length < 0) {
+        throw new OhValidationError("non-json-property", path, "array has an invalid length descriptor");
       }
-      const extraKeys = Reflect.ownKeys(value).filter((key) => key !== "length"
-        && (typeof key !== "string" || !/^(?:0|[1-9][0-9]*)$/u.test(key) || Number(key) >= value.length));
-      if (extraKeys.length > 0) {
+      const ownKeys = Reflect.ownKeys(value);
+      if (!ownKeys.includes("length")
+        || ownKeys.some((key) => key !== "length" && (typeof key !== "string"
+          || !/^(?:0|[1-9][0-9]*)$/u.test(key) || Number(key) >= length))) {
         throw new OhValidationError("non-json-property", path, "array has non-index properties");
       }
+      const elements: unknown[] = [];
+      for (let index = 0; index < length; index += 1) {
+        const descriptor = Object.getOwnPropertyDescriptor(value, String(index));
+        if (descriptor === undefined) {
+          throw new OhValidationError("sparse-array", `${path}[${index}]`, "must not contain holes");
+        }
+        if (!descriptor.enumerable || descriptor.get !== undefined || descriptor.set !== undefined) {
+          throw new OhValidationError("non-json-property", `${path}[${index}]`,
+            "must be an enumerable data property");
+        }
+        elements.push(descriptor.value);
+      }
+      const encoded = elements.map((element, index) =>
+        encodeCanonical(element, `${path}[${index}]`, ancestors));
       return `[${encoded.join(",")}]`;
     }
     if (!isPlainRecord(value)) {
@@ -90,19 +103,21 @@ function encodeCanonical(value: unknown, path: string, ancestors: Set<object>):
     if (ownKeys.some((key) => typeof key !== "string")) {
       throw new OhValidationError("non-json-property", path, "object has a symbol property");
     }
+    const entries: Array<readonly [string, unknown]> = [];
     const keys = ownKeys as string[];
     for (const key of keys) {
       const descriptor = Object.getOwnPropertyDescriptor(value, key);
       if (descriptor === undefined || !descriptor.enumerable || descriptor.get !== undefined || descriptor.set !== undefined) {
         throw new OhValidationError("non-json-property", `${path}.${key}`, "must be an enumerable data property");
       }
+      entries.push([key, descriptor.value]);
     }
-    keys.sort();
-    const entries = keys.map((key) => {
+    entries.sort(([left], [right]) => left < right ? -1 : left > right ? 1 : 0);
+    const encodedEntries = entries.map(([key, entryValue]) => {
       assertUnicodeScalarString(key, `${path}.<key>`);
-      return `${JSON.stringify(key)}:${encodeCanonical(value[key], `${path}.${key}`, ancestors)}`;
+      return `${JSON.stringify(key)}:${encodeCanonical(entryValue, `${path}.${key}`, ancestors)}`;
     });
-    return `{${entries.join(",")}}`;
+    return `{${encodedEntries.join(",")}}`;
   } finally {
     ancestors.delete(value);
   }

package/src/cli.ts

@@ -9,7 +9,7 @@ import { Oh } from "./sdk";
 import { OH_SQLITE_SCHEMA_VERSION } from "./sqlite/migrations";
 import { createOhSyncBundleV1, parseOhSyncBundleV1 } from "./sync";
 
-export const OH_PACKAGE_VERSION = "0.2.7" as const;
+export const OH_PACKAGE_VERSION = "0.3.1" as const;
 
 type ParsedArguments = { options: Map<string, string[]>; positionals: string[] };
 type ValidatedInvocation = Readonly<{

package/src/cloudflare-embedding.test.ts

@@ -0,0 +1,306 @@
+import { describe, expect, test } from "bun:test";
+
+import { canonicalSha256, sha256Hex, utf8ByteLength } from "./canonical";
+import {
+  OH_CLOUDFLARE_EMBEDDING_LIMITS_V1,
+  OH_CLOUDFLARE_EMBEDDING_PROFILE_V1,
+  OH_SEMANTIC_RENDERER_V1,
+  OhCloudflareEmbeddingClientV1,
+  OhCloudflareEmbeddingError,
+  renderOhCloudflareEmbeddingDocumentV1,
+  renderOhCloudflareEmbeddingQueryV1,
+  type OhEmbeddingFetchV1,
+  type OhRenderedEmbeddingInputV1,
+} from "./cloudflare-embedding";
+
+const accountId = "0123456789abcdef0123456789abcdef";
+const apiToken = "cloudflare-test-token-never-a-real-secret";
+
+function unitVector(index = 0): number[] {
+  return Array.from(
+    { length: OH_CLOUDFLARE_EMBEDDING_PROFILE_V1.dimensions },
+    (_, ordinal) => ordinal === index ? 1 : 0,
+  );
+}
+
+function success(vectors: readonly (readonly number[])[]): Response {
+  return Response.json({
+    result: {
+      data: vectors,
+      shape: [vectors.length, OH_CLOUDFLARE_EMBEDDING_PROFILE_V1.dimensions],
+    },
+    success: true,
+  });
+}
+
+describe("Cloudflare EmbeddingGemma profile", () => {
+  test("pins a provider-specific, rebuildable embedding and renderer identity", () => {
+    expect(OH_CLOUDFLARE_EMBEDDING_PROFILE_V1).toMatchObject({
+      dimensions: 768,
+      distance: "cosine",
+      inputUtf8Bytes: 448,
+      model: "@cf/google/embeddinggemma-300m",
+      normalization: "l2",
+      profileId: "oh.cloudflare.embeddinggemma.v1",
+      provider: "cloudflare.workers-ai",
+      v: 1,
+    });
+    const { profileSha256, ...profile } = OH_CLOUDFLARE_EMBEDDING_PROFILE_V1;
+    const { rendererSha256, ...renderer } = OH_SEMANTIC_RENDERER_V1;
+    expect(profileSha256).toBe(canonicalSha256(profile));
+    expect(rendererSha256).toBe(canonicalSha256(renderer));
+    expect(profileSha256).not.toBe(rendererSha256);
+  });
+
+  test("renders bounded query and complete Unicode-scalar document inputs", () => {
+    const query = renderOhCloudflareEmbeddingQueryV1("where did the session stop?");
+    expect(query).toEqual({
+      input: "task: search result | query: where did the session stop?",
+      inputSha256: sha256Hex(query.input),
+      kind: "query",
+      utf8Bytes: utf8ByteLength(query.input),
+      v: 1,
+    });
+
+    const source = `${"a".repeat(420)}🧠${"b".repeat(420)}`;
+    const rendered = renderOhCloudflareEmbeddingDocumentV1({
+      content: source,
+      title: "session",
+    });
+    expect(rendered.status).toBe("complete");
+    expect(rendered.diagnostic).toBeNull();
+    expect(rendered.chunks.length).toBeGreaterThan(1);
+    expect(rendered.chunks.map(({ content }) => content).join("")).toBe(source);
+    for (const [ordinal, chunk] of rendered.chunks.entries()) {
+      expect(chunk.ordinal).toBe(ordinal);
+      expect(chunk.input.kind).toBe("document");
+      expect(chunk.input.utf8Bytes).toBeLessThanOrEqual(448);
+      expect(chunk.input.inputSha256).toBe(sha256Hex(chunk.input.input));
+      expect(/[\ud800-\udbff]$/u.test(chunk.content)).toBeFalse();
+      expect(/^[\udc00-\udfff]/u.test(chunk.content)).toBeFalse();
+    }
+  });
+
+  test("makes truncation and an overlarge prefix explicit", () => {
+    const partial = renderOhCloudflareEmbeddingDocumentV1({
+      content: "x".repeat(2_000),
+      maximumChunks: 1,
+      title: "bounded",
+    });
+    expect(partial.status).toBe("partial");
+    expect(partial.diagnostic).toMatchObject({
+      code: "partial",
+      maximumChunks: 1,
+    });
+    expect(partial.diagnostic?.omittedUtf8Bytes).toBeGreaterThan(0);
+
+    const oversize = renderOhCloudflareEmbeddingDocumentV1({
+      content: "content",
+      title: "t".repeat(448),
+    });
+    expect(oversize).toMatchObject({
+      chunks: [],
+      diagnostic: { code: "oversize-prefix" },
+      status: "oversize",
+    });
+
+    const fixedPrefixBytes = utf8ByteLength("title:  | text: ");
+    for (const residualBytes of [1, 2, 3]) {
+      const scalarOversize = renderOhCloudflareEmbeddingDocumentV1({
+        content: "🧠",
+        title: "t".repeat(448 - fixedPrefixBytes - residualBytes),
+      });
+      expect(scalarOversize).toMatchObject({
+        chunks: [],
+        diagnostic: { code: "oversize-prefix", omittedUtf8Bytes: 4 },
+        status: "oversize",
+      });
+    }
+
+    const scalarPartial = renderOhCloudflareEmbeddingDocumentV1({
+      content: "a🧠",
+      title: "t".repeat(448 - fixedPrefixBytes - 1),
+    });
+    expect(scalarPartial).toMatchObject({
+      chunks: [{ content: "a", ordinal: 0 }],
+      diagnostic: { code: "partial", omittedUtf8Bytes: 4 },
+      status: "partial",
+    });
+  });
+
+  test("rejects noncanonical text and formatted inputs over the byte boundary", () => {
+    expect(() => renderOhCloudflareEmbeddingQueryV1("e\u0301"))
+      .toThrow(OhCloudflareEmbeddingError);
+    expect(() => renderOhCloudflareEmbeddingQueryV1("q".repeat(448)))
+      .toThrow("formatted embedding input exceeds");
+    expect(() => renderOhCloudflareEmbeddingDocumentV1({
+      content: "text",
+      maximumChunks: OH_CLOUDFLARE_EMBEDDING_LIMITS_V1.renderedChunks + 1,
+      title: "title",
+    })).toThrow(OhCloudflareEmbeddingError);
+  });
+});
+
+describe("Cloudflare Workers AI embedding client", () => {
+  test("sends one fixed no-redirect request and returns normalized vectors", async () => {
+    const requests: Array<Readonly<{ body: unknown; headers: Headers; url: string }>> = [];
+    const fetch: OhEmbeddingFetchV1 = async (input, init) => {
+      requests.push({
+        body: JSON.parse(String(init?.body)) as unknown,
+        headers: new Headers(init?.headers),
+        url: String(input),
+      });
+      expect(init?.method).toBe("POST");
+      expect(init?.redirect).toBe("error");
+      expect(init?.signal).toBeInstanceOf(AbortSignal);
+      return success([unitVector(3), unitVector(4).map((value) => value * 7)]);
+    };
+    const client = new OhCloudflareEmbeddingClientV1({ accountId, apiToken, fetch });
+    const inputs = [
+      renderOhCloudflareEmbeddingQueryV1("alpha"),
+      renderOhCloudflareEmbeddingQueryV1("beta"),
+    ] as const;
+    const vectors = await client.embed(inputs);
+    expect(vectors).toHaveLength(2);
+    expect(vectors[0]?.[3]).toBe(1);
+    expect(vectors[1]?.[4]).toBe(1);
+    expect(requests).toEqual([{
+      body: { text: inputs.map(({ input }) => input) },
+      headers: new Headers({
+        authorization: `Bearer ${apiToken}`,
+        "content-type": "application/json",
+      }),
+      url: `https://api.cloudflare.com/client/v4/accounts/${accountId}/ai/run/@cf/google/embeddinggemma-300m`,
+    }]);
+  });
+
+  test("rejects forged rendered inputs before network access", async () => {
+    let calls = 0;
+    const client = new OhCloudflareEmbeddingClientV1({
+      accountId,
+      apiToken,
+      fetch: async () => { calls += 1; return success([unitVector()]); },
+    });
+    const input = renderOhCloudflareEmbeddingQueryV1("safe");
+    await expect(client.embed([{ ...input, inputSha256: "0".repeat(64) as typeof input.inputSha256 }]))
+      .rejects.toMatchObject({ code: "invalid-input" });
+    const forge = (text: string): OhRenderedEmbeddingInputV1 => Object.freeze({
+      input: text,
+      inputSha256: sha256Hex(text),
+      kind: "query",
+      utf8Bytes: utf8ByteLength(text),
+      v: 1,
+    });
+    for (const forged of [
+      Object.freeze({ ...input }),
+      forge("task: search result | query: e\u0301"),
+      forge("task: search result | query: \ud800"),
+      forge("not a rendered query"),
+    ]) {
+      await expect(client.embed([forged])).rejects.toMatchObject({ code: "invalid-input" });
+    }
+    expect(calls).toBe(0);
+  });
+
+  test("normalizes extreme finite provider vectors without collapsing them to zero", async () => {
+    const extreme = unitVector();
+    extreme[0] = Number.MAX_VALUE;
+    const client = new OhCloudflareEmbeddingClientV1({
+      accountId,
+      apiToken,
+      fetch: async () => success([extreme]),
+    });
+    const [vector] = await client.embed([renderOhCloudflareEmbeddingQueryV1("query")]);
+    expect(vector?.[0]).toBe(1);
+    expect(vector?.reduce((sum, component) => sum + component * component, 0)).toBeCloseTo(1, 12);
+  });
+
+  test("bounds and validates every successful provider response", async () => {
+    const input = renderOhCloudflareEmbeddingQueryV1("query");
+    for (const response of [
+      Response.json({ success: false }),
+      Response.json({ result: { data: [unitVector()], shape: [1, 1] }, success: true }),
+      Response.json({ result: { data: [[...unitVector().slice(1)]], shape: [1, 768] }, success: true }),
+      Response.json({ result: { data: [[Number.NaN, ...unitVector().slice(1)]], shape: [1, 768] }, success: true }),
+      new Response("x".repeat(1_024), { status: 200 }),
+    ]) {
+      const client = new OhCloudflareEmbeddingClientV1({
+        accountId,
+        apiToken,
+        fetch: async () => response,
+        maximumResponseBytes: 512,
+      });
+      await expect(client.embed([input])).rejects.toMatchObject({ code: "invalid-response" });
+    }
+  });
+
+  test("surfaces only a bounded status classification on provider failures", async () => {
+    const client = new OhCloudflareEmbeddingClientV1({
+      accountId,
+      apiToken,
+      fetch: async () => new Response(`never echo ${apiToken}`, { status: 429 }),
+    });
+    const error = await client.embed([renderOhCloudflareEmbeddingQueryV1("query")])
+      .then(() => null, (caught: unknown) => caught);
+    expect(error).toMatchObject({ code: "provider-unavailable", status: 429 });
+    expect(String(error)).not.toContain(apiToken);
+    expect(String(error)).not.toContain(accountId);
+  });
+
+  test("classifies transport cancellation without retaining its cause", async () => {
+    const fetch: OhEmbeddingFetchV1 = async (_input, init) => await new Promise<Response>((_resolve, reject) => {
+      init?.signal?.addEventListener("abort", () => reject(new Error(`private ${apiToken}`)), { once: true });
+    });
+    const client = new OhCloudflareEmbeddingClientV1({
+      accountId,
+      apiToken,
+      deadlineMs: 1,
+      fetch,
+    });
+    const error = await client.embed([renderOhCloudflareEmbeddingQueryV1("query")])
+      .then(() => null, (caught: unknown) => caught);
+    expect(error).toMatchObject({ code: "aborted", status: null });
+    expect(String(error)).not.toContain(apiToken);
+  });
+
+  test("classifies cancellation while reading a response body as aborted", async () => {
+    const external = new AbortController();
+    const fetch: OhEmbeddingFetchV1 = async (_input, init) => new Response(
+      new ReadableStream<Uint8Array>({
+        start(controller) {
+          init?.signal?.addEventListener(
+            "abort",
+            () => controller.error(new Error(`private ${apiToken}`)),
+            { once: true },
+          );
+          setTimeout(() => external.abort(), 0);
+        },
+      }),
+      { status: 200 },
+    );
+    const client = new OhCloudflareEmbeddingClientV1({
+      accountId,
+      apiToken,
+      deadlineMs: 1_000,
+      fetch,
+    });
+    const error = await client.embed(
+      [renderOhCloudflareEmbeddingQueryV1("query")],
+      { signal: external.signal },
+    ).then(() => null, (caught: unknown) => caught);
+    expect(error).toMatchObject({ code: "aborted", status: null });
+    expect(String(error)).not.toContain(apiToken);
+  });
+
+  test("validates credential and request-budget configuration without network access", () => {
+    expect(() => new OhCloudflareEmbeddingClientV1({ accountId: "wrong", apiToken }))
+      .toThrow("credentials are malformed");
+    expect(() => new OhCloudflareEmbeddingClientV1({ accountId, apiToken: "short" }))
+      .toThrow("credentials are malformed");
+    expect(() => new OhCloudflareEmbeddingClientV1({
+      accountId,
+      apiToken,
+      maximumBatchInputs: 33,
+    })).toThrow(RangeError);
+  });
+});