latticesql 4.0.1 → 4.2.0

package/docs/api-reference.md

@@ -2,6 +2,13 @@
 
 Complete reference for all public classes, methods, and types exported by `latticesql`.
 
+> **v4.1 retrieval, query & data primitives** (eval/doctor/benchmark, chunked +
+> indexed vector, hybrid + ranking + rerank, graph, bounded reads / projection /
+> jsonPath / aggregate / keyset pagination / distinctOn / include, provenance +
+> trust, retry + resumable migrations, computed columns + rollups, keyless cloud
+> file presigning) are documented with examples in **[retrieval.md](retrieval.md)**.
+> All are additive and opt-in.
+
 ---
 
 ## Table of Contents
@@ -494,13 +501,23 @@ const results = await db.search('docs', 'deploy to production', { topK: 5, minSc
 
 **`SearchOptions`**:
 
-| Field      | Type     | Default | Description                         |
-| ---------- | -------- | ------- | ----------------------------------- |
-| `topK`     | `number` | `10`    | Max results to return               |
-| `minScore` | `number` | `0`     | Minimum cosine similarity threshold |
+| Field      | Type     | Default | Description                                                                       |
+| ---------- | -------- | ------- | --------------------------------------------------------------------------------- |
+| `topK`     | `number` | `10`    | Max results to return (clamped to `[1, 1000]` before the candidate fan-out; v4.2) |
+| `minScore` | `number` | `0`     | Minimum cosine similarity threshold                                               |
 
 **`SearchResult`**: `{ row: Row, score: number }`
 
+> **v4.2 — bounded retrieval.** `topK` is clamped (`clampTopK`,
+> `SEARCH_TOPK_MAX = 1000`) before the indexed arm over-fetches `topK * N`
+> candidates, so a single large `topK` can't turn one query into a whole-table
+> read. When a table has **no** native vector index, the in-process cosine scan
+> can be capped per-table via `EmbeddingsConfig.maxScanChunks`: if the scan would
+> read more than that many stored chunk vectors it throws
+> `EmbeddingScanTooLargeError` rather than load them all into memory — off by
+> default (unbounded scan, historical behavior), and never silently truncated. See
+> [retrieval.md](retrieval.md).
+
 ---
 
 ### Sync Methods
@@ -1396,6 +1413,52 @@ function cleanupEntityContexts(
 ): CleanupResult;
 ```
 
+### Structured import (4.2)
+
+Turn a JSON object or an Excel `.xlsx` workbook into a Lattice schema and
+materialize it (entities / dimensions / junctions), with point-in-time snapshots
+and re-import recognition. All exported from `latticesql`. In `lattice gui` these
+run automatically when you **drop a structured file into the assistant rail**;
+the same functions are available as a GUI-independent library API. See
+[importing.md](importing.md).
+
+```ts
+import {
+  inferSchema, // (data, opts?) => ProposedSchema  — entities/dimensions/junctions
+  inferFieldType, // (values) => InferredType
+  normalizeName, // (key) => string                  — source key → table/column name
+  sourceRecords, // (data, entity) => Record<string, unknown>[]
+  excelToRecords, // (absPath) => Promise<Record<string, unknown[]>>  — sheets → records
+  dedupeAndDetectViews, // (data, plan) => { ..., views: DetectedView[] }   — read-only per-slice views
+  detectAsOf, // (fileName) => string | null      — ISO YYYY-MM-DD
+  detectAsOfCandidates, // (inputs: AsOfInputs) => AsOfCandidate[]
+  detectAsOfColumns, // (data, plan) => AsOfColumnCandidate[]  — per-row date columns
+  parseCellDate, // (value) => string | null         — ISO YYYY-MM-DD
+  matchSchemaToExisting, // (existing, plan) => SchemaMatch  — fingerprint re-imports
+  renameEntities, // (plan, rename) => ProposedSchema
+  materializeImport, // (ctx, data, plan, views?, opts?) => Promise<MaterializeResult>
+  EmbeddingScanTooLargeError,
+} from 'latticesql';
+```
+
+`materializeImport(ctx, data, plan, views?, opts?)`:
+
+- `ctx`: `{ db: Lattice, configPath?: string | null }` — when `configPath` is
+  set, the inferred schema is persisted to the workspace config (canonical).
+- `opts.mode`: `'schema' | 'contents' | 'both'` (default `'both'`).
+- `opts.asOf`: file-level ISO date — stamps every row's `as_of` and folds it into
+  the row identity, so re-importing at a new date appends a snapshot.
+- `opts.asOfColumn`: a per-row date column name — dates each row individually.
+- `opts.onProgress`: streams `ImportProgress` steps for a live pipeline view.
+- Returns `MaterializeResult`:
+  `{ mode, asOf, asOfColumn, tablesCreated, rowsByTable, links, views }`.
+
+Types: `ProposedSchema`, `InferredEntity`, `InferredColumn`, `InferredDimension`,
+`InferredLinkage`, `InferredType`, `DetectedView`, `AsOfCandidate`, `AsOfInputs`,
+`AsOfColumnCandidate`, `SchemaMatch`, `EntityMatch`, `ExistingTable`,
+`MaterializeCtx`, `MaterializeResult`, `MaterializeOptions`, `ImportMode`,
+`ImportProgress`.
+
 ### Full-text search (1.16)
 
 ```ts

package/docs/architecture.md

@@ -187,6 +187,21 @@ Two modules:
 
 Standalone entry point compiled to `dist/cli.js` with a `#!/usr/bin/env node` shebang. Uses no external CLI framework — just manual `process.argv` parsing. Calls `generateAll()` and logs results.
 
+### Structured import (`src/import/`) _(v4.2)_
+
+Turns a structured source — a JSON object or an Excel `.xlsx` workbook — into a
+Lattice schema and materializes it. It is a self-contained module with no
+dependency on the GUI or any dashboard:
+
+- `infer.ts` — `inferSchema` / `inferFieldType` / `normalizeName` / `sourceRecords`: source → proposed entities, dimensions, junctions.
+- `excel.ts` — `excelToRecords`: sheets → records (header + data-region detection).
+- `dedupe-views.ts` — `dedupeAndDetectViews`: per-slice tabs that mirror a master become read-only views, not duplicate tables.
+- `asof.ts` / `asof-columns.ts` — `detectAsOf*` / `parseCellDate`: detect a file-level or per-row as-of date for point-in-time snapshots.
+- `match.ts` — `matchSchemaToExisting` / `renameEntities`: fingerprint a re-upload against existing tables so it lands as a new snapshot, not a duplicate set.
+- `materialize.ts` — `materializeImport`: create tables (idempotent), insert rows + links, persist the schema to config, build the detected views.
+
+In `lattice gui` the import is reachable only by dropping a structured file into the assistant rail; the confirmed proposal is applied via `POST /api/import/apply`. The functions are also exported from `latticesql` for library use.
+
 ---
 
 ## Data flow
@@ -314,6 +329,15 @@ src/
 │   └── loop.ts           # SyncLoop (+ cleanup integration, v0.5)
 ├── writeback/
 │   └── pipeline.ts       # WritebackPipeline
+├── import/               # v4.2 — structured-source import
+│   ├── infer.ts          # inferSchema / inferFieldType / normalizeName / sourceRecords
+│   ├── excel.ts          # excelToRecords
+│   ├── dedupe-views.ts   # dedupeAndDetectViews
+│   ├── asof.ts           # detectAsOf* / parseCellDate
+│   ├── asof-columns.ts   # detectAsOfColumns
+│   ├── match.ts          # matchSchemaToExisting / renameEntities
+│   ├── materialize.ts    # materializeImport
+│   └── types.ts          # ProposedSchema, InferredEntity, DetectedView, …
 └── security/
     └── sanitize.ts       # Sanitizer
 

package/docs/assistant.md

@@ -166,6 +166,29 @@ client): `organizeSource`, `describeImage`, `crawlUrl`, `enrichKnowledge`, and t
 A transient **"Analyzing…"** row shows while ingest runs; the add/enrich/link
 events stream into the feed as the server materializes them.
 
+### Structured-source import (drop a JSON / `.xlsx`) (4.2)
+
+The Context Constructor above turns _unstructured_ sources (documents, images,
+web pages) into a summarized, linked `files` row. **Dropping a structured source
+— a JSON object or an Excel `.xlsx` workbook — takes a different path:** Lattice
+infers a schema from it (entities, dimensions, junctions) and materializes it into
+real tables. Excel sheets become records (header + data-region detection);
+per-slice tabs that mirror a master become read-only **views** (no duplicated
+rows). An **as-of date** is detected (file contents → name → Excel preamble → a
+Claude fallback, or per-row from a date column), so re-importing a newer period
+keeps a **dated snapshot** beside the prior one; a re-upload is fingerprinted and
+matched to the tables already in the workspace, so it lands as a new snapshot
+rather than duplicate tables.
+
+A **recognized dataset with a confident date imports silently** as a dated
+snapshot (reported in the activity feed); a brand-new dataset, or a recognized one
+with no confident date, surfaces an **inline confirm card** that proposes the
+schema, the as-of date (and any per-row date column), and the mode before anything
+is written — applied via `POST /api/import/apply`. The same inference +
+materialization functions (`inferSchema`, `materializeImport`, `detectAsOf*`,
+`excelToRecords`, `dedupeAndDetectViews`, …) are exported from `latticesql` for
+library use. See [importing.md](importing.md) for the full walkthrough.
+
 ## Artifacts
 
 Ask the assistant to "write a doc / note / summary / write-up" and it calls the

package/docs/examples/dashboard.html

@@ -0,0 +1,284 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>My Dashboard</title>
+    <style>
+      /* A plain, mobile-friendly starting point. Restyle freely — only the
+         fetch() calls below matter. One system font, differentiated by
+         weight/size only. */
+      :root {
+        font-family:
+          ui-sans-serif,
+          system-ui,
+          -apple-system,
+          Segoe UI,
+          Roboto,
+          sans-serif;
+        color-scheme: light dark;
+      }
+      * {
+        box-sizing: border-box;
+      }
+      body {
+        margin: 0;
+        padding: 1.5rem 1rem;
+        max-width: 760px;
+        margin-inline: auto;
+        line-height: 1.5;
+        overflow-x: hidden;
+      }
+      h1 {
+        font-size: 1.4rem;
+        margin: 0 0 0.25rem;
+      }
+      p.sub {
+        margin: 0 0 1.5rem;
+        opacity: 0.7;
+      }
+      section {
+        border: 1px solid color-mix(in srgb, currentColor 18%, transparent);
+        border-radius: 12px;
+        padding: 1rem;
+        margin-bottom: 1.25rem;
+      }
+      label {
+        font-weight: 600;
+        display: block;
+        margin-bottom: 0.5rem;
+      }
+      textarea,
+      input[type='text'] {
+        width: 100%;
+        padding: 0.6rem;
+        border-radius: 8px;
+        border: 1px solid color-mix(in srgb, currentColor 25%, transparent);
+        background: transparent;
+        color: inherit;
+        font: inherit;
+      }
+      textarea {
+        min-height: 70px;
+        resize: vertical;
+      }
+      .row {
+        display: flex;
+        gap: 0.5rem;
+        flex-wrap: wrap;
+        align-items: center;
+        margin-top: 0.6rem;
+      }
+      button {
+        padding: 0.55rem 1rem;
+        border-radius: 8px;
+        border: 0;
+        background: #2d6cdf;
+        color: #fff;
+        font: inherit;
+        font-weight: 600;
+        cursor: pointer;
+      }
+      button:disabled {
+        opacity: 0.5;
+        cursor: default;
+      }
+      #drop {
+        border: 2px dashed color-mix(in srgb, currentColor 30%, transparent);
+        border-radius: 10px;
+        padding: 1.25rem;
+        text-align: center;
+        opacity: 0.85;
+      }
+      #drop.over {
+        border-color: #2d6cdf;
+        opacity: 1;
+      }
+      ul {
+        list-style: none;
+        padding: 0;
+        margin: 0;
+      }
+      li {
+        padding: 0.7rem 0;
+        border-top: 1px solid color-mix(in srgb, currentColor 12%, transparent);
+      }
+      li:first-child {
+        border-top: 0;
+      }
+      .name {
+        font-weight: 600;
+        word-break: break-word;
+      }
+      .meta {
+        font-size: 0.85rem;
+        opacity: 0.7;
+      }
+      .tag {
+        display: inline-block;
+        font-size: 0.75rem;
+        padding: 0.1rem 0.5rem;
+        border-radius: 999px;
+        background: color-mix(in srgb, currentColor 12%, transparent);
+        margin: 0.15rem 0.15rem 0 0;
+      }
+      #status {
+        min-height: 1.2rem;
+        font-size: 0.85rem;
+        opacity: 0.8;
+      }
+    </style>
+  </head>
+  <body>
+    <h1>My Dashboard</h1>
+    <p class="sub">
+      Upload a file or jot a note — Lattice reads it and files it against your data.
+    </p>
+
+    <section>
+      <label for="file">Upload files</label>
+      <div id="drop">
+        Drag files here, or
+        <button type="button" id="pick">choose files</button>
+      </div>
+      <input id="file" type="file" multiple hidden />
+    </section>
+
+    <section>
+      <label for="note">Add a note</label>
+      <textarea id="note" placeholder="Type a note, or paste a link to capture it…"></textarea>
+      <div class="row">
+        <button type="button" id="addNote">Add note</button>
+      </div>
+    </section>
+
+    <section>
+      <label>Recently captured</label>
+      <div id="status"></div>
+      <ul id="list"></ul>
+    </section>
+
+    <script>
+      // ---- Lattice client -------------------------------------------------
+      // These three calls are the whole integration. The dashboard is served by
+      // Lattice on the same origin, so plain relative fetch() works — no API key
+      // in the page, no CORS. Copy these into your own page to wire your own UI.
+
+      // Upload one file. Returns { id, extraction_status, suggestedLinks, ... }.
+      async function latticeUpload(file) {
+        const res = await fetch('/api/ingest/upload', {
+          method: 'POST',
+          headers: {
+            'content-type': file.type || 'application/octet-stream',
+            'x-filename': encodeURIComponent(file.name || 'file'),
+          },
+          body: file,
+        });
+        if (!res.ok) throw new Error('Upload failed: HTTP ' + res.status);
+        return res.json();
+      }
+
+      // Capture a note (or a pasted URL). Returns { id, extraction_status, suggestedLinks }.
+      async function latticeAddNote(text, title) {
+        const res = await fetch('/api/ingest/text', {
+          method: 'POST',
+          headers: { 'content-type': 'application/json' },
+          body: JSON.stringify(title ? { text, title } : { text }),
+        });
+        if (!res.ok) throw new Error('Add note failed: HTTP ' + res.status);
+        return res.json();
+      }
+
+      // List captured items (newest first). Returns an array of file rows.
+      async function latticeListFiles(limit = 25) {
+        const res = await fetch('/api/tables/files/rows?limit=' + limit);
+        if (!res.ok) throw new Error('List failed: HTTP ' + res.status);
+        const data = await res.json();
+        return Array.isArray(data.rows) ? data.rows : [];
+      }
+
+      // ---- Wiring (replace with your own UI) ------------------------------
+      const statusEl = document.getElementById('status');
+      const listEl = document.getElementById('list');
+      const fileInput = document.getElementById('file');
+      const drop = document.getElementById('drop');
+
+      function setStatus(msg) {
+        statusEl.textContent = msg || '';
+      }
+
+      function renderList(rows) {
+        listEl.innerHTML = '';
+        for (const r of rows) {
+          const li = document.createElement('li');
+          const name = document.createElement('div');
+          name.className = 'name';
+          name.textContent = r.original_name || r.name || '(untitled)';
+          const meta = document.createElement('div');
+          meta.className = 'meta';
+          meta.textContent =
+            (r.description || '').slice(0, 200) +
+            (r.extraction_status ? '  ·  ' + r.extraction_status : '');
+          li.append(name, meta);
+          listEl.append(li);
+        }
+        if (rows.length === 0) listEl.innerHTML = '<li class="meta">Nothing captured yet.</li>';
+      }
+
+      async function refresh() {
+        try {
+          renderList(await latticeListFiles());
+        } catch (e) {
+          setStatus(e.message);
+        }
+      }
+
+      async function handleFiles(files) {
+        for (const file of files) {
+          setStatus('Uploading ' + file.name + '…');
+          try {
+            const out = await latticeUpload(file);
+            const n = (out.suggestedLinks || []).length;
+            setStatus('Captured ' + file.name + (n ? ' · linked to ' + n + ' record(s)' : ''));
+          } catch (e) {
+            setStatus(e.message);
+          }
+        }
+        await refresh();
+      }
+
+      document.getElementById('pick').addEventListener('click', () => fileInput.click());
+      fileInput.addEventListener('change', () => {
+        if (fileInput.files.length) handleFiles(fileInput.files);
+        fileInput.value = '';
+      });
+      drop.addEventListener('dragover', (e) => {
+        e.preventDefault();
+        drop.classList.add('over');
+      });
+      drop.addEventListener('dragleave', () => drop.classList.remove('over'));
+      drop.addEventListener('drop', (e) => {
+        e.preventDefault();
+        drop.classList.remove('over');
+        if (e.dataTransfer.files.length) handleFiles(e.dataTransfer.files);
+      });
+
+      document.getElementById('addNote').addEventListener('click', async () => {
+        const ta = document.getElementById('note');
+        const text = ta.value.trim();
+        if (!text) return;
+        setStatus('Saving note…');
+        try {
+          await latticeAddNote(text);
+          ta.value = '';
+          setStatus('Note captured.');
+        } catch (e) {
+          setStatus(e.message);
+        }
+        await refresh();
+      });
+
+      refresh();
+    </script>
+  </body>
+</html>

package/docs/importing.md

@@ -0,0 +1,118 @@
+# Structured-source import (v4.2)
+
+latticesql 4.2 can turn a **structured file** — a JSON object or an Excel
+`.xlsx` workbook — into a Lattice schema (entities, dimensions, junctions) and
+materialize it into a workspace. Everything here is **additive and opt-in**:
+absent a file drop, behavior is byte-identical to 4.1.
+
+The feature is reachable **only by dropping a file into the assistant rail** in
+`lattice gui`. There is no CLI verb and no separate endpoint to call by hand —
+the upload pipeline builds a proposal, and a confirmed proposal is applied via
+`POST /api/import/apply`. The same inference and materialization functions are
+also exported from `latticesql` for library use (see [Library API](#library-api)).
+
+## What it does
+
+When you drop a recognized JSON / `.xlsx` source into the chat:
+
+1. **Infer a schema.** `inferSchema` reads the source and proposes **entities**
+   (record collections that become tables), **dimensions** (small repeated value
+   sets that become a shared taxonomy / dictionary), and **junctions** (the
+   many-to-many links between them). Field types are inferred per column
+   (`inferFieldType`), and source keys are normalized to table/column names
+   (`normalizeName`).
+2. **Read Excel natively.** `excelToRecords` turns each sheet into records by
+   detecting the header row and the data region. A per-slice tab that is just a
+   filtered view of a master sheet is recognized as a **read-only view** (no
+   duplicated rows) rather than a second table — see `dedupeAndDetectViews`.
+3. **Detect an as-of date for point-in-time snapshots.** `detectAsOf*` looks at
+   the file's contents, then its name, then an Excel preamble, then a Claude
+   fallback — or a per-row date **column** (`detectAsOfColumns`, `parseCellDate`).
+   When a date is found, every materialized row is stamped `as_of` and the row
+   identity folds it in, so **re-importing a newer period APPENDS a dated
+   snapshot beside the prior one** instead of overwriting it. Dimensions (the
+   shared taxonomy) are not dated.
+4. **Recognize a re-import.** `matchSchemaToExisting` fingerprints the inferred
+   schema and matches it against the tables already in the workspace, so a
+   re-upload lands as a **new snapshot of the existing tables**, not a duplicate
+   set. `renameEntities` applies any entity → table-name overrides.
+5. **Materialize.** `materializeImport` creates the tables (idempotently),
+   inserts the rows + links, persists the schema to the workspace config, and
+   builds the detected read-only views.
+
+## Silent import vs. the inline confirm card
+
+The chat drop chooses one of three paths automatically:
+
+- **Recognized dataset + a confident date → silent import.** The file matches
+  tables already in the workspace and a date was confidently detected, so it is
+  imported straight away as a dated snapshot and reported in the activity feed.
+- **Recognized dataset but no / ambiguous date → confirm card.** Importing
+  undated would overwrite the prior snapshot, so an **inline confirm card**
+  proposes the date (and any per-row date column) before anything is written.
+- **Brand-new structured data → confirm card.** Tables are never created
+  silently from a chat drop. The card proposes the full schema, the date, and
+  the mode for you to review and apply.
+
+Either way, nothing is written until a confident match resolves silently or you
+confirm the card; the confirmed proposal is applied via `POST /api/import/apply`,
+which streams the materialization progress back as NDJSON.
+
+## File-size cap
+
+A source file is capped at **50 MB**, and the cap is enforced **on both paths**:
+the streaming upload rejects an oversized file, and the apply route re-`statSync`s
+the retained bytes before reading them — so an oversized or swapped-on-disk
+source (including one reached via a `local_ref` that never went through the
+upload) cannot be streamed whole into memory.
+
+## Library API
+
+The inference + materialization functions are exported from `latticesql` and run
+GUI-independently:
+
+```ts
+import {
+  inferSchema,
+  inferFieldType,
+  normalizeName,
+  sourceRecords,
+  excelToRecords,
+  dedupeAndDetectViews,
+  detectAsOf,
+  detectAsOfCandidates,
+  detectAsOfColumns,
+  parseCellDate,
+  matchSchemaToExisting,
+  renameEntities,
+  materializeImport,
+} from 'latticesql';
+
+// JSON object → proposed schema
+const plan = inferSchema(data); // { entities, dimensions, junctions, skipped }
+
+// Detect the as-of date and any per-row date column
+const asOf = detectAsOf(fileName); // ISO YYYY-MM-DD | null
+const asOfColumns = detectAsOfColumns(data, plan);
+
+// Detect read-only views (per-slice tabs that mirror a master)
+const { views } = dedupeAndDetectViews(data, plan);
+
+// Materialize into a workspace
+const result = await materializeImport({ db, configPath }, data, plan, views, {
+  mode: 'both',
+  asOf,
+  asOfColumn: null,
+});
+// result: { mode, asOf, asOfColumn, tablesCreated, rowsByTable, links, views }
+```
+
+`materializeImport` takes a `mode` of `'schema'` (table structures + dimension
+values + views), `'contents'` (entity rows + links into existing tables), or
+`'both'` (the default). When `asOf` (a file-level ISO date) or `asOfColumn` (a
+per-row date column) is set, rows are stamped and the row identity folds the date
+in, so the same model imported at a new date is a distinct snapshot rather than an
+overwrite. `onProgress` streams the per-phase pipeline steps for a live view.
+
+See [CHANGELOG.md](../CHANGELOG.md) for the full 4.2 list and
+[assistant.md](assistant.md) for the chat-drop experience.