@vellumai/assistant 0.11.0-dev.202607301413.bdcf8bc → 0.11.0-dev.202607301449.cefb1f3

package/ARCHITECTURE.md

@@ -624,10 +624,10 @@ Real-time conversation chat message capture on macOS uses a WebSocket-based stre
 
 Two provider adapters are supported, each implementing the `StreamingTranscriber` interface from `src/stt/types.ts`:
 
-| Provider          | Adapter                                                     | Mode          | Mechanism                                                                                                                                                                                                                                             |
-| ----------------- | ----------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **Deepgram**      | `src/providers/speech-to-text/deepgram-realtime.ts`         | `realtime-ws` | Opens a WebSocket to Deepgram's `/v1/listen` endpoint, forwards raw PCM audio, normalizes Deepgram's `is_final`/`speech_final` semantics into `partial`/`final` events. Uses model `nova-2`.                                                          |
-| **Google Gemini** | `src/providers/speech-to-text/google-gemini-live-stream.ts` | `realtime-ws` | Opens a bidirectional streaming session against Gemini's Live API (`ai.live.connect`), forwards PCM audio frames, and normalizes `serverContent.inputTranscription` events into `partial`/`final` events. Uses model `gemini-live-2.5-flash-preview`. |
+| Provider          | Adapter                                                     | Mode          | Mechanism                                                                                                                                                                                                                                                    |
+| ----------------- | ----------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| **Deepgram**      | `src/providers/speech-to-text/deepgram-realtime.ts`         | `realtime-ws` | Opens a WebSocket to Deepgram's `/v1/listen` endpoint, forwards raw PCM audio, normalizes Deepgram's `is_final`/`speech_final` semantics into `partial`/`final` events. Uses model `nova-2`.                                                                 |
+| **Google Gemini** | `src/providers/speech-to-text/google-gemini-live-stream.ts` | `realtime-ws` | Opens a bidirectional streaming session against Gemini's Live API (`ai.live.connect`), forwards PCM audio frames, and normalizes `serverContent.inputTranscription` events into `partial`/`final` events. Uses model `gemini-2.5-flash-native-audio-latest`. |
 
 **Provider-specific behavior differences:**
 

package/docs/architecture/memory.md

@@ -67,6 +67,9 @@ graph LR
     CONS --> VIEWS["essentials / threads / recent"]
     CONS --> REEMBED["memory_v2_reembed →<br/>concept-page Qdrant collection"]
     CONS --> MAINT["memory_v3_maintain<br/>(v3-live follow-up)"]
+    ING["POST /v1/memory/ingest<br/>(deterministic batch import)"] --> PAGES
+    ING --> REEMBED
+    ING --> MAINT
 ```
 
 - `handleRemember` (`graph/tool-handlers.ts`) appends timestamped bullets to
@@ -85,6 +88,62 @@ graph LR
     failure-backoff-respecting);
   - manual "Run now" via `POST /v1/consolidation/run-now`.
     Failed runs enter an exponential backoff (transient vs billing curves).
+- **Ingestion** (`substrate/ingest.ts`, exposed as `POST /v1/memory/ingest`;
+  generated HTTP operation id `memory_ingest_post`, IPC method
+  `memory_ingest`) is the second sanctioned writer of
+  `memory/concepts/`: a deterministic batch import of fully-formed,
+  page-shaped markdown (frontmatter + body) that bypasses the buffer
+  entirely. The bypass is load-bearing, not a shortcut: buffer timestamps
+  have minute precision, so a bulk append lands every entry on one shared
+  minute stamp, the chunker's timestamp cutoff cannot split inside that
+  minute, and its same-minute burst guard falls back to processing the
+  whole buffer in a single oversized run. Ingest writes validated pages
+  directly instead. Purely mechanical: each page is validated and reported
+  individually, writes hold the consolidation lock so a batch cannot
+  interleave with a consolidation pass, and a batch that wrote at least one
+  page enqueues the same reindex follow-ups as consolidation
+  (`memory_v2_reembed`, `memory_v3_maintain`). Consolidation remains the only
+  LLM-driven writer.
+
+### Ingestion tracks and provenance
+
+Two user-facing flows feed the ingest route, chosen by the shape of the
+source material rather than its size:
+
+- **Distilled import** (`skills/assistant-migration`): memory exported from
+  another assistant is parsed, creator-reviewed, and shaped into concept
+  pages. The input is already distilled, so everything approved becomes
+  pages; assistant exports route here regardless of size because they need
+  candidate parsing and creator review.
+- **Large-corpus skim** (`skills/memory-corpus-ingest`): the raw dataset
+  (recording archives, document dumps) is cold-stored under a workspace
+  imports directory and never enters the memory corpus. Only a small map of
+  roughly 10 to 50 skim pages is ingested, each pointing back at the raw
+  files, and a drill-in retrieval skill authored alongside searches the cold
+  store on demand.
+
+Both stage pages on disk and write them through the `assistant memory ingest`
+CLI (`src/cli/commands/memory/memory-ingest.ts`: staging directory, JSON
+manifest, or stdin; dry-run; overwrite; 200-page batches). The one exception
+is a migration yielding only a handful of approved facts: assistant-migration
+saves those through the normal `remember` tool, so they enter via the buffer
+like any organic capture instead of the ingest route.
+
+Ingested pages carry provenance frontmatter with distinct consumers:
+
+- `source:` marks a page's origin, by convention `import:<provider>`
+  (e.g. `import:chatgpt`), so imported pages stay distinguishable from
+  consolidation-authored ones, which never carry the key.
+- `origin_date:` (ISO 8601 date or datetime) declares when the page's
+  content is from and drives `PageIndexEntry.freshAt`, the effective-recency
+  signal: any finite parse is accepted (pre-epoch dates included; an
+  offset-less datetime is read as UTC), and an absent or unparseable value
+  falls back to file mtime. The v3 fresh lane ranks on `freshAt` and the
+  card's `dated` stamp renders it, so a backdated import ranks by what its
+  content is about rather than when it reached disk. `modifiedAt` stays raw
+  mtime because the maintain job's re-embed delta diffs on it. Synthetic
+  entries (skills, CLI commands) carry `freshAt: null` and are skipped by
+  recency ranking.
 
 ### Read paths
 

package/openapi.yaml

@@ -8264,7 +8264,10 @@ paths:
     post:
       operationId: conversations_import_post
       summary: Import conversations
-      description: Import conversations from a standard JSON payload.
+      description:
+        Import conversations from a standard JSON payload. Created conversations record a provenance source of
+        `import:<provider>` derived from the `sourceKey` prefix (e.g. `chatgpt:abc123` -> `import:chatgpt`), or
+        `import:unknown` when no prefix is present.
       tags:
         - conversations
       requestBody:
@@ -17742,6 +17745,88 @@ paths:
                   - confident
                   - notes
                 additionalProperties: false
+  /v1/memory/ingest:
+    post:
+      operationId: memory_ingest_post
+      summary:
+        Batch-ingest fully-formed concept pages directly into memory/concepts/ (bypassing the consolidation buffer) and
+        enqueue reindex jobs
+      tags:
+        - memory
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                pages:
+                  minItems: 1
+                  maxItems: 200
+                  type: array
+                  items:
+                    type: object
+                    properties:
+                      slug:
+                        type: string
+                      content:
+                        type: string
+                    required:
+                      - slug
+                      - content
+                dryRun:
+                  type: boolean
+                overwrite:
+                  type: boolean
+              required:
+                - pages
+      responses:
+        "200":
+          description: Successful response
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  results:
+                    type: array
+                    items:
+                      type: object
+                      properties:
+                        slug:
+                          type: string
+                        action:
+                          type: string
+                          enum:
+                            - written
+                            - skipped_exists
+                            - invalid
+                        warnings:
+                          type: array
+                          items:
+                            type: string
+                        error:
+                          type: string
+                      required:
+                        - slug
+                        - action
+                        - warnings
+                      additionalProperties: false
+                  written:
+                    type: number
+                  skipped:
+                    type: number
+                  invalid:
+                    type: number
+                  dryRun:
+                    type: boolean
+                required:
+                  - results
+                  - written
+                  - skipped
+                  - invalid
+                  - dryRun
+                additionalProperties: false
   /v1/memory/remember:
     post:
       operationId: memory_remember_post

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vellumai/assistant",
-  "version": "0.11.0-dev.202607301413.bdcf8bc",
+  "version": "0.11.0-dev.202607301449.cefb1f3",
   "license": "MIT",
   "type": "module",
   "exports": {

package/src/__tests__/conversations-import-system-filter.test.ts

@@ -1,8 +1,12 @@
 /**
- * Tests that the conversations import route never persists non-renderable
- * roles. The messages store is UI-facing (`ConversationMessage`), so an
- * imported export carrying agent-context `system` rows must land only its
- * `user`/`assistant` turns — the `system` rows are dropped, not persisted.
+ * Route-level tests for the conversations import route.
+ *
+ * Covers two behaviors: non-renderable roles are never persisted (the
+ * messages store is UI-facing (`ConversationMessage`), so an imported export
+ * carrying agent-context `system` rows must land only its `user`/`assistant`
+ * turns; the `system` rows are dropped, not persisted), and imported
+ * conversations carry a provenance `source` derived from the `sourceKey`
+ * prefix (`import:<provider>`, or `import:unknown` when absent).
  */
 
 import { beforeEach, describe, expect, mock, test } from "bun:test";
@@ -82,3 +86,96 @@ describe("conversations import system-row filtering", () => {
     ).toBe(false);
   });
 });
+
+describe("conversations import provenance source", () => {
+  beforeEach(resetTables);
+
+  test("stamps import:<provider> from a prefixed sourceKey and keeps dedup", async () => {
+    // GIVEN an export whose sourceKey carries a provider prefix
+    const body = {
+      conversations: [
+        {
+          sourceKey: "chatgpt:abc123",
+          title: "ChatGPT import",
+          messages: [{ role: "user", content: "hello" }],
+        },
+      ],
+    };
+
+    // WHEN the conversation is imported
+    const result = (await importHandler({
+      body,
+    } as unknown as RouteHandlerArgs)) as { imported: number };
+    expect(result.imported).toBe(1);
+
+    // THEN the created row records the provider-derived provenance source
+    const db = getDb();
+    const conv = db.select().from(conversations).all()[0];
+    expect(conv.source).toBe("import:chatgpt");
+
+    // AND re-importing the same sourceKey still dedups (no second row)
+    const again = (await importHandler({
+      body,
+    } as unknown as RouteHandlerArgs)) as { imported: number; skipped: number };
+    expect(again.imported).toBe(0);
+    expect(again.skipped).toBe(1);
+    expect(db.select().from(conversations).all()).toHaveLength(1);
+  });
+
+  test("normalizes non-canonical prefixes instead of dropping them", async () => {
+    // GIVEN sourceKeys whose prefixes carry uppercase or underscore characters
+    const body = {
+      conversations: [
+        {
+          sourceKey: "OpenAI:abc",
+          title: "Uppercase prefix",
+          messages: [{ role: "user", content: "hello" }],
+        },
+        {
+          sourceKey: "chat_gpt:def",
+          title: "Underscore prefix",
+          messages: [{ role: "user", content: "hello" }],
+        },
+      ],
+    };
+
+    // WHEN the conversations are imported
+    const result = (await importHandler({
+      body,
+    } as unknown as RouteHandlerArgs)) as { imported: number };
+    expect(result.imported).toBe(2);
+
+    // THEN each prefix is normalized into import:<provider>, not import:unknown
+    const db = getDb();
+    const sources = db
+      .select()
+      .from(conversations)
+      .all()
+      .map((row) => row.source)
+      .sort();
+    expect(sources).toEqual(["import:chat-gpt", "import:openai"]);
+  });
+
+  test("falls back to import:unknown when sourceKey is absent", async () => {
+    // GIVEN an export entry with no sourceKey at all
+    const body = {
+      conversations: [
+        {
+          title: "Prefixless import",
+          messages: [{ role: "user", content: "hello" }],
+        },
+      ],
+    };
+
+    // WHEN the conversation is imported
+    const result = (await importHandler({
+      body,
+    } as unknown as RouteHandlerArgs)) as { imported: number };
+    expect(result.imported).toBe(1);
+
+    // THEN the created row falls back to the unknown-provider source
+    const db = getDb();
+    const conv = db.select().from(conversations).all()[0];
+    expect(conv.source).toBe("import:unknown");
+  });
+});

package/src/__tests__/plugin-import-boundary-guard.test.ts

@@ -147,6 +147,7 @@ const BASELINE: Record<string, readonly string[]> = {
     "../../../../runtime/auth/route-policy.js",
     "../../../../runtime/background-job-runner.js",
     "../../../../runtime/routes/errors.js",
+    "../../../../runtime/routes/parse-body.js",
     "../../../../runtime/routes/types.js",
     "../../../../security/secret-scanner.js",
     "../../../../skills/catalog-cache.js",

package/src/__tests__/plugin-import-boundary-reverse-guard.test.ts

@@ -73,6 +73,7 @@ const BASELINE: Record<string, readonly string[]> = {
   ],
   "image-recovery": ["src/daemon/conversation-error.ts"],
   memory: [
+    "src/cli/commands/memory/memory-ingest.ts",
     "src/cli/commands/memory/memory-retrospective.ts",
     "src/cli/commands/memory/memory-v2-compare-render.ts",
     "src/cli/commands/memory/memory-v2.ts",