dsh-session-index 0.0.0-stage → 0.1.0-beta.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 John Lam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,145 @@
-# Temporary Holding Version
+# dsh-session-index
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A derived SQLite index over **DeepSeek Harness session logs**, as a plugin: full-text search, listing, reading, and an
+incremental rebuild — offered as a **service** and as **agent-facing tools**.
+
+It is a Node program with a SQLite file. It imports **no dsh modules at all**: only `node:sqlite`,
+`node:child_process`, `node:fs`, `node:os`, `node:path` and `node:url`. What it knows about dsh is dsh's session
+*format*, which it reads.
+
+## Why this exists
+
+The harness already ships a SQLite FTS index (`@deepseek-ai/dsh-session-query-sqlite`) and agent tools that use it
+(`@deepseek-ai/dsh-tool-session-query`). **We tried to use them and it broke a deployment**: enabling that index made
+`api-session-controller` fail to start, its own code calling `provider.searchSessions`, and the cause was never
+reproduced — a probe of the same profile on another port booted cleanly. The deployment left the harness index at
+`openAt: never`, which also means the harness's five session tools have no backend to answer from.
+
+So this package carries the capability instead. The history is in the parent repository's findings register (`F98`,
+`F96`, `F99`, `F102`, `F103`), and it is written here so that a reader of this package alone knows why it is not just a
+wrapper around the thing that already exists.
+
+## Install
+
+```bash
+dsh plugin --profile <profile> add github:johnlam1968/dsh-session-index
+```
+
+or as a dependency plus a bundle entry in the profile's `package.json`:
+
+```json
+{ "dependencies": { "dsh-session-index": "github:johnlam1968/dsh-session-index" },
+  "dsh": { "profile": { "bundles": ["...", "dsh-session-index"] } } }
+```
+
+The row it inserts is `session-index`, which provides the service and registers the tools:
+
+```yaml
+- insert:
+    - id: session-index
+      name: 'dsh-session-index'
+```
+
+`Config`: `path` (the store; default `$DSH_HOME/session-index.db`), `sessionsDir` (default `$DSH_HOME/sessions`),
+`tokenizer` (`trigram` | `unicode61` | `none`).
+
+## The service
+
+`ctx.localSessionIndex` — the capability without the schema: no SQL, no table names and no file layout cross the
+boundary, so a consumer cannot come to depend on a store that is derived and free to change.
+
+| method | what it answers |
+|---|---|
+| `search(term, {limit})` | sessions whose **text, reasoning, tool results or tool-call arguments** match |
+| `find(term, {limit})` | sessions whose **title, id or working directory** match |
+| `list({cwd, search, limit})` | what the store holds |
+| `read(id, {kinds, lastMessages, offset, messageChars})` | one session's conversation, from its log |
+| `meta()` | what the store says about itself: `text_indexed`, `search_mode`, `tokenizer`, `fts_rows`, `sessions` |
+| `refresh({timeoutMs})` | an incremental rebuild in a **child process** |
+| `build({withText, incremental, tokenizer})` | a rebuild **in this process** — for a script or a migration |
+| `row(id)` | one stored row, for a caller that wants the file behind an id |
+
+## The agent-facing tools
+
+| tool | the question it answers |
+|---|---|
+| `session_index_list` | what is in the store: ids, titles, working directories, size, and whether a row is a SUBAGENT run. `subagents: include\|exclude\|only` filters them (measured: 318 of 499 rows are worker runs), and the answer always reports the mixture |
+| `session_index_read` | one session's conversation, as it was said — with how much session there is and what was left out |
+| `session_index_search` | where a phrase appears, and **which mechanism** answered |
+| `session_index_refresh` | bring the store current, incrementally, in a child process |
+
+### What they deliberately do NOT do
+
+**No measurement semantics.** The observer plugin that consumes this one composes the *subject a judgement would see*
+and marks the *OBSERVE allow-list*; both are facts about measuring a session, not about storing one. A session library
+that carried them would be useless to a deployment that measures nothing. Concretely, this package has:
+
+* no `state`/composer output, no judge's character budget;
+* no allow-list, no "which sessions will be measured";
+* no readings, no probabilities, no trace.
+
+### Where the tools came from, and what is next
+
+`list` and `read` were added when this plugin was split out of the observer, because a session library that cannot be
+asked what it holds, or what a session said, is not usable on its own. Remarks for the next reader:
+
+* **`session_index_stats`** is not a tool; `meta()` carries the same facts and `list` reports the counts. Add it only
+  if a model has a question that neither answers.
+* **Live sessions**: `read` reads the *persisted* log, and `list` reads the *index*, so a session still being written
+  is only as current as the last build. `refresh` is the way to close that gap, and a `session_index_read` of a *live*
+  in-memory session would need the harness's `sessionQuery` service — a dsh dependency this package deliberately does
+  not take.
+* **Other harnesses** (pi, minimax-code, zeroclaw, Hermes) write their own formats; the reader is dsh-format-specific
+  and a second format is a second reader, not a flag.
+* **The service is live but NOT CATALOGUED**, measured after the first real deployment: `cordis_inspect_query
+  { platform: 'host', provider: 'Service', method: 'listService' }` lists the harness's own services (including
+  `sessionQuery`) and does not list `localSessionIndex`, while `listTools` does list all four of this plugin's tools.
+  The catalogue appears to hold services that carry a declared contract, and this one is provided as a plain object --
+  a consumer can still inject it by name, but it is not discoverable or typed through the catalogue. Declaring it
+  properly is the first item for the next reader.
+
+## The store
+
+Derived and droppable: a refresh recreates it from the session logs, and nothing in it is a source of truth. Tables:
+`sessions`, `messages`, `tool_calls`, `tool_results`, `search_fts` (the FTS5 mirror), `meta` (the mode receipts).
+
+**The tokenizer decides what a query MEANS**, so it is chosen and recorded in `meta`:
+
+| tokenizer | `MATCH` finds | store size (measured: 499 sessions, 121.5 MB of text) |
+|---|---|---|
+| `trigram` (**default**) | substrings, exactly like a `LIKE` scan | 617.8 MB |
+| `unicode61` | words and phrases only — `oice-prox` finds nothing | 353.3 MB |
+| `none` (`build --no-fts`) | the scan over the stored tables | 154.7 MB |
+
+A query **shorter than three characters** is answered by the scan and reports `like`, because three is the smallest run
+a trigram index stores.
+
+## Measured costs (one host, 499 sessions)
+
+| what | cost |
+|---|---|
+| full build with text and the trigram mirror | 562 s, 617.8 MB, 91,158 mirrored rows |
+| refresh of ONE living 33 MB session | **2.7 s** — refold 2.2 s (decode 0.9, fold 1.0, insert 0.4), mirror **0.4 s** (append-only) |
+| search | 4–13 ms (mirror) against 242 ms (scan) |
+
+Two fixes are recorded in the numbers rather than in prose: writing a refolded session in **one transaction** took the
+insert phase from 41.3 s to 0.3 s, and maintaining the FTS mirror **per session** took a one-session refresh from
+110 s to 56 s. The mirror step is now **APPEND-ONLY**, and the remedy this README used to name was retired by measurement
+(`F106`): `DELETE … WHERE session_id = ?` on a 15,037-row session costs **4,087 ms**, deleting the same rows **by rowid**
+costs **4,129 ms**, and the lookup a `session_id → rowid` map would replace costs **92 ms** — the seconds are FTS5's
+trigram index work, not the search for the rows. So the work was removed instead of the lookup: `search_fts` carries
+`seq`, `mirror_state(session_id, high_water)` records how far each session's mirror was built, and maintenance appends
+**only the rows above that mark** — which a DSH log makes safe, because it only ever grows. A session with no receipt,
+a log that shrank, or a row with no `seq` is **replaced** rather than guessed at, and the build reports which path ran
+(`ftsAppended` / `ftsReplaced` / `ftsUnchanged`). `SCHEMA_VERSION` 4 costs one full rebuild, once.
+
+## Tests
+
+```bash
+npm test        # 28 tests: the store, the refresh, the plugin, the four tools
+npm run coverage
+```
+
+## License
+
+MIT

package/bin/session-index.mjs

@@ -0,0 +1,92 @@
+#!/usr/bin/env node
+// THE COMMAND LINE: build, find, search and stats, over the store in `../lib/`.
+//
+// It is deliberately thin. Everything it does is a call into `lib/build.js` (the builder) or `lib/store.js` (the read
+// side), so the CLI cannot drift from what the plugin's service does -- the same functions answer both.
+
+import { statSync } from 'node:fs'
+import { join } from 'node:path'
+import { DEFAULT_INDEX, DEFAULT_SESSIONS_DIR, buildIndex, readCounts } from '../lib/build.js'
+import { findSessions, searchSessions } from '../lib/store.js'
+
+const ms = (t) => `${(Number(process.hrtime.bigint() - t) / 1e6).toFixed(0)} ms`
+
+async function main(argv) {
+    const [command, ...rest] = argv
+    const flag = (name, fallback) => {
+        const at = rest.indexOf(`--${name}`)
+        return at === -1 ? fallback : rest[at + 1]
+    }
+    const out = flag('out', DEFAULT_INDEX)
+    if (command === 'build') {
+        const started = process.hrtime.bigint()
+        const result = buildIndex({
+            sessionsDir: flag('sessions', DEFAULT_SESSIONS_DIR),
+            out,
+            withText: rest.includes('--text'),
+            incremental: rest.includes('--incremental'),
+            // `--no-fts` builds the same store WITHOUT the mirror: a real store that searches by scan, which is how the
+            // scan path stays exercised rather than assumed.
+            fts: !rest.includes('--no-fts'),
+            tokenizer: flag('tokenizer', 'trigram'),
+            onProgress: (done, total) => process.stderr.write(`  ${done}/${total}\r`),
+        })
+        console.log(`${result.sessions} session(s) in the store; refolded ${result.refolded}, skipped ${result.skipped} unchanged`
+            + `${result.schemaMoved ? ' (the SCHEMA moved, so every receipt was void)' : ''}`
+            + `${result.modeChanged && !result.schemaMoved ? ' (the TEXT MODE changed, so every receipt was void)' : ''}`
+            + `, in ${ms(started)} -> ${result.out}`)
+        console.log(`  cost: refold ${(result.refoldMs / 1000).toFixed(1)} s (decode ${(result.decodeMs / 1000).toFixed(1)} s, fold ${(result.foldMs / 1000).toFixed(1)} s, insert ${(result.insertMs / 1000).toFixed(1)} s), mirror ${(result.mirrorMs / 1000).toFixed(1)} s`)
+        console.log(`  search: ${result.searchMode}${result.searchMode === 'fts5' ? ` (${result.tokenizer}), ${result.ftsRows} mirrored row(s)${result.ftsRebuilt ? ' (rebuilt whole)'
+                : result.ftsMaintained > 0 ? ` (maintained for ${result.ftsMaintained} session(s): ${result.ftsAppended} appended, ${result.ftsReplaced} replaced, ${result.ftsUnchanged} unchanged)`
+                : ' (unchanged)'}` : ' -- no FTS5 mirror, text is matched by scan'}`)
+        console.log(`  with a session/title event: ${result.titled} | without: ${result.untitled}` +
+            (result.untitled > 0 ? ' (those are the ones a title service must fold per request)' : ''))
+        console.log(`  size: ${(statSync(out).size / 1048576).toFixed(1)} MB`)
+        return 0
+    }
+    if (command === 'find') {
+        const term = rest.find((a) => !a.startsWith('--') && a !== flag('out', null) && a !== flag('limit', null))
+        if (term === undefined) { console.error('find needs a term'); return 2 }
+        const started = process.hrtime.bigint()
+        const rows = await findSessions(term, { path: out, limit: Number(flag('limit', 20)) })
+        console.log(`${rows.length} session(s) matching ${JSON.stringify(term)} in ${ms(started)}`)
+        for (const row of rows) {
+            console.log(`  ${row.id}`)
+            console.log(`    title: ${row.title === null ? '(none)' : row.title}${row.title_source === null ? '' : ` [${row.title_source}]`}`)
+            console.log(`    cwd:   ${row.cwd ?? '?'}`)
+            console.log(`    ${row.asks} asks, ${row.messages} messages, ${row.tool_calls} tool calls, ${new Date(Number(row.created_at ?? 0)).toISOString().slice(0, 16)}`)
+        }
+        return 0
+    }
+    if (command === 'search') {
+        const term = rest.find((a) => !a.startsWith('--') && a !== flag('out', null) && a !== flag('limit', null))
+        if (term === undefined) { console.error('search needs a term'); return 2 }
+        const started = process.hrtime.bigint()
+        const found = await searchSessions(term, { path: out, limit: Number(flag('limit', 20)) })
+        console.log(`${found.total} session(s) matching ${JSON.stringify(found.term)} in ${ms(started)}`
+            + (found.textIndexed
+                ? ` (${found.searchMode === 'fts5-trigram' ? 'FTS5 trigram mirror' : found.searchMode === 'fts5' ? 'FTS5 word mirror' : 'LIKE scan'}: message text, reasoning, tool results and tool arguments)`
+                : ' -- CONVERSATION TEXT IS NOT INDEXED in this store, so only titles, ids and directories were compared; rebuild with --text to search what was said and what tools returned'))
+        for (const row of found.rows) {
+            console.log(`  ${row.id}`)
+            console.log(`    title: ${row.title === null || row.title === undefined ? '(none)' : row.title}`)
+            console.log(`    cwd:   ${row.cwd ?? '?'}`)
+            console.log(`    matched in: ${row.matchedIn}${row.hits > 0 ? ` (${row.hits} message(s))` : ''}`)
+            if (row.snippet !== undefined && row.snippet !== '') console.log(`    …${row.snippet.split('\n').join(' ')}…`)
+        }
+        return 0
+    }
+    if (command === 'stats') {
+        const one = readCounts(out)
+        const size = (statSync(out).size / 1048576).toFixed(1)
+        console.log(`${one.sessions} session(s), ${one.titled} titled, ${one.asks} asks, ${one.shadowed} shadowed message(s), ${size} MB`)
+        return 0
+    }
+    console.error('usage: session-index.mjs build|find|search|stats [--out FILE] [--sessions DIR] [--text] [--incremental] [--limit N]')
+    return 2
+}
+
+// THE EXIT CODE, NOT `process.exit`. Measured: with stdout redirected to a file or a pipe, `process.exit()` after the
+// summary was built threw the summary away -- the rebuild's own lines vanished from its log while the build had
+// succeeded. Setting the code lets the process end when the loop drains, so everything written is flushed.
+if (process.argv[1] !== undefined && import.meta.url.endsWith(process.argv[1].split('/').pop())) process.exitCode = await main(process.argv.slice(2))

package/cordis.patch.yml

@@ -0,0 +1,6 @@
+# A HOST ROW: the store is a property of the deployment, not of one agent's preset. The config is omitted on purpose --
+# every default is the one a plain deployment wants (the store beside the harness home, `trigram` tokenization, the
+# harness's own sessions directory), and a default restated in two places is a default that drifts.
+- insert:
+    - id: session-index
+      name: 'dsh-session-index'

package/index.js

@@ -0,0 +1,105 @@
+// DSH-SESSION-INDEX: a session store of our own, as its own plugin.
+//
+// WHAT IT IS. A derived SQLite database over DSH session logs: FTS5 search (message text, reasoning, tool results and
+// tool-call arguments), title/cwd lookup, reading a session's conversation, and an INCREMENTAL rebuild. It provides a
+// `localSessionIndex` SERVICE and four agent-facing tools, and it imports no dsh modules at all.
+//
+// WHY IT EXISTS. The harness ships its own SQLite index (`@deepseek-ai/dsh-session-query-sqlite`) and its own agent
+// tools (`@deepseek-ai/dsh-tool-session-query`). Enabling that index in a live deployment made
+// `api-session-controller` fail to start, and the cause was never reproduced; the deployment left it at
+// `openAt: never` and this store carries the capability instead. That history is the reason this package exists and is
+// written down here so a reader of it alone does not have to discover it.
+
+import { Service } from '@deepseek-ai/cordis'
+import Schema from '@deepseek-ai/schemastery'
+import { defaultIndexPath, findSessions, listSessions, metaOf, searchSessions, sessionRow, subagentCounts } from './lib/store.js'
+import { refreshIndex } from './lib/refresh.js'
+import { readSession } from './lib/read.js'
+import { createTools } from './lib/tools.js'
+
+const name = 'session-index'
+
+/** The service name a consumer injects. */
+export const SESSION_INDEX_SERVICE = 'localSessionIndex'
+
+/**
+ * THE HOST SERVICES THIS PLUGIN REACHES, declared rather than discovered (the audit's rule (c), `F107`): `tools` is the
+ * whole of it. `test/session-index-plugin.test.js` checks this against the source rather than trusting it.
+ */
+export const HOST_SERVICES = ['tools']
+
+const Config = Schema.object({
+    path: Schema.string().description('The derived store. Droppable: a refresh recreates it.').default(defaultIndexPath()),
+    sessionsDir: Schema.string().description('Where the harness keeps its session logs. Defaults to `$DSH_HOME/sessions`.').default(''),
+    tokenizer: Schema.string().description('`trigram` (substring search, largest), `unicode61` (words), or `none` for no mirror.').default('trigram'),
+})
+
+/**
+ * The capability, as an object a plugin can call.
+ *
+ * EXACTLY WHAT A CALLER NEEDS, AND NOTHING ABOUT HOW IT IS STORED: no SQL, no table names and no file layout cross this
+ * boundary, so a consumer cannot come to depend on a schema that is free to change because the store is derived.
+ */
+export function createSessionIndex({ path = defaultIndexPath(), sessionsDir = undefined, tokenizer = 'trigram' } = {}) {
+    const options = sessionsDir === '' || sessionsDir === undefined ? {} : { sessionsDir }
+    return {
+        path,
+        search: (term, { limit = 20 } = {}) => searchSessions(term, { path, limit }),
+        find: (term, { limit = 20 } = {}) => findSessions(term, { path, limit }),
+        list: ({ cwd = null, search = null, subagents = 'include', limit = 20 } = {}) => listSessions({ path, cwd, search, subagents, limit }),
+        read: (id, options = {}) => readSession(id, { path, ...options }),
+        row: (id) => sessionRow(id, { path }),
+        meta: () => metaOf(path),
+        // HOW MANY OF THE ROWS ARE SUBAGENT RUNS, so a list can say what it is a list of (ROADMAP §14.4 item 3).
+        counts: () => subagentCounts({ path }),
+        refresh: ({ timeoutMs = 120000 } = {}) => refreshIndex({ out: path, ...options, timeoutMs }),
+        // THE BUILDER IS IMPORTED LAZILY: `lib/build.js` loads `node:sqlite` at module scope, and loading an experimental
+        // built-in while a host starts should not be a plugin's side effect.
+        build: async ({ withText = true, incremental = false, tokenizer: wanted = tokenizer } = {}) => {
+            const { buildIndex } = await import('./lib/build.js')
+            return buildIndex({ out: path, withText, incremental, tokenizer: wanted, ...options })
+        },
+    }
+}
+
+/**
+ * THE SERVICE, IN THE DOCUMENTED FORM.
+ *
+ * A compliance audit (`docs/findings.md`, F107) measured the consequence of providing it as a plain object: it is
+ * reachable by injection, and it does NOT appear in the live Service catalogue -- `listService` answers "no catalogued
+ * Service named localSessionIndex" -- where the harness's own services (`sessionQuery`, `sessionController`) do. The
+ * class form is what the contract asks for (`services-events.md` 2.1): a `Service` subclass whose constructor names the
+ * key, so `ctx.localSessionIndex` is a registered service rather than an anonymous value.
+ *
+ * THE IMPLEMENTATION IS UNCHANGED, and `createSessionIndex` stays: the same factory a test can call directly, with the
+ * class delegating to it, so the catalogue entry costs nothing in behaviour.
+ */
+export class LocalSessionIndex extends Service {
+    constructor(ctx, config = {}) {
+        super(ctx, SESSION_INDEX_SERVICE)
+        Object.assign(this, createSessionIndex({
+            path: config?.path ?? defaultIndexPath(),
+            sessionsDir: config?.sessionsDir ?? undefined,
+            tokenizer: config?.tokenizer ?? 'trigram',
+        }))
+    }
+}
+
+function apply(ctx, config) {
+    // MOUNTED, NOT PROVIDED: `ctx.plugin` runs the class's constructor, which registers the service under its key.
+    ctx.plugin(LocalSessionIndex, { ...(config ?? {}) })
+    // THE TOOLS ARE THE SAME CAPABILITY, for a model -- and they read the LIVE service, so a row that is replaced (or
+    // never mounts) is not answered by a stale reference.
+    const live = {
+        get path() { return ctx.get(SESSION_INDEX_SERVICE)?.path },
+    }
+    for (const method of ['search', 'find', 'list', 'read', 'row', 'meta', 'counts', 'refresh', 'build']) {
+        live[method] = (...args) => ctx.get(SESSION_INDEX_SERVICE)?.[method](...args)
+    }
+    ctx.inject(['tools'], (child) => {
+        if (child.tools === undefined || typeof child.tools.register !== 'function') return
+        for (const defined of createTools({ service: live })) child.tools.register(defined)
+    })
+}
+
+export { name, Config, apply }

package/lib/args.js

@@ -0,0 +1,20 @@
+// A SMALL ARGUMENT CHECK, because the runtime's schema subset does not refuse an unknown key by itself.
+//
+// Measured, in the plugin this was extracted from: a caller passed an argument the row did not declare and the answer
+// was about the missing parameter rather than about the call, which reads as a bug in the tool. So an unknown key is
+// NAMED, and so is a wrong type -- the same rule as everywhere else here: refuse by name rather than guess.
+
+export function checkAgainst(schema, args, toolName) {
+    const properties = schema?.properties ?? {}
+    for (const key of Object.keys(args)) {
+        if (properties[key] === undefined) throw new Error(`${toolName}: unknown parameter \`${key}\``)
+    }
+    for (const [key, spec] of Object.entries(properties)) {
+        const value = args[key]
+        if (value === undefined) continue
+        if (spec.type === 'number' && (typeof value !== 'number' || Number.isNaN(value))) throw new Error(`${toolName}: \`${key}\` must be a number`)
+        if (spec.type === 'string' && typeof value !== 'string') throw new Error(`${toolName}: \`${key}\` must be a string`)
+        if (spec.type === 'array' && !Array.isArray(value)) throw new Error(`${toolName}: \`${key}\` must be an array`)
+        if (spec.enum !== undefined && !spec.enum.includes(value)) throw new Error(`${toolName}: \`${key}\` must be one of ${spec.enum.join(', ')}`)
+    }
+}