sensemaking 0.27.1 → 0.28.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cli.js +16 -7
- package/dist/cjs/cli/constants.d.cts +2 -0
- package/dist/cjs/cli/constants.d.ts +2 -0
- package/dist/cjs/cli/constants.js +21 -0
- package/dist/cjs/cli/constants.js.map +1 -0
- package/dist/cjs/cli/named.js +2 -1
- package/dist/cjs/cli/named.js.map +1 -1
- package/dist/cjs/cli/shared.js +5 -4
- package/dist/cjs/cli/shared.js.map +1 -1
- package/dist/cjs/cli.js +10 -9
- package/dist/cjs/cli.js.map +1 -1
- package/dist/cjs/embed/query.js.map +1 -1
- package/dist/cjs/store/duckdb/engine.d.cts +11 -0
- package/dist/cjs/store/duckdb/engine.d.ts +11 -0
- package/dist/cjs/store/duckdb/engine.js +193 -0
- package/dist/cjs/store/duckdb/engine.js.map +1 -0
- package/dist/cjs/store/duckdb/lexical.js +34 -20
- package/dist/cjs/store/duckdb/lexical.js.map +1 -1
- package/dist/cjs/store/duckdb/native.d.cts +5 -2
- package/dist/cjs/store/duckdb/native.d.ts +5 -2
- package/dist/cjs/store/duckdb/native.js +157 -13
- package/dist/cjs/store/duckdb/native.js.map +1 -1
- package/dist/cjs/store/duckdb/open.js +151 -3
- package/dist/cjs/store/duckdb/open.js.map +1 -1
- package/dist/cjs/store/duckdb/ordered-bm25.d.cts +3 -3
- package/dist/cjs/store/duckdb/ordered-bm25.d.ts +3 -3
- package/dist/cjs/store/duckdb/ordered-bm25.js +4 -8
- package/dist/cjs/store/duckdb/ordered-bm25.js.map +1 -1
- package/dist/cjs/store/engine.d.cts +28 -0
- package/dist/cjs/store/engine.d.ts +28 -0
- package/dist/cjs/store/engine.js +362 -0
- package/dist/cjs/store/engine.js.map +1 -0
- package/dist/cjs/store/native.d.cts +8 -5
- package/dist/cjs/store/native.d.ts +8 -5
- package/dist/cjs/store/native.js +78 -73
- package/dist/cjs/store/native.js.map +1 -1
- package/dist/cjs/store/open.js +100 -40
- package/dist/cjs/store/open.js.map +1 -1
- package/dist/cjs/store/sqlite/engine.d.cts +12 -0
- package/dist/cjs/store/sqlite/engine.d.ts +12 -0
- package/dist/cjs/store/sqlite/engine.js +53 -0
- package/dist/cjs/store/sqlite/engine.js.map +1 -0
- package/dist/cjs/store/sqlite/lexical.js +3 -4
- package/dist/cjs/store/sqlite/lexical.js.map +1 -1
- package/dist/cjs/store/sqlite/open.js +119 -6
- package/dist/cjs/store/sqlite/open.js.map +1 -1
- package/dist/cjs/store/turso/engine.d.cts +9 -0
- package/dist/cjs/store/turso/engine.d.ts +9 -0
- package/dist/cjs/store/turso/engine.js +189 -0
- package/dist/cjs/store/turso/engine.js.map +1 -0
- package/dist/cjs/store/turso/lexical.js +16 -11
- package/dist/cjs/store/turso/lexical.js.map +1 -1
- package/dist/cjs/store/turso/native.d.cts +5 -1
- package/dist/cjs/store/turso/native.d.ts +5 -1
- package/dist/cjs/store/turso/native.js +157 -8
- package/dist/cjs/store/turso/native.js.map +1 -1
- package/dist/cjs/store/turso/open.js +172 -27
- package/dist/cjs/store/turso/open.js.map +1 -1
- package/dist/cjs/store/turso/reconcile.d.cts +0 -1
- package/dist/cjs/store/turso/reconcile.d.ts +0 -1
- package/dist/cjs/store/turso/reconcile.js +22 -23
- package/dist/cjs/store/turso/reconcile.js.map +1 -1
- package/dist/cjs/store/types.d.cts +2 -0
- package/dist/cjs/store/types.d.ts +2 -0
- package/dist/cjs/store/types.js.map +1 -1
- package/dist/esm/cli/constants.d.ts +2 -0
- package/dist/esm/cli/constants.js +2 -0
- package/dist/esm/cli/constants.js.map +1 -0
- package/dist/esm/cli/named.js +2 -1
- package/dist/esm/cli/named.js.map +1 -1
- package/dist/esm/cli/shared.js +6 -5
- package/dist/esm/cli/shared.js.map +1 -1
- package/dist/esm/cli.js +10 -9
- package/dist/esm/cli.js.map +1 -1
- package/dist/esm/embed/query.js +1 -1
- package/dist/esm/embed/query.js.map +1 -1
- package/dist/esm/store/duckdb/engine.d.ts +11 -0
- package/dist/esm/store/duckdb/engine.js +18 -0
- package/dist/esm/store/duckdb/engine.js.map +1 -0
- package/dist/esm/store/duckdb/lexical.js +11 -11
- package/dist/esm/store/duckdb/lexical.js.map +1 -1
- package/dist/esm/store/duckdb/native.d.ts +5 -2
- package/dist/esm/store/duckdb/native.js +16 -12
- package/dist/esm/store/duckdb/native.js.map +1 -1
- package/dist/esm/store/duckdb/open.js +40 -2
- package/dist/esm/store/duckdb/open.js.map +1 -1
- package/dist/esm/store/duckdb/ordered-bm25.d.ts +3 -3
- package/dist/esm/store/duckdb/ordered-bm25.js +4 -9
- package/dist/esm/store/duckdb/ordered-bm25.js.map +1 -1
- package/dist/esm/store/engine.d.ts +28 -0
- package/dist/esm/store/engine.js +85 -0
- package/dist/esm/store/engine.js.map +1 -0
- package/dist/esm/store/native.d.ts +8 -5
- package/dist/esm/store/native.js +59 -57
- package/dist/esm/store/native.js.map +1 -1
- package/dist/esm/store/open.js +24 -7
- package/dist/esm/store/open.js.map +1 -1
- package/dist/esm/store/sqlite/engine.d.ts +12 -0
- package/dist/esm/store/sqlite/engine.js +16 -0
- package/dist/esm/store/sqlite/engine.js.map +1 -0
- package/dist/esm/store/sqlite/lexical.js +2 -3
- package/dist/esm/store/sqlite/lexical.js.map +1 -1
- package/dist/esm/store/sqlite/open.js +38 -7
- package/dist/esm/store/sqlite/open.js.map +1 -1
- package/dist/esm/store/turso/engine.d.ts +9 -0
- package/dist/esm/store/turso/engine.js +14 -0
- package/dist/esm/store/turso/engine.js.map +1 -0
- package/dist/esm/store/turso/lexical.js +9 -9
- package/dist/esm/store/turso/lexical.js.map +1 -1
- package/dist/esm/store/turso/native.d.ts +5 -1
- package/dist/esm/store/turso/native.js +16 -10
- package/dist/esm/store/turso/native.js.map +1 -1
- package/dist/esm/store/turso/open.js +44 -3
- package/dist/esm/store/turso/open.js.map +1 -1
- package/dist/esm/store/turso/reconcile.d.ts +0 -1
- package/dist/esm/store/turso/reconcile.js +3 -8
- package/dist/esm/store/turso/reconcile.js.map +1 -1
- package/dist/esm/store/types.d.ts +2 -0
- package/dist/esm/store/types.js.map +1 -1
- package/package.json +21 -6
- package/skills/sense/SKILL.md +1 -1
- package/skills/sense/references/stores/duckdb.md +1 -1
- package/skills/sense/references/stores/sqlite.md +1 -1
- package/skills/sense/references/stores/turso.md +1 -1
- package/skills/sense-setup/references/stores/duckdb.md +1 -1
- package/skills/sense-setup/references/stores/sqlite.md +1 -1
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"sources":["/Users/kevin/Dev/open-source/ai/sensemaking/src/store/types.ts"],"sourcesContent":["// The backing-store interface: a minimal portable statement surface (exec/prepare), plus\n// dedicated interfaces exactly where engines diverge (lexical index, vector scan, raw sql).\n\nimport type { Config, ResolvedConfig, StoreName } from '../config/index.ts';\nimport type { ReconcileDelta } from '../features/types.ts';\nimport type { ParsedDoc } from '../scan/index.ts';\n\n// 'vectors': the store's VectorStore is functionally implemented, not present-but-inert; checked\n// at open (index.ts) before a config with an \"embed\" block is allowed onto a store lacking it.\n// 'segment': the engine can run segment() (Intl.Segmenter grapheme clustering) as a SQL function;\n// turso's client cannot register it and has no rewrite for it, unlike has()/basename(), which now\n// work everywhere. An array, not a bare union, so a runtime check reads the same list the type does.\nexport const CAPABILITY_NAMES = ['vectors', 'segment'] as const;\nexport type Capability = (typeof CAPABILITY_NAMES)[number];\n\nexport interface RunResult {\n changes: number | bigint;\n lastInsertRowid: number | bigint;\n}\n\n// A prepared statement's async surface: every supported engine crosses a real async boundary\n// for a query (DuckDB has no synchronous client), so run/get/all return Promises.\nexport interface Statement {\n run(...params: unknown[]): Promise<RunResult>;\n get(...params: unknown[]): Promise<unknown>;\n all(...params: unknown[]): Promise<unknown[]>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n columns(): Array<{ name: string }>;\n setReadBigInts(enabled: boolean): void;\n}\n\n// Connection surface feature-owned SQL runs against inside a store's hot loops (schema,\n// reconcile), portable so a feature never imports an engine-specific client type.\nexport interface Connection {\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Bulk-inserts rows that cannot conflict, through a path that binds no per-value parameters.\n // Optional: a store that has nothing faster than its own INSERT omits it and callers fall back\n // (appendRows in shared.ts). `columns` names the values each row carries; a table column the\n // caller does not write takes its default.\n appendRows?(table: string, columns: string[], rows: unknown[][]): Promise<void>;\n}\n\n// One reconcile algorithm (src/store/reconcile.ts), parameterised per engine. reconcileContent is\n// the load-bearing member: it lands every content-table change and owns its own multi-step\n// strategy (sqlite FTS5 incremental, duckdb combined delete/insert, turso incremental-or-rebuild).\nexport interface ReconcileDialect {\n // BEGIN mode for the whole reconcile: sqlite/turso 'BEGIN IMMEDIATE', duckdb 'BEGIN'.\n beginMode(): string;\n // Throws SenseError('COLUMN_LIMIT', ...) past this store's own column ceiling and reasoning.\n checkColumnLimit(count: number): void;\n // Adds `names` to frontmatter, already filtered to columns this connection doesn't have yet.\n // sqlite/turso loop (their ADD COLUMN is metadata-only); duckdb issues one statement per call.\n addColumns(conn: Connection, names: string[]): Promise<void>;\n // Replaces this engine's existing-row frontmatter upsert, inside the caller's\n // publication transaction. Preserve non-writable columns and repair missing rows.\n updateFrontmatter?(conn: Connection, writableColumns: string[], rows: unknown[][]): Promise<void>;\n // Reconciles searchable content for `touched` paths and parsed `docs`; `delta` carries the tree\n // state a strategy may need. Must not open its own transaction, and must not return before its own\n // multi-step strategy (e.g. turso's DROP/rebuild) completes.\n reconcileContent(conn: Connection, touched: string[], docs: ParsedDoc[], delta: ReconcileDelta, cfg: Config): Promise<void>;\n // Records this reconcile's write-transaction duration. sqlite/turso use it for open()'s derived\n // busy_timeout; duckdb has no such PRAGMA and omits it.\n recordDuration?(conn: Connection, ms: number): Promise<void>;\n}\n\n// One open algorithm (src/store/open.ts), parameterised per engine. `Handle` is whatever this\n// store needs to close the connection and construct its Store (sqlite: {db}; duckdb:\n// {instance, duckdb}; turso: db) -- opaque to the shared orchestration, threaded through unchanged.\nexport interface OpenDialect<Handle> {\n // Cache filename under STATE_DIR, e.g. 'cache.db'.\n filename: string;\n // Cache shape version, independent of the config's own `version`; bumping it rebuilds an existing tree.\n schemaVersion: string;\n reconcileDialect: ReconcileDialect;\n // Opens the physical connection and applies pragmas due before any SQL runs (sqlite: busy_timeout\n // + WAL; turso: connect-time timeout; duckdb: none).\n connect(dbPath: string, cfg: ResolvedConfig, options: { existingOnly: boolean; observational: boolean }): Promise<{ handle: Handle; conn: Connection }>;\n // Releases the handle, for both the rebuild-and-reopen branch and error cleanup on this attempt.\n close(handle: Handle, options?: { observational?: boolean }): Promise<void>;\n // True when connect() failed because another process holds the cache file, which the orchestration\n // retries. Each engine words it differently, so the match is the dialect's own (`native-not-emulated`).\n // Absent for sqlite, whose file lock is shared and whose concurrent-open failure is a different defect.\n // Matching on message text is forced, not chosen: measured 2026-09-02, duckdb throws a plain Error\n // whose only own properties are stack and message, and turso sets code to the constant\n // 'GenericFailure' on every failure alike with rawCode undefined. Neither exposes anything a\n // predicate could switch on, so each dialect pins its engine's wordings and unit-tests them.\n isLocked?(err: Error): boolean;\n // Minimal, configuration-independent schema needed before the builder can capture a planning\n // snapshot. This bootstrap may commit independently because no feature generation is published.\n ensureCoreSchema(handle: Handle, conn: Connection): Promise<void>;\n // Configuration-dependent persistent DDL. The builder invokes this only inside its fenced write\n // transaction, so a failed feature enable cannot leave catalog changes under the old signature.\n ensureFeatureSchema(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Connection-local preparation with no durable cache effect (DuckDB's TEMP vector stage).\n prepareConnection?(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Installs the derived busy_timeout PRAGMA right before reconcile (sqlite/turso); absent for\n // duckdb, which has no such PRAGMA.\n setDerivedBusyTimeout?(handle: Handle, conn: Connection, ms: number): Promise<void>;\n // Persistent lexical preparation belongs to build, while no-build calls only the readiness\n // check. SQLite and Turso maintain lexical state during reconcile and omit both hooks.\n prepareLexical?(conn: Connection): Promise<void>;\n assertLexicalReady?(conn: Connection): Promise<void>;\n createStore(handle: Handle, conn: Connection, cfg: ResolvedConfig, options: { observational: boolean }): Store;\n}\n\nexport interface FieldStat {\n field: string;\n coverage: number;\n type: string;\n}\n\nexport interface DocumentStore {\n // Frontmatter column names (including internal ones; callers filter).\n columns(): Promise<string[]>;\n // Per-column coverage (non-null count) and observed type set, aggregated in one SQL query,\n // never one row per note. `scopeWhere` is a caller-built WHERE fragment, per LexicalQueryOptions.\n fieldStats(columns: string[], scopeWhere: string): Promise<FieldStat[]>;\n}\n\n// A ranked match, nothing more. Snippets are cut from the document above the store\n// (commands/search.ts), identically for every engine, so no store carries them.\nexport interface LexicalHit {\n path: string;\n}\n\nexport interface LexicalQueryOptions {\n whereJoin: string;\n whereCond: string;\n scopeCond: string;\n limit: number;\n}\n\nexport interface LexicalIndex {\n // Ranked word-match query with snippet, scoped by the caller-built SQL fragments (the same\n // fragments narrowByWhere/materializeScope produce elsewhere).\n query(terms: string, opts: LexicalQueryOptions): Promise<LexicalHit[]>;\n}\n\nexport interface VectorCandidate {\n path: string;\n lines: string;\n similarity: number;\n}\n\nexport interface VectorSimilar {\n path: string;\n similarity: number;\n}\n\nexport interface VectorWriteRow {\n path: string;\n chunk: number;\n scale: number;\n vector: Buffer;\n}\n\nexport interface VectorStore {\n // Rows whose vector is NULL: never embedded, or added since.\n pending(): Promise<Array<{ path: string; chunk: number }>>;\n // One batch write per call (never per row) so a provider's embedding batch stays inside a\n // single store method.\n writeVectors(rows: VectorWriteRow[]): Promise<void>;\n candidates(queryVector: Float32Array, storeDims: number, fetch: number, allowed?: Set<string>): Promise<VectorCandidate[]>;\n similar(path: string, opts: { exclude: Set<string>; allowed?: Set<string>; k: number }): Promise<VectorSimilar[]>;\n hasVector(path: string): Promise<boolean>;\n}\n\n// The `sense sql` passthrough: string in, streamed rows out. Each store registers the same\n// sense-supplied functions and applies its own read-bigints/error-translation behavior.\nexport interface RawStatement {\n columns(): Array<{ name: string }>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n}\n\nexport interface SqlSession {\n prepare(sql: string): Promise<RawStatement>;\n}\n\nexport interface Store {\n readonly name: StoreName;\n readonly capabilities: ReadonlySet<Capability>;\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n // One crossing, N rows: every external write loop (search's candidate insert, graph ring's\n // temp-table writes) goes through this instead of looping over `run()`. Same contract as Connection.runBatch.\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Pins one snapshot across multi-statement reads; `map` and `peek` use it. A network-bound\n // write (search's embedding top-up) must stay outside it. Nesting joins the enclosing transaction.\n transaction<T>(fn: () => Promise<T>): Promise<T>;\n docs: DocumentStore;\n lexical: LexicalIndex;\n vectors: VectorStore;\n raw: SqlSession;\n // Engine-level facts for `sense status` (e.g. a derived busy_timeout PRAGMA reading); each\n // store owns what it reports and how it is worded. Empty when there is nothing to report.\n engineStatus(): Promise<Record<string, string>>;\n close(): Promise<void>;\n}\n"],"names":["CAPABILITY_NAMES"],"mappings":"AAAA,yFAAyF;AACzF,4FAA4F;AAM5F,iGAAiG;AACjG,+FAA+F;AAC/F,kGAAkG;AAClG,kGAAkG;AAClG,qGAAqG;AACrG,OAAO,MAAMA,mBAAmB;IAAC;IAAW;CAAU,CAAU"}
|
|
1
|
+
{"version":3,"sources":["/Users/kevin/Dev/open-source/ai/sensemaking/src/store/types.ts"],"sourcesContent":["// The backing-store interface: a minimal portable statement surface (exec/prepare), plus\n// dedicated interfaces exactly where engines diverge (lexical index, vector scan, raw sql).\n\nimport type { Config, ResolvedConfig, StoreName } from '../config/index.ts';\nimport type { ReconcileDelta } from '../features/types.ts';\nimport type { ParsedDoc } from '../scan/index.ts';\n\n// 'vectors': the store's VectorStore is functionally implemented, not present-but-inert; checked\n// at open (index.ts) before a config with an \"embed\" block is allowed onto a store lacking it.\n// 'segment': the engine can run segment() (Intl.Segmenter grapheme clustering) as a SQL function;\n// turso's client cannot register it and has no rewrite for it, unlike has()/basename(), which now\n// work everywhere. An array, not a bare union, so a runtime check reads the same list the type does.\nexport const CAPABILITY_NAMES = ['vectors', 'segment'] as const;\nexport type Capability = (typeof CAPABILITY_NAMES)[number];\n\nexport interface RunResult {\n changes: number | bigint;\n lastInsertRowid: number | bigint;\n}\n\n// A prepared statement's async surface: every supported engine crosses a real async boundary\n// for a query (DuckDB has no synchronous client), so run/get/all return Promises.\nexport interface Statement {\n run(...params: unknown[]): Promise<RunResult>;\n get(...params: unknown[]): Promise<unknown>;\n all(...params: unknown[]): Promise<unknown[]>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n columns(): Array<{ name: string }>;\n setReadBigInts(enabled: boolean): void;\n}\n\n// Connection surface feature-owned SQL runs against inside a store's hot loops (schema,\n// reconcile), portable so a feature never imports an engine-specific client type.\nexport interface Connection {\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Bulk-inserts rows that cannot conflict, through a path that binds no per-value parameters.\n // Optional: a store that has nothing faster than its own INSERT omits it and callers fall back\n // (appendRows in shared.ts). `columns` names the values each row carries; a table column the\n // caller does not write takes its default.\n appendRows?(table: string, columns: string[], rows: unknown[][]): Promise<void>;\n}\n\n// One reconcile algorithm (src/store/reconcile.ts), parameterised per engine. reconcileContent is\n// the load-bearing member: it lands every content-table change and owns its own multi-step\n// strategy (sqlite FTS5 incremental, duckdb combined delete/insert, turso incremental-or-rebuild).\nexport interface ReconcileDialect {\n // BEGIN mode for the whole reconcile: sqlite/turso 'BEGIN IMMEDIATE', duckdb 'BEGIN'.\n beginMode(): string;\n // Throws SenseError('COLUMN_LIMIT', ...) past this store's own column ceiling and reasoning.\n checkColumnLimit(count: number): void;\n // Adds `names` to frontmatter, already filtered to columns this connection doesn't have yet.\n // sqlite/turso loop (their ADD COLUMN is metadata-only); duckdb issues one statement per call.\n addColumns(conn: Connection, names: string[]): Promise<void>;\n // Replaces this engine's existing-row frontmatter upsert, inside the caller's\n // publication transaction. Preserve non-writable columns and repair missing rows.\n updateFrontmatter?(conn: Connection, writableColumns: string[], rows: unknown[][]): Promise<void>;\n // Reconciles searchable content for `touched` paths and parsed `docs`; `delta` carries the tree\n // state a strategy may need. Must not open its own transaction, and must not return before its own\n // multi-step strategy (e.g. turso's DROP/rebuild) completes.\n reconcileContent(conn: Connection, touched: string[], docs: ParsedDoc[], delta: ReconcileDelta, cfg: Config): Promise<void>;\n // Records this reconcile's write-transaction duration. sqlite/turso use it for open()'s derived\n // busy_timeout; duckdb has no such PRAGMA and omits it.\n recordDuration?(conn: Connection, ms: number): Promise<void>;\n}\n\n// One open algorithm (src/store/open.ts), parameterised per engine. `Handle` is whatever this\n// store needs to close the connection and construct its Store (sqlite: {db}; duckdb:\n// {instance, duckdb}; turso: db) -- opaque to the shared orchestration, threaded through unchanged.\nexport interface OpenDialect<Handle> {\n // Cache filename under STATE_DIR, e.g. 'cache.db'.\n filename: string;\n // Cache shape version, independent of the config's own `version`; bumping it rebuilds an existing tree.\n schemaVersion: string;\n reconcileDialect: ReconcileDialect;\n // Opens the physical connection and applies pragmas due before any SQL runs (sqlite: busy_timeout\n // + WAL; turso: connect-time timeout; duckdb: none).\n connect(dbPath: string, cfg: ResolvedConfig, options: { existingOnly: boolean; observational: boolean }): Promise<{ handle: Handle; conn: Connection }>;\n // Releases the handle, for both the rebuild-and-reopen branch and error cleanup on this attempt.\n close(handle: Handle, options?: { observational?: boolean }): Promise<void>;\n // True when connect() failed because another process holds the cache file, which the orchestration\n // retries. Each engine words it differently, so the match is the dialect's own (`native-not-emulated`).\n // Absent for sqlite, whose file lock is shared and whose concurrent-open failure is a different defect.\n // Matching on message text is forced, not chosen: measured 2026-09-02, duckdb throws a plain Error\n // whose only own properties are stack and message, and turso sets code to the constant\n // 'GenericFailure' on every failure alike with rawCode undefined. Neither exposes anything a\n // predicate could switch on, so each dialect pins its engine's wordings and unit-tests them.\n isLocked?(err: Error): boolean;\n // Minimal, configuration-independent schema needed before the builder can capture a planning\n // snapshot. This bootstrap may commit independently because no feature generation is published.\n ensureCoreSchema(handle: Handle, conn: Connection): Promise<void>;\n // Configuration-dependent persistent DDL. The builder invokes this only inside its fenced write\n // transaction, so a failed feature enable cannot leave catalog changes under the old signature.\n ensureFeatureSchema(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Connection-local preparation with no durable cache effect (DuckDB's TEMP vector stage).\n prepareConnection?(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Installs the derived busy_timeout PRAGMA right before reconcile (sqlite/turso); absent for\n // duckdb, which has no such PRAGMA.\n setDerivedBusyTimeout?(handle: Handle, conn: Connection, ms: number): Promise<void>;\n // The engine that builds this store's derived index (store/engine.ts): the npm package for duckdb/turso, Node's\n // bundled SQLite for sqlite. A recorded identity that differs rebuilds the cache; checkEngine runs before any build under a new one.\n engineIdentity?(): Promise<string>;\n checkEngine?(): Promise<void>;\n // Persistent lexical preparation belongs to build, while no-build calls only the readiness\n // check. SQLite and Turso maintain lexical state during reconcile and omit both hooks.\n prepareLexical?(conn: Connection): Promise<void>;\n assertLexicalReady?(conn: Connection): Promise<void>;\n createStore(handle: Handle, conn: Connection, cfg: ResolvedConfig, options: { observational: boolean }): Store;\n}\n\nexport interface FieldStat {\n field: string;\n coverage: number;\n type: string;\n}\n\nexport interface DocumentStore {\n // Frontmatter column names (including internal ones; callers filter).\n columns(): Promise<string[]>;\n // Per-column coverage (non-null count) and observed type set, aggregated in one SQL query,\n // never one row per note. `scopeWhere` is a caller-built WHERE fragment, per LexicalQueryOptions.\n fieldStats(columns: string[], scopeWhere: string): Promise<FieldStat[]>;\n}\n\n// A ranked match, nothing more. Snippets are cut from the document above the store\n// (commands/search.ts), identically for every engine, so no store carries them.\nexport interface LexicalHit {\n path: string;\n}\n\nexport interface LexicalQueryOptions {\n whereJoin: string;\n whereCond: string;\n scopeCond: string;\n limit: number;\n}\n\nexport interface LexicalIndex {\n // Ranked word-match query with snippet, scoped by the caller-built SQL fragments (the same\n // fragments narrowByWhere/materializeScope produce elsewhere).\n query(terms: string, opts: LexicalQueryOptions): Promise<LexicalHit[]>;\n}\n\nexport interface VectorCandidate {\n path: string;\n lines: string;\n similarity: number;\n}\n\nexport interface VectorSimilar {\n path: string;\n similarity: number;\n}\n\nexport interface VectorWriteRow {\n path: string;\n chunk: number;\n scale: number;\n vector: Buffer;\n}\n\nexport interface VectorStore {\n // Rows whose vector is NULL: never embedded, or added since.\n pending(): Promise<Array<{ path: string; chunk: number }>>;\n // One batch write per call (never per row) so a provider's embedding batch stays inside a\n // single store method.\n writeVectors(rows: VectorWriteRow[]): Promise<void>;\n candidates(queryVector: Float32Array, storeDims: number, fetch: number, allowed?: Set<string>): Promise<VectorCandidate[]>;\n similar(path: string, opts: { exclude: Set<string>; allowed?: Set<string>; k: number }): Promise<VectorSimilar[]>;\n hasVector(path: string): Promise<boolean>;\n}\n\n// The `sense sql` passthrough: string in, streamed rows out. Each store registers the same\n// sense-supplied functions and applies its own read-bigints/error-translation behavior.\nexport interface RawStatement {\n columns(): Array<{ name: string }>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n}\n\nexport interface SqlSession {\n prepare(sql: string): Promise<RawStatement>;\n}\n\nexport interface Store {\n readonly name: StoreName;\n readonly capabilities: ReadonlySet<Capability>;\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n // One crossing, N rows: every external write loop (search's candidate insert, graph ring's\n // temp-table writes) goes through this instead of looping over `run()`. Same contract as Connection.runBatch.\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Pins one snapshot across multi-statement reads; `map` and `peek` use it. A network-bound\n // write (search's embedding top-up) must stay outside it. Nesting joins the enclosing transaction.\n transaction<T>(fn: () => Promise<T>): Promise<T>;\n docs: DocumentStore;\n lexical: LexicalIndex;\n vectors: VectorStore;\n raw: SqlSession;\n // Engine-level facts for `sense status` (e.g. a derived busy_timeout PRAGMA reading); each\n // store owns what it reports and how it is worded. Empty when there is nothing to report.\n engineStatus(): Promise<Record<string, string>>;\n close(): Promise<void>;\n}\n"],"names":["CAPABILITY_NAMES"],"mappings":"AAAA,yFAAyF;AACzF,4FAA4F;AAM5F,iGAAiG;AACjG,+FAA+F;AAC/F,kGAAkG;AAClG,kGAAkG;AAClG,qGAAqG;AACrG,OAAO,MAAMA,mBAAmB;IAAC;IAAW;CAAU,CAAU"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sensemaking",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.28.0",
|
|
4
4
|
"description": "Query and search your markdown notes with context-aware progressive disclosure: SQL over frontmatter, links, and text, plus semantic search and link-graph ranking. No server, no build step.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"markdown",
|
|
@@ -72,7 +72,7 @@
|
|
|
72
72
|
"schema.json"
|
|
73
73
|
],
|
|
74
74
|
"scripts": {
|
|
75
|
-
"benchmark": "tsds build && node benchmark/gate.
|
|
75
|
+
"benchmark": "tsds build && node benchmark/gate.ts",
|
|
76
76
|
"build": "tsds build",
|
|
77
77
|
"format": "tsds format",
|
|
78
78
|
"prepublishOnly": "tsds validate",
|
|
@@ -84,26 +84,41 @@
|
|
|
84
84
|
"dependencies": {
|
|
85
85
|
"@huggingface/tokenizers": "^0.2.0",
|
|
86
86
|
"franc-min": "^6.2.0",
|
|
87
|
-
"install-module-linked": "^2.
|
|
87
|
+
"install-module-linked": "^2.1.0",
|
|
88
88
|
"markdown-it": "^15.0.1",
|
|
89
89
|
"markdown-it-footnote": "^4.0.0",
|
|
90
90
|
"markdown-it-task-lists": "^2.1.1",
|
|
91
|
+
"semver": "^7.0.0",
|
|
91
92
|
"stemmer": "^2.0.1",
|
|
92
93
|
"tinypool": "^2.1.2",
|
|
93
94
|
"yaml": "^2.9.0"
|
|
94
95
|
},
|
|
95
96
|
"devDependencies": {
|
|
96
|
-
"@duckdb/node-api": "
|
|
97
|
-
"@tursodatabase/database": "
|
|
97
|
+
"@duckdb/node-api": "^1.5.6-r.1",
|
|
98
|
+
"@tursodatabase/database": "^0.8.2",
|
|
98
99
|
"@types/markdown-it-footnote": "^3.0.4",
|
|
99
100
|
"@types/mocha": "*",
|
|
100
|
-
"@types/node": "
|
|
101
|
+
"@types/node": "^22",
|
|
102
|
+
"@types/semver": "^7.8.0",
|
|
101
103
|
"cr": "^0.1.0",
|
|
102
104
|
"fs-remove-compat": "^1.0.4",
|
|
103
105
|
"node-version-use": "*",
|
|
106
|
+
"portable-env": "^1.4.0",
|
|
104
107
|
"ts-dev-stack": "*",
|
|
105
108
|
"tsds-config": "*"
|
|
106
109
|
},
|
|
110
|
+
"peerDependencies": {
|
|
111
|
+
"@duckdb/node-api": "^1.5.6-r.1",
|
|
112
|
+
"@tursodatabase/database": "^0.8.2"
|
|
113
|
+
},
|
|
114
|
+
"peerDependenciesMeta": {
|
|
115
|
+
"@duckdb/node-api": {
|
|
116
|
+
"optional": true
|
|
117
|
+
},
|
|
118
|
+
"@tursodatabase/database": {
|
|
119
|
+
"optional": true
|
|
120
|
+
}
|
|
121
|
+
},
|
|
107
122
|
"engines": {
|
|
108
123
|
"node": ">=22.20"
|
|
109
124
|
}
|
package/skills/sense/SKILL.md
CHANGED
|
@@ -112,7 +112,7 @@ Save a query in `sense.config.json` only when it will be reused:
|
|
|
112
112
|
|
|
113
113
|
Run these as `sense by-tag urgent` and `sense hot`. Invocation flags override a saved search's `preset`, `k`, or `where` value.
|
|
114
114
|
|
|
115
|
-
Running a saved entry validates it. Exit code 0 means it ran,
|
|
115
|
+
Running a saved entry validates it. Exit code 0 means it ran, 6 means the invocation needs different arguments, and 4 means an operational failure, such as a query, store or CLI bootstrap failure. An empty result can be valid data, so interpret it from the query's purpose.
|
|
116
116
|
|
|
117
117
|
## Tables
|
|
118
118
|
|
|
@@ -4,7 +4,7 @@ Read this guide when `store` is `duckdb`.
|
|
|
4
4
|
|
|
5
5
|
Sense stores the cache at `.sense/cache.duckdb`. It is a DuckDB database file with frontmatter values in `VARIANT` columns and embeddings in native fixed-width float arrays. Use this store when another DuckDB tool needs to inspect or analyze the cache.
|
|
6
6
|
|
|
7
|
-
The first open installs `@duckdb/node-api` if needed. Its native download is about 110 MB. Under DuckDB's [concurrency model](https://duckdb.org/docs/current/connect/concurrency), Sense opens an existing cache for no-build observation in read-only mode, so several independent reader processes can coexist. A build-enabled command or watcher opens it read-write and excludes every other process until it closes. Contenders wait within Sense's bounded lock budget, then fail with `STORE_BUSY`. This cross-process rule does not promise writer/reader coexistence or unrestricted sharing by independently configured DuckDB instances in one process.
|
|
7
|
+
The first open installs `@duckdb/node-api` if needed, at the newest version in sensemaking's declared range. Its native download is about 110 MB. A build after the installed version changes rebuilds the cache, and a no-build query on that cache fails with `INDEX_NOT_READY` until it does. Under DuckDB's [concurrency model](https://duckdb.org/docs/current/connect/concurrency), Sense opens an existing cache for no-build observation in read-only mode, so several independent reader processes can coexist. A build-enabled command or watcher opens it read-write and excludes every other process until it closes. Contenders wait within Sense's bounded lock budget, then fail with `STORE_BUSY`. This cross-process rule does not promise writer/reader coexistence or unrestricted sharing by independently configured DuckDB instances in one process.
|
|
8
8
|
|
|
9
9
|
## `sense search` grammar
|
|
10
10
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Read this guide when `store` is `sqlite` or omitted.
|
|
4
4
|
|
|
5
|
-
SQLite is the default store. It uses Node's built-in SQLite, keeps the cache at `.sense/cache.db`, supports concurrent sense commands through WAL, and implements the full search grammar documented here.
|
|
5
|
+
SQLite is the default store. It uses Node's built-in SQLite, keeps the cache at `.sense/cache.db`, supports concurrent sense commands through WAL, and implements the full search grammar documented here. A build after a Node upgrade that changes the bundled SQLite version rebuilds the cache, and a no-build query on that cache fails with `INDEX_NOT_READY` until it does.
|
|
6
6
|
|
|
7
7
|
## `sense search` grammar
|
|
8
8
|
|
|
@@ -4,7 +4,7 @@ Read this guide when `store` is `turso`.
|
|
|
4
4
|
|
|
5
5
|
Sense stores the cache at `.sense/cache.turso.db` and uses `@tursodatabase/database` with Tantivy full-text indexes. Use this experimental adapter when the tree benefits from the embedded Turso engine or its text-index behavior.
|
|
6
6
|
|
|
7
|
-
The first open installs the native package if needed. Turso holds the cache file for the connection's lifetime, so another Sense command waits for the current command or watcher cycle to close it. The upstream engine supports more deployment and concurrency options than this adapter currently exposes. Sense opens a local file and does not configure sync, remote access, concurrent writers, or encryption.
|
|
7
|
+
The first open installs the native package if needed, at the newest version in sensemaking's declared range. A build after the installed version changes rebuilds the cache, and a no-build query on that cache fails with `INDEX_NOT_READY` until it does. Turso holds the cache file for the connection's lifetime, so another Sense command waits for the current command or watcher cycle to close it. The upstream engine supports more deployment and concurrency options than this adapter currently exposes. Sense opens a local file and does not configure sync, remote access, concurrent writers, or encryption.
|
|
8
8
|
|
|
9
9
|
## `sense search` grammar
|
|
10
10
|
|
|
@@ -17,7 +17,7 @@ Sense's [store benchmark summary](../store-benchmarks.md) measures the current a
|
|
|
17
17
|
## Current Sense adapter boundaries
|
|
18
18
|
|
|
19
19
|
- DuckDB support is experimental.
|
|
20
|
-
- The first open installs `@duckdb/node-api
|
|
20
|
+
- The first open installs the newest `@duckdb/node-api` in the range sensemaking's `peerDependencies` declares, a native download of about 110 MB. An already-installed version is used, and one outside that range is reported. A build after the version changes rebuilds the cache.
|
|
21
21
|
- Under DuckDB's [concurrency model](https://duckdb.org/docs/current/connect/concurrency), Sense opens an existing cache for no-build observation in read-only mode, so several independent reader processes can coexist. A build-enabled command or watcher opens it read-write and excludes every other process until it closes. Contenders wait within Sense's bounded lock budget, then fail with `STORE_BUSY`. This cross-process rule does not promise writer/reader coexistence or unrestricted sharing by independently configured DuckDB instances in one process.
|
|
22
22
|
- `sense search` accepts bare words and quoted phrases. It rejects FTS5 boolean, prefix, `NEAR`, initial-token, grouping, and column-filter syntax.
|
|
23
23
|
- SQLite's raw `MATCH`, `snippet()`, and `bm25()` queries do not port.
|
|
@@ -12,7 +12,7 @@ Choose it when the tree has no engine-specific requirement, when saved searches
|
|
|
12
12
|
- The sense SQL functions `has()`, `basename()`, and `segment()`.
|
|
13
13
|
- WAL-backed concurrent sense commands.
|
|
14
14
|
|
|
15
|
-
The cache is `.sense/cache.db`. It is derived data and can be rebuilt from the markdown files.
|
|
15
|
+
The cache is `.sense/cache.db`. It is derived data and can be rebuilt from the markdown files. It records the bundled SQLite version that built it, so a Node upgrade that changes that version rebuilds it on the next build.
|
|
16
16
|
|
|
17
17
|
## When another store fits the workflow
|
|
18
18
|
|