sensemaking 0.27.1 → 0.28.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/bin/cli.js +16 -7
- package/dist/cjs/cli/constants.d.cts +2 -0
- package/dist/cjs/cli/constants.d.ts +2 -0
- package/dist/cjs/cli/constants.js +21 -0
- package/dist/cjs/cli/constants.js.map +1 -0
- package/dist/cjs/cli/named.js +2 -1
- package/dist/cjs/cli/named.js.map +1 -1
- package/dist/cjs/cli/shared.js +5 -4
- package/dist/cjs/cli/shared.js.map +1 -1
- package/dist/cjs/cli.js +10 -9
- package/dist/cjs/cli.js.map +1 -1
- package/dist/cjs/embed/query.js.map +1 -1
- package/dist/cjs/store/duckdb/engine.d.cts +11 -0
- package/dist/cjs/store/duckdb/engine.d.ts +11 -0
- package/dist/cjs/store/duckdb/engine.js +193 -0
- package/dist/cjs/store/duckdb/engine.js.map +1 -0
- package/dist/cjs/store/duckdb/lexical.js +34 -20
- package/dist/cjs/store/duckdb/lexical.js.map +1 -1
- package/dist/cjs/store/duckdb/native.d.cts +5 -2
- package/dist/cjs/store/duckdb/native.d.ts +5 -2
- package/dist/cjs/store/duckdb/native.js +157 -13
- package/dist/cjs/store/duckdb/native.js.map +1 -1
- package/dist/cjs/store/duckdb/open.js +151 -3
- package/dist/cjs/store/duckdb/open.js.map +1 -1
- package/dist/cjs/store/duckdb/ordered-bm25.d.cts +3 -3
- package/dist/cjs/store/duckdb/ordered-bm25.d.ts +3 -3
- package/dist/cjs/store/duckdb/ordered-bm25.js +4 -8
- package/dist/cjs/store/duckdb/ordered-bm25.js.map +1 -1
- package/dist/cjs/store/engine.d.cts +28 -0
- package/dist/cjs/store/engine.d.ts +28 -0
- package/dist/cjs/store/engine.js +362 -0
- package/dist/cjs/store/engine.js.map +1 -0
- package/dist/cjs/store/native.d.cts +8 -5
- package/dist/cjs/store/native.d.ts +8 -5
- package/dist/cjs/store/native.js +78 -73
- package/dist/cjs/store/native.js.map +1 -1
- package/dist/cjs/store/open.js +100 -40
- package/dist/cjs/store/open.js.map +1 -1
- package/dist/cjs/store/sqlite/engine.d.cts +12 -0
- package/dist/cjs/store/sqlite/engine.d.ts +12 -0
- package/dist/cjs/store/sqlite/engine.js +53 -0
- package/dist/cjs/store/sqlite/engine.js.map +1 -0
- package/dist/cjs/store/sqlite/lexical.js +3 -4
- package/dist/cjs/store/sqlite/lexical.js.map +1 -1
- package/dist/cjs/store/sqlite/open.js +119 -6
- package/dist/cjs/store/sqlite/open.js.map +1 -1
- package/dist/cjs/store/turso/engine.d.cts +9 -0
- package/dist/cjs/store/turso/engine.d.ts +9 -0
- package/dist/cjs/store/turso/engine.js +189 -0
- package/dist/cjs/store/turso/engine.js.map +1 -0
- package/dist/cjs/store/turso/lexical.js +16 -11
- package/dist/cjs/store/turso/lexical.js.map +1 -1
- package/dist/cjs/store/turso/native.d.cts +5 -1
- package/dist/cjs/store/turso/native.d.ts +5 -1
- package/dist/cjs/store/turso/native.js +157 -8
- package/dist/cjs/store/turso/native.js.map +1 -1
- package/dist/cjs/store/turso/open.js +172 -27
- package/dist/cjs/store/turso/open.js.map +1 -1
- package/dist/cjs/store/turso/reconcile.d.cts +0 -1
- package/dist/cjs/store/turso/reconcile.d.ts +0 -1
- package/dist/cjs/store/turso/reconcile.js +22 -23
- package/dist/cjs/store/turso/reconcile.js.map +1 -1
- package/dist/cjs/store/types.d.cts +2 -0
- package/dist/cjs/store/types.d.ts +2 -0
- package/dist/cjs/store/types.js.map +1 -1
- package/dist/esm/cli/constants.d.ts +2 -0
- package/dist/esm/cli/constants.js +2 -0
- package/dist/esm/cli/constants.js.map +1 -0
- package/dist/esm/cli/named.js +2 -1
- package/dist/esm/cli/named.js.map +1 -1
- package/dist/esm/cli/shared.js +6 -5
- package/dist/esm/cli/shared.js.map +1 -1
- package/dist/esm/cli.js +10 -9
- package/dist/esm/cli.js.map +1 -1
- package/dist/esm/embed/query.js +1 -1
- package/dist/esm/embed/query.js.map +1 -1
- package/dist/esm/store/duckdb/engine.d.ts +11 -0
- package/dist/esm/store/duckdb/engine.js +18 -0
- package/dist/esm/store/duckdb/engine.js.map +1 -0
- package/dist/esm/store/duckdb/lexical.js +11 -11
- package/dist/esm/store/duckdb/lexical.js.map +1 -1
- package/dist/esm/store/duckdb/native.d.ts +5 -2
- package/dist/esm/store/duckdb/native.js +16 -12
- package/dist/esm/store/duckdb/native.js.map +1 -1
- package/dist/esm/store/duckdb/open.js +40 -2
- package/dist/esm/store/duckdb/open.js.map +1 -1
- package/dist/esm/store/duckdb/ordered-bm25.d.ts +3 -3
- package/dist/esm/store/duckdb/ordered-bm25.js +4 -9
- package/dist/esm/store/duckdb/ordered-bm25.js.map +1 -1
- package/dist/esm/store/engine.d.ts +28 -0
- package/dist/esm/store/engine.js +85 -0
- package/dist/esm/store/engine.js.map +1 -0
- package/dist/esm/store/native.d.ts +8 -5
- package/dist/esm/store/native.js +59 -57
- package/dist/esm/store/native.js.map +1 -1
- package/dist/esm/store/open.js +24 -7
- package/dist/esm/store/open.js.map +1 -1
- package/dist/esm/store/sqlite/engine.d.ts +12 -0
- package/dist/esm/store/sqlite/engine.js +16 -0
- package/dist/esm/store/sqlite/engine.js.map +1 -0
- package/dist/esm/store/sqlite/lexical.js +2 -3
- package/dist/esm/store/sqlite/lexical.js.map +1 -1
- package/dist/esm/store/sqlite/open.js +38 -7
- package/dist/esm/store/sqlite/open.js.map +1 -1
- package/dist/esm/store/turso/engine.d.ts +9 -0
- package/dist/esm/store/turso/engine.js +14 -0
- package/dist/esm/store/turso/engine.js.map +1 -0
- package/dist/esm/store/turso/lexical.js +9 -9
- package/dist/esm/store/turso/lexical.js.map +1 -1
- package/dist/esm/store/turso/native.d.ts +5 -1
- package/dist/esm/store/turso/native.js +16 -10
- package/dist/esm/store/turso/native.js.map +1 -1
- package/dist/esm/store/turso/open.js +44 -3
- package/dist/esm/store/turso/open.js.map +1 -1
- package/dist/esm/store/turso/reconcile.d.ts +0 -1
- package/dist/esm/store/turso/reconcile.js +3 -8
- package/dist/esm/store/turso/reconcile.js.map +1 -1
- package/dist/esm/store/types.d.ts +2 -0
- package/dist/esm/store/types.js.map +1 -1
- package/package.json +21 -7
- package/skills/sense/EXAMPLES.md +0 -99
- package/skills/sense/SKILL.md +0 -145
- package/skills/sense/references/search.md +0 -63
- package/skills/sense/references/sql.md +0 -120
- package/skills/sense/references/stores/duckdb.md +0 -35
- package/skills/sense/references/stores/sqlite.md +0 -66
- package/skills/sense/references/stores/turso.md +0 -36
- package/skills/sense/references/terminology.md +0 -32
- package/skills/sense-bases/EXAMPLES.md +0 -103
- package/skills/sense-bases/SKILL.md +0 -147
- package/skills/sense-setup/EXAMPLES.md +0 -129
- package/skills/sense-setup/SKILL.md +0 -112
- package/skills/sense-setup/references/embeddings.md +0 -51
- package/skills/sense-setup/references/store-benchmarks.md +0 -16
- package/skills/sense-setup/references/stores/duckdb.md +0 -28
- package/skills/sense-setup/references/stores/sqlite.md +0 -21
- package/skills/sense-setup/references/stores/turso.md +0 -25
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"sources":["/Users/kevin/Dev/open-source/ai/sensemaking/src/store/types.ts"],"sourcesContent":["// The backing-store interface: a minimal portable statement surface (exec/prepare), plus\n// dedicated interfaces exactly where engines diverge (lexical index, vector scan, raw sql).\n\nimport type { Config, ResolvedConfig, StoreName } from '../config/index.ts';\nimport type { ReconcileDelta } from '../features/types.ts';\nimport type { ParsedDoc } from '../scan/index.ts';\n\n// 'vectors': the store's VectorStore is functionally implemented, not present-but-inert; checked\n// at open (index.ts) before a config with an \"embed\" block is allowed onto a store lacking it.\n// 'segment': the engine can run segment() (Intl.Segmenter grapheme clustering) as a SQL function;\n// turso's client cannot register it and has no rewrite for it, unlike has()/basename(), which now\n// work everywhere. An array, not a bare union, so a runtime check reads the same list the type does.\nexport const CAPABILITY_NAMES = ['vectors', 'segment'] as const;\nexport type Capability = (typeof CAPABILITY_NAMES)[number];\n\nexport interface RunResult {\n changes: number | bigint;\n lastInsertRowid: number | bigint;\n}\n\n// A prepared statement's async surface: every supported engine crosses a real async boundary\n// for a query (DuckDB has no synchronous client), so run/get/all return Promises.\nexport interface Statement {\n run(...params: unknown[]): Promise<RunResult>;\n get(...params: unknown[]): Promise<unknown>;\n all(...params: unknown[]): Promise<unknown[]>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n columns(): Array<{ name: string }>;\n setReadBigInts(enabled: boolean): void;\n}\n\n// Connection surface feature-owned SQL runs against inside a store's hot loops (schema,\n// reconcile), portable so a feature never imports an engine-specific client type.\nexport interface Connection {\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Bulk-inserts rows that cannot conflict, through a path that binds no per-value parameters.\n // Optional: a store that has nothing faster than its own INSERT omits it and callers fall back\n // (appendRows in shared.ts). `columns` names the values each row carries; a table column the\n // caller does not write takes its default.\n appendRows?(table: string, columns: string[], rows: unknown[][]): Promise<void>;\n}\n\n// One reconcile algorithm (src/store/reconcile.ts), parameterised per engine. reconcileContent is\n// the load-bearing member: it lands every content-table change and owns its own multi-step\n// strategy (sqlite FTS5 incremental, duckdb combined delete/insert, turso incremental-or-rebuild).\nexport interface ReconcileDialect {\n // BEGIN mode for the whole reconcile: sqlite/turso 'BEGIN IMMEDIATE', duckdb 'BEGIN'.\n beginMode(): string;\n // Throws SenseError('COLUMN_LIMIT', ...) past this store's own column ceiling and reasoning.\n checkColumnLimit(count: number): void;\n // Adds `names` to frontmatter, already filtered to columns this connection doesn't have yet.\n // sqlite/turso loop (their ADD COLUMN is metadata-only); duckdb issues one statement per call.\n addColumns(conn: Connection, names: string[]): Promise<void>;\n // Replaces this engine's existing-row frontmatter upsert, inside the caller's\n // publication transaction. Preserve non-writable columns and repair missing rows.\n updateFrontmatter?(conn: Connection, writableColumns: string[], rows: unknown[][]): Promise<void>;\n // Reconciles searchable content for `touched` paths and parsed `docs`; `delta` carries the tree\n // state a strategy may need. Must not open its own transaction, and must not return before its own\n // multi-step strategy (e.g. turso's DROP/rebuild) completes.\n reconcileContent(conn: Connection, touched: string[], docs: ParsedDoc[], delta: ReconcileDelta, cfg: Config): Promise<void>;\n // Records this reconcile's write-transaction duration. sqlite/turso use it for open()'s derived\n // busy_timeout; duckdb has no such PRAGMA and omits it.\n recordDuration?(conn: Connection, ms: number): Promise<void>;\n}\n\n// One open algorithm (src/store/open.ts), parameterised per engine. `Handle` is whatever this\n// store needs to close the connection and construct its Store (sqlite: {db}; duckdb:\n// {instance, duckdb}; turso: db) -- opaque to the shared orchestration, threaded through unchanged.\nexport interface OpenDialect<Handle> {\n // Cache filename under STATE_DIR, e.g. 'cache.db'.\n filename: string;\n // Cache shape version, independent of the config's own `version`; bumping it rebuilds an existing tree.\n schemaVersion: string;\n reconcileDialect: ReconcileDialect;\n // Opens the physical connection and applies pragmas due before any SQL runs (sqlite: busy_timeout\n // + WAL; turso: connect-time timeout; duckdb: none).\n connect(dbPath: string, cfg: ResolvedConfig, options: { existingOnly: boolean; observational: boolean }): Promise<{ handle: Handle; conn: Connection }>;\n // Releases the handle, for both the rebuild-and-reopen branch and error cleanup on this attempt.\n close(handle: Handle, options?: { observational?: boolean }): Promise<void>;\n // True when connect() failed because another process holds the cache file, which the orchestration\n // retries. Each engine words it differently, so the match is the dialect's own (`native-not-emulated`).\n // Absent for sqlite, whose file lock is shared and whose concurrent-open failure is a different defect.\n // Matching on message text is forced, not chosen: measured 2026-09-02, duckdb throws a plain Error\n // whose only own properties are stack and message, and turso sets code to the constant\n // 'GenericFailure' on every failure alike with rawCode undefined. Neither exposes anything a\n // predicate could switch on, so each dialect pins its engine's wordings and unit-tests them.\n isLocked?(err: Error): boolean;\n // Minimal, configuration-independent schema needed before the builder can capture a planning\n // snapshot. This bootstrap may commit independently because no feature generation is published.\n ensureCoreSchema(handle: Handle, conn: Connection): Promise<void>;\n // Configuration-dependent persistent DDL. The builder invokes this only inside its fenced write\n // transaction, so a failed feature enable cannot leave catalog changes under the old signature.\n ensureFeatureSchema(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Connection-local preparation with no durable cache effect (DuckDB's TEMP vector stage).\n prepareConnection?(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Installs the derived busy_timeout PRAGMA right before reconcile (sqlite/turso); absent for\n // duckdb, which has no such PRAGMA.\n setDerivedBusyTimeout?(handle: Handle, conn: Connection, ms: number): Promise<void>;\n // Persistent lexical preparation belongs to build, while no-build calls only the readiness\n // check. SQLite and Turso maintain lexical state during reconcile and omit both hooks.\n prepareLexical?(conn: Connection): Promise<void>;\n assertLexicalReady?(conn: Connection): Promise<void>;\n createStore(handle: Handle, conn: Connection, cfg: ResolvedConfig, options: { observational: boolean }): Store;\n}\n\nexport interface FieldStat {\n field: string;\n coverage: number;\n type: string;\n}\n\nexport interface DocumentStore {\n // Frontmatter column names (including internal ones; callers filter).\n columns(): Promise<string[]>;\n // Per-column coverage (non-null count) and observed type set, aggregated in one SQL query,\n // never one row per note. `scopeWhere` is a caller-built WHERE fragment, per LexicalQueryOptions.\n fieldStats(columns: string[], scopeWhere: string): Promise<FieldStat[]>;\n}\n\n// A ranked match, nothing more. Snippets are cut from the document above the store\n// (commands/search.ts), identically for every engine, so no store carries them.\nexport interface LexicalHit {\n path: string;\n}\n\nexport interface LexicalQueryOptions {\n whereJoin: string;\n whereCond: string;\n scopeCond: string;\n limit: number;\n}\n\nexport interface LexicalIndex {\n // Ranked word-match query with snippet, scoped by the caller-built SQL fragments (the same\n // fragments narrowByWhere/materializeScope produce elsewhere).\n query(terms: string, opts: LexicalQueryOptions): Promise<LexicalHit[]>;\n}\n\nexport interface VectorCandidate {\n path: string;\n lines: string;\n similarity: number;\n}\n\nexport interface VectorSimilar {\n path: string;\n similarity: number;\n}\n\nexport interface VectorWriteRow {\n path: string;\n chunk: number;\n scale: number;\n vector: Buffer;\n}\n\nexport interface VectorStore {\n // Rows whose vector is NULL: never embedded, or added since.\n pending(): Promise<Array<{ path: string; chunk: number }>>;\n // One batch write per call (never per row) so a provider's embedding batch stays inside a\n // single store method.\n writeVectors(rows: VectorWriteRow[]): Promise<void>;\n candidates(queryVector: Float32Array, storeDims: number, fetch: number, allowed?: Set<string>): Promise<VectorCandidate[]>;\n similar(path: string, opts: { exclude: Set<string>; allowed?: Set<string>; k: number }): Promise<VectorSimilar[]>;\n hasVector(path: string): Promise<boolean>;\n}\n\n// The `sense sql` passthrough: string in, streamed rows out. Each store registers the same\n// sense-supplied functions and applies its own read-bigints/error-translation behavior.\nexport interface RawStatement {\n columns(): Array<{ name: string }>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n}\n\nexport interface SqlSession {\n prepare(sql: string): Promise<RawStatement>;\n}\n\nexport interface Store {\n readonly name: StoreName;\n readonly capabilities: ReadonlySet<Capability>;\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n // One crossing, N rows: every external write loop (search's candidate insert, graph ring's\n // temp-table writes) goes through this instead of looping over `run()`. Same contract as Connection.runBatch.\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Pins one snapshot across multi-statement reads; `map` and `peek` use it. A network-bound\n // write (search's embedding top-up) must stay outside it. Nesting joins the enclosing transaction.\n transaction<T>(fn: () => Promise<T>): Promise<T>;\n docs: DocumentStore;\n lexical: LexicalIndex;\n vectors: VectorStore;\n raw: SqlSession;\n // Engine-level facts for `sense status` (e.g. a derived busy_timeout PRAGMA reading); each\n // store owns what it reports and how it is worded. Empty when there is nothing to report.\n engineStatus(): Promise<Record<string, string>>;\n close(): Promise<void>;\n}\n"],"names":["CAPABILITY_NAMES"],"mappings":"AAAA,yFAAyF;AACzF,4FAA4F;AAM5F,iGAAiG;AACjG,+FAA+F;AAC/F,kGAAkG;AAClG,kGAAkG;AAClG,qGAAqG;AACrG,OAAO,MAAMA,mBAAmB;IAAC;IAAW;CAAU,CAAU"}
|
|
1
|
+
{"version":3,"sources":["/Users/kevin/Dev/open-source/ai/sensemaking/src/store/types.ts"],"sourcesContent":["// The backing-store interface: a minimal portable statement surface (exec/prepare), plus\n// dedicated interfaces exactly where engines diverge (lexical index, vector scan, raw sql).\n\nimport type { Config, ResolvedConfig, StoreName } from '../config/index.ts';\nimport type { ReconcileDelta } from '../features/types.ts';\nimport type { ParsedDoc } from '../scan/index.ts';\n\n// 'vectors': the store's VectorStore is functionally implemented, not present-but-inert; checked\n// at open (index.ts) before a config with an \"embed\" block is allowed onto a store lacking it.\n// 'segment': the engine can run segment() (Intl.Segmenter grapheme clustering) as a SQL function;\n// turso's client cannot register it and has no rewrite for it, unlike has()/basename(), which now\n// work everywhere. An array, not a bare union, so a runtime check reads the same list the type does.\nexport const CAPABILITY_NAMES = ['vectors', 'segment'] as const;\nexport type Capability = (typeof CAPABILITY_NAMES)[number];\n\nexport interface RunResult {\n changes: number | bigint;\n lastInsertRowid: number | bigint;\n}\n\n// A prepared statement's async surface: every supported engine crosses a real async boundary\n// for a query (DuckDB has no synchronous client), so run/get/all return Promises.\nexport interface Statement {\n run(...params: unknown[]): Promise<RunResult>;\n get(...params: unknown[]): Promise<unknown>;\n all(...params: unknown[]): Promise<unknown[]>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n columns(): Array<{ name: string }>;\n setReadBigInts(enabled: boolean): void;\n}\n\n// Connection surface feature-owned SQL runs against inside a store's hot loops (schema,\n// reconcile), portable so a feature never imports an engine-specific client type.\nexport interface Connection {\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Bulk-inserts rows that cannot conflict, through a path that binds no per-value parameters.\n // Optional: a store that has nothing faster than its own INSERT omits it and callers fall back\n // (appendRows in shared.ts). `columns` names the values each row carries; a table column the\n // caller does not write takes its default.\n appendRows?(table: string, columns: string[], rows: unknown[][]): Promise<void>;\n}\n\n// One reconcile algorithm (src/store/reconcile.ts), parameterised per engine. reconcileContent is\n// the load-bearing member: it lands every content-table change and owns its own multi-step\n// strategy (sqlite FTS5 incremental, duckdb combined delete/insert, turso incremental-or-rebuild).\nexport interface ReconcileDialect {\n // BEGIN mode for the whole reconcile: sqlite/turso 'BEGIN IMMEDIATE', duckdb 'BEGIN'.\n beginMode(): string;\n // Throws SenseError('COLUMN_LIMIT', ...) past this store's own column ceiling and reasoning.\n checkColumnLimit(count: number): void;\n // Adds `names` to frontmatter, already filtered to columns this connection doesn't have yet.\n // sqlite/turso loop (their ADD COLUMN is metadata-only); duckdb issues one statement per call.\n addColumns(conn: Connection, names: string[]): Promise<void>;\n // Replaces this engine's existing-row frontmatter upsert, inside the caller's\n // publication transaction. Preserve non-writable columns and repair missing rows.\n updateFrontmatter?(conn: Connection, writableColumns: string[], rows: unknown[][]): Promise<void>;\n // Reconciles searchable content for `touched` paths and parsed `docs`; `delta` carries the tree\n // state a strategy may need. Must not open its own transaction, and must not return before its own\n // multi-step strategy (e.g. turso's DROP/rebuild) completes.\n reconcileContent(conn: Connection, touched: string[], docs: ParsedDoc[], delta: ReconcileDelta, cfg: Config): Promise<void>;\n // Records this reconcile's write-transaction duration. sqlite/turso use it for open()'s derived\n // busy_timeout; duckdb has no such PRAGMA and omits it.\n recordDuration?(conn: Connection, ms: number): Promise<void>;\n}\n\n// One open algorithm (src/store/open.ts), parameterised per engine. `Handle` is whatever this\n// store needs to close the connection and construct its Store (sqlite: {db}; duckdb:\n// {instance, duckdb}; turso: db) -- opaque to the shared orchestration, threaded through unchanged.\nexport interface OpenDialect<Handle> {\n // Cache filename under STATE_DIR, e.g. 'cache.db'.\n filename: string;\n // Cache shape version, independent of the config's own `version`; bumping it rebuilds an existing tree.\n schemaVersion: string;\n reconcileDialect: ReconcileDialect;\n // Opens the physical connection and applies pragmas due before any SQL runs (sqlite: busy_timeout\n // + WAL; turso: connect-time timeout; duckdb: none).\n connect(dbPath: string, cfg: ResolvedConfig, options: { existingOnly: boolean; observational: boolean }): Promise<{ handle: Handle; conn: Connection }>;\n // Releases the handle, for both the rebuild-and-reopen branch and error cleanup on this attempt.\n close(handle: Handle, options?: { observational?: boolean }): Promise<void>;\n // True when connect() failed because another process holds the cache file, which the orchestration\n // retries. Each engine words it differently, so the match is the dialect's own (`native-not-emulated`).\n // Absent for sqlite, whose file lock is shared and whose concurrent-open failure is a different defect.\n // Matching on message text is forced, not chosen: measured 2026-09-02, duckdb throws a plain Error\n // whose only own properties are stack and message, and turso sets code to the constant\n // 'GenericFailure' on every failure alike with rawCode undefined. Neither exposes anything a\n // predicate could switch on, so each dialect pins its engine's wordings and unit-tests them.\n isLocked?(err: Error): boolean;\n // Minimal, configuration-independent schema needed before the builder can capture a planning\n // snapshot. This bootstrap may commit independently because no feature generation is published.\n ensureCoreSchema(handle: Handle, conn: Connection): Promise<void>;\n // Configuration-dependent persistent DDL. The builder invokes this only inside its fenced write\n // transaction, so a failed feature enable cannot leave catalog changes under the old signature.\n ensureFeatureSchema(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Connection-local preparation with no durable cache effect (DuckDB's TEMP vector stage).\n prepareConnection?(handle: Handle, conn: Connection, cfg: Config): Promise<void>;\n // Installs the derived busy_timeout PRAGMA right before reconcile (sqlite/turso); absent for\n // duckdb, which has no such PRAGMA.\n setDerivedBusyTimeout?(handle: Handle, conn: Connection, ms: number): Promise<void>;\n // The engine that builds this store's derived index (store/engine.ts): the npm package for duckdb/turso, Node's\n // bundled SQLite for sqlite. A recorded identity that differs rebuilds the cache; checkEngine runs before any build under a new one.\n engineIdentity?(): Promise<string>;\n checkEngine?(): Promise<void>;\n // Persistent lexical preparation belongs to build, while no-build calls only the readiness\n // check. SQLite and Turso maintain lexical state during reconcile and omit both hooks.\n prepareLexical?(conn: Connection): Promise<void>;\n assertLexicalReady?(conn: Connection): Promise<void>;\n createStore(handle: Handle, conn: Connection, cfg: ResolvedConfig, options: { observational: boolean }): Store;\n}\n\nexport interface FieldStat {\n field: string;\n coverage: number;\n type: string;\n}\n\nexport interface DocumentStore {\n // Frontmatter column names (including internal ones; callers filter).\n columns(): Promise<string[]>;\n // Per-column coverage (non-null count) and observed type set, aggregated in one SQL query,\n // never one row per note. `scopeWhere` is a caller-built WHERE fragment, per LexicalQueryOptions.\n fieldStats(columns: string[], scopeWhere: string): Promise<FieldStat[]>;\n}\n\n// A ranked match, nothing more. Snippets are cut from the document above the store\n// (commands/search.ts), identically for every engine, so no store carries them.\nexport interface LexicalHit {\n path: string;\n}\n\nexport interface LexicalQueryOptions {\n whereJoin: string;\n whereCond: string;\n scopeCond: string;\n limit: number;\n}\n\nexport interface LexicalIndex {\n // Ranked word-match query with snippet, scoped by the caller-built SQL fragments (the same\n // fragments narrowByWhere/materializeScope produce elsewhere).\n query(terms: string, opts: LexicalQueryOptions): Promise<LexicalHit[]>;\n}\n\nexport interface VectorCandidate {\n path: string;\n lines: string;\n similarity: number;\n}\n\nexport interface VectorSimilar {\n path: string;\n similarity: number;\n}\n\nexport interface VectorWriteRow {\n path: string;\n chunk: number;\n scale: number;\n vector: Buffer;\n}\n\nexport interface VectorStore {\n // Rows whose vector is NULL: never embedded, or added since.\n pending(): Promise<Array<{ path: string; chunk: number }>>;\n // One batch write per call (never per row) so a provider's embedding batch stays inside a\n // single store method.\n writeVectors(rows: VectorWriteRow[]): Promise<void>;\n candidates(queryVector: Float32Array, storeDims: number, fetch: number, allowed?: Set<string>): Promise<VectorCandidate[]>;\n similar(path: string, opts: { exclude: Set<string>; allowed?: Set<string>; k: number }): Promise<VectorSimilar[]>;\n hasVector(path: string): Promise<boolean>;\n}\n\n// The `sense sql` passthrough: string in, streamed rows out. Each store registers the same\n// sense-supplied functions and applies its own read-bigints/error-translation behavior.\nexport interface RawStatement {\n columns(): Array<{ name: string }>;\n iterate(...params: unknown[]): AsyncIterable<unknown>;\n}\n\nexport interface SqlSession {\n prepare(sql: string): Promise<RawStatement>;\n}\n\nexport interface Store {\n readonly name: StoreName;\n readonly capabilities: ReadonlySet<Capability>;\n exec(sql: string): Promise<void>;\n prepare(sql: string): Promise<Statement>;\n // One crossing, N rows: every external write loop (search's candidate insert, graph ring's\n // temp-table writes) goes through this instead of looping over `run()`. Same contract as Connection.runBatch.\n runBatch(sql: string, paramRows: unknown[][]): Promise<void>;\n // Pins one snapshot across multi-statement reads; `map` and `peek` use it. A network-bound\n // write (search's embedding top-up) must stay outside it. Nesting joins the enclosing transaction.\n transaction<T>(fn: () => Promise<T>): Promise<T>;\n docs: DocumentStore;\n lexical: LexicalIndex;\n vectors: VectorStore;\n raw: SqlSession;\n // Engine-level facts for `sense status` (e.g. a derived busy_timeout PRAGMA reading); each\n // store owns what it reports and how it is worded. Empty when there is nothing to report.\n engineStatus(): Promise<Record<string, string>>;\n close(): Promise<void>;\n}\n"],"names":["CAPABILITY_NAMES"],"mappings":"AAAA,yFAAyF;AACzF,4FAA4F;AAM5F,iGAAiG;AACjG,+FAA+F;AAC/F,kGAAkG;AAClG,kGAAkG;AAClG,qGAAqG;AACrG,OAAO,MAAMA,mBAAmB;IAAC;IAAW;CAAU,CAAU"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sensemaking",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.28.1",
|
|
4
4
|
"description": "Query and search your markdown notes with context-aware progressive disclosure: SQL over frontmatter, links, and text, plus semantic search and link-graph ranking. No server, no build step.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"markdown",
|
|
@@ -68,11 +68,10 @@
|
|
|
68
68
|
},
|
|
69
69
|
"files": [
|
|
70
70
|
"dist",
|
|
71
|
-
"skills",
|
|
72
71
|
"schema.json"
|
|
73
72
|
],
|
|
74
73
|
"scripts": {
|
|
75
|
-
"benchmark": "tsds build && node benchmark/gate.
|
|
74
|
+
"benchmark": "tsds build && node benchmark/gate.ts",
|
|
76
75
|
"build": "tsds build",
|
|
77
76
|
"format": "tsds format",
|
|
78
77
|
"prepublishOnly": "tsds validate",
|
|
@@ -84,26 +83,41 @@
|
|
|
84
83
|
"dependencies": {
|
|
85
84
|
"@huggingface/tokenizers": "^0.2.0",
|
|
86
85
|
"franc-min": "^6.2.0",
|
|
87
|
-
"install-module-linked": "^2.
|
|
86
|
+
"install-module-linked": "^2.1.0",
|
|
88
87
|
"markdown-it": "^15.0.1",
|
|
89
88
|
"markdown-it-footnote": "^4.0.0",
|
|
90
89
|
"markdown-it-task-lists": "^2.1.1",
|
|
90
|
+
"semver": "^7.0.0",
|
|
91
91
|
"stemmer": "^2.0.1",
|
|
92
92
|
"tinypool": "^2.1.2",
|
|
93
93
|
"yaml": "^2.9.0"
|
|
94
94
|
},
|
|
95
95
|
"devDependencies": {
|
|
96
|
-
"@duckdb/node-api": "
|
|
97
|
-
"@tursodatabase/database": "
|
|
96
|
+
"@duckdb/node-api": "^1.5.6-r.1",
|
|
97
|
+
"@tursodatabase/database": "^0.8.2",
|
|
98
98
|
"@types/markdown-it-footnote": "^3.0.4",
|
|
99
99
|
"@types/mocha": "*",
|
|
100
|
-
"@types/node": "
|
|
100
|
+
"@types/node": "^22",
|
|
101
|
+
"@types/semver": "^7.8.0",
|
|
101
102
|
"cr": "^0.1.0",
|
|
102
103
|
"fs-remove-compat": "^1.0.4",
|
|
103
104
|
"node-version-use": "*",
|
|
105
|
+
"portable-env": "^1.4.0",
|
|
104
106
|
"ts-dev-stack": "*",
|
|
105
107
|
"tsds-config": "*"
|
|
106
108
|
},
|
|
109
|
+
"peerDependencies": {
|
|
110
|
+
"@duckdb/node-api": "^1.5.6-r.1",
|
|
111
|
+
"@tursodatabase/database": "^0.8.2"
|
|
112
|
+
},
|
|
113
|
+
"peerDependenciesMeta": {
|
|
114
|
+
"@duckdb/node-api": {
|
|
115
|
+
"optional": true
|
|
116
|
+
},
|
|
117
|
+
"@tursodatabase/database": {
|
|
118
|
+
"optional": true
|
|
119
|
+
}
|
|
120
|
+
},
|
|
107
121
|
"engines": {
|
|
108
122
|
"node": ">=22.20"
|
|
109
123
|
}
|
package/skills/sense/EXAMPLES.md
DELETED
|
@@ -1,99 +0,0 @@
|
|
|
1
|
-
# sense: worked examples
|
|
2
|
-
|
|
3
|
-
Outputs below are illustrative. Every result is a reference; reading happens afterward, through the filesystem, on the paths that earned it.
|
|
4
|
-
|
|
5
|
-
## A. "Do the notes say anything about X?"
|
|
6
|
-
|
|
7
|
-
```
|
|
8
|
-
sense search "pricing OR billing OR invoicing" --k 10 --format json
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
```json
|
|
12
|
-
[
|
|
13
|
-
{ "path": "notes/pricing-model.md", "title": "Pricing model",
|
|
14
|
-
"summary": "Tiered per-seat pricing; floor and discount rules", "snippets": ["«Pricing» floor for annual…"], "via": "match+link", "score": 0.0333, "lines": null },
|
|
15
|
-
{ "path": "notes/renewal-playbook.md", "title": "Renewal playbook",
|
|
16
|
-
"summary": "Renewal sequence and owners", "snippets": ["…«billing» contact confirms the PO…"], "via": "match", "score": 0.0313, "lines": "L81-140" }
|
|
17
|
-
]
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
Tens of tokens per row; often the `summary` answers the question with no read at all. A row with `via: "link"` has link evidence rather than a word match. `lines`, when set, is a section or embedding chunk's line range, a direct `Read` range. See [Sense terminology](references/terminology.md) for these distinctions.
|
|
21
|
-
|
|
22
|
-
## B. Known-field filtering (no search)
|
|
23
|
-
|
|
24
|
-
```
|
|
25
|
-
sense sql "SELECT path, title, status FROM frontmatter WHERE status = 'active' AND has(tags, ?)" pricing --format json
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
Plain SQL on discovered columns. Combine with search by joining `content` and adding `AND content MATCH ?`: filter and rank in one query.
|
|
29
|
-
|
|
30
|
-
Per-member counts on an array field (GROUP BY on the raw column would split `["a","b"]` from `["b","a"]`):
|
|
31
|
-
|
|
32
|
-
```
|
|
33
|
-
sense sql "SELECT j.value AS tag, COUNT(*) n FROM frontmatter, json_each(frontmatter.tags) j GROUP BY j.value ORDER BY n DESC"
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
## C. Structure before reading
|
|
37
|
-
|
|
38
|
-
```
|
|
39
|
-
sense peek notes/architecture-decisions.md
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
```
|
|
43
|
-
notes/architecture-decisions.md (~4400 tokens)
|
|
44
|
-
title: Architecture decisions
|
|
45
|
-
sections:
|
|
46
|
-
## Storage layer: why SQLite [L41-88, ~610t]
|
|
47
|
-
## Queue: rejected options [L89-120, ~380t]
|
|
48
|
-
...
|
|
49
|
-
links out (7): notes/pricing-model.md, ...
|
|
50
|
-
backlinks (2): notes/_index.md, notes/roadmap.md
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
The note is ~4,400 tokens; the peek is ~500. If only one section matters, `Read` its line range (~400 tokens), a tenth of the file.
|
|
54
|
-
|
|
55
|
-
## D. The graph
|
|
56
|
-
|
|
57
|
-
```
|
|
58
|
-
sense sql "SELECT src FROM links WHERE dst = ?" notes/pricing-model.md # who cites this
|
|
59
|
-
sense sql "SELECT src, target FROM links WHERE dst IS NULL" # unresolved links
|
|
60
|
-
sense sql "SELECT path, round(_rank*100,2) r FROM frontmatter ORDER BY _rank DESC LIMIT 5" # load-bearing notes
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
`sense map` prints the hub list without SQL; use these when you need it filtered or joined.
|
|
64
|
-
|
|
65
|
-
## E. Cold start on an unknown notes tree
|
|
66
|
-
|
|
67
|
-
```
|
|
68
|
-
sense map # fields, hubs, recent. Read this first
|
|
69
|
-
sense --list # named queries someone already saved
|
|
70
|
-
sense sql "SELECT DISTINCT type FROM frontmatter" # what a field's values are
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
## F. "The notes say it in different words"
|
|
74
|
-
|
|
75
|
-
```
|
|
76
|
-
sense search "children dying from poor nutrition" --k 3 --format json
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
```json
|
|
80
|
-
[
|
|
81
|
-
{ "path": "notes/malnutrition-outcomes.md", "title": "Malnutrition outcomes",
|
|
82
|
-
"summary": "Stunting and mortality by region", "snippets": [], "via": "vector", "score": 0.0167, "similarity": 0.61, "lines": "L14-52" }
|
|
83
|
-
]
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
A `via: "vector"` row has vector evidence rather than a word match. `similarity` is the cosine against the chunk identified by `lines`, a direct `Read` range. Vector search requires a configured model, enabled vector signal and prepared embeddings. Vector-only results do not prove the query words are absent; use the word-search guidance when absence matters.
|
|
87
|
-
|
|
88
|
-
## Consequences
|
|
89
|
-
|
|
90
|
-
| query | what happens | bounded alternative |
|
|
91
|
-
|---|---|---|
|
|
92
|
-
| `SELECT text FROM content` | returns the tree's entire prose | `snippet(content, -1, '«', '»', '…', 10)` excerpts the match |
|
|
93
|
-
| `snippet()` on a tree holding a megabyte-scale note | re-tokenizes the whole document per matched row: seconds per query | bound it: `CASE WHEN length(content.text) <= 16384 THEN snippet(...) END`, or select `summary` instead |
|
|
94
|
-
| `sense search "pricing"` | lexically matches only that word's stem (vector rows still widen by meaning) | OR-in synonyms and instances: `"pricing OR billing OR invoicing"` |
|
|
95
|
-
| `sense search "pricing model details"` | bare words AND-join; one absent word = zero lexical rows | OR the words, or quote an exact phrase |
|
|
96
|
-
| `sense search "customer-facing OR on-site"` | bare punctuation is FTS5 syntax (`-` reads as a column filter) | double-quote: `"customer-facing" OR "on-site"` |
|
|
97
|
-
| `Read` of a large file for one section | costs the whole file | `peek`, then `Read` the line range |
|
|
98
|
-
| row queries without `LIMIT` | unbounded output (aggregates are already bounded) | `LIMIT n` |
|
|
99
|
-
| saving one-off queries to config | config churn | ad-hoc `sense sql`; save reusable views |
|
package/skills/sense/SKILL.md
DELETED
|
@@ -1,145 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: sense
|
|
3
|
-
description: "Query a markdown tree with the sense CLI: filter notes by frontmatter, search prose, follow wikilinks and backlinks, trace link paths, find similar but unlinked notes, and inspect note outlines. Use when a task needs to query, filter, count, search, or report on a markdown directory; when a directory has sense.config.json; or when adding a saved sense query."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# sense
|
|
7
|
-
|
|
8
|
-
Use sense to locate evidence in a markdown tree before reading files. With the config default `"build": true`, CLI queries incrementally scan the configured tree for the capabilities they need; vector preparation is limited to the eligible scope. Config `"build": false` or a one-query `--no-build` reads the last completed indexed generation, checks requested readiness and does not scan live files. Explicit build and watch prepare it regardless of that default. Results contain paths, metadata, snippets, and line ranges. Read the returned files or ranges when the task needs their prose.
|
|
9
|
-
|
|
10
|
-
Setup, store selection, presets, and note design belong to the `sense-setup` skill. Translating an Obsidian Bases file belongs to `sense-bases`.
|
|
11
|
-
|
|
12
|
-
Use the shared [Sense terminology](references/terminology.md) when explaining results or output controls. Read it when distinguishing a snippet, section, section description, outline, chunk or link type; those names are not interchangeable.
|
|
13
|
-
|
|
14
|
-
## Start with the tree
|
|
15
|
-
|
|
16
|
-
Run these when the tree or its configuration is unfamiliar:
|
|
17
|
-
|
|
18
|
-
```sh
|
|
19
|
-
sense status # config, store, cache, document count, preset coverage
|
|
20
|
-
sense map # fields, hubs, recent notes
|
|
21
|
-
sense --list # saved queries
|
|
22
|
-
```
|
|
23
|
-
|
|
24
|
-
If a command reports a missing config, run `sense init` only when the user wants the tree configured. A one-off query does not authorize changing the tree.
|
|
25
|
-
|
|
26
|
-
## Choose the command
|
|
27
|
-
|
|
28
|
-
| Need | Command |
|
|
29
|
-
|---|---|
|
|
30
|
-
| Exact count, filter, grouping, or known-field report | `sense sql` or a saved SQL query |
|
|
31
|
-
| Notes about a subject | `sense search` |
|
|
32
|
-
| One note's frontmatter, outline, links, and backlinks | `sense peek` |
|
|
33
|
-
| Shortest link chain between two notes | `sense path` |
|
|
34
|
-
| Similar notes that a note does not link to | `sense related` |
|
|
35
|
-
| Tree shape and available fields | `sense map` |
|
|
36
|
-
|
|
37
|
-
Start with a bounded result. Raise `--k`, widen the preset, or broaden the words only when the first result does not answer the question.
|
|
38
|
-
|
|
39
|
-
```sh
|
|
40
|
-
sense search "pricing" --k 10
|
|
41
|
-
sense search "sourcing quotes" --preset raw
|
|
42
|
-
sense peek notes/pricing-model.md
|
|
43
|
-
sense path onboarding.md pricing-model.md
|
|
44
|
-
sense related notes/pricing-model.md --k 10
|
|
45
|
-
sense sql "SELECT path FROM frontmatter WHERE status = ? LIMIT 50" active
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
## Read the store guide before composing syntax
|
|
49
|
-
|
|
50
|
-
The config's `store` key selects the SQL dialect and text-search grammar. An omitted key means `sqlite`. Bare words and quoted phrases work in `sense search` on every store. Advanced operators and raw text-index SQL differ.
|
|
51
|
-
|
|
52
|
-
Read the matching guide before writing raw SQL that uses engine functions, dates, JSON, text matching, or native types. Also read it before using advanced search operators.
|
|
53
|
-
|
|
54
|
-
- [SQLite query guide](references/stores/sqlite.md)
|
|
55
|
-
- [DuckDB query guide](references/stores/duckdb.md)
|
|
56
|
-
- [Turso query guide](references/stores/turso.md)
|
|
57
|
-
|
|
58
|
-
The tables, `?` placeholders, quoted identifiers, `has()`, `basename()`, and preset `scope` binding are shared. [Portable SQL](references/sql.md) documents the schema and queries that work without depending on one engine.
|
|
59
|
-
|
|
60
|
-
## Search and evidence
|
|
61
|
-
|
|
62
|
-
`search` combines the signals enabled by the selected preset:
|
|
63
|
-
|
|
64
|
-
| Signal | Evidence |
|
|
65
|
-
|---|---|
|
|
66
|
-
| `match` | The note contains the search words. `snippets` shows the matching passages. |
|
|
67
|
-
| `link` | A note that matched links to this note. |
|
|
68
|
-
| `vector` | The embedding model placed this note near the query. The search words may be absent. |
|
|
69
|
-
|
|
70
|
-
Combinations such as `match+link` mean that more than one signal produced the row. `score` ranks rows within that result only. Do not compare it across searches. `similarity` ranks vector evidence within the current result and model. Do not carry a fixed similarity cutoff between trees.
|
|
71
|
-
|
|
72
|
-
The `lines` value gives a line range in the indexed note. It can identify a section around a word match or an embedding chunk. Read that range when it is present. A null range means the whole note is the reference. A vector-only row has no lexical snippet and is a lead, not proof that the note contains the query terms.
|
|
73
|
-
|
|
74
|
-
Read [search evidence and troubleshooting](references/search.md) when a search will support a factual claim, when absence matters, when results look noisy, or when tuning signal weights.
|
|
75
|
-
|
|
76
|
-
`sense search --explain --format json` adds the actual ranking contributions for returned notes. It does not change ranking or explain every missing result. The search guide describes how contributions differ from the `via` evidence label.
|
|
77
|
-
|
|
78
|
-
## Scope and output
|
|
79
|
-
|
|
80
|
-
Bare commands use the `default` preset. `--preset <name>` chooses another. For `search`, `map`, `peek`, `path`, and `related`, `--include`, `--exclude`, and `--no-exclude` change the query scope for one invocation, but cannot reach files that no preset indexes. `sense status` shows actual coverage. Scope selects results; presets are not a security boundary.
|
|
81
|
-
|
|
82
|
-
`peek` rejects an exact path outside scope before trying a basename, resolves basenames within scope, and shows only scoped resolved links and backlinks; unresolved targets remain as written. `path` resolves its endpoints across the indexed tree and scopes intermediate notes. `related` resolves its seed across the indexed tree and scopes result candidates.
|
|
83
|
-
|
|
84
|
-
`peek --section-count-limit n` bounds section descriptions. `--link-count-limit n` independently bounds each group of outbound links, backlinks and unresolved links. Both default to 20 and require positive safe integers. Request more when the outline or links you need are truncated, or fewer for triage. Totals still describe the full selected groups; the limits do not widen scope. Section descriptions identify where to read; they do not return section prose.
|
|
85
|
-
|
|
86
|
-
`--where` filters search and graph commands against frontmatter alias `f`:
|
|
87
|
-
|
|
88
|
-
```sh
|
|
89
|
-
sense search "pricing" --where "f.status = 'active' AND has(f.tags, 'sales')"
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
`sense sql` is index-wide by default. With `--preset`, the command binds a temporary `scope(path)` table. The SQL must join it:
|
|
93
|
-
|
|
94
|
-
```sh
|
|
95
|
-
sense sql "SELECT f.path FROM frontmatter f JOIN scope ON scope.path = f.path" --preset default
|
|
96
|
-
```
|
|
97
|
-
|
|
98
|
-
Table output is for people. Use `--format json` when code or an agent will parse rows. Use `--format csv` when redirecting a large row set to a file. `sql`, `search`, `related`, and saved queries support all three formats. `map`, `peek`, `status`, and `path` support table and JSON.
|
|
99
|
-
|
|
100
|
-
## Saved queries
|
|
101
|
-
|
|
102
|
-
Save a query in `sense.config.json` only when it will be reused:
|
|
103
|
-
|
|
104
|
-
```json
|
|
105
|
-
{
|
|
106
|
-
"queries": {
|
|
107
|
-
"by-tag": { "sql": "SELECT path, title FROM frontmatter WHERE has(tags, ?) ORDER BY path" },
|
|
108
|
-
"hot": { "search": "pricing", "preset": "raw", "k": 20 }
|
|
109
|
-
}
|
|
110
|
-
}
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
Run these as `sense by-tag urgent` and `sense hot`. Invocation flags override a saved search's `preset`, `k`, or `where` value.
|
|
114
|
-
|
|
115
|
-
Running a saved entry validates it. Exit code 0 means it ran, 2 means the invocation needs different arguments, and 1 means the query or store failed. An empty result can be valid data, so interpret it from the query's purpose.
|
|
116
|
-
|
|
117
|
-
## Tables
|
|
118
|
-
|
|
119
|
-
| Table | Holds |
|
|
120
|
-
|---|---|
|
|
121
|
-
| `frontmatter` | One discovered column per frontmatter key, plus `path`, `_mtime`, `_ctime`, `_size`, `_rank`, and `_parse_error` |
|
|
122
|
-
| `content` | `path`, `title`, `summary`, and authored text, plus store-owned search columns |
|
|
123
|
-
| `links` | `src`, written `target`, resolved `dst`, and `embed` |
|
|
124
|
-
| `tags` | Merged and deduplicated frontmatter and inline tags |
|
|
125
|
-
| `sections` | Heading, level, line range, and token estimate |
|
|
126
|
-
| `preset_files` | Paths covered by each preset |
|
|
127
|
-
|
|
128
|
-
Features can add tables. `sense map` and `sense status` show which features are active.
|
|
129
|
-
|
|
130
|
-
A non-null `_parse_error` means the file has no recovered frontmatter values. To distinguish a missing field from invalid frontmatter, include `_parse_error IS NULL` in the filter. Fixes appear on the next command because reconciliation runs first.
|
|
131
|
-
|
|
132
|
-
## Reading discipline
|
|
133
|
-
|
|
134
|
-
Select only the columns needed for the answer. Use `LIMIT` for row-returning exploration. Prefer `path`, `title`, `summary`, and bounded snippets over `content.text`. Aggregates such as `COUNT` and `GROUP BY` are already bounded by their result shape.
|
|
135
|
-
|
|
136
|
-
When a result identifies a large note, use `peek` and then read the relevant line range. Small files are often cheaper to read whole.
|
|
137
|
-
|
|
138
|
-
Worked command traces are in [EXAMPLES.md](EXAMPLES.md).
|
|
139
|
-
|
|
140
|
-
## Upkeep
|
|
141
|
-
|
|
142
|
-
- Install a missing CLI with `npm install -g sensemaking`.
|
|
143
|
-
- `sense status` prints the cache path and watcher state.
|
|
144
|
-
- Use `sense build --force` to recreate a derived index whose state is in doubt; source files, configuration, and the watcher ownership claim remain in place.
|
|
145
|
-
- Use `sense watch` when another process should keep all configured capabilities prepared during frequent edits. A default query scans the configured tree for its needed capabilities and limits vector preparation to eligible paths; `--no-build` reads the completed generation only.
|
|
@@ -1,63 +0,0 @@
|
|
|
1
|
-
# Search evidence and troubleshooting
|
|
2
|
-
|
|
3
|
-
Read this guide when search results will support a factual claim, when absence matters, when results look noisy, or when changing signal weights.
|
|
4
|
-
|
|
5
|
-
[Sense terminology](terminology.md) defines the result objects. In particular, snippets, sections and embedding chunks are different units.
|
|
6
|
-
|
|
7
|
-
## What each signal establishes
|
|
8
|
-
|
|
9
|
-
`words` ranks literal occurrences after stemming. It is the right signal for identifiers, error strings, names, and quoted phrases. A `match` row is lexical evidence because its `snippets` show the occurrence.
|
|
10
|
-
|
|
11
|
-
`links` expands from matching notes through links written by the authors. A `link` row establishes that relationship, but does not establish that the row contains the query words.
|
|
12
|
-
|
|
13
|
-
`vectors` ranks the meaning of chunks. It can find paraphrases and related concepts with no shared vocabulary. A `vector` row is a lead to read. It cannot prove that a word or claim appears in the note.
|
|
14
|
-
|
|
15
|
-
Search uses every signal named by the preset. If `signals` is absent, every signal whose prerequisites hold has weight 1. A number changes that signal's contribution to reciprocal-rank fusion:
|
|
16
|
-
|
|
17
|
-
```json
|
|
18
|
-
"signals": { "words": 1, "links": 1, "vectors": 4 }
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
Weights are corpus and model choices. Compare representative queries before changing them. One weight does not transfer reliably between unrelated trees or embedding models.
|
|
22
|
-
|
|
23
|
-
## Reading the rows
|
|
24
|
-
|
|
25
|
-
- `via` names the evidence that produced the row.
|
|
26
|
-
- `snippets` contains marked lexical passages. It is empty when no word matched.
|
|
27
|
-
- `lines` is the selected section or embedding chunk's line range in the indexed note. Null means the whole note is the reference.
|
|
28
|
-
- `score` orders the fused result. Its scale changes with the participating signals and ranks, so compare rows only inside one result.
|
|
29
|
-
- `similarity` is cosine similarity against the best chunk. Compare it inside one result and one model. Small trees can give unrelated text a moderately close nearest neighbor.
|
|
30
|
-
|
|
31
|
-
Relay the evidence label when reporting a result. "The note contains these words" and "the note is semantically related" support different claims.
|
|
32
|
-
|
|
33
|
-
## Finite retrieval and paging
|
|
34
|
-
|
|
35
|
-
`k` is the maximum number of notes returned by one finite retrieval. Changing `k` can change the candidate pools, link seeds and fused ranking, including the earlier rows. It starts a new retrieval rather than continuing a stable result set. When you need pages, retain one returned array or JSON result and page that locally.
|
|
36
|
-
|
|
37
|
-
## Explain a ranking
|
|
38
|
-
|
|
39
|
-
Add `--explain` to a direct `sense search` invocation when you need to inspect how the configured signals contributed. With `--format json`, each returned row adds an `explanation` array containing `signal`, `rank`, `weight` and `contribution`. Entries follow words, links, vectors order and include only signals that contributed to that candidate. Without the flag, ordinary output is unchanged.
|
|
40
|
-
|
|
41
|
-
Ranks are one-based within each signal's candidate list. A contribution is `weight / (60 + rank - 1)`, using the actual unrounded value accumulated during ranking. The sum is the fused score before its public display rounding. These values are not native BM25 scores and do not make lexical rankings comparable across stores.
|
|
42
|
-
|
|
43
|
-
An isolated word-match seed can receive link-signal restart mass while its `via` label remains `match`. The explanation reports the numeric contribution; `via` remains the evidence label, not an exhaustive score breakdown. An explanation does not establish that a claim is true or describe why an absent note failed to enter the candidate set.
|
|
44
|
-
|
|
45
|
-
Table and CSV formats encode the explanation cell as JSON text. Table output follows the existing cell clipping rules; use JSON or CSV for the complete values. Explanation reuses the current search and does not make an additional embedding request. It is not a saved-query configuration field.
|
|
46
|
-
|
|
47
|
-
## When a search misses
|
|
48
|
-
|
|
49
|
-
For word search, try concrete terms that the notes may use, widen the selected preset, or raise `--k`. Search grammar depends on the selected store. Read its guide before adding operators.
|
|
50
|
-
|
|
51
|
-
For vector search, restate the concept in different words. Then use `related <path>` on the nearest useful hit to inspect its semantic neighborhood.
|
|
52
|
-
|
|
53
|
-
Each widening step adds candidates and can dilute the ranking. Inspect the new rows before widening again.
|
|
54
|
-
|
|
55
|
-
## Absence
|
|
56
|
-
|
|
57
|
-
A words-only preset returns no rows when the terms do not occur in its indexed scope. With vectors enabled, nearest-neighbor search can still return vector-only rows for any input. Those rows show conceptual proximity, not lexical presence.
|
|
58
|
-
|
|
59
|
-
When absence matters, use a preset whose `signals` excludes `vectors`, or query the selected store's text index as described in its guide. Confirm the preset's coverage with `sense status` before concluding that the terms are absent from the tree.
|
|
60
|
-
|
|
61
|
-
## Template-heavy trees
|
|
62
|
-
|
|
63
|
-
Vectors rank whole chunks, including repeated boilerplate. In a tree whose notes share a large template and contain little unique prose, unrelated seeds can return the same neighbors with almost identical similarities. Compare two unlike seed notes. If their neighbor lists barely change, search the fields or words that distinguish the notes instead.
|
|
@@ -1,120 +0,0 @@
|
|
|
1
|
-
# Portable SQL
|
|
2
|
-
|
|
3
|
-
Read this guide for raw SQL over sense tables. These patterns use the shared schema and avoid native text-index syntax. Read the selected store's query guide as well when the statement uses dates, JSON table functions, native types, or text matching.
|
|
4
|
-
|
|
5
|
-
## Discover the tree before assuming fields
|
|
6
|
-
|
|
7
|
-
Frontmatter columns come from the indexed notes. Inspect them before writing a field query:
|
|
8
|
-
|
|
9
|
-
```sh
|
|
10
|
-
sense map
|
|
11
|
-
sense sql "SELECT name FROM pragma_table_info('frontmatter')"
|
|
12
|
-
sense sql "SELECT DISTINCT status FROM frontmatter ORDER BY status"
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Punctuated field names need double quotes. Values should use `?` parameters:
|
|
16
|
-
|
|
17
|
-
```sh
|
|
18
|
-
sense sql 'SELECT path FROM frontmatter WHERE "plugin-id" = ? LIMIT 50' example
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
`content.title` and `content.summary` always exist. A frontmatter column with either name exists only when at least one indexed note declares it.
|
|
22
|
-
|
|
23
|
-
## Shared tables
|
|
24
|
-
|
|
25
|
-
| Table | Main columns |
|
|
26
|
-
|---|---|
|
|
27
|
-
| `frontmatter` | `path`, discovered fields, `_mtime`, `_ctime`, `_size`, `_rank`, `_parse_error` |
|
|
28
|
-
| `content` | `path`, `title`, `summary`, `text` |
|
|
29
|
-
| `links` | `src`, `target`, `dst`, `embed` |
|
|
30
|
-
| `tags` | `path`, `tag` |
|
|
31
|
-
| `sections` | `path`, `heading`, `level`, `start_line`, `end_line`, `tokens` |
|
|
32
|
-
| `preset_files` | `path`, `preset` |
|
|
33
|
-
|
|
34
|
-
`links.dst` is null when the written target does not resolve to indexed markdown. `links.embed` is 1 for an embed and 0 for an ordinary wikilink. The `tags` table merges frontmatter and inline tags and stores a nested tag such as `book/scifi` in full.
|
|
35
|
-
|
|
36
|
-
## Shared query patterns
|
|
37
|
-
|
|
38
|
-
```sql
|
|
39
|
-
SELECT COUNT(*) AS notes FROM frontmatter;
|
|
40
|
-
|
|
41
|
-
SELECT path, title
|
|
42
|
-
FROM frontmatter
|
|
43
|
-
WHERE status = ?
|
|
44
|
-
ORDER BY path
|
|
45
|
-
LIMIT 50;
|
|
46
|
-
|
|
47
|
-
SELECT src
|
|
48
|
-
FROM links
|
|
49
|
-
WHERE dst = ?
|
|
50
|
-
ORDER BY src;
|
|
51
|
-
|
|
52
|
-
SELECT heading, start_line, end_line, tokens
|
|
53
|
-
FROM sections
|
|
54
|
-
WHERE path = ?
|
|
55
|
-
ORDER BY start_line;
|
|
56
|
-
|
|
57
|
-
SELECT path, tag
|
|
58
|
-
FROM tags
|
|
59
|
-
WHERE tag = ? OR tag LIKE ?
|
|
60
|
-
ORDER BY path;
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
For a nested tag family, bind the same value twice as `book` and `book/%`.
|
|
64
|
-
|
|
65
|
-
`has(field, value)` works on every store. It tests membership for a JSON array and substring presence for a scalar string. Use `field = ?` for exact scalar equality because `has(status, 'active')` also matches `inactive`.
|
|
66
|
-
|
|
67
|
-
`basename(path[, suffix])` works on every store. `segment(terms)` is available on SQLite and DuckDB only; the store guides explain when it is needed.
|
|
68
|
-
|
|
69
|
-
## Preset scope
|
|
70
|
-
|
|
71
|
-
`sense sql` covers the whole index unless it receives `--preset`. With that flag, sense binds a temporary `scope(path)` table and requires the statement to join it:
|
|
72
|
-
|
|
73
|
-
```sh
|
|
74
|
-
sense sql "SELECT f.path FROM frontmatter f JOIN scope ON scope.path = f.path ORDER BY f.path" --preset default
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
A saved SQL query written against `scope` can run under different presets without duplicating the statement. Without `--preset`, join `preset_files` directly and name the preset in SQL.
|
|
78
|
-
|
|
79
|
-
## Graph queries
|
|
80
|
-
|
|
81
|
-
Bound recursive walks. An unrestricted walk on a dense graph can enumerate paths faster than it eliminates them.
|
|
82
|
-
|
|
83
|
-
```sql
|
|
84
|
-
WITH RECURSIVE hop(path, d) AS (
|
|
85
|
-
SELECT ?, 0
|
|
86
|
-
UNION
|
|
87
|
-
SELECT CASE WHEN l.src = hop.path THEN l.dst ELSE l.src END, hop.d + 1
|
|
88
|
-
FROM hop
|
|
89
|
-
JOIN links l ON (l.src = hop.path OR l.dst = hop.path) AND l.dst IS NOT NULL
|
|
90
|
-
WHERE hop.d < 2
|
|
91
|
-
)
|
|
92
|
-
SELECT DISTINCT path FROM hop WHERE d > 0;
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
Use `sense path` for the shortest chain between two known notes. Use raw recursion when the task needs a set of neighbors to filter or join.
|
|
96
|
-
|
|
97
|
-
## Unresolved links
|
|
98
|
-
|
|
99
|
-
`dst IS NULL` includes links to attachments that sense never indexes, such as images, PDFs, and `.base` files. Exclude the attachment extensions used by the tree before treating the remaining rows as broken links. Trees with dotted markdown titles need an explicit extension list instead of a blanket "contains a dot" filter.
|
|
100
|
-
|
|
101
|
-
Templates and examples can also contain deliberately unresolved wikilinks. Scope the query to authored content when those files are indexed.
|
|
102
|
-
|
|
103
|
-
## Types and parse errors
|
|
104
|
-
|
|
105
|
-
YAML strings remain text, whole numbers and booleans remain integer-like values, fractions remain real-like values, and lists and maps remain structured or JSON-backed values according to the store. Use `sense map` to see the types observed for each field. Read the store guide before comparing a field that contains more than one type.
|
|
106
|
-
|
|
107
|
-
A malformed frontmatter block produces `_parse_error` and no recovered field values. This query lists the files to fix:
|
|
108
|
-
|
|
109
|
-
```sql
|
|
110
|
-
SELECT path, _parse_error
|
|
111
|
-
FROM frontmatter
|
|
112
|
-
WHERE _parse_error IS NOT NULL
|
|
113
|
-
ORDER BY path;
|
|
114
|
-
```
|
|
115
|
-
|
|
116
|
-
To find notes that genuinely omit `status`, use `status IS NULL AND _parse_error IS NULL`.
|
|
117
|
-
|
|
118
|
-
## Bound the output
|
|
119
|
-
|
|
120
|
-
Select only the columns required by the task and add `LIMIT` while exploring. Avoid `SELECT text FROM content` because it returns the tree's prose. Use `search` for bounded passages, or select `path`, `title`, and `summary` and read the relevant files afterward.
|
|
@@ -1,35 +0,0 @@
|
|
|
1
|
-
# DuckDB query guide
|
|
2
|
-
|
|
3
|
-
Read this guide when `store` is `duckdb`.
|
|
4
|
-
|
|
5
|
-
Sense stores the cache at `.sense/cache.duckdb`. It is a DuckDB database file with frontmatter values in `VARIANT` columns and embeddings in native fixed-width float arrays. Use this store when another DuckDB tool needs to inspect or analyze the cache.
|
|
6
|
-
|
|
7
|
-
The first open installs `@duckdb/node-api` if needed. Its native download is about 110 MB. Under DuckDB's [concurrency model](https://duckdb.org/docs/current/connect/concurrency), Sense opens an existing cache for no-build observation in read-only mode, so several independent reader processes can coexist. A build-enabled command or watcher opens it read-write and excludes every other process until it closes. Contenders wait within Sense's bounded lock budget, then fail with `STORE_BUSY`. This cross-process rule does not promise writer/reader coexistence or unrestricted sharing by independently configured DuckDB instances in one process.
|
|
8
|
-
|
|
9
|
-
## `sense search` grammar
|
|
10
|
-
|
|
11
|
-
Bare words and quoted phrases work. Bare words are joined with AND, and phrases are verified against authored text.
|
|
12
|
-
|
|
13
|
-
The portable search command does not accept FTS5 prefix, boolean, `NEAR`, initial-token, column-filter, or grouping operators. Examples include `foo*`, `a OR b`, `NEAR(a b)`, `^term`, `title:term`, and `(a b)`. Sense raises `STORE_CAPABILITY_MISSING` instead of interpreting them as literals.
|
|
14
|
-
|
|
15
|
-
To express alternatives, run separate bounded searches and combine the paths, or use a frontmatter filter when the distinction is structured.
|
|
16
|
-
|
|
17
|
-
## Raw text search
|
|
18
|
-
|
|
19
|
-
`content` is a plain table. SQLite's `content MATCH`, `snippet()`, and `bm25()` do not run here. Use `sense search` for ranked text retrieval and combine it with `--where` for frontmatter filters.
|
|
20
|
-
|
|
21
|
-
DuckDB maintains its own native FTS index for the search command. Its generated index name and scoring adapter are implementation details. Saved queries should not call them.
|
|
22
|
-
|
|
23
|
-
## Functions and types
|
|
24
|
-
|
|
25
|
-
`has()`, `basename()`, and `segment()` are registered SQL functions. `segment()` is available for portable saved queries, although `sense search` already handles unspaced scripts.
|
|
26
|
-
|
|
27
|
-
Frontmatter columns use DuckDB `VARIANT`. Homogeneous values compare normally. A numeric comparison against a field that contains numbers in some notes and text in others can raise a type error. Check `sense map` for mixed observed types before writing the predicate.
|
|
28
|
-
|
|
29
|
-
DuckDB has its own date, JSON, list, and casting syntax. Use DuckDB SQL rather than copying SQLite functions into a saved query. For ISO 8601 timestamps with offsets, cast to `TIMESTAMPTZ` before comparing:
|
|
30
|
-
|
|
31
|
-
```sql
|
|
32
|
-
WHERE created::TIMESTAMPTZ >= ?::TIMESTAMPTZ
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
DuckDB's dialect closely follows PostgreSQL but has documented differences and its own extensions. Read the official [DuckDB SQL introduction](https://duckdb.org/docs/current/sql/introduction) for engine SQL. Queries written with DuckDB-only syntax stay tied to this store.
|