@deepagents/text2sql 7.3.0 → 7.5.0

package/README.md

@@ -4,12 +4,12 @@ AI-powered natural language to SQL. Ask questions in plain English, get executab
 
 ## Features
 
-- **Natural Language to SQL** - Convert questions to validated, executable queries
-- **Multi-Database Support** - PostgreSQL, SQLite, DuckDB, SQL Server, MySQL/MariaDB, BigQuery, ClickHouse, and PostHog HogQL adapters
-- **Schema-Aware** - Automatic introspection of tables, relationships, indexes, and constraints
-- **Domain Knowledge** - Inject business terms, guardrails, and query patterns via fragments
+- **Natural language to SQL** - Convert questions to validated, executable queries
+- **Database adapters** - PostgreSQL, SQLite, DuckDB, SQL Server, MySQL/MariaDB, BigQuery, ClickHouse, and PostHog HogQL adapters
+- **Schema introspection** - Automatic introspection of tables, relationships, indexes, and constraints
+- **Domain knowledge** - Inject business terms, guardrails, and query patterns via fragments
 - **Conversational** - Multi-turn conversations with context persistence
-- **Safe by Default** - Read-only queries, validation, and configurable guardrails
+- **Safe by default** - Read-only queries, validation, and configurable guardrails
 
 ## Installation
 
@@ -29,9 +29,24 @@ npm install @clickhouse/client       # ClickHouse, or use your preferred client
 # PostHog uses native fetch and needs no additional client package
 ```
 
-Requires Node.js LTS
+The spreadsheet adapter requires `xlsx` for Excel, CSV, and TSV files. It is
+an optional peer dependency; install it only if you use
+`@deepagents/text2sql/spreadsheet`. With npm 12, add this to your project's
+`.npmrc` to allow directly declared URL dependencies:
 
-## Quick Start
+```ini
+allow-remote=root
+```
+
+Then install SheetJS `xlsx` 0.20.3 from its CDN:
+
+```bash
+npm install https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz
+```
+
+Requires Node.js LTS.
+
+## Quick start
 
 ```typescript
 import { groq } from '@ai-sdk/groq';
@@ -96,11 +111,11 @@ maps dynamically, use `isValidAdapterName(name)` to check one key or
 `validateAdapterNames(names)` to fail fast before constructing `Text2Sql` or
 the sandbox-side adapter module used by the `sql` CLI.
 
-## AI Model Providers
+## AI model providers
 
 Text2SQL works with any model provider supported by the [Vercel AI SDK](https://sdk.vercel.ai/docs), including OpenAI, Anthropic, Google, Groq, and more.
 
-## Building a Conversational Agent
+## Building a conversational agent
 
 `Text2Sql` owns SQL generation and schema indexing, but you own the chat loop.
 Build the agent by composing a `ContextEngine`, a sandbox, and `agent` +
@@ -144,6 +159,7 @@ const backend = await createDockerSandbox({
   ],
   env: {
     TEXT2SQL_ADAPTERS: '/workspace/text2sql-adapters.ts',
+    TEXT2SQL_OUT_DIR: '/tmp/text2sql',
   },
 });
 const sandbox = await createBashTool({
@@ -179,7 +195,12 @@ for await (const chunk of stream) {
 
 The `/workspace/text2sql-adapters.ts` module must exist in the sandbox and
 default-export your adapter map. Mount your project into `/workspace` (as above)
-or upload/write that module before you call `sql index` or `chat()`.
+or upload or write that module before calling `sql index`. With a remote Docker
+daemon, the bind mount's `hostPath` must exist on the daemon's host.
+
+`TEXT2SQL_OUT_DIR` keeps schema and query artifacts in the sandbox's writable
+`/tmp/text2sql` directory. The default `./sql` would be inside the read-only
+`/workspace` mount.
 
 `instructions()` returns the SQL-flavored system fragments (policies, workflows,
 clarifications, style guides) — spread them into `context.set()` alongside the
@@ -197,7 +218,7 @@ turns, the reminder is persisted as a synthetic user message for replay and
 prompt caching; bare `createPrepareStep()` integrations and
 `agent.generate()` own persistence of their generated assistant history.
 
-## Advanced: SQL CLI in Sandboxes
+## Advanced: SQL CLI in sandboxes
 
 `sql validate <db> "..."` and `sql run <db> "..."` are real commands from the
 `@deepagents/text2sql` package. `<db>` is the configured adapter name. With a
@@ -228,11 +249,13 @@ changes). With neither set, every `sql index` introspects fresh. See
 [Schema index caching](#schema-index-caching) for
 the underlying injectable primitives.
 
-For in-process or virtual-sandbox usage (without installing the package CLI),
-wrap an existing `Text2Sql` instance as a just-bash custom command:
+For in-process or virtual-sandbox usage, install `just-bash` and wrap an existing
+`Text2Sql` instance as a custom command:
 
 ```typescript
-import { InMemoryFs, createVirtualSandbox } from '@deepagents/context';
+import { InMemoryFs } from 'just-bash';
+
+import { createVirtualSandbox } from '@deepagents/context';
 import {
   type CreateSqlCommandOptions,
   type CreateSqlCommandResult,
@@ -396,6 +419,26 @@ for await (const chunk of followUp) {
 }
 ```
 
+## Checkpoint files
+
+`Checkpoint` from `@deepagents/text2sql` saves completed computations so a
+pipeline can resume without repeating them. Load a file with
+`Checkpoint.load({ path, configHash, clock: globalThis })`, then use `run()` for
+one computation or `each()` for a sequence of inputs. Both require a codec to
+encode outputs and validate them on resume; `jsonCodec(schema)` handles plain
+JSON values. The optional `configHash` invalidates saved results when both the
+file and the caller supply nonempty hashes that differ.
+
+`clock` is required and controls the delay between file replacement retries
+when a file is busy. Pass `globalThis` to use Node.js timers.
+
+Writes from one `Checkpoint` instance run in order. Each write captures the
+latest state when it starts, so an earlier snapshot cannot overwrite a later
+one. The checkpoint uses `durableWrite` to flush the file before replacing it and
+sync the parent directory where supported. Await `run()`, `each()`, or
+`point().through()` before exiting. Separate instances or processes do not
+coordinate writes to the same file.
+
 ## Schema index caching
 
 Schema introspection is the expensive part of indexing. `Text2Sql` and
@@ -420,13 +463,20 @@ import { FileIndexCache, Text2Sql } from '@deepagents/text2sql';
 
 const text2sql = new Text2Sql({
   adapters: { main: adapter },
-  cache: new FileIndexCache({ dir: '/var/cache/text2sql', namespace: 'v1' }),
+  cache: new FileIndexCache({
+    clock: globalThis,
+    dir: '/var/cache/text2sql',
+    namespace: 'v1',
+  }),
 });
 ```
 
-`FileIndexCache` writes atomically (temp + rename). It treats a file that does
-not parse to context fragments as a miss, so a torn read causes a new
-introspection. `dir` defaults to the OS temp directory.
+`FileIndexCache` writes a temporary file and renames it over the cache file. A
+missing file, invalid JSON, or data that fails context-fragment validation is
+a cache miss, so the next `index()` call introspects again. `clock` is required
+and controls file replacement retry delays; pass `globalThis` to use Node.js
+timers. `dir` defaults to the OS temp directory. The optional `namespace`
+separates cache generations; change it when the schema changes.
 
 ### Introspect once for concurrent callers
 
@@ -441,16 +491,18 @@ uses [`@zukhruf/mutex`](https://www.npmjs.com/package/@zukhruf/mutex):
 import { Mutex } from '@zukhruf/mutex';
 import { SqliteStore } from '@zukhruf/mutex/sqlite';
 
-const mutex = new Mutex(new SqliteStore('/var/lib/my-app/locks'));
+const mutex = new Mutex(
+  new SqliteStore('/var/lib/my-app/locks', { clock: globalThis }),
+);
 
 const fragments = await mutex.acquire('text2sql:main', () =>
   text2sql.index({ names: ['main'] }),
 );
 ```
 
-The first caller introspects the database and writes the cache. The other
-callers wait. When each waiter gets the key, its `index()` call reads the cache
-that the first caller wrote.
+Callers must share the cache directory, namespace, and adapter names. The first
+caller introspects the database and writes the cache. When each waiting caller
+acquires the same lock key, its `index()` call reads that cached schema.
 
 `SqliteStore` shares keys between the processes on one host. For a fleet of
 hosts, acquire the key from a service that all hosts share, for example a
@@ -460,7 +512,7 @@ Postgres advisory lock, and give all hosts the same cache.
 > `FileIndexCache` parse-as-miss behavior tolerates a torn write: the next
 > `index()` call introspects again.
 
-## Streaming Index Progress
+## Streaming index progress
 
 `AdapterIndexer#index({ onProgress })` emits progress events while it warms or reads
 the schema cache for host-side indexing. The sandbox CLI writes the same event
@@ -510,7 +562,7 @@ imposing one duration policy.
 `toSql()` is stateless. It reads the current schema fragments, but it does not
 write messages, titles, or usage into your context store.
 
-## Direct SQL Generation with Extra Fragments
+## Direct SQL generation with extra fragments
 
 This lower-level helper still takes a single `adapter` because it bypasses the
 multi-adapter `Text2Sql` wrapper entirely.
@@ -536,11 +588,11 @@ console.log(result.sql);
 
 Full documentation available at [januarylabs.github.io/deepagents](https://januarylabs.github.io/deepagents/docs/text2sql):
 
-- [Getting Started](https://januarylabs.github.io/deepagents/docs/text2sql/getting-started)
+- [Getting started](https://januarylabs.github.io/deepagents/docs/text2sql/getting-started)
 - [Generate SQL](https://januarylabs.github.io/deepagents/docs/text2sql/to-sql)
-- [Agent + SQLite Store Recipe](https://januarylabs.github.io/deepagents/docs/text2sql/recipes/agent-sqlite-store)
-- [Teach the System](https://januarylabs.github.io/deepagents/docs/text2sql/teach-the-system)
-- [Build Conversations](https://januarylabs.github.io/deepagents/docs/text2sql/build-conversations)
+- [Agent + SQLite store recipe](https://januarylabs.github.io/deepagents/docs/text2sql/recipes/agent-sqlite-store)
+- [Teach the system](https://januarylabs.github.io/deepagents/docs/text2sql/teach-the-system)
+- [Build conversations](https://januarylabs.github.io/deepagents/docs/text2sql/build-conversations)
 - [Grounding](https://januarylabs.github.io/deepagents/docs/text2sql/grounding)
 - [PostgreSQL](https://januarylabs.github.io/deepagents/docs/text2sql/postgresql)
 - [SQLite](https://januarylabs.github.io/deepagents/docs/text2sql/sqlite)

package/dist/bin/sql.js

@@ -18,9 +18,11 @@ import { tmpdir } from "node:os";
 import path from "node:path";
 var FileCache = class {
   path;
-  constructor(watermark, extension = ".txt", baseDir = tmpdir()) {
+  #clock;
+  constructor(watermark, { clock, extension = ".txt", baseDir = tmpdir() }) {
     const hash = createHash("md5").update(watermark).digest("hex");
     this.path = path.join(baseDir, `text2sql-${hash}${extension}`);
+    this.#clock = clock;
   }
   async get() {
     if (existsSync(this.path)) {
@@ -30,7 +32,7 @@ var FileCache = class {
   }
   async set(content) {
     await mkdir(path.dirname(this.path), { recursive: true });
-    await atomicWrite(this.path, content);
+    await atomicWrite(this.path, content, { clock: this.#clock });
   }
 };
 var JsonCache = class extends FileCache {
@@ -39,8 +41,8 @@ var JsonCache = class extends FileCache {
    * @param isValid - Checks that a parsed cache file holds a T; a file that
    * fails it is read as a miss.
    */
-  constructor(watermark, isValid, baseDir) {
-    super(watermark, ".json", baseDir);
+  constructor(watermark, isValid, { clock, baseDir }) {
+    super(watermark, { clock, extension: ".json", baseDir });
     this.#isValid = isValid;
   }
   async read() {
@@ -62,9 +64,11 @@ var JsonCache = class extends FileCache {
 
 // packages/text2sql/src/lib/index-cache.ts
 var FileIndexCache = class {
+  #clock;
   #dir;
   #namespace;
-  constructor(options = {}) {
+  constructor(options) {
+    this.#clock = options.clock;
     this.#dir = options.dir;
     this.#namespace = options.namespace;
   }
@@ -76,7 +80,10 @@ var FileIndexCache = class {
   }
   #cacheFor(key) {
     const watermark = this.#namespace ? `${this.#namespace}-${key}` : key;
-    return new JsonCache(watermark, isFragmentList, this.#dir);
+    return new JsonCache(watermark, isFragmentList, {
+      clock: this.#clock,
+      baseDir: this.#dir
+    });
   }
 };
 function isFragmentList(value) {
@@ -107,6 +114,7 @@ function validateAdapterNames(names) {
 }
 
 // packages/text2sql/src/lib/adapters/adapter.ts
+import pMemoize from "p-memoize";
 import { format as formatSql } from "sql-formatter";
 import { z } from "zod";
 
@@ -887,7 +895,7 @@ function resolveIndexCache() {
   if (!dir && !namespace) {
     return void 0;
   }
-  return new FileIndexCache({ dir, namespace });
+  return new FileIndexCache({ clock: globalThis, dir, namespace });
 }
 async function runCommand(command, positional, options) {
   let ctx;

package/dist/index.js

@@ -19,6 +19,7 @@ function validateAdapterNames(names) {
 }
 
 // packages/text2sql/src/lib/adapters/adapter.ts
+import pMemoize from "p-memoize";
 import { format as formatSql } from "sql-formatter";
 import { z } from "zod";
 
@@ -431,12 +432,7 @@ var Adapter = class {
   #decodeShellEscapes(sql) {
     return sql.replace(/\\n/g, "\n").replace(/\\t/g, "	").replace(/\\([().*$])/g, "$1").replace(/\\(?=$)/gm, "");
   }
-  #cachedAllowedEntities = null;
-  async #resolveScope() {
-    if (this.#cachedAllowedEntities) return this.#cachedAllowedEntities;
-    this.#cachedAllowedEntities = await this.resolveAllowedEntities();
-    return this.#cachedAllowedEntities;
-  }
+  #resolveScope = pMemoize(() => this.resolveAllowedEntities());
   async #analyzePolicy(sql) {
     return this.#policyAnalyzer.analyze(sql, {
       resolveAllowedEntities: () => this.#resolveScope()
@@ -1137,9 +1133,11 @@ import { tmpdir } from "node:os";
 import path from "node:path";
 var FileCache = class {
   path;
-  constructor(watermark, extension = ".txt", baseDir = tmpdir()) {
+  #clock;
+  constructor(watermark, { clock, extension = ".txt", baseDir = tmpdir() }) {
     const hash2 = createHash("md5").update(watermark).digest("hex");
     this.path = path.join(baseDir, `text2sql-${hash2}${extension}`);
+    this.#clock = clock;
   }
   async get() {
     if (existsSync(this.path)) {
@@ -1149,7 +1147,7 @@ var FileCache = class {
   }
   async set(content) {
     await mkdir(path.dirname(this.path), { recursive: true });
-    await atomicWrite(this.path, content);
+    await atomicWrite(this.path, content, { clock: this.#clock });
   }
 };
 var JsonCache = class extends FileCache {
@@ -1158,8 +1156,8 @@ var JsonCache = class extends FileCache {
    * @param isValid - Checks that a parsed cache file holds a T; a file that
    * fails it is read as a miss.
    */
-  constructor(watermark, isValid, baseDir) {
-    super(watermark, ".json", baseDir);
+  constructor(watermark, isValid, { clock, baseDir }) {
+    super(watermark, { clock, extension: ".json", baseDir });
     this.#isValid = isValid;
   }
   async read() {
@@ -1181,9 +1179,11 @@ var JsonCache = class extends FileCache {
 
 // packages/text2sql/src/lib/index-cache.ts
 var FileIndexCache = class {
+  #clock;
   #dir;
   #namespace;
-  constructor(options = {}) {
+  constructor(options) {
+    this.#clock = options.clock;
     this.#dir = options.dir;
     this.#namespace = options.namespace;
   }
@@ -1195,7 +1195,10 @@ var FileIndexCache = class {
   }
   #cacheFor(key) {
     const watermark = this.#namespace ? `${this.#namespace}-${key}` : key;
-    return new JsonCache(watermark, isFragmentList, this.#dir);
+    return new JsonCache(watermark, isFragmentList, {
+      clock: this.#clock,
+      baseDir: this.#dir
+    });
   }
 };
 function isFragmentList(value) {
@@ -1388,27 +1391,29 @@ var Checkpoint = class _Checkpoint {
   points;
   path;
   configHash;
+  #clock;
   #writes = pLimit(1);
   #changes = 0;
   #writtenChanges = 0;
-  constructor(path6, configHash, points) {
+  constructor(path6, configHash, clock, points) {
     this.points = points;
     this.path = path6;
     this.configHash = configHash;
+    this.#clock = clock;
   }
   /**
    * Load checkpoint from file, or return empty checkpoint if none exists.
    * Handles corrupted files and config changes gracefully.
    */
   static async load(options) {
-    const { path: path6, configHash } = options;
+    const { path: path6, configHash, clock } = options;
     if (existsSync2(path6)) {
       try {
         const content = readFileSync(path6, "utf-8");
         const file = checkpointFile.parse(JSON.parse(content));
         if (configHash && file.configHash && file.configHash !== configHash) {
           console.log("\u26A0 Config changed, starting fresh");
-          return new _Checkpoint(path6, configHash, {});
+          return new _Checkpoint(path6, configHash, clock, {});
         }
         const points = file.points;
         const totalEntries = Object.values(points).reduce(
@@ -1416,14 +1421,14 @@ var Checkpoint = class _Checkpoint {
           0
         );
         console.log(`\u2713 Resuming from checkpoint (${totalEntries} entries)`);
-        return new _Checkpoint(path6, configHash, points);
+        return new _Checkpoint(path6, configHash, clock, points);
       } catch {
         console.log("\u26A0 Checkpoint corrupted, starting fresh");
-        return new _Checkpoint(path6, configHash, {});
+        return new _Checkpoint(path6, configHash, clock, {});
       }
     }
     console.log("Starting new checkpoint");
-    return new _Checkpoint(path6, configHash, {});
+    return new _Checkpoint(path6, configHash, clock, {});
   }
   /**
    * Run a single computation with checkpointing.
@@ -1501,7 +1506,9 @@ var Checkpoint = class _Checkpoint {
         configHash: this.configHash,
         points: this.points
       };
-      await durableWrite(this.path, JSON.stringify(file, null, 2));
+      await durableWrite(this.path, JSON.stringify(file, null, 2), {
+        clock: this.#clock
+      });
       this.#writtenChanges = changes;
     });
   }
@@ -1557,9 +1564,9 @@ function hashConfig(config) {
 
 // packages/text2sql/src/lib/fs/sqlite/sqlite-fs.ts
 import { Declaration } from "@zukhruf/sqlite-drift";
+import { open } from "@zukhruf/sqlite-open";
 import { transaction } from "@zukhruf/sqlite-transaction";
 import * as path2 from "node:path";
-import { DatabaseSync } from "node:sqlite";
 import z6 from "zod";
 
 // packages/text2sql/src/lib/fs/columns.ts
@@ -1569,7 +1576,6 @@ var bigintColumn = z5.union([z5.number(), z5.string().regex(/^-?\d+$/)]).transfo
 
 // packages/text2sql/src/lib/fs/sqlite/ddl.sqlite-fs.ts
 var SETUP = `
-PRAGMA journal_mode = WAL;
 PRAGMA synchronous = NORMAL;
 PRAGMA cache_size = -64000;
 PRAGMA temp_store = MEMORY;
@@ -1631,7 +1637,7 @@ var SqliteFs = class {
     this.#chunkSize = options.chunkSize ?? 1024 * 1024;
     const normalizedRoot = this.#normalizeRoot(options.root);
     this.#root = normalizedRoot === "/" ? "" : normalizedRoot;
-    this.#db = new DatabaseSync(options.dbPath);
+    this.#db = open(options.dbPath);
     this.#db.exec(SETUP);
     tables.apply(this.#db);
     this.#sql = this.#db.createTagStore();