@dotdotgod/context 0.2.25 → 0.2.27

package/README.md

@@ -10,7 +10,8 @@ The package exposes a local stdio MCP server and reusable core modules. Claude C
 
 - `execute`, `batch_execute`, `execute_file`
 - `index`, `search`, `fetch_and_index`
-- `stats`, `doctor`, `purge`
+- `session_resume`, `ingestion_job_start`, `ingestion_job_status`, `ingestion_job_cancel`
+- `context_heal`, `stats`, `doctor`, `purge`
 - `dotdotgod_project_load`, `dotdotgod_project_impact`, `dotdotgod_project_initialize`
 
 Run the stdio MCP server directly with:
@@ -36,6 +37,8 @@ Each command has one **10 MiB combined capture ceiling** shared by stdout and st
 
 `batch_execute` accepts up to 100 commands, uses concurrency from 1 through 8, and preserves input order. Execution and explicit shell mode remain write-capable operations subject to the host's approval model.
 
+`execute`, `batch_execute`, and `execute_file` share an `inherit-filtered-v1` child-environment policy. It preserves compatibility-oriented inheritance while filtering `NODE_OPTIONS`, `PYTHONPATH`, `RUBYOPT`, `LD_PRELOAD`, and supported-platform `DYLD_*` variables. Callers may provide string overrides or remove inherited variables with `null`, but cannot restore reserved variables. Results report filtered names without values. This policy applies only to these dotdotgod tools and does not isolate ordinary inherited credentials.
+
 ## Local Storage And Retrieval
 
 Indexed content is stored in the project-local SQLite FTS5 database:
@@ -44,7 +47,7 @@ Indexed content is stored in the project-local SQLite FTS5 database:
 .dotdotgod/context/context.sqlite
 ```
 
-The store is local ignored state, not maintained project truth. Sources can use transient, session, or project scope and carry optional expiry metadata. Search applies scope, session, and source predicates before candidate fusion, then:
+The store is local ignored state, not maintained project truth. Sources can use transient, session, or project scope and carry optional expiry metadata. Writable connections use WAL and a 1-second busy timeout. Source replacement, expiry, and purge update source and FTS rows transactionally. Recognized older schemas migrate transactionally through a version ledger; unknown, newer, incomplete, or corrupt schemas fail closed before explicit healing. Search applies scope, session, and source predicates before candidate fusion, then:
 
 1. retrieves bounded Porter-tokenized FTS5 candidates;
 2. retrieves bounded label/path candidates;
@@ -56,7 +59,17 @@ This is bounded indexing and retrieval, not LLM summarization. Search does not a
 
 ## Structural Ingestion
 
-Local `index` accepts one regular text file up to **25 MiB**. Directory ingestion is not supported.
+Local `index` accepts one project-contained regular text file or directory. A file is limited to **25 MiB**. Directory defaults allow at most depth 16, 10,000 visited entries, 1,000 indexed files, 25 MiB per file, and 100 MiB in aggregate; callers may configure the traversal limits explicitly.
+
+Directory ingestion:
+
+- walks paths in deterministic lexical order and returns metadata-only indexed, skipped, and failed entries;
+- accepts explicit `includeExtensions` and `excludePaths` filters but does not interpret `.gitignore` files;
+- skips directory and file symlinks by default; optional file-symlink following remains limited to targets inside the project;
+- rejects special files and verifies file identity through a bounded descriptor read before indexing;
+- stops further traversal/indexing on cancellation while reporting already committed sources.
+
+Content chunking then applies these rules:
 
 - Markdown is split around headings, prose blocks, and fenced blocks while retaining heading and fence metadata.
 - JSON is parsed into deterministic key-path chunks; invalid JSON falls back to bounded text chunks.
@@ -91,7 +104,7 @@ The application-level fetch policy:
 - supports identity, gzip, deflate, and Brotli encodings and rejects unsupported encodings;
 - accepts bounded text-oriented MIME types and fails closed for unsupported types or HTTP statuses.
 
-HTML is handled as bounded untrusted text; the package does not claim rich HTML extraction. These controls provide application-level URL and address validation, not a network namespace or process sandbox.
+Accepted HTML is normalized by a bounded `html-v1` extractor. It validates supported UTF-8/US-ASCII declarations, removes scripts, styles, noscript and hidden/non-content elements, and preserves basic titles, headings, paragraphs, lists, tables, code, and link text. Link metadata is separately bounded. The extractor does not invoke a browser, execute JavaScript, load subresources, follow links, or provide standards-complete HTML rendering. HTML remains `external-untrusted`; normalization is not rendering sanitization or a prompt-injection guarantee. These controls provide application-level URL and address validation, not a network namespace or process sandbox.
 
 ## Diagnostics And Deletion
 
@@ -100,7 +113,7 @@ HTML is handled as bounded untrusted text; the package does not claim rich HTML
 - the minimum Node.js runtime;
 - project and storage-path readiness;
 - SQLite FTS5 with the Porter tokenizer;
-- compatibility of an existing context database schema;
+- compatibility of an existing context database schema without migration or repair;
 - current store statistics when supplied;
 - the configured strict fetch policy.
 
@@ -114,6 +127,12 @@ The MCP server also exposes bounded project load, graph impact, and initializer
 
 Claude Code and Codex adapter hooks may tell the model to call these MCP tools and retry a denied operation. Hooks do not transform an ordinary host shell call into an MCP call. Pi retains its native project-memory lifecycle and calls the shared runtime directly.
 
+## Phase 3 Capabilities
+
+Search includes a bounded typo-tolerant trigram lane while Porter FTS remains primary. Databases use transactional version-ledger migrations; explicit healing backs up first and accepts only recognized profiles. Opaque session IDs can be resumed without a history-listing API. Durable background ingestion has bounded queue, status, cancellation, and restart recovery.
+
+Execution keeps compatibility-oriented filtered inheritance by default; this mode is not credential isolation. Opt-in `allowlist-v1` starts from an empty child environment and copies only explicit names, including caller overrides only when separately allowlisted, with names-only reporting. Strict fetch remains default. Browser rendering is available only through an injected renderer plus `browser: true`; this package bundles no browser, keeps rendered content `external-untrusted`, enforces limits/abort, and makes no sandbox claim.
+
 ## Learn More
 
 - [Behavior specification](https://github.com/dotdotgod/dotdotgod-kit/blob/main/docs/spec/CONTEXT_EXECUTION.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dotdotgod/context",
-  "version": "0.2.25",
+  "version": "0.2.27",
   "description": "Local-first MCP execution, indexing, and retrieval runtime for dotdotgod adapters.",
   "type": "module",
   "license": "Elastic-2.0",
@@ -28,7 +28,7 @@
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.25.2",
     "zod": "^3.25.76",
-    "@dotdotgod/cli": "0.2.25"
+    "@dotdotgod/cli": "0.2.27"
   },
   "engines": {
     "node": ">=22.5.0"

package/src/content.mjs

@@ -1,41 +1,129 @@
 import { readFileSync, statSync } from 'node:fs';
+import { readManifestFile, walkDirectoryManifest } from './directory-ingestion.mjs';
+import { normalizeHtml } from './html-normalize.mjs';
 import { resolveExistingWithinRoot } from './paths.mjs';
 import { safeFetch } from './safe-fetch.mjs';
 
 const MAX_FILE_BYTES = 25 * 1024 * 1024;
 const MAX_FETCH_BYTES = 10 * 1024 * 1024;
 
-export function indexFile(store, input, sessionId) {
-  const path = resolveExistingWithinRoot(input.root || process.cwd(), input.path);
-  const stat = statSync(path);
-  if (!stat.isFile()) throw new Error('Only regular files are supported in the first release.');
+function indexRegularFile(store, input, sessionId, path, stat, label = input.source ?? path, bytes) {
   const maxBytes = Math.min(input.maxBytes ?? MAX_FILE_BYTES, MAX_FILE_BYTES);
   if (stat.size > maxBytes) throw new Error(`File exceeds maximum size: ${stat.size} bytes`);
+  const content = bytes ?? readFileSync(path);
+  if (content.length > maxBytes) throw new Error(`File exceeds maximum size: ${content.length} bytes`);
   return store.index({
     scope: input.scope ?? 'session',
     sessionId,
-    label: input.source ?? path,
+    label,
     kind: 'file',
-    text: readFileSync(path, 'utf8'),
+    text: content.toString('utf8'),
     metadata: { path, size: stat.size },
     ttlMs: input.ttlMs ?? null,
   });
 }
 
-export async function fetchAndIndex(store, input, sessionId, signal) {
+function indexDirectory(store, input, sessionId, signal) {
+  const manifest = walkDirectoryManifest({
+    root: input.root || process.cwd(),
+    path: input.path,
+    includeExtensions: input.includeExtensions,
+    excludePaths: input.excludePaths,
+    followFileSymlinks: input.followFileSymlinks,
+    maxDepth: input.maxDepth,
+    maxVisitedEntries: input.maxVisitedEntries,
+    maxFiles: input.maxFiles,
+    maxFileBytes: Math.min(input.maxBytes ?? MAX_FILE_BYTES, MAX_FILE_BYTES),
+    maxAggregateBytes: input.maxAggregateBytes,
+    signal,
+  });
+  const indexed = [];
+  const failed = [...manifest.failed];
+  const aggregateLimit = input.maxAggregateBytes ?? 100 * 1024 * 1024;
+  let indexedBytes = 0;
+  for (const entry of manifest.files) {
+    if (signal?.aborted) break;
+    try {
+      const remaining = aggregateLimit - indexedBytes;
+      if (remaining < 0) throw new Error('Directory aggregate byte limit exceeded.');
+      const bytes = readManifestFile(entry, manifest.root, Math.min(input.maxBytes ?? MAX_FILE_BYTES, MAX_FILE_BYTES, remaining));
+      const label = input.source ? `${input.source}/${entry.path}` : entry.absolutePath;
+      indexed.push({ path: entry.path, ...indexRegularFile(store, input, sessionId, entry.absolutePath, entry, label, bytes) });
+      indexedBytes += bytes.length;
+    } catch (error) {
+      failed.push({ path: entry.path, reason: 'index-failed', error: error instanceof Error ? error.message : String(error) });
+    }
+  }
+  return {
+    directory: manifest.directory,
+    indexed,
+    skipped: manifest.skipped,
+    failed,
+    aborted: manifest.aborted || signal?.aborted === true,
+    truncated: manifest.truncated,
+    visitedEntries: manifest.visitedEntries,
+    aggregateBytes: indexedBytes,
+  };
+}
+
+export function indexFile(store, input, sessionId, signal) {
+  const path = resolveExistingWithinRoot(input.root || process.cwd(), input.path);
+  const stat = statSync(path);
+  if (stat.isDirectory()) return indexDirectory(store, input, sessionId, signal);
+  if (!stat.isFile()) throw new Error('Only regular files and directories are supported.');
+  return indexRegularFile(store, input, sessionId, path, stat);
+}
+
+export async function fetchAndIndex(store, input, sessionId, signal, capabilities = {}) {
   const maxBytes = Math.min(input.maxBytes ?? MAX_FETCH_BYTES, MAX_FETCH_BYTES);
-  const fetched = await safeFetch(input.url, {
+  let fetched;
+  if (input.browser === true) {
+    if (typeof capabilities.renderer !== 'function') throw new Error('Browser rendering capability is unavailable; strict fetch remains the default.');
+    const url = new URL(input.url);
+    if (!['http:', 'https:'].includes(url.protocol)) throw new Error('Only HTTP(S) browser rendering is supported.');
+    const timeoutMs = Math.min(input.timeoutMs ?? 10_000, 120_000);
+    const controller = new AbortController();
+    let rejectAbort;
+    const aborted = new Promise((_, reject) => { rejectAbort = reject; });
+    const abort = () => { const reason = signal?.reason instanceof Error ? signal.reason : new Error('Browser rendering aborted.'); controller.abort(reason); rejectAbort(reason); };
+    if (signal?.aborted) abort(); else signal?.addEventListener('abort', abort, { once: true });
+    let timer;
+    const timeout = new Promise((_, reject) => { timer = setTimeout(() => { controller.abort(new Error('Browser rendering timed out.')); reject(new Error(`Browser rendering timed out after ${timeoutMs}ms.`)); }, timeoutMs); timer.unref?.(); });
+    let rendered;
+    try { rendered = await Promise.race([capabilities.renderer({ url: url.href, signal: controller.signal, timeoutMs, maxBytes }), timeout, aborted]); }
+    finally { clearTimeout(timer); signal?.removeEventListener('abort', abort); }
+    if (!rendered || typeof rendered !== 'object' || Array.isArray(rendered)) throw new Error('Browser renderer returned an invalid result.');
+    if (rendered.body !== undefined && !Buffer.isBuffer(rendered.body)) throw new Error('Browser renderer body must be a Buffer.');
+    if (rendered.text !== undefined && typeof rendered.text !== 'string') throw new Error('Browser renderer text must be a string.');
+    if (rendered.body === undefined && rendered.text === undefined) throw new Error('Browser renderer must return body or text.');
+    const body = rendered.body ?? Buffer.from(rendered.text, 'utf8');
+    if (body.length > maxBytes) throw new Error(`Rendered content exceeds maximum size: ${body.length} bytes`);
+    const finalUrl = String(rendered.url ?? url.href);
+    if (finalUrl.length > 4096 || !['http:', 'https:'].includes(new URL(finalUrl).protocol)) throw new Error('Browser renderer returned an invalid final URL.');
+    const contentType = String(rendered.contentType ?? 'text/html; charset=utf-8');
+    if (contentType.length > 512 || /[\r\n]/u.test(contentType)) throw new Error('Browser renderer returned invalid content type metadata.');
+    const status = rendered.status ?? 200;
+    if (!Number.isInteger(status) || status < 100 || status > 599) throw new Error('Browser renderer returned an invalid HTTP status.');
+    fetched = { body, text: body.toString('utf8'), url: finalUrl, originalUrl: url.href, redirects: [], contentType, contentEncoding: 'identity', status, bytes: body.length, wireLength: body.length };
+  } else fetched = await safeFetch(input.url, {
     signal,
     timeoutMs: input.timeoutMs,
     maxBytes,
     maxWireBytes: maxBytes,
   });
+  const isHtml = /^(?:text\/html|application\/xhtml\+xml)(?:;|$)/iu.test(fetched.contentType);
+  const normalized = isHtml ? normalizeHtml(fetched.body, {
+    contentType: fetched.contentType,
+    maxInputBytes: maxBytes,
+    maxOutputBytes: maxBytes,
+  }) : null;
+  const text = normalized?.text ?? fetched.text;
   return store.index({
     scope: input.scope ?? 'project',
     sessionId,
-    label: input.source ?? fetched.url,
+    label: input.source ?? normalized?.title ?? fetched.url,
     kind: 'url',
-    text: fetched.text,
+    text,
     metadata: {
       url: fetched.url,
       originalUrl: fetched.originalUrl,
@@ -46,8 +134,21 @@ export async function fetchAndIndex(store, input, sessionId, signal) {
       status: fetched.status,
       bytes: fetched.bytes,
       wireLength: fetched.wireLength,
+      ...(normalized ? {
+        htmlTitle: normalized.title,
+        htmlInputBytes: normalized.inputBytes,
+        htmlOutputBytes: normalized.outputBytes,
+        htmlTruncated: normalized.truncated,
+        htmlFallbackReason: normalized.fallbackReason,
+      } : {}),
+      acquisitionMode: input.browser === true ? 'browser-renderer' : 'strict-fetch',
+    },
+    provenance: {
+      sourceType: 'fetched-url',
+      origin: fetched.url,
+      contentType: normalized ? 'text/markdown' : fetched.contentType,
+      extractor: normalized?.extractor,
     },
-    provenance: { sourceType: 'fetched-url', origin: fetched.url, contentType: fetched.contentType },
     ttlMs: input.ttlMs ?? 24 * 60 * 60 * 1000,
   });
 }

package/src/directory-ingestion.mjs

@@ -0,0 +1,182 @@
+import { closeSync, constants, fstatSync, lstatSync, openSync, readFileSync, readdirSync, realpathSync, statSync } from 'node:fs';
+import { isAbsolute, relative, resolve, sep } from 'node:path';
+
+const DEFAULT_LIMITS = Object.freeze({
+  maxDepth: 16,
+  maxVisitedEntries: 10_000,
+  maxFiles: 1_000,
+  maxFileBytes: 25 * 1024 * 1024,
+  maxAggregateBytes: 100 * 1024 * 1024,
+});
+
+function inside(base, target) {
+  const rel = relative(base, target);
+  return rel !== '..' && !rel.startsWith(`..${sep}`) && !isAbsolute(rel);
+}
+
+function relativePath(base, target) {
+  return relative(base, target).split(sep).join('/');
+}
+
+function boundedInteger(value, fallback, name) {
+  const selected = value ?? fallback;
+  if (!Number.isSafeInteger(selected) || selected < 0) throw new TypeError(`${name} must be a non-negative safe integer`);
+  return selected;
+}
+
+function normalizeOptions(options) {
+  const includeExtensions = options.includeExtensions?.map((value) => {
+    const extension = String(value).toLowerCase();
+    return extension.startsWith('.') ? extension : `.${extension}`;
+  });
+  const excludePaths = (options.excludePaths ?? []).map((value) => {
+    const normalized = String(value).replaceAll('\\', '/').replace(/^\.\//, '').replace(/\/$/, '');
+    if (!normalized || normalized === '..' || normalized.startsWith('../') || normalized.includes('/../')) {
+      throw new TypeError(`Invalid excluded path: ${value}`);
+    }
+    return normalized;
+  });
+  return {
+    includeExtensions: includeExtensions ? new Set(includeExtensions) : null,
+    excludePaths,
+    followFileSymlinks: options.followFileSymlinks === true,
+    maxDepth: boundedInteger(options.maxDepth, DEFAULT_LIMITS.maxDepth, 'maxDepth'),
+    maxVisitedEntries: boundedInteger(options.maxVisitedEntries, DEFAULT_LIMITS.maxVisitedEntries, 'maxVisitedEntries'),
+    maxFiles: boundedInteger(options.maxFiles, DEFAULT_LIMITS.maxFiles, 'maxFiles'),
+    maxFileBytes: boundedInteger(options.maxFileBytes, DEFAULT_LIMITS.maxFileBytes, 'maxFileBytes'),
+    maxAggregateBytes: boundedInteger(options.maxAggregateBytes, DEFAULT_LIMITS.maxAggregateBytes, 'maxAggregateBytes'),
+  };
+}
+
+function excluded(path, prefixes) {
+  return prefixes.some((prefix) => path === prefix || path.startsWith(`${prefix}/`));
+}
+
+function extensionAllowed(path, extensions) {
+  if (!extensions) return true;
+  const slash = path.lastIndexOf('/');
+  const dot = path.lastIndexOf('.');
+  return dot > slash && extensions.has(path.slice(dot).toLowerCase());
+}
+
+function snapshot(path, target, stat, symlink) {
+  return {
+    path,
+    absolutePath: target,
+    size: stat.size,
+    identity: { dev: stat.dev, ino: stat.ino, size: stat.size, mtimeMs: stat.mtimeMs },
+    symlink,
+  };
+}
+
+export function walkDirectoryManifest(options = {}) {
+  const root = realpathSync(resolve(options.root || process.cwd()));
+  const requested = resolve(root, options.path ?? '.');
+  if (!inside(root, requested)) throw new Error(`Directory escapes project root: ${options.path}`);
+  if (lstatSync(requested).isSymbolicLink()) throw new Error(`Directory symlinks are not followed: ${options.path ?? '.'}`);
+  const directory = realpathSync(requested);
+  if (!inside(root, directory)) throw new Error(`Directory escapes project root: ${options.path}`);
+  if (!statSync(directory).isDirectory()) throw new Error(`Not a directory: ${options.path ?? '.'}`);
+
+  const policy = normalizeOptions(options);
+  const files = [];
+  const skipped = [];
+  const failed = [];
+  let visitedEntries = 0;
+  let aggregateBytes = 0;
+  let aborted = false;
+  let truncated = false;
+
+  function skip(path, reason) { skipped.push({ path, reason }); }
+  function fail(path, reason) { failed.push({ path, reason }); }
+
+  function visit(current, depth) {
+    if (aborted || options.signal?.aborted) { aborted = true; return; }
+    let names;
+    try { names = readdirSync(current).sort((a, b) => (a < b ? -1 : a > b ? 1 : 0)); }
+    catch { fail(relativePath(root, current), 'directory-read-failed'); return; }
+
+    for (const name of names) {
+      if (options.signal?.aborted) { aborted = true; return; }
+      const absolutePath = resolve(current, name);
+      const path = relativePath(root, absolutePath);
+      if (visitedEntries >= policy.maxVisitedEntries) { skip(path, 'visited-entry-limit'); truncated = true; return; }
+      visitedEntries += 1;
+      if (excluded(path, policy.excludePaths)) { skip(path, 'excluded-path'); continue; }
+
+      let entry;
+      try { entry = lstatSync(absolutePath); }
+      catch { fail(path, 'entry-stat-failed'); continue; }
+
+      if (entry.isSymbolicLink()) {
+        let target;
+        let targetStat;
+        try {
+          target = realpathSync(absolutePath);
+          if (!inside(root, target)) { skip(path, 'symlink-outside-root'); continue; }
+          targetStat = statSync(target);
+        } catch { skip(path, 'symlink-unresolvable'); continue; }
+        if (targetStat.isDirectory()) { skip(path, 'directory-symlink'); continue; }
+        if (!policy.followFileSymlinks) { skip(path, 'file-symlink'); continue; }
+        if (!targetStat.isFile()) { skip(path, 'special-file'); continue; }
+        addFile(path, target, targetStat, true);
+        continue;
+      }
+
+      if (entry.isDirectory()) {
+        if (depth >= policy.maxDepth) { skip(path, 'depth-limit'); continue; }
+        visit(absolutePath, depth + 1);
+        if (aborted || truncated) return;
+        continue;
+      }
+      if (!entry.isFile()) { skip(path, 'special-file'); continue; }
+      addFile(path, absolutePath, entry, false);
+    }
+  }
+
+  function addFile(path, absolutePath, stat, symlink) {
+    if (!extensionAllowed(path, policy.includeExtensions)) { skip(path, 'extension-filter'); return; }
+    if (stat.size > policy.maxFileBytes) { skip(path, 'file-byte-limit'); return; }
+    if (files.length >= policy.maxFiles) { skip(path, 'file-count-limit'); return; }
+    if (aggregateBytes + stat.size > policy.maxAggregateBytes) { skip(path, 'aggregate-byte-limit'); return; }
+    files.push(snapshot(path, absolutePath, stat, symlink));
+    aggregateBytes += stat.size;
+  }
+
+  visit(directory, 0);
+  return { root, directory, files, skipped, failed, aborted, truncated, visitedEntries, aggregateBytes };
+}
+
+function matchesIdentity(stat, identity) {
+  return stat.dev === identity.dev && stat.ino === identity.ino && stat.size === identity.size && stat.mtimeMs === identity.mtimeMs;
+}
+
+export function verifyManifestFile(entry, root) {
+  const base = realpathSync(resolve(root));
+  const target = realpathSync(entry.absolutePath);
+  if (!inside(base, target)) throw new Error(`Manifest file escapes project root: ${entry.path}`);
+  const stat = statSync(target);
+  if (!stat.isFile()) throw new Error(`Manifest entry is not a regular file: ${entry.path}`);
+  if (!matchesIdentity(stat, entry.identity ?? {})) throw new Error(`Manifest file changed before indexing: ${entry.path}`);
+  return target;
+}
+
+export function readManifestFile(entry, root, maxBytes = DEFAULT_LIMITS.maxFileBytes) {
+  const target = verifyManifestFile(entry, root);
+  const noFollow = constants.O_NOFOLLOW ?? 0;
+  const descriptor = openSync(target, constants.O_RDONLY | noFollow);
+  try {
+    const before = fstatSync(descriptor);
+    if (!before.isFile() || !matchesIdentity(before, entry.identity ?? {})) throw new Error(`Manifest file changed before indexing: ${entry.path}`);
+    if (before.size > maxBytes) throw new Error(`Manifest file exceeds maximum size: ${entry.path}`);
+    const bytes = readFileSync(descriptor);
+    const after = fstatSync(descriptor);
+    if (!matchesIdentity(after, entry.identity ?? {}) || bytes.length !== before.size) throw new Error(`Manifest file changed while indexing: ${entry.path}`);
+    if (bytes.length > maxBytes) throw new Error(`Manifest file exceeds maximum size: ${entry.path}`);
+    return bytes;
+  } finally {
+    closeSync(descriptor);
+  }
+}
+
+export { DEFAULT_LIMITS as DIRECTORY_INGESTION_LIMITS };

package/src/doctor.mjs

@@ -92,11 +92,20 @@ function schemaCheck(dbPath) {
     const names = new Set(db.prepare("SELECT name FROM sqlite_master WHERE type IN ('table', 'view')").all().map((row) => row.name));
     const missing = [...EXPECTED_TABLES].filter((name) => !names.has(name));
     if (missing.length > 0) return check('schema-compatibility', 'FAIL', 'Context database is missing required tables.', { dbPath, missing });
-    const sourceColumns = new Set(db.prepare('PRAGMA table_info(sources)').all().map((row) => row.name));
-    const requiredColumns = ['id', 'scope', 'session_id', 'label', 'kind', 'metadata', 'created_at', 'expires_at'];
-    const missingColumns = requiredColumns.filter((name) => !sourceColumns.has(name));
-    if (missingColumns.length > 0) return check('schema-compatibility', 'FAIL', 'Sources table is incompatible.', { dbPath, missingColumns });
-    return check('schema-compatibility', 'OK', 'Existing context database has the required schema.', { dbPath });
+    const sourceRows = db.prepare('PRAGMA table_info(sources)').all();
+    const chunkRows = db.prepare('PRAGMA table_info(chunks)').all();
+    const requiredSources = ['id', 'scope', 'session_id', 'label', 'kind', 'metadata', 'created_at', 'expires_at'];
+    const requiredChunks = ['source_id', 'scope', 'session_id', 'ordinal', 'body'];
+    const missingColumns = [...requiredSources.filter((name) => !sourceRows.some((row) => row.name === name)), ...requiredChunks.filter((name) => !chunkRows.some((row) => row.name === name))];
+    const sourceShape = sourceRows.length === requiredSources.length && sourceRows.every((row, index) => row.name === requiredSources[index])
+      && sourceRows[0].pk === 1
+      && sourceRows.every((row) => ['scope', 'label', 'kind', 'metadata', 'created_at'].includes(row.name) ? row.notnull === 1 : true);
+    const chunkShape = chunkRows.length === requiredChunks.length && chunkRows.every((row, index) => row.name === requiredChunks[index]);
+    const chunkSql = String(db.prepare("SELECT sql FROM sqlite_master WHERE name = 'chunks'").get()?.sql ?? '').toLowerCase();
+    if (missingColumns.length > 0 || !sourceShape || !chunkShape || !chunkSql.includes('using fts5') || !chunkSql.includes("tokenize='porter unicode61'")) {
+      return check('schema-compatibility', 'FAIL', 'Context database schema is incompatible.', { dbPath, schemaProfile: 'unknown', missingColumns });
+    }
+    return check('schema-compatibility', 'OK', 'Existing context database matches compatibility profile 1.', { dbPath, schemaProfile: 1 });
   } catch (error) {
     return check('schema-compatibility', 'FAIL', 'Context database could not be inspected read-only.', { dbPath, error: error.message });
   } finally {

package/src/environment-policy.mjs

@@ -0,0 +1,85 @@
+const COMMON_FILTERED_NAMES = new Set(['NODE_OPTIONS', 'PYTHONPATH', 'RUBYOPT']);
+const VALID_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/;
+
+function platformFamily(platform) {
+  if (platform === 'win32') return 'win32';
+  if (platform === 'darwin') return 'darwin';
+  return 'posix';
+}
+
+function canonicalName(name, family) {
+  return family === 'win32' ? name.toUpperCase() : name;
+}
+
+function validateName(name) {
+  if (!VALID_NAME.test(name)) throw new TypeError(`Invalid environment variable name: ${JSON.stringify(name)}`);
+}
+
+function isFilteredName(name, family) {
+  const comparable = family === 'win32' ? name.toUpperCase() : name;
+  if (COMMON_FILTERED_NAMES.has(comparable)) return true;
+  if (family === 'darwin') return comparable === 'LD_PRELOAD' || comparable.startsWith('DYLD_');
+  if (family === 'posix') return comparable === 'LD_PRELOAD';
+  return false;
+}
+
+function entriesOf(value, label) {
+  if (value === null || typeof value !== 'object' || Array.isArray(value)) {
+    throw new TypeError(`${label} must be an object`);
+  }
+  return Object.entries(value).sort(([left], [right]) => left.localeCompare(right, 'en'));
+}
+
+/**
+ * Compose a child-process environment without exposing filtered values.
+ * Override values may be strings or null; null explicitly deletes a variable.
+ */
+export function composeEnvironment({ inherited = {}, overrides = {}, platform = process.platform, mode = 'inherit-filtered-v1', allow = [] } = {}) {
+  const family = platformFamily(platform);
+  if (!['inherit-filtered-v1', 'allowlist-v1'].includes(mode)) throw new TypeError(`Unsupported environment policy mode: ${mode}`);
+  if (!Array.isArray(allow)) throw new TypeError('allow must be an array');
+  const allowed = new Set(allow.map((name) => { validateName(name); return canonicalName(name, family); }));
+  const env = {};
+  const filtered = new Set();
+
+  for (const [name, value] of entriesOf(inherited, 'inherited')) {
+    validateName(name);
+    if (typeof value !== 'string' && value !== undefined) {
+      throw new TypeError(`Inherited environment variable ${name} must be a string or undefined`);
+    }
+    if (value === undefined) continue;
+    const outputName = canonicalName(name, family);
+    if (isFilteredName(outputName, family) || (mode === 'allowlist-v1' && !allowed.has(outputName))) {
+      filtered.add(outputName);
+      continue;
+    }
+    env[outputName] = value;
+  }
+
+  for (const [name, value] of entriesOf(overrides, 'overrides')) {
+    validateName(name);
+    const outputName = canonicalName(name, family);
+    if (isFilteredName(outputName, family)) {
+      throw new TypeError(`Environment variable ${name} is reserved by the execution policy`);
+    }
+    if (mode === 'allowlist-v1' && !allowed.has(outputName)) throw new TypeError(`Environment variable ${name} is not in the explicit allowlist`);
+    if (value === null) {
+      delete env[outputName];
+      continue;
+    }
+    if (typeof value !== 'string') {
+      throw new TypeError(`Environment override ${name} must be a string or null`);
+    }
+    env[outputName] = value;
+  }
+
+  return {
+    env,
+    policy: {
+      mode,
+      platform: family,
+      ...(mode === 'allowlist-v1' ? { allowedNames: [...allowed].sort((left, right) => left.localeCompare(right, 'en')) } : {}),
+      filteredNames: [...filtered].sort((left, right) => left.localeCompare(right, 'en')),
+    },
+  };
+}

package/src/execute.mjs

@@ -3,7 +3,9 @@ import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import { spawn } from 'node:child_process';
 import { Transform } from 'node:stream';
+import { finished } from 'node:stream/promises';
 import { resolveWithinRoot } from './paths.mjs';
+import { composeEnvironment } from './environment-policy.mjs';
 
 const DIRECT_LIMIT = 12_000;
 const MAX_DIRECT_RETURN = 1024 * 1024;
@@ -32,6 +34,7 @@ function killProcess(child) {
 }
 
 export async function executeCommand(input, options = {}) {
+  if (options.signal?.aborted) throw options.signal.reason ?? new Error('Execution aborted.');
   const spec = commandSpec(input);
   const cwd = resolveWithinRoot(options.root || process.cwd(), input.cwd || '.');
   const timeoutMs = Math.min(Math.max(1, input.timeoutMs ?? 120_000), 30 * 60_000);
@@ -45,6 +48,12 @@ export async function executeCommand(input, options = {}) {
   let aborted = false;
   let captureLimitExceeded = false;
   const captureLimitBytes = Math.min(HARD_LIMIT, Math.max(1024, options.captureLimitBytes ?? HARD_LIMIT));
+  const environment = composeEnvironment({
+    inherited: options.env ?? process.env,
+    overrides: input.env ?? {},
+    mode: input.environmentMode ?? 'inherit-filtered-v1',
+    allow: input.allowedEnv ?? [],
+  });
 
   try {
     const result = await new Promise((resolvePromise, reject) => {
@@ -53,7 +62,7 @@ export async function executeCommand(input, options = {}) {
         shell: spec.shell,
         detached: process.platform !== 'win32',
         stdio: ['ignore', 'pipe', 'pipe'],
-        env: options.env ?? process.env,
+        env: environment.env,
       });
       let capturedBytes = 0;
       const boundedCapture = () => new Transform({
@@ -74,14 +83,22 @@ export async function executeCommand(input, options = {}) {
       const timer = setTimeout(() => { timedOut = true; killProcess(child); }, timeoutMs);
       const onAbort = () => { aborted = true; killProcess(child); };
       options.signal?.addEventListener('abort', onAbort, { once: true });
-      child.once('error', (error) => { clearTimeout(timer); reject(error); });
+      let spawnError = false;
+      child.once('error', (error) => {
+        spawnError = true;
+        clearTimeout(timer);
+        options.signal?.removeEventListener('abort', onAbort);
+        stdout.end(() => stderr.end(() => reject(error)));
+      });
       child.once('close', (code, signal) => {
+        if (spawnError) return;
         clearTimeout(timer);
         options.signal?.removeEventListener('abort', onAbort);
         stdout.end(() => stderr.end(() => resolvePromise({ code, signal })));
       });
     });
 
+    await Promise.all([finished(stdout), finished(stderr)]);
     const outputLimit = Math.min(MAX_DIRECT_RETURN, Math.max(1, input.outputLimit ?? DIRECT_LIMIT));
     const directLimit = Math.min(HARD_LIMIT, Math.max(1, input.directLimit ?? DIRECT_LIMIT));
     const stdoutResult = readBounded(stdoutPath, outputLimit);
@@ -99,6 +116,7 @@ export async function executeCommand(input, options = {}) {
       durationMs: Date.now() - startedAt,
       stdoutBytes: stdoutResult.bytes,
       stderrBytes: stderrResult.bytes,
+      environmentPolicy: environment.policy,
     };
     const mode = input.outputMode ?? 'auto';
     const shouldIndex = mode === 'indexed' || (mode === 'auto' && totalBytes > directLimit);
@@ -122,12 +140,13 @@ export async function executeCommand(input, options = {}) {
       ...(mode === 'discard' ? {} : shouldIndex ? { indexed } : { stdout: stdoutResult.text, stderr: stderrResult.text, truncated: stdoutResult.truncated || stderrResult.truncated }),
     };
   } finally {
-    rmSync(dir, { recursive: true, force: true });
+    rmSync(dir, { recursive: true, force: true, maxRetries: 3, retryDelay: 10 });
   }
 }
 
 export async function executeFile(input, options = {}) {
   const path = resolveWithinRoot(options.root || process.cwd(), input.path);
+  const selectorEnvironment = composeEnvironment({ inherited: options.env ?? process.env, overrides: input.env ?? {}, mode: input.environmentMode ?? 'inherit-filtered-v1', allow: input.allowedEnv ?? [] }).env;
   const dir = mkdtempSync(join(tmpdir(), 'dotdotgod-context-file-'));
   try {
     let executable;
@@ -135,12 +154,12 @@ export async function executeFile(input, options = {}) {
     if (input.language === 'python') {
       const wrapper = join(dir, 'run.py');
       writeFileSync(wrapper, `from pathlib import Path\nFILE_CONTENT = Path(${JSON.stringify(path)}).read_text(encoding='utf-8', errors='replace')\n${input.code}\n`);
-      executable = process.env.PYTHON || 'python3';
+      executable = selectorEnvironment.PYTHON || 'python3';
       args = [wrapper];
     } else if (input.language === 'shell') {
       const wrapper = join(dir, 'run.sh');
       writeFileSync(wrapper, `FILE_PATH=${JSON.stringify(path)}\nexport FILE_PATH\n${input.code}\n`);
-      executable = process.env.SHELL || '/bin/sh';
+      executable = selectorEnvironment.SHELL || '/bin/sh';
       args = [wrapper];
     } else {
       const wrapper = join(dir, 'run.mjs');
@@ -150,7 +169,7 @@ export async function executeFile(input, options = {}) {
     }
     return await executeCommand({ ...input, executable, args, command: undefined, shell: false, cwd: input.cwd ?? options.root }, options);
   } finally {
-    rmSync(dir, { recursive: true, force: true });
+    rmSync(dir, { recursive: true, force: true, maxRetries: 3, retryDelay: 10 });
   }
 }