@fadhilp/stateql 0.13.0 → 0.13.1

package/docs/databases.md

@@ -0,0 +1,225 @@
+# Database support
+
+[Home](../README.md) · [Usage](usage.md) · [Database support](databases.md) · [TypeScript library](library.md)
+
+- [Driver notes](#driver-notes)
+- [SQL statement support](#sql-statement-support)
+- [Dialect upserts](#dialect-upserts)
+- [PostgreSQL diagnostics and maintenance](#postgresql-diagnostics-and-maintenance)
+- [SQLite and MySQL diagnostics and maintenance](#sqlite-and-mysql-diagnostics-and-maintenance)
+- [Native MongoDB](#native-mongodb)
+- [Native Redis](#native-redis)
+
+Connect using the [usage guide](usage.md#connections-and-profiles). All writes
+require a read-write connection; SQL/MongoDB approvals and retry rules are
+covered under [write safety](usage.md#write-safety).
+
+## Driver notes
+
+- **SQLite:** use a filesystem path for direct connections or `sqlite:` for an
+  environment-backed path.
+- **PostgreSQL:** StateQL preserves strict TLS verification by normalizing
+  `sslmode=prefer`, `require`, and `verify-ca` to `verify-full` before opening
+  the adapter. Use `sslmode=verify-full` explicitly for clarity. Setting
+  `uselibpqcompat=true` opts out and keeps libpq-compatible SSL semantics.
+- **MySQL:** uses positional `?` parameters. MariaDB compatibility is not
+  currently claimed.
+- **MongoDB:** supports `mongodb://` and `mongodb+srv://` URLs with an explicit
+  database path. SQL methods are rejected; use the native MongoDB methods below.
+- **Redis:** supports `redis://` and TLS-backed `rediss://` URLs. SQL methods and
+  legacy `inspect` commands are unavailable; use native commands and catalog
+  inspection instead.
+
+## SQL statement support
+
+StateQL parses and classifies one `SELECT`, `INSERT`, `REPLACE`, `UPDATE`,
+`DELETE`, `CREATE`, `ALTER`, `DROP`, or `TRUNCATE` statement through
+`node-sql-parser`, subject to dialect support and safety validation. Reads use
+`query`; mutations and DDL use durable `exec`/`plan`/`apply` operations. Raw
+transaction-control SQL is unsupported: use
+[staged transactions](usage.md#transactions), not `BEGIN` or `COMMIT` statements.
+MongoDB and Redis use separate native command APIs.
+
+Unrecognized or parser-unsupported forms remain blocked. The
+[SQL command roadmap](../SQL_COMMAND_ROADMAP.md) describes future categories,
+not permission to execute them.
+
+## Dialect upserts
+
+PostgreSQL `INSERT ... ON CONFLICT DO NOTHING|UPDATE` and MySQL `INSERT ... ON
+DUPLICATE KEY UPDATE` are structurally validated and recorded with statement
+type `upsert`. Finite `VALUES` and MySQL `INSERT ... SET` sources use normal
+write policy. An update-upsert fed by `SELECT` requires `--allow-unbounded`
+because its candidate row count is not statically bounded. Upserts support direct
+`exec`, `plan`/`apply`, and staged transactions; hidden additional writes are rejected.
+
+MySQL `INSERT IGNORE` and SQLite `INSERT OR IGNORE` remain non-overwriting
+inserts. SQLite `INSERT OR REPLACE` retains destructive-operation approval,
+while SQLite modern `ON CONFLICT ... DO UPDATE` and every `MERGE` form remain
+blocked until the parser can expose their complete mutation structure.
+
+## PostgreSQL diagnostics and maintenance
+
+Run PostgreSQL plans through `query`:
+
+```bash
+stql query "EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) SELECT * FROM jobs WHERE id = 42"
+```
+
+Plain `EXPLAIN` may plan a structurally validated `SELECT`, `INSERT`, `UPDATE`,
+or `DELETE`. Because `EXPLAIN ANALYZE` executes its inner statement, StateQL
+accepts only a validated read-only `SELECT`; `SELECT INTO`, writing CTEs, and
+mutations are rejected. Diagnostics execute inside PostgreSQL `BEGIN READ ONLY`
+and are never reused from cache. `--cache require` therefore returns
+`CACHE_MISS` without executing the diagnostic.
+
+Legacy `EXPLAIN [ANALYZE] [VERBOSE] statement` is also supported. Option lists
+allow PostgreSQL boolean diagnostic options plus `FORMAT` and `SERIALIZE`;
+unknown, duplicate, or malformed options fail closed. PostgreSQL's read-only
+transaction protects database writes, but cannot contain external effects from
+user-defined or incorrectly labelled functions.
+
+StateQL supports PostgreSQL 14–18. Top-level `VALUES` is a bounded read and
+accepts normal PostgreSQL positional parameters. It is conservatively
+non-cacheable because expressions may be volatile. The following narrow `SHOW`
+allowlist is also available as non-cacheable diagnostics:
+`server_version`, `server_version_num`, `transaction_read_only`,
+`transaction_isolation`, and `default_transaction_isolation`. `SHOW ALL` and
+other settings remain blocked. Syntax accepted by StateQL but introduced by a
+newer PostgreSQL release may be rejected safely by an older server.
+
+`VACUUM`, `ANALYZE`, `REINDEX`, and `CLUSTER` are PostgreSQL maintenance writes:
+
+```bash
+stql exec "VACUUM (ANALYZE) public.jobs" --allow-destructive
+stql plan "REINDEX TABLE public.jobs" --allow-destructive
+```
+
+The PostgreSQL 14–18 grammar includes parenthesized `REINDEX CONCURRENTLY`,
+PostgreSQL 16 `BUFFER_USAGE_LIMIT` for `VACUUM`/`ANALYZE`, optional
+`DATABASE`/`SYSTEM` reindex names, and PostgreSQL 18 `ONLY table *` maintenance
+targets. Memory sizes accept an integer number of kilobytes or a quoted
+`B|kB|MB|GB|TB` value. Older servers may reject newer forms after dispatch, so
+StateQL retains conservative unknown-outcome handling.
+
+They require a read-write connection and `--allow-destructive`, reject StateQL
+parameters and optimistic row-count preconditions, and run as individually
+tracked autocommit operations. They cannot be staged in a StateQL transaction,
+even where a PostgreSQL variant could run inside a database transaction. A
+timeout, cancellation, or error after dispatch is reported as `OUTCOME_UNKNOWN`;
+inspect database state before replaying it.
+
+## SQLite and MySQL diagnostics and maintenance
+
+SQLite supports `EXPLAIN QUERY PLAN` for structurally read-only `SELECT`
+statements. MySQL supports `EXPLAIN SELECT` plus bare `SHOW TABLES`,
+`SHOW COLUMNS FROM table`, and `SHOW INDEX|INDEXES FROM table`. Broader
+`EXPLAIN`, `SHOW`, and write-bearing forms remain blocked.
+
+MySQL executable comments (`/*! ... */`) are rejected throughout SQL. Because
+StateQL does not assume a server `sql_mode`, quoting that could expose these
+comments under `ANSI_QUOTES` or `NO_BACKSLASH_ESCAPES` is also rejected.
+
+These diagnostics use `query`, work with read-only connections, preserve the
+original statement instead of applying StateQL's limiting SQL wrapper, and are
+never reused from cache. Materialized results still receive StateQL's row and
+byte checks.
+
+SQLite also supports bare `VACUUM`, plus `ANALYZE [target]` and
+`REINDEX [target]` with at most one unqualified or double-quoted target:
+
+```bash
+stql exec "ANALYZE jobs" --allow-destructive
+stql exec "REINDEX jobs_created_at_idx" --allow-destructive
+stql exec "VACUUM" --allow-destructive
+```
+
+These commands require a read-write connection, reject parameters, run as
+individually tracked autocommit operations, and cannot be staged. A timeout,
+cancellation, or error after dispatch is reported as `OUTCOME_UNKNOWN`.
+`VACUUM INTO`, schema-qualified targets, paths, `ATTACH`, and arbitrary `PRAGMA`
+remain blocked.
+
+MySQL supports one optionally qualified bare or backtick-quoted target for
+`ANALYZE TABLE`, `OPTIMIZE TABLE`, and `CHECK TABLE`:
+
+```bash
+stql exec "ANALYZE TABLE jobs" --allow-destructive
+stql plan "OPTIMIZE TABLE jobs" --allow-destructive
+stql query "CHECK TABLE jobs"
+```
+
+`ANALYZE` and `OPTIMIZE` are durable autocommit writes requiring a read-write
+connection and destructive approval; server-reported error rows become known
+failed operations, while timeout or cancellation after dispatch remains
+`OUTCOME_UNKNOWN`. `CHECK TABLE` is an unwrapped, non-cacheable autocommit read
+that works on read-only connections and retains normal result limits. All three
+reject StateQL parameters, options, multiple targets, and additional
+statements, and none can run during a staged transaction.
+## Native MongoDB
+
+MongoDB commands use official Extended JSON (EJSON), so BSON values survive the
+CLI boundary:
+
+```bash
+stql mongo query '{"operation":"find","collection":"users","filter":{"_id":{"$oid":"507f1f77bcf86cd799439011"}}}'
+stql mongo exec '{"operation":"updateOne","collection":"users","filter":{"_id":{"$oid":"507f1f77bcf86cd799439011"}},"update":{"$set":{"seen_at":{"$date":"2026-01-01T00:00:00Z"}}}}'
+stql mongo plan '{"operation":"deleteMany","collection":"users","filter":{"disabled":true}}' --allow-destructive
+```
+
+The TypeScript equivalents are `mongoQuery(command)`, `mongoExec(command)`, and
+`mongoPlan(command)`. Supported reads are `find` and `aggregate`; writes are
+`insertOne`, `insertMany`, `updateOne`, `updateMany`, `replaceOne`, `deleteOne`,
+and `deleteMany`. Result documents are JSON-safe, order-preserving EJSON: for example,
+ObjectIds and dates appear as `{ "$oid": "..." }` and
+`{ "$date": { "$numberLong": "..." } }`.
+
+Empty update, replacement, or delete filters require `--allow-unbounded`;
+deletes and replacements also require `--allow-destructive`. Mongo inspection
+accepts `collections`, `collection`, `columns`, `indexes`, and `constraints`
+(`schema` and `table` remain aliases shared with SQL drivers). MongoDB cache
+confidence is TTL-based: external writes are not detected, so use
+`--cache bypass` for a fresh read.
+
+```ts
+const result = await stateql.mongoQuery({
+  operation: "find",
+  collection: "users",
+  filter: { active: true },
+  options: { sort: { _id: 1 }, limit: 50 },
+});
+```
+
+## Native Redis
+
+Redis/Rediss URLs support URL database selection, password or ACL username,
+and TLS (`rediss`). Supply credential-bearing URLs through `--env` or a trusted
+full-URL credential reference. Library/batch callers may instead use a
+[password reference](library.md#password-references) with a password-free target.
+Native commands accept `{command: string, args?: string[]}`:
+
+```bash
+stql redis query '{"command":"GET","args":["app:status"]}'
+stql redis exec '{"command":"SET","args":["app:status","ready"]}' --idempotency-key status-ready
+stql redis plan '{"command":"SET","args":["app:status","paused"]}'
+stql objects key --search app --limit 50
+stql object key app:status
+```
+
+Writes and plans require a read-write connection. The TypeScript methods are:
+
+- `redisQuery`: `GET`, `MGET`, `TYPE`, `EXISTS`, `TTL`, `PTTL`, `HGET`, `HMGET`,
+  bounded `LRANGE`, and bounded `SCAN`/`HSCAN`/`SSCAN`/`ZSCAN`.
+- `redisExec` and `redisPlan`: one-key `SET`, `DEL`, `HSET`, `HDEL`, `LPUSH`,
+  `RPUSH`, `SADD`, `SREM`, `ZADD`, or `ZREM` mutation.
+- `describeObject({kind:"key",name})`: bounded string/hash/list/set/zset value
+  inspection with TTL and continuation metadata where applicable.
+
+Arguments are UTF-8 strings, at most 100 values/256 KiB; materialized replies are
+at most 1 MiB. `KEYS`, scripts, modules, pub/sub, blocking commands, admin/flush,
+and arbitrary commands are rejected. Key discovery always uses SCAN. A Redis
+plan snapshots one bounded key and `apply` uses an isolated `WATCH` + one-command
+`MULTI/EXEC`; a pre-apply content or expiry change returns `ROW_CONFLICT` and is
+never retried automatically. Direct `redisExec` has Redis single-command
+atomicity only. Redis has no SQL rollback or StateQL staged transaction support;
+a lost write/EXEC reply is reported as `OUTCOME_UNKNOWN` and remains blocked.

package/docs/library.md

@@ -0,0 +1,268 @@
+# TypeScript library
+
+[Home](../README.md) · [Usage](usage.md) · [Database support](databases.md) · [TypeScript library](library.md)
+
+- [Getting started](#getting-started)
+- [Responses and command context](#responses-and-command-context)
+- [Actor workspaces](#actor-workspaces)
+- [Credential resolution](#credential-resolution)
+- [Safe profile updates](#safe-profile-updates)
+
+## Getting started
+
+Install with `npm install @fadhilp/stateql`. This example connects to an existing
+SQLite database containing a `users` table, queries it, and filters the stored
+result. `forActor()` resolves or creates a session; it does not establish a
+database connection for a new session.
+
+```ts
+import { StateQL } from "@fadhilp/stateql";
+
+const stateql = StateQL.forActor({
+  home: "./.stql",
+  actor: "audit",
+});
+
+try {
+  const connected = await stateql.connect("./app.sqlite", { readOnly: true });
+  if (!connected.ok) throw new Error(connected.error.message);
+
+  const result = await stateql.query(
+    "SELECT id, email FROM users WHERE status = ? ORDER BY id LIMIT 50",
+    { params: ["active"], timeoutMs: 5_000 },
+  );
+  if (!result.ok) throw new Error(result.error.message);
+
+  const filtered = await stateql.filter(result.data.result_id, "email LIKE ?", {
+    params: ["%@example.com"],
+  });
+  if (!filtered.ok) throw new Error(filtered.error.message);
+
+  console.log(filtered.data.preview);
+} finally {
+  stateql.close();
+}
+```
+
+`close()` releases the client's local store; it does not disconnect or delete
+the durable session. Await active commands before closing. Use `disconnect()`
+only when you intend to remove the session's shared active connection.
+
+Options include `timeoutMs`, `credentialTimeoutMs`, `maxResultBytes`,
+`maxStateBytes`, `cacheTtlSeconds`, and `resultTtlSeconds`. Database calls also
+accept `signal` for cancellation. See [limits](usage.md#result-lifetime-and-limits)
+and [deadlines](usage.md#deadlines-and-cancellation) for defaults and behavior.
+The same [write safety](usage.md#write-safety) and
+[database restrictions](databases.md) apply to library calls.
+
+## Responses and command context
+
+Library responses retain the full response envelope regardless of the CLI
+output mode. Check `response.ok` before reading `response.data`; failures expose
+`response.error`. Queries return typed `ResultData` with `result_id`, `preview`,
+and result metadata. Reuse `result_id` with methods such as `filter`, `rows`,
+`count`, and `exportResult`; the CLI instead presents its primary ID as `handle`.
+
+The following snippets assume an open client and an appropriate connection.
+
+Hosts that dispatch batch-shaped commands can attach trusted metadata out of
+band. `origin` is audit/source metadata only; it never changes actor membership,
+workspace access, or write authorization.
+
+```ts
+const controller = new AbortController();
+await stateql.executeCommand(
+  { command: "query", sql: "SELECT id, email FROM users ORDER BY id LIMIT 50", cache: "bypass" },
+  { origin: "user", signal: controller.signal },
+);
+
+const userHistory = await stateql.history(50, { origin: "user" });
+await stateql.executeCommand(
+  { command: "history", limit: 50, history_origin: "user" },
+  { origin: "model" },
+);
+```
+
+Supported origins are `legacy`, `user`, `model`, `system`, and `api`. Existing
+direct calls and `executeCommand(command)` calls are recorded as `legacy`.
+`history_origin` is only a retrieval filter; putting an `origin` field in a
+`BatchCommand` cannot attribute the command. `batch` accepts the same trusted
+context as `options.executionContext` for all commands in that batch.
+
+## Actor workspaces
+
+`StateQL.forWorkspace(...)` is a trusted-host primitive that atomically creates
+or reopens a durable workspace, attaches the requested actor, and returns a
+client bound to that actor:
+
+```ts
+const stateql = StateQL.forWorkspace({
+  home: "./.stql",
+  workspace: "team-audit",
+  actor: "audit-worker-1",
+  credentialResolver,
+  signal,
+});
+```
+
+Repeated opens of the same actor and workspace are idempotent. An actor already
+attached elsewhere fails with a `StateQLError` whose code is
+`PERMISSION_DENIED`; StateQL never moves or merges it. All actor options,
+including limits, credential resolution, cancellation, `home`, and `now`, are
+preserved. The workspace name also reserves a same-named actor identity for
+legacy compatibility, so workspace and actor identifiers must be globally
+collision-free. The returned client is still bound only to `actor`, preserving
+plan, transaction, operation, and history ownership.
+
+`StateQL.forActor(...)` retains its existing behavior: it resolves the actor's
+attached session directly from StateQL storage and creates a legacy-compatible
+session named after the actor on first use. Use `new StateQL({ session, actor })`
+when the session and membership are already known.
+
+Membership management and `forWorkspace` are library-only host capabilities,
+not batch or CLI commands. Existing member-authorized management remains
+available through `linkActor(session, actorId)`, `unlinkActor(session, actorId)`,
+`listActors(session)`, and `resolveActor(actorId)`. Integrations should ask for
+user confirmation before changing membership or the shared connection; a host
+calling `forWorkspace` is responsible for authorizing that workspace access.
+
+## Credential resolution
+
+Library integrations can resolve environment-variable names, opaque full-URL
+credential references, or password-only references through a trusted approval
+or secret-storage layer instead of mutating `process.env`.
+
+Integrations pinned to an older published package should gate setup before
+sending `password_ref`:
+
+```ts
+if ((StateQL.passwordReferenceVersion ?? 0) < 1) {
+  throw new Error("Installed StateQL does not support password references.");
+}
+```
+
+`passwordReferenceVersion = 1` guarantees the password-only resolver request,
+validation, persistence, reconnect, and redaction contract documented below.
+
+In this adapter example, `credentialBroker` is supplied by the host application;
+it is not a StateQL export.
+
+```ts
+import {
+  CredentialResolutionError,
+  StateQL,
+  type CredentialRequest,
+} from "@fadhilp/stateql";
+
+async function resolveCredential(
+  request: CredentialRequest,
+): Promise<string | undefined> {
+  const approved = await credentialBroker.request({
+    reference: request.reference,
+    source: request.source ?? "secret_env",
+    actor: request.actorId,
+    session: request.session.id,
+    operation: request.operation,
+    access: request.access,
+    signal: request.signal,
+  });
+
+  if (approved.denied) throw new CredentialResolutionError("denied");
+  return approved.value;
+}
+
+const stateql = StateQL.forActor({
+  actor: "agent-session-id",
+  credentialResolver: resolveCredential,
+});
+```
+
+Credential resolution has its own two-minute default deadline
+(`credentialTimeoutMs`) and remains cancellable through `request.signal`.
+The database-operation timeout begins after a credential is resolved.
+
+When no custom resolver is configured, StateQL reads only `secret_env`
+references from `process.env`; `credential_ref` and `password_ref` never fall
+back to the environment. A configured resolver is authoritative for all
+sources: returning `undefined` produces `CREDENTIAL_UNAVAILABLE` and never falls
+back to the process environment. Resolver requests retain `reference` and
+include `source` (`secret_env`, `credential_ref`, or `password_ref`); source may
+be omitted only on legacy secret-environment request objects. A `password_ref`
+request additionally includes the exact password-free effective `target`.
+Resolvers may throw `CredentialResolutionError` with `denied`, `cancelled`,
+`timeout`, or `unavailable` to produce controlled, secret-free failures. Unknown
+resolver errors are replaced with a generic `CREDENTIAL_RESOLUTION_FAILED`
+response.
+
+StateQL calls the resolver only immediately before database access, after SQL
+safety and duplicate checks. Requests contain actor and session identity, the
+operation's effective read/write access, an abort signal, and sanitized
+connection metadata.
+
+For `secret_env` and `credential_ref`, returned values must be complete
+PostgreSQL, MySQL, MongoDB, or Redis URLs, or explicit `sqlite:` sources. For
+`password_ref`, the resolver returns only the password; an explicit empty string
+is a resolved password, while `undefined` fails closed. StateQL percent-encodes
+and injects only that password into the original target for adapter use, leaving
+all nonsecret URL/TLS/CA bytes unchanged. It persists only the original target
+and reference. Resolved credentials never enter connection metadata, history,
+snapshots, cache keys, responses, or stored errors.
+
+Hosts remain responsible for approval policy, binding lifetime, revocation,
+and keeping values out of their own logs and model-visible data.
+
+For writes, credential resolution happens after StateQL atomically reserves the
+operation for duplicate protection. A resolution failure keeps a non-executed
+`failed` audit record, does not consume the idempotency key, and permits a safe
+retry.
+
+### Password references
+
+Library callers can keep nonsecret endpoint, username, database, TLS, and CA
+options in the literal URL while resolving only its password:
+
+```ts
+await stateql.connect(
+  "postgres://app@db.example/app?sslmode=verify-full&sslrootcert=/etc/app-ca.pem",
+  { passwordRef: "vault://database/app/password", readOnly: true },
+);
+```
+
+The same field is accepted by `addProfile`, `updateProfile`, and batch
+`connect`/`profile.add`/`profile.update` commands (snake case in batch input).
+Targets with an embedded password or query parameters that override endpoint or
+credential fields are rejected before credential resolution or driver access.
+
+## Safe profile updates
+
+`updateProfile(name, options)` accepts optional `target`, `secretEnv`,
+`credentialRef`, `passwordRef`, and `readOnly` fields. The four string fields
+also accept `null` for explicit clearing.
+
+```ts
+const updated = await stateql.updateProfile("production", { readOnly: true });
+if (!updated.ok) throw new Error(updated.error.message);
+```
+
+Omitting all source and password-reference fields keeps the existing source and
+adjunct reference. Supplying any source field replaces the source atomically:
+exactly one non-null source is required and the other source columns are
+cleared. An omitted `passwordRef` is preserved when the existing literal target
+is unchanged; changing the target clears it unless the update explicitly supplies
+a replacement. Replacing the source with `secretEnv` or `credentialRef` clears
+it. An explicit non-null password reference combined with either
+reference-backed source is rejected. Direct URLs
+with embedded passwords or secret-like query parameters are rejected.
+`profile.list/show/update` return only
+`{profile,target,secret_env,credential_ref,password_ref,read_only}`. Profile
+changes affect subsequent `connect` calls and do not silently mutate an
+already-open connection.
+
+The `password_refs_v1` migration adds nullable `password_ref` columns to both
+profiles and connections and enforces that they accompany only literal target
+configuration. It rejects incompatible profile schemas/rows instead of dropping
+references. Downgrading a state home containing password references is
+unsupported: older binaries do not resolve this source and may attempt the
+password-free target using ambient/trust authentication; constraint-protected
+source replacements may also fail. Use the same or newer StateQL binary, or
+explicitly clear all password references before downgrade.