@vaultgradient/pq-agent 0.1.0

package/README.md

@@ -0,0 +1,152 @@
+# @vaultgradient/pq-agent
+
+**The agent that runs inside your network so AI agents can query your data — safely, audibly, and over a single outbound connection.**
+
+`pq-agent` is half of [PipeQuery Cloud](https://pipequery.cloud). The other half is the hosted control plane your AI tools (Claude Desktop, Cursor, Copilot, custom MCP clients) connect to. The agent sits between them and your data sources, so:
+
+- **Your data stays on your infra.** Source credentials live in *your* `pipequery.yaml`. The control plane never sees them.
+- **No inbound ports.** The agent dials *out* to the control plane over WSS:443. Your firewall stays closed.
+- **Every read is audited.** Each query produces a tamper-evident audit entry persisted in the control plane.
+
+---
+
+## Install
+
+```bash
+npm install -g @vaultgradient/pq-agent
+```
+
+Requires Node 20 or later.
+
+## Configure
+
+Create a `pipequery.yaml` next to your data:
+
+```yaml
+sources:
+  orders:
+    type: postgres
+    url: ${DATABASE_URL}
+    query: SELECT * FROM orders
+    interval: 30s
+
+endpoints:
+  /top-orders:
+    query: orders | sort(total desc) | first(10)
+```
+
+`pq-agent` reuses the [PipeQuery](https://github.com/andreadito/pipequery) source library: Postgres, MySQL, SQLite, Snowflake, ClickHouse, MongoDB, Kafka, REST APIs, WebSocket streams, CSV / JSON files, inline data. Connection strings support `${ENV_VAR}` interpolation so secrets stay out of yaml.
+
+## Run
+
+You need two things from your control-plane operator (or your own `pq-cloud-admin` if you self-host):
+
+1. **Control-plane URL** — e.g. `wss://pipequery-cloud.fly.dev/agent`
+2. **Agent token** — issued via `pq-cloud-admin issue-agent-token`. The token is shown **once**.
+
+Then:
+
+```bash
+export PQ_AGENT_CONTROL_PLANE_URL=wss://your-cp.fly.dev/agent
+export PQ_AGENT_ENROLLMENT_TOKEN=<your-agent-token>
+export PQ_AGENT_CONFIG=./pipequery.yaml
+
+pq-agent run
+```
+
+You should see:
+
+```
+[agent] loading config from /path/to/pipequery.yaml
+[agent] loaded 1 source(s), 1 endpoint(s)
+[agent] connected to wss://your-cp.fly.dev/agent (auth=enrollment)
+```
+
+The agent now serves MCP traffic from the control plane against your sources. Your AI client uses the **MCP URL** (e.g. `https://your-cp.fly.dev/mcp`) plus an **api_key** — issued separately, also via the admin CLI.
+
+### As a service (systemd / launchd)
+
+For production, run `pq-agent` under a process supervisor. Example systemd unit:
+
+```ini
+[Unit]
+Description=PipeQuery agent
+After=network.target
+
+[Service]
+Environment=PQ_AGENT_CONTROL_PLANE_URL=wss://your-cp.fly.dev/agent
+Environment=PQ_AGENT_ENROLLMENT_TOKEN=...
+Environment=PQ_AGENT_CONFIG=/etc/pipequery/pipequery.yaml
+ExecStart=/usr/bin/pq-agent run
+Restart=on-failure
+User=pipequery
+
+[Install]
+WantedBy=multi-user.target
+```
+
+---
+
+## How it works
+
+```
+┌─────────────────────┐         ┌──────────────────────────┐
+│   Claude Desktop    │ HTTPS   │   PipeQuery Cloud        │
+│   Cursor / Copilot  │ ──────► │   <your-cp>.fly.dev/mcp  │
+│   (any MCP client)  │         │                          │
+└─────────────────────┘         │   ┌────────────────────┐ │
+                                │   │ Tenant registry    │ │
+                                │   │ Audit log (Postgres)│ │
+                                │   └────────────────────┘ │
+                                │             │            │
+                                │             │ WSS (long-lived)
+                                │             ▼            │
+                                └─────────────┬────────────┘
+                                              │
+                                  outbound :443 only
+                                              │
+                                              ▼
+                                ┌──────────────────────────┐
+                                │  pq-agent (this package) │
+                                │  Inside YOUR network     │
+                                │                          │
+                                │  ┌────────────────────┐  │
+                                │  │ pipequery.yaml     │  │
+                                │  │ source credentials │  │
+                                │  └────────────────────┘  │
+                                │             │            │
+                                │             ▼            │
+                                │   Postgres / Snowflake /  │
+                                │   Kafka / REST APIs / etc │
+                                └──────────────────────────┘
+```
+
+The agent dials the control plane over WSS:443. When the AI client makes an MCP tool call, the control plane routes it through the WS to the agent, the agent executes the query locally using whichever source adapter is configured (with push-down to native SQL where supported), and the result streams back. Source credentials never cross the public internet — only the query results do.
+
+## CLI reference
+
+```
+pq-agent run [options]
+
+Options:
+  --url <url>           Control-plane WSS URL
+                        (env: PQ_AGENT_CONTROL_PLANE_URL)
+  --config <path>       Path to pipequery.yaml
+                        (env: PQ_AGENT_CONFIG, default: ./pipequery.yaml)
+  --token-file <path>   Where to read/write the agent token
+                        (env: PQ_AGENT_TOKEN_FILE,
+                         default: $XDG_DATA_HOME/pipequery/agent.token)
+  --insecure            Allow ws:// for localhost-only dev. Refused
+                        against any non-localhost host.
+```
+
+## Troubleshooting
+
+- **`Refusing to connect: URL must use wss://`** — the production control plane uses TLS; `ws://` is allowed only for localhost dev.
+- **`WS closed 4001`** — your agent token is invalid or revoked. Get a fresh one from your operator (`pq-cloud-admin issue-agent-token`).
+- **`WS closed 4003`** — protocol version mismatch. Upgrade `pq-agent` to the latest.
+- **`AUDIT CHAIN BREAK` in the CP logs after a restart** — known v1 limitation: the agent's chain resets to zero on restart. Fix is on the roadmap (disk-persistence of the chain head).
+
+## License
+
+Proprietary. © Vault Gradient. Commercial licensing inquiries: [pipequery.cloud](https://pipequery.cloud).

package/dist/client.d.ts

@@ -0,0 +1,22 @@
+import { AgentRuntime } from './runtime.js';
+import '@vaultgradient/pipequery-cli/sources';
+
+interface ClientOptions {
+    url: string;
+    token: string;
+    runtime: AgentRuntime;
+    tenantId?: string;
+    agentId?: string;
+    allowInsecure?: boolean;
+    requestTimeoutMs?: number;
+}
+interface Client {
+    start(): Promise<void>;
+    stop(): Promise<void>;
+    on(event: 'open', listener: () => void): void;
+    on(event: 'close', listener: (code: number) => void): void;
+    on(event: 'error', listener: (err: Error) => void): void;
+}
+declare function createClient(opts: ClientOptions): Client;
+
+export { type Client, type ClientOptions, createClient };

package/dist/client.js

@@ -0,0 +1,501 @@
+// src/client.ts
+import { randomUUID as randomUUID2 } from "crypto";
+import { EventEmitter } from "events";
+import WebSocket from "ws";
+
+// ../protocol/dist/errors.js
+var WS_CLOSE_CODES = {
+  GOING_AWAY: 1001,
+  INTERNAL_ERROR: 1011,
+  INVALID_AGENT_TOKEN: 4001,
+  TENANT_NOT_ENROLLED: 4002,
+  PROTOCOL_VERSION_MISMATCH: 4003
+};
+var NO_RECONNECT_CLOSE_CODES = /* @__PURE__ */ new Set([
+  WS_CLOSE_CODES.INVALID_AGENT_TOKEN,
+  WS_CLOSE_CODES.TENANT_NOT_ENROLLED,
+  WS_CLOSE_CODES.PROTOCOL_VERSION_MISMATCH
+]);
+
+// ../protocol/dist/version.js
+var PROTOCOL_VERSION = 1;
+var PROTOCOL_VERSION_HEADER = "x-pq-protocol-version";
+
+// src/audit.ts
+import { createHash } from "crypto";
+var ZERO_HASH = "0".repeat(64);
+function canonicalJson(value) {
+  if (value === null) return "null";
+  if (typeof value === "number" || typeof value === "boolean") {
+    return JSON.stringify(value);
+  }
+  if (typeof value === "string") return JSON.stringify(value);
+  if (Array.isArray(value)) {
+    return "[" + value.map(canonicalJson).join(",") + "]";
+  }
+  if (typeof value === "object") {
+    const obj = value;
+    const keys = Object.keys(obj).filter((k) => obj[k] !== void 0).sort();
+    const parts = keys.map(
+      (k) => `${JSON.stringify(k)}:${canonicalJson(obj[k])}`
+    );
+    return "{" + parts.join(",") + "}";
+  }
+  throw new Error(`canonicalJson: unsupported value of type ${typeof value}`);
+}
+function computeHash(prevHash, entry) {
+  return createHash("sha256").update(prevHash).update(canonicalJson(entry)).digest("hex");
+}
+var AuditEmitter = class {
+  constructor(cfg) {
+    this.cfg = cfg;
+  }
+  cfg;
+  prevHash = ZERO_HASH;
+  emit(partial) {
+    const { ts: tsOverride, ...rest } = partial;
+    const entry = {
+      tenant_id: this.cfg.tenantId,
+      agent_id: this.cfg.agentId,
+      ts: tsOverride ?? (/* @__PURE__ */ new Date()).toISOString(),
+      ...rest
+    };
+    const prev_hash = this.prevHash;
+    const hash = computeHash(prev_hash, entry);
+    this.prevHash = hash;
+    return { entry, prev_hash, hash };
+  }
+  // For tests / observability.
+  currentHash() {
+    return this.prevHash;
+  }
+};
+
+// src/dispatch.ts
+import { randomUUID } from "crypto";
+var REQUEST_TIMEOUT_MS = 3e4;
+var MAX_ROWS = 1e5;
+var MAX_BYTES = 10 * 1024 * 1024;
+async function dispatch(inbound, opts) {
+  switch (inbound.type) {
+    case "ping":
+      return { outbound: [{ type: "pong", id: inbound.id, payload: {} }] };
+    case "request.list_sources":
+      return wrapRequest(inbound.id, opts, () => listSources(opts.runtime));
+    case "request.describe_source":
+      return wrapRequest(
+        inbound.id,
+        opts,
+        () => describeSource(opts.runtime, inbound.payload.name, inbound.payload.sample_size)
+      );
+    case "request.list_endpoints":
+      return wrapRequest(inbound.id, opts, () => listEndpoints(opts.runtime));
+    case "request.call_endpoint":
+      return wrapRequest(
+        inbound.id,
+        opts,
+        () => callEndpoint(opts.runtime, inbound.payload.path)
+      );
+    case "request.query":
+      return wrapRequest(
+        inbound.id,
+        opts,
+        () => runQuery(opts.runtime, inbound.payload.expression)
+      );
+    case "control.rotate_token":
+    case "control.update_config":
+    case "control.shutdown":
+      return { outbound: [] };
+  }
+}
+async function wrapRequest(requestId, opts, handler) {
+  const timeoutMs = opts.timeoutMs ?? REQUEST_TIMEOUT_MS;
+  const startedAt = Date.now();
+  let result;
+  try {
+    result = await Promise.race([
+      handler(),
+      timeoutAfter(timeoutMs)
+    ]);
+  } catch (err) {
+    if (err instanceof TimeoutError) {
+      return {
+        outbound: [
+          {
+            type: "response.error",
+            id: requestId,
+            payload: {
+              code: "timeout",
+              message: `request exceeded ${timeoutMs}ms`
+            }
+          }
+        ]
+      };
+    }
+    return {
+      outbound: [
+        {
+          type: "response.error",
+          id: requestId,
+          payload: {
+            code: "execution",
+            message: err instanceof Error ? err.message : String(err)
+          }
+        }
+      ]
+    };
+  }
+  const latency_ms = Date.now() - startedAt;
+  if (!result.ok) {
+    return {
+      outbound: [
+        {
+          type: "response.error",
+          id: requestId,
+          payload: { code: result.code, message: result.message }
+        }
+      ]
+    };
+  }
+  const bytes = byteLengthOfJson(result.rows);
+  if (result.rows.length > MAX_ROWS || bytes > MAX_BYTES) {
+    return {
+      outbound: [
+        {
+          type: "response.error",
+          id: requestId,
+          payload: {
+            code: "too_large",
+            message: `result exceeded cap (rows=${result.rows.length}, bytes=${bytes})`,
+            details: {
+              row_count: result.rows.length,
+              bytes,
+              max_rows: MAX_ROWS,
+              max_bytes: MAX_BYTES
+            }
+          }
+        }
+      ]
+    };
+  }
+  return {
+    outbound: [
+      {
+        type: "response.chunk",
+        id: requestId,
+        payload: { rows: result.rows, chunk_index: 0 }
+      },
+      {
+        type: "response.end",
+        id: requestId,
+        payload: {
+          total_rows: result.rows.length,
+          latency_ms,
+          // The published SourceManager.runQuery() returns rows without
+          // exposing whether push-down was used. Slice 4 reports false until
+          // the OSS API surfaces this; the wire field is still in the spec
+          // and consumed by future audit/billing observability.
+          pushed_down: false
+        }
+      }
+    ]
+  };
+}
+var TimeoutError = class extends Error {
+};
+function timeoutAfter(ms) {
+  return new Promise(
+    (_, reject) => setTimeout(() => reject(new TimeoutError()), ms)
+  );
+}
+function byteLengthOfJson(value) {
+  return Buffer.byteLength(JSON.stringify(value), "utf8");
+}
+async function listSources(runtime) {
+  const names = runtime.sources.getSourceNames();
+  const statuses = runtime.sources.getAllStatuses();
+  const rows = names.map((name) => ({
+    name,
+    status: statuses[name] ?? null
+  }));
+  return { ok: true, rows };
+}
+async function describeSource(runtime, name, sampleSize) {
+  const status = runtime.sources.getSourceStatus(name);
+  if (!status) {
+    return { ok: false, code: "not_found", message: `source "${name}" is not configured` };
+  }
+  const data = runtime.sources.getSourceData(name) ?? [];
+  const size = Math.min(data.length, Math.max(1, sampleSize ?? 5));
+  const sample = data.slice(0, size);
+  const fields = /* @__PURE__ */ new Set();
+  for (const row of sample) {
+    if (row && typeof row === "object") {
+      for (const k of Object.keys(row)) fields.add(k);
+    }
+  }
+  return {
+    ok: true,
+    rows: [
+      {
+        name,
+        status,
+        fields: [...fields],
+        sample
+      }
+    ]
+  };
+}
+async function listEndpoints(runtime) {
+  const rows = [...runtime.endpoints.values()];
+  return { ok: true, rows };
+}
+async function callEndpoint(runtime, path) {
+  const endpoint = runtime.endpoints.get(path);
+  if (!endpoint) {
+    return { ok: false, code: "not_found", message: `endpoint "${path}" is not registered` };
+  }
+  return runQuery(runtime, endpoint.query);
+}
+async function runQuery(runtime, expression) {
+  try {
+    const result = await runtime.sources.runQuery(expression);
+    const rows = Array.isArray(result) ? result : [result];
+    return { ok: true, rows };
+  } catch (err) {
+    const message = err instanceof Error ? err.message : String(err);
+    return { ok: false, code: "execution", message };
+  }
+}
+function isInboundEnvelope(value) {
+  if (typeof value !== "object" || value === null) return false;
+  const env = value;
+  return typeof env.type === "string" && typeof env.id === "string" && typeof env.payload === "object" && env.payload !== null;
+}
+
+// src/reconnect.ts
+var Backoff = class {
+  initialMs;
+  ceilingMs;
+  resetAfterStableMs;
+  currentMs;
+  constructor(opts = {}) {
+    this.initialMs = opts.initialMs ?? 1e3;
+    this.ceilingMs = opts.ceilingMs ?? 6e4;
+    this.resetAfterStableMs = opts.resetAfterStableMs ?? 5 * 6e4;
+    this.currentMs = this.initialMs;
+  }
+  // Returns the delay (ms) to wait before the next reconnect attempt, then
+  // doubles the cap for the following attempt (up to ceiling).
+  nextDelayMs(random = Math.random) {
+    const delay = Math.floor(random() * this.currentMs);
+    this.currentMs = Math.min(this.currentMs * 2, this.ceilingMs);
+    return delay;
+  }
+  reset() {
+    this.currentMs = this.initialMs;
+  }
+};
+
+// src/client.ts
+function createClient(opts) {
+  const emitter = new EventEmitter();
+  const backoff = new Backoff();
+  const audit = new AuditEmitter({
+    tenantId: opts.tenantId ?? "dev-tenant",
+    agentId: opts.agentId ?? "dev-agent"
+  });
+  let socket = null;
+  let stopped = false;
+  let reconnectTimer = null;
+  let stableTimer = null;
+  function assertSafeUrl(url) {
+    if (url.startsWith("wss://")) return;
+    if (url.startsWith("ws://")) {
+      if (!opts.allowInsecure) {
+        throw new Error(
+          `Refusing to connect: URL must use wss:// (got ${url}). Pass allowInsecure for local dev only.`
+        );
+      }
+      let host;
+      try {
+        host = new URL(url).hostname;
+      } catch {
+        throw new Error(`Refusing to connect: invalid URL (${url}).`);
+      }
+      const localish = host === "localhost" || host === "127.0.0.1" || host === "::1" || host === "0.0.0.0" || host.endsWith(".local");
+      if (!localish) {
+        throw new Error(
+          `Refusing to connect: --insecure is only allowed for localhost targets, got host="${host}". Use wss:// for ${url}.`
+        );
+      }
+      return;
+    }
+    throw new Error(
+      `Refusing to connect: URL must be ws:// or wss:// (got ${url}).`
+    );
+  }
+  async function connect() {
+    assertSafeUrl(opts.url);
+    const ws = new WebSocket(opts.url, {
+      headers: {
+        Authorization: `Bearer ${opts.token}`,
+        [PROTOCOL_VERSION_HEADER]: String(PROTOCOL_VERSION)
+      }
+    });
+    socket = ws;
+    ws.on("open", () => {
+      stableTimer = setTimeout(
+        () => backoff.reset(),
+        backoff.resetAfterStableMs
+      );
+      emitter.emit("open");
+    });
+    ws.on("message", (raw) => {
+      void handleFrame(ws, raw.toString());
+    });
+    ws.on("close", (code) => {
+      if (stableTimer) {
+        clearTimeout(stableTimer);
+        stableTimer = null;
+      }
+      socket = null;
+      emitter.emit("close", code);
+      if (stopped) return;
+      if (NO_RECONNECT_CLOSE_CODES.has(code)) {
+        emitter.emit(
+          "error",
+          new Error(
+            `WS closed ${code} \u2014 non-recoverable per \xA77. Re-enrollment or operator action required.`
+          )
+        );
+        return;
+      }
+      scheduleReconnect();
+    });
+    ws.on("error", (err) => {
+      emitter.emit("error", err);
+    });
+  }
+  async function handleFrame(ws, raw) {
+    let parsed;
+    try {
+      parsed = JSON.parse(raw);
+    } catch (err) {
+      emitter.emit("error", new Error(`Invalid JSON frame: ${String(err)}`));
+      return;
+    }
+    if (!isInboundEnvelope(parsed)) {
+      emitter.emit("error", new Error("Frame missing required envelope fields"));
+      return;
+    }
+    const inbound = parsed;
+    const shouldAudit = inbound.type.startsWith("request.");
+    const start = Date.now();
+    const result = await dispatch(inbound, {
+      runtime: opts.runtime,
+      ...opts.requestTimeoutMs !== void 0 && {
+        timeoutMs: opts.requestTimeoutMs
+      }
+    });
+    const latency_ms = Date.now() - start;
+    for (const outbound of result.outbound) {
+      send(ws, outbound);
+    }
+    if (shouldAudit) {
+      const auditEnvelope = buildAuditEvent(
+        inbound,
+        result.outbound,
+        latency_ms
+      );
+      send(ws, auditEnvelope);
+    }
+  }
+  function buildAuditEvent(inbound, outbound, latency_ms) {
+    const summary = summarizeOutbound(outbound);
+    let expression;
+    let endpoint_path;
+    if (inbound.type === "request.query") expression = inbound.payload.expression;
+    if (inbound.type === "request.call_endpoint") endpoint_path = inbound.payload.path;
+    const auditEvent = audit.emit({
+      request_id: inbound.id,
+      caller_id: "unknown",
+      // §13 spec gap — see audit.ts header
+      message_type: inbound.type,
+      ...expression !== void 0 && { expression },
+      ...endpoint_path !== void 0 && { endpoint_path },
+      pushed_down: summary.pushed_down,
+      row_count: summary.row_count,
+      bytes: summary.bytes,
+      latency_ms,
+      outcome: summary.outcome,
+      ...summary.error_code !== void 0 && { error_code: summary.error_code }
+    });
+    return {
+      type: "event.audit",
+      id: randomUUID2(),
+      payload: auditEvent
+    };
+  }
+  function send(ws, envelope) {
+    if (ws.readyState !== WebSocket.OPEN) return;
+    ws.send(JSON.stringify(envelope));
+  }
+  function scheduleReconnect() {
+    const delay = backoff.nextDelayMs();
+    reconnectTimer = setTimeout(() => {
+      reconnectTimer = null;
+      if (stopped) return;
+      void connect().catch((err) => {
+        emitter.emit("error", err);
+        scheduleReconnect();
+      });
+    }, delay);
+  }
+  return {
+    async start() {
+      stopped = false;
+      await connect();
+    },
+    async stop() {
+      stopped = true;
+      if (reconnectTimer) {
+        clearTimeout(reconnectTimer);
+        reconnectTimer = null;
+      }
+      if (stableTimer) {
+        clearTimeout(stableTimer);
+        stableTimer = null;
+      }
+      if (socket) {
+        socket.close();
+        socket = null;
+      }
+    },
+    on(event, listener) {
+      emitter.on(event, listener);
+    }
+  };
+}
+function summarizeOutbound(outbound) {
+  let row_count = 0;
+  let bytes = 0;
+  let pushed_down = false;
+  let outcome = "success";
+  let error_code;
+  for (const env of outbound) {
+    if (env.type === "response.error") {
+      outcome = "error";
+      error_code = env.payload.code;
+    } else if (env.type === "response.chunk") {
+      bytes += JSON.stringify(env.payload).length;
+    } else if (env.type === "response.end") {
+      row_count = env.payload.total_rows;
+      pushed_down = env.payload.pushed_down;
+    }
+  }
+  return error_code !== void 0 ? { outcome, error_code, row_count, bytes, pushed_down } : { outcome, row_count, bytes, pushed_down };
+}
+export {
+  createClient
+};