@vaultgradient/pipequery-cli 0.3.2 → 0.4.0

package/README.md

@@ -2,14 +2,16 @@
 
 A command-line tool for building data pipelines, REST APIs, and terminal dashboards using PipeQuery expressions.
 
-Connect to any data source (REST APIs, WebSockets, files), run pipe-based queries to transform and aggregate data, then expose results as API endpoints or visualize them as rich terminal charts.
+Connect to any data source (REST APIs, WebSockets, files, Postgres, MySQL, SQLite, Kafka), run pipe-based queries to transform and aggregate data, then expose results as API endpoints or visualize them as rich terminal charts.
 
 **Create API endpoints on the fly** — run `pq endpoint add /api/prices -q "crypto | sort(price desc)"` and instantly get a live JSON endpoint, no config file needed.
 
+**Give your AI agent live data** — `pq mcp serve` exposes every configured source to any Model Context Protocol client (Claude Desktop, Claude Code, Cursor, Copilot). Your AI can now query your REST APIs, files, Postgres / MySQL / SQLite databases, Kafka streams, and WebSocket feeds directly.
+
 ## Installation
 
 ```bash
-npm install -g @andreadito/pq
+npm install -g @vaultgradient/pipequery-cli
 ```
 
 Or install from source:
@@ -80,6 +82,29 @@ pq source add coins -t rest -u "https://api.example.com/coins" -i 30s
 # Add a file source with live watching
 pq source add orders -t file -p ./data/orders.csv -w
 
+# Add a Postgres source (credentials interpolated from env at runtime)
+pq source add users -t postgres \
+  -u "postgres://\${DB_USER}:\${DB_PASS}@db.internal:5432/app" \
+  -q "SELECT id, email, created_at FROM users WHERE created_at > NOW() - INTERVAL '7 days'" \
+  -i 60s
+
+# Add a MySQL source
+pq source add orders -t mysql \
+  -u "mysql://\${DB_USER}:\${DB_PASS}@db.internal:3306/shop" \
+  -q "SELECT id, total, status FROM orders WHERE created_at > NOW() - INTERVAL 1 DAY" \
+  -i 60s
+
+# Add a SQLite source (path is relative to pipequery.yaml; opens read-only by default)
+pq source add events -t sqlite -p ./events.db \
+  -q "SELECT * FROM events WHERE ts > strftime('%s', 'now', '-1 hour')" \
+  -i 30s
+
+# Add a Kafka / Redpanda source (streaming — last 1000 messages kept in a ring buffer)
+pq source add live_orders -t kafka \
+  --brokers "broker1:9092,broker2:9092" \
+  --topic order-events \
+  --value-format json
+
 # Test a source (fetch sample data)
 pq source test coins
 
@@ -92,6 +117,10 @@ Source types:
 - **websocket** — Streams data from a WebSocket connection
 - **file** — Reads JSON or CSV files, optionally watches for changes
 - **static** — Inline JSON data defined in config
+- **postgres** — Polls a Postgres query. Supports `${ENV_VAR}` interpolation in the connection URL so credentials stay out of `pipequery.yaml`. Options: `-u` URL, `-q` query, `-i` interval, `--ssl <require\|no-verify\|false>`, `--max-rows <n>` (default 10000).
+- **mysql** — Polls a MySQL (or MariaDB) query. Same env-var interpolation and `--ssl` / `--max-rows` options as `postgres`.
+- **sqlite** — Polls a query against a local SQLite file (or `:memory:`). Opens read-only by default (pass `--no-readonly` to open read-write). Options: `-p` path, `-q` query, `-i` interval, `--max-rows <n>`, `--no-readonly`.
+- **kafka** — Streams messages from a Kafka (or Kafka-compatible, e.g. Redpanda) topic into a bounded ring buffer. Each message is decoded per `valueFormat` and spread into a row alongside `_kafka_topic`/`_kafka_partition`/`_kafka_offset`/`_kafka_timestamp`/`_kafka_key` metadata. Auto-generates a per-process consumer group so each pipequery instance sees the full firehose; set `groupId` explicitly to load-balance across replicas. Options: `--brokers` (comma list, supports `${ENV_VAR}`), `--topic`, `--group-id`, `--from-beginning`, `--value-format <json\|string\|raw>`, `--max-buffer <n>` (default 1000), `--ssl true`. SASL is config-only (not CLI-exposed yet).
 
 ### `pq endpoint`
 
@@ -181,6 +210,63 @@ pq remote connect https://my-server.example.com
 pq remote status
 ```
 
+## Use with AI (MCP)
+
+PipeQuery ships a [Model Context Protocol](https://modelcontextprotocol.io) server so any MCP-aware AI client — Claude Desktop, Claude Code, Cursor, Copilot, custom agents — can query your configured sources directly.
+
+```bash
+# stdio mode — what Claude Desktop / Cursor want
+pq mcp serve
+
+# HTTP/SSE mode — for remote clients or future hosted deployments
+pq mcp serve --http --port 3001
+
+# HTTP with bearer-token auth (recommended for anything non-localhost)
+PIPEQUERY_MCP_TOKEN=$(openssl rand -hex 32) pq mcp serve --http --port 3001
+# or inline:
+pq mcp serve --http --port 3001 --auth-token "my-secret"
+
+# Attach to a running `pq serve` instance instead of loading pipequery.yaml locally
+pq mcp serve --attach http://localhost:3000
+
+# Inspect the tool schemas (useful for directory submission / debugging)
+pq mcp inspect
+```
+
+**HTTP authentication** — when a token is set (`PIPEQUERY_MCP_TOKEN` env var or `--auth-token` flag), clients must send `Authorization: Bearer <token>` on every request; unauthenticated requests are rejected with `401`. If no token is set, the endpoint is open and the server prints a loud startup warning — fine for localhost, never OK for a publicly reachable deployment.
+
+**Tools exposed** to the AI:
+
+| Tool | What it does |
+|------|-------------|
+| `query` | Run any PipeQuery expression against live sources |
+| `list_sources` | List configured sources and their health |
+| `describe_source` | Return a sample + inferred field names for one source |
+| `list_endpoints` | List pre-configured endpoints and their stored queries |
+| `call_endpoint` | Execute a pre-configured endpoint by path |
+
+### Claude Desktop setup
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
+
+```json
+{
+  "mcpServers": {
+    "pipequery": {
+      "command": "pq",
+      "args": ["mcp", "serve"],
+      "cwd": "/path/to/your/project-with-pipequery.yaml"
+    }
+  }
+}
+```
+
+Restart Claude Desktop; ask "what pipequery sources are configured?" to verify.
+
+### Claude Code / Cursor setup
+
+Same idea — consult each tool's MCP config section. The stdio command is `pq mcp serve` with `cwd` pointing at a directory containing `pipequery.yaml`.
+
 ## Docker & Remote Deployment
 
 The `pq` server and CLI are fully decoupled. You can run the server in Docker (locally or on a remote machine) and control it from your local terminal.