@powerduck/openapi-cli 0.2.0

package/README.md

@@ -0,0 +1,702 @@
+# @powerduck/openapi-cli
+
+> CI-ready command-line tool for batch-testing OpenAPI 3.2 documents across HTTP, SSE, WebSocket, GraphQL, gRPC, and MCP.
+
+Built on top of [@powerduck/request](https://www.npmjs.com/package/@powerduck/request). Point it at an OpenAPI JSON file or URL, get three report formats.
+
+---
+
+## Features
+
+- **Multi-protocol out of the box** — HTTP, SSE (auto-detected), WebSocket, GraphQL, gRPC (all four streaming modes), and MCP (Streamable HTTP + stdio)
+- **Two assertion styles** — declarative `x-tests` arrays and full Postman `pm.test()` / `pm.expect()` scripts
+- **Three report formats** — machine-readable JSON, colored CLI output, and a self-contained interactive HTML report
+- **Local or remote specs** — load an OpenAPI document from a file path or an `http(s)://` URL
+- **CI-friendly** — non-zero exit on failure, configurable timeouts, bounded concurrency, `.env` support
+- **Flexible filtering** — run a subset by method, path, tag, or operationId
+- **Minimal runtime dependencies** — `@powerduck/request`, `commander`, and `@apidevtools/json-schema-ref-parser` for `$ref` resolution
+
+---
+
+## Installation
+
+### Global
+
+```bash
+npm install -g @powerduck/openapi-cli
+```
+
+### Project-local (recommended for CI)
+
+```bash
+npm install --save-dev @powerduck/openapi-cli
+```
+
+### npx (one-off)
+
+```bash
+npx @powerduck/openapi-cli --spec openapi.json
+```
+
+**Requirements:** Node.js >= 18.17.
+
+---
+
+## Quick Start
+
+```bash
+# 1. Test a local OpenAPI document against its first server URL
+openapi-cli --spec openapi.json
+
+# 2. Override the server and write reports to a custom directory
+openapi-cli --spec openapi.json \
+  --server https://api.staging.example.com \
+  --output ./reports
+
+# 3. Test a remote spec, only GET requests, 10 concurrent workers
+openapi-cli --spec https://docs.example.com/openapi.json \
+  --method get \
+  --concurrency 10
+
+# 4. Use a config file for complex setups
+openapi-cli --config openapi-cli.config.json
+```
+
+---
+
+## CLI Reference
+
+```
+openapi-cli [options]
+```
+
+### Core
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--spec <path-or-url>` | `-s` | **required** | Path or `http(s)://` URL to an OpenAPI 3.2 JSON document |
+| `--server <url>` | `-S` | *(from spec)* | Override the server base URL |
+| `--output <dir>` | `-o` | `./openapi-cli-report` | Directory for generated reports |
+| `--format <formats>` | `-f` | `json,cli,html` | Comma-separated report formats: `json`, `cli`, `html` |
+| `--config <path>` | `-c` | — | Path to a JSON config file (CLI args take precedence) |
+
+### Execution
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--concurrency <n>` | `-n` | `5` | Maximum number of concurrent requests |
+| `--timeout <ms>` | `-t` | `30000` | Per-request timeout in milliseconds (also applies to remote spec fetch) |
+| `--no-fail-on-error` | — | off | Exit with code 0 even when tests fail |
+
+### Filtering
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--method <methods>` | — | Comma-separated HTTP methods to include (e.g. `get,post`) |
+| `--path <patterns>` | — | Comma-separated regex patterns matched against the path |
+| `--tag <tags>` | — | Comma-separated OpenAPI tags to include |
+| `--operation-id <ids>` | — | Comma-separated operationIds to include |
+
+### Network & TLS
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--proxy <url>` | `-p` | — | HTTP(S) proxy URL (e.g. `http://proxy.corp:8080`) |
+| `--ca <path>` | — | — | Path to a CA certificate bundle (PEM) |
+| `--cert <path>` | — | — | Path to a client certificate (PEM) for mTLS |
+| `--key <path>` | — | — | Path to a client private key (PEM) for mTLS |
+| `--insecure` | `-k` | off | Skip TLS certificate verification |
+
+### Authentication & Headers
+
+| Option | Short | Default | Description |
+|--------|-------|---------|-------------|
+| `--header <headers...>` | `-H` | — | Extra request headers, `Key: Value` format. Repeatable. |
+| `--bearer <token>` | — | — | Bearer token for the `Authorization` header |
+| `--variable <vars...>` | `-v` | — | Postman variables, `KEY=VALUE` format. Repeatable. |
+| `--env <path>` | — | — | Path to a `.env` file (does not override existing env vars) |
+
+### gRPC
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--grpc-no-reflection` | off | Disable gRPC server reflection; use proto files instead |
+| `--grpc-proto <paths...>` | — | Paths to `.proto` files or directories containing them |
+
+### MCP
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--mcp-transport <transport>` | `streamable-http` | MCP transport: `streamable-http` or `stdio` |
+| `--mcp-command <command>` | — | MCP stdio command (e.g. `npx`) |
+| `--mcp-args <args>` | — | MCP stdio arguments, space- or comma-separated |
+| `--mcp-cwd <dir>` | — | Working directory for the MCP stdio subprocess |
+
+### Informational
+
+| Option | Short | Description |
+|--------|-------|-------------|
+| `--help` | `-h` | Show help |
+| `--version` | `-V` | Show version |
+
+---
+
+## Configuration File
+
+A JSON config file can be used instead of (or in addition to) CLI arguments. CLI arguments always take precedence.
+
+```json
+{
+  "specPath": "./openapi.json",
+  "serverUrl": "https://api.staging.example.com",
+  "outputDir": "./reports",
+  "formats": ["json", "html"],
+  "concurrency": 10,
+  "timeout": 15000,
+  "failOnError": true,
+  "proxy": "http://proxy.corp:8080",
+  "tls": {
+    "strictSSL": true,
+    "caCert": "./ca-bundle.pem",
+    "clientCert": "./client.crt",
+    "clientKey": "./client.key"
+  },
+  "headers": {
+    "X-API-Key": "${API_KEY}",
+    "X-Environment": "staging"
+  },
+  "auth": {
+    "type": "bearer",
+    "token": "${BEARER_TOKEN}"
+  },
+  "variables": {
+    "host": "api.staging.example.com"
+  },
+  "filter": {
+    "methods": ["get", "post"],
+    "tags": ["v2", "public"],
+    "paths": ["^/api/v2"],
+    "operationIds": ["getUser", "createUser"]
+  },
+  "grpcReflection": true,
+  "grpcProtoPaths": ["./proto"],
+  "mcpTransport": "streamable-http",
+  "mcpCommand": "npx",
+  "mcpArgs": ["-y", "@modelcontextprotocol/server-everything"],
+  "mcpCwd": "./mcp-servers"
+}
+```
+
+### Config field reference
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `specPath` | `string` | Local path or remote URL to the OpenAPI spec |
+| `serverUrl` | `string` | Override the server base URL |
+| `outputDir` | `string` | Output directory for reports |
+| `formats` | `string[]` | Report formats: `json`, `cli`, `html` |
+| `concurrency` | `number` | Max concurrent requests |
+| `timeout` | `number` | Per-request timeout in ms |
+| `failOnError` | `boolean` | Exit non-zero when tests fail |
+| `proxy` | `string` | HTTP(S) proxy URL |
+| `tls.strictSSL` | `boolean` | Verify TLS certificates (default `true`) |
+| `tls.caCert` | `string` | CA certificate bundle path |
+| `tls.clientCert` | `string` | Client certificate path (mTLS) |
+| `tls.clientKey` | `string` | Client private key path (mTLS) |
+| `headers` | `Record<string,string>` | Extra request headers |
+| `auth.type` | `string` | Auth type (currently `bearer`) |
+| `auth.token` | `string` | Bearer token |
+| `variables` | `Record<string,string>` | Postman variables |
+| `filter.methods` | `string[]` | HTTP method filter |
+| `filter.paths` | `string[]` | Path regex filter |
+| `filter.tags` | `string[]` | OpenAPI tag filter |
+| `filter.operationIds` | `string[]` | operationId filter |
+| `grpcReflection` | `boolean` | Use gRPC server reflection |
+| `grpcProtoPaths` | `string[]` | Proto file/directory paths |
+| `mcpTransport` | `string` | `streamable-http` or `stdio` |
+| `mcpCommand` | `string` | MCP stdio command |
+| `mcpArgs` | `string[]` | MCP stdio arguments |
+| `mcpCwd` | `string` | MCP stdio working directory |
+
+---
+
+## Spec Sources
+
+### Local file
+
+```bash
+openapi-cli --spec ./api/openapi.json
+```
+
+### Remote URL
+
+```bash
+openapi-cli --spec https://docs.example.com/openapi.json
+```
+
+Remote specs are fetched with:
+- A configurable timeout (`--timeout`, default 30s)
+- Up to 5 automatic redirects
+- A descriptive error on HTTP failure (status code + message)
+- JSON validation after download
+
+The `User-Agent` header is set to `@powerduck/openapi-cli`.
+
+---
+
+## $ref Resolution
+
+Internal `$ref` pointers (e.g. `#/components/schemas/User`) are automatically resolved before test execution. This means request body schemas, response schemas, and path items that reference `components` are fully inlined.
+
+- **Supported**: internal refs to any location in the same document (`#/...`)
+- **Not resolved**: external file or URL refs (`./other.json#/...`, `https://...`) — these are left as-is
+- **Circular refs**: safely ignored (no infinite recursion)
+
+Resolution is powered by [`@apidevtools/json-schema-ref-parser`](https://www.npmjs.com/package/@apidevtools/json-schema-ref-parser).
+
+---
+
+## Assertions
+
+### Declarative assertions (`x-tests`)
+
+Add an `x-tests` array to any operation. Each assertion is an object with a `name` and an `assert` type.
+
+```json
+{
+  "paths": {
+    "/users/{id}": {
+      "get": {
+        "operationId": "getUser",
+        "x-tests": [
+          { "name": "status is 200", "assert": "status", "value": 200 },
+          { "name": "content-type is JSON", "assert": "header", "key": "content-type", "contains": "application/json" },
+          { "name": "user id exists", "assert": "jsonPath", "path": "$.id", "exists": true },
+          { "name": "user is active", "assert": "jsonPath", "path": "$.status", "equals": "active" },
+          { "name": "response under 2s", "assert": "responseTime", "max": 2000 },
+          { "name": "body contains email", "assert": "bodyContains", "contains": "@" }
+        ]
+      }
+    }
+  }
+}
+```
+
+#### Assertion types
+
+| Type | Required fields | Optional fields | Description |
+|------|----------------|-----------------|-------------|
+| `status` | `value` (number) | — | HTTP status code must equal `value` |
+| `header` | `key` (string) | `contains` (string) | Header must exist; if `contains` is set, the header value must include the substring (case-insensitive) |
+| `bodyContains` | `contains` (string) | — | Response body must include the substring |
+| `bodyEquals` | `body` (string) | — | Response body must exactly equal `body` |
+| `jsonPath` | `path` (string) | `equals`, `exists` (boolean) | Resolve a JSONPath expression (dot notation + array indices). With `equals`, compare the resolved value. With `exists: true`, verify the path resolves. |
+| `responseTime` | `max` (number) | — | Response duration must be <= `max` milliseconds |
+
+#### JSONPath syntax
+
+Supports dot notation and array indices:
+
+- `$.data.id` → `body.data.id`
+- `$.users[0].name` → `body.users[0].name`
+- `data.items[2].value` → leading `$` is optional
+
+### Postman scripts (`x-postman-scripts`)
+
+Full Postman test scripts are executed by `@powerduck/request`'s built-in script sandbox. Use the standard `pm.test()` and `pm.expect()` API.
+
+```json
+{
+  "paths": {
+    "/users": {
+      "get": {
+        "x-postman-scripts": {
+          "test": "pm.test('status is 200', () => pm.response.to.have.status(200)); pm.test('users is an array', () => pm.expect(pm.response.json().users).to.be.an('array')); pm.test('at least one user', () => pm.expect(pm.response.json().users.length).to.be.above(0));"
+        }
+      }
+    }
+  }
+}
+```
+
+Both `x-tests` and `x-postman-scripts` can coexist on the same operation. Declarative assertions run first, then Postman scripts.
+
+### Implicit assertions (no user assertions defined)
+
+When an operation defines **no** `x-tests` and **no** `x-postman-scripts`, the CLI adds an implicit success check using each protocol's conventional success criterion. This prevents 404s, 500s, and gRPC errors from being silently marked as passed.
+
+| Protocol | Implicit success criterion | Failure example |
+|----------|---------------------------|-----------------|
+| HTTP | Status code is 2xx (200–299) | 404, 500, 301 |
+| SSE | Status code is 2xx (200–299) | 404, 500 |
+| GraphQL | Status 2xx **and** no top-level `errors` array | 200 with `{"errors": [...]}` |
+| gRPC | Status code is `0` (OK) | code 13 (INTERNAL), code 5 (NOT_FOUND) |
+| MCP | JSON-RPC response has `result` and no `error` | `{"error": {"code": -32601}}` |
+| WebSocket | Connection reached `open` state with no error | connection refused, handshake failure |
+
+If you need to test a non-success response (e.g., asserting that an endpoint returns 404), define an explicit `x-tests` assertion — implicit checks are skipped when any user assertion exists.
+
+---
+
+## Multi-Protocol Support
+
+### HTTP / SSE
+
+Standard OpenAPI operations are executed as HTTP requests. SSE is auto-detected when:
+- The response content type is `text/event-stream`, or
+- The operation has `x-response-stream: true`
+
+SSE responses are collected as a list of events.
+
+### GraphQL
+
+```json
+{
+  "paths": {
+    "/graphql": {
+      "post": {
+        "x-protocol": "graphql",
+        "x-graphql": {
+          "endpoint": "https://api.example.com/graphql",
+          "query": "query GetUser($id: ID!) { user(id: $id) { name email } }",
+          "operationName": "GetUser",
+          "variablesSchema": {
+            "type": "object",
+            "properties": { "id": { "type": "string" } },
+            "required": ["id"]
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+### gRPC
+
+All four gRPC method modes are supported: unary, server-streaming, client-streaming, and bidirectional-streaming.
+
+```json
+{
+  "paths": {
+    "/grpc": {
+      "post": {
+        "x-protocol": "grpc",
+        "x-grpc": {
+          "address": "127.0.0.1:50051",
+          "service": "demo.echo.Echo",
+          "method": "Say",
+          "kind": "unary",
+          "reflection": true
+        }
+      }
+    }
+  }
+}
+```
+
+| Field | Description |
+|-------|-------------|
+| `address` | gRPC server `host:port` |
+| `service` | Fully-qualified service name |
+| `method` | Method name |
+| `kind` | `unary`, `server_streaming`, `client_streaming`, `bidi_streaming` |
+| `reflection` | Use server reflection (default `true`) |
+| `protoPaths` | Paths to `.proto` files when reflection is off |
+| `metadata` | Initial metadata (headers) |
+| `sample` | Sample request message (JSON) |
+
+### MCP
+
+```json
+{
+  "paths": {
+    "/mcp": {
+      "post": {
+        "x-protocol": "mcp",
+        "x-mcp": {
+          "endpoint": "http://127.0.0.1:4200/mcp",
+          "method": "tools/call",
+          "name": "get_weather",
+          "arguments": { "city": "San Francisco" }
+        }
+      }
+    }
+  }
+}
+```
+
+Supports both Streamable HTTP and stdio transports (configured via `--mcp-transport`).
+
+### WebSocket
+
+```json
+{
+  "paths": {
+    "/ws": {
+      "get": {
+        "x-protocol": "websocket",
+        "x-ws": {
+          "url": "wss://api.example.com/ws",
+          "subprotocols": ["chat"],
+          "send": [{ "type": "subscribe", "channel": "updates" }]
+        }
+      }
+    }
+  }
+}
+```
+
+---
+
+## Report Formats
+
+### JSON (`report.json`)
+
+A complete machine-readable report:
+
+```json
+{
+  "summary": {
+    "total": 10,
+    "passed": 8,
+    "failed": 1,
+    "errors": 1,
+    "skipped": 0,
+    "durationMs": 2340,
+    "passRate": 80.0
+  },
+  "results": [
+    {
+      "operationId": "getUser",
+      "path": "/users/{id}",
+      "method": "GET",
+      "protocol": "http",
+      "status": "passed",
+      "durationMs": 142,
+      "response": { "status": 200, "contentType": "application/json", "body": { "id": 1 } },
+      "assertions": [
+        { "name": "status is 200", "passed": true }
+      ],
+      "timestamp": "2024-01-01T00:00:00.000Z"
+    }
+  ],
+  "config": { "specPath": "./openapi.json" },
+  "generatedAt": "2024-01-01T00:00:05.000Z",
+  "version": "0.1.0"
+}
+```
+
+### CLI (stdout)
+
+Colored terminal output with a progress bar, per-test status, and assertion details. Failed assertions show error messages inline.
+
+### HTML (`report.html`)
+
+A self-contained HTML file with:
+- Summary strip (total, passed, failed, errors, pass rate, duration)
+- Thin progress bar
+- Filter buttons (All / Passed / Failed / Errors)
+- Expandable test rows with assertion details and response body preview
+- Automatic light/dark mode based on system preference
+- No external dependencies — opens in any browser
+
+---
+
+## CI Integration
+
+### GitHub Actions
+
+```yaml
+name: API Tests
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+      - name: Run API tests
+        run: npx @powerduck/openapi-cli --spec openapi.json --server ${{ secrets.API_URL }} --output ./reports
+        env:
+          API_KEY: ${{ secrets.API_KEY }}
+      - name: Upload reports
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: openapi-cli-reports
+          path: ./reports
+```
+
+### GitLab CI
+
+```yaml
+api_test:
+  stage: test
+  image: node:20
+  script:
+    - npm ci
+    - npx @powerduck/openapi-cli --spec openapi.json --output ./reports
+  artifacts:
+    when: always
+    paths:
+      - ./reports/
+    expire_in: 30 days
+```
+
+### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | All tests passed (or `--no-fail-on-error` is set) |
+| `1` | One or more tests failed |
+| `2` | Configuration error, spec load failure, or runtime error |
+
+---
+
+## Programmatic API
+
+```ts
+import {
+  runTests,
+  resolveConfig,
+  generateJsonReport,
+  printCliReport,
+  generateHtmlReport,
+  loadSpec,
+  isRemoteSpec,
+} from "@powerduck/openapi-cli";
+
+// Build a config from args
+const config = resolveConfig({
+  spec: "./openapi.json",
+  server: "https://api.staging.example.com",
+  concurrency: 10,
+});
+
+// Run all tests
+const report = await runTests(config);
+
+// Generate reports
+printCliReport(report);
+generateJsonReport(report, "./reports");
+generateHtmlReport(report, "./reports");
+
+// Load a spec directly (local or remote)
+const spec = await loadSpec(config);
+
+// Check if a path is remote
+console.log(isRemoteSpec("https://example.com/spec.json")); // true
+
+// Exit with the right code
+if (report.summary.failed > 0 || report.summary.errors > 0) {
+  process.exit(1);
+}
+```
+
+---
+
+## Environment Variables
+
+### `.env` files
+
+Use `--env .env` to load variables from a file into `process.env` before config resolution. Format is `KEY=VALUE` per line.
+
+```env
+# Lines starting with # are ignored
+API_KEY=sk-live-abc123
+BEARER_TOKEN="eyJhbGciOi..."
+API_HOST=api.staging.example.com
+```
+
+Behavior:
+- Existing `process.env` variables are **not** overwritten.
+- Surrounding single or double quotes are stripped from values.
+- Blank lines and comment lines (`#`) are ignored.
+- A missing `.env` file is silently ignored (no error).
+
+Loaded variables are available for `${VAR_NAME}` expansion in CLI config values (see below). To make a variable available as a Postman `{{variable}}` inside the OpenAPI spec, pass it with `--variable KEY=${KEY}`.
+
+### Variable expansion
+
+`${VAR_NAME}` syntax is expanded in these config fields:
+- `serverUrl`
+- `proxy`
+- `headers` (values)
+- `auth.token`
+- `variables` (values)
+
+Example:
+
+```bash
+openapi-cli --spec openapi.json \
+  --server 'https://${API_HOST}/v1' \
+  --header 'Authorization: Bearer ${BEARER_TOKEN}'
+```
+
+---
+
+## Troubleshooting
+
+### "OpenAPI spec not found"
+
+Verify the path exists. For remote specs, check the URL is reachable and returns HTTP 200.
+
+### "Spec is not valid JSON"
+
+The spec must be a JSON object with a `paths` property. YAML specs are not supported — convert to JSON first.
+
+### "Failed to fetch spec: HTTP 401"
+
+The remote spec URL requires authentication. Download it locally first or use a URL that is publicly accessible.
+
+### gRPC "server reflection unavailable"
+
+Either enable reflection on the server, or use `--grpc-no-reflection` with `--grpc-proto` pointing to your `.proto` files.
+
+### MCP stdio "command not found"
+
+Ensure `--mcp-command` is on your `PATH`, or use an absolute path. For `npx`-based servers, make sure the package is installed or use `npx -y <package>`.
+
+### Tests are slow
+
+Reduce `--concurrency` if the target server rate-limits. Increase `--timeout` for slow endpoints. Use `--method` or `--tag` to test a subset.
+
+### Tests fail with "(implicit)" in the assertion name
+
+This means the operation has no user-defined assertions and the implicit protocol check failed. For HTTP this means the status was not 2xx; for gRPC the code was not 0; for MCP the response contained a JSON-RPC error. Add an explicit `x-tests` assertion if the non-success response is expected (e.g. testing a 404 endpoint).
+
+### All tests show "error" status
+
+Check the `error` field in `report.json` for details. Common causes: wrong server URL, network connectivity, missing auth headers, or TLS certificate issues (try `--insecure` for testing).
+
+---
+
+## Development
+
+```bash
+# Install dependencies
+npm install
+
+# Type check
+npm run typecheck
+
+# Build (ESM + CJS + type declarations)
+npm run build
+
+# Run tests
+npm test
+
+# Watch mode
+npm run test:watch
+```
+
+---
+
+## License
+
+MIT