clyops-api 0.0.0-stage → 0.2.0

package/README.md

@@ -1,3 +1,163 @@
-# Temporary Holding Version
+# clyops-api
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Serve a directory of [clyops](https://github.com/wankdanker/clyops) tools as an HTTP API. Each tool
+becomes a `POST` endpoint whose JSON body is validated against the tool's own
+`--help-json-schema` and documented in an OpenAPI document, built with
+[plus-express](https://www.npmjs.com/package/plus-express). Calls wait for the tool by default, or
+return a job to poll.
+
+```sh
+npm install -g clyops-api
+clyops-api --root ~/mytool/scripts            # http://127.0.0.1:8080
+```
+
+```
+scripts/
+  check.sh            → POST /tools/check
+  media/
+    to-pcm.sh         → POST /tools/media/to-pcm
+```
+
+The directory is read like [clyops-dispatch](../dispatch) reads it (groups, `.clyops` files,
+ignore lists; `--root` may also be a dispatcher definition file). Only programs built on a clyops
+library are served, since their schema is what the endpoint is made from.
+
+## Calling a tool
+
+The body has one key per option and argument: the option's long name with `-` as `_` (`dry_run`),
+and the argument's name. Flags are booleans, repeatable options and variadic arguments are arrays,
+and numbers are numbers. The mapping to a command line is [spec §13](../../spec/SPEC.md#13-json-input-toargv).
+
+```sh
+curl -s localhost:8080/tools/media/to-pcm -H content-type:application/json \
+     -d '{"input": "in.wav", "rate": 16000, "verbose": true}'
+```
+
+```json
+{ "ok": true, "exitCode": 0, "signal": null, "timedOut": false, "durationMs": 412,
+  "stdout": "...", "stderr": "...", "command": ["/home/me/mytool/scripts/media/to-pcm.sh", "..."],
+  "json": { "...": "stdout parsed, when it is JSON" } }
+```
+
+A tool that runs but fails answers `200` with `"ok": false` and its exit code and stderr. Input
+that doesn't match the schema answers `400` with the validation issues and doesn't run anything.
+If the client disconnects, the tool is stopped.
+
+A program with commands (`tasks db migrate`) has an endpoint per command: `POST /tools/tasks/db/migrate`.
+
+**Streaming stdin.** Any body that isn't JSON is streamed to the tool's stdin as it arrives, and
+the input comes from the query string instead (typed by the schema: `?count=3&verbose=true`,
+repeated keys for arrays: `?tag=a&tag=b`). The query is validated before the tool starts.
+
+```sh
+curl -s 'localhost:8080/tools/transcribe?model=base' -H content-type:audio/wav --data-binary @in.wav
+```
+
+**Multipart.** `multipart/form-data` takes the input as JSON in an `args` part, stdin in a `stdin`
+part, and files for path inputs in parts named after them (saved to a temporary directory for the
+run, then removed):
+
+```sh
+curl -s localhost:8080/tools/media/to-pcm -F args='{"rate":16000}' -F input=@in.wav
+```
+
+**Streaming stdout.** When a tool declares binary output (`stdout` with a type such as
+`audio/mpeg`), the response *is* that output, sent as the tool writes it with the declared
+`Content-Type`: `curl ... > out.mp3` works, and a player can start before the tool is done. The
+exit status isn't known when the headers go out, so it follows as HTTP trailers,
+`X-Clyops-Exit-Code` and `X-Clyops-Stderr` (the end of stderr); a tool that fails before writing
+anything answers `500` with the usual JSON. `Accept: application/json` asks for the JSON envelope
+instead (`stdout` base64-encoded, `"stdoutEncoding": "base64"`), and `Accept:
+application/octet-stream` streams any tool's output.
+
+**Async.** `POST /tools/...?async=true` answers `202` with a job record and a `Location` header:
+
+```sh
+curl -s -XPOST 'localhost:8080/tools/media/to-pcm?async=true' -H content-type:application/json -d '{"input":"in.wav"}'
+# {"job_id":"job-1f2e3d4c5b6a","status":"pending",...}
+curl -s localhost:8080/jobs/job-1f2e3d4c5b6a
+# {"job_id":"...","status":"done","result":{"ok":true,...},...}
+curl -s -XDELETE localhost:8080/jobs/job-1f2e3d4c5b6a    # cancel
+```
+
+Job status moves `pending` → `processing` → `done` | `error` (`error: "cancelled"` when
+cancelled). Jobs are kept in memory (the latest 1000) and run `--concurrency` at a time. A job
+with a streamed body keeps it in a file until it runs; a job with binary output writes it to a
+file served at `GET /jobs/<id>/stdout` (its result has `stdoutUrl`). A job's files are removed when
+it is dropped.
+
+## Endpoints
+
+| | |
+| --- | --- |
+| `GET /openapi.json` | OpenAPI 3 document of everything below |
+| `GET /tools` | The tools: name, words, path, description |
+| `GET /tools/<words>` | A tool's clyops schema and the JSON Schema of its input |
+| `POST /tools/<words>[?async=true]` | Run it |
+| `GET /jobs`, `GET /jobs/<id>`, `DELETE /jobs/<id>` | Async jobs |
+| `GET /jobs/<id>/stdout` | A finished job's binary output |
+| `POST /mcp` | The same tools over MCP (streamable HTTP, stateless) for agents; see [clyops-mcp](../mcp). `--no-mcp` turns it off. |
+
+## Options
+
+```
+clyops-api --root DIR [--name NAME] [--cwd DIR] [--timeout SECONDS] [--concurrency N]
+           [--host 127.0.0.1] [--port 8080] [--api-key KEY] [--keys FILE] [--no-mcp] [--no-watch]
+           [--allow GLOB]... [--deny GLOB]... [--read-only] [--paths-within DIR]...
+           [--max-body BYTES] [--max-output BYTES] [--audit FILE]
+```
+
+It listens on localhost unless told otherwise. Every option can also come from the environment as
+`CLYOPS_API_<OPTION>`. Tools run in `--cwd` (default: where the server was started), so relative
+paths in the input resolve there. Each operation in the OpenAPI document carries the tool's
+declared effects as `x-clyops-effects`.
+
+## Security
+
+Anyone who can call the API can run every tool it serves with any arguments their schemas accept.
+Serve only what callers should be able to run:
+
+- **Keys.** With `--api-key` (or `CLYOPS_API_API_KEY`), every request needs `Authorization: Bearer
+  KEY` or `X-API-Key: KEY`; that key may run everything. `--keys FILE` adds named keys, each with
+  its own scope:
+  ```json
+  { "ci": { "key": "…", "allow": ["media/*"] }, "ops": { "key": "…", "deny": ["admin/**"] } }
+  ```
+  A key sees and runs only its tools (`403` otherwise), in the REST endpoints and at `/mcp`, and
+  sees only its own jobs.
+- **Which tools.** `--allow media/*` and `--deny admin/**` (repeatable) are globs over a tool's
+  words: `*` within a word, `**` across words. A tool must match an `--allow` pattern when there
+  are any, and no `--deny` pattern; `allow:` and `deny:` lines in the root `.clyops` file apply as
+  well. `--read-only` serves only tools declaring the `read-only` effect. Hot reload applies the
+  same rules to new tools.
+- **Paths.** `--paths-within DIR` (repeatable): path inputs (`path`, `file:*`, `dir:*`) must
+  resolve inside one of these directories, symlinks followed, or the request is a `400`.
+- **Limits.** `--timeout`, `--concurrency`, `--max-body` (JSON, multipart and spooled bodies;
+  default 10 MiB) and `--max-output` (stdout and stderr kept per run; default 16 MiB).
+- **Secrets.** Options a tool marks secret are passed to it in its environment rather than on
+  its command line (where `ps` shows them), and `command` in responses, job records and the audit
+  log shows them as `***`.
+- **Audit.** `--audit FILE` (`-` for stderr) appends one JSON line per run: time, key, tool,
+  command line, exit status and duration.
+
+**Hot reload.** The server watches the tools directory: add, change or remove a tool (or a
+`.clyops` file) and its endpoint, the OpenAPI document and the MCP tool list follow within a
+moment, without a restart. Jobs already running keep running. `--no-watch` reads the tools once
+at startup instead.
+
+## As a library
+
+```js
+import express from 'express';
+import { createApi } from 'clyops-api';
+
+const { app, current, close } = await createApi({ root: './scripts', apiKey: process.env.KEY, watch: true });
+app.listen(8080);
+current().tools;   // what is being served now
+close();           // stop watching
+```
+
+Watching is off by default in the library (`watch: true` turns it on, `onReload` reports each
+new set) and on by default in the `clyops-api` command.
+
+`loadTools(root)` and `runTool(tool, input)` are exported for other servers.

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,62 @@
+#!/usr/bin/env node
+// clyops-api --root DIR: serve a directory of clyops tools over HTTP.
+import { Cli, die, info } from 'clyops';
+import { auditLog } from 'clyops-tools';
+import { readFileSync } from 'node:fs';
+import { createApi } from './server.js';
+const { version } = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
+const cli = new Cli({ name: 'clyops-api' });
+cli.setDescription('Serve a directory of clyops tools as an HTTP API: one POST endpoint per tool, validated and documented (OpenAPI at /openapi.json) from each tool\'s --help-json-schema.');
+cli.setEpilog('Every option can also be set in the environment as CLYOPS_API_<OPTION> (CLYOPS_API_API_KEY for the key).\n\nExamples:\n  clyops-api --root ~/mytool/scripts\n  curl -X POST localhost:8080/tools/media/to-pcm -H content-type:application/json -d \'{"input":"a.wav"}\'');
+cli.opt('CLYOPS_API_ROOT', 'root', 'r', '', 'Tools directory or dispatcher definition file', 'Tools', 'path');
+cli.opt('CLYOPS_API_NAME', 'name', 'n', 'optional', 'API title (default: the directory name)', 'Tools');
+cli.opt('CLYOPS_API_CWD', 'cwd', '', 'optional', 'Working directory for the tools (default: the current one)', 'Tools', 'dir:exists');
+cli.opt('CLYOPS_API_TIMEOUT', 'timeout', 't', '0', 'Kill a tool after this many seconds (0: never)', 'Tools', 'int:0-');
+cli.opt('CLYOPS_API_CONCURRENCY', 'concurrency', 'j', 'optional', 'Async jobs run at the same time (default: CPUs)', 'Tools', 'int:1-');
+cli.opt('CLYOPS_API_MCP', 'mcp', '', 'true', 'Also serve the tools over MCP (streamable HTTP) at /mcp', 'Server', 'bool');
+cli.opt('CLYOPS_API_WATCH', 'watch', 'w', 'true', 'Pick up added, changed and removed tools without a restart', 'Tools', 'bool');
+cli.opt('CLYOPS_API_HOST', 'host', 'H', '127.0.0.1', 'Address to listen on', 'Server');
+cli.opt('CLYOPS_API_PORT', 'port', 'p', '8080', 'Port to listen on', 'Server', 'port');
+cli.opt('CLYOPS_API_API_KEY', 'api-key', 'k', 'optional', 'Require this key (Authorization: Bearer KEY or X-API-Key); it may run every tool', 'Server', 'secret');
+cli.opt('CLYOPS_API_KEYS', 'keys', 'K', 'optional', 'JSON file of named keys, each with its own scope: {"ci": {"key": "...", "allow": ["media/*"]}}', 'Security', 'file:readable');
+cli.optArray('CLYOPS_API_ALLOW', 'allow', 'a', 'Serve only tools matching this glob over their words (media/*, media/**)', 'Security');
+cli.optArray('CLYOPS_API_DENY', 'deny', 'D', 'Leave out tools matching this glob', 'Security');
+cli.opt('CLYOPS_API_READ_ONLY', 'read-only', '', 'flag', 'Serve only tools that declare the read-only effect', 'Security');
+cli.optArray('CLYOPS_API_PATHS_WITHIN', 'paths-within', '', 'Path inputs must resolve inside this directory', 'Security', 'dir:exists');
+cli.opt('CLYOPS_API_MAX_BODY', 'max-body', '', '10485760', 'Largest request body in bytes: JSON, multipart or spooled for an async job', 'Security', 'int:1-');
+cli.opt('CLYOPS_API_MAX_OUTPUT', 'max-output', '', '16777216', 'Keep at most this many bytes of a tool\'s stdout and stderr (0: all)', 'Security', 'int:0-');
+cli.opt('CLYOPS_API_AUDIT', 'audit', '', 'optional', 'Append a JSON line per run to this file (-: stderr)', 'Security', 'path');
+const args = cli.run();
+let keys;
+if (args.CLYOPS_API_KEYS) {
+    try {
+        keys = JSON.parse(readFileSync(args.CLYOPS_API_KEYS, 'utf8'));
+    }
+    catch (err) {
+        die(1, 'cannot read --keys: %s', err.message);
+    }
+}
+const within = args.CLYOPS_API_PATHS_WITHIN;
+const { app, current } = await createApi({
+    root: args.CLYOPS_API_ROOT,
+    name: args.CLYOPS_API_NAME ?? undefined,
+    cwd: args.CLYOPS_API_CWD ?? undefined,
+    timeoutMs: args.CLYOPS_API_TIMEOUT * 1000,
+    concurrency: args.CLYOPS_API_CONCURRENCY ?? undefined,
+    apiKey: args.CLYOPS_API_API_KEY ?? undefined,
+    keys,
+    filter: { allow: args.CLYOPS_API_ALLOW, deny: args.CLYOPS_API_DENY, readOnly: args.CLYOPS_API_READ_ONLY },
+    within: within.length ? within : undefined,
+    maxBody: args.CLYOPS_API_MAX_BODY,
+    maxOutput: args.CLYOPS_API_MAX_OUTPUT,
+    audit: args.CLYOPS_API_AUDIT ? auditLog(args.CLYOPS_API_AUDIT) : undefined,
+    mcp: args.CLYOPS_API_MCP,
+    watch: args.CLYOPS_API_WATCH,
+    onReload: ({ tools }) => info('reloaded: %d tool(s)', tools.length),
+    version,
+});
+const server = app.listen(args.CLYOPS_API_PORT, args.CLYOPS_API_HOST, () => {
+    info('serving %d tool(s) on http://%s:%d (OpenAPI: /openapi.json%s)', current().tools.length, args.CLYOPS_API_HOST, args.CLYOPS_API_PORT, args.CLYOPS_API_MCP ? ', MCP: /mcp' : '');
+});
+for (const signal of ['SIGINT', 'SIGTERM'])
+    process.on(signal, () => server.close(() => process.exit(0)));

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+export * from './server.js';
+export { toZod } from './zod.js';
+export { loadTools, runTool, type Tool, type ToolResult } from 'clyops-tools';

package/dist/index.js

@@ -0,0 +1,4 @@
+// clyops-api: an HTTP API over a directory of clyops tools.
+export * from './server.js';
+export { toZod } from './zod.js';
+export { loadTools, runTool } from 'clyops-tools';

package/dist/server.d.ts

@@ -0,0 +1,75 @@
+import { type AuditEntry, type Group, type Tool, type ToolFilter, type ToolResult } from 'clyops-tools';
+import { JobQueue } from 'clyops-jobs';
+import { type NextFunction, type Request, type Response } from 'express';
+/** A named API key and what it may run: globs over tool words (see clyops-tools' ToolFilter). */
+export interface ApiKey {
+    key: string;
+    allow?: string[];
+    deny?: string[];
+}
+export interface ApiOptions {
+    /** Tools directory or dispatcher definition file. */
+    root: string;
+    /** API title (default: the root's name). */
+    name?: string;
+    /** Working directory tools run in (default: the server's). */
+    cwd?: string;
+    /** Required as `Authorization: Bearer KEY` or `X-API-Key: KEY` when set; it may run every tool. */
+    apiKey?: string;
+    /** Named keys, each limited to the tools its allow/deny globs let through. */
+    keys?: Record<string, ApiKey>;
+    /** Which tools to serve at all (with the root's `allow`/`deny` settings). */
+    filter?: ToolFilter;
+    /** Path-valued inputs must resolve inside these directories. */
+    within?: string[];
+    /** Kill a tool after this long (0: never). */
+    timeoutMs?: number;
+    /** Largest request body accepted: JSON, multipart, or spooled for an async job (default 10 MiB). */
+    maxBody?: number;
+    /** Keep at most this many bytes of a tool's stdout and stderr (0 or unset: all). Streamed stdout is not kept. */
+    maxOutput?: number;
+    /** Called after every run, for an audit log. */
+    audit?: (entry: Omit<AuditEntry, 'time'>) => void;
+    /** Async jobs run at the same time (default: the number of CPUs). */
+    concurrency?: number;
+    /** Also serve the tools over MCP (streamable HTTP) at /mcp (default: true). */
+    mcp?: boolean;
+    /** Pick up added, changed and removed tools without a restart (default: false; call close() to stop). */
+    watch?: boolean;
+    /** Called after each reload when watching. */
+    onReload?: (loaded: {
+        tree: Group;
+        tools: ApiTool[];
+    }) => void;
+    version?: string;
+}
+/** A tool the API serves, with the URL path of its endpoint, `/tools/<group>/.../<name>`. */
+export type ApiTool = Tool & {
+    path: string;
+};
+type ApiResult = ToolResult & {
+    stdoutEncoding?: 'base64';
+    stdoutUrl?: string;
+};
+/**
+ * Build the Express app. With `watch`, the tool endpoints, the OpenAPI
+ * document and /mcp follow the tools directory as it changes; otherwise they
+ * are fixed when the app is built.
+ */
+export declare function createApi(opts: ApiOptions): Promise<{
+    app: import("express-serve-static-core").Express;
+    queue: JobQueue<ApiResult>;
+    /** The tree and tools being served now. */
+    current: () => {
+        tree: Group;
+        tools: ApiTool[];
+    };
+    /** Stop watching (the app keeps serving the last set). */
+    close: () => void | undefined;
+}>;
+/** Errors as JSON: validation failures (400) with zod's issues, anything else 500. */
+export declare function errorHandler(err: Error & {
+    status?: number;
+    errors?: unknown;
+}, _req: Request, res: Response, _next: NextFunction): void;
+export {};