@datanimbus/dnio-mcp 1.0.0

package/Dockerfile

@@ -0,0 +1,20 @@
+FROM node:22-alpine
+
+RUN apk update && apk upgrade && apk add --no-cache dumb-init
+
+WORKDIR /app
+
+COPY package.json package-lock.json ./
+
+RUN npm ci --omit=dev
+
+COPY src/ src/
+
+ENV NODE_ENV=production
+ENV TRANSPORT=http
+ENV MCP_PORT=3100
+ENV LOG_LEVEL=info
+
+EXPOSE 3100
+
+CMD ["dumb-init", "node", "src/index.js"]

package/docs/README.md

@@ -0,0 +1,35 @@
+# DNIO MCP Server — Documentation
+
+This directory documents the DNIO MCP server in depth. The top-level [`readme.md`](../readme.md) is the entry point for users; these docs are for anyone extending the server, debugging tool behaviour, or needing the full surface reference.
+
+## Index
+
+### Overview
+- [`architecture.md`](architecture.md) — codebase organization, master-client + per-domain pattern, file layout.
+- [`authentication.md`](authentication.md) — admin + MCP-user dual-token model, refresh lifecycle, app provisioning.
+
+### Tool reference (one file per domain)
+- [`tools/apps.md`](tools/apps.md) — app & service discovery (5 tools)
+- [`tools/records.md`](tools/records.md) — record CRUD, generic + per-service typed tools (6 + dynamic)
+- [`tools/data-services.md`](tools/data-services.md) — data-service definitions (7 tools)
+- [`tools/connectors.md`](tools/connectors.md) — connector instances + marketplace (3 tools)
+- [`tools/plugins.md`](tools/plugins.md) — workflow-node plugins from marketplace (5 tools)
+- [`tools/data-pipes.md`](tools/data-pipes.md) — flows: triggers, processes, mappings, branching (10 tools)
+- [`tools/deployment-groups.md`](tools/deployment-groups.md) — K8s deployment lifecycle (12 tools)
+
+### Recipes
+- [`workflows.md`](workflows.md) — end-to-end flows that combine multiple tools.
+
+### Design history
+- [`superpowers/specs/`](superpowers/specs/) — design docs for past restructures.
+
+## How the docs are organized
+
+Each `tools/<domain>.md` file lists every tool in that domain with:
+- the tool name as registered with the MCP server,
+- a one-line purpose,
+- the inputs (parameters & types),
+- behaviour notes (validation, auto-injection, server-side defaults),
+- the underlying HTTP endpoint hit on the DNIO platform.
+
+The docs intentionally describe **only what is in the code today**. Stub domains (functions, formulas, users, user-groups, api-keys, bots, data-formats) are not documented because no tools are registered for them yet.

package/docs/architecture.md

@@ -0,0 +1,171 @@
+# Architecture
+
+## Two parallel trees
+
+The codebase splits around two parallel concerns:
+
+```
+src/
+├── clients/          # HTTP layer, one file per product domain
+│   ├── base-client.js          # shared got + token mgmt
+│   ├── apps.js
+│   ├── services.js
+│   ├── records.js
+│   ├── connectors.js
+│   ├── plugins.js
+│   ├── data-pipes.js
+│   ├── deployment-groups.js
+│   └── <stubs>.js              # data-formats, formulas, functions, users, …
+└── tools/            # MCP tool registrations, one file per product domain
+    ├── _helpers.js             # toolError + resolveServiceOrError
+    ├── mcp-tools-registry.js   # master: imports every domain registrar
+    ├── apps.js
+    ├── services.js
+    ├── records.js
+    ├── connectors.js
+    ├── plugins.js
+    ├── data-pipes.js
+    ├── deployment-groups.js
+    ├── data-service-registry.js # generates per-service typed tools dynamically
+    └── <stubs>.js
+```
+
+Each domain owns:
+- a **client class** in `src/clients/<domain>.js` that defines the HTTP methods,
+- a **tool registrar** in `src/tools/<domain>.js` that exports a `(ctx) => void` function registering MCP tools.
+
+The masters (`src/services/dnio-client.js` and `src/tools/mcp-tools-registry.js`) just compose the per-domain modules.
+
+## Master client
+
+`src/services/dnio-client.js`:
+
+```js
+class DNIOClient extends BaseClient {
+    constructor(opts) {
+        super(opts);
+        this.apps             = new AppsClient(this);
+        this.services         = new ServicesClient(this);
+        this.records          = new RecordsClient(this);
+        this.connectors       = new ConnectorsClient(this);
+        this.plugins          = new PluginsClient(this);
+        this.dataPipes        = new DataPipesClient(this);
+        this.deploymentGroups = new DeploymentGroupsClient(this);
+        // stubs
+        this.functions        = new FunctionsClient(this);
+        this.dataFormats      = new DataFormatsClient(this);
+        this.formulas         = new FormulasClient(this);
+        this.users            = new UsersClient(this);
+        this.userGroups       = new UserGroupsClient(this);
+        this.apiKeys          = new ApiKeysClient(this);
+        this.bots             = new BotsClient(this);
+    }
+}
+```
+
+`BaseClient` (in `src/clients/base-client.js`) owns:
+- the `got` instance (with `prefixUrl`, timeout, default headers),
+- `setToken(token)` — rebuilds the `got` instance with the new `Authorization` header,
+- `_request(method, path, options)` — internal request helper with error normalization,
+- `get` / `post` / `put` / `delete` — exposed HTTP verbs,
+- `_logRequest(method, url, options)` — emits one `logger.debug` line per outbound request, with method, decoded query params, and JSON-stringified body. Lives in one place so every domain client gets it for free.
+
+Each domain client receives the master in its constructor and uses `this.http.get/post/...` for HTTP. Domain clients are stateless w.r.t. auth — `dnioClient.setToken(...)` mutates the master and every domain call automatically picks up the new token.
+
+Cross-domain calls go through the master. Example: `services.verifyPod(app, path)` calls `this.http.records.count(app, path)` to test whether the K8s pod for a service is up.
+
+## Master tool registrar
+
+`src/tools/mcp-tools-registry.js`:
+
+```js
+function registerAllTools(server, dnioClient, authManager, registry, userContext) {
+    const ctx = {server, dnioClient, authManager, registry, userContext};
+
+    registerAppsTools(ctx);
+    registerServicesTools(ctx);
+    registerRecordsTools(ctx);
+    registerConnectorsTools(ctx);
+    registerPluginsTools(ctx);
+    registerDataPipesTools(ctx);
+    registerDeploymentGroupsTools(ctx);
+    // stubs (no-op until those domains get tools):
+    registerFunctionsTools(ctx);
+    registerDataFormatsTools(ctx);
+    registerFormulasTools(ctx);
+    registerUsersTools(ctx);
+    registerUserGroupsTools(ctx);
+    registerApiKeysTools(ctx);
+    registerBotsTools(ctx);
+}
+```
+
+Each domain registrar destructures from the same `ctx`, so they all share the same `registry` instance — when `select_app` mutates `registry.selectedApp`, every other tool sees it.
+
+## App-state model
+
+The `ServiceRegistry` (in `src/services/service-registry.js`) is the single source of truth for the currently selected app:
+
+- `registry.selectedApp` — the app name (`null` until `select_app` succeeds).
+- `registry.services` — `Map<serviceId, ServiceEntry>` of every service discovered in that app.
+- `registry.prefixToServiceId`, `registry.nameToServiceId` — fast lookups for `resolveService(identifier)`.
+
+`select_app` triggers `registry.loadApp(appName, adminToken, userToken)`, which:
+1. Lists active services via `client.services.list(appName, {filter: {status: 'Active'}, …})`.
+2. For each service, fetches its full schema if the list response didn't include it.
+3. Computes a `toolPrefix` (slugified service name).
+4. Stores everything in the registry maps.
+5. Triggers `data-service-registry.js` to register per-service typed CRUD tools.
+
+`refresh_services` re-runs the same logic without restarting.
+
+## Static vs. dynamic tools
+
+There are two layers of record CRUD tools:
+
+- **Static, generic** — `list_records`, `get_record`, `create_record`, `update_record`, `delete_record`, `count_records` (in `tools/records.js`). They take a `serviceName` argument and resolve it via `resolveServiceOrError(registry, serviceName)`.
+- **Dynamic, typed** — registered by `tools/data-service-registry.js` after `select_app`. Each service gets six tools named `<toolPrefix>_list/_get/_create/_update/_delete/_count`. Their input schemas are converted from the service's `definition` array via `src/schemas/schema-converter.js`.
+
+Both layers exist so the agent has the option of working generically against any service (when it doesn't know the schema yet) or with strongly-typed inputs once the service is known.
+
+## Transports
+
+`src/index.js` supports two transports:
+
+### stdio (`TRANSPORT=stdio`, default)
+- Single shared `DNIOClient`, `ServiceRegistry`, and MCP server for the lifetime of the process.
+- Used by Claude Desktop's local MCP integration and Cursor's MCP config.
+
+### HTTP (`TRANSPORT=http`)
+- An Express server at `MCP_PORT` (default 3100).
+- Per-session `DNIOClient` + `ServiceRegistry` + MCP server, keyed by `Mcp-Session-Id` header.
+- `SessionManager` (in `src/services/session-manager.js`) handles session creation, expiry (`SESSION_TTL`, default 2h), and cleanup.
+- Routes:
+  - `POST /mcp` — tool calls / MCP requests; creates a new session if no header present.
+  - `GET /mcp` — SSE notifications (requires session).
+  - `DELETE /mcp` — close session.
+  - `GET /health` — health + active session count.
+  - `GET /sessions?key=<admin_pass>` — debug listing of active sessions.
+  - `GET /.well-known/oauth-*` — return 404; the server is authless.
+
+## Authentication
+
+All MCP-server-to-DNIO API calls use a JWT in the `Authorization: JWT <token>` header. The token is set on the master client via `setToken`. There are two tokens, both managed by `src/services/auth-manager.js`:
+
+- **Admin token** — for management endpoints (listing apps, fetching schemas, creating data services, deploying flows).
+- **MCP-user token** — for data plane (record CRUD).
+
+See [`authentication.md`](authentication.md) for the lifecycle.
+
+## Where things live (cheat sheet)
+
+| Concern | File |
+|---|---|
+| Add a new domain client | `src/clients/<domain>.js` (plus mount in `services/dnio-client.js`) |
+| Add a new domain tool file | `src/tools/<domain>.js` (plus require + call in `tools/mcp-tools-registry.js`) |
+| HTTP plumbing / token lifecycle | `src/clients/base-client.js`, `src/services/auth-manager.js` |
+| App / service discovery state | `src/services/service-registry.js` |
+| Per-service typed tools | `src/tools/data-service-registry.js` |
+| HTTP-transport sessions | `src/services/session-manager.js`, `src/index.js` |
+| Shared tool helpers | `src/tools/_helpers.js` (`toolError`, `resolveServiceOrError`) |
+| Spec / docs surfaced to the LLM | `src/utils/constants.js` (data-service creation spec, connector/role docs) |

package/docs/authentication.md

@@ -0,0 +1,74 @@
+# Authentication
+
+The MCP server manages two separate JWTs against the DNIO platform and switches between them based on the operation. Tokens auto-refresh; consumers of the MCP tools never see them.
+
+## Two tokens
+
+| Token | Used for | Provisioned by |
+|---|---|---|
+| **Admin token** | Management / control plane: `list_apps`, `get_service_schema`, anything that lists apps or fetches data-service definitions across apps. | `DNIO_USERNAME` / `DNIO_PASSWORD` env vars. |
+| **MCP-user token** | Data plane: every record CRUD, plus most domain-specific operations (creating connectors, designing flows, managing deployment groups). | `MCP_USER_EMAIL` / `MCP_USER_PASSWORD` env vars — the server looks up this account; if it doesn't exist on the platform, it creates it on first boot. |
+
+`auth-manager.js` (`src/services/auth-manager.js`) owns both lifecycles. It exits the process at startup if `MCP_USER_EMAIL` or `MCP_USER_PASSWORD` is unset; `src/index.js` does the same for `DNIO_BASE_URL` / `DNIO_USERNAME` / `DNIO_PASSWORD`. There are no hardcoded credentials anywhere in the code.
+
+## Boot sequence
+
+1. **Login as admin.** POST `/api/a/rbac/auth/login` with `{username, password, isSuperAdmin: true}`. Stores the admin token + computes its expiry from `DNIO_TOKEN_TTL`.
+2. **Look up MCP user.** GET `/api/a/rbac/admin/user?filter={"email":"mcp-user@dnio.com"}` (with admin token). If found, reuse; otherwise:
+3. **Create MCP user.** POST `/api/a/rbac/${DNIO_NAMESPACE}/user` with the MCP user payload. Records the user's `_id`.
+4. **Add MCP user to all apps.** GET `/api/a/rbac/admin/app?count=-1&select=_id` (admin token), then PUT `/api/a/rbac/admin/user/utils/addToApps/mcp-user@dnio.com` with the full app list. This is what gives the MCP user data-plane access across the platform.
+5. **Login as MCP user.** Same `/auth/login` endpoint with the MCP user's stored credentials. Stores the MCP-user token + expiry.
+
+After step 5, the server is ready; both tokens are warm.
+
+## Per-tool token switching
+
+Tools call `dnioClient.setToken(token)` before each HTTP request to point the master client at the right JWT for that operation.
+
+- `tools/apps.js` `list_apps` — uses admin token, then resets to MCP-user token before returning.
+- `tools/apps.js` `select_app` — uses both: admin to list services + fetch schemas, MCP-user for pod verification.
+- `tools/apps.js` `get_service_schema` — admin token (then resets).
+- Almost every other tool — MCP-user token.
+
+The pattern is always `dnioClient.setToken(...)` immediately before the API call inside the tool handler. Because every domain client shares the same `BaseClient` `got` instance, one `setToken` updates the auth header for every subsequent call.
+
+## Refresh
+
+`AuthManager` schedules a refresh just before each token expires (`DNIO_TOKEN_TTL`, default 1800 s):
+
+- For the admin token: re-run the admin login.
+- For the MCP-user token: re-run the MCP-user login.
+
+If a tool is mid-flight when refresh fires, the in-flight call uses whatever token the master client had at the moment of the request. Subsequent calls pick up the refreshed token because `setToken` rebuilds the `got` instance with the new header.
+
+If the admin password changes externally, restart the server — there's no UI for re-prompting credentials.
+
+## HTTP-transport sessions
+
+In HTTP mode (`TRANSPORT=http`), every Mcp-Session creates its own `DNIOClient` initialized with the current MCP-user token, and that per-session client lives inside `SessionManager`. Token refresh on `AuthManager` does not propagate into already-issued session clients — a session that lives for hours will keep using its initial token until either (a) a tool inside that session calls `setToken` again with the latest from `authManager.getUserToken()`, or (b) the session expires (`SESSION_TTL`, default 7200 s).
+
+Practically every tool handler does call `dnioClient.setToken(userContext.token)` and `userContext.token` is refreshed on `select_app`, so this rarely surfaces. If you see auth failures after a session has been idle for >30 minutes, the `SESSION_TTL` cleanup is the simplest fix.
+
+## stdio-transport
+
+In stdio mode there's a single `DNIOClient`, `ServiceRegistry`, and MCP server for the process. Token refresh on `AuthManager` is enough — every tool handler calls `setToken` against that single client.
+
+## Endpoints used
+
+All against `${DNIO_BASE_URL}`:
+
+| Method | Path | Purpose |
+|---|---|---|
+| POST | `/api/a/rbac/auth/login` | Admin and MCP-user login |
+| GET | `/api/a/rbac/admin/user?filter=…` | Find the MCP user |
+| POST | `/api/a/rbac/${namespace}/user` | Create MCP user |
+| GET | `/api/a/rbac/admin/app?count=-1&select=_id` | List apps for provisioning |
+| PUT | `/api/a/rbac/admin/user/utils/addToApps/${email}` | Grant MCP user app access |
+
+The MCP user's password is generated on first creation and persisted by the platform; the server doesn't store it locally beyond memory.
+
+## Security notes
+
+- The MCP server's HTTP transport is **authless** — anyone with the URL can connect. If you deploy it externally, put it behind a network policy or auth proxy.
+- `GET /sessions?key=<admin_pass>` is a debug endpoint that uses the admin password as a shared secret. Disable it in production by removing the route or wrapping it in middleware.
+- Tokens are kept in memory only. They never hit disk and aren't logged (the request logger logs path/query/body but never headers).

package/docs/tools/apps.md

@@ -0,0 +1,59 @@
+# Apps & Service Discovery
+
+Tools registered by `src/tools/apps.js`. They drive the entry-point flow (`list_apps` → `select_app`) and provide read-only inspection of services in the selected app.
+
+Backed by `AppsClient` (`src/clients/apps.js`) and `ServicesClient.getSchema` (`src/clients/services.js`).
+
+## `list_apps`
+
+| | |
+|---|---|
+| **Purpose** | List every app on the platform. The first tool to run in any session. |
+| **Inputs** | (none) |
+| **Endpoint** | `GET /api/a/rbac/admin/app?count=-1&select=_id,description` (admin token) |
+| **Auth** | Admin token, then resets to MCP-user token before returning. |
+| **Returns** | `{ platform, user, selectedApp, apps: [{ appName, description }] }` |
+
+## `select_app`
+
+| | |
+|---|---|
+| **Purpose** | Switch the active app. Discovers every active data service in the app, verifies pods, loads schemas, and registers per-service typed CRUD tools. Required before any data-plane tool. |
+| **Inputs** | `appName` (string, required) |
+| **Behaviour** | 1. Calls `authManager.ensureUserAppAccess(appName)` to make sure the MCP user has access. 2. Refreshes `userContext.token`. 3. Calls `registry.loadApp(appName, adminToken, userToken)`, which lists active services, fetches each service's `definition`, and stores them in the registry. 4. Triggers per-service tool registration via `data-service-registry.js`. |
+| **Endpoints** | `GET /api/a/sm/${app}/service?filter={"app":"${app}","status":"Active"}&count=-1&select=…` (admin) for listing; `GET /api/a/sm/${app}/service/${serviceId}` (admin) per service that needs schema fetch. |
+| **Returns** | `{ action: 'app_selected', appName, services: [...], skipped: [...]?, totalLoaded, usage }` |
+
+## `list_services`
+
+| | |
+|---|---|
+| **Purpose** | Show every loaded data service for the currently selected app, including the parsed schema description used for tool generation. Read-only — uses the in-memory `ServiceRegistry`. |
+| **Inputs** | (none) |
+| **Endpoint** | (none — reads `registry.getServiceList()`) |
+| **Returns** | `{ appName, services: [{ name, serviceId, apiPath, schema }] }` |
+
+## `refresh_services`
+
+| | |
+|---|---|
+| **Purpose** | Re-scan the selected app for new/changed services without restarting the server. Useful after creating a new data service in the same session. |
+| **Inputs** | (none) |
+| **Behaviour** | Re-runs `registry.loadApp` for the currently selected app. |
+| **Returns** | `{ action: 'refreshed', appName, loaded: [name…], skipped: [...] }` |
+
+## `get_service_schema`
+
+| | |
+|---|---|
+| **Purpose** | Fetch the full schema of a single service — useful before constructing a record payload for `create_record`. |
+| **Inputs** | `serviceName` (string, required) — name, slug prefix, or `_id`. |
+| **Behaviour** | Resolves `serviceName` via `registry.resolveService` (matches by `_id`, lowercase name, or slugified prefix). If not found, returns the available list. |
+| **Endpoint** | `GET /api/a/sm/${app}/service/${serviceId}` (admin token) |
+| **Returns** | `{ serviceId, name, api, attributeCount, status, definition }` |
+
+## Notes
+
+- `select_app` is the only tool in this domain that mutates server-side state — it sets `registry.selectedApp` and rebuilds the per-service typed tool set.
+- `list_services` and `get_service_schema` will return errors when no app is selected.
+- `select_app` fetches schemas individually only when the list response did not already include `definition`. The platform's list endpoint generally returns it; the per-service GET is a safety net.

package/docs/tools/connectors.md

@@ -0,0 +1,76 @@
+# Connectors
+
+Tools registered by `src/tools/connectors.js`. A **connector** is a configured backing system (a database, an S3 bucket, an SFTP server, an SMTP relay, an ActiveMQ queue, …) that data services and flow nodes bind to.
+
+The platform separates **connector types** (templates from a marketplace catalog) from **connector instances** (concrete configured ones in the selected app).
+
+Backed by `ConnectorsClient` (`src/clients/connectors.js`).
+
+## Where connectors come from
+
+```
+list_connector_types     →  marketplace catalog (templates)
+        │
+        ▼
+create_connector         →  POST /api/a/bm/${app}/connector
+        │                   creates an instance in this app
+        ▼
+list_connectors          →  GET /api/a/rbac/${app}/connector
+                            lists existing instances
+```
+
+The instance API and the marketplace API live under different namespaces (`/api/a/rbac` vs `/api/a/bm`).
+
+## Tool reference
+
+### `list_connectors`
+
+| | |
+|---|---|
+| **Purpose** | List existing connector instances in the selected app. Used by `create_data_service` (auto-fill) and as the source for connector IDs throughout the platform. |
+| **Inputs** | `category` (optional, one of `DB` or `STORAGE`). Omit for all categories (`MESSAGING`, `STORAGE`, `DB`, etc.). |
+| **Endpoint** | `GET /api/a/rbac/${app}/connector?filter={"app":"${app}"[, "category":"…"]}&count=-1&select=_id,name,category,subCategory,type,options,_metadata` |
+| **Returns** | `{ app, count, connectors: [{ _id, name, category, subCategory, type, isDefault, isValid }], usage }` |
+
+`isDefault` is true for the platform's pre-provisioned default connectors (a default MongoDB DB connector and a default GridFS storage connector). `create_data_service` auto-attaches these when the LLM doesn't specify a connector.
+
+### `list_connector_types`
+
+| | |
+|---|---|
+| **Purpose** | List connector **templates** in the marketplace. Each item shows the marketplace `_id` (used as `marketItemId` for `create_connector`), the `type` / `label` / `category`, and a `fields[]` schema describing what credentials each type needs. |
+| **Inputs** | `category` (optional, e.g. `DB`, `STORAGE`, `MESSAGING`), `search` (optional, substring on label/type/tags). |
+| **Endpoint** | `GET /api/a/bm/${app}/marketplace/connector?count=1000&select=label,thumbnail,type,fields,category,tags,version` |
+| **Behaviour** | Strips the `thumbnail` SVG from the response to keep payloads small. Filtering is client-side. |
+| **Returns** | `{ app, count, types: [{ marketItemId, type, label, category, tags, version, fields: [{ key, label, type, htmlInputType, required, encrypted }] }], usage }` |
+
+The `fields[]` schema tells the agent exactly what to ask the user for. Examples seen in the wild:
+
+| Type | Fields |
+|---|---|
+| `MONGODB` | `connectionString` (encrypted) |
+| `S3` | `accessKeyId`, `secretAccessKey` (encrypted), `region`, `bucket` |
+| `ACTIVEMQ` | `host`, `port`, `username`, `password` (encrypted) |
+| `SFTP` / `SMTP` / etc. | various |
+
+### `create_connector`
+
+| | |
+|---|---|
+| **Purpose** | Create a new connector instance in the selected app. Always preceded by `list_connector_types`. |
+| **Inputs** | `name` (required, display name), `marketItemId` (required, from `list_connector_types`), `values` (required, JSON string keyed by each field's `key`). |
+| **Behaviour** | Builds payload `{ values, name, app: registry.selectedApp, marketItemId }` — the `app` field is server-injected from the selected app, regardless of what the LLM passes. Encrypted fields (e.g. passwords, secret keys) are sent as-is; the platform encrypts them at rest. |
+| **Endpoint** | `POST /api/a/bm/${app}/connector` |
+| **Returns** | The platform's response (the created connector). |
+
+## Typical sequence
+
+```
+list_connector_types(category: 'DB', search: 'mongo')
+  → user picks one (marketItemId, fields)
+  → ask the user for each field's value (passwords for encrypted ones)
+create_connector(name, marketItemId, values)
+list_connectors                # confirm it shows up
+```
+
+The created instance can then be used as `connectors.data._id` (for DB) or `connectors.file._id` (for STORAGE) when calling `create_data_service`.