executor 1.0.6 → 1.1.1

package/README.md

@@ -1,111 +1,314 @@
-# eXecutor
-### **_A powerful "short-cutter" to your console to you and your team!!_**
+# executor
 
-[![npm](https://img.shields.io/npm/v/executor.svg?style=flat-square)](https://www.npmjs.com/package/executor) 
-[![CircleCI](https://circleci.com/gh/crystian/executor/tree/master.svg?style=shield)](https://circleci.com/gh/crystian/executor/tree/master)
-[![Codacy grade](https://img.shields.io/codacy/grade/d3f65a1bc7604109843a0b9bda912c3b.svg?style=flat-square)](https://www.codacy.com/app/crystian/executor?utm_source=github.com&utm_medium=referral&utm_content=crystian/executor&utm_campaign=Badge_Grade)
-[![Codacy coverage](https://img.shields.io/codacy/coverage/d3f65a1bc7604109843a0b9bda912c3b.svg?style=flat-square)](https://www.codacy.com/app/crystian/executor?utm_source=github.com&utm_medium=referral&utm_content=crystian/executor&utm_campaign=Badge_Coverage)
-[![npm](https://img.shields.io/npm/dm/executor.svg?style=flat-square)](https://www.npmjs.com/package/executor)
-[![HitCount](http://hits.dwyl.com/crystian/executor.svg)](http://hits.dwyl.com/crystian/executor)
+`executor` is a local-first execution environment for AI agents.
 
-[![license](https://img.shields.io/npm/l/executor.svg?style=flat-square)](https://raw.githubusercontent.com/crystian/executor/master/LICENSE)
-[![David](https://img.shields.io/david/crystian/executor.svg?style=flat-square)](https://github.com/crystian/executor/blob/master/package.json)
-[![Gitter](https://img.shields.io/gitter/room/crystian/executor.svg?style=flat-square)](https://gitter.im/crystian/executor/??utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+It gives an agent a TypeScript runtime, a discoverable tool catalog, and a single local place to connect external systems such as MCP servers, OpenAPI APIs, and GraphQL APIs. Instead of pasting large MCP manifests into every chat or giving an agent broad shell access, you run code inside `executor` and let it call typed `tools.*` functions.
 
-[ english ![](docs/img/en.png) ] [ [spanish ![](docs/img/sp.png) ](docs/README-ES.md) ]
+At runtime, `executor` behaves like one local product:
 
----
+- a CLI for starting the runtime and executing code
+- a local API server
+- a local web UI for connecting sources, inspecting tools, and managing secrets
+- an MCP endpoint for hosts that want to drive `executor` through MCP
 
-## TL;DR
+The current codebase is the v3 rewrite. Older experiments live in `legacy/` and `legacy2/`, but the active architecture is the one in `apps/` and `packages/`.
 
-This tool allows you to save a lot of time writing commands with totally custom shortcuts, that you can share on the project with your "human" team or machines, with a unique code on the same place. #agile #needForSpeed. 
- 
-> Like `scripts` of `package.json` but with _steroids_.
+## Why this exists
 
-**Notes:** The idea is not to drop the console; it's for optimising it!
-  
-### Features
+`executor` is built around a simple idea: agents should work against a structured tool environment instead of guessing at raw HTTP calls, carrying huge MCP definitions in context, or running arbitrary local commands with broad permissions.
 
-* Multi-platform: An only code that works on all platforms. Powered by [nodejs](https://nodejs.org)
-* Templates for easy reuse
-* It does not matter what kind of project is, if you write commands this tool can be helpful for you
-* There are five source types of variables: Predefined, environment, package.json, templates and shortcuts.
-* Devs and DevOps can be friends with it because they can share their scripts.
-* Typo on long command lines are past problems!
-* Do away with one thousand of similar scripts on your `package.json`!  
-(`build`, `build-ci`, `build-prod`, `build-prod-ci`, `build-qa`, `build-qa-ci`, `build-ci-cache`, etc...)
-* The `scripts` on the `package.json` were modified and have to `rebuild` your docker image, never again!
-* **Without dependencies from other modules!**
+In practice that means:
 
-![](docs/img/jake.gif)
+- sources are connected once and turned into a reusable workspace tool catalog
+- the agent discovers tools by intent, inspects schemas, and then calls typed functions
+- secrets and OAuth flows stay in the local runtime and web UI instead of being pasted into chat
+- human interaction can pause an execution and resume it cleanly
 
----
-## Documentation
+## Mental model
 
-### Spoiler
+Think of `executor` as a local control plane for agent tool use.
 
-##### Before
-On `package.json`
+1. You start a local `executor` daemon.
+2. You connect sources such as an MCP server, an OpenAPI document, or a GraphQL endpoint.
+3. `executor` indexes those sources into a workspace tool catalog.
+4. An agent runs TypeScript against that catalog through `executor call` or through the MCP bridge.
+5. If a tool needs credentials or user input, execution pauses, opens a local flow, and then resumes.
 
+## What it does today
+
+### Connect external tool sources
+
+`executor` currently supports these source types:
+
+- `mcp`: remote MCP servers, including transport selection for streamable HTTP or SSE
+- `openapi`: REST APIs described by an OpenAPI document
+- `graphql`: GraphQL endpoints that can be introspected into callable tools
+
+The add-source flow can:
+
+- probe a URL and infer what kind of source it is
+- infer likely authentication requirements
+- prompt for credentials when discovery or connection needs them
+- start OAuth when a source requires it
+- persist the source and its indexed tool metadata in the local workspace
+
+The web app also includes templates for common providers so you can start from real examples instead of filling every field by hand.
+
+### Run agent code against tools
+
+The main CLI workflow is `executor call`.
+
+The runtime expects the agent to use the built-in discovery workflow:
+
+```ts
+const matches = await tools.discover({ query: "github issues", limit: 5 });
+const path = matches.bestPath;
+const detail = await tools.describe.tool({ path, includeSchemas: true });
+
+return await tools.github.issues.list({
+  owner: "vercel",
+  repo: "next.js",
+});
 ```
-"scripts": {
-  "build": "docker run --rm -it -p 4200:4200 -v /choclo:... --name myProject-dev myProject ng build",
-  "build-prod": "docker run --rm -it -p 4200:4200 -v /choclo:... --name myProject-dev myProject ng build --prod",
-  "server": "docker run --rm -it -p 4200:4200 -v /choclo:... --name myProject-dev myProject ng serve"
-}
-```
-##### Executor
+
+A few important rules shape the execution model:
+
+- write TypeScript, not raw shell pipelines
+- use `tools.*`, not direct `fetch`
+- discover first when the exact tool path is not known
+- inspect schemas before calling complex tools
+
+### Handle credentials and user interaction
+
+When a source or tool needs human input, `executor` can pause the execution and create an interaction record.
+
+That interaction may ask you to:
+
+- open a secure local credential page
+- complete an OAuth flow in the browser
+- respond to a structured elicitation from a tool host
+- resume a paused execution from the CLI
+
+This is the core human-in-the-loop behavior that lets `executor` keep secrets and approvals outside the agent's raw context.
+
+### Inspect the connected tool model
+
+The web UI is not just a setup surface. It is also where you can inspect what `executor` learned from a source.
+
+For each source you can:
+
+- browse its tool tree
+- search for tools by intent
+- inspect input and output schemas
+- view generated manifests, definitions, and raw source documents when available
+- edit source settings and authentication details
+
+## Quick start
+
+If you are working from this repository, the easiest path is:
+
+```bash
+bun install
+bun run executor doctor --json
+bun run executor up
 ```
-"templates": {
-  "docker-common": "docker run --rm -it -p 4200:4200 -v /choclo:... --name ${pkg.name}-dev ${pkg.name}" 
-},
-"shortcuts": {
-  "build": "${docker-common} ng build",
-  "build-prod": "${docker-common} ng build --prod",
-  "server": "${docker-common} ng serve"
-}
+
+That starts the local runtime. The default base URL is:
+
+```text
+http://127.0.0.1:8788
 ```
 
-### Table of content
+From there:
 
-* [Intro](https://github.com/crystian/executor/wiki/Home)
-* [Why?](https://github.com/crystian/executor/wiki/1-why#why)
-* [Installation](https://github.com/crystian/executor/wiki/2-installation#installation)
-* [Configuration](https://github.com/crystian/executor/wiki/3-configuration#configuration)
-  * [sources](https://github.com/crystian/executor/wiki/3-configuration#sources)
-  * [configuration](https://github.com/crystian/executor/wiki/3-configuration#configuration-1)
-    * [options](https://github.com/crystian/executor/wiki/3-configuration#options)
-    * [data](https://github.com/crystian/executor/wiki/3-configuration#data)
-      * [predefined (def)](https://github.com/crystian/executor/wiki/3-configuration#predefined-def)
-      * [environments (env)](https://github.com/crystian/executor/wiki/3-configuration#environments-env)
-      * [package.json (pkg)](https://github.com/crystian/executor/wiki/3-configuration#packagejson-pkg)
-      * [templates](https://github.com/crystian/executor/wiki/3-configuration#templates)
-      * [shortcuts](https://github.com/crystian/executor/wiki/3-configuration#shortcuts)
-* [Bonus track](https://github.com/crystian/executor/wiki/4-bonus)
-* [contributing](https://github.com/crystian/executor/wiki/5-contributing)
+1. Open the web UI in your browser.
+2. Add a source from `/sources/add`.
+3. If needed, store credentials in `/secrets`.
+4. Run TypeScript with `bun run executor call ...`.
 
----
+If you are using a packaged distribution, the command name is simply `executor` instead of `bun run executor`.
 
-* [Changelog](CHANGELOG.md)
+## Core CLI commands
 
-And remember: RTFM! :)
+```bash
+executor up
+executor down
+executor status --json
+executor doctor --json
+executor call --file script.ts
+executor resume --execution-id exec_123
+```
 
-## Installation
+`executor call` accepts code in three ways:
 
+- inline as a positional argument
+- from `--file`
+- from standard input with `--stdin`
+
+Examples:
+
+```bash
+executor call 'const matches = await tools.discover({ query: "repo details", limit: 1 }); return matches;'
+executor call --file script.ts
+cat script.ts | executor call --stdin
+executor call --no-open --file script.ts
 ```
-npm i -g executor
+
+If an execution pauses, resume it with:
+
+```bash
+executor resume --execution-id exec_123
 ```
 
-Test to check the installation:
+## Adding a source
+
+There are two main ways to add a source.
+
+### In the web UI
+
+Use the Add Source flow to:
+
+- paste a URL
+- run discovery
+- review the inferred kind, namespace, transport, and auth
+- connect the source
+- complete credential or OAuth setup if required
+
+This is the easiest path for most users.
 
+### From inside an execution
+
+The runtime also exposes `tools.executor.sources.add(...)`, which lets an agent add a source from code.
+
+Examples:
+
+```ts
+return await tools.executor.sources.add({
+  endpoint: "https://example.com/mcp",
+  name: "Example",
+  namespace: "example",
+});
 ```
-x hello
+
+```ts
+return await tools.executor.sources.add({
+  kind: "openapi",
+  endpoint: "https://api.github.com",
+  specUrl: "https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json",
+  name: "GitHub",
+  namespace: "github",
+});
 ```
 
-It should show a message with installed version number.
+For HTTP-style sources, `executor` can drive the credential flow for you.
+
+## How execution works
+
+At a high level, every execution follows the same loop:
+
+1. `executor` resolves the current local installation and workspace.
+2. It builds a tool catalog from built-in tools plus all connected workspace sources.
+3. It runs your TypeScript inside the in-process execution runtime.
+4. Tool calls are dispatched through `executor` rather than directly from your code.
+5. If a tool needs interaction, the run pauses and records a pending interaction.
+6. Once the interaction is resolved, the execution continues and eventually completes or fails.
+
+This gives you a stable surface for agent automation:
+
+- the agent sees a coherent catalog
+- connected sources become reusable namespace-based tools
+- auth stays attached to sources and secret material
+- the runtime can track execution state instead of losing it inside a one-shot prompt
+
+## Web UI overview
+
+The React web app is served from the same local server as the API.
+
+Main screens:
+
+- `/`: list connected sources in the current local workspace
+- `/sources/add`: discover and connect new sources
+- `/sources/:sourceId`: inspect tools, search tools, and browse source artifacts
+- `/sources/:sourceId/edit`: edit source settings and auth
+- `/secrets`: create, update, and delete locally stored secrets
+
+The UI uses the same control-plane API as the CLI, so both surfaces are operating on the same local runtime state.
+
+## Local-first runtime behavior
+
+By default `executor` runs as a single local daemon process.
+
+It serves:
+
+- `/v1` for the local control-plane API
+- `/mcp` for the `executor` MCP endpoint
+- the web UI for normal browser routes
+
+Default network location:
+
+- host: `127.0.0.1`
+- port: `8788`
+
+Default data locations are OS-aware:
+
+- Linux data: `~/.local/share/executor`
+- Linux runtime state: `~/.local/state/executor/run`
+- macOS: `~/Library/Application Support/Executor`
+- Windows: `%LOCALAPPDATA%\Executor`
+
+The server also maintains local PID and log files in its runtime directory.
+
+## Persistence and data
+
+`executor` persists the local control plane to SQL.
+
+By default it uses embedded PGlite for a local install, which keeps the product self-contained. If you provide a Postgres URL, the persistence layer can target real Postgres instead.
+
+Persisted concepts include:
+
+- the local installation record
+- organizations and workspaces provisioned for that install
+- connected sources
+- indexed tool artifacts and related metadata
+- credentials and secret material bindings
+- source auth sessions
+- executions and execution interactions
+- policies
+
+On first start, `executor` provisions a local account, a personal organization, and a default workspace automatically.
+
+## Security and trust model
+
+`executor` is designed to narrow how agents interact with external systems.
+
+Compared with direct shell or raw API usage, the model is intentionally more structured:
+
+- tool calls are routed through a controlled runtime
+- secrets are stored separately from prompt text
+- OAuth and credential capture happen through local flows
+- executions can pause for interaction instead of guessing or failing silently
+- source auth and tool metadata live with the workspace rather than inside each prompt
+
+This does not make the system magically risk-free, but it gives the runtime places to enforce policy, collect approvals, and keep sensitive material out of the agent's immediate context.
+
+## Repository layout
+
+If you are exploring the repo, these are the directories that matter most:
+
+- `apps/executor`: packaged CLI entrypoint and daemon lifecycle commands
+- `apps/web`: local React web UI
+- `packages/server`: local HTTP server that serves API, MCP, and UI
+- `packages/control-plane`: source management, secrets, persistence, execution, and inspection
+- `packages/runtime-local-inproc`: in-process TypeScript execution runtime
+- `packages/executor-mcp`: MCP bridge for `execute` and `resume`
+- `packages/codemode-*`: core tool abstractions plus MCP and OpenAPI adapters
+
+## Project status
 
-[More info about installation.](https://github.com/crystian/executor/wiki/2-installation#installation)
+This repository is explicitly on its third major architecture iteration.
 
----
+- `apps/` and `packages/` are the active implementation
+- `legacy/` is the original codebase
+- `legacy2/` is the second generation
 
-MIT © 2018 [Crystian](https://github.com/crystian), made with love for you <3!
+If you want the system view instead of the product view, read [`ARCHITECTURE.md`](./ARCHITECTURE.md).

package/bin/deno-subprocess-worker.mjs

@@ -0,0 +1,205 @@
+// @ts-nocheck
+// This script runs inside the Deno subprocess.
+// It communicates with the host process via line-delimited JSON over stdin/stdout.
+// All IPC messages are prefixed with @@executor-ipc@@ to distinguish from user output.
+
+const encoder = new TextEncoder();
+const IPC_PREFIX = "@@executor-ipc@@";
+
+const pendingToolCalls = new Map();
+let started = false;
+
+/** @type {string[]} */
+const logs = [];
+
+const writeIpcMessage = (message) => {
+  const payload = `${IPC_PREFIX}${JSON.stringify(message)}\n`;
+  Deno.stdout.writeSync(encoder.encode(payload));
+};
+
+const toErrorMessage = (error) => {
+  if (error instanceof Error) {
+    return error.stack ?? error.message;
+  }
+
+  return String(error);
+};
+
+const createToolCaller = (toolPath) => (args) =>
+  new Promise((resolve, reject) => {
+    const requestId = crypto.randomUUID();
+    pendingToolCalls.set(requestId, { resolve, reject });
+
+    writeIpcMessage({
+      type: "tool_call",
+      requestId,
+      toolPath,
+      args: args === undefined ? {} : args,
+    });
+  });
+
+const createToolsProxy = (path = []) => {
+  const callable = () => undefined;
+
+  return new Proxy(callable, {
+    get(_target, prop) {
+      if (prop === "then") return undefined;
+      if (typeof prop !== "string") return undefined;
+      return createToolsProxy([...path, prop]);
+    },
+    apply(_target, _thisArg, args) {
+      const toolPath = path.join(".");
+      if (!toolPath) {
+        throw new Error("Tool path missing in invocation");
+      }
+
+      return createToolCaller(toolPath)(args.length > 0 ? args[0] : undefined);
+    },
+  });
+};
+
+const formatLogArg = (value) => {
+  if (typeof value === "string") {
+    return value;
+  }
+
+  try {
+    return JSON.stringify(value);
+  } catch {
+    return String(value);
+  }
+};
+
+const formatLogLine = (args) => args.map(formatLogArg).join(" ");
+
+const sandboxConsole = {
+  log: (...args) => {
+    logs.push(`[log] ${formatLogLine(args)}`);
+  },
+  warn: (...args) => {
+    logs.push(`[warn] ${formatLogLine(args)}`);
+  },
+  error: (...args) => {
+    logs.push(`[error] ${formatLogLine(args)}`);
+  },
+  info: (...args) => {
+    logs.push(`[info] ${formatLogLine(args)}`);
+  },
+  debug: (...args) => {
+    logs.push(`[debug] ${formatLogLine(args)}`);
+  },
+};
+
+const runUserCode = async (code) => {
+  const tools = createToolsProxy();
+
+  const execute = new Function(
+    "tools",
+    "console",
+    `"use strict"; return (async () => {\n${code}\n})();`,
+  );
+
+  return await execute(tools, sandboxConsole);
+};
+
+const handleStart = (message) => {
+  if (started) {
+    writeIpcMessage({
+      type: "failed",
+      error: "start message already received",
+      logs,
+    });
+    return;
+  }
+
+  started = true;
+
+  runUserCode(message.code)
+    .then((result) => {
+      writeIpcMessage({
+        type: "completed",
+        result,
+        logs,
+      });
+    })
+    .catch((error) => {
+      writeIpcMessage({
+        type: "failed",
+        error: toErrorMessage(error),
+        logs,
+      });
+    });
+};
+
+const handleToolResult = (message) => {
+  const pending = pendingToolCalls.get(message.requestId);
+  if (!pending) {
+    return;
+  }
+
+  pendingToolCalls.delete(message.requestId);
+
+  if (message.ok) {
+    pending.resolve(message.value);
+    return;
+  }
+
+  pending.reject(new Error(message.error));
+};
+
+const handleHostMessage = (message) => {
+  if (!message || typeof message !== "object") {
+    return;
+  }
+
+  if (message.type === "start") {
+    handleStart(message);
+    return;
+  }
+
+  if (message.type === "tool_result") {
+    handleToolResult(message);
+  }
+};
+
+const decodeLines = async () => {
+  const reader = Deno.stdin.readable.getReader();
+  const decoder = new TextDecoder();
+  let buffer = "";
+
+  while (true) {
+    const { value, done } = await reader.read();
+    if (done) {
+      break;
+    }
+
+    buffer += decoder.decode(value, { stream: true });
+
+    while (true) {
+      const lineEnd = buffer.indexOf("\n");
+      if (lineEnd === -1) {
+        break;
+      }
+
+      const line = buffer.slice(0, lineEnd).trim();
+      buffer = buffer.slice(lineEnd + 1);
+
+      if (line.length === 0) {
+        continue;
+      }
+
+      try {
+        const message = JSON.parse(line);
+        handleHostMessage(message);
+      } catch (error) {
+        writeIpcMessage({
+          type: "failed",
+          error: `invalid host message: ${toErrorMessage(error)}`,
+          logs,
+        });
+      }
+    }
+  }
+};
+
+await decodeLines();

package/bin/executor.js

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+import "./executor.mjs";