@kadi.build/core 0.0.1-alpha.1 → 0.0.1-alpha.11

package/README.md

@@ -1,306 +1,437 @@
 # @kadi.build/core
 
----
-
-The `@kadi.build/core` module is a comprehensive toolkit for developers integrating with the KADI infrastructure. This module simplifies tasks such as managing `agent.json` files, spawning processes, interacting with brokers, and handling interprocess communication (IPC).
-
-## Features
+> Framework for building distributed abilities with multiple transport protocols
 
-- **Agent JSON Management**: Manage configurations for tools, agents, or systems using KADI.
-- **Process Management**: Support launching and managing subprocesses.
-- **Multi-Broker Support**: Dynamic broker selection and management with environment/runtime configuration.
-- **Broker Interaction**: Facilitate communications with WebSocket brokers.
-- **IPC Support**: Tools for interprocess communication across various programming languages.
+[![npm version](https://img.shields.io/npm/v/@kadi.build/core.svg)](https://www.npmjs.com/package/@kadi.build/core)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 ## Installation
 
-Install `@kadi.build/core` using npm:
-
 ```bash
 npm install @kadi.build/core
 ```
 
-## Documentation
+## Quick Start
 
-### Agent JSON Functions
+### Creating an Ability
 
-- **`getAbilityJSON(abilityName, abilityVersion)`**: Retrieves the `agent.json` for a specified ability.
-- **`getAbilityJSONPath(abilityName, abilityVersion)`**: Provides the file path to the `agent.json` for a specific ability.
-- **`getAbilityVersionFromArray(abilities, name)`**: Searches ability array provided, and returns the version number for name provided.
-- **`getAbilitiesDir()`**: Returns the directory path where abilities are stored.
-- **`getProjectJSON()`**: Fetches the `agent.json` for the current project.
-- **`getProjectJSONPath()`**: Gets the file path for the project's `agent.json`.
-- **`getKadiCoreJSON()`**: Retrieves the `agent.json` for the Kadi core.
-- **`getKadiCoreJSONPath()`**: Provides the file path to the Kadi core's `agent.json`.
-- **`getKadiJSON()`**: Fetches the `agent.json` for the Kadi system.
-- **`getKadiJSONPath()`**: Returns the file path for the Kadi system's `agent.json`.
+```javascript
+import { KadiClient, z } from '@kadi.build/core';
 
-### Broker Management Functions
+const ability = new KadiClient({
+  name: 'math-ability',
+  version: '1.0.0'
+});
 
-The broker management system allows dynamic selection from multiple configured brokers defined in `agent.json`:
+// Register tool with Zod schemas (recommended)
+ability.registerTool({
+  name: 'add',
+  description: 'Add two numbers',
+  input: z.object({
+    a: z.number().describe('First number'),
+    b: z.number().describe('Second number')
+  }),
+  output: z.object({
+    result: z.number().describe('Sum of a and b')
+  })
+}, async ({ a, b }) => {
+  return { result: a + b };
+});
 
-```json
-{
-  "brokers": {
-    "local": "ws://127.0.0.1:8080",
-    "remote": "ws://production.example.com:8080",
-    "staging": "ws://staging.example.com:8080"
-  }
-}
+// Start serving (native, stdio, or broker)
+await ability.serve('stdio');
 ```
 
-Available functions:
+### Loading and Using Abilities
+
+```javascript
+import { loadAbility } from '@kadi.build/core';
 
-- **`KADI_BROKERS`**: Object containing all configured brokers with their parsed URLs.
-- **`KADI_BROKER_URL`**: Default broker URL (first one defined, for backward compatibility).
-- **`getBrokerUrl(brokerName)`**: Get URL for a specific broker by name. Returns `null` if not found.
-- **`getBrokerNames()`**: Get array of all available broker names.
-- **`setActiveBroker(brokerName)`**: Set the active broker for the session. Returns `true` if successful.
-- **`getActiveBrokerName()`**: Get the name of the currently active broker.
-- **`getActiveBrokerUrl()`**: Get the URL of the currently active broker.
-- **`getDefaultBrokerName()`**: Get the name of the default broker (first one defined).
-- **`selectBrokerFromEnv()`**: Set active broker from `KADI_BROKER` environment variable.
+const math = await loadAbility('math-ability', 'stdio');
+const result = await math.add({ a: 5, b: 3 });
+console.log(result); // { result: 8 }
+```
 
-### Process Management Functions
+## Tool Registration (Three Ways)
 
-- **`runExecCommand(name, version, command)`**: Executes a command for initializing abilities.
-- **`runSpawnCommand(name, version, command)`**: Uses `spawn` to execute commands for subprocesses.
+### 1. With Zod Schemas (Recommended)
 
-## Usage Examples
+Zod provides type-safe schemas with 70% less code than JSON Schema:
 
 ```javascript
-import {
-  getProjectJSON,
-  runExecCommand,
-  IPCManager,
-  Broker
-} from '@kadi.build/core';
-
-async function setupProject() {
-  const projectConfig = getProjectJSON();
-  console.log(projectConfig);
-
-  await runExecCommand('example', '1.0', 'npm install');
-
-  // Broker management examples
-  console.log('Available brokers:', getBrokerNames());
-  console.log('All brokers:', KADI_BROKERS);
-
-  // Set active broker
-  setActiveBroker('remote');
-  console.log('Active broker:', getActiveBrokerUrl());
-
-  // IPC setup
-  const ipc = new IPCManager();
-  ipc.createInstance('python', 'pythonScript.py', 'pythonInstance');
-
-  // Traditional broker setup
-  Broker.addBroker('ws://example.com', 'exampleBroker');
-  let broker = Broker.getBroker('default');
-  console.log(`Connected to ${broker.url}`);
-  broker.send(
-    BrokerMessageBuilder.setup(
-      'TestAgent',
-      'A Broker Testing Agent',
-      null,
-      null
-    )
-  );
-}
+import { KadiClient, z } from '@kadi.build/core';
+
+const ability = new KadiClient({ name: 'example' });
+
+ability.registerTool({
+  name: 'processData',
+  description: 'Process user data',
+  version: '1.0.0',
+  input: z.object({
+    name: z.string().min(1).describe('User name'),
+    age: z.number().int().positive().optional().describe('User age'),
+    tags: z.array(z.string()).describe('User tags')
+  }),
+  output: z.object({
+    processed: z.boolean(),
+    data: z.record(z.unknown())
+  })
+}, async (params) => {
+  // Automatic validation of inputs and outputs
+  return {
+    processed: true,
+    data: params
+  };
+});
 ```
 
-### Broker Functions
+### 2. With createTool() Helper
 
-- **`Broker`**: Manages interactions with broker systems.
-- **`IBroker`**: Broker instance interface with methods to manage its lifecycle and communications.
-- **`BrokerMessageBuilder`**: Assists in constructing messages for broker communication.
+For advanced use cases with runtime validation:
 
-## Broker Functions
+```javascript
+import { createTool, z } from '@kadi.build/core';
+
+const { definition, handler } = createTool({
+  name: 'convert',
+  description: 'Convert units',
+  input: z.object({
+    value: z.number(),
+    from: z.enum(['km', 'miles']),
+    to: z.enum(['km', 'miles'])
+  }),
+  output: z.object({
+    result: z.number(),
+    unit: z.string()
+  }),
+  handler: async ({ value, from, to }) => {
+    // Your logic here
+    return { result: value * 1.6, unit: to };
+  }
+});
 
-The Broker system facilitates interaction between different agents and services within the Kadi environment.
+ability.registerTool(definition, handler);
+```
 
-### `Broker` Interface
+### 3. With JSON Schema (Backward Compatible)
 
-- **`addBroker(url, name = 'default')`**: Adds a new broker connection. If a broker with the same name exists, it will be reused.
-- **`disconnect(name = 'default')`**: Disconnects the broker specified by the name.
-- **`deleteBroker(name = 'default')`**: Deletes the broker specified by the name and closes its connection.
-- **`send(message, brokerName = 'default')`**: Sends a message through the broker specified by the name.
-- **`addEventListener(event, listener, brokerName = 'default')`**: Adds an event listener to the specified broker.
-- **`removeEventListener(event, listener, brokerName = 'default')`**: Removes an event listener from the specified broker.
-- **`removeAllListeners(brokerName = 'default')`**: Removes all event listeners from the specified broker.
-- **`getBroker(brokerName = 'default')`**: Retrieves a broker instance by name.
+Traditional approach still supported:
 
-### `IBroker` Instance
+```javascript
+ability.registerTool({
+  name: 'greet',
+  description: 'Greet someone',
+  inputSchema: {
+    type: 'object',
+    properties: {
+      name: { type: 'string', description: 'Name to greet' }
+    },
+    required: ['name']
+  },
+  outputSchema: {
+    type: 'object',
+    properties: {
+      greeting: { type: 'string' }
+    }
+  }
+}, async ({ name }) => {
+  return { greeting: `Hello, ${name}!` };
+});
+```
 
-This class extends `EventEmitter` and manages individual broker connections:
+## Migration from JSON Schema to Zod
 
-- **`constructor(url, name)`**: Initializes a new broker connection.
-- **`send(message)`**: Sends a message through the WebSocket connection.
-- **`getConnectedAgents()`**: Retrieves a list of currently connected agents.
+**Before (JSON Schema):**
+```javascript
+// 64 lines of boilerplate
+ability.registerTool({
+  name: 'deployToAkash',
+  description: 'Deploy to Akash Network',
+  inputSchema: {
+    type: 'object',
+    properties: {
+      profile: {
+        type: 'string',
+        description: 'Deployment profile name'
+      },
+      dryRun: {
+        type: 'boolean',
+        description: 'Preview without deploying',
+        default: false
+      },
+      verbose: {
+        type: 'boolean',
+        description: 'Enable verbose output',
+        default: false
+      }
+    },
+    required: ['profile']
+  },
+  outputSchema: {
+    type: 'object',
+    properties: {
+      success: { type: 'boolean' },
+      dseq: { type: 'string' },
+      services: {
+        type: 'array',
+        items: { type: 'string' }
+      }
+    }
+  }
+}, handler);
+```
 
-### `BrokerMessageBuilder`
+**After (Zod):**
+```javascript
+// 15 lines - clean and type-safe!
+ability.registerTool({
+  name: 'deployToAkash',
+  description: 'Deploy to Akash Network',
+  input: z.object({
+    profile: z.string().describe('Deployment profile name'),
+    dryRun: z.boolean().default(false).describe('Preview without deploying'),
+    verbose: z.boolean().default(false).describe('Enable verbose output')
+  }),
+  output: z.object({
+    success: z.boolean(),
+    dseq: z.string(),
+    services: z.array(z.string())
+  })
+}, handler);
+```
 
-Utility class for constructing broker messages:
+**Result: 77% less code!**
 
-- **`create_message(type, data)`**: Creates a JSON string with the specified type and data.
-- **`message(to, content)`**: Constructs a message directed to a specific peer.
-- **`setup(name, description, limit, uuid)`**: Creates a setup message for registering the broker.
-- **`suspend()`**: Constructs a suspend message.
-- **`finish()`**: Constructs a finish message.
-- **`list()`**: Constructs a list message to request a list of connected agents.
+## Transport Protocols
 
-### IPC Functions
+KADI supports three transport protocols for different use cases:
 
-- **`IPCMessageBuilder`**: Constructs messages for IPC interactions.
-- **`IPCManager`**: Manages setup and lifecycle of IPC connections.
-- **`IAbilityIPC`**: Represents an IPC ability instance with methods to manage its lifecycle and communications.
+### Native (In-Process)
+```javascript
+const ability = await loadAbility('my-ability', 'native');
+// Direct function calls, zero IPC overhead
+```
 
-### `IPCManager`
+### Stdio (Child Process)
+```javascript
+const ability = await loadAbility('my-ability', 'stdio');
+// JSON-RPC over stdin/stdout, language-agnostic
+```
 
-Manages the lifecycle and setup of interprocess communications.
+### Broker (Distributed)
+```javascript
+const ability = await loadAbility('my-ability', 'broker', {
+  brokerUrl: 'ws://localhost:8080',
+  networks: ['global']
+});
+// WebSocket-based distributed communication
+```
 
-- **`createInstance(language, commandString, name = 'default')`**: Creates and manages a new IPC instance for the specified language.
-- **`getInstance(name)`**: Retrieves an existing IPC instance by name.
-- **`shutdownInstance(name)`**: Shuts down an existing IPC instance by name.
+## Events
 
-### `IAbilityIPC` Instance
+Publish and subscribe to events across all protocols:
 
-Manages an individual interprocess communication instance:
+```javascript
+// Publishing events
+ability.registerTool({
+  name: 'process',
+  input: z.object({ data: z.string() }),
+  output: z.object({ result: z.string() })
+}, async (params) => {
+  // Publish progress events
+  await ability.publishEvent('process.started', { data: params.data });
+
+  const result = await doWork(params.data);
+
+  await ability.publishEvent('process.completed', { result });
+  return { result };
+});
 
-- **`start()`**: Starts the child process associated with this IPC instance.
-- **`launch()`**: Sends the launch command to the associated process.
-- **`sendMessage(message)`**: Sends a JSON-formatted message to the child process.
-- **`shutdown()`**: Sends a shutdown command to the child process and terminates it.
+// Subscribing to events
+ability.subscribeToEvent('process.*', (data) => {
+  console.log('Process event:', data);
+});
+```
 
-### `IPCMessageBuilder`
+## Broker Mode
 
-Utility class for constructing IPC messages:
+Connect to KADI broker for distributed communication:
 
-- **`createMessage(type, contents)`**: Creates a JSON string with the specified type and contents.
-- **`launch(commandString)`**: Constructs a launch message with the specified command string.
-- **`shutdown()`**: Constructs a shutdown message.
-- **`message(content)`**: Constructs a general message with the specified content.
+```javascript
+const client = new KadiClient({
+  name: 'my-agent',
+  role: 'agent',
+  brokers: {
+    dev: 'ws://localhost:8080',
+    prod: 'ws://prod.example.com:8080'
+  },
+  defaultBroker: 'dev',
+  networks: ['global']
+});
 
-## Usage Examples
+// Register tools
+client.registerTool({
+  name: 'greet',
+  input: z.object({ name: z.string() }),
+  output: z.object({ greeting: z.string() })
+}, async ({ name }) => {
+  return { greeting: `Hello, ${name}!` };
+});
+
+// Connect to broker
+await client.serve('broker');
+
+// Call remote tools
+const result = await client.callTool('translator', 'translate', {
+  text: 'Hello',
+  to: 'es'
+});
+```
 
-Here's how you might use the IPC and Broker functionalities:
+## API Reference
 
-### Interacting with IPC
+### KadiClient
+
+Main class for all KADI operations:
 
 ```javascript
-console.log('Starting IPC Ability Interface test...');
-
-// Create Python IPC instance
-const pythonIPC = IPCManager.createInstance(
-  'python',
-  'echo "Hello, World!"',
-  'pythonTest'
-);
-
-pythonIPC.on('error', (error) => {
-  console.error('Handled Python Error:', error);
-  // Consider appropriate error handling or recovery actions here
+const client = new KadiClient({
+  name: 'my-ability',
+  version: '1.0.0',
+  role: 'ability',  // 'agent' or 'ability'
+  brokers: { /* broker configs */ },
+  defaultBroker: 'dev',
+  networks: ['global']
 });
+```
 
-pythonIPC.on('critical-error', (error) => {
-  console.error('Critical Error from Python:', error);
-  // Handle critical errors, potentially restarting the process or alerting administrators
-});
+**Methods:**
+- `registerTool(definition, handler)` - Register a tool
+- `serve(mode)` - Start serving ('native', 'stdio', 'broker')
+- `callTool(agent, tool, params)` - Call remote tool
+- `publishEvent(name, data)` - Publish event
+- `subscribeToEvent(pattern, callback)` - Subscribe to events
+- `loadAbility(name, protocol, options)` - Load another ability
 
-pythonIPC.on('message', (msg) => {
-  console.log('Message from Python:', msg);
-});
+### loadAbility
 
-pythonIPC.on('broker', (msg) => {
-  console.log('Broker message from Python:', msg);
-});
+Load an ability by name and protocol:
 
-pythonIPC.on('shutdown', () => {
-  console.log('Python IPC has shut down.');
+```javascript
+const ability = await loadAbility('ability-name', 'protocol', {
+  brokerUrl: 'ws://localhost:8080',
+  networks: ['global']
 });
+```
+
+### createTool
+
+Helper for creating tools with validation:
+
+```javascript
+import { createTool, z } from '@kadi.build/core';
 
-Broker.addBroker('http://your.broker.com', 'default');
-let brk = Broker.getBroker('default');
-broker = brk;
-
-pythonIPC.attachBroker(brk);
-
-brk.on('open', function open() {
-  console.log(`Connected to ${brk.url}`);
-  const jsonObject = BrokerMessageBuilder.setup(
-    'TestAgent',
-    'A Broker Testing Agent',
-    null,
-    null
-  );
-  brk.send(jsonObject);
+const { definition, handler } = createTool({
+  name: 'toolName',
+  input: z.object({ /* ... */ }),
+  output: z.object({ /* ... */ }),
+  handler: async (params) => { /* ... */ }
 });
+```
 
-brk.on('message', function incoming(data) {
-  let msg = JSON.parse(data);
-  if (msg.type == 'setup') {
-    (async () => {
-      try {
-        await GetLLMProxy();
-      } catch (error) {
-        console.error('Error during GetLLMProxy:', error.message);
-      }
-    })();
+## Common Patterns
+
+### Error Handling
+
+```javascript
+ability.registerTool({
+  name: 'riskyOperation',
+  input: z.object({ data: z.string() }),
+  output: z.object({
+    success: z.boolean(),
+    result: z.string().optional(),
+    error: z.string().optional()
+  })
+}, async (params) => {
+  try {
+    const result = await performOperation(params.data);
+    return { success: true, result };
+  } catch (error) {
+    await ability.publishEvent('operation.error', {
+      error: error.message,
+      params
+    });
+    return { success: false, error: error.message };
   }
-  // console.log('Received:', formatJSON(data));
 });
+```
 
-brk.on('error', function error(error) {
-  console.error('WebSocket error:', error);
-});
+### Health Check
 
-brk.on('close', function close() {
-  console.log('Disconnected from the server');
-  process.exit(0); // Exit the process when the WebSocket connection is closed
+```javascript
+ability.registerTool({
+  name: 'health',
+  input: z.object({}),
+  output: z.object({
+    status: z.string(),
+    uptime: z.number(),
+    memory: z.record(z.number())
+  })
+}, async () => {
+  return {
+    status: 'healthy',
+    uptime: process.uptime(),
+    memory: process.memoryUsage()
+  };
 });
+```
 
-// Send launch command
+## Environment Variables
 
-//Utilize log files
-// pythonIPC.sendMessage(IPCMessageBuilder.launch(pythonIPC.commandString, process.cwd() + "/logs"));
+```bash
+# Protocol selection
+KADI_PROTOCOL=stdio
 
-//No log files created
-pythonIPC.sendMessage(IPCMessageBuilder.launch(pythonIPC.commandString));
+# Broker configuration
+KADI_BROKER_URL=ws://localhost:8080
+KADI_AGENT_SCOPE=project-123
 ```
 
-### Handling IPC
+## Debug Mode
 
-```javascript
-const ipcManager = new IPCManager();
-const ipcInstance = ipcManager.createInstance('node', 'app.js', 'nodeApp');
-ipcInstance.on('message', (message) => console.log(message));
+```bash
+DEBUG=kadi:* node index.js
 ```
 
+## Troubleshooting
+
+### Broker Connection Failures
+
+- Verify broker URL is correct
+- Check network connectivity
+- Ensure broker is running
+
+### Events Not Arriving
+
+- Subscribe BEFORE triggering events
+- Verify pattern matching
+- Check that publisher and subscriber are on same network
+
+### Type Errors with Zod
+
+- Ensure Zod schemas match handler types
+- Use `z.infer<typeof schema>` for TypeScript types
+- Check that all required fields are provided
+
+## License
+
+MIT
+
+## Related Projects
+
+- [@kadi.build/cli](https://gitlab.com/humin-game-lab/kadi/kadi) - Command-line interface
+- [@kadi.build/broker](https://gitlab.com/humin-game-lab/kadi/kadi-broker) - KADI broker
+
 ---
 
-## TODO:
-
-- Once @kadi.build/core and dependencies are done, @kadi.build/cli (and all commands), need to be rewritten, using the @kadi.build/core module. this will be v1.0 release
-
-* [ ] add getConnectedAgents to the Broker, it should call the IBroker method
-
-* [x] add key to IBroker object
-* [ ] IBroker on 'message' should listen for setup event, and update teh uuid and key for the IBroker object
-
-* [ ] Extract Broker and IPC to their own node module and host on npm
-* [x] Publish @kadi.build/core to npm
-* [x] Setup of python environment needs to happen once an IPC Inteface is created
-      This include setting up the virtual environment and installing the required packages
-* [x] Add method to IPCManger to set IPCInterfaces from agent.json. If IPCManger has empty list, it thorws an error that no IAbilityIPC's can be created. IPCManger passes the proper interface command to IAbilityIPC on creation from this list, IAbilityIPC does not need access directly to agent.json. Controlling app can update this list anytime a new IAbilityInterface is created, allowing the agent.json to be updated during runtime.
-* [x] Need to adapt ipc.js to read from the @kadi.build/core folder when trying to open IPC instances, instead of project folder. @kadi.build/core should be in the npm_modules folder (when deployed)
-* [x] The test code needs to fully test the IPCManger interface, and not just the IPCAbility
-* [x] If node fails, we need to make sure python IPC interface and child processes also close out.
-* [x] Add buffer to error handeling in IAbilityIPC, mimic same process used in std input
-* [x] Verify if Python IAbilityIPC needs to utilize buffering, and node passing '\n' at end of jso
-* [x] Setup passthrough for broker messages to/from Language IPC interface
-* [x] Add broker message builder/parser in python IAbilityIPC
-* [x] Update BrokerMessageBuilder to use same format at IPCMessageBuilder
-* [x] Update Broker so it extends EventEmitter, rather than have a property of event emitter (similar to IAbilityIPC)
-* [x] update python IAbility so that it creates a timestamped command_output.log file, new file for each run, in the project root directory... not the @kadi.build/core directory
-* [x] Comment out debug messages
-* [x] Add ability to allow for launched process to be piped to event handler (launch command should default to process, but allow user to set flag that will save all outputs to file in the event handler)
+Built with ❤️ by the KADI team