diodejs 0.3.0 → 0.4.1

package/README.md

@@ -38,16 +38,124 @@ DIODE_TICKET_BYTES_THRESHOLD=512000
 DIODE_TICKET_UPDATE_INTERVAL=30000
 ```
 
-These settings can also be configured programmatically:
+These settings can also be configured programmatically on the relay connections managed by `DiodeClientManager`:
 ```javascript
-connection.setReconnectOptions({
-  maxRetries: 10,
-  retryDelay: 2000,
-  maxRetryDelay: 20000,
-  autoReconnect: true,
-  ticketBytesThreshold: 512000,
-  ticketUpdateInterval: 30000
+const { DiodeClientManager } = require('diodejs');
+
+async function main() {
+  const client = new DiodeClientManager({ keyLocation: './db/keys.json' });
+  await client.connect();
+
+  for (const connection of client.getConnections()) {
+    connection.setReconnectOptions({
+      maxRetries: 10,
+      retryDelay: 2000,
+      maxRetryDelay: 20000,
+      autoReconnect: true
+    });
+
+    connection.setTicketBatchingOptions({
+      threshold: 512000,
+      interval: 30000
+    });
+  }
+}
+
+main();
+```
+
+### Multi-Relay Connections (Recommended)
+
+You can connect to multiple Diode relays and automatically route binds to the relay where the target device is connected.
+
+`DiodeClientManager` now ranks relays by observed latency. On startup it probes the required candidate set, persists relay scores to disk, and prefers the lowest-latency connected relay for control-plane RPC calls. It does not ping the full network at startup.
+
+When neither `host` nor `hosts` is specified, the manager starts from the default seed pool, any discovery-provider candidates, built-in `dio_network` candidates, and previously successful non-provider relays saved in `relay-scores.json`. Without live network discovery it probes all default seeds once. When live network discovery returns usable relays, startup resolves from a region-diverse seed bootstrap subset plus the bounded discovery sample, then continues measuring the remaining seeds in the background while keeping region diversity in the warm set. If you pass `host` or `hosts`, startup stays constrained to those configured relays unless you explicitly opt into using the discovery provider alongside them.
+
+```javascript
+const { DiodeClientManager, BindPort } = require('diodejs');
+
+async function main() {
+  // Connect to default relay pool (pre-net defaults)
+  const client = new DiodeClientManager({ keyLocation: './db/keys.json' });
+  await client.connect();
+
+  const bind = new BindPort(client, {
+    3003: { targetPort: 8080, deviceIdHex: '0x...', protocol: 'tcp' }
+  });
+  bind.bind();
+}
+```
+
+You can tune relay selection if needed:
+
+```javascript
+const client = new DiodeClientManager({
+  keyLocation: './db/keys.json',
+  relaySelection: {
+    startupConcurrency: 2,
+    minReadyConnections: 2,
+    probeTimeoutMs: 1200,
+    warmConnectionBudget: 3,
+    probeAllInitialCandidates: true,
+    continueProbingUntestedSeeds: true,
+    regionDiverseSeedOrdering: true,
+    discoveryProviderTimeoutMs: 1500,
+    backgroundProbeIntervalMs: 300000,
+    slowRelayThresholdMs: 250,
+    slowDeviceRetryTtlMs: 5000,
+    scoreCachePath: './db/relay-scores.json'
+  }
+});
+```
+
+Routing note: the initiating client can prefer a better local relay, but it cannot override the relay encoded in the remote device ticket. Best results come when both clients use the manager's relay ranking so each side reconnects toward a closer relay over time.
+
+Discovery note: startup discovery sources are now `configured/seed`, `discoveryProvider`, built-in live `dio_network` discovery, cached non-provider/non-network relays, and target-on-demand relay resolution from `getNode(serverId)`. Provider and network membership are not cached independently; only RTT/history is persisted in `relay-scores.json`.
+
+Example discovery provider:
+
+```javascript
+const fs = require('fs/promises');
+
+const client = new DiodeClientManager({
+  keyLocation: './db/keys.json',
+  relaySelection: {
+    discoveryProvider: async () => {
+      const data = JSON.parse(await fs.readFile('./relays.json', 'utf8'));
+      return data;
+    },
+    networkDiscovery: {
+      endpoint: 'wss://prenet.diode.io:8443/ws',
+      startupProbeCount: 2,
+      backgroundBatchSize: 12
+    }
+  }
+});
+```
+
+`discoveryProvider(context)` may return either relay strings such as `'relay.example.com:41046'` or objects like `{ host, port, priority, region, metadata }`. The callback receives a read-only `context` object with:
+
+- `defaultPort`
+- `keyLocation`
+- `explicitHost`
+- `explicitHosts`
+- `initialHosts`
+- `knownRelayScores`
+
+`knownRelayScores` contains the manager's current score snapshot for previously seen relays. Provider membership itself is not cached across runs; only relay score history is persisted.
+
+Built-in network discovery uses the Diode JSON-RPC websocket endpoint and requests `dio_network`. It is enabled by default only when neither `host` nor `hosts` is set. Only connected `server` nodes are considered, and private/unroutable addresses are filtered unless `includePrivateAddresses` is enabled. When discovery succeeds, startup keeps the seed safety baseline by probing one seed per region before adding the configured discovery sample, instead of blocking on every seed before ready.
+
+If you provide a host, only that relay is used initially (similar to `-diodeaddrs`):
+
+```javascript
+const client = new DiodeClientManager({
+  host: 'us2.prenet.diode.io',
+  port: 41046,
+  keyLocation: './db/keys.json'
 });
+await client.connect();
 ```
 
 ### Test RPC
@@ -55,25 +163,35 @@ connection.setReconnectOptions({
 Here's a quick example to get you started with RPC functions using `DiodeRPC` Class
 
 ```javascript
-const { DiodeConnection, DiodeRPC, makeReadable } = require('diodejs');
+const { DiodeClientManager, DiodeRPC, makeReadable } = require('diodejs');
 
 async function main() {
   const host = 'eu2.prenet.diode.io';
   const port = 41046;
   const keyLocation = './db/keys.json'; // Optional, defaults to './db/keys.json'
 
-  const connection = new DiodeConnection(host, port, keyLocation);
-  
+  const client = new DiodeClientManager({ host, port, keyLocation });
+  await client.connect();
+
+  const connection = client.getConnections()[0];
+  if (!connection) {
+    throw new Error('No relay connection available');
+  }
+
   // Configure reconnection (optional - overrides environment variables)
   connection.setReconnectOptions({
     maxRetries: Infinity, // Unlimited reconnection attempts
     retryDelay: 1000,     // Initial delay of 1 second
     maxRetryDelay: 30000, // Maximum delay of 30 seconds
-    autoReconnect: true,  // Automatically reconnect on disconnection
-    ticketBytesThreshold: 512000, // Bytes threshold for ticket updates
-    ticketUpdateInterval: 30000   // Time interval for ticket updates
+    autoReconnect: true   // Automatically reconnect on disconnection
   });
-  
+
+  // Configure ticket batching (optional - overrides environment variables)
+  connection.setTicketBatchingOptions({
+    threshold: 512000, // Bytes threshold for ticket updates
+    interval: 30000    // Time interval for ticket updates
+  });
+
   // Listen for reconnection events (optional)
   connection.on('reconnecting', (info) => {
     console.log(`Reconnecting... Attempt #${info.attempt} in ${info.delay}ms`);
@@ -84,8 +202,6 @@ async function main() {
   connection.on('reconnect_failed', () => {
     console.log('Failed to reconnect after maximum attempts');
   });
-  
-  await connection.connect();
 
   const rpc = new DiodeRPC(connection);
 
@@ -100,7 +216,7 @@ async function main() {
   } catch (error) {
     console.error('RPC Error:', error);
   } finally {
-    connection.close();
+    client.close();
   }
 }
 
@@ -113,15 +229,15 @@ Here's a quick example to get you started with port forwarding using the `BindPo
 
 #### Port Binding
 ```javascript
-const { DiodeConnection, BindPort } = require('diodejs');
+const { DiodeClientManager, BindPort } = require('diodejs');
 
 async function main() {
     const host = 'eu2.prenet.diode.io';
     const port = 41046;
     const keyLocation = './db/keys.json';
   
-    const connection = new DiodeConnection(host, port, keyLocation);
-    await connection.connect();
+    const client = new DiodeClientManager({ host, port, keyLocation });
+    await client.connect();
   
     // Multiple or single port binding with configuration object
     const portsConfig = {
@@ -133,11 +249,12 @@ async function main() {
       3003: { 
         targetPort: 443, 
         deviceIdHex: "0x5365baf29cb7ab58de588dfc448913cb609283e2",
-        protocol: "tcp" // Can be "tls", "tcp", or "udp"
+        protocol: "tcp", // Can be "tls", "tcp", or "udp"
+        transport: "native" // Optional - "api" (default) or "native" for portopen2 (tcp/udp only)
       }
     };
     
-    const portForward = new BindPort(connection, portsConfig);
+    const portForward = new BindPort(client, portsConfig);
     portForward.bind();
     
     // You can also dynamically add ports with protocol specification
@@ -150,18 +267,18 @@ main();
 
 #### Single Port Binding (Legacy)
 ```javascript
-const { DiodeConnection, BindPort } = require('diodejs');
+const { DiodeClientManager, BindPort } = require('diodejs');
 
 async function main() {
     const host = 'eu2.prenet.diode.io';
     const port = 41046;
     const keyLocation = './db/keys.json';
   
-    const connection = new DiodeConnection(host, port, keyLocation);
-    await connection.connect();
+    const client = new DiodeClientManager({ host, port, keyLocation });
+    await client.connect();
   
     // Legacy method - single port binding (defaults to TLS protocol)
-    const portForward = new BindPort(connection, 3002, 80, "5365baf29cb7ab58de588dfc448913cb609283e2");
+    const portForward = new BindPort(client, 3002, 80, "5365baf29cb7ab58de588dfc448913cb609283e2");
     portForward.bind();
 }
 
@@ -173,15 +290,15 @@ main();
 Here's a quick example to get you started with publishing ports using the `PublishPort` class:
 
 ```javascript
-const { DiodeConnection, PublishPort } = require('diodejs');
+const { DiodeClientManager, PublishPort } = require('diodejs');
 
 async function main() {
   const host = 'us2.prenet.diode.io';
   const port = 41046;
   const keyLocation = './db/keys.json';
 
-  const connection = new DiodeConnection(host, port, keyLocation);
-  await connection.connect();
+  const client = new DiodeClientManager({ host, port, keyLocation });
+  await client.connect();
 
   // Option 1: Simple array of ports (all public)
   const publishedPorts = [8080, 3000]; 
@@ -196,7 +313,7 @@ async function main() {
   };
   
   // certPath parameter is maintained for backward compatibility but not required
-  const publishPort = new PublishPort(connection, publishedPortsWithConfig);
+  const publishPort = new PublishPort(client, publishedPortsWithConfig);
 }
 
 main();
@@ -206,46 +323,69 @@ main();
 
 ### Classes and Methods
 
-#### `DiodeConnection`
-
-- **Constructor**: `new DiodeConnection(host, port, keyLocation)`
-  - `host` (string): The host address of the Diode server.
-  - `port` (number): The port number of the Diode server.
-  - `keyLocation` (string)(default: './db/keys.json'): The path to the key storage file. If the file doesn't exist, keys are generated automatically.
+#### `DiodeClientManager`
+
+- **Constructor**: `new DiodeClientManager(options)`
+  - `options.host` (string, optional): Single relay host (with or without port). If provided, only this relay is used initially.
+  - `options.port` (number, optional): Port to use when `options.host` has no port. Defaults to `41046`.
+  - `options.hosts` (string[] or comma-separated string, optional): Explicit relay list.
+  - `options.keyLocation` (string, optional): Key storage path (default: `./db/keys.json`).
+  - `options.deviceCacheTtlMs` (number, optional): Cache TTL for device relay resolution (default: `30000`).
+  - `options.relaySelection` (object, optional): Relay ranking and probing options.
+    - `enabled` (boolean, optional): Enables smart relay ranking. Defaults to `true`.
+    - `startupConcurrency` (number, optional): Parallel startup probe limit. Defaults to `2`.
+    - `minReadyConnections` (number, optional): Minimum target connection count for the startup probe set. Defaults to `2`.
+    - `probeTimeoutMs` (number, optional): Ping timeout for relay probes. Defaults to `1200`.
+    - `warmConnectionBudget` (number, optional): Maximum number of idle control relays to keep after startup coverage completes. Defaults to `3`.
+    - `probeAllInitialCandidates` (boolean, optional): Probes all initial configured/default candidates once before final startup ranking is trusted. Defaults to `true`.
+    - `continueProbingUntestedSeeds` (boolean, optional): Continues probing untested candidates after startup coverage completes. Defaults to `true`.
+    - `regionDiverseSeedOrdering` (boolean, optional): Interleaves seed regions during first-pass probing and warm retention. Defaults to `true`.
+    - `discoveryProvider` (function, optional): Async or sync callback that returns extra relay candidates as strings or `{ host, port, priority, region, metadata }` objects. The callback receives `{ defaultPort, keyLocation, explicitHost, explicitHosts, initialHosts, knownRelayScores }`.
+    - `discoveryProviderTimeoutMs` (number, optional): Timeout for `discoveryProvider` results. Defaults to `1500`.
+    - `useProviderWithExplicitHost` (boolean, optional): Allows provider candidates when `options.host` is set. Defaults to `false`.
+    - `useProviderWithExplicitHosts` (boolean, optional): Allows provider candidates when `options.hosts` is set. Defaults to `false`.
+    - `networkDiscovery` (object, optional): Built-in live directory discovery via the JSON-RPC websocket endpoint.
+      - `enabled` (boolean, optional): Enables live network discovery when no explicit `host` or `hosts` is configured. Defaults to `true` in default mode.
+      - `endpoint` (string, optional): Directory endpoint. Defaults to `wss://prenet.diode.io:8443/ws`.
+      - `method` (string, optional): JSON-RPC method name. Defaults to `dio_network`.
+      - `timeoutMs` (number, optional): Timeout for the discovery websocket request. Defaults to `1500`.
+      - `startupProbeCount` (number, optional): Number of discovered network relays added to the startup coverage probe set. Defaults to `2`.
+      - `backgroundBatchSize` (number, optional): Number of additional discovered relays measured in the background queue per run. Defaults to `12`.
+      - `includePrivateAddresses` (boolean, optional): Allows private or unroutable discovery results to be used. Defaults to `false`.
+    - `backgroundProbeIntervalMs` (number, optional): Minimum interval before background refresh probes are retried for a relay. Defaults to `300000`.
+    - `slowRelayThresholdMs` (number, optional): RTT threshold used to shorten target relay cache entries. Defaults to `250`.
+    - `slowDeviceRetryTtlMs` (number, optional): TTL used for slow target relays. Defaults to `5000`.
+    - `deviceRelayReconciliation` (object, optional): Bounded fallback that re-resolves a device through alternate connected control relays when the initially resolved target relay is much slower than the current control relay baseline.
+      - `enabled` (boolean, optional): Enables reconciliation. Defaults to `true`.
+      - `maxControlRelays` (number, optional): Maximum number of alternate connected control relays queried per reconciliation attempt. Defaults to `2`.
+      - `timeoutMs` (number, optional): Per-control-relay timeout for reconciliation lookups. Defaults to `probeTimeoutMs`.
+      - `minLatencyDeltaMs` (number, optional): Minimum RTT gap between the target relay and the control relay baseline before reconciliation triggers. Defaults to `150`.
+      - `slowdownFactor` (number, optional): Minimum multiple by which the target relay RTT must exceed the control relay baseline before reconciliation triggers. Defaults to `4`.
+    - `scoreCachePath` (string|null, optional): Relay score cache file. Defaults to `./db/relay-scores.json` next to `keyLocation`. Set to `null` to disable persistence.
 
 - **Methods**:
-  - `connect()`: Connects to the Diode server. Returns a promise.
-  - `sendCommand(commandArray)`: Sends a command to the Diode server. Returns a promise.
-  - `sendCommandWithSessionId(commandArray, sessionId)`: Sends a command with a session ID. Returns a promise.
-  - `getEthereumAddress()`: Returns the Ethereum address derived from the device keys.
-  - `getServerEthereumAddress()`: Returns the Ethereum address of the server.
-  - `createTicketCommand()`: Creates a ticket command for authentication. Returns a promise.
-  - `close()`: Closes the connection to the Diode server.
-  - `getDeviceCertificate()`: Returns the generated certificate PEM.
-  - `setReconnectOptions(options)`: Configures reconnection behavior with the following options:
-    - `maxRetries` (number): Maximum reconnection attempts (default: Infinity)
-    - `retryDelay` (number): Initial delay between retries in ms (default: 1000)
-    - `maxRetryDelay` (number): Maximum delay between retries in ms (default: 30000)
-    - `autoReconnect` (boolean): Whether to automatically reconnect on disconnection (default: true)
-    - `ticketBytesThreshold` (number): Bytes threshold for ticket updates (default: 512000)
-    - `ticketUpdateInterval` (number): Time interval for ticket updates in ms (default: 30000)
-
-- **Events**:
-  - `reconnecting`: Emitted when a reconnection attempt is about to start, with `attempt` and `delay` information
-  - `reconnected`: Emitted when reconnection is successful
-  - `reconnect_failed`: Emitted when all reconnection attempts have failed
+  - `connect()`: Probes the required initial relays, merges provider and optional `dio_network` candidates, ranks the successful relays by latency, trims the warm relay set, and returns a promise. In live network discovery mode this starts from a region-diverse seed bootstrap subset plus the bounded discovery sample, then continues probing the remaining seeds in the background.
+  - `getNearestConnection()`: Returns the preferred connected relay. With relay selection enabled, this is the lowest-latency scored relay.
+  - `getConnectionForDevice(deviceId)`: Resolves and returns a relay connection for the device. Returns a promise.
+  - `getConnections()`: Returns a list of active connections.
+  - `close()`: Closes all managed connections.
+
+Connections returned by `getConnections()` or `getConnectionForDevice()` emit `reconnecting`, `reconnected`, and `reconnect_failed` events, and support `setReconnectOptions(...)` and `setTicketBatchingOptions({ threshold, interval })`.
 
 #### `DiodeRPC`
 
 - **Constructor**: `new DiodeRPC(connection)`
-  - `connection` (DiodeConnection): An instance of `DiodeConnection`.
+  - `connection` (object): A relay connection from `DiodeClientManager.getConnections()` or `DiodeClientManager.getConnectionForDevice()`.
 
 - **Methods**:
   - `getBlockPeak()`: Retrieves the current block peak. Returns a promise.
   - `getBlockHeader(index)`: Retrieves the block header for a given index. Returns a promise.
   - `getBlock(index)`: Retrieves the block for a given index. Returns a promise.
+  - `getObject(deviceId)`: Retrieves a device ticket object. Returns a promise.
+  - `getNode(nodeId)`: Retrieves relay node information. Returns a promise.
   - `ping()`: Sends a ping command. Returns a promise.
   - `portOpen(deviceId, port, flags)`: Opens a port on the device. Returns a promise.
+  - `portOpen2(deviceId, port, flags)`: Opens a native relay port on the device (TCP/UDP). Returns the server relay port.
   - `portSend(ref, data)`: Sends data to the device. Returns a promise.
   - `portClose(ref)`: Closes a port on the device. Returns a promise.
   - `sendError(sessionId, ref, error)`: Sends an error response. Returns a promise.
@@ -259,23 +399,25 @@ main();
   
   Legacy Constructor:
   - `new BindPort(connection, localPort, targetPort, deviceIdHex)`
-    - `connection` (DiodeConnection): An instance of `DiodeConnection`.
+    - `connection` (DiodeClientManager): An instance of `DiodeClientManager`.
     - `localPort` (number): The local port to bind.
     - `targetPort` (number): The target port on the device.
     - `deviceIdHex` (string): The device ID in hexadecimal format (with or without '0x' prefix).
   
   New Constructor:
   - `new BindPort(connection, portsConfig)`
-    - `connection` (DiodeConnection): An instance of `DiodeConnection`.
+    - `connection` (DiodeClientManager): An instance of `DiodeClientManager`.
     - `portsConfig` (object): A configuration object where keys are local ports and values are objects with:
       - `targetPort` (number): The target port on the device.
       - `deviceIdHex` (string): The device ID in hexadecimal format (with or without '0x' prefix).
       - `protocol` (string, optional): The protocol to use ("tls", "tcp", or "udp"). Defaults to "tls".
+      - `transport` (string, optional): The relay transport to use ("api" or "native"). Defaults to "api". Native uses `portopen2` and supports TCP/UDP only.
 
 - **Methods**:
   - `bind()`: Binds all configured local ports to their target ports on the devices.
-  - `addPort(localPort, targetPort, deviceIdHex, protocol)`: Adds a new port binding configuration.
+  - `addPort(localPort, targetPort, deviceIdHex, protocol, transport)`: Adds a new port binding configuration.
     - `protocol` (string, optional): The protocol to use. Can be "tls", "tcp", or "udp". Defaults to "tls".
+    - `transport` (string, optional): The relay transport to use ("api" or "native"). Defaults to "api".
   - `removePort(localPort)`: Removes a port binding configuration.
   - `bindSinglePort(localPort)`: Binds a single local port to its target.
   - `closeAllServers()`: Closes all active server instances.
@@ -283,7 +425,7 @@ main();
 #### `PublishPort`
 
 - **Constructor**: `new PublishPort(connection, publishedPorts, _certPath)`
-  - `connection` (DiodeConnection): An instance of `DiodeConnection`.
+  - `connection` (DiodeClientManager): An instance of `DiodeClientManager`.
   - `publishedPorts` (array|object): Either:
     - An array of ports to publish (all public mode)
     - An object mapping ports to their configuration: `{ port: { mode: 'public'|'private', whitelist: ['0x123...'] } }`