holosphere 1.1.6 → 1.1.8

package/FEDERATION.md

@@ -1,143 +1,111 @@
 # HoloSphere Federation
 
-This document explains how to use the federation functionality in HoloSphere, which allows different spaces to share data with each other.
+HoloSphere's federation system allows different holons (spaces) to share and access data from each other. Federation creates a relationship between spaces that enables data propagation and cross-space access.
 
-## What is Federation?
+## Key Concepts
 
-Federation in HoloSphere allows different spaces (data stores) to connect and share data with each other. This enables:
+- **Federation Relationship**: A connection between two spaces that allows data to flow between them.
+- **Soul References**: Lightweight references that point to data in its original location (single source of truth).
+- **Notification Flow**: Data notifications flow from source spaces to target spaces that are in the notify list.
+- **Source-Target Relationship**: Each federation sets up a source space (federation list) and a target space (notify list).
 
-- Creating networks of connected spaces
-- Propagating data across these networks
-- Subscribing to changes in federated spaces
+## Federation Data Flow
 
-Federation can be either public (no authentication required) or private (password-protected).
+The HoloSphere federation system works with a clear source-target relationship:
 
-## Core Federation Functions
+1. **Federation List**: When space A federates with space B, space A adds B to its federation list.
+2. **Notify List**: Space B adds space A to its notify list.
+3. **Data Flow**: When space A changes, space B gets notified (but not vice versa unless bidirectional).
 
-### Creating a Federation
+## Creating Federation
 
-To create a federation between two spaces:
+Create federation relationships between spaces:
 
 ```javascript
-// Public federation (no passwords)
+// Create federation between space1 and space2
 await holoSphere.federate('space1', 'space2');
 
-// Private federation (with passwords)
-await holoSphere.federate('space1', 'space2', 'password1', 'password2');
+// This sets up:
+// - space1.federation includes space2
+// - space2.notify includes space1
 ```
 
-### Setting Up Bidirectional Notification (Important!)
+The bidirectional parameter is largely unused in the current implementation since the federation system naturally sets up the correct notification flow. The default relationship allows space2 to be notified of changes in space1.
 
-After creating a federation, you need to set up the notification settings for proper bidirectional data propagation:
+## Storing and Propagating Data
+
+Data must be explicitly propagated to federated spaces:
 
 ```javascript
-// 1. Get federation settings for first space
-const fedSettings1 = await holoSphere.getGlobal('federation', 'space1');
-if (fedSettings1) {
-  // 2. Configure notify settings
-  fedSettings1.notify = fedSettings1.notify || [];
-  if (!fedSettings1.notify.includes('space2')) {
-    fedSettings1.notify.push('space2');
-  }
-  // 3. Save updated settings
-  await holoSphere.putGlobal('federation', fedSettings1);
-}
+const data = {
+  id: 'item1',
+  title: 'Federation Example',
+  value: 42
+};
 
-// Repeat for the second space
-const fedSettings2 = await holoSphere.getGlobal('federation', 'space2');
-if (fedSettings2) {
-  fedSettings2.notify = fedSettings2.notify || [];
-  if (!fedSettings2.notify.includes('space1')) {
-    fedSettings2.notify.push('space1');
-  }
-  await holoSphere.putGlobal('federation', fedSettings2);
-}
-```
+// Store data in space1
+await holoSphere.put('space1', 'items', data);
 
-### Getting Federation Information
+// Propagate to federated spaces
+await holoSphere.propagate('space1', 'items', data);
+```
 
-To retrieve information about a space's federation:
+You can also enable automatic propagation in the `put()` method:
 
 ```javascript
-// Public space
-const fedInfo = await holoSphere.getFederation('space1');
-
-// Private space
-const fedInfo = await holoSphere.getFederation('space1', 'password1');
+// Store data and automatically propagate
+await holoSphere.put('space1', 'items', data, null, {
+  autoPropagate: true
+});
 ```
 
-The returned object contains:
-- `id`: The space ID
-- `name`: The space name
-- `federation`: Array of space IDs that this space is federated with
-- `notify`: Array of space IDs that this space will propagate data to
+## Accessing Federated Data
 
-### Propagating Data to Federated Spaces
+### Direct Retrieval
 
-To propagate data to all federated spaces:
+You can access data directly from any space:
 
 ```javascript
-const data = { id: 'item1', value: 42 };
-
-// 1. Store data locally first
-await holoSphere.put('space1', 'items', data);
-
-// 2. Explicitly propagate to federated spaces
-await holoSphere.propagateToFederation('space1', 'items', data);
-
-// 3. (Optional) Add a short delay to ensure propagation completes
-await new Promise(resolve => setTimeout(resolve, 1000));
+// Retrieve data from space2 (will resolve reference if it's a reference)
+const data = await holoSphere.get('space2', 'items', 'item1', null, {
+  resolveReferences: true  // Default is true
+});
 ```
 
-### Accessing Data Across Federated Spaces
+### Aggregate Federated Data
 
-There are multiple ways to access data across federated spaces:
-
-#### Method 1: Using getFederated (Recommended)
-
-This retrieves data from both the local space and all federated spaces:
+Use `getFederated()` to get data from multiple federated spaces:
 
 ```javascript
-// Get all items from local and federated spaces
-const allItems = await holoSphere.getFederated('space2', 'items');
-
-// Find a specific item by ID
-const item1 = allItems.find(item => item.id === 'item1');
+// Get combined data from the local space and all its federated spaces
+const federatedData = await holoSphere.getFederated('space2', 'items', {
+  resolveReferences: true,  // Default: true
+  idField: 'id'             // Field to use as the unique identifier
+});
 ```
 
-#### Method 2: Direct Access
+## Soul References
 
-After propagation, data may also be accessible directly:
+HoloSphere uses a simplified reference system based on soul paths:
 
-```javascript
-// Attempt to get item1 from space2 directly
-const item1FromSpace2 = await holoSphere.get('space2', 'items', 'item1');
-```
-
-### Subscribing to Federation Changes
+1. A reference contains only an `id` and a `soul` property
+2. The soul path is in the format: `appname/holon/lens/key`
+3. When resolving a reference, HoloSphere follows the soul path to retrieve the original data
 
-To subscribe to changes in a federation:
+By default, federation propagation uses references instead of duplicating data. This can be controlled:
 
 ```javascript
-// Subscribe
-const subscription = await holoSphere.subscribeFederation('space1', null, (data, originSpace, lens) => {
-  console.log(`Received data from ${originSpace}/${lens}:`, data);
+// Propagate with full data copy instead of references
+await holoSphere.propagate('space1', 'items', data, {
+  useReferences: false
 });
-
-// Later, unsubscribe
-subscription.unsubscribe();
 ```
 
-### Removing a Federation
-
-To remove a federation between two spaces:
+## Removing Federation
 
 ```javascript
-// Public federation
+// Remove federation relationship
 await holoSphere.unfederate('space1', 'space2');
-
-// Private federation
-await holoSphere.unfederate('space1', 'space2', 'password1', 'password2');
 ```
 
 ## Complete Example
@@ -154,61 +122,55 @@ async function federationExample() {
     const space1 = 'public-space1';
     const space2 = 'public-space2';
     
-    // Step 1: Create federation
+    // Step 1: Create federation relationship
     await holoSphere.federate(space1, space2);
     
-    // Step 2: Set up bidirectional notify settings (critical!)
-    const fedSettings1 = await holoSphere.getGlobal('federation', space1);
-    if (fedSettings1) {
-      fedSettings1.notify = fedSettings1.notify || [];
-      if (!fedSettings1.notify.includes(space2)) {
-        fedSettings1.notify.push(space2);
-        await holoSphere.putGlobal('federation', fedSettings1);
-      }
-    }
+    // Step 2: Verify federation is set up properly
+    const fedInfo1 = await holoSphere.getFederation(space1);
+    const fedInfo2 = await holoSphere.getFederation(space2);
     
-    const fedSettings2 = await holoSphere.getGlobal('federation', space2);
-    if (fedSettings2) {
-      fedSettings2.notify = fedSettings2.notify || [];
-      if (!fedSettings2.notify.includes(space1)) {
-        fedSettings2.notify.push(space1);
-        await holoSphere.putGlobal('federation', fedSettings2);
-      }
-    }
+    console.log(`Federation info for ${space1}:`, fedInfo1);
+    // Should include: federation: ['space2']
     
-    // Step 3: Verify federation is set up properly
-    const updatedFedInfo = await holoSphere.getFederation(space1);
-    console.log(`Federation info for ${space1}:`, updatedFedInfo);
+    console.log(`Federation info for ${space2}:`, fedInfo2);
+    // Should include: notify: ['space1']
     
-    // Step 4: Store data in space1
+    // Step 3: Store data in space1
     const item = { 
       id: 'item1', 
       title: 'Federation Test', 
       value: 42 
     };
+    
     await holoSphere.put(space1, 'items', item);
     
-    // Step 5: Propagate to federation
-    await holoSphere.propagateToFederation(space1, 'items', item);
+    // Step 4: Propagate data to federated spaces
+    await holoSphere.propagate(space1, 'items', item);
     
-    // Step 6: Allow time for propagation
+    // Allow time for propagation
     await new Promise(resolve => setTimeout(resolve, 1000));
     
-    // Step 7: Access data from both spaces
-    const itemFromSpace1 = await holoSphere.get(space1, 'items', 'item1');
-    console.log('Item from space1:', itemFromSpace1);
+    // Step 5: Access data from space2 (resolves reference)
+    const itemFromSpace2 = await holoSphere.get(space2, 'items', 'item1');
+    console.log('Item from space2:', itemFromSpace2);
+    
+    // Step 6: Update item in space1
+    const updatedItem = {
+      ...item,
+      value: 100,
+      updated: true
+    };
+    
+    await holoSphere.put(space1, 'items', updatedItem);
     
-    // Step 8: Access federated data
-    // Method 1: Using getFederated
-    const federatedData = await holoSphere.getFederated(space2, 'items');
-    const itemFromFederation = federatedData.find(item => item.id === 'item1');
-    console.log('Item from federation:', itemFromFederation);
+    // Since we're using soul references, the update is immediately visible
+    // through the reference without needing to propagate again
     
-    // Method 2: Direct access (if propagation worked correctly)
-    const directAccess = await holoSphere.get(space2, 'items', 'item1');
-    console.log('Direct access from space2:', directAccess);
+    // Verify update is visible through the reference
+    const updatedItemFromSpace2 = await holoSphere.get(space2, 'items', 'item1');
+    console.log('Updated item from space2:', updatedItemFromSpace2);
     
-    // Step 9: Clean up
+    // Step 7: Clean up
     await holoSphere.unfederate(space1, space2);
   } finally {
     // Always close the HoloSphere instance
@@ -219,47 +181,33 @@ async function federationExample() {
 federationExample().catch(console.error);
 ```
 
-## Running the Tests
-
-HoloSphere includes test scripts to verify federation functionality:
-
-### Public Federation Tests
-
-Run the public federation tests with:
-
-```bash
-node test-federation.js
-```
-
-This tests:
-- Creating public federations
-- Storing and retrieving data
-- Propagating data to federated spaces
-- Subscribing to federation changes
-- Removing federations
-
 ## Troubleshooting
 
 ### Common Issues
 
-1. **Missing Notify Settings**: The most common issue is not properly configuring notify settings. Always ensure both spaces have each other in their notify arrays.
+1. **Federation Relationship**: Make sure to check both the federation list and notify list to understand data flow.
 
-2. **No Explicit Propagation**: Data doesn't automatically propagate between spaces. You must explicitly call `propagateToFederation()` after storing data.
+2. **Data Propagation**: If data isn't appearing in federated spaces, check:
+   - The federation relationship was created correctly
+   - The data was propagated explicitly or `autoPropagate` was set to `true`
+   - The notify list includes the target space
 
-3. **Authentication Errors**: When working with private federations, ensure passwords are correct and consistent.
+3. **Reference Resolution**: If you're getting reference objects instead of the actual data:
+   - Make sure `resolveReferences` is set to `true` (it's the default)
+   - Check that the original data still exists at the referenced location
 
 4. **Timing Issues**: Data propagation is asynchronous. Add small delays (500-1000ms) between operations to allow propagation to complete.
 
-5. **Missing Federation Metadata**: After propagation, federated items should have a `federation` property containing the origin space and timestamp.
-
 ### Best Practices
 
-1. **Verify Federation Setup**: After creating a federation, always check the federation info to ensure it includes both the federation relationship and notify settings.
-
-2. **Error Handling**: Wrap federation operations in try/catch blocks and handle errors gracefully.
+1. **Verify Federation Structure**: After creating a federation, check both spaces to ensure:
+   - Source space has the target in its federation list
+   - Target space has the source in its notify list
 
-3. **Bidirectional Setup**: For proper data sharing, both spaces need notify settings pointing to each other.
+2. **Explicit Propagation**: Unless you're using `autoPropagate`, always call `propagate()` explicitly after storing data that should be shared.
 
-4. **Propagation Timing**: Allow sufficient time for propagation operations to complete before attempting to access data.
+3. **Choose the Right Propagation Method**:
+   - Use `useReferences: true` (default) to keep a single source of truth
+   - Use `useReferences: false` only when you need independent copies
 
-5. **Cleanup**: Always close the HoloSphere instance when done to prevent resource leaks. 
+4. **Cleanup**: Always close the HoloSphere instance when done to prevent resource leaks. 

package/README.md

@@ -329,3 +329,143 @@ Data in HoloSphere is organized by:
 
 GPL-3.0-or-later
 
+# HoloSphere Federation
+
+HoloSphere provides a federation system that allows spaces to share data and messages across different holons. This document outlines the federation functionality and how to use it.
+
+## Core Federation Features
+
+### Space Federation
+- Create relationships between spaces with clear source-target connections
+- Use soul references to maintain a single source of truth
+- Control data propagation between federated spaces
+- Manage notification settings for each space
+
+### Message Federation
+- Track messages across federated spaces
+- Maintain message relationships between original and federated copies
+- Update messages consistently across all federated spaces
+
+## API Reference
+
+### Space Federation
+
+#### `federate(spaceId1, spaceId2, password1, password2, bidirectional)`
+Creates a federation relationship between two spaces.
+
+```javascript
+await holosphere.federate('space1', 'space2', 'pass1', 'pass2');
+```
+
+This sets up:
+- space1.federation includes space2
+- space2.notify includes space1
+
+Parameters:
+- `spaceId1`: First space ID (source space)
+- `spaceId2`: Second space ID (target space)
+- `password1`: Optional password for first space
+- `password2`: Optional password for second space
+- `bidirectional`: Whether to set up bidirectional notifications (default: true, but generally not needed)
+
+### Data Propagation
+
+#### `propagate(holon, lens, data, options)`
+Propagates data to federated spaces.
+
+```javascript
+await holosphere.propagate('space1', 'items', data, {
+  useReferences: true,         // Default: uses soul references 
+  targetSpaces: ['space2']     // Optional: specific targets
+});
+```
+
+Parameters:
+- `holon`: The holon identifier
+- `lens`: The lens identifier
+- `data`: The data to propagate
+- `options`: Propagation options
+
+Alternatively, you can use auto-propagation:
+
+```javascript
+await holosphere.put('space1', 'items', data, null, {
+  autoPropagate: true  // Enable auto-propagation
+});
+```
+
+### Message Federation
+
+#### `federateMessage(originalChatId, messageId, federatedChatId, federatedMessageId, type)`
+Tracks a federated message across different chats.
+
+```javascript
+await holosphere.federateMessage('chat1', 'msg1', 'chat2', 'msg2', 'quest');
+```
+
+#### `getFederatedMessages(originalChatId, messageId)`
+Gets all federated messages for a given original message.
+
+```javascript
+const messages = await holosphere.getFederatedMessages('chat1', 'msg1');
+```
+
+#### `updateFederatedMessages(originalChatId, messageId, updateCallback)`
+Updates a message across all federated chats.
+
+```javascript
+await holosphere.updateFederatedMessages('chat1', 'msg1', async (chatId, messageId) => {
+    // Update message in this chat
+});
+```
+
+## Usage Example
+
+```javascript
+// Create federation between spaces
+await holosphere.federate('space1', 'space2');
+
+// Store data in space1
+const data = { id: 'item1', value: 42 };
+await holosphere.put('space1', 'items', data);
+
+// Propagate to federated spaces
+await holosphere.propagate('space1', 'items', data);
+
+// Retrieve data from space2 (will resolve the reference)
+const item = await holosphere.get('space2', 'items', 'item1');
+
+// Track a federated message
+await holosphere.federateMessage('chat1', 'msg1', 'chat2', 'msg2', 'quest');
+
+// Update message across all federated chats
+await holosphere.updateFederatedMessages('chat1', 'msg1', async (chatId, messageId) => {
+    await updateMessageInChat(chatId, messageId);
+});
+```
+
+## Soul References
+
+When using the default `useReferences: true` with propagation:
+
+1. Only a lightweight reference is stored in the federated space
+2. The reference contains the original item's ID and soul path
+3. When accessed, the reference is automatically resolved to the original data
+4. Changes to the original data are immediately visible through references
+
+This maintains a single source of truth while keeping storage efficient.
+
+## Federation Structure
+
+The federation system uses two key arrays to manage relationships:
+
+```javascript
+{
+    id: string,
+    name: string,
+    federation: string[],  // Source spaces this holon federates with
+    notify: string[],      // Target spaces to notify of changes
+    timestamp: number
+}
+```
+

package/examples/federation.js

@@ -0,0 +1,154 @@
+/**
+ * Organization Federation Example for HoloSphere
+ * 
+ * This example demonstrates a real-world use case where a tech team creates tasks
+ * that are federated to the parent organization for visibility.
+ */
+
+import HoloSphere from './holosphere.js';
+
+async function organizationFederationExample() {
+  const holoSphere = new HoloSphere('organization-example');
+  
+  try {
+    console.log('Starting Organization Federation Example...');
+    
+    // Define our spaces/holons
+    const orgHolon = 'acme-organization';
+    const techTeamHolon = 'acme-tech-team';
+    
+    // Step 1: Create federation relationship between tech team and organization
+    console.log('\nStep 1: Creating federation relationship...');
+    
+    await holoSphere.federate(techTeamHolon, orgHolon);
+    console.log('Federation created between tech team and organization');
+    
+    // Step 2: Set up bidirectional notification settings (critical!)
+    console.log('\nStep 2: Setting up bidirectional notification...');
+    
+    // First set up tech team to notify organization
+    const techTeamFedSettings = await holoSphere.getGlobal('federation', techTeamHolon);
+    if (techTeamFedSettings) {
+      techTeamFedSettings.notify = techTeamFedSettings.notify || [];
+      if (!techTeamFedSettings.notify.includes(orgHolon)) {
+        techTeamFedSettings.notify.push(orgHolon);
+        await holoSphere.putGlobal('federation', techTeamFedSettings);
+        console.log('Tech team set to notify organization');
+      }
+    }
+    
+    // Then set up organization to notify tech team (if needed)
+    const orgFedSettings = await holoSphere.getGlobal('federation', orgHolon);
+    if (orgFedSettings) {
+      orgFedSettings.notify = orgFedSettings.notify || [];
+      if (!orgFedSettings.notify.includes(techTeamHolon)) {
+        orgFedSettings.notify.push(techTeamHolon);
+        await holoSphere.putGlobal('federation', orgFedSettings);
+        console.log('Organization set to notify tech team');
+      }
+    }
+    
+    // Step 3: Verify federation is set up properly
+    console.log('\nStep 3: Verifying federation setup...');
+    
+    const techTeamFedInfo = await holoSphere.getFederation(techTeamHolon);
+    console.log('Tech team federation info:', techTeamFedInfo);
+    
+    // Step 4: Create a task in the tech team holon
+    console.log('\nStep 4: Creating a task in the tech team holon...');
+    
+    const task = {
+      id: 'task-123',
+      title: 'Implement new authentication system',
+      description: 'Replace the current auth system with OAuth2',
+      assignee: 'dev@example.com',
+      status: 'in_progress',
+      priority: 'high',
+      dueDate: '2023-12-31',
+      createdAt: new Date().toISOString(),
+      tags: ['security', 'infrastructure']
+    };
+    
+    // Store the task in the tech team holon
+    await holoSphere.put(techTeamHolon, 'tasks', task);
+    console.log('Task created in tech team holon:', task.id);
+    
+    // Step 5: Propagate the task to the organization holon
+    console.log('\nStep 5: Propagating task to organization holon...');
+    
+    await holoSphere.propagate(techTeamHolon, 'tasks', task);
+    console.log('Task propagated to organization holon');
+    
+    // Step 6: Allow time for propagation
+    console.log('\nStep 6: Waiting for propagation to complete...');
+    await new Promise(resolve => setTimeout(resolve, 1000));
+    
+    // Step 7: Verify task in both holons
+    console.log('\nStep 7: Verifying task is in both holons...');
+    
+    // Check tech team holon directly
+    const techTeamTask = await holoSphere.get(techTeamHolon, 'tasks', 'task-123');
+    console.log('Task in tech team holon:', techTeamTask ? 'Found' : 'Not found');
+    if (techTeamTask) console.log('Tech team task status:', techTeamTask.status);
+    
+    // Check organization holon directly
+    const orgTask = await holoSphere.get(orgHolon, 'tasks', 'task-123');
+    console.log('Task in organization holon:', orgTask ? 'Found' : 'Not found');
+    if (orgTask) {
+      console.log('Organization task status:', orgTask.status);
+      console.log('Federation metadata:', orgTask.federation);
+    }
+    
+    // Step 8: Use getFederated to view all tasks across holons
+    console.log('\nStep 8: Using getFederated to view all tasks...');
+    
+    const allOrgTasks = await holoSphere.getFederated(orgHolon, 'tasks');
+    console.log(`Organization holon has access to ${allOrgTasks.length} tasks`);
+    
+    const allTechTasks = await holoSphere.getFederated(techTeamHolon, 'tasks');
+    console.log(`Tech team holon has access to ${allTechTasks.length} tasks`);
+    
+    // Step 9: Update the task in tech team and propagate the change
+    console.log('\nStep 9: Updating task in tech team holon...');
+    
+    const updatedTask = {
+      ...task,
+      status: 'completed',
+      completedAt: new Date().toISOString()
+    };
+    
+    await holoSphere.put(techTeamHolon, 'tasks', updatedTask);
+    await holoSphere.propagate(techTeamHolon, 'tasks', updatedTask);
+    console.log('Task updated and propagated');
+    
+    // Step 10: Allow time for propagation
+    console.log('\nStep 10: Waiting for propagation to complete...');
+    await new Promise(resolve => setTimeout(resolve, 1000));
+    
+    // Step 11: Verify updates in both holons
+    console.log('\nStep 11: Verifying updated task in both holons...');
+    
+    const updatedTechTeamTask = await holoSphere.get(techTeamHolon, 'tasks', 'task-123');
+    console.log('Updated task in tech team holon status:', updatedTechTeamTask?.status);
+    
+    const updatedOrgTask = await holoSphere.get(orgHolon, 'tasks', 'task-123');
+    console.log('Updated task in organization holon status:', updatedOrgTask?.status);
+    
+    // Step 12: Clean up - remove federation
+    console.log('\nStep 12: Cleaning up - removing federation...');
+    
+    await holoSphere.unfederate(techTeamHolon, orgHolon);
+    console.log('Federation removed between tech team and organization');
+    
+    console.log('\nOrganization federation example completed successfully!');
+  } catch (error) {
+    console.error('Error in organization federation example:', error);
+  } finally {
+    // Always close the HoloSphere instance
+    await holoSphere.close();
+    console.log('HoloSphere instance closed');
+  }
+}
+
+// Run the example
+organizationFederationExample().catch(console.error);