interaqt 0.7.3 → 0.8.0

package/agent/agentspace/knowledge/generator/integration-implementation-handler.md

@@ -0,0 +1,550 @@
+---
+name: integration-implementation-handler
+description: Guide for implementing integrations to connect reactive backend with imperative external systems
+model: inherit
+color: purple
+---
+
+# Integration Implementation Guide
+
+## Overview
+
+**Integrations** bridge the gap between the reactive backend framework and imperative external systems (APIs, services, databases, message queues, etc.). They allow external non-reactive systems to participate in the reactive data flow.
+
+## Integration Interface
+
+Every integration must implement the `IIntegration` interface:
+
+```typescript
+export type IIntegration = {
+    configure?:() => Promise<any>
+    setup?:(controller: Controller) => Promise<any>
+    createSideEffects:() => RecordMutationSideEffect[]
+    createAPIs?: () => APIs
+}
+```
+
+### Constructor Arguments
+
+```typescript
+export type IIntegrationConstructorArgs = {
+    entities: EntityInstance[],
+    relations: RelationInstance[],
+    activities: ActivityInstance[],
+    interactions: InteractionInstance[],
+    dict: DictionaryInstance[]
+}
+
+export type IIntegrationHandles = {
+    [k:string]: any  // External handles like websocketServer, etc.
+}
+
+class MyIntegration implements IIntegration {
+    constructor(
+        public args: IIntegrationConstructorArgs, 
+        public handles: IIntegrationHandles
+    ) {}
+}
+```
+
+## Lifecycle Methods
+
+### 1. `configure()` - Schema Augmentation
+
+**Purpose**: Modify the reactive schema before system initialization.
+
+**Use Cases**:
+- Inject new entities into the system
+- Add computed properties to existing entities
+- Configure state machines for reactive properties
+- Inject computations into relations
+
+**Execution Timing**: Before Controller initialization
+
+**Example - Injecting Entity**:
+```typescript
+async configure() {
+    // Create and inject a new entity for external events
+    const TaskEvent = Entity.create({
+        name: 'LLMPicGenAsyncTaskEvent',
+        properties: [
+            Property.create({ name: 'taskId', type: 'string' }),
+            Property.create({ name: 'status', type: 'string' }),
+            Property.create({ name: 'result', type: 'object' }),
+        ]
+    });
+    
+    // Inject into entities array
+    this.args.entities.push(TaskEvent);
+}
+```
+
+**Example - Injecting Property Computation**:
+```typescript
+async configure() {
+    // Find target entities
+    const streamEntities = Stream.instances;
+    
+    for (const entity of streamEntities) {
+        // Find and inject computation into property
+        const urlProperty = entity.properties.find(p => p.name === 'url')!;
+        
+        urlProperty.computation = Custom.create({
+            name: 'generateStreamUrl',
+            async getInitialValue(this: Controller, record?: any) {
+                // Call external API to generate URL
+                const timestamp = Math.floor(Date.now() / 1000) + 3600;
+                const authString = `/${appName}/${streamName}${key}${timestamp}`;
+                const sign = crypto.createHash('md5').update(authString).digest('hex');
+                return `rtmp://${domain}/${appName}/${streamName}?volcTime=${timestamp}&volcSecret=${sign}`;
+            },
+            incrementalCompute: async function(this: { controller: Controller, state: any }, lastValue: any, mutationEvent: any, record: any, dataDeps: any) {
+                // Skip recomputation if URL should remain constant
+                return ComputationResult.skip();
+            }
+        });
+    }
+}
+```
+
+**Example - Injecting State Machine**:
+```typescript
+async configure() {
+    const taskEntities = AsyncTask.instances;
+    
+    for (const taskEntity of taskEntities) {
+        const statusProperty = taskEntity.properties.find(p => p.name === 'status');
+        
+        // Create state node
+        const statusState = StateNode.create({
+            name: 'status',
+            computeValue: (lastValue, mutationEvent) => {
+                return mutationEvent?.record?.status || lastValue || 'pending';
+            }
+        });
+        
+        // Configure state machine
+        statusProperty.computation = StateMachine.create({
+            states: [statusState],
+            initialState: statusState,
+            transfers: [
+                StateTransfer.create({
+                    trigger: {
+                        recordName: 'TaskEvent',  // Event entity name
+                        type: 'create',
+                        record: { taskType: taskEntity.name }
+                    },
+                    current: statusState,
+                    next: statusState,
+                    computeTarget: async function(this: Controller, mutationEvent: any) {
+                        const event = mutationEvent.record;
+                        if (event.status) {
+                            return { id: event.taskId };  // Target record to update
+                        }
+                        return undefined;
+                    }
+                })
+            ]
+        });
+    }
+}
+```
+
+### 2. `setup()` - Runtime Initialization
+
+**Purpose**: Initialize external connections and register runtime handlers.
+
+**Use Cases**:
+- Connect to external services (Redis, databases, message queues)
+- Register event listeners on external handles
+- Set up WebSocket connection handlers
+- Initialize API clients
+
+**Execution Timing**: After Controller initialization, before server starts
+
+**Example - External Service Connection**:
+```typescript
+async setup(controller: Controller) {
+    const redisUrl = process.env.REDIS_URL || 'redis://localhost:6379';
+    
+    this.subscriber = createClient({ url: redisUrl });
+    this.subscriber.on('error', (err) => console.error('Redis Error:', err));
+    await this.subscriber.connect();
+    
+    this.publisher = createClient({ url: redisUrl });
+    await this.publisher.connect();
+}
+```
+
+**Example - WebSocket Handler Registration**:
+```typescript
+async setup(controller: Controller) {
+    // Register handler for user connection
+    this.handles.websocketServer.on('connection-setup-completed', async (ws, request) => {
+        const user = ws.user;
+        if (!user) return;
+        
+        // Query user's channels
+        const userChannels = await controller.system.storage.find(
+            'UserChannelRelation',
+            MatchExp.atom({ key: 'source.id', value: ['=', user.id] }),
+            undefined,
+            ['id', ['target', { attributeQuery: ['id'] }]]
+        );
+        
+        // Subscribe to Redis channels for this user
+        for (let relation of userChannels) {
+            this.subscriber?.subscribe(relation.target.id, (message) => {
+                ws.send(message);
+            });
+        }
+    });
+}
+```
+
+### 3. `createSideEffects()` - Reactive Synchronization
+
+**Purpose**: Define side effects that synchronize reactive data changes to external systems.
+
+**Use Cases**:
+- Push data mutations to external APIs
+- Publish messages to message queues
+- Update external database records
+- Trigger external workflows
+
+**Return Value**: Array of `RecordMutationSideEffect`
+
+**Example - Publishing to Message Queue**:
+```typescript
+createSideEffects(): RecordMutationSideEffect[] {
+    const dispatcher = this;
+    
+    return [
+        RecordMutationSideEffect.create({
+            name: 'channel_message_publish_sideeffect',
+            record: {
+                name: 'ChannelMessageRelation',  // Target relation/entity
+            },
+            async content(this: Controller, event: RecordMutationEvent) {
+                if (event.type === 'create') {
+                    const channelId = event.record?.source.id;
+                    const message = event.record?.target;
+                    
+                    if (message && channelId) {
+                        const messageData = {
+                            type: 'message',
+                            channelId,
+                            message
+                        };
+                        
+                        // Publish to external system
+                        await dispatcher.publisher?.publish(
+                            channelId, 
+                            JSON.stringify(messageData)
+                        );
+                        console.log('Published to Redis:', channelId);
+                    }
+                }
+            }
+        })
+    ];
+}
+```
+
+**Example - Multiple Side Effects**:
+```typescript
+createSideEffects(): RecordMutationSideEffect[] {
+    return [
+        // Handle relation creation
+        ...UserChannelRelation.instances.map(instance => 
+            RecordMutationSideEffect.create({
+                name: `channel_user_relation_${instance.name}_create`,
+                record: { name: instance.name! },
+                async content(this: Controller, event: RecordMutationEvent) {
+                    if (event.type === 'create') {
+                        // Subscribe online users to new channels
+                        const clients = Array.from(websocketServer.clients);
+                        const wsClient = clients.find(c => c.user.id === event.record?.source.id);
+                        if (wsClient) {
+                            subscriber?.subscribe(event.record?.target.id, (msg) => {
+                                wsClient.send(msg);
+                            });
+                        }
+                    }
+                }
+            })
+        ),
+        
+        // Handle relation deletion
+        ...UserChannelRelation.instances.map(instance => 
+            RecordMutationSideEffect.create({
+                name: `channel_user_relation_${instance.name}_delete`,
+                record: { name: instance.name! },
+                async content(this: Controller, event: RecordMutationEvent) {
+                    if (event.type === 'delete') {
+                        // Unsubscribe user from channel
+                        // ... implementation
+                    }
+                }
+            })
+        )
+    ];
+}
+```
+
+### 4. `createAPIs()` - Custom Endpoints
+
+**Purpose**: Create custom API endpoints for external system interactions.
+
+**Use Cases**:
+- Query external service status
+- Trigger external operations
+- Proxy requests to external APIs
+- Implement custom business logic that doesn't fit interactions
+
+**Return Value**: Object mapping API names to API definitions
+
+**Example - Status Query API**:
+```typescript
+createAPIs(): APIs {
+    const apis: APIs = {};
+    
+    apis.queryTaskStatus = createAPI(
+        async function(this: Controller, context, params: { taskId: string, taskType: string }) {
+            const { taskId, taskType } = params;
+            
+            // Query internal state
+            const task = await this.system.storage.findOne(
+                taskType,
+                MatchExp.atom({ key: 'id', value: ['=', taskId] }),
+                undefined,
+                ['id', 'executionId', 'status', 'result']
+            );
+            
+            if (!task) {
+                return { error: 'Task not found' };
+            }
+            
+            // Query external system
+            const externalStatus = await queryExternalAPI(task.executionId);
+            
+            // Update internal state via event creation
+            if (externalStatus.status !== task.status) {
+                await this.system.storage.create('TaskEvent', {
+                    taskId,
+                    eventType: 'statusUpdated',
+                    status: externalStatus.status,
+                    result: externalStatus.result,
+                    taskType
+                });
+            }
+            
+            return {
+                success: true,
+                taskId,
+                status: externalStatus.status,
+                result: externalStatus.result
+            };
+        },
+        {
+            params: { taskId: 'string', taskType: 'string' },
+            useNamedParams: true,
+            allowAnonymous: false
+        }
+    );
+    
+    return apis;
+}
+```
+
+## Integration Patterns
+
+### Pattern 1: External Service Synchronization (Redis, MQ)
+
+**Characteristics**:
+- Bidirectional data flow
+- Real-time synchronization
+- Connection management
+
+**Implementation**:
+- `setup()`: Establish connections, register listeners
+- `createSideEffects()`: Push internal changes to external system
+- External events → Create records in reactive system
+
+**Example**: RedisChannelIntegration
+
+### Pattern 2: External Resource Generation (URLs, Tokens)
+
+**Characteristics**:
+- One-way data flow (internal → external)
+- Lazy evaluation
+- Resource lifecycle management
+
+**Implementation**:
+- `configure()`: Inject Custom computation into properties
+- `getInitialValue()`: Call external API to generate resource
+- `incrementalCompute()`: Usually skip (resources are immutable)
+
+**Example**: VolcStreamIntegration
+
+### Pattern 3: Async Task Management
+
+**Characteristics**:
+- Async operation lifecycle
+- Status polling
+- Result synchronization
+
+**Implementation**:
+- `configure()`: Inject event entity, configure state machines
+- `createAPIs()`: Provide status query endpoint
+- Event-driven state updates via injected entity
+
+**Example**: VolcPicGenIntegration
+
+## Best Practices
+
+### 1. Separation of Concerns
+
+- **configure()**: Schema modifications only
+- **setup()**: Connection establishment only
+- **createSideEffects()**: Data synchronization only
+- **createAPIs()**: Query/command operations only
+
+### 2. Error Handling
+
+```typescript
+async setup(controller: Controller) {
+    try {
+        this.client = await connectToService();
+        this.client.on('error', (err) => {
+            console.error('Service error:', err);
+            // Implement reconnection logic
+        });
+    } catch (error) {
+        console.error('Failed to connect:', error);
+        throw error;  // Fail fast if critical
+    }
+}
+```
+
+### 3. Event-Driven State Updates
+
+**❌ BAD - Direct State Mutation**:
+```typescript
+// Don't directly update entity properties
+await this.system.storage.update(taskType, taskId, {
+    status: newStatus  // This bypasses reactive system
+});
+```
+
+**✅ GOOD - Event-Driven Updates**:
+```typescript
+// Create event records to trigger state machines
+await this.system.storage.create('TaskEvent', {
+    taskId,
+    eventType: 'statusUpdated',
+    status: newStatus,
+    taskType
+});
+// State machine will reactively update the target entity
+```
+
+### 4. Resource Cleanup
+
+```typescript
+async cleanup() {
+    if (this.subscriber?.isOpen) {
+        await this.subscriber.disconnect();
+    }
+    if (this.publisher?.isOpen) {
+        await this.publisher.disconnect();
+    }
+}
+```
+
+### 5. Environment Configuration
+
+```typescript
+async setup(controller: Controller) {
+    const apiKey = process.env.EXTERNAL_API_KEY;
+    if (!apiKey) {
+        throw new Error('EXTERNAL_API_KEY must be set');
+    }
+    // ... use apiKey
+}
+```
+
+### 6. Type Safety with External APIs
+
+```typescript
+// Define external API types
+export type ExternalTaskStatus = 'pending' | 'processing' | 'success' | 'failed';
+
+// Map to internal types
+function mapExternalStatus(external: string): TaskStatus {
+    switch (external) {
+        case 'in_queue': return 'pending';
+        case 'generating': return 'processing';
+        case 'done': return 'success';
+        default: return 'failed';
+    }
+}
+```
+
+## Integration Registration
+
+After creating an integration, register it in `integrations/index.ts`:
+
+```typescript
+import { MyIntegration } from './myintegration';
+
+const AggregatedIntegrationClass = createAggregatedIntegration([
+    RedisChannelIntegration,
+    VolcStreamIntegration,
+    VolcPicGenIntegration,
+    MyIntegration  // Add your integration
+]);
+
+export default AggregatedIntegrationClass;
+```
+
+## Testing Integrations
+
+1. **Unit Tests**: Test integration logic in isolation
+2. **Integration Tests**: Test with mock external services
+3. **E2E Tests**: Test with real external services in staging
+
+```typescript
+// Mock external service for testing
+class MockExternalService {
+    async query(id: string) {
+        return { status: 'success', result: { data: 'test' } };
+    }
+}
+
+// Use in tests
+const integration = new MyIntegration(args, { 
+    externalService: new MockExternalService() 
+});
+```
+
+## Common Pitfalls
+
+1. **❌ Modifying schema in setup()**: Schema must be finalized before Controller initialization
+2. **❌ Blocking operations in configure()**: Avoid heavy I/O, keep it fast
+3. **❌ Forgetting error handlers**: Always handle connection errors
+4. **❌ Direct state mutations**: Use event-driven updates instead
+5. **❌ Ignoring cleanup**: Implement proper resource cleanup
+6. **❌ Hardcoding configuration**: Use environment variables
+
+## Summary
+
+Integrations enable reactive systems to communicate with external imperative systems through four key mechanisms:
+
+1. **configure()**: Augment reactive schema
+2. **setup()**: Initialize external connections
+3. **createSideEffects()**: Synchronize data to external systems
+4. **createAPIs()**: Expose custom endpoints
+
+Choose the appropriate pattern based on your integration needs, and always prioritize event-driven design for state synchronization.
+

package/agent/agentspace/prompt/integration_sub_agent_refactor.md

@@ -0,0 +1,19 @@
+我们的项目所使用的框架是一个叫做 Interaqt 的后端响应式数据框架，它会自动根据应用中的数据变化及响应式数据的定义执行相应的数据变化。它只负责处理一般的业务逻辑中表达的数据逻辑，例如一般业务逻辑中会用到 平均/综合 等计算，还有常见的基于状态机等业务逻辑表达等。对于一些非一般的能力需求，例如 大模型生图、大模型生视频、tts、发送邮件、发送信息、完整支付系统等。它需要借助外部系统/api 来完成。
+我们设计了一个叫做 integration 的概念，专门用来对接当前系统和外部的api/系统。它通过 interaqt 框架提供的数据观察机制，来观察数据变化，根据数据变化来决定如何调用外部的 api。同时通过 webhook 等机制来接受外部的事件，将外部事件同步回系统中，触发响应式的业务逻辑。
+
+我们将 integration 需要集成的功能分成了三种类别：
+1. 调用外部的 api，为了获得一个具体的返回。例如 tts，大模型生图等。
+2. 执行某种副作用，例如发送邮件、发送 im 消息等。
+3. 对接其他有状态的系统，例如支付系统等。
+
+现在我们需要指导 claude code 的 sub agent 合理地识别需要的外部服务以及如何自己实现 integration。
+1. 指导 `.claude/agents/requirements-analysis-handler.md` 在需求分析阶段，正确分析出 integration 的类型。并在相应的输出的文档中，设计一个字段来表达 integration 的类型。
+2. 指导 `.claude/agents/implement-design-handler.md` 在设计数据的时候，根据如下原则进行设计：
+  2.1. 不管是哪种类型，都会涉及到对外部 api 的调用，例如执行副作用，也会有副作用 api 的调用。所以我们应该对每一个 api 的调用都设计一个 `{xxx}APICall` 的 entity，它负责记录这次 api 调用的参数、状态、返回值、调用时间等。
+  2.2. 同时设计一个相应的 integraion event entity，当我们通过 webhook 或者自己通过接口查询到 api 调用状态的变化时，在系统内创建相应的 api call result event 事件。并且将上一步创建的 `{xxx}APICall` entity 的 status 和 data 字段写成基于 integration event entity 的 computation，这样就完整符合了框架的响应式范式。也记录了所有应该记录的数据，增强了系统的健壮性。
+  2.3. 如果当前场景是希望基于这个 integration 获得具体的返回值，那么意味着我们系统内的业务数据对这个 `{xxx}APICall` 的 entity 是有依赖的，应该写成基于 `{xxx}APICall` 的 computation。例如我们的有一个 `Greeting` entity，其中有个 `voiceUrl` property 是需要利用外部 tts 能力将文本转化为语音。那么 `Greeting.voiceUrl` 就应该表达为基于 `{ttsAPICall}` entity 的 computation。如果是纯副作用类型等的调用，就不需要了。注意，这种情况下，还需要建立相应的 entity 和 api call entity 之间的关系，才能查找到正确的数据。
+  2.4. `.claude/agents/implement-design-handler.md` 在做 data-design 的时候，应该明确表达出来：1. 设计的哪些实体是 api call 类型的 entity，哪些实体是 api call result event 实体。2. 系统内的业务数据如果需要 api 的返回结果，那么应该依赖正确的 api call entity。
+3. 指导 `.claude/agents/code-generation-handler.md` 在实现阶段，在写测试用例时，完全可以通过创建正确的 api call result event 来模拟 api 的调用，完整验证系统的内部逻辑的正确性。不需要等到 integration 的真实实现。
+4. 指导 `.claude/agents/error-check-handler.md` 在合适的阶段对 integration 相关的设计做错误检查。
+
+你充分理解上面的所有思路，并且修改相应的 sub agent 文件来达成目标。

package/agent/agentspace/prompt/requirement_analysis_refactor.md

@@ -29,6 +29,6 @@
 6. 最终产出的 interactions-design.json 仍然使用原文档里的 interaction 数据结构，但以流程为组来组织。
 
 注意，在每一个步骤中，都要给出清晰的数据结构定义，用 json 来写。
-整体用简洁的英语完成文档重写。注意原本文档中的在关键步骤 update STATUS.json 仍然按照原文档的方式写。
+整体用简洁的英语完成文档重写。注意原本文档中的在关键步骤 update docs/{module}.status.json 仍然按照原文档的方式写。