interaqt 0.7.3 → 0.7.4

package/agent/agentspace/knowledge/generator/data-analysis.md

@@ -2,7 +2,12 @@
 
 ## Overview
 
-This guide explains how to analyze data requirements using the new requirement artifacts (`requirements/data-concepts.json` and `requirements/interactions-design.json`) to produce a comprehensive data design for interaqt implementations. The analysis leverages pre-extracted data concepts and interaction specifications to determine entity lifecycles, property dependencies, and computation methods.
+This guide explains how to analyze data requirements using the new requirement artifacts (`requirements/{module}.data-concepts.json`, `requirements/{module}.interactions-design.json`, and `requirements/{module}.integration.json`) to produce a comprehensive data design for interaqt implementations. The analysis leverages pre-extracted data concepts, interaction specifications, and external integration requirements to determine entity lifecycles, property dependencies, and computation methods.
+
+**IMPORTANT**: Read the current module name from `.currentmodule` file in project root, and use it to construct the file paths. For example, if `.currentmodule` contains `user-management`, then the paths would be:
+- `requirements/user-management.data-concepts.json`
+- `requirements/user-management.interactions-design.json`
+- `requirements/user-management.integration.json`
 
 ## Important Note on Interaction References
 
@@ -19,26 +24,41 @@ This applies to all interaction-related fields including:
 
 ## Input Artifacts
 
-### 1. requirements/data-concepts.json
+### 1. requirements/{module}.data-concepts.json
 Contains pre-extracted data concepts:
 - **Entities**: Core business objects with their properties
 - **Relations**: Connections between entities 
 - **Dictionaries**: Global data and system-wide configurations
 - **Views**: Pre-defined data queries and filters
 
-### 2. requirements/interactions-design.json
+Note: Replace `{module}` with the value from `.currentmodule` file.
+
+### 2. requirements/{module}.interactions-design.json
 Contains interaction specifications showing:
 - **Data operations**: What each interaction creates, reads, updates, or deletes
 - **Data constraints**: How data is modified or created
 - **Validation rules**: Business rules and constraints
 
+Note: Replace `{module}` with the value from `.currentmodule` file.
+
+### 3. requirements/{module}.integration.json
+Contains external system integration requirements:
+- **Integration flows**: Descriptions of interactions with external systems
+- **Asynchronous operations**: Payment processing, AIGC generation, file storage, etc.
+- **System boundaries**: What happens in current system vs external systems
+- **Data flow**: How data moves between systems
+
+Note: Replace `{module}` with the value from `.currentmodule` file. This file is used to identify integration event entities for tracking asynchronous external system responses.
+
 ## Analysis Process
 
 ### Step 1: Import Core Data Concepts
 
-#### 1.1 Import Entities from requirements/data-concepts.json
+#### 1.1 Import Entities from requirements/{module}.data-concepts.json
 
-Read the entities directly from `requirements/data-concepts.json`. Each entity already includes:
+First, read the module name from `.currentmodule` file in project root to get the current module name.
+
+Then read the entities directly from `requirements/{module}.data-concepts.json`. Each entity already includes:
 - Name and description
 - Properties with types and purposes
 - Computed property indicators
@@ -48,19 +68,100 @@ Read the entities directly from `requirements/data-concepts.json`. Each entity a
 
 #### 1.2 Import Dictionaries
 
-Read dictionaries from `requirements/data-concepts.json`. These represent:
+Read dictionaries from `requirements/{module}.data-concepts.json`. These represent:
 - System-wide configurations
 - Global statistics 
 - Shared validation rules
 - Cross-entity aggregates
 
+#### 1.3 Identify Integration Event Entities
+
+Read integration requirements from `requirements/{module}.integration.json` (using the module name from `.currentmodule`).
+
+**Identify Asynchronous External Calls:**
+For each integration in the file, analyze whether it involves asynchronous responses:
+- Webhook callbacks from external systems
+- Polling-based status checks
+- Delayed processing results (payment confirmations, AIGC generation results, etc.)
+
+**Create Event Entities for Asynchronous Integrations:**
+For each asynchronous integration identified, create a corresponding event entity following this pattern:
+
+**Naming Convention:**
+- `{IntegrationName}Event` (e.g., `PaymentEvent`, `AIGCEvent`, `FileUploadEvent`)
+- Use the integration name from `requirements/{module}.integration.json`
+
+**Event Entity Characteristics:**
+- **Immutable**: Can only be created, never updated or deleted
+- **Append-only**: New events are added to track state changes
+- **Source of Truth**: Business data in the system should be computed based on these events
+
+**Event Entity Properties:**
+Based on the integration's `flow_description` and `current_system_data`, determine what properties the event should capture:
+- External system's response data (transaction IDs, status codes, results)
+- Timestamp of when the event was received
+- Reference to related entities in current system (User.id, Order.id, etc.)
+- Any relevant metadata from the external system
+
+**Example:**
+```json
+// From requirements/{module}.integration.json:
+{
+  "id": "INT001",
+  "name": "PaymentProcessing",
+  "external_system": "Stripe",
+  "flow_description": "...Stripe processes payment...returns payment result..."
+}
+
+// Generated Event Entity:
+"PaymentEvent": {
+  "purpose": "Records payment status updates received from Stripe",
+  "isIntegrationEvent": true,
+  "properties": {
+    "transactionId": {
+      "type": "string",
+      "purpose": "External payment transaction ID from Stripe"
+    },
+    "eventType": {
+      "type": "string",
+      "purpose": "Type of External payment event type"
+    },
+    "paymentStatus": {
+      "type": "string", 
+      "purpose": "Status returned by Stripe (success, failed, pending)"
+    },
+    "amount": {
+      "type": "number",
+      "purpose": "Payment amount"
+    },
+    "timestamp": {
+      "type": "date",
+      "purpose": "When the payment result was received"
+    },
+    "userId": {
+      "type": "string",
+      "purpose": "Reference to User who initiated payment"
+    },
+    "orderId": {
+      "type": "string",
+      "purpose": "Reference to Order being paid for"
+    }
+  }
+}
+```
+
+**Documentation:**
+- Document the relationship between event entities and business entities
+- Explain how business data should be computed from these events
+- Note that other entities' properties may depend on these event entities
+
 ### Step 2: Analyze Entity Lifecycles Using Interactions
 
-For **EACH entity** in data-concepts.json:
+For **EACH entity** in `requirements/{module}.data-concepts.json`:
 
 #### 2.1 Determine Creation Pattern
 
-Analyze `requirements/interactions-design.json` to identify how entities are created:
+Analyze `requirements/{module}.interactions-design.json` (using the module name from `.currentmodule`) to identify how entities are created:
 
 **Step A: Find Creation Interactions**
 1. Search all interactions where the entity appears in `interactions.specification.data.creates`
@@ -73,6 +174,10 @@ Analyze `requirements/interactions-design.json` to identify how entities are cre
 **Step B: Determine Creation Type**
 Analyze the creation pattern:
 
+- **integration-event**: Entity is an event entity for external system integration
+  - Identified in Step 1.3 from `requirements/{module}.integration.json`
+  - Created when external system sends asynchronous responses (webhooks, callbacks)
+  
 - **interaction-created**: Entity is created independently by interactions
   - Entity appears alone in `data.creates` 
   - Or is the primary entity being created
@@ -84,7 +189,7 @@ Analyze the creation pattern:
   
 - **derived**: Entity is filtered/computed from other entities
   - No interactions directly create it
-  - Views in data-concepts.json are typically derived
+  - Views in `requirements/{module}.data-concepts.json` are typically derived
   
 - **mutation-derived**: Entity is created from record mutation events
   - Not directly in any interaction's `data.creates`
@@ -114,7 +219,7 @@ Analyze the creation pattern:
 // Bed is created-with-parent (parent: Dormitory) with creation details
 
 // For mutation-derived entity (not in any interaction's creates):
-// In data-concepts.json: "UserActivityLog: Records all user actions"
+// In requirements/{module}.data-concepts.json: "UserActivityLog: Records all user actions"
 // In interactions: No interaction directly creates UserActivityLog
 // In descriptions: "Activity logs are automatically created when users perform actions"
 // Result: UserActivityLog is mutation-derived
@@ -139,7 +244,7 @@ For **EACH property** of every entity:
 
 #### 3.1 Identify Data Dependencies
 
-From data-concepts.json, check if property is marked as `computed`:
+From `requirements/{module}.data-concepts.json`, check if property is marked as `computed`:
 - If `computed: true`, examine the `computation` field
 - List all entities/relations/properties mentioned in `computation.dependencies`
 - These become the property's `dataDependencies`
@@ -151,7 +256,7 @@ From data-concepts.json, check if property is marked as `computed`:
 
 #### 3.2 Identify Interaction Dependencies
 
-Search interactions-design.json to find which interactions modify this property:
+Search `requirements/{module}.interactions-design.json` (using the module name from `.currentmodule`) to find which interactions modify this property:
 1. Look for entity property in `data.updates` (e.g., "User.behaviorScore")
 2. Look for entity in `data.creates` (properties set at creation)
 3. List all matching interactions as `interactionDependencies` (use interaction **names**, not IDs)
@@ -159,7 +264,7 @@ Search interactions-design.json to find which interactions modify this property:
 #### 3.3 Determine Computation Method
 
 Transform the computation description using semantic best practices:
-- **Don't copy directly** from data-concepts.json
+- **Don't copy directly** from `requirements/{module}.data-concepts.json`
 - Apply the "Best Practices for Computation Design" principles
 - Use semantic computations (Count, Every, Any, Summation, etc.) where possible
 - Decompose complex calculations into intermediate properties
@@ -176,7 +281,7 @@ Based on the analysis:
 
 #### 4.1 Import Relation Structure
 
-Read relations from data-concepts.json:
+Read relations from `requirements/{module}.data-concepts.json` (using the module name from `.currentmodule`):
 - Source and target entities
 - Cardinality
 - Relation properties
@@ -193,18 +298,64 @@ Similar to entities, analyze how relations are created:
    - Dependencies from the creates entry
 3. Analyze creation context
 
-**Determine Creation Type**:
-- **interaction-created**: Relation created independently
-- **created-with-entity**: Created when source/target entity is created
+**Determine Creation Type - MANDATORY ALGORITHM**:
+
+**🔴 CRITICAL: Follow this exact decision algorithm for EACH relation:**
+
+```
+FOR each Relation found in interaction's data.creates:
+
+STEP 1: Check if relation NOT in any interaction's data.creates
+  → IF TRUE: Type = "mutation-derived"
+  → STOP
+
+STEP 2: Check the SAME interaction's data.creates array
+  → Find all Entities in the same creates array
+  
+STEP 3: Check relation's dependencies field
+  → IF dependencies contains ANY Entity from STEP 2:
+    → Type = "created-with-entity"
+    → parent = that Entity name
+    → STOP
+    
+STEP 4: Check description for keywords
+  → IF description contains "newly created" OR "just created" OR "connecting to created":
+    → Type = "created-with-entity"
+    → parent = the Entity being referenced
+    → STOP
+
+STEP 5: Default case (relation only, no entity in same creates)
+  → Type = "interaction-created"
+  → parent = null
+```
+
+**Type Definitions**:
+- **integration-event**: Relation created from external system events (rare, most integration events are entities)
+  - Would be identified if a relation needs to track external system relationships
+  - Immutable and append-only like integration-event entities
+  
+- **created-with-entity**: Relation created together with an entity in the SAME interaction
+  - **Key signal**: Relation's dependencies include an Entity that is ALSO in the same `data.creates` array
+  - **Key signal**: Description mentions "newly created" or "connecting to created" entity
+  - The relation should be created automatically by the entity's Transform computation
+  - Set `parent` to the entity name
+  
+- **interaction-created**: Relation created independently (no entity being created in same interaction)
+  - **Key signal**: Relation is the ONLY item in `data.creates`, or
+  - **Key signal**: All entities in dependencies already exist (not being created in same interaction)
+  - The relation needs its own Transform computation
+  
 - **derived**: Computed from data conditions
+  
 - **mutation-derived**: Created from record mutation events
   - Not directly in any interaction's `data.creates`
   - Created by reactive computations responding to entity/relation changes
   - Common for maintaining referential integrity or creating audit trails
 
-**Example**:
+**Examples with Algorithm Applied**:
+
 ```json
-// UserBedAssignment appears in:
+// Example 1: interaction-created
 // "AssignUserToBed": "data": { 
 //   "creates": [{
 //     "target": "UserBedAssignment",
@@ -212,26 +363,47 @@ Similar to entities, analyze how relations are created:
 //     "dependencies": ["User", "Bed", "AvailabilityCheck"]
 //   }]
 // }
-// Result: interaction-created with detailed creation info
+// Analysis:
+// - STEP 2: Same creates array has: [] (no entities)
+// - STEP 3: Dependencies ["User", "Bed"] are NOT in creates array
+// - STEP 5: Result = interaction-created
 
-// If it appeared with entity creation:
+// Example 2: created-with-entity
 // "CreatePost": "data": { 
 //   "creates": [
 //     {"target": "Post", "description": "Create new post", "dependencies": ["User"]},
 //     {"target": "PostAuthorRelation", "description": "Link post to author", "dependencies": ["Post", "User"]}
 //   ]
 // }
-// Result: PostAuthorRelation is created-with-entity (Post) with creation details
+// Analysis for PostAuthorRelation:
+// - STEP 2: Same creates array has: ["Post"]
+// - STEP 3: Dependencies contain "Post" which IS in creates array
+// - Result = created-with-entity (parent: "Post")
+
+// Example 3: created-with-entity (donate module case)
+// "RechargeGifts": "data": { 
+//   "creates": [
+//     {"target": "RechargeRecord", "description": "Create new recharge record...", "dependencies": []},
+//     {"target": "UserRechargeRelation", "description": "Create relation connecting current User to the newly created RechargeRecord", "dependencies": ["User", "RechargeRecord"]}
+//   ]
+// }
+// Analysis for UserRechargeRelation:
+// - STEP 2: Same creates array has: ["RechargeRecord"]
+// - STEP 3: Dependencies contain "RechargeRecord" which IS in creates array
+// - STEP 4: Description contains "newly created"
+// - Result = created-with-entity (parent: "RechargeRecord")
 
-// For mutation-derived relation:
+// Example 4: mutation-derived
 // UserFollowRelation not in any interaction's creates
 // Description: "Automatically created when user likes multiple posts by same author"
-// Result: UserFollowRelation is mutation-derived
+// Analysis:
+// - STEP 1: NOT in any creates array
+// - Result = mutation-derived
 ```
 
 ### Step 5: Transform Dictionaries to Analysis Format
 
-For each dictionary in data-concepts.json:
+For each dictionary in `requirements/{module}.data-concepts.json` (using the module name from `.currentmodule`):
 
 #### 5.1 Analyze Usage Patterns
 
@@ -330,14 +502,6 @@ Decompose into intermediate semantic computations:
 }
 ```
 
-### Benefits of This Approach
-
-1. **Clarity**: Each property has a single, clear purpose
-2. **Reusability**: Intermediate properties can be used by multiple consumers
-3. **Performance**: System can optimize semantic computations
-4. **Maintainability**: Changes to business logic are localized
-5. **Testability**: Each computation can be validated independently
-
 ### When to Create Intermediate Properties
 
 Create intermediate computed properties when you find yourself:
@@ -358,12 +522,13 @@ Transform the analyzed data into the standard output format:
 {
   "entities": {
     "[EntityName]": {
-      "purpose": "[From data-concepts.json description]",
+      "purpose": "[From requirements/{module}.data-concepts.json description]",
+      "isIntegrationEvent": "[true if this is an integration event entity, false otherwise]",
       "dataDependencies": "[Dependencies identified in Step 2]",
       "computationMethod": "[Creation pattern description]",
       "lifecycle": {
         "creation": {
-          "type": "[interaction-created | derived | created-with-parent | mutation-derived]",
+          "type": "[integration-event | interaction-created | derived | created-with-parent | mutation-derived]",
           "parent": "[Parent entity name if created-with-parent]",
           "creationInteractions": [
             {
@@ -374,8 +539,8 @@ Transform the analyzed data into the standard output format:
           ]
         },
         "deletion": {
-          "canBeDeleted": "[true/false based on Step 2.2]",
-          "deletionType": "[soft-delete | hard-delete | auto-delete]",
+          "canBeDeleted": "[true/false based on Step 2.2, always false for integration-event entities]",
+          "deletionType": "[soft-delete | hard-delete | auto-delete | none for integration-event]",
           "deletionInteractions": [
             {
               "name": "[Interaction name]",
@@ -387,8 +552,8 @@ Transform the analyzed data into the standard output format:
       },
       "properties": {
         "[propertyName]": {
-          "type": "[From data-concepts.json]",
-          "purpose": "[From data-concepts.json or inferred]",
+          "type": "[From requirements/{module}.data-concepts.json]",
+          "purpose": "[From requirements/{module}.data-concepts.json or inferred]",
           "controlType": "[From Step 3.4]",
           "dataDependencies": "[From Step 3.1]",
           "interactionDependencies": "[From Step 3.2]",
@@ -400,17 +565,17 @@ Transform the analyzed data into the standard output format:
   },
   "relations": {
     "[RelationName]": {
-      "type": "[From data-concepts.json cardinality]",
-      "purpose": "[From data-concepts.json description]",
-      "sourceEntity": "[From data-concepts.json]",
-      "targetEntity": "[From data-concepts.json]",
+      "type": "[From requirements/{module}.data-concepts.json cardinality]",
+      "purpose": "[From requirements/{module}.data-concepts.json description]",
+      "sourceEntity": "[From requirements/{module}.data-concepts.json]",
+      "targetEntity": "[From requirements/{module}.data-concepts.json]",
       "sourceProperty": "[Inferred or specified]",
       "targetProperty": "[Inferred or specified]",
       "dataDependencies": "[Always includes source and target entities]",
       "computationMethod": "[From Step 4.2]",
       "lifecycle": {
         "creation": {
-          "type": "[interaction-created | created-with-entity | derived | mutation-derived]",
+          "type": "[integration-event | interaction-created | created-with-entity | derived | mutation-derived]",
           "parent": "[If created-with-entity]",
           "creationInteractions": [
             {
@@ -441,7 +606,7 @@ Transform the analyzed data into the standard output format:
   },
   "dictionaries": {
     "[DictionaryName]": {
-      "purpose": "[From data-concepts.json description]",
+      "purpose": "[From requirements/{module}.data-concepts.json description]",
       "type": "[object with key types]",
       "dataDependencies": "[From Step 5.2]",
       "interactionDependencies": "[From Step 5.2]",
@@ -551,13 +716,75 @@ Entity created by different interactions with different logic:
 }
 ```
 
+### 6. Integration Event Entity
+Event entity for tracking asynchronous external system responses:
+```json
+"PaymentEvent": {
+  "purpose": "Records payment status updates received from Stripe payment gateway",
+  "isIntegrationEvent": true,
+  "dataDependencies": [],
+  "computationMethod": "Created when webhook receives payment status from Stripe",
+  "lifecycle": {
+    "creation": {
+      "type": "integration-event",
+      "parent": null,
+      "creationInteractions": []
+    },
+    "deletion": {
+      "canBeDeleted": false,
+      "deletionType": "none",
+      "deletionInteractions": []
+    }
+  },
+  "properties": {
+    "transactionId": {
+      "type": "string",
+      "purpose": "External payment transaction ID from Stripe",
+      "controlType": "creation-only",
+      "dataDependencies": [],
+      "interactionDependencies": [],
+      "computationMethod": "Set from Stripe webhook payload",
+      "initialValue": "From external system response"
+    },
+    "paymentStatus": {
+      "type": "string",
+      "purpose": "Payment status (success, failed, pending)",
+      "controlType": "creation-only",
+      "dataDependencies": [],
+      "interactionDependencies": [],
+      "computationMethod": "Set from Stripe webhook payload",
+      "initialValue": "From external system response"
+    },
+    "timestamp": {
+      "type": "date",
+      "purpose": "When the payment event was received",
+      "controlType": "creation-only",
+      "dataDependencies": [],
+      "interactionDependencies": [],
+      "computationMethod": "Current server timestamp when event is created",
+      "initialValue": "now()"
+    }
+  }
+}
+```
+
+Note: Other business entities (like `Order.paymentStatus`, `User.premiumUntil`) should be computed based on these event entities to maintain reactive consistency.
+
 ## Validation Checklist
 
-- [ ] All entities from data-concepts.json are analyzed
-- [ ] All relations from data-concepts.json are analyzed
-- [ ] All dictionaries from data-concepts.json are analyzed
+- [ ] Read module name from `.currentmodule` file in project root
+- [ ] All entities from `requirements/{module}.data-concepts.json` are analyzed
+- [ ] All relations from `requirements/{module}.data-concepts.json` are analyzed
+- [ ] All dictionaries from `requirements/{module}.data-concepts.json` are analyzed
+- [ ] Integration event entities identified from `requirements/{module}.integration.json`
+- [ ] Event entities properly marked with `isIntegrationEvent: true`
+- [ ] Event entities have lifecycle.creation.type set to "integration-event"
+- [ ] Event entities have deletion.canBeDeleted set to false
 - [ ] Creation patterns identified for each entity/relation
-- [ ] Interaction dependencies found by searching interactions-design.json
+- [ ] **🔴 CRITICAL: For EACH relation, executed the 5-step algorithm in Step 4.2 to determine lifecycle type**
+- [ ] **🔴 CRITICAL: For relations with type "created-with-entity", verified parent field is set correctly**
+- [ ] **🔴 CRITICAL: For relations in same creates array as entities, checked dependencies to identify created-with-entity pattern**
+- [ ] Interaction dependencies found by searching `requirements/{module}.interactions-design.json`
 - [ ] Data dependencies match computed property definitions
 - [ ] Lifecycle patterns are consistent with business logic
 - [ ] Parent-child relationships properly identified