jinaga-server 3.5.2 → 3.5.4

package/documentation/streaming-feeds-full-results-solution.md

@@ -1,230 +0,0 @@
-# Solution: Full Results Delivery for Streamed Feeds Endpoint
-
-## Problem Analysis
-
-The current streaming feeds endpoint has a critical limitation: it only delivers the first 100 results from the initial query and then switches to push-only mode. This means clients never receive complete historical data for feeds with more than 100 matching results.
-
-## Proposed Solution: Continuous Initial Query with Pagination
-
-### Core Design Principle
-
-Modify the streaming implementation to **continue fetching and delivering all historical results** before transitioning to real-time push mode, while maintaining the streaming connection throughout the process.
-
-### Implementation Strategy
-
-#### 1. Enhanced `streamFeed` Method
-
-```typescript
-private async streamFeed(user: RequestUser | null, params: { [key: string]: string }, query: qs.ParsedQs): Promise<Stream<FeedResponse> | null> {
-    const feedHash = params["hash"];
-    if (!feedHash) {
-        return null;
-    }
-
-    const feedDefinition = await this.feedCache.getFeed(feedHash);
-    if (!feedDefinition) {
-        return null;
-    }
-
-    let bookmark = query["b"] as string ?? "";
-    const userIdentity = serializeUserIdentity(user);
-    const start = feedDefinition.feed.given.map(label => feedDefinition.namedStart[label.name]);
-    const givenHash = computeObjectHash(feedDefinition.namedStart);
-
-    const stream = new Stream<FeedResponse>();
-    
-    // NEW: Continuous initial query until exhausted
-    bookmark = await this.streamAllInitialResults(userIdentity, feedDefinition, start, bookmark, stream);
-    
-    // Set up real-time listeners after initial data is complete
-    const inverses = invertSpecification(feedDefinition.feed);
-    const listeners = inverses.map(inverse => this.factManager.addSpecificationListener(
-        inverse.inverseSpecification,
-        async (results) => {
-            const matchingResults = results.filter(pr =>
-                givenHash === computeTupleSubsetHash(pr.tuple, inverse.givenSubset));
-            if (matchingResults.length != 0) {
-                bookmark = await this.streamFeedResponse(userIdentity, feedDefinition, start, bookmark, stream, true);
-            }
-        }
-    ));
-    
-    stream.done(() => {
-        for (const listener of listeners) {
-            this.factManager.removeSpecificationListener(listener);
-        }
-    });
-    
-    return stream;
-}
-```
-
-#### 2. New `streamAllInitialResults` Method
-
-```typescript
-private async streamAllInitialResults(
-    userIdentity: UserIdentity | null, 
-    feedDefinition: FeedObject, 
-    start: FactReference[], 
-    initialBookmark: string, 
-    stream: Stream<FeedResponse>
-): Promise<string> {
-    let bookmark = initialBookmark;
-    let hasMoreResults = true;
-    let pageCount = 0;
-    const maxPages = 1000; // Safety limit to prevent infinite loops
-    
-    while (hasMoreResults && pageCount < maxPages) {
-        const results = await this.authorization.feed(userIdentity, feedDefinition.feed, start, bookmark);
-        
-        // Check if we got results
-        if (results.tuples.length === 0) {
-            hasMoreResults = false;
-            break;
-        }
-        
-        // Process and send results
-        const references = results.tuples.flatMap(t => t.facts).filter((value, index, self) => 
-            self.findIndex(f => f.hash === value.hash && f.type === value.type) === index
-        );
-        
-        if (references.length > 0) {
-            const response: FeedResponse = {
-                references,
-                bookmark: results.bookmark
-            };
-            stream.feed(response);
-        }
-        
-        // Update bookmark for next iteration
-        const newBookmark = results.bookmark;
-        if (newBookmark === bookmark) {
-            // No progress made, avoid infinite loop
-            hasMoreResults = false;
-        } else {
-            bookmark = newBookmark;
-        }
-        
-        // Check if we got fewer results than the page size (indicates end)
-        if (results.tuples.length < 100) {
-            hasMoreResults = false;
-        }
-        
-        pageCount++;
-        
-        // Add small delay to prevent overwhelming the database
-        if (hasMoreResults && pageCount % 10 === 0) {
-            await new Promise(resolve => setTimeout(resolve, 10));
-        }
-    }
-    
-    return bookmark;
-}
-```
-
-### Key Design Features
-
-#### 1. **Continuous Pagination**
-- Loops through all available pages until no more results
-- Uses bookmark progression to detect completion
-- Includes safety limits to prevent infinite loops
-
-#### 2. **Stream Integrity**
-- Maintains single persistent connection throughout
-- Delivers results incrementally as pages are fetched
-- No interruption to client connection
-
-#### 3. **Performance Safeguards**
-- Maximum page limit (1000 pages = 100,000 results max)
-- Periodic delays to prevent database overload
-- Early termination on duplicate bookmarks
-
-#### 4. **Backward Compatibility**
-- Preserves existing API contract
-- Maintains same response format
-- No changes required to client implementations
-
-### Alternative Approaches Considered
-
-#### Option A: Async Background Fetching
-**Rejected**: Would complicate error handling and resource cleanup
-
-#### Option B: Configurable Page Limits
-**Rejected**: Adds complexity without solving the core issue
-
-#### Option C: Separate Historical vs Real-time Endpoints
-**Rejected**: Breaks the unified streaming model
-
-### Implementation Considerations
-
-#### 1. **Database Impact**
-- Multiple sequential queries instead of single query
-- Potential increased load on connection pool
-- Mitigation: Rate limiting and connection reuse
-
-#### 2. **Memory Management**
-- Stream queue may grow during rapid pagination
-- Mitigation: Client processing should keep pace with delivery
-- Consider: Optional memory pressure monitoring
-
-#### 3. **Timeout Handling**
-- 5-minute stream timeout may be insufficient for large datasets
-- Consider: Dynamic timeout based on data volume
-- Alternative: Configurable timeout per feed
-
-#### 4. **Error Recovery**
-- Database errors during pagination should not break stream
-- Implement: Retry logic for individual page failures
-- Fallback: Continue from last successful bookmark
-
-### Testing Strategy
-
-#### 1. **Unit Tests**
-- Test pagination logic with various result set sizes
-- Verify bookmark progression and termination conditions
-- Test error scenarios and recovery
-
-#### 2. **Integration Tests**
-- Large dataset streaming (1000+ results)
-- Concurrent stream handling
-- Database connection pool behavior under load
-
-#### 3. **Performance Tests**
-- Memory usage during large result streaming
-- Database query performance impact
-- Client processing capability validation
-
-### Deployment Considerations
-
-#### 1. **Gradual Rollout**
-- Feature flag to enable/disable full pagination
-- Monitor database performance impact
-- Rollback capability if issues arise
-
-#### 2. **Monitoring**
-- Track average pages per stream
-- Monitor database connection pool utilization
-- Alert on streams exceeding safety limits
-
-#### 3. **Configuration**
-- Make page limit configurable
-- Allow timeout adjustment per deployment
-- Enable/disable rate limiting
-
-## Expected Outcomes
-
-### Benefits
-1. **Complete Data Delivery**: Clients receive all historical results
-2. **Unified Experience**: Single endpoint for both historical and real-time data
-3. **Streaming Efficiency**: Maintains connection throughout data delivery
-4. **Backward Compatibility**: No breaking changes to existing clients
-
-### Trade-offs
-1. **Increased Database Load**: More queries per stream initialization
-2. **Longer Initial Response Time**: Complete data delivery takes longer
-3. **Memory Usage**: Larger stream queues during pagination
-4. **Complexity**: More sophisticated error handling required
-
-## Conclusion
-
-This solution addresses the core limitation while maintaining the streaming architecture's benefits. The continuous pagination approach ensures complete data delivery without breaking the real-time streaming model, providing a robust foundation for applications requiring both historical completeness and live updates.