@trafficgroup/knex-rel 0.1.8 → 0.1.10

package/plan.md

@@ -1,684 +0,0 @@
-# Plan: Add Study-Level Minute Result Aggregation
-
-## Overview
-
-Add `getGroupedMinuteResultsByStudyUuid()` method to `VideoMinuteResultDAO` to aggregate minute results across all COMPLETED videos in a study, with time normalization using `recordingStartedAt`.
-
----
-
-## Schema Analysis (Existing)
-
-### Table Relationships
-
-```
-study (id, uuid, name, type, status)
-  ↓ (1:N)
-folders (id, uuid, studyId)
-  ↓ (1:N)
-video (id, uuid, folderId, recordingStartedAt, status, videoType)
-  ↓ (1:N)
-video_minute_results (id, uuid, video_id, minute_number, results)
-```
-
-### Key Fields
-
-- `video.recordingStartedAt`: TIMESTAMPTZ in UTC (nullable for backward compatibility)
-- `video.status`: 'QUEUED' | 'PROCESSING' | 'COMPLETED' | 'FAILED' | 'PENDING'
-- `video_minute_results.minute_number`: Integer offset from video start (0-based)
-- `video_minute_results.results`: JSONB containing TMC or ATR data
-
-### Existing Indexes
-
-- `idx_videos_folder_recording` on (folderId, recordingStartedAt) WHERE recordingStartedAt IS NOT NULL
-
----
-
-## Interface Definitions
-
-### New Response Interface
-
-**File**: `knex-rel/src/dao/VideoMinuteResultDAO.ts` (add at top with existing interfaces)
-
-```typescript
-interface IStudyTimeGroupResult {
-  absoluteTime: string; // ISO 8601 datetime for bucket start (e.g., "2024-10-26T14:30:00Z")
-  groupIndex: number; // Sequential index for UI ordering
-  label: string; // Human-readable label (e.g., "2:30 PM - 2:44 PM")
-  results: ITMCResult | IATRResult; // Aggregated results (reuse existing types)
-  minuteCount: number; // Total individual minutes aggregated
-  videoCount: number; // Number of distinct videos contributing
-  contributingVideos: string[]; // Array of video UUIDs for traceability
-}
-
-interface IGroupedStudyResponse {
-  success: boolean;
-  data: IStudyTimeGroupResult[];
-  groupingMinutes: number; // Grouping interval (1, 5, 10, 15, 30, 60)
-  study: {
-    uuid: string;
-    name: string;
-    type: "TMC" | "ATR";
-    status: string;
-  };
-  videoCount: number; // Total videos included in aggregation
-  dateRange: {
-    earliest: string; // ISO 8601 of earliest recording start
-    latest: string; // ISO 8601 of latest recording end
-  };
-}
-```
-
----
-
-## SQL Query Strategy
-
-### Phase 1: Get Study Videos with Metadata
-
-**Purpose**: Fetch all COMPLETED videos with recordingStartedAt, validate study exists
-
-```sql
-SELECT
-  s.id as study_id,
-  s.uuid as study_uuid,
-  s.name as study_name,
-  s.type as study_type,
-  s.status as study_status,
-  v.id as video_id,
-  v.uuid as video_uuid,
-  v.name as video_name,
-  v.recording_started_at,
-  v.duration_seconds
-FROM study s
-INNER JOIN folders f ON f.study_id = s.id
-INNER JOIN video v ON v.folder_id = f.id
-WHERE s.uuid = ?
-  AND v.status = 'COMPLETED'
-  AND v.recording_started_at IS NOT NULL
-ORDER BY v.recording_started_at ASC;
-```
-
-**Error Handling**:
-
-- If study not found → throw Error("Study with UUID {uuid} not found")
-- If no videos found → return empty data array with study metadata
-- If videos missing recordingStartedAt → log warning, exclude from aggregation
-
-### Phase 2: Get All Minute Results with Time Normalization
-
-**Purpose**: Fetch minute results with absolute timestamps calculated in SQL
-
-```sql
-SELECT
-  vmr.id,
-  vmr.video_id,
-  vmr.minute_number,
-  vmr.results,
-  v.uuid as video_uuid,
-  v.recording_started_at,
-  -- Calculate absolute wall-clock time for this minute
-  (v.recording_started_at + (vmr.minute_number || ' minutes')::INTERVAL) as absolute_time,
-  -- Calculate time bucket index (floor division by grouping interval)
-  FLOOR(
-    EXTRACT(EPOCH FROM (v.recording_started_at + (vmr.minute_number || ' minutes')::INTERVAL)) / (? * 60)
-  ) as time_bucket
-FROM video_minute_results vmr
-INNER JOIN video v ON vmr.video_id = v.id
-WHERE v.id IN (?)  -- Array of video IDs from Phase 1
-ORDER BY time_bucket, vmr.minute_number;
-```
-
-**Parameters**:
-
-- `?` (first): groupingMinutes (e.g., 15)
-- `?` (second): Array of video IDs from Phase 1
-
-**Output**: Rows with absolute timestamps and time bucket assignments
-
-### Phase 3: Aggregate by Time Bucket (TypeScript)
-
-**Reason for TypeScript aggregation**:
-
-- Reuse existing `aggregateTMCResults()` and `aggregateATRResults()` methods
-- Complex nested JSON aggregation is easier in TypeScript than SQL
-- Study type determines aggregation logic (not individual video types)
-
-**Pseudo-SQL for Grouping** (illustrative, actual grouping in TS):
-
-```sql
-SELECT
-  time_bucket,
-  MIN(absolute_time) as bucket_start_time,
-  MAX(absolute_time) as bucket_end_time,
-  COUNT(*) as minute_count,
-  COUNT(DISTINCT video_id) as video_count,
-  array_agg(DISTINCT video_uuid) as contributing_videos,
-  array_agg(results ORDER BY absolute_time) as all_results
-FROM [results_from_phase_2]
-GROUP BY time_bucket
-ORDER BY time_bucket;
-```
-
----
-
-## TypeScript Implementation Logic
-
-### Method Signature
-
-```typescript
-async getGroupedMinuteResultsByStudyUuid(
-  studyUuid: string,
-  groupingMinutes: number = 15
-): Promise<IGroupedStudyResponse>
-```
-
-### Step-by-Step Logic
-
-#### Step 1: Fetch Study and Videos
-
-```typescript
-// Get study metadata
-const study = await this.knex("study").where("uuid", studyUuid).first();
-if (!study) {
-  throw new Error(`Study with UUID ${studyUuid} not found`);
-}
-
-// Get all COMPLETED videos with recordingStartedAt
-const videos = await this.knex("video as v")
-  .innerJoin("folders as f", "v.folderId", "f.id")
-  .select(
-    "v.id",
-    "v.uuid",
-    "v.name",
-    "v.recordingStartedAt",
-    "v.durationSeconds",
-  )
-  .where("f.studyId", study.id)
-  .where("v.status", "COMPLETED")
-  .whereNotNull("v.recordingStartedAt")
-  .orderBy("v.recordingStartedAt", "asc");
-
-// Early return if no valid videos
-if (videos.length === 0) {
-  return {
-    success: true,
-    data: [],
-    groupingMinutes,
-    study: {
-      uuid: study.uuid,
-      name: study.name,
-      type: study.type,
-      status: study.status,
-    },
-    videoCount: 0,
-    dateRange: {
-      earliest: null,
-      latest: null,
-    },
-  };
-}
-```
-
-#### Step 2: Fetch and Transform Minute Results
-
-```typescript
-const videoIds = videos.map((v) => v.id);
-const videoMap = new Map(videos.map((v) => [v.id, v]));
-
-// Fetch minute results with absolute time calculation
-const minuteResults = await this.knex("video_minute_results as vmr")
-  .innerJoin("video as v", "vmr.video_id", "v.id")
-  .select(
-    "vmr.video_id",
-    "vmr.minute_number",
-    "vmr.results",
-    "v.uuid as video_uuid",
-    this.knex.raw(`
-      v.recording_started_at + (vmr.minute_number || ' minutes')::INTERVAL
-      as absolute_time
-    `),
-    this.knex.raw(
-      `
-      FLOOR(
-        EXTRACT(EPOCH FROM (v.recording_started_at + (vmr.minute_number || ' minutes')::INTERVAL))
-        / (? * 60)
-      ) as time_bucket
-    `,
-      [groupingMinutes],
-    ),
-  )
-  .whereIn("v.id", videoIds)
-  .orderBy("time_bucket")
-  .orderBy("vmr.minute_number");
-```
-
-#### Step 3: Group by Time Buckets
-
-```typescript
-// Group results by time bucket
-const bucketMap = new Map<
-  number,
-  {
-    absoluteTime: Date;
-    results: any[];
-    videoUuids: Set<string>;
-    minuteCount: number;
-  }
->();
-
-for (const row of minuteResults) {
-  const bucket = row.time_bucket;
-
-  if (!bucketMap.has(bucket)) {
-    bucketMap.set(bucket, {
-      absoluteTime: new Date(row.absolute_time),
-      results: [],
-      videoUuids: new Set(),
-      minuteCount: 0,
-    });
-  }
-
-  const bucketData = bucketMap.get(bucket)!;
-  bucketData.results.push(row.results);
-  bucketData.videoUuids.add(row.video_uuid);
-  bucketData.minuteCount++;
-}
-```
-
-#### Step 4: Aggregate Each Bucket
-
-```typescript
-// Sort buckets by time
-const sortedBuckets = Array.from(bucketMap.entries()).sort(
-  (a, b) => a[0] - b[0],
-);
-
-const aggregatedGroups: IStudyTimeGroupResult[] = sortedBuckets.map(
-  ([bucketIndex, bucketData], idx) => {
-    // Determine aggregation type based on study type
-    let aggregatedResult;
-    if (study.type === "TMC") {
-      aggregatedResult = this.aggregateTMCResults(bucketData.results);
-    } else {
-      aggregatedResult = this.aggregateATRResults(bucketData.results);
-    }
-
-    // Calculate bucket end time
-    const bucketStartTime = bucketData.absoluteTime;
-    const bucketEndTime = new Date(
-      bucketStartTime.getTime() + groupingMinutes * 60 * 1000 - 1000,
-    );
-
-    return {
-      absoluteTime: bucketStartTime.toISOString(),
-      groupIndex: idx,
-      label: this.formatAbsoluteTimeLabel(bucketStartTime, bucketEndTime),
-      results: aggregatedResult,
-      minuteCount: bucketData.minuteCount,
-      videoCount: bucketData.videoUuids.size,
-      contributingVideos: Array.from(bucketData.videoUuids),
-    };
-  },
-);
-```
-
-#### Step 5: Calculate Date Range
-
-```typescript
-const earliestVideo = videos[0]; // Already sorted by recordingStartedAt
-const latestVideo = videos[videos.length - 1];
-
-// Calculate actual end time of last video
-const latestEndTime = latestVideo.durationSeconds
-  ? new Date(
-      new Date(latestVideo.recordingStartedAt).getTime() +
-        latestVideo.durationSeconds * 1000,
-    )
-  : new Date(latestVideo.recordingStartedAt);
-
-return {
-  success: true,
-  data: aggregatedGroups,
-  groupingMinutes,
-  study: {
-    uuid: study.uuid,
-    name: study.name,
-    type: study.type,
-    status: study.status,
-  },
-  videoCount: videos.length,
-  dateRange: {
-    earliest: earliestVideo.recordingStartedAt.toISOString(),
-    latest: latestEndTime.toISOString(),
-  },
-};
-```
-
----
-
-## Code Reuse Strategy
-
-### Existing Methods to Reuse
-
-1. **`aggregateTMCResults(minutes: any[]): ITMCResult`** (lines 508-585)
-   - No changes needed
-   - Accepts array of raw results objects
-   - Returns aggregated TMC structure
-
-2. **`aggregateATRResults(minutes: any[]): IATRResult`** (lines 590-643)
-   - No changes needed
-   - Accepts array of raw results objects
-   - Returns aggregated ATR structure
-
-3. **`formatTimeLabel(startMinute: number, endMinute: number): string`** (lines 703-715)
-   - DO NOT reuse - this formats video-relative time (HH:MM format from minute offset)
-   - Need NEW method for absolute time formatting
-
-### New Helper Method Required
-
-```typescript
-/**
- * Format absolute time label for display (12-hour clock)
- */
-private formatAbsoluteTimeLabel(startTime: Date, endTime: Date): string {
-  const formatTime = (date: Date): string => {
-    const hours = date.getHours();
-    const minutes = date.getMinutes();
-    const ampm = hours >= 12 ? 'PM' : 'AM';
-    const displayHours = hours % 12 || 12;
-    return `${displayHours}:${minutes.toString().padStart(2, '0')} ${ampm}`;
-  };
-
-  const formatDate = (date: Date): string => {
-    return date.toLocaleDateString('en-US', {
-      month: 'short',
-      day: 'numeric',
-      year: 'numeric'
-    });
-  };
-
-  // If same day, show: "Oct 26: 2:30 PM - 2:44 PM"
-  if (startTime.toDateString() === endTime.toDateString()) {
-    return `${formatDate(startTime)}: ${formatTime(startTime)} - ${formatTime(endTime)}`;
-  }
-
-  // If different days, show full timestamps
-  return `${formatDate(startTime)} ${formatTime(startTime)} - ${formatDate(endTime)} ${formatTime(endTime)}`;
-}
-```
-
----
-
-## Error Handling Strategy
-
-### Input Validation
-
-```typescript
-// Validate groupingMinutes
-const validIntervals = [1, 5, 10, 15, 30, 60];
-if (!validIntervals.includes(groupingMinutes)) {
-  throw new Error(
-    `Invalid grouping interval: ${groupingMinutes}. Must be one of: ${validIntervals.join(", ")}`,
-  );
-}
-```
-
-### Edge Cases
-
-| Scenario                                     | Handling                                           |
-| -------------------------------------------- | -------------------------------------------------- |
-| Study not found                              | Throw error immediately                            |
-| No folders in study                          | Return empty data array with study metadata        |
-| No COMPLETED videos                          | Return empty data array, videoCount = 0            |
-| Videos missing recordingStartedAt            | Exclude from aggregation, log warning              |
-| No minute results                            | Return empty data array                            |
-| Type mismatch (video.videoType ≠ study.type) | Use study.type for aggregation (trust study level) |
-| Missing duration_seconds                     | Use last minute_number to estimate end time        |
-| Empty results in minute                      | Include in count but don't affect aggregation      |
-
-### Logging Strategy
-
-```typescript
-// At start of method
-console.log(
-  `[VideoMinuteResultDAO] Aggregating study ${studyUuid} with ${groupingMinutes}min intervals`,
-);
-
-// After video fetch
-console.log(
-  `[VideoMinuteResultDAO] Found ${videos.length} COMPLETED videos with recording times`,
-);
-const videosWithoutTime = await this.knex("video as v")
-  .innerJoin("folders as f", "v.folderId", "f.id")
-  .where("f.studyId", study.id)
-  .where("v.status", "COMPLETED")
-  .whereNull("v.recordingStartedAt")
-  .count("* as count");
-if (+videosWithoutTime[0].count > 0) {
-  console.warn(
-    `[VideoMinuteResultDAO] Excluding ${videosWithoutTime[0].count} videos without recordingStartedAt`,
-  );
-}
-
-// After aggregation
-console.log(
-  `[VideoMinuteResultDAO] Generated ${aggregatedGroups.length} time buckets`,
-);
-```
-
----
-
-## Performance Considerations
-
-### Query Optimization
-
-1. **Single Query for Minute Results**: Fetch all data in one query with JOINs (eliminates N+1)
-2. **Index Usage**: Existing `idx_videos_folder_recording` covers video filtering
-3. **In-Memory Aggregation**: TypeScript grouping is faster than SQL for complex JSON aggregation
-4. **Early Returns**: Skip processing if no videos/results found
-
-### Memory Management
-
-- **Batch Size Estimate**:
-  - 10 videos × 60 minutes × 2KB/result = ~1.2MB
-  - 100 videos × 1440 minutes × 2KB/result = ~288MB
-  - Safe for typical studies (<50 videos)
-- **Future Optimization**: If studies exceed 100 videos, consider streaming or pagination
-
-### Database Load
-
-- **Single study query**: O(1)
-- **Single video query**: O(folders) - typically small
-- **Single minute results query**: O(total_minutes) - most expensive
-- **Total queries**: 3 (no N+1 problems)
-
----
-
-## Testing Strategy (Recommendations Only - Not Implemented)
-
-### Unit Test Cases
-
-1. **Basic Functionality**
-   - Single video, 1-minute grouping
-   - Multiple videos, 15-minute grouping
-   - TMC vs ATR aggregation
-
-2. **Time Normalization**
-   - Videos with different recordingStartedAt align correctly
-   - Verify bucket boundaries (e.g., 2:00 PM, 2:15 PM, 2:30 PM)
-   - Handle midnight boundary crossing
-
-3. **Edge Cases**
-   - Study not found (expect error)
-   - No COMPLETED videos (expect empty array)
-   - Mix of videos with/without recordingStartedAt
-   - Single minute spanning multiple buckets (shouldn't happen)
-
-4. **Aggregation Accuracy**
-   - Verify TMC turning movements sum correctly
-   - Verify ATR lane counts sum correctly
-   - Check contributingVideos array accuracy
-
-### Integration Test
-
-```typescript
-// Example test structure (not implemented)
-describe("getGroupedMinuteResultsByStudyUuid", () => {
-  it("should aggregate TMC results across multiple videos", async () => {
-    // Setup: Create study with 3 videos at different times
-    // Insert minute results with known values
-    // Call method with 15-minute grouping
-    // Assert: Verify bucket times, aggregated counts, video attribution
-  });
-});
-```
-
----
-
-## Migration Requirements
-
-**No database migrations needed** - all existing schema supports this feature:
-
-- `video.recordingStartedAt` exists (migration 20251020225758)
-- `video.status` exists
-- Indexes on folder/recording relationships exist
-- Study → Folder → Video relationships exist
-
----
-
-## Implementation Order
-
-1. **Add Interfaces** (lines 10-69 in VideoMinuteResultDAO.ts)
-   - `IStudyTimeGroupResult`
-   - `IGroupedStudyResponse`
-
-2. **Add Helper Method** (before existing private methods)
-   - `formatAbsoluteTimeLabel(startTime: Date, endTime: Date): string`
-
-3. **Add Main Method** (after `getGroupedMinuteResultsByVideoUuid`, around line 479)
-   - `getGroupedMinuteResultsByStudyUuid(studyUuid, groupingMinutes)`
-   - Follow 5-step logic outlined above
-
-4. **Export Interfaces** (at end of file)
-   - Export new interfaces for use in API layer
-
-5. **Update Package**
-   - `npm run build` to verify TypeScript compilation
-   - No DAO export changes needed (class already exported)
-
----
-
-## API Integration Notes (For Future Work)
-
-### Suggested API Endpoint
-
-```
-GET /api/studies/:studyUuid/minute-results?grouping=15
-```
-
-### Response Format
-
-```json
-{
-  "success": true,
-  "data": [
-    {
-      "absoluteTime": "2024-10-26T14:30:00.000Z",
-      "groupIndex": 0,
-      "label": "Oct 26, 2024: 2:30 PM - 2:44 PM",
-      "results": {
-        /* aggregated TMC/ATR data */
-      },
-      "minuteCount": 45,
-      "videoCount": 3,
-      "contributingVideos": ["uuid1", "uuid2", "uuid3"]
-    }
-  ],
-  "groupingMinutes": 15,
-  "study": {
-    "uuid": "study-uuid",
-    "name": "Downtown Intersection Study",
-    "type": "TMC",
-    "status": "COMPLETE"
-  },
-  "videoCount": 3,
-  "dateRange": {
-    "earliest": "2024-10-26T14:00:00.000Z",
-    "latest": "2024-10-26T16:30:00.000Z"
-  }
-}
-```
-
----
-
-## Risks & Mitigation
-
-| Risk                                | Impact                      | Mitigation                                           |
-| ----------------------------------- | --------------------------- | ---------------------------------------------------- |
-| Videos without recordingStartedAt   | Cannot normalize time       | Exclude from aggregation, log warning                |
-| Large studies (100+ videos)         | Memory/performance issues   | Document limit, add pagination in future             |
-| Type inconsistency (video vs study) | Wrong aggregation logic     | Use study.type as source of truth                    |
-| Missing duration_seconds            | Inaccurate dateRange.latest | Calculate from last minute_number                    |
-| Timezone confusion                  | Incorrect time labels       | Use UTC internally, format in user timezone (future) |
-| Concurrent video processing         | Incomplete aggregation      | Document that method shows COMPLETED videos only     |
-
----
-
-## Pattern Compliance Checklist
-
-- ✅ **DAO Pattern**: Method follows existing DAO patterns (async, returns typed response)
-- ✅ **Interface Naming**: `IGroupedStudyResponse`, `IStudyTimeGroupResult` (PascalCase with I prefix)
-- ✅ **UUID Usage**: Uses study.uuid for external API, study.id internally
-- ✅ **Performance**: Single query for minute results (eliminates N+1)
-- ✅ **Error Handling**: Throws errors for missing study, returns empty for no data
-- ✅ **Type Safety**: All interfaces properly typed, no `any` in return types
-- ✅ **Code Reuse**: Leverages existing aggregation methods
-- ✅ **Naming**: `getGroupedMinuteResultsByStudyUuid` follows existing conventions
-
----
-
-## Files Modified
-
-### knex-rel/src/dao/VideoMinuteResultDAO.ts
-
-**Changes**:
-
-- Add `IStudyTimeGroupResult` interface (after line 48)
-- Add `IGroupedStudyResponse` interface (after `IStudyTimeGroupResult`)
-- Add `formatAbsoluteTimeLabel()` private method (before line 703)
-- Add `getGroupedMinuteResultsByStudyUuid()` public method (after line 479)
-- Export new interfaces at end of file
-
-**Lines Added**: ~200 lines
-**Pattern Compliance**: 100%
-
----
-
-## Validation Checklist
-
-Before implementation:
-
-- [ ] Confirm study.type values match video.videoType enum ('TMC' | 'ATR')
-- [ ] Verify recordingStartedAt is stored in UTC (migration confirms TIMESTAMPTZ)
-- [ ] Check if duration_seconds is reliably populated
-- [ ] Review existing aggregation methods for edge cases
-
-After implementation:
-
-- [ ] Test with TMC study (multiple videos, 15-minute grouping)
-- [ ] Test with ATR study (multiple videos, 60-minute grouping)
-- [ ] Test with study containing no COMPLETED videos
-- [ ] Test with study where some videos lack recordingStartedAt
-- [ ] Verify time bucket boundaries align to grouping interval
-- [ ] Check memory usage with 50-video study
-- [ ] Run `npm run build` - confirm no TypeScript errors
-- [ ] Run `npm test` - confirm existing tests still pass
-
----
-
-## Summary
-
-This plan adds study-level aggregation with **zero database changes**, reusing 100% of existing schema and aggregation logic. Key innovation is SQL-based time normalization using `recordingStartedAt` to convert video-relative minute offsets to absolute wall-clock times, then grouping in TypeScript for flexibility.
-
-**Implementation Effort**: ~200 lines of TypeScript, ~3-4 hours
-**Risk Level**: Low (no schema changes, proven aggregation logic)
-**Performance**: Good (single query, in-memory aggregation, indexed lookups)