@xiboplayer/cache 0.6.3 → 0.6.5

package/README.md

@@ -1,22 +1,57 @@
 # @xiboplayer/cache
 
-**Offline caching and download management with durable filesystem storage.**
+**Offline caching and download management with parallel chunk downloads, resume support, and storage health monitoring.**
 
 ## Overview
 
-Manages media downloads and offline storage for Xibo players:
+Manages the complete lifecycle of media files from CMS to local storage:
 
-- **StoreClient** — pure REST client for reading/writing content in the ContentStore (filesystem via proxy)
-- **DownloadClient** — Service Worker postMessage client for managing background downloads
-- **Parallel chunk downloads** — large files (100MB+) split into configurable chunks, downloaded concurrently
-- **Header+trailer first** — MP4 moov atom fetched first for instant playback start before full download
-- **MD5 verification** — integrity checking with CRC32-based skip optimization
-- **Download queue** — flat queue with barriers for layout-ordered downloading
-- **CacheAnalyzer** — stale media detection and storage-pressure eviction
-- **Widget data via enriched RequiredFiles** — RSS/dataset widget data is fetched through server-side enriched RequiredFiles paths (CMS adds download URLs), not via client-side pre-fetching
-- **Dynamic BASE path** — widget HTML `<base>` tag uses a dynamic path within the Service Worker scope for correct relative URL resolution
+- **REST-based file store** -- StoreClient provides CRUD operations (has, get, put, remove, list) via proxy endpoints
+- **Flat-queue download orchestration** -- DownloadManager with per-file and global concurrency limits
+- **Intelligent chunking** -- files > 100MB split into 50MB chunks with prioritized headers and trailers for fast video start
+- **Download resume** -- partial downloads resume automatically; expired URLs defer to next collection cycle
+- **Stale media detection** -- CacheAnalyzer identifies orphaned files and evicts when storage exceeds threshold
+- **Widget HTML preprocessing** -- base tag injection, dependency rewriting for iframe sandboxing
 
-No Cache API is used anywhere. All content is stored on the filesystem via the proxy's ContentStore.
+No Cache API is used. All content is stored on the filesystem via the proxy's ContentStore.
+
+## Architecture
+
+```
+CMS (RequiredFiles: size, URL, MD5)
+        |
+        v
++-------------------------------------+
+|      DownloadManager (Facade)       |
+|  +- enqueue(fileInfo)               |
+|  +- prioritizeLayoutFiles(mediaIds) |
+|  +- urgentChunk(type, id, idx)      |
++-------------------------------------+
+             |
+             v
+      DownloadQueue (Flat)
+    +-----------------------------+
+    | [Task, Task, BARRIER,       |
+    |  Task, Task, Task]          |
+    |                             |
+    | Global concurrency: 6       |
+    | Per-file chunks: 2-3        |
+    +-----------------------------+
+       |
+       +-> DownloadTask (chunk-0, high priority)
+       +-> DownloadTask (chunk-last, high priority)
+       +-> BARRIER (gate: waits for above to finish)
+       +-> DownloadTask (bulk chunks, normal priority)
+               |
+               v
+         HTTP Range GET
+               |
+               v
+       ContentStore (proxy)
+         /store/media/{id}
+         /store/layout/{id}
+         /store/widget/{L}/{R}/{M}
+```
 
 ## Installation
 
@@ -26,35 +61,176 @@ npm install @xiboplayer/cache
 
 ## Usage
 
+### StoreClient -- direct REST access to ContentStore
+
 ```javascript
-import { StoreClient, DownloadClient } from '@xiboplayer/cache';
+import { StoreClient } from '@xiboplayer/cache';
 
-// Storage operations (pure REST, no SW needed)
 const store = new StoreClient();
-const { exists, size } = await store.has('media', '123');
-const blob = await store.get('media', '123');
-await store.put('widget', '472/221/190', htmlBlob, 'text/html');
 
-// Download management (SW postMessage)
-const downloads = new DownloadClient();
-await downloads.init();
-await downloads.download(files);
-const progress = await downloads.getProgress();
+// Check existence
+const exists = await store.has('media', '123');
+
+// Retrieve file
+const blob = await store.get('media', '456');
+
+// Store widget HTML
+const html = new Blob(['<h1>Widget</h1>'], { type: 'text/html' });
+await store.put('widget', 'layout/1/region/2/media/3', html, 'text/html');
+
+// List all cached files
+const files = await store.list();
+
+// Delete orphaned files
+await store.remove([
+  { type: 'media', id: '999' },
+  { type: 'media', id: '1000' },
+]);
+```
+
+### DownloadManager -- orchestrated downloads
+
+```javascript
+import { DownloadManager } from '@xiboplayer/cache';
+
+const dm = new DownloadManager({
+  concurrency: 6,
+  chunkSize: 50 * 1024 * 1024,
+  chunksPerFile: 2,
+});
+
+// Enqueue a file
+const media = dm.enqueue({
+  id: '123',
+  type: 'media',
+  path: 'https://cdn.example.com/video.mp4',
+  size: 250 * 1024 * 1024,
+  md5: 'abc123def456',
+});
+
+// Wait for completion
+const blob = await media.wait();
+
+// Get progress
+const progress = dm.getProgress();
+// { 'media/123': { downloaded: 50M, total: 250M, percent: '20.0', state: 'downloading' } }
+```
+
+### Prioritization and emergency chunks
+
+```javascript
+// Boost files needed by current layout
+dm.prioritizeLayoutFiles(['123', '456']);
+
+// Emergency: chunk needed for video playback is stalled
+// Moves to front of queue, reduces global concurrency to 2
+dm.urgentChunk('media', '789', 3);
+```
+
+### CacheAnalyzer -- storage health
+
+```javascript
+import { CacheAnalyzer } from '@xiboplayer/cache';
+
+const analyzer = new CacheAnalyzer(store, { threshold: 80 });
+
+const report = await analyzer.analyze(requiredFiles);
+console.log(`Storage: ${report.storage.percent}%`);
+console.log(`Orphaned: ${report.files.orphaned} files`);
+console.log(`Evicted: ${report.evicted.length} files`);
+```
+
+## Download Pipeline
+
+Files flow through stages:
+
+1. **Enqueueing** -- deduplication via `type/id` key, URL refresh if new URL has longer expiry
+2. **Preparation** -- HEAD request determines chunking (> 100MB = chunked)
+3. **Task creation** -- chunk-0 and chunk-last get high priority (video headers/trailers); BARRIER separates from bulk chunks
+4. **Processing** -- concurrency-aware: 6 global connections, 2-3 per file
+5. **Execution** -- HTTP Range GET with retries and exponential backoff
+6. **Storage** -- proxy ContentStore saves chunks to filesystem
+7. **Assembly** -- chunks concatenated; progressive callback enables playback before full download
+
+### Chunk strategy
+
+- **Default chunk size:** 50MB
+- **Threshold:** files > 100MB get chunked
+- **Header+trailer first:** MP4 moov atom in chunk-0 and chunk-last allows instant playback start
+- **Barriers:** critical chunks download before bulk chunks begin
+- **Resume:** cached chunks tracked in `skipChunks` Set; only missing chunks downloaded
+- **Expired URLs:** deferred (not failed) -- next collection provides fresh URL
+
+### Retry strategy by type
+
+| Type | Max retries | Backoff | Notes |
+|------|------------|---------|-------|
+| media | 3 | 500ms | Large, cacheable files |
+| layout | 3 | 500ms | Layout XML, stable URL |
+| dataset | 4 | 15s, 30s, 60s, 120s | Re-enqueues 5x for "cache not ready" |
+| static | 3 | 500ms | Widget dependencies (CSS, fonts) |
+
+## API Reference
+
+### StoreClient
+
+| Method | Signature | Returns | Description |
+|--------|-----------|---------|-------------|
+| `has()` | `has(type, id)` | `Promise<boolean>` | Check if file exists |
+| `get()` | `get(type, id)` | `Promise<Blob \| null>` | Retrieve file as Blob |
+| `put()` | `put(type, id, body, contentType?)` | `Promise<boolean>` | Store file |
+| `remove()` | `remove(files)` | `Promise<{deleted, total}>` | Delete files |
+| `list()` | `list()` | `Promise<Array>` | List all cached files |
+
+### DownloadManager
+
+```javascript
+new DownloadManager({ concurrency?, chunkSize?, chunksPerFile? })
+```
+
+| Method | Signature | Returns | Description |
+|--------|-----------|---------|-------------|
+| `enqueue()` | `enqueue(fileInfo)` | `FileDownload` | Add file to download queue |
+| `getTask()` | `getTask(key)` | `FileDownload \| null` | Get task by "type/id" key |
+| `getProgress()` | `getProgress()` | `Record<string, Progress>` | All in-progress downloads |
+| `prioritizeLayoutFiles()` | `prioritizeLayoutFiles(mediaIds)` | `void` | Boost priority for layout files |
+| `urgentChunk()` | `urgentChunk(type, id, chunkIndex)` | `boolean` | Move chunk to front, reduce concurrency |
+| `clear()` | `clear()` | `void` | Cancel all, clear queue |
+
+### CacheAnalyzer
+
+```javascript
+new CacheAnalyzer(storeClient, { threshold: 80 })
+```
+
+| Method | Signature | Returns | Description |
+|--------|-----------|---------|-------------|
+| `analyze()` | `analyze(requiredFiles)` | `Promise<Report>` | Compare cache vs required, evict if needed |
+
+**Report:**
+```javascript
+{
+  storage: { usage, quota, percent },
+  files: { required, orphaned, total },
+  orphaned: [{ id, type, size, cachedAt }],
+  evicted: [{ id, type, size }],
+  threshold: 80
+}
 ```
 
-## Exports
+### CacheManager
 
-| Export | Description |
-|--------|-------------|
-| `StoreClient` | Pure REST client for ContentStore — has/get/put/remove/list |
-| `DownloadClient` | SW postMessage client for background downloads |
-| `DownloadManager` | Core download queue with barrier-based ordering |
-| `CacheAnalyzer` | Stale media detection and eviction |
+| Method | Signature | Returns | Description |
+|--------|-----------|---------|-------------|
+| `getCacheKey()` | `getCacheKey(type, id)` | `string` | Get cache key path |
+| `addDependant()` | `addDependant(mediaId, layoutId)` | `void` | Track layout -> media reference |
+| `removeLayoutDependants()` | `removeLayoutDependants(layoutId)` | `string[]` | Remove layout, return orphaned media IDs |
+| `isMediaReferenced()` | `isMediaReferenced(mediaId)` | `boolean` | Check if media is still used |
 
 ## Dependencies
 
-- `@xiboplayer/utils` — logger, events
-- `spark-md5` — MD5 hashing for file verification
+- `@xiboplayer/utils` -- logger
+- `spark-md5` -- MD5 hashing for file verification
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xiboplayer/cache",
-  "version": "0.6.3",
+  "version": "0.6.5",
   "description": "Offline caching and download management with parallel chunk downloads",
   "type": "module",
   "main": "./src/index.js",
@@ -16,7 +16,7 @@
   },
   "dependencies": {
     "spark-md5": "^3.0.2",
-    "@xiboplayer/utils": "0.6.3"
+    "@xiboplayer/utils": "0.6.5"
   },
   "devDependencies": {
     "vitest": "^2.0.0",

package/src/widget-html.js

@@ -7,7 +7,7 @@
  * - CSS object-position fix for CMS template alignment
  *
  * URL rewriting is no longer needed — the CMS serves CSS with relative paths
- * (/api/v2/player/dependencies/font.otf), and the <base> tag resolves widget
+ * (${PLAYER_API}/dependencies/font.otf), and the <base> tag resolves widget
  * media references via mirror routes. Zero translation, zero regex.
  *
  * Runs on the main thread (needs window.location for URL construction).
@@ -25,7 +25,7 @@ const BASE = (typeof window !== 'undefined')
 
 /**
  * Store widget HTML in ContentStore for iframe loading.
- * Stored at mirror path /api/v2/player/widgets/{L}/{R}/{M} — same URL the
+ * Stored at mirror path ${PLAYER_API}/widgets/{L}/{R}/{M} — same URL the
  * CMS serves from, so iframes load directly from Express mirror routes.
  *
  * @param {string} layoutId - Layout ID
@@ -63,12 +63,13 @@ export async function cacheWidgetHtml(layoutId, regionId, mediaId, html) {
   }
 
   // Rewrite dependency URLs to local mirror paths. CMS sends absolute URLs
-  // like https://cms.example.com/api/v2/player/dependencies/bundle.min.js
-  // which fail due to CORS/auth. Replace with local /api/v2/player/dependencies/...
-  modifiedHtml = modifiedHtml.replace(
-    /https?:\/\/[^"'\s)]+?(\/api\/v2\/player\/dependencies\/[^"'\s?)]+)(\?[^"'\s)]*)?/g,
-    (_, path) => path
+  // like https://cms.example.com${PLAYER_API}/dependencies/bundle.min.js
+  // which fail due to CORS/auth. Replace with local PLAYER_API/dependencies/...
+  const depsPattern = new RegExp(
+    `https?://[^"'\\s)]+?(${PLAYER_API.replace(/\//g, '\\/')}/dependencies/[^"'\\s?)]+)(\\?[^"'\\s)]*)?`,
+    'g'
   );
+  modifiedHtml = modifiedHtml.replace(depsPattern, (_, path) => path);
 
   // Inject xiboICTargetId — XIC library reads this global before its IIFE runs
   // to set _lib.targetId, which is included in every IC HTTP request as {id: ...}

package/src/widget-html.test.js

@@ -8,9 +8,9 @@
 
 import { describe, it, expect, beforeEach, vi } from 'vitest';
 import { cacheWidgetHtml } from './widget-html.js';
+import { PLAYER_API } from '@xiboplayer/utils';
 
 const CMS_BASE = 'https://displays.superpantalles.com';
-const SIGNED_PARAMS = 'displayId=152&type=P&itemId=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260222T000000Z&X-Amz-Expires=1771803983&X-Amz-SignedHeaders=host&X-Amz-Signature=abc123';
 
 /**
  * RSS ticker widget HTML (layout 472, region 223, widget 193).
@@ -22,8 +22,8 @@ function makeRssTickerHtml() {
 <head>
 <meta charset="utf-8">
 <title>RSS Ticker</title>
-<script src="/api/v2/player/dependencies/bundle.min.js"></script>
-<link rel="stylesheet" href="/api/v2/player/dependencies/fonts.css">
+<script src="${PLAYER_API}/dependencies/bundle.min.js"></script>
+<link rel="stylesheet" href="${PLAYER_API}/dependencies/fonts.css">
 <style>.rss-item { padding: 10px; }</style>
 </head>
 <body>
@@ -47,8 +47,8 @@ function makePdfWidgetHtml() {
 <html>
 <head>
 <meta charset="utf-8">
-<script src="/api/v2/player/dependencies/bundle.min.js"></script>
-<link rel="stylesheet" href="/api/v2/player/dependencies/fonts.css"></link>
+<script src="${PLAYER_API}/dependencies/bundle.min.js"></script>
+<link rel="stylesheet" href="${PLAYER_API}/dependencies/fonts.css"></link>
 </head>
 <body>
 <object data="11.pdf" type="application/pdf" width="100%" height="100%"></object>
@@ -86,9 +86,11 @@ describe('cacheWidgetHtml', () => {
     global.fetch = createFetchMock();
   });
 
+  const storePrefix = `/store${PLAYER_API}/widgets/`;
+
   function getStoredWidget() {
     for (const [key, value] of storeContents) {
-      if (key.startsWith('/store/api/v2/player/widgets/')) return value;
+      if (key.startsWith(storePrefix)) return value;
     }
     return undefined;
   }
@@ -101,7 +103,7 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '223', '193', html);
 
       const stored = getStoredWidget();
-      expect(stored).toContain('<base href="/api/v2/player/media/file/">');
+      expect(stored).toContain(`<base href="${PLAYER_API}/media/file/">`);
     });
 
     it('injects <base> tag when no <head> tag exists', async () => {
@@ -109,7 +111,7 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '223', '193', html);
 
       const stored = getStoredWidget();
-      expect(stored).toContain('<base href="/api/v2/player/media/file/">');
+      expect(stored).toContain(`<base href="${PLAYER_API}/media/file/">`);
     });
   });
 
@@ -120,8 +122,8 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '223', '193', makeRssTickerHtml());
 
       const stored = getStoredWidget();
-      expect(stored).toContain('/api/v2/player/dependencies/bundle.min.js');
-      expect(stored).toContain('/api/v2/player/dependencies/fonts.css');
+      expect(stored).toContain(`${PLAYER_API}/dependencies/bundle.min.js`);
+      expect(stored).toContain(`${PLAYER_API}/dependencies/fonts.css`);
     });
 
     it('preserves the data URL (193.json) for resolution via base tag', async () => {
@@ -143,7 +145,7 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '223', '193', makeRssTickerHtml());
 
       // Only the widget HTML PUT should be fetched — no proactive resource fetches
-      const putCalls = fetchedUrls.filter(u => u.startsWith('/store/api/v2/player/widgets/'));
+      const putCalls = fetchedUrls.filter(u => u.startsWith(storePrefix));
       expect(putCalls.length).toBe(1);
     });
 
@@ -151,7 +153,7 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '223', '193', makeRssTickerHtml());
 
       const keys = [...storeContents.keys()];
-      expect(keys.some(k => k.includes('/store/api/v2/player/widgets/472/223/193'))).toBe(true);
+      expect(keys.some(k => k.includes(`${storePrefix}472/223/193`))).toBe(true);
     });
   });
 
@@ -169,7 +171,7 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '221', '190', makePdfWidgetHtml());
 
       const keys = [...storeContents.keys()];
-      expect(keys.some(k => k.includes('/store/api/v2/player/widgets/472/221/190'))).toBe(true);
+      expect(keys.some(k => k.includes(`${storePrefix}472/221/190`))).toBe(true);
     });
   });
 
@@ -257,8 +259,8 @@ describe('cacheWidgetHtml', () => {
       await cacheWidgetHtml('472', '223', '193', makeRssTickerHtml());
 
       const keys = [...storeContents.keys()];
-      expect(keys.some(k => k.includes('/store/api/v2/player/widgets/472/221/190'))).toBe(true);
-      expect(keys.some(k => k.includes('/store/api/v2/player/widgets/472/223/193'))).toBe(true);
+      expect(keys.some(k => k.includes(`${storePrefix}472/221/190`))).toBe(true);
+      expect(keys.some(k => k.includes(`${storePrefix}472/223/193`))).toBe(true);
     });
   });
 });