@djangocfg/ui-nextjs 2.1.81 → 2.1.83

package/src/tools/AudioPlayer/@refactoring3/01-WAVESURFER-STREAMING-ANALYSIS.md

@@ -0,0 +1,611 @@
+# WaveSurfer.js Streaming and Chunked Audio Analysis
+
+> **Analysis Date:** 2025-12-30
+> **WaveSurfer Version:** 7.x (based on source code analysis)
+> **Source Location:** `@sources/wavesurfer.js-main/`
+
+---
+
+## Table of Contents
+
+1. [Executive Summary](#executive-summary)
+2. [How WaveSurfer Loads Audio](#how-wavesurfer-loads-audio)
+3. [Streaming Support Analysis](#streaming-support-analysis)
+4. [Fetch and Progress Tracking](#fetch-and-progress-tracking)
+5. [Working with Partially Loaded Audio](#working-with-partially-loaded-audio)
+6. [Pre-decoded Peaks Pattern](#pre-decoded-peaks-pattern)
+7. [MediaStream Support (Recording)](#mediastream-support-recording)
+8. [Limitations and Workarounds](#limitations-and-workarounds)
+9. [Recommendations for Streaming Audio](#recommendations-for-streaming-audio)
+10. [Code Snippets Reference](#code-snippets-reference)
+
+---
+
+## Executive Summary
+
+**Key Finding:** WaveSurfer.js does NOT natively support true audio streaming (chunked/progressive loading with immediate playback). It requires the **entire audio file** to be downloaded and decoded before rendering the waveform.
+
+### Critical Limitations:
+
+| Feature | Support | Notes |
+|---------|---------|-------|
+| True streaming (MediaSource API) | **NO** | Not implemented |
+| Chunked fetch with progressive rendering | **NO** | Full download required |
+| Partial audio playback | **LIMITED** | Only with pre-decoded peaks |
+| Pre-computed peaks + streaming URL | **YES** | Recommended approach |
+| Real-time microphone input | **YES** | Via Record plugin |
+| Large file support | **WORKAROUND** | Use pre-decoded peaks |
+
+---
+
+## How WaveSurfer Loads Audio
+
+### Main Loading Flow
+
+The loading process in WaveSurfer follows this sequence:
+
+```
+load(url) -> Fetcher.fetchBlob() -> Full Download -> audioContext.decodeAudioData() -> Renderer.render()
+```
+
+From `src/wavesurfer.ts` (lines 502-566):
+
+```typescript
+private async loadAudio(url: string, blob?: Blob, channelData?: WaveSurferOptions['peaks'], duration?: number) {
+  this.emit('load', url)
+
+  if (!this.options.media && this.isPlaying()) this.pause()
+
+  this.decodedData = null
+  this.stopAtPosition = null
+
+  // Abort any ongoing fetch before starting a new one
+  this.abortController?.abort()
+  this.abortController = null
+
+  // Fetch the entire audio as a blob if pre-decoded data is not provided
+  if (!blob && !channelData) {
+    const fetchParams = this.options.fetchParams || {}
+    if (window.AbortController && !fetchParams.signal) {
+      this.abortController = new AbortController()
+      fetchParams.signal = this.abortController.signal
+    }
+    const onProgress = (percentage: number) => this.emit('loading', percentage)
+    blob = await Fetcher.fetchBlob(url, onProgress, fetchParams)  // <-- FULL DOWNLOAD
+    const overridenMimeType = this.options.blobMimeType
+    if (overridenMimeType) {
+      blob = new Blob([blob], { type: overridenMimeType })
+    }
+  }
+
+  // Set the mediaelement source
+  this.setSrc(url, blob)
+
+  // Wait for the audio duration
+  const audioDuration = await new Promise<number>((resolve) => {
+    const staticDuration = duration || this.getDuration()
+    if (staticDuration) {
+      resolve(staticDuration)
+    } else {
+      this.mediaSubscriptions.push(
+        this.onMediaEvent('loadedmetadata', () => resolve(this.getDuration()), { once: true }),
+      )
+    }
+  })
+
+  // Decode the audio data or use user-provided peaks
+  if (channelData) {
+    this.decodedData = Decoder.createBuffer(channelData, audioDuration || 0)
+  } else if (blob) {
+    const arrayBuffer = await blob.arrayBuffer()
+    this.decodedData = await Decoder.decode(arrayBuffer, this.options.sampleRate)  // <-- FULL DECODE
+  }
+
+  if (this.decodedData) {
+    this.emit('decode', this.getDuration())
+    this.renderer.render(this.decodedData)
+  }
+
+  this.emit('ready', this.getDuration())
+}
+```
+
+### Key Observations:
+
+1. **Full Download Required:** `Fetcher.fetchBlob()` downloads the entire file
+2. **Full Decode Required:** `Decoder.decode()` decodes the entire ArrayBuffer
+3. **No Progressive Rendering:** Waveform only renders after complete decode
+4. **Memory Intensive:** Large files may fail due to memory constraints
+
+---
+
+## Streaming Support Analysis
+
+### Official Statement from README.md (lines 118-121):
+
+```markdown
+<details>
+  <summary>What about streaming audio?</summary>
+  Streaming audio is supported only with <a href="https://wavesurfer.xyz/examples/?predecoded.js">pre-decoded peaks and duration</a>.
+</details>
+```
+
+### What "Streaming Support" Actually Means:
+
+WaveSurfer's "streaming support" is **NOT true streaming**. It means:
+
+1. You provide pre-computed peaks data
+2. You provide the known duration
+3. The actual audio URL can be a streaming source
+4. WaveSurfer renders the waveform immediately from peaks
+5. Audio playback happens via HTMLMediaElement (which supports streaming natively)
+
+**This is NOT:**
+- Progressive waveform rendering as audio loads
+- Chunked audio decoding
+- MediaSource API integration
+- Real-time waveform updates during playback of streaming content
+
+---
+
+## Fetch and Progress Tracking
+
+### Fetcher Implementation (`src/fetcher.ts`):
+
+```typescript
+async function watchProgress(response: Response, progressCallback: (percentage: number) => void) {
+  if (!response.body || !response.headers) return
+  const reader = response.body.getReader()
+
+  const contentLength = Number(response.headers.get('Content-Length')) || 0
+  let receivedLength = 0
+
+  // Process the data
+  const processChunk = (value: Uint8Array | undefined) => {
+    // Add to the received length
+    receivedLength += value?.length || 0
+    const percentage = Math.round((receivedLength / contentLength) * 100)
+    progressCallback(percentage)
+  }
+
+  // Use iteration instead of recursion to avoid stack issues
+  try {
+    while (true) {
+      const data = await reader.read()
+
+      if (data.done) {
+        break
+      }
+
+      processChunk(data.value)
+    }
+  } catch (err) {
+    // Ignore errors because we can only handle the main response
+    console.warn('Progress tracking error:', err)
+  }
+}
+
+async function fetchBlob(
+  url: string,
+  progressCallback: (percentage: number) => void,
+  requestInit?: RequestInit,
+): Promise<Blob> {
+  // Fetch the resource
+  const response = await fetch(url, requestInit)
+
+  if (response.status >= 400) {
+    throw new Error(`Failed to fetch ${url}: ${response.status} (${response.statusText})`)
+  }
+
+  // Read the data to track progress
+  watchProgress(response.clone(), progressCallback)
+
+  return response.blob()  // <-- Returns complete blob
+}
+```
+
+### Key Limitations in Fetcher:
+
+1. **Chunks are read but NOT processed** - only for progress percentage
+2. **Returns complete blob** - `response.blob()` waits for full download
+3. **No progressive decoding** - chunks are discarded after progress tracking
+4. **Response cloning** - original response goes to blob(), clone is for progress
+
+### Progress Events:
+
+The `loading` event provides download progress but NOT decode/render progress:
+
+```typescript
+// Usage
+wavesurfer.on('loading', (percent) => {
+  console.log(`Downloaded: ${percent}%`)
+})
+
+// This fires during download, NOT during decode
+```
+
+---
+
+## Working with Partially Loaded Audio
+
+### Short Answer: Not Directly Supported
+
+WaveSurfer cannot render or play partially downloaded audio. However, there are workarounds:
+
+### Pattern 1: Pre-decoded Peaks (Recommended)
+
+```typescript
+const wavesurfer = WaveSurfer.create({
+  container: '#waveform',
+  url: '/audio/long-file.mp3',  // Can be streaming URL
+  peaks: preComputedPeaksArray,  // Pre-generated peaks
+  duration: 3600,  // Known duration in seconds
+})
+```
+
+From `examples/predecoded.js`:
+
+```javascript
+const wavesurfer = WaveSurfer.create({
+  container: document.body,
+  waveColor: 'rgb(200, 0, 200)',
+  progressColor: 'rgb(100, 0, 100)',
+  barWidth: 10,
+  barRadius: 10,
+  barGap: 2,
+  url: '/examples/audio/demo.wav',
+  peaks: [
+    [
+      0, 0.0023595101665705442, 0.012107174843549728, 0.005919494666159153,
+      -0.31324470043182373, 0.1511787623167038, 0.2473851442337036,
+      // ... more peak values
+    ],
+  ],
+  duration: 22,
+})
+```
+
+### Pattern 2: WebAudio Shim (For Precise Timing)
+
+From `examples/webaudio-shim.js`:
+
+```javascript
+import WaveSurfer from 'wavesurfer.js'
+import WebAudioPlayer from 'wavesurfer.js/dist/webaudio.js'
+
+const webAudioPlayer = new WebAudioPlayer()
+webAudioPlayer.src = '/examples/audio/audio.wav'
+
+webAudioPlayer.addEventListener('loadedmetadata', () => {
+  const wavesurfer = WaveSurfer.create({
+    container: document.body,
+    media: webAudioPlayer,
+    peaks: webAudioPlayer.getChannelData(),
+    duration: webAudioPlayer.duration,
+  })
+})
+```
+
+### Pattern 3: Dynamic Peaks Update
+
+You can call `setOptions()` with new peaks as they become available:
+
+```typescript
+// Initial load with placeholder
+const wavesurfer = WaveSurfer.create({
+  container: '#waveform',
+  url: streamingUrl,
+  peaks: [[0, 0, 0]],  // Minimal placeholder
+  duration: estimatedDuration,
+})
+
+// Later, update with real peaks
+wavesurfer.setOptions({
+  peaks: actualPeaksData,
+  duration: actualDuration,
+})
+```
+
+---
+
+## Pre-decoded Peaks Pattern
+
+### How It Works:
+
+1. **Server-side:** Generate peaks using tools like `audiowaveform`
+2. **Client-side:** Pass peaks and duration to WaveSurfer
+3. **Rendering:** Waveform renders immediately without downloading audio
+4. **Playback:** HTMLMediaElement handles streaming playback
+
+### Decoder.createBuffer (`src/decoder.ts`):
+
+```typescript
+/** Create an audio buffer from pre-decoded audio data */
+function createBuffer(channelData: Array<Float32Array | number[]>, duration: number): AudioBuffer {
+  // Validate inputs
+  if (!channelData || channelData.length === 0) {
+    throw new Error('channelData must be a non-empty array')
+  }
+  if (duration <= 0) {
+    throw new Error('duration must be greater than 0')
+  }
+
+  // If a single array of numbers is passed, make it an array of arrays
+  if (typeof channelData[0] === 'number') channelData = [channelData as unknown as number[]]
+
+  // Normalize to -1..1
+  normalize(channelData)
+
+  // Convert to Float32Array for consistency
+  const float32Channels = channelData.map((channel) =>
+    channel instanceof Float32Array ? channel : Float32Array.from(channel),
+  )
+
+  return {
+    duration,
+    length: float32Channels[0].length,
+    sampleRate: float32Channels[0].length / duration,
+    numberOfChannels: float32Channels.length,
+    getChannelData: (i: number) => {
+      const channel = float32Channels[i]
+      if (!channel) {
+        throw new Error(`Channel ${i} not found`)
+      }
+      return channel
+    },
+    copyFromChannel: AudioBuffer.prototype.copyFromChannel,
+    copyToChannel: AudioBuffer.prototype.copyToChannel,
+  } as AudioBuffer
+}
+```
+
+### Peak Generation Tools:
+
+- **audiowaveform** (recommended): https://github.com/bbc/audiowaveform
+- **ffmpeg** with audio filter
+- Custom server-side processing
+
+---
+
+## MediaStream Support (Recording)
+
+### Real-time Waveform During Recording
+
+The **Record Plugin** (`src/plugins/record.ts`) provides real-time waveform rendering from microphone input:
+
+```typescript
+public renderMicStream(stream: MediaStream): MicStream {
+  const audioContext = new AudioContext()
+  const source = audioContext.createMediaStreamSource(stream)
+  const analyser = audioContext.createAnalyser()
+  source.connect(analyser)
+
+  // Use smaller FFT size for more responsive peak detection
+  if (this.options.continuousWaveform || this.options.scrollingWaveform) {
+    analyser.fftSize = 32
+  }
+  const bufferLength = analyser.frequencyBinCount
+  const dataArray = new Float32Array(bufferLength)
+
+  // ... drawing logic at 100 FPS ...
+
+  const drawWaveform = () => {
+    if (this.isWaveformPaused) return
+
+    analyser.getFloatTimeDomainData(dataArray)
+
+    if (this.options.scrollingWaveform) {
+      // Scrolling waveform - use peak values
+      // ... accumulate peaks ...
+    } else if (this.options.continuousWaveform) {
+      // Continuous waveform
+      // ... accumulate all data ...
+    }
+
+    // Render using wavesurfer.load() with peaks
+    if (this.wavesurfer) {
+      this.wavesurfer
+        .load('', [this.dataWindow], totalDuration)
+        .then(() => { /* ... */ })
+    }
+  }
+
+  const intervalId = setInterval(drawWaveform, 1000 / FPS)  // 100 FPS
+  // ...
+}
+```
+
+### Key Features:
+
+- **Scrolling Waveform:** Fixed window showing last N seconds
+- **Continuous Waveform:** Growing waveform as recording progresses
+- **100 FPS update rate** for smooth visualization
+- Uses `load('', [peaks], duration)` pattern internally
+
+---
+
+## Limitations and Workarounds
+
+### Memory Constraints for Large Files
+
+From README.md (lines 114-116):
+
+```markdown
+<details>
+  <summary>Does wavesurfer support large files?</summary>
+  Since wavesurfer decodes audio entirely in the browser using Web Audio, large clips may fail to decode due to memory constraints. We recommend using pre-decoded peaks for large files.
+</details>
+```
+
+### VBR Audio Mismatch
+
+Variable bit rate files can cause waveform/audio sync issues:
+
+```markdown
+<details>
+  <summary>There is a mismatch between my audio and the waveform. How do I fix it?</summary>
+  If you're using a VBR (variable bit rate) audio file, there might be a mismatch between the audio and the waveform. This can be fixed by converting your file to CBR (constant bit rate).
+</details>
+```
+
+### No MediaSource API Integration
+
+WaveSurfer does not use MediaSource Extensions (MSE) for:
+- Progressive audio loading
+- Adaptive bitrate streaming
+- HLS/DASH support
+
+---
+
+## Recommendations for Streaming Audio
+
+### Recommended Architecture for Streaming:
+
+```
+                                   +------------------+
+                                   |  Audio Server    |
+                                   |  (HLS/DASH/MP3)  |
+                                   +--------+---------+
+                                            |
+           +--------------------------------+--------------------------------+
+           |                                                                 |
+           v                                                                 v
++----------+----------+                                          +-----------+-----------+
+|  Pre-computed Peaks |                                          |  Audio Streaming URL  |
+|  (JSON/Binary)      |                                          |  (direct or chunked)  |
++----------+----------+                                          +-----------+-----------+
+           |                                                                 |
+           v                                                                 v
++----------+----------+                                          +-----------+-----------+
+|  WaveSurfer         |                                          |  HTMLMediaElement     |
+|  (waveform render)  |<---------------------------------------->|  (audio playback)     |
++---------------------+                                          +-----------------------+
+```
+
+### Implementation Pattern:
+
+```typescript
+// 1. Fetch peaks separately (fast, small payload)
+const peaksResponse = await fetch('/api/audio/peaks/123')
+const { peaks, duration } = await peaksResponse.json()
+
+// 2. Create WaveSurfer with streaming URL
+const wavesurfer = WaveSurfer.create({
+  container: '#waveform',
+  url: 'https://streaming.example.com/audio/123.mp3',  // Can be HLS, etc.
+  peaks: peaks,
+  duration: duration,
+  backend: 'MediaElement',  // Use HTMLMediaElement for streaming
+})
+
+// 3. Audio streams progressively via HTMLMediaElement
+// Waveform is instantly rendered from pre-computed peaks
+```
+
+### For Progressive Loading Without Pre-computed Peaks:
+
+If you cannot pre-compute peaks, consider:
+
+1. **Chunked Peak Generation:** Generate peaks in chunks on server, send progressively
+2. **Placeholder Waveform:** Show loading indicator, then render when ready
+3. **Estimated Waveform:** Use audio analysis on first few seconds, extrapolate
+4. **Hybrid Approach:** Quick low-resolution peaks first, high-res after full load
+
+---
+
+## Code Snippets Reference
+
+### WaveSurfer Options Interface (relevant streaming options):
+
+```typescript
+export type WaveSurferOptions = {
+  // ... other options ...
+
+  /** Audio URL */
+  url?: string
+
+  /** Pre-computed audio data, arrays of floats for each channel */
+  peaks?: Array<Float32Array | number[]>
+
+  /** Pre-computed audio duration in seconds */
+  duration?: number
+
+  /** Use an existing media element instead of creating one */
+  media?: HTMLMediaElement
+
+  /** Options to pass to the fetch method */
+  fetchParams?: RequestInit
+
+  /** Playback "backend" to use, defaults to MediaElement */
+  backend?: 'WebAudio' | 'MediaElement'
+
+  /** Override the Blob MIME type */
+  blobMimeType?: string
+}
+```
+
+### Events for Monitoring Load Progress:
+
+```typescript
+// During download
+wavesurfer.on('load', (url) => {
+  console.log('Started loading:', url)
+})
+
+wavesurfer.on('loading', (percent) => {
+  console.log(`Download progress: ${percent}%`)
+})
+
+// After decode
+wavesurfer.on('decode', (duration) => {
+  console.log('Audio decoded, duration:', duration)
+})
+
+// Ready for playback
+wavesurfer.on('ready', (duration) => {
+  console.log('Ready to play, duration:', duration)
+})
+```
+
+### WebAudioPlayer (Custom Backend):
+
+```typescript
+import WebAudioPlayer from 'wavesurfer.js/dist/webaudio.js'
+
+// WebAudioPlayer decodes entire file but provides precise timing
+const player = new WebAudioPlayer()
+player.src = '/audio.mp3'
+
+player.addEventListener('loadedmetadata', () => {
+  // Access decoded channel data
+  const channelData = player.getChannelData()  // Float32Array[]
+  const duration = player.duration
+})
+```
+
+---
+
+## Conclusion
+
+WaveSurfer.js is designed for **pre-loaded audio visualization**, not streaming. For streaming audio applications:
+
+1. **Always use pre-decoded peaks** for large or streaming audio
+2. **Let HTMLMediaElement handle streaming** playback
+3. **Generate peaks server-side** using audiowaveform or similar
+4. **Consider chunked peak delivery** for very long files
+
+The library's architecture fundamentally requires full audio data for waveform rendering, making true progressive/streaming waveform visualization impossible without the pre-decoded peaks pattern.
+
+---
+
+## Related Files
+
+- `/src/wavesurfer.ts` - Main WaveSurfer class
+- `/src/fetcher.ts` - Fetch with progress tracking
+- `/src/decoder.ts` - Audio decoding and buffer creation
+- `/src/webaudio.ts` - WebAudio backend
+- `/src/player.ts` - Base player class
+- `/src/renderer.ts` - Waveform rendering
+- `/src/plugins/record.ts` - Real-time microphone recording
+- `/examples/predecoded.js` - Pre-decoded peaks example
+- `/examples/webaudio-shim.js` - WebAudio backend example