@djangocfg/ui-nextjs 2.1.83 → 2.1.85

package/src/tools/AudioPlayer/@refactoring3/01-WAVESURFER-STREAMING-ANALYSIS.md

@@ -1,611 +0,0 @@
-# WaveSurfer.js Streaming and Chunked Audio Analysis
-
-> **Analysis Date:** 2025-12-30
-> **WaveSurfer Version:** 7.x (based on source code analysis)
-> **Source Location:** `@sources/wavesurfer.js-main/`
-
----
-
-## Table of Contents
-
-1. [Executive Summary](#executive-summary)
-2. [How WaveSurfer Loads Audio](#how-wavesurfer-loads-audio)
-3. [Streaming Support Analysis](#streaming-support-analysis)
-4. [Fetch and Progress Tracking](#fetch-and-progress-tracking)
-5. [Working with Partially Loaded Audio](#working-with-partially-loaded-audio)
-6. [Pre-decoded Peaks Pattern](#pre-decoded-peaks-pattern)
-7. [MediaStream Support (Recording)](#mediastream-support-recording)
-8. [Limitations and Workarounds](#limitations-and-workarounds)
-9. [Recommendations for Streaming Audio](#recommendations-for-streaming-audio)
-10. [Code Snippets Reference](#code-snippets-reference)
-
----
-
-## Executive Summary
-
-**Key Finding:** WaveSurfer.js does NOT natively support true audio streaming (chunked/progressive loading with immediate playback). It requires the **entire audio file** to be downloaded and decoded before rendering the waveform.
-
-### Critical Limitations:
-
-| Feature | Support | Notes |
-|---------|---------|-------|
-| True streaming (MediaSource API) | **NO** | Not implemented |
-| Chunked fetch with progressive rendering | **NO** | Full download required |
-| Partial audio playback | **LIMITED** | Only with pre-decoded peaks |
-| Pre-computed peaks + streaming URL | **YES** | Recommended approach |
-| Real-time microphone input | **YES** | Via Record plugin |
-| Large file support | **WORKAROUND** | Use pre-decoded peaks |
-
----
-
-## How WaveSurfer Loads Audio
-
-### Main Loading Flow
-
-The loading process in WaveSurfer follows this sequence:
-
-```
-load(url) -> Fetcher.fetchBlob() -> Full Download -> audioContext.decodeAudioData() -> Renderer.render()
-```
-
-From `src/wavesurfer.ts` (lines 502-566):
-
-```typescript
-private async loadAudio(url: string, blob?: Blob, channelData?: WaveSurferOptions['peaks'], duration?: number) {
-  this.emit('load', url)
-
-  if (!this.options.media && this.isPlaying()) this.pause()
-
-  this.decodedData = null
-  this.stopAtPosition = null
-
-  // Abort any ongoing fetch before starting a new one
-  this.abortController?.abort()
-  this.abortController = null
-
-  // Fetch the entire audio as a blob if pre-decoded data is not provided
-  if (!blob && !channelData) {
-    const fetchParams = this.options.fetchParams || {}
-    if (window.AbortController && !fetchParams.signal) {
-      this.abortController = new AbortController()
-      fetchParams.signal = this.abortController.signal
-    }
-    const onProgress = (percentage: number) => this.emit('loading', percentage)
-    blob = await Fetcher.fetchBlob(url, onProgress, fetchParams)  // <-- FULL DOWNLOAD
-    const overridenMimeType = this.options.blobMimeType
-    if (overridenMimeType) {
-      blob = new Blob([blob], { type: overridenMimeType })
-    }
-  }
-
-  // Set the mediaelement source
-  this.setSrc(url, blob)
-
-  // Wait for the audio duration
-  const audioDuration = await new Promise<number>((resolve) => {
-    const staticDuration = duration || this.getDuration()
-    if (staticDuration) {
-      resolve(staticDuration)
-    } else {
-      this.mediaSubscriptions.push(
-        this.onMediaEvent('loadedmetadata', () => resolve(this.getDuration()), { once: true }),
-      )
-    }
-  })
-
-  // Decode the audio data or use user-provided peaks
-  if (channelData) {
-    this.decodedData = Decoder.createBuffer(channelData, audioDuration || 0)
-  } else if (blob) {
-    const arrayBuffer = await blob.arrayBuffer()
-    this.decodedData = await Decoder.decode(arrayBuffer, this.options.sampleRate)  // <-- FULL DECODE
-  }
-
-  if (this.decodedData) {
-    this.emit('decode', this.getDuration())
-    this.renderer.render(this.decodedData)
-  }
-
-  this.emit('ready', this.getDuration())
-}
-```
-
-### Key Observations:
-
-1. **Full Download Required:** `Fetcher.fetchBlob()` downloads the entire file
-2. **Full Decode Required:** `Decoder.decode()` decodes the entire ArrayBuffer
-3. **No Progressive Rendering:** Waveform only renders after complete decode
-4. **Memory Intensive:** Large files may fail due to memory constraints
-
----
-
-## Streaming Support Analysis
-
-### Official Statement from README.md (lines 118-121):
-
-```markdown
-<details>
-  <summary>What about streaming audio?</summary>
-  Streaming audio is supported only with <a href="https://wavesurfer.xyz/examples/?predecoded.js">pre-decoded peaks and duration</a>.
-</details>
-```
-
-### What "Streaming Support" Actually Means:
-
-WaveSurfer's "streaming support" is **NOT true streaming**. It means:
-
-1. You provide pre-computed peaks data
-2. You provide the known duration
-3. The actual audio URL can be a streaming source
-4. WaveSurfer renders the waveform immediately from peaks
-5. Audio playback happens via HTMLMediaElement (which supports streaming natively)
-
-**This is NOT:**
-- Progressive waveform rendering as audio loads
-- Chunked audio decoding
-- MediaSource API integration
-- Real-time waveform updates during playback of streaming content
-
----
-
-## Fetch and Progress Tracking
-
-### Fetcher Implementation (`src/fetcher.ts`):
-
-```typescript
-async function watchProgress(response: Response, progressCallback: (percentage: number) => void) {
-  if (!response.body || !response.headers) return
-  const reader = response.body.getReader()
-
-  const contentLength = Number(response.headers.get('Content-Length')) || 0
-  let receivedLength = 0
-
-  // Process the data
-  const processChunk = (value: Uint8Array | undefined) => {
-    // Add to the received length
-    receivedLength += value?.length || 0
-    const percentage = Math.round((receivedLength / contentLength) * 100)
-    progressCallback(percentage)
-  }
-
-  // Use iteration instead of recursion to avoid stack issues
-  try {
-    while (true) {
-      const data = await reader.read()
-
-      if (data.done) {
-        break
-      }
-
-      processChunk(data.value)
-    }
-  } catch (err) {
-    // Ignore errors because we can only handle the main response
-    console.warn('Progress tracking error:', err)
-  }
-}
-
-async function fetchBlob(
-  url: string,
-  progressCallback: (percentage: number) => void,
-  requestInit?: RequestInit,
-): Promise<Blob> {
-  // Fetch the resource
-  const response = await fetch(url, requestInit)
-
-  if (response.status >= 400) {
-    throw new Error(`Failed to fetch ${url}: ${response.status} (${response.statusText})`)
-  }
-
-  // Read the data to track progress
-  watchProgress(response.clone(), progressCallback)
-
-  return response.blob()  // <-- Returns complete blob
-}
-```
-
-### Key Limitations in Fetcher:
-
-1. **Chunks are read but NOT processed** - only for progress percentage
-2. **Returns complete blob** - `response.blob()` waits for full download
-3. **No progressive decoding** - chunks are discarded after progress tracking
-4. **Response cloning** - original response goes to blob(), clone is for progress
-
-### Progress Events:
-
-The `loading` event provides download progress but NOT decode/render progress:
-
-```typescript
-// Usage
-wavesurfer.on('loading', (percent) => {
-  console.log(`Downloaded: ${percent}%`)
-})
-
-// This fires during download, NOT during decode
-```
-
----
-
-## Working with Partially Loaded Audio
-
-### Short Answer: Not Directly Supported
-
-WaveSurfer cannot render or play partially downloaded audio. However, there are workarounds:
-
-### Pattern 1: Pre-decoded Peaks (Recommended)
-
-```typescript
-const wavesurfer = WaveSurfer.create({
-  container: '#waveform',
-  url: '/audio/long-file.mp3',  // Can be streaming URL
-  peaks: preComputedPeaksArray,  // Pre-generated peaks
-  duration: 3600,  // Known duration in seconds
-})
-```
-
-From `examples/predecoded.js`:
-
-```javascript
-const wavesurfer = WaveSurfer.create({
-  container: document.body,
-  waveColor: 'rgb(200, 0, 200)',
-  progressColor: 'rgb(100, 0, 100)',
-  barWidth: 10,
-  barRadius: 10,
-  barGap: 2,
-  url: '/examples/audio/demo.wav',
-  peaks: [
-    [
-      0, 0.0023595101665705442, 0.012107174843549728, 0.005919494666159153,
-      -0.31324470043182373, 0.1511787623167038, 0.2473851442337036,
-      // ... more peak values
-    ],
-  ],
-  duration: 22,
-})
-```
-
-### Pattern 2: WebAudio Shim (For Precise Timing)
-
-From `examples/webaudio-shim.js`:
-
-```javascript
-import WaveSurfer from 'wavesurfer.js'
-import WebAudioPlayer from 'wavesurfer.js/dist/webaudio.js'
-
-const webAudioPlayer = new WebAudioPlayer()
-webAudioPlayer.src = '/examples/audio/audio.wav'
-
-webAudioPlayer.addEventListener('loadedmetadata', () => {
-  const wavesurfer = WaveSurfer.create({
-    container: document.body,
-    media: webAudioPlayer,
-    peaks: webAudioPlayer.getChannelData(),
-    duration: webAudioPlayer.duration,
-  })
-})
-```
-
-### Pattern 3: Dynamic Peaks Update
-
-You can call `setOptions()` with new peaks as they become available:
-
-```typescript
-// Initial load with placeholder
-const wavesurfer = WaveSurfer.create({
-  container: '#waveform',
-  url: streamingUrl,
-  peaks: [[0, 0, 0]],  // Minimal placeholder
-  duration: estimatedDuration,
-})
-
-// Later, update with real peaks
-wavesurfer.setOptions({
-  peaks: actualPeaksData,
-  duration: actualDuration,
-})
-```
-
----
-
-## Pre-decoded Peaks Pattern
-
-### How It Works:
-
-1. **Server-side:** Generate peaks using tools like `audiowaveform`
-2. **Client-side:** Pass peaks and duration to WaveSurfer
-3. **Rendering:** Waveform renders immediately without downloading audio
-4. **Playback:** HTMLMediaElement handles streaming playback
-
-### Decoder.createBuffer (`src/decoder.ts`):
-
-```typescript
-/** Create an audio buffer from pre-decoded audio data */
-function createBuffer(channelData: Array<Float32Array | number[]>, duration: number): AudioBuffer {
-  // Validate inputs
-  if (!channelData || channelData.length === 0) {
-    throw new Error('channelData must be a non-empty array')
-  }
-  if (duration <= 0) {
-    throw new Error('duration must be greater than 0')
-  }
-
-  // If a single array of numbers is passed, make it an array of arrays
-  if (typeof channelData[0] === 'number') channelData = [channelData as unknown as number[]]
-
-  // Normalize to -1..1
-  normalize(channelData)
-
-  // Convert to Float32Array for consistency
-  const float32Channels = channelData.map((channel) =>
-    channel instanceof Float32Array ? channel : Float32Array.from(channel),
-  )
-
-  return {
-    duration,
-    length: float32Channels[0].length,
-    sampleRate: float32Channels[0].length / duration,
-    numberOfChannels: float32Channels.length,
-    getChannelData: (i: number) => {
-      const channel = float32Channels[i]
-      if (!channel) {
-        throw new Error(`Channel ${i} not found`)
-      }
-      return channel
-    },
-    copyFromChannel: AudioBuffer.prototype.copyFromChannel,
-    copyToChannel: AudioBuffer.prototype.copyToChannel,
-  } as AudioBuffer
-}
-```
-
-### Peak Generation Tools:
-
-- **audiowaveform** (recommended): https://github.com/bbc/audiowaveform
-- **ffmpeg** with audio filter
-- Custom server-side processing
-
----
-
-## MediaStream Support (Recording)
-
-### Real-time Waveform During Recording
-
-The **Record Plugin** (`src/plugins/record.ts`) provides real-time waveform rendering from microphone input:
-
-```typescript
-public renderMicStream(stream: MediaStream): MicStream {
-  const audioContext = new AudioContext()
-  const source = audioContext.createMediaStreamSource(stream)
-  const analyser = audioContext.createAnalyser()
-  source.connect(analyser)
-
-  // Use smaller FFT size for more responsive peak detection
-  if (this.options.continuousWaveform || this.options.scrollingWaveform) {
-    analyser.fftSize = 32
-  }
-  const bufferLength = analyser.frequencyBinCount
-  const dataArray = new Float32Array(bufferLength)
-
-  // ... drawing logic at 100 FPS ...
-
-  const drawWaveform = () => {
-    if (this.isWaveformPaused) return
-
-    analyser.getFloatTimeDomainData(dataArray)
-
-    if (this.options.scrollingWaveform) {
-      // Scrolling waveform - use peak values
-      // ... accumulate peaks ...
-    } else if (this.options.continuousWaveform) {
-      // Continuous waveform
-      // ... accumulate all data ...
-    }
-
-    // Render using wavesurfer.load() with peaks
-    if (this.wavesurfer) {
-      this.wavesurfer
-        .load('', [this.dataWindow], totalDuration)
-        .then(() => { /* ... */ })
-    }
-  }
-
-  const intervalId = setInterval(drawWaveform, 1000 / FPS)  // 100 FPS
-  // ...
-}
-```
-
-### Key Features:
-
-- **Scrolling Waveform:** Fixed window showing last N seconds
-- **Continuous Waveform:** Growing waveform as recording progresses
-- **100 FPS update rate** for smooth visualization
-- Uses `load('', [peaks], duration)` pattern internally
-
----
-
-## Limitations and Workarounds
-
-### Memory Constraints for Large Files
-
-From README.md (lines 114-116):
-
-```markdown
-<details>
-  <summary>Does wavesurfer support large files?</summary>
-  Since wavesurfer decodes audio entirely in the browser using Web Audio, large clips may fail to decode due to memory constraints. We recommend using pre-decoded peaks for large files.
-</details>
-```
-
-### VBR Audio Mismatch
-
-Variable bit rate files can cause waveform/audio sync issues:
-
-```markdown
-<details>
-  <summary>There is a mismatch between my audio and the waveform. How do I fix it?</summary>
-  If you're using a VBR (variable bit rate) audio file, there might be a mismatch between the audio and the waveform. This can be fixed by converting your file to CBR (constant bit rate).
-</details>
-```
-
-### No MediaSource API Integration
-
-WaveSurfer does not use MediaSource Extensions (MSE) for:
-- Progressive audio loading
-- Adaptive bitrate streaming
-- HLS/DASH support
-
----
-
-## Recommendations for Streaming Audio
-
-### Recommended Architecture for Streaming:
-
-```
-                                   +------------------+
-                                   |  Audio Server    |
-                                   |  (HLS/DASH/MP3)  |
-                                   +--------+---------+
-                                            |
-           +--------------------------------+--------------------------------+
-           |                                                                 |
-           v                                                                 v
-+----------+----------+                                          +-----------+-----------+
-|  Pre-computed Peaks |                                          |  Audio Streaming URL  |
-|  (JSON/Binary)      |                                          |  (direct or chunked)  |
-+----------+----------+                                          +-----------+-----------+
-           |                                                                 |
-           v                                                                 v
-+----------+----------+                                          +-----------+-----------+
-|  WaveSurfer         |                                          |  HTMLMediaElement     |
-|  (waveform render)  |<---------------------------------------->|  (audio playback)     |
-+---------------------+                                          +-----------------------+
-```
-
-### Implementation Pattern:
-
-```typescript
-// 1. Fetch peaks separately (fast, small payload)
-const peaksResponse = await fetch('/api/audio/peaks/123')
-const { peaks, duration } = await peaksResponse.json()
-
-// 2. Create WaveSurfer with streaming URL
-const wavesurfer = WaveSurfer.create({
-  container: '#waveform',
-  url: 'https://streaming.example.com/audio/123.mp3',  // Can be HLS, etc.
-  peaks: peaks,
-  duration: duration,
-  backend: 'MediaElement',  // Use HTMLMediaElement for streaming
-})
-
-// 3. Audio streams progressively via HTMLMediaElement
-// Waveform is instantly rendered from pre-computed peaks
-```
-
-### For Progressive Loading Without Pre-computed Peaks:
-
-If you cannot pre-compute peaks, consider:
-
-1. **Chunked Peak Generation:** Generate peaks in chunks on server, send progressively
-2. **Placeholder Waveform:** Show loading indicator, then render when ready
-3. **Estimated Waveform:** Use audio analysis on first few seconds, extrapolate
-4. **Hybrid Approach:** Quick low-resolution peaks first, high-res after full load
-
----
-
-## Code Snippets Reference
-
-### WaveSurfer Options Interface (relevant streaming options):
-
-```typescript
-export type WaveSurferOptions = {
-  // ... other options ...
-
-  /** Audio URL */
-  url?: string
-
-  /** Pre-computed audio data, arrays of floats for each channel */
-  peaks?: Array<Float32Array | number[]>
-
-  /** Pre-computed audio duration in seconds */
-  duration?: number
-
-  /** Use an existing media element instead of creating one */
-  media?: HTMLMediaElement
-
-  /** Options to pass to the fetch method */
-  fetchParams?: RequestInit
-
-  /** Playback "backend" to use, defaults to MediaElement */
-  backend?: 'WebAudio' | 'MediaElement'
-
-  /** Override the Blob MIME type */
-  blobMimeType?: string
-}
-```
-
-### Events for Monitoring Load Progress:
-
-```typescript
-// During download
-wavesurfer.on('load', (url) => {
-  console.log('Started loading:', url)
-})
-
-wavesurfer.on('loading', (percent) => {
-  console.log(`Download progress: ${percent}%`)
-})
-
-// After decode
-wavesurfer.on('decode', (duration) => {
-  console.log('Audio decoded, duration:', duration)
-})
-
-// Ready for playback
-wavesurfer.on('ready', (duration) => {
-  console.log('Ready to play, duration:', duration)
-})
-```
-
-### WebAudioPlayer (Custom Backend):
-
-```typescript
-import WebAudioPlayer from 'wavesurfer.js/dist/webaudio.js'
-
-// WebAudioPlayer decodes entire file but provides precise timing
-const player = new WebAudioPlayer()
-player.src = '/audio.mp3'
-
-player.addEventListener('loadedmetadata', () => {
-  // Access decoded channel data
-  const channelData = player.getChannelData()  // Float32Array[]
-  const duration = player.duration
-})
-```
-
----
-
-## Conclusion
-
-WaveSurfer.js is designed for **pre-loaded audio visualization**, not streaming. For streaming audio applications:
-
-1. **Always use pre-decoded peaks** for large or streaming audio
-2. **Let HTMLMediaElement handle streaming** playback
-3. **Generate peaks server-side** using audiowaveform or similar
-4. **Consider chunked peak delivery** for very long files
-
-The library's architecture fundamentally requires full audio data for waveform rendering, making true progressive/streaming waveform visualization impossible without the pre-decoded peaks pattern.
-
----
-
-## Related Files
-
-- `/src/wavesurfer.ts` - Main WaveSurfer class
-- `/src/fetcher.ts` - Fetch with progress tracking
-- `/src/decoder.ts` - Audio decoding and buffer creation
-- `/src/webaudio.ts` - WebAudio backend
-- `/src/player.ts` - Base player class
-- `/src/renderer.ts` - Waveform rendering
-- `/src/plugins/record.ts` - Real-time microphone recording
-- `/examples/predecoded.js` - Pre-decoded peaks example
-- `/examples/webaudio-shim.js` - WebAudio backend example