@mohanscodex/spectra-code 0.4.9 → 0.5.0

package/skills/cloudflare-deploy/references/realtime-sfu/configuration.md

@@ -0,0 +1,137 @@
+# Configuration & Deployment
+
+## Dashboard Setup
+
+1. Navigate to https://dash.cloudflare.com/?to=/:account/calls
+2. Click "Create Application" (or use existing app)
+3. Copy `CALLS_APP_ID` from dashboard
+4. Generate and copy `CALLS_APP_SECRET` (treat as sensitive credential)
+5. Use credentials in Wrangler config or environment variables below
+
+## Dependencies
+
+**Backend (Workers):** Built-in fetch API, no additional packages required
+
+**Client (PartyTracks):**
+```bash
+npm install partytracks @cloudflare/calls
+```
+
+**Client (React + PartyTracks):**
+```bash
+npm install partytracks @cloudflare/calls observable-hooks
+# Observable hooks: useObservableAsValue, useValueAsObservable
+```
+
+**Client (Raw API):** Native browser WebRTC API only
+
+## Wrangler Setup
+
+```jsonc
+{
+  "name": "my-calls-app",
+  "main": "src/index.ts",
+  "compatibility_date": "2025-01-01", // Use current date for new projects
+  "vars": {
+    "CALLS_APP_ID": "your-app-id",
+    "MAX_WEBCAM_BITRATE": "1200000",
+    "MAX_WEBCAM_FRAMERATE": "24",
+    "MAX_WEBCAM_QUALITY_LEVEL": "1080"
+  },
+  // Set secret: wrangler secret put CALLS_APP_SECRET
+  "durable_objects": {
+    "bindings": [
+      {
+        "name": "ROOM",
+        "class_name": "Room"
+      }
+    ]
+  }
+}
+```
+
+## Deploy
+
+```bash
+wrangler login
+wrangler secret put CALLS_APP_SECRET
+wrangler deploy
+```
+
+## Environment Variables
+
+**Required:**
+- `CALLS_APP_ID`: From dashboard
+- `CALLS_APP_SECRET`: From dashboard (secret)
+
+**Optional:**
+- `MAX_WEBCAM_BITRATE` (default: 1200000)
+- `MAX_WEBCAM_FRAMERATE` (default: 24)
+- `MAX_WEBCAM_QUALITY_LEVEL` (default: 1080)
+- `TURN_SERVICE_ID`: TURN service
+- `TURN_SERVICE_TOKEN`: TURN auth (secret)
+
+## TURN Configuration
+
+```javascript
+const pc = new RTCPeerConnection({
+  iceServers: [
+    { urls: 'stun:stun.cloudflare.com:3478' },
+    {
+      urls: [
+        'turn:turn.cloudflare.com:3478?transport=udp',
+        'turn:turn.cloudflare.com:3478?transport=tcp',
+        'turns:turn.cloudflare.com:5349?transport=tcp'
+      ],
+      username: turnUsername,
+      credential: turnCredential
+    }
+  ],
+  bundlePolicy: 'max-bundle', // Recommended: reduces overhead
+  iceTransportPolicy: 'all'    // Use 'relay' to force TURN (testing only)
+});
+```
+
+**Ports:** 3478 (UDP/TCP), 53 (UDP), 80 (TCP), 443 (TLS), 5349 (TLS)
+
+**When to use TURN:** Required for restrictive corporate firewalls/networks that block UDP. ~5-10% of connections fallback to TURN. STUN works for most users.
+
+**ICE candidate filtering:** Cloudflare handles candidate filtering automatically. No need to manually filter candidates.
+
+## Durable Object Boilerplate
+
+Minimal presence system:
+
+```typescript
+export class Room {
+  private sessions = new Map<string, {userId: string, tracks: string[]}>();
+
+  async fetch(req: Request) {
+    const {pathname} = new URL(req.url);
+    const body = await req.json();
+    
+    if (pathname === '/join') {
+      this.sessions.set(body.sessionId, {userId: body.userId, tracks: []});
+      return Response.json({participants: this.sessions.size});
+    }
+    
+    if (pathname === '/publish') {
+      this.sessions.get(body.sessionId)?.tracks.push(...body.tracks);
+      // Broadcast to others via WebSocket (not shown)
+      return new Response('OK');
+    }
+    
+    return new Response('Not found', {status: 404});
+  }
+}
+```
+
+## Environment Validation
+
+Check credentials before first API call:
+
+```typescript
+if (!env.CALLS_APP_ID || !env.CALLS_APP_SECRET) {
+  throw new Error('CALLS_APP_ID and CALLS_APP_SECRET required');
+}
+```

package/skills/cloudflare-deploy/references/realtime-sfu/gotchas.md

@@ -0,0 +1,133 @@
+# Gotchas & Troubleshooting
+
+## Common Errors
+
+### "Slow initial connect (~1.8s)"
+
+**Cause:** First STUN delayed during consensus forming (normal behavior)
+**Solution:** Subsequent connections are faster. CF detects DTLS ClientHello early to compensate.
+
+### "No media flow"
+
+**Cause:** SDP exchange incomplete, connection not established, tracks not added before offer, browser permissions missing
+**Solution:** 
+1. Verify SDP exchange complete
+2. Check `pc.connectionState === 'connected'`
+3. Ensure tracks added before creating offer
+4. Confirm browser permissions granted
+5. Use `chrome://webrtc-internals` for debugging
+
+### "Track not receiving"
+
+**Cause:** Track not published, track ID not shared, session IDs mismatch, `pc.ontrack` not set, renegotiation needed
+**Solution:** 
+1. Verify track published successfully
+2. Confirm track ID shared between peers
+3. Check session IDs match
+4. Set `pc.ontrack` handler before answer
+5. Trigger renegotiation if needed
+
+### "ICE connection failed"
+
+**Cause:** Network changed, firewall blocked UDP, TURN needed, transient network issue
+**Solution:**
+```typescript
+pc.oniceconnectionstatechange = async () => {
+  if (pc.iceConnectionState === 'failed') {
+    console.warn('ICE failed, attempting restart');
+    await pc.restartIce(); // Triggers new ICE gathering
+    
+    // Create new offer with ICE restart flag
+    const offer = await pc.createOffer({iceRestart: true});
+    await pc.setLocalDescription(offer);
+    
+    // Send to backend → Cloudflare API
+    await fetch(`/api/sessions/${sessionId}/renegotiate`, {
+      method: 'PUT',
+      body: JSON.stringify({sdp: offer.sdp})
+    });
+  }
+};
+```
+
+### "Track stuck/frozen"
+
+**Cause:** Sender paused track, network congestion, codec mismatch, mobile browser backgrounded
+**Solution:**
+1. Check `track.enabled` and `track.readyState === 'live'`
+2. Verify sender active: `pc.getSenders().find(s => s.track === track)`
+3. Check stats for packet loss/jitter (see patterns.md)
+4. On mobile: Re-acquire tracks when app foregrounded
+5. Test with different codecs if persistent
+
+### "Network change disconnects call"
+
+**Cause:** Mobile switching WiFi↔cellular, laptop changing networks
+**Solution:**
+```typescript
+// Listen for network changes
+if ('connection' in navigator) {
+  (navigator as any).connection.addEventListener('change', async () => {
+    console.log('Network changed');
+    await pc.restartIce(); // Use ICE restart pattern above
+  });
+}
+
+// Or use PartyTracks (handles automatically)
+```
+
+## Retry with Exponential Backoff
+
+```typescript
+async function fetchWithRetry(url: string, options: RequestInit, maxRetries = 3) {
+  for (let i = 0; i < maxRetries; i++) {
+    try {
+      const res = await fetch(url, options);
+      if (res.ok) return res;
+      if (res.status >= 500) throw new Error('Server error');
+      return res; // Client error, don't retry
+    } catch (err) {
+      if (i === maxRetries - 1) throw err;
+      const delay = Math.min(1000 * 2 ** i, 10000); // Cap at 10s
+      await new Promise(resolve => setTimeout(resolve, delay));
+    }
+  }
+}
+```
+
+## Debugging with chrome://webrtc-internals
+
+1. Open `chrome://webrtc-internals` in Chrome/Edge
+2. Find your PeerConnection in the list
+3. Check **Stats graphs** for packet loss, jitter, bandwidth
+4. Check **ICE candidate pairs**: Look for `succeeded` state, relay vs host candidates
+5. Check **getStats**: Raw metrics for inbound/outbound RTP
+6. Look for errors in **Event log**: `iceConnectionState`, `connectionState` changes
+7. Export data with "Download the PeerConnection updates and stats data" button
+8. Common issues visible here: ICE failures, high packet loss, bitrate drops
+
+## Limits
+
+| Resource/Limit | Value | Notes |
+|----------------|-------|-------|
+| Egress (Free) | 1TB/month | Per account |
+| Egress (Paid) | $0.05/GB | After free tier |
+| Inbound traffic | Free | All plans |
+| TURN service | Free | Included with SFU |
+| Participants | No hard limit | Client bandwidth/CPU bound (typically 10-50 tracks) |
+| Tracks per session | No hard limit | Client resources limited |
+| Session duration | No hard limit | Production calls run for hours |
+| WebRTC ports | UDP 1024-65535 | Outbound only, required for media |
+| API rate limit | 600 req/min | Per app, burst allowed |
+
+## Security Checklist
+
+- ✅ **Never expose** `CALLS_APP_SECRET` to client
+- ✅ **Validate user identity** in backend before creating sessions
+- ✅ **Implement auth tokens** for session access (JWT in custom header)
+- ✅ **Rate limit** session creation endpoints
+- ✅ **Expire sessions** server-side after inactivity
+- ✅ **Validate track IDs** before subscribing (prevent unauthorized access)
+- ✅ **Use HTTPS** for all signaling (API calls)
+- ✅ **Enable DTLS-SRTP** (automatic with Cloudflare, encrypts media)
+- ⚠️ **Consider E2EE** for sensitive content (implement client-side with Insertable Streams API)

package/skills/cloudflare-deploy/references/realtime-sfu/patterns.md

@@ -0,0 +1,174 @@
+# Patterns & Use Cases
+
+## Architecture
+
+```
+Client (WebRTC) <---> CF Edge <---> Backend (HTTP)
+                           |
+                    CF Backbone (310+ DCs)
+                           |
+                    Other Edges <---> Other Clients
+```
+
+Anycast: Last-mile <50ms (95%), no region select, NACK shield, distributed consensus
+
+Cascading trees auto-scale to millions:
+```
+Publisher -> Edge A -> Edge B -> Sub1
+                    \-> Edge C -> Sub2,3
+```
+
+## Use Cases
+
+**1:1:** A creates session+publishes, B creates+subscribes to A+publishes, A subscribes to B
+**N:N:** All create session+publish, backend broadcasts track IDs, all subscribe to others
+**1:N:** Publisher creates+publishes, viewers each create+subscribe (no fan-out limit)
+**Breakout:** Same PeerConnection! Backend closes/adds tracks, no recreation
+
+## PartyTracks (Recommended)
+
+Observable-based client with automatic device/network handling:
+
+```typescript
+import {PartyTracks} from 'partytracks';
+
+// Create client
+const pt = new PartyTracks({
+  apiUrl: '/api/calls',
+  sessionId: 'my-session',
+  onTrack: (track, peer) => {
+    const video = document.getElementById(`video-${peer.id}`) as HTMLVideoElement;
+    video.srcObject = new MediaStream([track]);
+  }
+});
+
+// Publish camera (push API)
+const camera = await pt.getCamera(); // Auto-requests permissions, handles device changes
+await pt.publishTrack(camera, {trackName: 'my-camera'});
+
+// Subscribe to remote track (pull API)
+await pt.subscribeToTrack({trackName: 'remote-camera', sessionId: 'other-session'});
+
+// React hook example
+import {useObservableAsValue} from 'observable-hooks';
+
+function VideoCall() {
+  const localTracks = useObservableAsValue(pt.localTracks$);
+  const remoteTracks = useObservableAsValue(pt.remoteTracks$);
+  
+  return <div>{/* Render tracks */}</div>;
+}
+
+// Screenshare
+const screen = await pt.getScreenshare();
+await pt.publishTrack(screen, {trackName: 'my-screen'});
+
+// Handle device changes (automatic)
+// PartyTracks detects device changes (e.g., Bluetooth headset) and renegotiates
+```
+
+## Backend
+
+Express:
+```js
+app.post('/api/new-session', async (req, res) => {
+  const r = await fetch(`${CALLS_API}/apps/${process.env.CALLS_APP_ID}/sessions/new`,
+    {method: 'POST', headers: {'Authorization': `Bearer ${process.env.CALLS_APP_SECRET}`}});
+  res.json(await r.json());
+});
+```
+
+Workers: Same pattern, use `env.CALLS_APP_ID` and `env.CALLS_APP_SECRET`
+
+DO Presence: See configuration.md for boilerplate
+
+## Audio Level Detection
+
+```typescript
+// Attach analyzer to audio track
+function attachAudioLevelDetector(track: MediaStreamTrack) {
+  const ctx = new AudioContext();
+  const analyzer = ctx.createAnalyser();
+  const src = ctx.createMediaStreamSource(new MediaStream([track]));
+  src.connect(analyzer);
+  
+  const data = new Uint8Array(analyzer.frequencyBinCount);
+  const checkLevel = () => {
+    analyzer.getByteFrequencyData(data);
+    const level = data.reduce((a, b) => a + b) / data.length;
+    if (level > 30) console.log('Speaking:', level); // Trigger UI update
+    requestAnimationFrame(checkLevel);
+  };
+  checkLevel();
+}
+```
+
+## Connection Quality Monitoring
+
+```typescript
+pc.getStats().then(stats => {
+  stats.forEach(report => {
+    if (report.type === 'inbound-rtp' && report.kind === 'video') {
+      const {packetsLost, packetsReceived, jitter} = report;
+      const lossRate = packetsLost / (packetsLost + packetsReceived);
+      if (lossRate > 0.05) console.warn('High packet loss:', lossRate);
+      if (jitter > 100) console.warn('High jitter:', jitter);
+    }
+  });
+});
+```
+
+## Stage Management (Limit Visible Participants)
+
+```typescript
+// Subscribe to top 6 active speakers only
+let activeSubscriptions = new Set<string>();
+
+function updateStage(topSpeakers: string[]) {
+  const toAdd = topSpeakers.filter(id => !activeSubscriptions.has(id)).slice(0, 6);
+  const toRemove = [...activeSubscriptions].filter(id => !topSpeakers.includes(id));
+  
+  toRemove.forEach(id => {
+    pc.getSenders().find(s => s.track?.id === id)?.track?.stop();
+    activeSubscriptions.delete(id);
+  });
+  
+  toAdd.forEach(async id => {
+    await fetch(`/api/subscribe`, {method: 'POST', body: JSON.stringify({trackId: id})});
+    activeSubscriptions.add(id);
+  });
+}
+```
+
+## Advanced
+
+Bandwidth mgmt:
+```ts
+const s = pc.getSenders().find(s => s.track?.kind === 'video');
+const p = s.getParameters();
+if (!p.encodings) p.encodings = [{}];
+p.encodings[0].maxBitrate = 1200000; p.encodings[0].maxFramerate = 24;
+await s.setParameters(p);
+```
+
+Simulcast (CF auto-forwards best layer):
+```ts
+pc.addTransceiver('video', {direction: 'sendonly', sendEncodings: [
+  {rid: 'high', maxBitrate: 1200000},
+  {rid: 'med', maxBitrate: 600000, scaleResolutionDownBy: 2},
+  {rid: 'low', maxBitrate: 200000, scaleResolutionDownBy: 4}
+]});
+```
+
+DataChannel:
+```ts
+const dc = pc.createDataChannel('chat', {ordered: true, maxRetransmits: 3});
+dc.onopen = () => dc.send(JSON.stringify({type: 'chat', text: 'Hi'}));
+dc.onmessage = (e) => console.log('RX:', JSON.parse(e.data));
+```
+
+**WHIP/WHEP:** For streaming interop (OBS → SFU, SFU → video players), use WHIP (ingest) and WHEP (egress) protocols. See Cloudflare Stream integration docs.
+
+Integrations: R2 for recording `env.R2_BUCKET.put(...)`, Queues for analytics
+
+Perf: 100-250ms connect, ~50ms latency (95%), 200-400ms glass-to-glass, no participant limit (client: 10-50 tracks)

package/skills/cloudflare-deploy/references/realtimekit/README.md

@@ -0,0 +1,113 @@
+# Cloudflare RealtimeKit
+
+Expert guidance for building real-time video and audio applications using **Cloudflare RealtimeKit** - a comprehensive SDK suite for adding customizable live video and voice to web or mobile applications.
+
+## Overview
+
+RealtimeKit is Cloudflare's SDK suite built on Realtime SFU, abstracting WebRTC complexity with fast integration, pre-built UI components, global performance (300+ cities), and production features (recording, transcription, chat, polls).
+
+**Use cases**: Team meetings, webinars, social video, audio calls, interactive plugins
+
+## Core Concepts
+
+- **App**: Workspace grouping meetings, participants, presets, recordings. Use separate Apps for staging/production
+- **Meeting**: Re-usable virtual room. Each join creates new **Session**
+- **Session**: Live meeting instance. Created on first join, ends after last leave
+- **Participant**: User added via REST API. Returns `authToken` for client SDK. **Do not reuse tokens**
+- **Preset**: Reusable permission/UI template (permissions, meeting type, theme). Applied at participant creation
+- **Peer ID** (`id`): Unique per session, changes on rejoin
+- **Participant ID** (`userId`): Persistent across sessions
+
+## Quick Start
+
+### 1. Create App & Meeting (Backend)
+
+```bash
+# Create app
+curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<account_id>/realtime/kit/apps' \
+  -H 'Authorization: Bearer <api_token>' \
+  -d '{"name": "My RealtimeKit App"}'
+
+# Create meeting
+curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<account_id>/realtime/kit/<app_id>/meetings' \
+  -H 'Authorization: Bearer <api_token>' \
+  -d '{"title": "Team Standup"}'
+
+# Add participant
+curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<account_id>/realtime/kit/<app_id>/meetings/<meeting_id>/participants' \
+  -H 'Authorization: Bearer <api_token>' \
+  -d '{"name": "Alice", "preset_name": "host"}'
+# Returns: { authToken }
+```
+
+### 2. Client Integration
+
+**React**:
+```tsx
+import { RtkMeeting } from '@cloudflare/realtimekit-react-ui';
+
+function App() {
+  return <RtkMeeting authToken="<participant_auth_token>" onLeave={() => {}} />;
+}
+```
+
+**Core SDK**:
+```typescript
+import RealtimeKitClient from '@cloudflare/realtimekit';
+
+const meeting = new RealtimeKitClient({ authToken: '<token>', video: true, audio: true });
+await meeting.join();
+```
+
+## Reading Order
+
+| Task | Files |
+|------|-------|
+| Quick integration | README only |
+| Custom UI | README → patterns → api |
+| Backend setup | README → configuration |
+| Debug issues | gotchas |
+| Advanced features | patterns → api |
+
+## RealtimeKit vs Realtime SFU
+
+| Choose | When |
+|--------|------|
+| **RealtimeKit** | Need pre-built UI, fast integration, React/Angular/HTML |
+| **Realtime SFU** | Building from scratch, custom WebRTC, full control |
+
+RealtimeKit is built on Realtime SFU but abstracts WebRTC complexity with UI components and SDKs.
+
+## Which Package?
+
+Need pre-built meeting UI?
+- React → `@cloudflare/realtimekit-react-ui` (`<RtkMeeting>`)
+- Angular → `@cloudflare/realtimekit-angular-ui`
+- HTML/Vanilla → `@cloudflare/realtimekit-ui`
+
+Need custom UI?
+- Core SDK → `@cloudflare/realtimekit` (RealtimeKitClient) - full control
+
+Need raw WebRTC control?
+- See `realtime-sfu/` reference
+
+## In This Reference
+
+- [Configuration](./configuration.md) - Setup, installation, wrangler config
+- [API](./api.md) - Meeting object, REST API, SDK methods
+- [Patterns](./patterns.md) - Common workflows, code examples
+- [Gotchas](./gotchas.md) - Common issues, troubleshooting
+
+## See Also
+
+- [Workers](../workers/) - Backend integration
+- [D1](../d1/) - Meeting metadata storage
+- [R2](../r2/) - Recording storage
+- [KV](../kv/) - Session management
+
+## Reference Links
+
+- **Official Docs**: https://developers.cloudflare.com/realtime/realtimekit/
+- **API Reference**: https://developers.cloudflare.com/api/resources/realtime_kit/
+- **Examples**: https://github.com/cloudflare/realtimekit-web-examples
+- **Dashboard**: https://dash.cloudflare.com/?to=/:account/realtime/kit