lucille-node 0.0.1

package/README.md

@@ -0,0 +1,31 @@
+# Lucille
+
+Named for the incomporable Lucille Ball, lucille is a part of the 4's Entertainment System providing for streaming, and downloadable video.
+It utilizes bittorrent via the webtorrent protocol to enable P2P capabilites to save bandwidth, and provide scalability that other distributed systems cannot while simultaneously saving on bandwidth costs obviating the need for ad-supported revenue. 
+The point is for it to get so popular it breaks. 
+Then we'll know we're doing something right.
+
+## Overview
+
+This first iteration of lucille is built on Digital Ocean infrastructure utilizing the following stack:
+
+-------------
+|| Storage ||
+|| Seeder  ||
+|| Tracker ||
+|| Viewer  ||
+-------------
+
+If you're unfamiliar torrents, no worries!
+All they mean is that while you're streaming your video, you're enabling others to stream it as well.
+This happens seemlessly for you in the background, and lucille doesn't track whether you do or don't, so feel free to turn it off if you'd like.
+
+### Video posters
+
+Posters will likely interface with lucille through a client application that will handle the upload and storage of their video. 
+The first such application will be part of the shoppe plugin for the federated wiki, and can be found here: https://github.com/planet-nine-app/wiki-plugin-shoppe.
+
+### Video watchers
+
+Watchers will likewise interface through a client application, and the first one will probably be at the main lucille hub, which is tbd, or in fed wikis as well. 
+We shall see.

package/STORAGE.md

@@ -0,0 +1,419 @@
+# Storage and Seeding: Build Guide
+
+This covers the two backend Node services you need before the wiki and frontend
+can do anything useful: the DO Spaces storage layer and the WebTorrent seeder +
+tracker. Both are designed to run as long-lived fedwiki plugin processes.
+
+---
+
+## Part 1 — DO Spaces (Storage)
+
+### What it does
+
+Spaces is S3-compatible object storage. It holds the raw video files and
+optionally serves them through a built-in CDN. The seeder pulls from Spaces
+when it first seeds a file; after that, WebTorrent peers take over the majority
+of delivery.
+
+### Setup
+
+Create a Space in the DigitalOcean control panel. Pick the region closest to
+your users. Enable the CDN endpoint — you'll use this URL to serve the initial
+seed bytes cheaply before P2P kicks in.
+
+Generate a Spaces access key under **API → Spaces Keys**. You'll get an access
+key ID and a secret. Store both as environment variables — never in code.
+
+```
+DO_SPACES_KEY=your_access_key
+DO_SPACES_SECRET=your_secret_key
+DO_SPACES_REGION=nyc3
+DO_SPACES_BUCKET=your-bucket-name
+DO_SPACES_CDN_ENDPOINT=https://your-bucket.nyc3.cdn.digitaloceanspaces.com
+```
+
+### Install
+
+```bash
+npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner
+```
+
+### Client setup
+
+```js
+// spaces.js
+const { S3Client, PutObjectCommand, GetObjectCommand,
+        ListObjectsV2Command } = require('@aws-sdk/client-s3')
+const { getSignedUrl } = require('@aws-sdk/s3-request-presigner')
+
+const client = new S3Client({
+  endpoint: `https://${process.env.DO_SPACES_REGION}.digitaloceanspaces.com`,
+  region: process.env.DO_SPACES_REGION,
+  credentials: {
+    accessKeyId: process.env.DO_SPACES_KEY,
+    secretAccessKey: process.env.DO_SPACES_SECRET
+  }
+})
+
+const BUCKET = process.env.DO_SPACES_BUCKET
+```
+
+### Uploading a video
+
+Stream the file into Spaces rather than buffering the whole thing in memory.
+For large files you'll want multipart upload, but the SDK handles that
+automatically when you pass a stream body.
+
+```js
+const fs = require('fs')
+
+async function uploadVideo(localPath, key) {
+  const stream = fs.createReadStream(localPath)
+  const stat = fs.statSync(localPath)
+
+  await client.send(new PutObjectCommand({
+    Bucket: BUCKET,
+    Key: key,               // e.g. 'videos/my-film.mp4'
+    Body: stream,
+    ContentLength: stat.size,
+    ContentType: 'video/mp4',
+    ACL: 'private'          // keep private; serve via presigned URL or CDN
+  }))
+
+  console.log(`Uploaded ${key} (${(stat.size / 1e9).toFixed(2)} GB)`)
+}
+```
+
+### Generating a presigned download URL
+
+The auth plugin will call this before handing the viewer a magnet link. The
+presigned URL lets the seeder fetch the file without making it fully public.
+Expiry of 1 hour is plenty — the seeder only needs it once at startup.
+
+```js
+async function presignedGet(key, expiresIn = 3600) {
+  const command = new GetObjectCommand({ Bucket: BUCKET, Key: key })
+  return getSignedUrl(client, command, { expiresIn })
+}
+```
+
+### Listing available videos
+
+```js
+async function listVideos(prefix = 'videos/') {
+  const response = await client.send(new ListObjectsV2Command({
+    Bucket: BUCKET,
+    Prefix: prefix
+  }))
+  return (response.Contents || []).map(obj => ({
+    key: obj.Key,
+    size: obj.Size,
+    lastModified: obj.LastModified
+  }))
+}
+
+module.exports = { uploadVideo, presignedGet, listVideos }
+```
+
+### CDN delivery
+
+Once a file is in Spaces, its CDN URL is:
+
+```
+https://<bucket>.<region>.cdn.digitaloceanspaces.com/<key>
+```
+
+You can use this CDN URL directly as the download source when creating a
+torrent. The CDN absorbs the initial cold-start bandwidth before P2P takes
+over, and it caches globally so the seeder Droplet's egress bill stays low.
+
+---
+
+## Part 2 — WebTorrent Seeder
+
+### What it does
+
+A long-lived Node process that seeds every video file, maintains an in-memory
+torrent registry, and exposes a small HTTP API so the auth plugin can get
+magnet links without spawning its own WebTorrent client.
+
+### Install
+
+```bash
+npm install webtorrent express
+```
+
+### The seeder process
+
+```js
+// seeder.js
+const WebTorrent = require('webtorrent')
+const express = require('express')
+const https = require('https')
+const fs = require('fs')
+const path = require('path')
+const os = require('os')
+
+const { presignedGet, listVideos } = require('./spaces')
+
+const client = new WebTorrent()
+const app = express()
+app.use(express.json())
+
+// Map of Spaces key → torrent instance
+const torrents = new Map()
+
+// Download a file from a URL into a temp directory and return the local path
+function downloadToTemp(url, filename) {
+  return new Promise((resolve, reject) => {
+    const dest = path.join(os.tmpdir(), filename)
+    // Skip if already cached locally
+    if (fs.existsSync(dest)) return resolve(dest)
+
+    const file = fs.createWriteStream(dest)
+    https.get(url, res => {
+      res.pipe(file)
+      file.on('finish', () => file.close(() => resolve(dest)))
+    }).on('error', err => {
+      fs.unlink(dest, () => {})
+      reject(err)
+    })
+  })
+}
+
+// Seed a single video by its Spaces key
+async function seedVideo(key) {
+  if (torrents.has(key)) return torrents.get(key)
+
+  console.log(`Seeding: ${key}`)
+  const url = await presignedGet(key)
+  const filename = path.basename(key)
+  const localPath = await downloadToTemp(url, filename)
+
+  return new Promise((resolve, reject) => {
+    client.seed(localPath, {
+      // Point peers at your tracker
+      announce: [process.env.TRACKER_URL || 'ws://localhost:8000']
+    }, torrent => {
+      torrents.set(key, torrent)
+      console.log(`Seeding ${filename} — magnet: ${torrent.magnetURI}`)
+      resolve(torrent)
+    })
+  })
+}
+
+// Seed everything in Spaces at startup
+async function seedAll() {
+  const videos = await listVideos()
+  for (const video of videos) {
+    await seedVideo(video.key).catch(err =>
+      console.error(`Failed to seed ${video.key}:`, err.message)
+    )
+  }
+}
+
+// API routes consumed by the auth plugin
+
+// GET /magnet?key=videos/my-film.mp4
+app.get('/magnet', async (req, res) => {
+  const { key } = req.query
+  if (!key) return res.status(400).json({ error: 'key required' })
+
+  try {
+    const torrent = await seedVideo(key)
+    res.json({
+      magnetURI: torrent.magnetURI,
+      infoHash: torrent.infoHash,
+      name: torrent.name
+    })
+  } catch (err) {
+    res.status(500).json({ error: err.message })
+  }
+})
+
+// GET /status — health check and active torrent list
+app.get('/status', (req, res) => {
+  const active = []
+  for (const [key, torrent] of torrents.entries()) {
+    active.push({
+      key,
+      infoHash: torrent.infoHash,
+      peers: torrent.numPeers,
+      uploadSpeed: torrent.uploadSpeed,
+      downloaded: torrent.downloaded
+    })
+  }
+  res.json({ torrents: active, totalPeers: client.torrents.reduce((n, t) => n + t.numPeers, 0) })
+})
+
+// POST /seed — add a newly uploaded video without restarting
+app.post('/seed', async (req, res) => {
+  const { key } = req.body
+  if (!key) return res.status(400).json({ error: 'key required' })
+  try {
+    const torrent = await seedVideo(key)
+    res.json({ magnetURI: torrent.magnetURI })
+  } catch (err) {
+    res.status(500).json({ error: err.message })
+  }
+})
+
+const PORT = process.env.SEEDER_PORT || 7000
+app.listen(PORT, async () => {
+  console.log(`Seeder API listening on port ${PORT}`)
+  await seedAll()
+})
+
+// Graceful shutdown — flush torrents before exit
+process.on('SIGTERM', () => {
+  console.log('Shutting down seeder...')
+  client.destroy(() => process.exit(0))
+})
+```
+
+### Notes on the temp cache
+
+The seeder downloads each file from Spaces once and caches it in `/tmp`. On
+a 7TB Storage-Optimized Droplet you could cache everything locally and skip
+the Spaces download on restart entirely — just point `client.seed()` at a
+permanent local path. For smaller Droplets, the tmpdir approach is fine; the
+file is re-downloaded from Spaces on each cold start.
+
+If your library is large and cold starts are slow, pre-warm the cache on
+Droplet boot using a startup script that calls `POST /seed` for each key.
+
+### Environment variables
+
+```
+TRACKER_URL=ws://your-tracker-host:8000
+SEEDER_PORT=7000
+# Plus the DO_SPACES_* vars from Part 1
+```
+
+---
+
+## Part 3 — Tracker / Signaling Server
+
+### What it does
+
+A WebTorrent-compatible tracker that brokers peer discovery. Viewers register
+their info-hash here and the tracker tells them who else has the same torrent.
+This is a thin WebSocket server — it does no streaming itself.
+
+### Install
+
+```bash
+npm install bittorrent-tracker
+```
+
+### The tracker process
+
+```js
+// tracker.js
+const Tracker = require('bittorrent-tracker').Server
+
+const server = new Tracker({
+  udp: false,   // browsers can't speak UDP; WebRTC/WS only
+  http: true,   // fallback for non-WebSocket clients
+  ws: true,     // WebRTC signaling over WebSocket — the main path
+  stats: true   // enables the /stats endpoint
+})
+
+server.on('error', err => console.error('Tracker error:', err.message))
+
+server.on('warning', err => console.warn('Tracker warning:', err.message))
+
+server.on('listening', () => {
+  const wsAddr = server.ws.address()
+  const httpAddr = server.http.address()
+  console.log(`Tracker WS  listening on ws://0.0.0.0:${wsAddr.port}`)
+  console.log(`Tracker HTTP listening on http://0.0.0.0:${httpAddr.port}`)
+})
+
+// Optional: log peer join/leave for debugging
+server.on('start', (addr, params) => {
+  console.log(`Peer joined  infoHash=${params.info_hash.toString('hex').slice(0,8)}`)
+})
+server.on('stop', (addr, params) => {
+  console.log(`Peer left    infoHash=${params.info_hash.toString('hex').slice(0,8)}`)
+})
+
+const PORT = process.env.TRACKER_PORT || 8000
+server.listen(PORT)
+
+process.on('SIGTERM', () => server.close(() => process.exit(0)))
+```
+
+That's the entire tracker. The `bittorrent-tracker` package does the rest —
+peer list management, announce intervals, scrape responses.
+
+### Pointing the seeder at the tracker
+
+In `seeder.js` above the `announce` array already reads from `TRACKER_URL`.
+Set it to your tracker's public address:
+
+```
+TRACKER_URL=ws://your-tracker-droplet-ip:8000
+```
+
+Viewers pass the same URL via the magnet link's `&tr=` parameter, which the
+auth plugin appends before returning the magnet URI to the browser.
+
+---
+
+## Part 4 — Wiring it together
+
+### Request flow at stream time
+
+```
+viewer pays → auth plugin → GET /magnet?key=videos/film.mp4 → seeder API
+           → seeder returns magnetURI (with tracker URL embedded)
+           → auth plugin returns magnetURI to viewer
+           → viewer's WebTorrent (browser) opens magnet
+           → connects to tracker → discovers seeder + other peers
+           → streams from peer mesh, seeder fills gaps
+```
+
+### Startup order
+
+1. Tracker starts first (seeder needs its URL)
+2. Seeder starts, seeds all Spaces content, registers with tracker
+3. Auth plugin starts, ready to serve magnet links
+
+### Running as services
+
+Both processes are long-lived. On a plain Droplet, use `pm2`:
+
+```bash
+npm install -g pm2
+pm2 start tracker.js --name tracker
+pm2 start seeder.js  --name seeder
+pm2 save
+pm2 startup   # generates the systemd hook so they survive reboots
+```
+
+### Firewall rules (ufw)
+
+```bash
+ufw allow 7000/tcp   # seeder API (internal only — restrict to your auth droplet IP)
+ufw allow 8000/tcp   # tracker HTTP
+ufw allow 8000/udp   # tracker UDP (optional, not needed for browser clients)
+```
+
+The seeder API (port 7000) should only be reachable from the auth plugin's
+Droplet, not the public internet. Use DigitalOcean's VPC or a firewall rule
+scoped to your auth Droplet's private IP.
+
+---
+
+## Part 5 — Deployment checklist
+
+- [ ] DO Spaces bucket created, CDN enabled
+- [ ] Spaces access key generated, stored as env vars
+- [ ] Tracker Droplet running `tracker.js`, port 8000 open
+- [ ] Seeder Droplet (Premium CPU-Optimized) running `seeder.js`
+- [ ] `TRACKER_URL` env var set on seeder pointing to tracker's public IP
+- [ ] `POST /seed` called after every new upload to Spaces
+- [ ] Seeder API (port 7000) firewalled to auth Droplet only
+- [ ] `pm2 startup` run on both Droplets so processes survive reboots
+- [ ] `/status` endpoint smoke-tested to confirm peers are connecting

package/data/lucille/50a84a5a-6b2b-492e-b279-61eddb392e34_video:test-video-1773549/834/196

@@ -0,0 +1 @@
+{"title":"test-video-1773549834196","description":"A test video","tags":["test","integration"],"uuid":"50a84a5a-6b2b-492e-b279-61eddb392e34","videoId":"1bc176d66a289f8dc346048841aa54e05a3664f83e830e6189a73b961f565fe4","createdAt":"2026-03-15T04:43:54.199Z"}

package/data/lucille/dfa790d7-c4f5-43a9-b7ff-f166f0f344c9_video:test-video-1773549/541/230

@@ -0,0 +1 @@
+{"title":"test-video-1773549541230","description":"A test video","tags":["test","integration"],"uuid":"dfa790d7-c4f5-43a9-b7ff-f166f0f344c9","videoId":"6b2c45b7f4f89b1bda0d627adfe01287f83b34aeb919faacc6d4dc2b67c56b64","createdAt":"2026-03-15T04:39:01.233Z"}

package/data/lucille/k/eys

@@ -0,0 +1 @@
+{"privateKey":"c1446ecd82f3d199e6a15660f57a2294214ee10b6ebf86431ca6acfa5bd5b23d","pubKey":"035195b65ea792314cba1400189d6dccc0ceb074a592e22500c3e94dead25ecc00"}

package/data/lucille/pubKey_02b0071b6d90ce7598eabdcfa0596f95359799fd413e300dcbdca823095a/4cf/884

@@ -0,0 +1 @@
+50a84a5a-6b2b-492e-b279-61eddb392e34

package/data/lucille/pubKey_030e4e2f145c4b68863ca42868e08b1559c3c9c10774e61c3f4ef9766d0c/df7/e30

@@ -0,0 +1 @@
+dfa790d7-c4f5-43a9-b7ff-f166f0f344c9

package/data/lucille/user_50a84a5a-6b2b-492e-b279-61eddb/392/e34

@@ -0,0 +1 @@
+{"pubKey":"02b0071b6d90ce7598eabdcfa0596f95359799fd413e300dcbdca823095a4cf884","videos":{},"uuid":"50a84a5a-6b2b-492e-b279-61eddb392e34"}

package/data/lucille/user_dfa790d7-c4f5-43a9-b7ff-f166f0/f34/4c9

@@ -0,0 +1 @@
+{"pubKey":"030e4e2f145c4b68863ca42868e08b1559c3c9c10774e61c3f4ef9766d0cdf7e30","videos":{},"uuid":"dfa790d7-c4f5-43a9-b7ff-f166f0f344c9"}

package/data/lucille/videos_50a84a5a-6b2b-492e-b279-61eddb/392/e34

@@ -0,0 +1 @@
+{"test-video-1773549834196":{"title":"test-video-1773549834196","description":"A test video","tags":["test","integration"],"uuid":"50a84a5a-6b2b-492e-b279-61eddb392e34","videoId":"1bc176d66a289f8dc346048841aa54e05a3664f83e830e6189a73b961f565fe4","createdAt":"2026-03-15T04:43:54.199Z"}}

package/data/lucille/videos_dfa790d7-c4f5-43a9-b7ff-f166f0/f34/4c9

@@ -0,0 +1 @@
+{"test-video-1773549541230":{"title":"test-video-1773549541230","description":"A test video","tags":["test","integration"],"uuid":"dfa790d7-c4f5-43a9-b7ff-f166f0f344c9","videoId":"6b2c45b7f4f89b1bda0d627adfe01287f83b34aeb919faacc6d4dc2b67c56b64","createdAt":"2026-03-15T04:39:01.233Z"}}