@tricoteuses/transcription-videos 0.1.1

package/LICENSE.md

@@ -0,0 +1,25 @@
+# Tricoteuses-Transcription-Videos
+
+## _Node.js/TypeScript pipeline to **transcribe French Parliament videos**, from either a **`.m3u8`** URL or a **WAV** file extracted via `ffmpeg`._
+
+By:
+
+- Henry Boisgibault <mailto:henry.boisgibault@proton.me>
+- Pierre Drege <mailto:dregop@proton.me>
+
+Copyright © 2025, 2026 Logora
+
+https://git.tricoteuses.fr/logiciels/tricoteuses-transcription-videos
+
+> Tricoteuses-Transcription-Videos is free software; you can redistribute
+> it and/or modify it under the terms of the GNU Affero General Public
+> License as published by the Free Software Foundation, either version 3
+> of the License, or (at your option) any later version.
+>
+> Tricoteuses-Transcription-Videos is distributed in the hope that it
+> will be useful, but WITHOUT ANY WARRANTY; without even the implied
+> warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+> See the GNU Affero General Public License for more details.
+>
+> You should have received a copy of the GNU Affero General Public License
+> along with this program. If not, see <http://www.gnu.org/licenses/>.

package/README.md

@@ -0,0 +1,275 @@
+# tricoteuses-transcription-videos
+
+Node.js/TypeScript pipeline to **transcribe French Parliament videos** (with speaker diarization), from either a **`.m3u8`** URL or a **WAV** file extracted via `ffmpeg`.
+
+- **Output**: a **JSON array** in the Compte-Rendu format of Assemblée's Data:
+
+  ```json
+  [
+    {
+      "code_grammaire": "PAROLE_GENERIQUE",
+      "ordre_absolu_seance": "4",
+      "orateurs": {
+        "orateur": {
+          "nom": "speaker A",
+          "id": "",
+          "qualite": ""
+        }
+      },
+      "texte": {
+        "_": "Merci monsieur le rapporteur général."
+      }
+    },
+    {
+      "code_grammaire": "PAROLE_GENERIQUE",
+      "ordre_absolu_seance": "5",
+      "orateurs": {
+        "orateur": {
+          "nom": "speaker D",
+          "id": "",
+          "qualite": ""
+        }
+      },
+      "texte": {
+        "_": "Merci monsieur le président, mesdames et messieurs, ..."
+      }
+    }
+  ]
+  ```
+
+  > Timestamps are **milliseconds**. `speaker` is a **letter** (A, B, C…).
+
+- **Plug-and-play architecture** via **providers**: currently `AssemblyAI` and `Deepgram`. You can plug additional models later without changing application code.
+
+---
+
+## Table of Contents
+
+- [Prerequisites](#prerequisites)
+- [Installation](#installation)
+- [Usage (single video)](#usage-single-video)
+- [Usage (batch by reunion UIDs)](#usage-batch-by-reunion-uids)
+- [Code Architecture](#code-architecture)
+- [Swap providers later](#swap-providers-later)
+- [License](#license)
+
+---
+
+## Prerequisites
+
+- **Node.js** ≥ 20
+- **npm**
+- **ffmpeg** available in your `PATH` (to extract audio from `.m3u8`):
+  ```bash
+  ffmpeg -version
+  ```
+- **Assemblée dataset prepared** :
+  - Must contain: Agenda\_<LEGISLATURE>\_nettoye/ for the target legislature.
+
+---
+
+## Installation
+
+```bash
+npm install
+cp .env.example .env   # add your models key
+```
+
+---
+
+## Usage (single video, useful for model testing)
+
+### 1) Select your provider in `.env`
+
+Set `TRANSCRIPTION_PROVIDER` to one of: `deepgram` or `assemblyai`
+
+### 2) From a `.m3u8` URL (audio extraction + transcription)
+
+```bash
+# create the output folders if needed
+mkdir -p ./out ./audios
+
+npm run transcribe --   --m3u8 "https://videos-an.vodalys.com/.../master.m3u8"   --out ./audios/reunion.wav   --ss 0   --t 800   --save ./out/transcript-{model_name}.json
+```
+
+### CLI Options
+
+- `--ss` : start offset (seconds)
+- `--t` : duration (seconds)
+- `--out`: WAV path; if omitted, we default to `os.tmpdir()`
+- `--save`: output **JSON** path (default: `./transcript.json`)
+- `--lang fr` to force language (otherwise uses `.env` default)
+- `--diarize false` to disable diarization (enabled by default)
+
+### 3) From an existing audio file **WAV**
+
+```bash
+npm run transcribe --   --file C:/path/to/reunion.wav   --save ./out/transcript-{model_name}.json
+```
+
+---
+
+## Usage (batch by reunion UIDs, useful in prod)
+
+Process only specific Assemblée **reunion UIDs** using the dataset loaders. For each UID:
+
+1. read `reunion.urlVideo`,
+2. extract audio to **`./audios/<uid>.wav`** (skip ffmpeg if the WAV already exists),
+3. transcribe + diarize the **full** video with the current provider,
+4. write segments to **`$ASSEMBLEE_DATA_DIR/Videos_<ROMAN_LEGISLATURE>_nettoye/<uid>/transcript.json`**  
+   (+ `info.json` with basic metadata).
+
+### CLI Options
+
+- `--dataDir`: Absolute path to Assemblée dataset (or as 1st positional). **Required.**
+- `-l, --legislature`: Legislature number (e.g., `16` or `17`).
+- `-s, --fromSession`: Session number to start from (Senat only)
+- `--uids`: Comma-separated UIDs (e.g., `uid1,uid2`).
+- `--uid`: Repeatable UID flag (can be used multiple times).
+- `--lang`, `--language`: Language code (e.g., `fr`).
+- `--diarize`: Enable diarization (default: `true`).
+- `--no-diarize`: Disable diarization.
+- `--keepWav`: Keep extracted WAV files (default: `true`).
+- `--no-keepWav`: Delete WAV after successful transcription.
+- `--audioDir`: Directory for WAV files (default: `./audios`).
+- `--reextract`: Force re-extraction even if WAV exists (default: `false`).
+- `--ss`: Start offset (seconds).
+- `--t`: Duration (seconds).
+- `-p, --provider`: Transcription provider (`assemblyai` | `deepgram`).
+
+### Examples
+
+**Transcribe all Reunions from 17th legislature (max 50):**
+
+```bash
+npm run transcribe:reunions ../assemblee-data -- --legislature 17 --provider assemblyai --max 50
+```
+
+**Force re-extraction + startTimecode to optimize wav:**
+
+```bash
+npm run transcribe:reunions -- --dataDir /abs/path/assemblee-data -l 17 --uid RUANR5...   --reextract true --ss 796
+
+# Transcribe one AN
+npm run transcribe:reunions ../assemblee-data -- --legislature 17 --transcriptsDir ../assemblee-data/transcripts --audioDir ../assemblee-data/audios --provider deepgram --chambre AN --uid RUANR5L17S2025IDC453375
+# Transcribe one SN
+npm run transcribe:reunions ../senat-data -- --fromSession 2025 --transcriptsDir ../senat-data/transcripts --audioDir ../senat-data/audios --provider deepgram --chambre SN --uid RUSN20251016IDODDF-900
+```
+
+## Usage - Transcription Live
+
+Live transcription continuously transcribes an HLS `.m3u8` stream, with automatic retries and a clean stop when the stream ends.  
+It is designed for **one job per live** (e.g. one Kubernetes pod per debate).
+
+### Basic CLI usage
+
+```bash
+npm run transcribe:live --   --url "https://videos-an.vodalys.com/live/.../index.m3u8"   --out ./live-transcripts/live-$(date +%s).ndjson
+```
+
+### Live CLI options
+
+- `--url` (**required**): HLS `.m3u8` live URL
+- `--out`: NDJSON output file (default: `./live-transcripts/live-<timestamp>.ndjson`)
+- `--lang`: language code (default from `.env`)
+- `--diarize` / `--no-diarize`: enable/disable diarization (default: enabled)
+- `--provider`: transcription provider (`deepgram`, `assemblyai`, …)
+- `--model`: provider-specific model (optional)
+- `--punctuate` / `--no-punctuate`: enable/disable punctuation
+- `--maxMinutes`: stop automatically after N minutes (POC / safety)
+
+### Output format (NDJSON)
+
+The output file is append-only, one JSON object per line:
+
+```json
+{"type":"meta","msg":"live transcription start","url":"..."}
+{"type":"segment","start_ms":123400,"end_ms":127800,"speaker":"Speaker A","text":"Hello everyone"}
+{"type":"segment","start_ms":128000,"end_ms":132200,"speaker":"Speaker B","text":"Thank you"}
+{"type":"meta","msg":"session ended","durSec":6400}
+```
+
+This format allows streaming ingestion, retries without duplicates, and easy replay.
+
+### Error handling & retries
+
+- On stream errors or disconnects, the script retries automatically with a short backoff.
+- If a session ends too quickly, it is retried until the minimum valid duration is reached.
+
+### Production integration
+
+Typical flow:
+
+1. API detects a new `DebatDirect`
+2. A job/pod is started for this live
+3. `transcribe:live` runs for this single stream
+4. Segments are pushed incrementally to the API
+5. When the live ends, the job exits and the debate is marked `TERMINE`
+
+**Rule:** 1 live = 1 process.
+
+## Code Architecture
+
+```
+src/
+├─ config/
+│  └─ env.ts                   # .env loading & validation
+├─ types/
+│  └─ transcription.ts         # common types (segments in ms, speakers, metadata)
+├─ providers/
+│  ├─ TranscriptionProvider.ts # generic interface
+│  ├─ assemblyai.ts            # AssemblyAI implementation
+│  ├─ deepgram.ts              # Deepgram implementation
+│  ├─ mistral.ts               # Mistral implementation
+│  └─ index.ts                 # provider factory based on .env
+├─ utils/
+│  └─ ffmpeg.ts                # .m3u8 → WAV mono 16k extraction
+│  └─ transcribe.ts            # single function used by scripts/services
+├─ scripts/
+│  └─ transcribe_reunions.ts
+└─ └─ transcribe_live.ts
+```
+
+---
+
+## Swap providers later
+
+Application code always calls:
+
+```ts
+const result = await transcribeVideo({
+  filePath: '/tmp/reunion.wav',
+  language: 'fr',
+  diarize: true,
+});
+```
+
+To add another provider:
+
+1. Create `src/providers/myProvider.ts` implementing `TranscriptionProvider`.
+2. Add a `case` in `src/providers/index.ts` and a `.env` value (`TRANSCRIPTION_PROVIDER=myProvider`).
+3. Map the new API’s response to the **same types** (segments in ms, letter speakers).
+
+---
+
+## Docker :
+
+### Build the image
+
+```bash
+docker build -t transcriber:dev .
+```
+
+### Run (change the ABSOLUTE_PATH_TO_ASSEMBLEE_DATA)
+
+```bash
+docker run \
+  --env-file .env \
+  -e LEGISLATURE=17 \
+  -v "/ABSOLUTE_PATH_TO_ASSEMBLEE_DATA:/app/assemblee-data" \
+  transcriber:dev
+```
+
+## License
+
+AGPL-3.0-or-later

package/package.json

@@ -0,0 +1,70 @@
+{
+  "name": "@tricoteuses/transcription-videos",
+  "version": "0.1.1",
+  "description": "Permet d'obtenir la transcription des vidéos de l'assemblée/sénat en fournissant un lien vidéo m3u8 en entrée",
+  "keywords": [
+    "Assemblée nationale",
+    "Sénat",
+    "France",
+    "m3u8",
+    "transcription",
+    "vidéo",
+    "modèle IA"
+  ],
+  "author": "Pierre Drege <dregop@proton.me>",
+  "contributors": [
+    "Henry Boisgibault <henry.boisgibault@proton.me>"
+  ],
+  "homepage": "https://tricoteuses.fr/",
+  "license": "AGPL-3.0-or-later",
+  "repository": {
+    "type": "git",
+    "url": "https://git.tricoteuses.fr/logiciels/tricoteuses-transcription-videos.git"
+  },
+  "type": "module",
+  "engines": {
+    "node": ">=20.0.0"
+  },
+  "files": [
+    "lib"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "transcribe:reunions": "tsx src/scripts/transcribe_reunions.ts",
+    "test": "vitest run",
+    "gen:expected": "tsx tests/generate-expected.ts",
+    "transcribe:live": "tsx src/scripts/transcribe_live.ts",
+    "lint": "eslint .",
+    "prepare": "npm run build",
+    "prettier": "prettier --write \"src/**/*.ts\""
+  },
+  "dependencies": {
+    "@deepgram/sdk": "^4.11.2",
+    "@mistralai/mistralai": "^1.14.0",
+    "@speechmatics/batch-client": "^5.1.0",
+    "@tricoteuses/assemblee": "^2.5.18",
+    "@tricoteuses/senat": "^2.20.11",
+    "assemblyai": "^4.15.0",
+    "dotenv": "^17.2.1",
+    "execa": "^9.6.0",
+    "fast-xml-parser": "^5.2.5",
+    "fs-extra": "^11.3.2",
+    "sade": "^1.8.1",
+    "vitest": "^4.0.17",
+    "ws": "^8.19.0",
+    "zod": "^4.0.17"
+  },
+  "devDependencies": {
+    "@types/fs-extra": "^11.0.4",
+    "@types/node": "^24.3.0",
+    "@types/ws": "^8.18.1",
+    "eslint": "^9.27.0",
+    "prettier": "^3.5.3",
+    "tslib": "^2.0.2",
+    "tsx": "^4.0.0",
+    "typescript": "^5.8.3"
+  }
+}