offcourse 1.2.1 → 1.3.0

package/README.md

@@ -9,382 +9,83 @@
 
 Download online courses for offline access – of course! 📚
 
-Saves video content and lesson text as Markdown files, organized by module structure.
+**[→ View Documentation & Homepage](https://sebastian-software.github.io/offcourse/)**
 
-## Features
-
-- 🔐 **Browser-based authentication** – Log in once, sessions are cached
-- 📚 **Course structure preservation** – Maintains module/lesson hierarchy
-- 🎬 **Video downloads** – HLS, Vimeo, Loom, native MP4/WebM
-- 📎 **Attachments** – Downloads PDFs and other course materials
-- 📝 **Content extraction** – Converts lesson text to clean Markdown
-- ⏸️ **Resumable syncs** – Skips already downloaded content
-- ⚡ **Concurrent downloads** – Configurable parallelism
-- 🔍 **Auto-detection** – Automatically detects platform from URL
-
-## Supported Platforms
-
-| Platform | Status | Notes |
-|----------|--------|-------|
-| [Skool.com](https://skool.com) | ✅ Supported | Community courses |
-| [HighLevel (GoHighLevel)](https://gohighlevel.com) | ✅ Supported | Membership portals, ClientClub |
-| [LearningSuite.io](https://learningsuite.io) | ✅ Supported | German LMS platform |
-
-## Installation
+## Quick Start
 
 ```bash
+# Install
 npm install -g offcourse
-```
-
-Or run directly with npx:
-
-```bash
-npx offcourse <command>
-```
-
-Requires Node.js 22+.
-
-For HLS video downloads (HighLevel native videos), [ffmpeg](https://ffmpeg.org/) must be installed:
-
-```bash
-# macOS
-brew install ffmpeg
 
-# Ubuntu/Debian
-sudo apt install ffmpeg
+# Download a course
+offcourse sync <course-url>
 
-# Windows (via Chocolatey)
-choco install ffmpeg
+# Or run without installing
+npx offcourse sync <course-url>
 ```
 
-## Usage
+Requires Node.js 22+ and [ffmpeg](https://ffmpeg.org/) for HLS videos.
 
-### Login
-
-```bash
-# Opens browser for interactive login
-offcourse login
-
-# Force re-login
-offcourse login --force
-```
+## Supported Platforms
 
-### Sync a Course
+| Platform | URL Pattern |
+|----------|-------------|
+| [Skool](https://skool.com) | `skool.com/community/classroom` |
+| [HighLevel](https://gohighlevel.com) | `member.*.com/courses/...` |
+| [LearningSuite](https://learningsuite.io) | `*.learningsuite.io/student/...` |
 
-The `sync` command auto-detects the platform from the URL:
+## Key Commands
 
 ```bash
-# Auto-detect platform and download
+# Sync a course (auto-detects platform)
 offcourse sync <url>
 
-# Skip video downloads
-offcourse sync <url> --skip-videos
-
-# Skip text content
-offcourse sync <url> --skip-content
-
-# Preview without downloading
-offcourse sync <url> --dry-run
-
-# Limit to first N lessons (for testing)
-offcourse sync <url> --limit 5
-
-# Override course name (useful when auto-detection fails)
-offcourse sync <url> --course-name "My Course Name"
-
-# Prefer specific video quality
-offcourse sync <url> --quality 720p
-```
-
-### Platform-Specific Commands
-
-```bash
-# Skool courses
-offcourse sync-skool https://www.skool.com/your-community/classroom
+# Sync with options
+offcourse sync <url> --skip-videos      # Text only
+offcourse sync <url> --dry-run          # Preview
+offcourse sync <url> --limit 5          # Test with 5 lessons
 
-# HighLevel/GoHighLevel membership portals
-offcourse sync-highlevel https://member.example.com/courses/products/<id>
-offcourse sync-highlevel <url> --course-name "Course Name"
-
-# LearningSuite courses
-offcourse sync-learningsuite https://subdomain.learningsuite.io/student/course/<id>
-```
-
-### Complete Command (LearningSuite)
-
-Some platforms lock lessons sequentially – you must complete lesson 1 before accessing lesson 2. The `complete` command automatically marks all accessible lessons as complete to unlock more content:
-
-```bash
-# Mark all lessons as complete (iterates until no new content unlocks)
+# Unlock sequential content (LearningSuite)
 offcourse complete <url>
 
-# Show browser window
-offcourse complete <url> --visible
-```
-
-The command runs in rounds:
-1. Scans course structure
-2. Starts any unstarted modules
-3. Marks accessible lessons as complete
-4. Re-scans for newly unlocked content
-5. Repeats until nothing changes
-
-### Configuration
-
-```bash
-# Show current config
-offcourse config show
-
-# Set output directory
+# Configuration
 offcourse config set outputDir ~/Courses
-
-# Set video quality (highest, lowest, 1080p, 720p, 480p)
 offcourse config set videoQuality 720p
-
-# Set download concurrency (1-5)
-offcourse config set concurrency 3
-
-# Run headless (no browser window)
-offcourse config set headless true
-```
-
-### Inspect (Debugging)
-
-```bash
-# Analyze page structure
-offcourse inspect <url>
-
-# Save analysis to files
-offcourse inspect <url> --output ./analysis
-
-# Include full HTML dump
-offcourse inspect <url> --full
-```
-
-## Output Structure
-
-```
-~/Downloads/offcourse/
-└── course-name/
-    ├── 01-module-name/
-    │   ├── 01-lesson-name/
-    │   │   ├── content.md
-    │   │   └── video.mp4
-    │   └── 02-another-lesson/
-    │       ├── content.md
-    │       └── video.mp4
-    └── 02-next-module/
-        └── ...
-```
-
-## Video Providers
-
-Offcourse supports downloading videos from various providers, regardless of which platform hosts the course:
-
-| Provider | Status | Notes |
-|----------|--------|-------|
-| **HLS Streams** | ✅ Supported | Requires ffmpeg. Used by HighLevel, LearningSuite (via Bunny CDN) |
-| **Vimeo** | ✅ Supported | Embedded players, extracts best quality |
-| **Loom** | ✅ Supported | Share links and embeds |
-| **Native MP4/WebM** | ✅ Supported | Direct file downloads |
-| **YouTube** | 🚧 Planned | Requires yt-dlp |
-| **Wistia** | 🚧 Planned | Requires special handling |
-
-### CDN Support
-
-Videos are often served through CDNs for better performance:
-
-- **Bunny CDN** – Used by LearningSuite for HLS streams (requires session cookies)
-- **Cloudflare Stream** – Common for HighLevel native videos
-- **Vimeo CDN** – For embedded Vimeo players
-
-## Platform Notes
-
-### HighLevel (GoHighLevel)
-
-HighLevel is an all-in-one marketing platform with a "Memberships" feature for hosting courses.
-
-| Feature | Support |
-|---------|---------|
-| Authentication | Firebase-based login via browser |
-| Course structure | API-based extraction (products, categories, posts) |
-| Videos | Native HLS with quality selection, Vimeo, Loom embeds |
-| Attachments | ✅ Supported |
-
-**URL patterns:**
-- `https://member.yourdomain.com/courses/...`
-- `https://portal.yourdomain.com/courses/...`
-- `https://courses.yourdomain.com/...`
-
-### LearningSuite
-
-LearningSuite is a German LMS platform popular with coaches and course creators.
-
-| Feature | Support |
-|---------|---------|
-| Authentication | Browser-based with session caching |
-| Course structure | DOM-based extraction |
-| Videos | HLS streams via Bunny CDN (requires ffmpeg + cookies) |
-| Attachments | ✅ PDFs and course materials |
-| Sequential unlocking | Use `offcourse complete <url>` |
-
-**URL format:** `https://{subdomain}.learningsuite.io/student/course/{slug}/{courseId}`
-
-**Note:** Videos require session cookies which are automatically extracted from the browser session.
-
-## Architecture
-
-```
-src/
-├── cli/              # Command-line interface
-│   ├── commands/     # Individual commands (sync, login, config, etc.)
-│   └── index.ts      # CLI entry point
-├── config/           # Configuration management
-├── downloader/       # Video download handlers
-│   ├── hlsDownloader.ts    # HLS/m3u8 streams (ffmpeg)
-│   ├── vimeoDownloader.ts  # Vimeo video extraction
-│   ├── loomDownloader.ts   # Loom video extraction
-│   └── queue.ts            # Download queue with concurrency
-├── scraper/          # Platform-specific scrapers
-│   ├── highlevel/    # HighLevel/GoHighLevel support
-│   ├── learningsuite/# LearningSuite support
-│   ├── extractor.ts  # Common content extraction
-│   └── navigator.ts  # Common navigation utilities
-├── shared/           # Shared utilities
-│   ├── auth.ts       # Authentication helpers
-│   ├── url.ts        # URL parsing utilities
-│   └── slug.ts       # Filename sanitization
-├── state/            # SQLite database for tracking
-└── storage/          # File system operations
 ```
 
-### Adding a New Platform
-
-1. Create a new directory under `src/scraper/` (e.g., `src/scraper/newplatform/`)
-2. Implement required modules:
-   - `auth.ts` – Session detection and validation
-   - `navigator.ts` – Course structure extraction
-   - `extractor.ts` – Lesson content extraction
-   - `schemas.ts` – Zod schemas for API responses
-3. Add CLI command in `src/cli/commands/`
-4. Register in `src/cli/index.ts`
-
 ## Development
 
-### Setup
-
 ```bash
-# Clone the repository
 git clone https://github.com/sebastian-software/offcourse.git
 cd offcourse
-
-# Install dependencies
 npm install
-
-# Build
 npm run build
-
-# Link globally (optional)
-npm link
+npm link  # optional: link globally
 ```
 
 ### Commands
 
 ```bash
-# Watch mode
-npm run dev
-
-# Run directly (without build)
-npx tsx src/cli/index.ts <command>
-
-# Lint
-npm run lint
-
-# Format
-npm run format
-
-# Type check
-npm run typecheck
-
-# Test
-npm test
-```
-
-### Git Hooks
-
-This project uses [Husky](https://typicode.github.io/husky/) for Git hooks:
-
-- **pre-commit**: Runs Prettier on staged files via lint-staged
-- **pre-push**: Runs ESLint and TypeScript type checking
-- **commit-msg**: Validates commit messages follow [Conventional Commits](https://www.conventionalcommits.org/)
-
-### Commit Convention
-
-We follow [Conventional Commits](https://www.conventionalcommits.org/). Commit messages must follow this format:
-
+npm run dev        # Watch mode
+npm run lint       # ESLint
+npm run typecheck  # TypeScript
+npm test           # Tests
+npm run release    # Release to npm
 ```
-<type>[optional scope]: <description>
-
-[optional body]
-
-[optional footer(s)]
-```
-
-**Types:**
-
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes (formatting, semicolons, etc.)
-- `refactor`: Code refactoring
-- `perf`: Performance improvements
-- `test`: Adding or updating tests
-- `chore`: Maintenance tasks
-- `ci`: CI/CD changes
 
-**Examples:**
-
-```bash
-git commit -m "feat: add support for Vimeo downloads"
-git commit -m "fix: handle missing video URLs gracefully"
-git commit -m "docs: update installation instructions"
-```
-
-### Releasing
-
-Releases are managed with [release-it](https://github.com/release-it/release-it). The release process:
-
-1. Runs linting, type checking, and tests
-2. Bumps version based on conventional commits
-3. Generates/updates `CHANGELOG.md`
-4. Creates a Git tag and GitHub release
-5. Publishes to npm
-
-```bash
-# Interactive release (will prompt for version bump)
-npm run release
-
-# Dry run (preview what would happen)
-npm run release -- --dry-run
+### Adding a New Platform
 
-# Specific version bump
-npm run release -- --minor
-npm run release -- --major
-```
+1. Create `src/scraper/newplatform/` with:
+   - `auth.ts` – Session detection
+   - `navigator.ts` – Course structure
+   - `extractor.ts` – Content extraction
+   - `schemas.ts` – Zod schemas
+2. Add CLI command in `src/cli/commands/`
+3. Register in `src/cli/index.ts`
 
 ## Acknowledgments
 
-A huge thank you to [Sindre Sorhus](https://github.com/sindresorhus) 🙏 for creating and maintaining so many excellent packages that power this project:
-
-- [`@sindresorhus/slugify`](https://github.com/sindresorhus/slugify) – Slugify a string
-- [`conf`](https://github.com/sindresorhus/conf) – Simple config handling
-- [`delay`](https://github.com/sindresorhus/delay) – Delay a promise
-- [`execa`](https://github.com/sindresorhus/execa) – Process execution for humans
-- [`ky`](https://github.com/sindresorhus/ky) – Tiny & elegant HTTP client
-- [`p-queue`](https://github.com/sindresorhus/p-queue) – Promise queue with concurrency control
-- [`p-retry`](https://github.com/sindresorhus/p-retry) – Retry a promise-returning function
-
-His commitment to high-quality, well-documented, and beautifully designed open source software is truly inspiring. If you find his work useful, consider [sponsoring him](https://github.com/sponsors/sindresorhus).
+Thanks to [Sindre Sorhus](https://github.com/sindresorhus) for the excellent packages powering this project: `slugify`, `conf`, `delay`, `execa`, `ky`, `p-queue`, `p-retry`.
 
 ## License
 

package/dist/cli/commands/syncLearningSuite.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"syncLearningSuite.d.ts","sourceRoot":"","sources":["../../../src/cli/commands/syncLearningSuite.ts"],"names":[],"mappings":"AA2EA,MAAM,WAAW,wBAAwB;IACvC,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,WAAW,4BAA4B;IAC3C,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAaD;;;GAGG;AACH,eAAO,MAAM,wBAAwB,0BAKnC,CAAC;AAoEH;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAE1D;AAED;;;GAGG;AACH,wBAAsB,wBAAwB,CAC5C,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,wBAAwB,GAChC,OAAO,CAAC,IAAI,CAAC,CA4Vf;AA+HD;;GAEG;AACH,wBAAgB,2BAA2B,CACzC,KAAK,EAAE,MAAM,EACb,WAAW,EAAE,MAAM,GAAG,IAAI,EAC1B,WAAW,EAAE,MAAM,GAAG,IAAI,GACzB,MAAM,CAkCR;AA8BD;;GAEG;AACH,wBAAsB,4BAA4B,CAChD,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,4BAA4B,GACpC,OAAO,CAAC,IAAI,CAAC,CAwPf"}
+{"version":3,"file":"syncLearningSuite.d.ts","sourceRoot":"","sources":["../../../src/cli/commands/syncLearningSuite.ts"],"names":[],"mappings":"AA4EA,MAAM,WAAW,wBAAwB;IACvC,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,WAAW,4BAA4B;IAC3C,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAaD;;;GAGG;AACH,eAAO,MAAM,wBAAwB,0BAKnC,CAAC;AAoEH;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAE1D;AAED;;;GAGG;AACH,wBAAsB,wBAAwB,CAC5C,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,wBAAwB,GAChC,OAAO,CAAC,IAAI,CAAC,CA+Wf;AAoID;;GAEG;AACH,wBAAgB,2BAA2B,CACzC,KAAK,EAAE,MAAM,EACb,WAAW,EAAE,MAAM,GAAG,IAAI,EAC1B,WAAW,EAAE,MAAM,GAAG,IAAI,GACzB,MAAM,CAkCR;AA8BD;;GAEG;AACH,wBAAsB,4BAA4B,CAChD,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,4BAA4B,GACpC,OAAO,CAAC,IAAI,CAAC,CAqQf"}

package/dist/cli/commands/syncLearningSuite.js

@@ -5,7 +5,7 @@ import { join } from "node:path";
 import { loadConfig } from "../../config/configManager.js";
 import { downloadVideo } from "../../downloader/index.js";
 import { getAuthenticatedSession, createLoginChecker } from "../../shared/auth.js";
-import { buildLearningSuiteCourseStructure, createFolderName, extractLearningSuitePostContent, getLearningSuiteLessonUrl, slugify, } from "../../scraper/learningsuite/index.js";
+import { buildLearningSuiteCourseStructure, createFolderName, extractLearningSuitePostContent, getAuthToken, getLearningSuiteLessonUrl, slugify, } from "../../scraper/learningsuite/index.js";
 import { createCourseDirectory, createModuleDirectory, getVideoPath, saveMarkdown, isLessonSynced, downloadFile, } from "../../storage/fileSystem.js";
 /**
  * Tracks if shutdown has been requested (Ctrl+C).
@@ -325,11 +325,12 @@ export async function syncLearningSuiteCommand(url, options) {
                             }
                             // Queue video download
                             if (!options.skipVideos && !syncStatus.video && content.video?.url) {
+                                const videoUrl = content.video.hlsUrl ?? content.video.url;
                                 videoTasks.push({
                                     lessonId: lesson.id,
                                     lessonName: lesson.title,
-                                    videoUrl: content.video.hlsUrl ?? content.video.url,
-                                    videoType: mapVideoType(content.video.type),
+                                    videoUrl,
+                                    videoType: mapVideoType(content.video.type, videoUrl),
                                     outputPath: getVideoPath(moduleDir, lessonIndex, lesson.title),
                                     preferredQuality: options.quality,
                                 });
@@ -350,11 +351,12 @@ export async function syncLearningSuiteCommand(url, options) {
                             lesson.id);
                             const content = await extractLearningSuitePostContent(session.page, lessonUrl, courseStructure.tenantId, courseStructure.course.id, lesson.id);
                             if (content?.video?.url) {
+                                const videoUrl = content.video.hlsUrl ?? content.video.url;
                                 videoTasks.push({
                                     lessonId: lesson.id,
                                     lessonName: lesson.title,
-                                    videoUrl: content.video.hlsUrl ?? content.video.url,
-                                    videoType: mapVideoType(content.video.type),
+                                    videoUrl,
+                                    videoType: mapVideoType(content.video.type, videoUrl),
                                     outputPath: getVideoPath(moduleDir, lessonIndex, lesson.title),
                                     preferredQuality: options.quality,
                                 });
@@ -382,31 +384,54 @@ export async function syncLearningSuiteCommand(url, options) {
         console.log(`   Content: ${contentParts.join(", ")}`);
         // Phase 3: Download videos
         if (!options.skipVideos && videoTasks.length > 0) {
-            // Extract cookies from session for authenticated video downloads
+            // Extract cookies and auth token from session for authenticated video downloads
             const browserCookies = await session.page.context().cookies();
             const cookieString = browserCookies.map((c) => `${c.name}=${c.value}`).join("; ");
             const refererUrl = `https://${courseStructure.domain}/`;
-            // Add cookies and referer to all video tasks
+            const authToken = await getAuthToken(session.page);
+            // Add cookies, referer, and auth token to all video tasks
             for (const task of videoTasks) {
                 task.cookies = cookieString;
                 task.referer = refererUrl;
+                if (authToken) {
+                    task.authToken = authToken;
+                }
             }
             await downloadVideos(videoTasks, config);
         }
         console.log(chalk.green("\n✅ Sync complete!\n"));
         console.log(chalk.gray(`   Output: ${courseDir}\n`));
     }
+    catch (error) {
+        console.error(chalk.red("\n❌ Sync failed"));
+        if (error instanceof Error) {
+            console.error(chalk.gray(`   Error: ${error.message}`));
+        }
+        throw error;
+    }
     finally {
-        await browser.close();
+        // Always close the browser to prevent hanging
+        if (browser) {
+            try {
+                await browser.close();
+            }
+            catch {
+                // Ignore errors during browser close
+            }
+        }
     }
 }
 /**
  * Maps LearningSuite video type to downloader video type.
  */
-function mapVideoType(type) {
+function mapVideoType(type, url) {
+    // Special case: segments:... URLs use direct HLS segment download
+    if (url?.startsWith("segments:")) {
+        return "hls";
+    }
     switch (type) {
         case "hls":
-            return "highlevel"; // Use HLS downloader
+            return "highlevel"; // Use HLS downloader for standard HLS
         case "vimeo":
             return "vimeo";
         case "loom":
@@ -758,8 +783,23 @@ export async function completeLearningSuiteCommand(url, options) {
         }
         console.log(chalk.green("✅ Complete finished!\n"));
     }
+    catch (error) {
+        console.error(chalk.red("\n❌ Complete failed"));
+        if (error instanceof Error) {
+            console.error(chalk.gray(`   Error: ${error.message}`));
+        }
+        throw error;
+    }
     finally {
-        await browser.close();
+        // Always close the browser to prevent hanging
+        if (browser) {
+            try {
+                await browser.close();
+            }
+            catch {
+                // Ignore errors during browser close
+            }
+        }
     }
 }
 //# sourceMappingURL=syncLearningSuite.js.map