@mogulmoretti/skrape 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -3,7 +3,7 @@
3
3
  Read Skool course content instead of watching it.
4
4
 
5
5
  `skrape` pulls the classroom of a Skool community **you already pay for and
6
- belong to**, and writes clean, readable, timestamped transcripts to disk —
6
+ belong to**, and writes clean, readable, timestamped transcripts to disk,
7
7
  so you can read a lesson in two minutes instead of watching a 20-minute video.
8
8
 
9
9
  It works because Skool ships the whole course tree as structured data inside
@@ -15,13 +15,13 @@ that isn't already sitting on the page you're allowed to see.
15
15
 
16
16
  **This is a personal tool for your own communities.** It signs in as *you*,
17
17
  using *your* real browser session, and only ever touches courses your account
18
- already has access to — it can't and won't bypass Skool's own paywall or
18
+ already has access to. It can't and won't bypass Skool's own paywall or
19
19
  membership checks.
20
20
 
21
21
  Automating access to a platform like this generally sits outside most
22
22
  platforms' terms of service, even when you're a legitimate paying member and
23
23
  even when nothing is being redistributed. Running this is a decision about
24
- your own account, and it's yours to make — this tool doesn't make it for you,
24
+ your own account, and it's yours to make. This tool doesn't make it for you,
25
25
  and it doesn't hide what it's doing (see [How it works](#how-it-works)).
26
26
  Don't use it on communities you don't belong to, don't share or republish
27
27
  what it produces, and don't run it in a way designed to look like normal
@@ -39,33 +39,49 @@ one-time, ~150MB download) if it isn't already on your machine, then walks
39
39
  you through everything:
40
40
 
41
41
  1. Checks whether you're signed in, and if not, opens a real Chrome window for
42
- you to sign in yourself — nothing is typed on your behalf, and your
42
+ you to sign in yourself. Nothing is typed on your behalf, and your
43
43
  password never passes through this tool.
44
44
  2. Looks up the communities your account belongs to and lets you pick one
45
45
  with the arrow keys (an "enter a slug manually" option is always there
46
46
  too, in case discovery doesn't find one).
47
- 3. Shows you what's about to happen — the community, how many courses were
48
- found, where files will land — and waits for you to confirm before any
49
- work starts.
47
+ 3. Shows you what's about to happen (the community, how many courses were
48
+ found, where files will land) and waits for you to confirm before any
49
+ work starts. Pick **Transcripts** or **Transcripts + videos**.
50
50
  4. Runs the sync with a live progress bar. You can Ctrl+C at any point; it
51
51
  cleans up after itself.
52
52
  5. Prints a summary: outcome counts, total words, the output path, and a
53
- readable list of anything that wasn't transcribed and why — nothing is
53
+ readable list of anything that wasn't transcribed and why. Nothing is
54
54
  silently dropped from the count.
55
55
 
56
56
  From there you can sync another community, re-sync the same one (it's
57
- incremental — already-fetched lessons are skipped), open the output folder,
57
+ incremental: already-fetched lessons are skipped), open the output folder,
58
58
  or quit.
59
59
 
60
- Output lands in `./out/<slug>/transcripts/<course>/NN-lesson.md`.
60
+ Output lands in `~/skrape/<slug>/transcripts/<course>/NN-lesson.md`, and videos
61
+ (if you asked for them) in `~/skrape/<slug>/videos/<course>/NN-lesson.mp4`,
62
+ wherever you run skrape from.
63
+
64
+ ### Downloading videos
65
+
66
+ Video downloads are opt-in and use [yt-dlp](https://github.com/yt-dlp/yt-dlp),
67
+ which you install once yourself:
68
+
69
+ brew install yt-dlp ffmpeg # macOS; see yt-dlp's docs for other platforms
70
+
71
+ skrape hands each lesson's video link to yt-dlp, which covers Loom, YouTube,
72
+ Vimeo, Wistia and most other hosts. A video that's already on disk is skipped,
73
+ so re-running a sync only fetches what's missing. Videos are big (a 20-minute
74
+ Loom lesson is ~150MB), so check your free disk space before pulling a whole
75
+ community.
61
76
 
62
77
  ### Scripting / advanced use
63
78
 
64
79
  The underlying subcommands work directly too, without the guided flow:
65
80
 
66
- skrape login # once — sign in by hand, session persists
81
+ skrape login # once: sign in by hand, session persists
67
82
  skrape sync <slug> # slug is the part after skool.com/
68
- skrape sync <slug> -o ./out -c 4 # custom output dir / concurrency
83
+ skrape sync <slug> -o ~/notes -c 4 # custom output dir / concurrency
84
+ skrape sync <slug> --videos # transcripts + every lesson video
69
85
 
70
86
  ### Building from source
71
87
 
@@ -79,7 +95,7 @@ The underlying subcommands work directly too, without the guided flow:
79
95
 
80
96
  ## How it works
81
97
 
82
- Skool ships the full course tree as JSON inside each classroom page — no
98
+ Skool ships the full course tree as JSON inside each classroom page, with no
83
99
  clicking through lesson by lesson. `skrape` reads that structure the way any
84
100
  browser or crawler would: an authenticated request to a page your account
85
101
  can already open.
@@ -89,8 +105,8 @@ moment). Where a caption track exists, it's downloaded and cleaned into
89
105
  readable prose with timestamp anchors. Where one doesn't, the lesson is
90
106
  reported as not transcribed rather than silently skipped.
91
107
 
92
- When a page doesn't come back with the data the tool expects — an expired
93
- session, or Skool changing how a page is built — it falls back to driving an
108
+ When a page doesn't come back with the data the tool expects (an expired
109
+ session, or Skool changing how a page is built), it falls back to driving an
94
110
  actual browser window using your already-authenticated profile, rather than
95
111
  guessing or failing silently.
96
112
 
@@ -100,18 +116,22 @@ run the command.
100
116
 
101
117
  ## What this doesn't do
102
118
 
103
- - Doesn't work on communities you're not a member of — there's no bypass for
119
+ - Doesn't work on communities you're not a member of. There's no bypass for
104
120
  Skool's own access checks, because none of the data is reachable without
105
121
  them.
106
122
  - Doesn't redistribute or publish anything. Output is written to your local
107
123
  disk for you to read.
108
124
  - Doesn't store your password. Sign-in happens in a real Chrome window that
109
125
  only you interact with.
126
+ - Doesn't download videos unless you ask it to (`--videos`, or **Transcripts + videos**
127
+ in the guided flow).
128
+ - Doesn't download lesson `resources`: those are links to outside docs
129
+ (Google Docs, etc.), not files hosted on Skool.
110
130
  - Doesn't transcribe video hosts it doesn't recognize. Those lessons show up
111
131
  in the summary as not transcribed, with a reason.
112
132
 
113
133
  ## License
114
134
 
115
- MIT — see [LICENSE](LICENSE). In short: do what you like with the code, no
135
+ MIT. See [LICENSE](LICENSE). In short: do what you like with the code, no
116
136
  warranty, use it at your own risk and your own judgment about the platforms
117
137
  you point it at.
@@ -1,4 +1,6 @@
1
1
  export declare function profileDir(): string;
2
+ /** Where synced communities land unless -o says otherwise: one fixed place, whatever the cwd. */
3
+ export declare function defaultOutRoot(): string;
2
4
  export declare function dbPath(): string;
3
5
  /**
4
6
  * Marker file written once the one-time Chrome/Playwright browser install
@@ -6,6 +8,12 @@ export declare function dbPath(): string;
6
8
  * Its presence lets later runs skip re-probing with a real browser launch.
7
9
  */
8
10
  export declare function chromeMarkerPath(): string;
11
+ /** Launch the browser first-run setup confirmed; before any setup ran, assume the user's Chrome. */
12
+ export declare function browserLaunchOptions(): {
13
+ executablePath: string;
14
+ } | {
15
+ channel: string;
16
+ };
9
17
  export declare function ensureRoot(): Promise<void>;
10
18
  export declare function isLoggedIn(html: string): boolean;
11
19
  /** A handle callers can use to force-close a launched login context, e.g. on Ctrl+C. */
@@ -2,10 +2,15 @@ import { homedir } from 'node:os';
2
2
  import { join } from 'node:path';
3
3
  import { mkdir } from 'node:fs/promises';
4
4
  import { extractNextData, PayloadParseError } from '../fetch/nextdata.js';
5
+ import { markedExecutablePath } from '../fetch/chromeSetup.js';
5
6
  const ROOT = join(homedir(), '.skool-skrape');
6
7
  export function profileDir() {
7
8
  return join(ROOT, 'chrome-profile');
8
9
  }
10
+ /** Where synced communities land unless -o says otherwise: one fixed place, whatever the cwd. */
11
+ export function defaultOutRoot() {
12
+ return join(homedir(), 'skrape');
13
+ }
9
14
  export function dbPath() {
10
15
  return join(ROOT, 'skool.db');
11
16
  }
@@ -17,6 +22,11 @@ export function dbPath() {
17
22
  export function chromeMarkerPath() {
18
23
  return join(ROOT, 'chrome-installed');
19
24
  }
25
+ /** Launch the browser first-run setup confirmed; before any setup ran, assume the user's Chrome. */
26
+ export function browserLaunchOptions() {
27
+ const executablePath = markedExecutablePath(chromeMarkerPath());
28
+ return executablePath ? { executablePath } : { channel: 'chrome' };
29
+ }
20
30
  export async function ensureRoot() {
21
31
  await mkdir(ROOT, { recursive: true });
22
32
  }
@@ -46,14 +56,12 @@ export async function login(onContext) {
46
56
  await ensureRoot();
47
57
  const { chromium } = await import('playwright');
48
58
  const context = await chromium.launchPersistentContext(profileDir(), {
49
- channel: 'chrome',
59
+ ...browserLaunchOptions(),
50
60
  headless: false,
51
61
  });
52
62
  onContext?.(context);
53
63
  const page = context.pages()[0] ?? (await context.newPage());
54
64
  await page.goto('https://www.skool.com/login', { waitUntil: 'domcontentloaded' });
55
- console.log('\nA browser window is open. Sign in to Skool there.');
56
- console.log('Waiting for you to reach a logged-in page (Ctrl+C to cancel)...\n');
57
65
  await page.waitForFunction(() => {
58
66
  const el = document.getElementById('__NEXT_DATA__');
59
67
  if (!el?.textContent)
@@ -65,6 +73,5 @@ export async function login(onContext) {
65
73
  return false;
66
74
  }
67
75
  }, undefined, { timeout: 0 });
68
- console.log('Signed in. Session saved — future runs will not need this.');
69
76
  await context.close();
70
77
  }
package/dist/cli.js CHANGED
@@ -1,14 +1,19 @@
1
1
  #!/usr/bin/env node
2
+ import { createRequire } from 'node:module';
3
+ import { join, resolve } from 'node:path';
4
+ import { styleText } from 'node:util';
2
5
  import { Command } from 'commander';
3
6
  import { HttpFetcher } from './fetch/http.js';
4
7
  import { BrowserFetcher } from './fetch/browser.js';
5
8
  import { ResilientFetcher } from './fetch/resilient.js';
6
9
  import { openDb } from './store/db.js';
7
10
  import { syncClassroom } from './sync.js';
8
- import { login, profileDir, dbPath, ensureRoot, isLoggedIn } from './auth/session.js';
9
- const MARKS = {
10
- ok: '+', skipped: '=', 'no-video': '.', 'no-access': '-', unavailable: '!', failed: 'x',
11
- };
11
+ import { formatSummary } from './tui/summary.js';
12
+ import { OUTCOME_STYLE, displayPath, formatDuration } from './tui/theme.js';
13
+ import { login, profileDir, dbPath, defaultOutRoot, ensureRoot, isLoggedIn } from './auth/session.js';
14
+ // Color only for a human at a terminal; pipes and NO_COLOR get plain text.
15
+ const useColor = Boolean(process.stdout.isTTY) && !process.env.NO_COLOR;
16
+ const paint = (color, text) => (useColor ? styleText(color, text) : text);
12
17
  const MAX_CONCURRENCY = 16;
13
18
  /** Parse the --concurrency flag to a positive integer, or return null if invalid. */
14
19
  function parseConcurrency(raw) {
@@ -18,19 +23,27 @@ function parseConcurrency(raw) {
18
23
  return Math.min(n, MAX_CONCURRENCY);
19
24
  }
20
25
  const program = new Command();
21
- program.name('skrape').description('Read Skool content instead of watching it');
26
+ const { version } = createRequire(import.meta.url)('../package.json');
27
+ program
28
+ .name('skrape')
29
+ .description('Read Skool course content instead of watching it')
30
+ .version(version, '-v, --version')
31
+ .addHelpText('after', '\nRun `skrape` with no arguments for the guided, interactive flow.');
22
32
  program
23
33
  .command('login')
24
34
  .description('Sign in to Skool once; the session is reused by later runs')
25
35
  .action(async () => {
36
+ console.log('A browser window is opening. Sign in to Skool there (Ctrl+C to cancel)…');
26
37
  await login();
38
+ console.log(`${paint('green', '✓')} Signed in. Future runs reuse this session.`);
27
39
  });
28
40
  program
29
41
  .command('sync')
30
42
  .argument('<slug>', 'community slug, e.g. demo from skool.com/demo')
31
- .option('-o, --out <dir>', 'output directory', './out')
43
+ .option('-o, --out <dir>', 'output directory', defaultOutRoot())
32
44
  .option('-c, --concurrency <n>', 'parallel requests', '4')
33
- .description('Pull a community classroom to disk as transcripts')
45
+ .option('--videos', 'also download every lesson video (needs yt-dlp)')
46
+ .description('Pull a community classroom to disk as transcripts (and optionally videos)')
34
47
  .action(async (slug, options) => {
35
48
  const concurrency = parseConcurrency(options.concurrency);
36
49
  if (concurrency === null) {
@@ -40,10 +53,12 @@ program
40
53
  return;
41
54
  }
42
55
  await ensureRoot();
43
- const outDir = `${options.out}/${slug}`;
56
+ const outDir = join(resolve(options.out), slug);
44
57
  const db = openDb(dbPath());
45
58
  const browser = new BrowserFetcher(profileDir());
46
59
  const fetcher = new ResilientFetcher(new HttpFetcher(), async () => browser, isLoggedIn);
60
+ const startedAt = Date.now();
61
+ console.log(`${paint('bold', '◆ skrape')} ${paint('dim', `syncing skool.com/${slug}${options.videos ? ' · with videos' : ''}`)}\n`);
47
62
  try {
48
63
  const summary = await syncClassroom({
49
64
  slug,
@@ -51,39 +66,35 @@ program
51
66
  db,
52
67
  fetcher,
53
68
  concurrency,
69
+ videos: options.videos,
54
70
  onProgress: (event) => {
55
- const mark = MARKS[event.outcome] ?? '?';
56
- console.log(`${mark} [${event.done}/${event.total}] ${event.course.slice(0, 28).padEnd(28)} ${event.title.slice(0, 50)}`);
71
+ const style = OUTCOME_STYLE[event.outcome];
72
+ const counter = `${event.done}/${event.total}`.padStart(String(event.total).length * 2 + 1);
73
+ console.log(`${paint(style.color, style.icon)} ${paint('dim', counter)} ${paint('dim', `${event.course.slice(0, 28)} ›`)} ${event.title.slice(0, 60)}`);
57
74
  },
58
75
  });
59
- console.log('\n--- summary ---');
60
- for (const [outcome, count] of Object.entries(summary.counts)) {
61
- if (count > 0)
62
- console.log(` ${outcome.padEnd(14)} ${count}`);
63
- }
64
- console.log(` ${'words'.padEnd(14)} ${summary.totalWords.toLocaleString('en-US')}`);
76
+ console.log('');
77
+ for (const line of formatSummary(summary, displayPath(outDir), paint))
78
+ console.log(line);
79
+ console.log(paint('dim', `Finished in ${formatDuration(Date.now() - startedAt)}`));
65
80
  if (fetcher.escalatedRoutes.size > 0) {
66
- console.log(`\n escalated to browser: ${[...fetcher.escalatedRoutes].join(', ')}`);
67
- }
68
- if (summary.problems.length > 0) {
69
- console.log('\n--- not transcribed ---');
70
- for (const problem of summary.problems) {
71
- console.log(` [${problem.outcome}] ${problem.course} / ${problem.title}: ${problem.reason}`);
72
- }
81
+ console.log(paint('dim', `Escalated to browser: ${[...fetcher.escalatedRoutes].join(', ')}`));
73
82
  }
74
- console.log(`\nOutput: ${outDir}`);
75
83
  }
76
84
  catch (error) {
77
- console.error(`\nSync failed: ${error.message}`);
78
- if (fetcher.escalatedRoutes.size > 0) {
85
+ console.error(`\n${paint('red', '✗ Sync failed:')} ${error.message}`);
86
+ if (error.message.includes('HTTP 404')) {
87
+ console.error(`No classroom at skool.com/${slug}. Check the slug (the part after skool.com/).`);
88
+ }
89
+ else if (fetcher.escalatedRoutes.size > 0) {
79
90
  console.error(`\n escalated to browser: ${[...fetcher.escalatedRoutes].join(', ')}`);
80
91
  console.error('If this mentions an unexpected payload, the authenticated browser path was already tried ' +
81
92
  "for the route(s) above, so Skool's page structure most likely changed rather than the " +
82
- 'session being expired. A fresh `skool login` is still worth trying, but treat it as a ' +
93
+ 'session being expired. A fresh `skrape login` is still worth trying, but treat it as a ' +
83
94
  'secondary guess.');
84
95
  }
85
96
  else {
86
- console.error('If this mentions an unexpected payload, your session may have expired — run: skool login. ' +
97
+ console.error('If this mentions an unexpected payload, your session may have expired. Run: skrape login. ' +
87
98
  "It is also possible Skool's page structure changed; if a retry after login fails the same " +
88
99
  'way, that is more likely.');
89
100
  }
@@ -1,3 +1,4 @@
1
+ import { browserLaunchOptions } from '../auth/session.js';
1
2
  /**
2
3
  * Playwright-backed fallback. Playwright is imported dynamically so that users
3
4
  * who never hit the fallback never pay its startup cost — and so the package
@@ -13,14 +14,14 @@ export class BrowserFetcher {
13
14
  }
14
15
  async ensureContext() {
15
16
  if (this.closed) {
16
- throw new Error('BrowserFetcher is closed — cannot open a new context after close()');
17
+ throw new Error('BrowserFetcher is closed, cannot open a new context after close()');
17
18
  }
18
19
  if (!this.context) {
19
20
  try {
20
21
  this.contextPromise ??= (async () => {
21
22
  const { chromium } = await import('playwright');
22
23
  return await chromium.launchPersistentContext(this.profileDir, {
23
- channel: 'chrome',
24
+ ...browserLaunchOptions(),
24
25
  headless: true,
25
26
  });
26
27
  })();
@@ -15,6 +15,8 @@
15
15
  * A missing/unreadable/empty marker is just treated as "not installed".
16
16
  */
17
17
  export declare function isChromeMarkedInstalled(markerPath: string): boolean;
18
+ /** The browser executable a past run confirmed, if the marker exists and that path still does. */
19
+ export declare function markedExecutablePath(markerPath: string): string | null;
18
20
  /** Records the confirmed Chrome executable's path, so later runs can skip
19
21
  * the probe by cheaply checking that path still exists on disk. */
20
22
  export declare function markChromeInstalled(markerPath: string, executablePath: string): Promise<void>;
@@ -37,6 +39,8 @@ export declare function markChromeInstalled(markerPath: string, executablePath:
37
39
  * run) never pays Playwright's module-load cost at all.
38
40
  */
39
41
  export declare function probeChromeLaunchable(): Promise<string | undefined>;
42
+ /** Which browser to install when none is launchable: Chrome where that needs no root, Chromium on Linux. */
43
+ export declare const INSTALL_TARGET: string;
40
44
  /**
41
45
  * Runs the equivalent of `npx playwright install chrome` in-process, by
42
46
  * spawning Playwright's own bundled CLI script directly. Preferred over
@@ -20,16 +20,20 @@ import { fileURLToPath } from 'node:url';
20
20
  * A missing/unreadable/empty marker is just treated as "not installed".
21
21
  */
22
22
  export function isChromeMarkedInstalled(markerPath) {
23
+ return markedExecutablePath(markerPath) !== null;
24
+ }
25
+ /** The browser executable a past run confirmed, if the marker exists and that path still does. */
26
+ export function markedExecutablePath(markerPath) {
23
27
  if (!existsSync(markerPath))
24
- return false;
28
+ return null;
25
29
  let storedPath;
26
30
  try {
27
31
  storedPath = readFileSync(markerPath, 'utf8').trim();
28
32
  }
29
33
  catch {
30
- return false;
34
+ return null;
31
35
  }
32
- return storedPath.length > 0 && existsSync(storedPath);
36
+ return storedPath.length > 0 && existsSync(storedPath) ? storedPath : null;
33
37
  }
34
38
  /** Records the confirmed Chrome executable's path, so later runs can skip
35
39
  * the probe by cheaply checking that path still exists on disk. */
@@ -55,20 +59,36 @@ export async function markChromeInstalled(markerPath, executablePath) {
55
59
  * run) never pays Playwright's module-load cost at all.
56
60
  */
57
61
  export async function probeChromeLaunchable() {
62
+ let chromium;
58
63
  try {
59
- const { chromium } = await import('playwright');
60
- const server = await chromium.launchServer({ channel: 'chrome', headless: true });
61
- try {
62
- return server.process().spawnfile;
63
- }
64
- finally {
65
- await server.close();
66
- }
64
+ ({ chromium } = await import('playwright'));
67
65
  }
68
66
  catch {
69
67
  return undefined;
70
68
  }
69
+ // Prefer the user's real Chrome; fall back to Playwright's own Chromium (what gets
70
+ // installed on Linux, where installing Chrome itself needs root).
71
+ const candidates = [{ channel: 'chrome' }];
72
+ if (existsSync(chromium.executablePath()))
73
+ candidates.push({ executablePath: chromium.executablePath() });
74
+ for (const options of candidates) {
75
+ try {
76
+ const server = await chromium.launchServer({ ...options, headless: true });
77
+ try {
78
+ return server.process().spawnfile;
79
+ }
80
+ finally {
81
+ await server.close();
82
+ }
83
+ }
84
+ catch {
85
+ // try the next candidate
86
+ }
87
+ }
88
+ return undefined;
71
89
  }
90
+ /** Which browser to install when none is launchable: Chrome where that needs no root, Chromium on Linux. */
91
+ export const INSTALL_TARGET = process.platform === 'linux' ? 'chromium' : 'chrome';
72
92
  /** Resolves the on-disk path to Playwright's own bundled CLI script, via the
73
93
  * package's exported `./package.json` subpath (not `./cli.js`, which isn't
74
94
  * in Playwright's `exports` map and so can't be resolved directly) — this
@@ -94,7 +114,7 @@ async function resolvePlaywrightCliPath() {
94
114
  export async function installChromeViaCli(onProgress) {
95
115
  const cliPath = await resolvePlaywrightCliPath();
96
116
  await new Promise((resolve, reject) => {
97
- const child = spawn(process.execPath, [cliPath, 'install', 'chrome'], {
117
+ const child = spawn(process.execPath, [cliPath, 'install', INSTALL_TARGET], {
98
118
  stdio: ['ignore', 'pipe', 'pipe'],
99
119
  });
100
120
  let stdout = '';
@@ -0,0 +1,12 @@
1
+ export declare function hasYtDlp(): Promise<boolean>;
2
+ export type DownloadResult = {
3
+ status: 'downloaded' | 'exists';
4
+ } | {
5
+ status: 'failed';
6
+ reason: string;
7
+ };
8
+ /**
9
+ * Download one lesson video to `destPath` (an .mp4). An existing file is treated as complete:
10
+ * yt-dlp writes to a .part file and only renames on success, so a partial never lands at destPath.
11
+ */
12
+ export declare function downloadVideo(videoUrl: string, destPath: string): Promise<DownloadResult>;
@@ -0,0 +1,54 @@
1
+ import { spawn } from 'node:child_process';
2
+ import { access, mkdir } from 'node:fs/promises';
3
+ import { dirname } from 'node:path';
4
+ /** Run yt-dlp, resolving with its exit code and the tail of stderr. Never rejects on a bad exit. */
5
+ function run(args) {
6
+ return new Promise((resolve, reject) => {
7
+ const child = spawn(process.env.SKRAPE_YT_DLP ?? 'yt-dlp', args, { stdio: ['ignore', 'ignore', 'pipe'] });
8
+ let stderr = '';
9
+ child.stderr.on('data', (chunk) => {
10
+ stderr = (stderr + chunk.toString()).slice(-2000);
11
+ });
12
+ child.on('error', reject);
13
+ child.on('close', (code) => resolve({ code: code ?? 1, stderr }));
14
+ });
15
+ }
16
+ export async function hasYtDlp() {
17
+ try {
18
+ return (await run(['--version'])).code === 0;
19
+ }
20
+ catch {
21
+ return false;
22
+ }
23
+ }
24
+ /**
25
+ * Download one lesson video to `destPath` (an .mp4). An existing file is treated as complete:
26
+ * yt-dlp writes to a .part file and only renames on success, so a partial never lands at destPath.
27
+ */
28
+ export async function downloadVideo(videoUrl, destPath) {
29
+ try {
30
+ await access(destPath);
31
+ return { status: 'exists' };
32
+ }
33
+ catch {
34
+ // not there yet — download it
35
+ }
36
+ await mkdir(dirname(destPath), { recursive: true });
37
+ try {
38
+ const { code, stderr } = await run([
39
+ '--quiet', '--no-progress', '--no-playlist',
40
+ '--referer', 'https://www.skool.com/',
41
+ '--merge-output-format', 'mp4', '--remux-video', 'mp4',
42
+ // Let yt-dlp pick the source extension; --remux-video lands the final file at destPath.
43
+ '-o', destPath.replace(/\.mp4$/, '.%(ext)s'),
44
+ videoUrl,
45
+ ]);
46
+ if (code === 0)
47
+ return { status: 'downloaded' };
48
+ const lastError = stderr.split('\n').filter((line) => line.includes('ERROR')).pop();
49
+ return { status: 'failed', reason: (lastError ?? stderr.trim().split('\n').pop() ?? `yt-dlp exit ${code}`).trim() };
50
+ }
51
+ catch (error) {
52
+ return { status: 'failed', reason: error.message };
53
+ }
54
+ }
@@ -1,6 +1,7 @@
1
1
  import type { ContentItem } from '../types.js';
2
2
  export declare function slugify(text: string, maxLength?: number): string;
3
3
  export declare function transcriptPath(outDir: string, item: ContentItem, padWidth?: number): string;
4
+ export declare function videoPath(outDir: string, item: ContentItem, padWidth?: number): string;
4
5
  export declare function writeTranscript(outDir: string, item: ContentItem, body: string, meta: {
5
6
  wordCount: number;
6
7
  sourceUrl: string;
@@ -12,18 +12,24 @@ export function slugify(text, maxLength = 60) {
12
12
  .replace(/-+$/g, '');
13
13
  return cleaned || 'untitled';
14
14
  }
15
- export function transcriptPath(outDir, item, padWidth = 2) {
15
+ function lessonPath(outDir, kind, ext, item, padWidth) {
16
16
  const course = slugify(item.course ?? 'uncategorized');
17
17
  const width = Math.max(2, padWidth);
18
- const name = `${String(item.index).padStart(width, '0')}-${slugify(item.title)}.md`;
19
- return join(outDir, 'transcripts', course, name);
18
+ const name = `${String(item.index).padStart(width, '0')}-${slugify(item.title)}.${ext}`;
19
+ return join(outDir, kind, course, name);
20
+ }
21
+ export function transcriptPath(outDir, item, padWidth = 2) {
22
+ return lessonPath(outDir, 'transcripts', 'md', item, padWidth);
23
+ }
24
+ export function videoPath(outDir, item, padWidth = 2) {
25
+ return lessonPath(outDir, 'videos', 'mp4', item, padWidth);
20
26
  }
21
27
  export async function writeTranscript(outDir, item, body, meta, padWidth = 2) {
22
28
  const path = transcriptPath(outDir, item, padWidth);
23
29
  await mkdir(dirname(path), { recursive: true });
24
30
  const minutes = Math.round(item.durationMs / 60_000);
25
31
  const cleanTitle = item.title.replace(/\s+/g, ' ');
26
- const cleanCourse = (item.course ?? '—').replace(/\s+/g, ' ');
32
+ const cleanCourse = (item.course ?? '-').replace(/\s+/g, ' ');
27
33
  const cleanSection = item.section?.replace(/\s+/g, ' ');
28
34
  const header = [
29
35
  `# ${cleanTitle}`,
package/dist/sync.d.ts CHANGED
@@ -18,6 +18,13 @@ export interface SyncSummary {
18
18
  counts: Record<Outcome, number>;
19
19
  problems: Problem[];
20
20
  totalWords: number;
21
+ /** Present only when the sync was asked to download videos. */
22
+ videos?: VideoCounts;
23
+ }
24
+ export interface VideoCounts {
25
+ downloaded: number;
26
+ existing: number;
27
+ failed: number;
21
28
  }
22
29
  export interface SyncOptions {
23
30
  slug: string;
@@ -25,6 +32,8 @@ export interface SyncOptions {
25
32
  db: Db;
26
33
  fetcher: Fetcher;
27
34
  concurrency?: number;
35
+ /** Also download each lesson's video (needs yt-dlp on PATH). */
36
+ videos?: boolean;
28
37
  onProgress?: (event: ProgressEvent) => void;
29
38
  }
30
39
  export declare function syncClassroom(options: SyncOptions): Promise<SyncSummary>;
package/dist/sync.js CHANGED
@@ -1,6 +1,8 @@
1
1
  import { listCourses, listLessons } from './discover/skool.js';
2
2
  import { getTranscript } from './media/index.js';
3
- import { writeTranscript } from './store/markdown.js';
3
+ import { downloadVideo, hasYtDlp } from './media/download.js';
4
+ import { existsSync } from 'node:fs';
5
+ import { transcriptPath, videoPath, writeTranscript } from './store/markdown.js';
4
6
  /** Run tasks with a bounded number in flight. */
5
7
  async function pooled(tasks, limit) {
6
8
  const results = new Array(tasks.length);
@@ -31,7 +33,12 @@ function errorMessage(error) {
31
33
  }
32
34
  }
33
35
  export async function syncClassroom(options) {
34
- const { slug, outDir, db, fetcher, concurrency = 4, onProgress } = options;
36
+ const { slug, outDir, db, fetcher, concurrency = 4, videos = false, onProgress } = options;
37
+ if (videos && !(await hasYtDlp())) {
38
+ throw new Error('Video downloads need yt-dlp, which was not found on your PATH. ' +
39
+ 'Install it (macOS: brew install yt-dlp ffmpeg; others: https://github.com/yt-dlp/yt-dlp#installation) and try again.');
40
+ }
41
+ const videoCounts = videos ? { downloaded: 0, existing: 0, failed: 0 } : undefined;
35
42
  const counts = {
36
43
  ok: 0, skipped: 0, 'no-video': 0, 'no-access': 0, unavailable: 0, failed: 0,
37
44
  };
@@ -44,7 +51,7 @@ export async function syncClassroom(options) {
44
51
  counts['no-access']++;
45
52
  problems.push({
46
53
  outcome: 'no-access', course: locked.title, title: '(entire course)',
47
- reason: 'hasAccess is 0 — not entitled, skipped without probing',
54
+ reason: 'hasAccess is 0, not entitled, skipped without probing',
48
55
  });
49
56
  }
50
57
  // One course's listing failing (network blip, malformed payload) must not take down the whole
@@ -90,12 +97,12 @@ export async function syncClassroom(options) {
90
97
  const record = (outcome, item, reason) => {
91
98
  counts[outcome]++;
92
99
  if (reason) {
93
- problems.push({ outcome, course: item.course ?? '—', title: item.title, reason });
100
+ problems.push({ outcome, course: item.course ?? '-', title: item.title, reason });
94
101
  }
95
102
  done++;
96
103
  // Progress reporting is cosmetic. A caller's callback throwing must never fail the sync.
97
104
  try {
98
- onProgress?.({ outcome, course: item.course ?? '—', title: item.title, done, total });
105
+ onProgress?.({ outcome, course: item.course ?? '-', title: item.title, done, total });
99
106
  }
100
107
  catch {
101
108
  // swallow — the caller's callback is not our concern
@@ -129,6 +136,20 @@ export async function syncClassroom(options) {
129
136
  return record('no-access', item, 'lesson locked');
130
137
  if (!item.videoUrl)
131
138
  return record('no-video', item, 'lesson has no video attached');
139
+ // Download before the transcript step so an already-transcribed lesson still gets its video.
140
+ if (videoCounts) {
141
+ const download = await downloadVideo(item.videoUrl, videoPath(outDir, item, padWidth));
142
+ if (download.status === 'failed') {
143
+ videoCounts.failed++;
144
+ problems.push({
145
+ outcome: 'failed', course: item.course ?? '-', title: item.title,
146
+ reason: `video download: ${download.reason}`,
147
+ });
148
+ }
149
+ else {
150
+ videoCounts[download.status === 'downloaded' ? 'downloaded' : 'existing']++;
151
+ }
152
+ }
132
153
  let status;
133
154
  try {
134
155
  status = db.getTranscriptStatus(id);
@@ -136,7 +157,8 @@ export async function syncClassroom(options) {
136
157
  catch (error) {
137
158
  return record('failed', item, `could not read transcript status: ${errorMessage(error)}`);
138
159
  }
139
- if (status === 'ok')
160
+ // 'ok' in the db only means it was written *somewhere*; skip only when the file is in this outDir.
161
+ if (status === 'ok' && existsSync(transcriptPath(outDir, item, padWidth)))
140
162
  return record('skipped', item);
141
163
  let result;
142
164
  try {
@@ -174,5 +196,5 @@ export async function syncClassroom(options) {
174
196
  return record(result.status, item, result.reason);
175
197
  }), concurrency);
176
198
  db.markSynced(slug);
177
- return { counts, problems, totalWords };
199
+ return { counts, problems, totalWords, videos: videoCounts };
178
200
  }
package/dist/tui/App.d.ts CHANGED
@@ -2,13 +2,15 @@ import React from 'react';
2
2
  import type { CommunityRef } from '../discover/communities.js';
3
3
  import type { ProgressEvent, SyncSummary } from '../sync.js';
4
4
  export interface AppControllers {
5
- /** Output root directory, e.g. './out' — the community slug is appended. */
5
+ /** Absolute output root, e.g. ~/skrape; the community slug is appended. */
6
6
  outRoot: string;
7
7
  checkLoggedIn: () => Promise<boolean>;
8
8
  login: () => Promise<void>;
9
9
  discoverCommunities: () => Promise<CommunityRef[] | null>;
10
10
  countAccessibleCourses: (slug: string) => Promise<number>;
11
- runSync: (slug: string, outDir: string, onProgress: (event: ProgressEvent) => void) => Promise<SyncSummary>;
11
+ /** Checked when the user picks videos, so installing it in another terminal works without a restart. */
12
+ hasYtDlp: () => Promise<boolean>;
13
+ runSync: (slug: string, outDir: string, onProgress: (event: ProgressEvent) => void, videos: boolean) => Promise<SyncSummary>;
12
14
  }
13
15
  /**
14
16
  * Caps error text shown in the TUI. Some failures (e.g. a raw Playwright
package/dist/tui/App.js CHANGED
@@ -3,7 +3,8 @@ import React from 'react';
3
3
  import { Box, Text, useApp, useInput } from 'ink';
4
4
  import { SelectList } from './SelectList.js';
5
5
  import { applyProgress, createProgressState, formatProgressBar } from './progress.js';
6
- import { formatSummary } from './summary.js';
6
+ import { formatVideoLine, nothingTranscribed, sortedProblems } from './summary.js';
7
+ import { ACCENT, OUTCOME_STYLE, displayPath, estimateRemainingMs, formatDuration } from './theme.js';
7
8
  import { isValidSlug, normalizeSlug } from './slug.js';
8
9
  import { MENU_ITEMS, nextStepForMenuChoice } from './flow.js';
9
10
  import { revealOutputFolder } from './reveal.js';
@@ -35,8 +36,46 @@ function ConfirmPrompt({ onConfirm }) {
35
36
  else if (input === 'q' || key.escape)
36
37
  exit();
37
38
  });
38
- return _jsx(Text, { dimColor: true, children: "[enter] continue [q] quit" });
39
+ return _jsx(Text, { dimColor: true, children: "enter continue \u00B7 q quit" });
39
40
  }
41
+ function Header() {
42
+ return (_jsxs(Box, { marginBottom: 1, children: [_jsx(Text, { color: ACCENT, bold: true, children: "\u25C6 skrape" }), _jsx(Text, { dimColor: true, children: " read the course, skip the video" })] }));
43
+ }
44
+ const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
45
+ function Spinner({ label }) {
46
+ const [frame, setFrame] = React.useState(0);
47
+ React.useEffect(() => {
48
+ const timer = setInterval(() => setFrame((current) => (current + 1) % SPINNER_FRAMES.length), 80);
49
+ return () => clearInterval(timer);
50
+ }, []);
51
+ return (_jsxs(Text, { children: [_jsx(Text, { color: ACCENT, children: SPINNER_FRAMES[frame] }), " ", label] }));
52
+ }
53
+ /** Label/value rows inside a rounded card, labels aligned. */
54
+ function Card({ rows }) {
55
+ const width = Math.max(...rows.map(([label]) => label.length));
56
+ return (_jsx(Box, { flexDirection: "column", borderStyle: "round", borderColor: "gray", paddingX: 1, alignSelf: "flex-start", children: rows.map(([label, value]) => (_jsxs(Box, { children: [_jsx(Text, { dimColor: true, children: label.padEnd(width + 3) }), _jsx(Text, { children: value })] }, label))) }));
57
+ }
58
+ function OutcomeChips({ counts, always = [] }) {
59
+ const shown = Object.keys(OUTCOME_STYLE).filter((o) => counts[o] > 0 || always.includes(o));
60
+ return (_jsx(Text, { children: shown.map((outcome, index) => (_jsxs(Text, { children: [index > 0 && ' ', _jsx(Text, { color: OUTCOME_STYLE[outcome].color, children: OUTCOME_STYLE[outcome].icon }), " ", counts[outcome], ' ', _jsx(Text, { dimColor: true, children: OUTCOME_STYLE[outcome].label })] }, outcome))) }));
61
+ }
62
+ /** Ticks once a second on its own so the rest of the screen only re-renders on progress. */
63
+ function Timing({ startedAt, done, total }) {
64
+ const [now, setNow] = React.useState(Date.now());
65
+ React.useEffect(() => {
66
+ const timer = setInterval(() => setNow(Date.now()), 1000);
67
+ return () => clearInterval(timer);
68
+ }, []);
69
+ const elapsed = now - startedAt;
70
+ const remaining = estimateRemainingMs(elapsed, done, total);
71
+ return (_jsxs(Text, { dimColor: true, children: [formatDuration(elapsed), " elapsed", remaining !== null && ` · ~${formatDuration(remaining)} left`] }));
72
+ }
73
+ const YT_DLP_MISSING = 'Video downloads need yt-dlp, which isn\'t installed. In another terminal run: brew install yt-dlp ffmpeg (or see github.com/yt-dlp/yt-dlp), then pick again.';
74
+ const MODE_ITEMS = [
75
+ { label: 'Transcripts', value: 'transcripts', hint: 'fast · text only' },
76
+ { label: 'Transcripts + videos', value: 'videos', hint: 'needs yt-dlp · large download' },
77
+ { label: 'Quit', value: 'quit' },
78
+ ];
40
79
  function ExitPrompt({ label }) {
41
80
  const { exit } = useApp();
42
81
  useInput(() => exit());
@@ -166,16 +205,16 @@ export function App({ controllers }) {
166
205
  if (step.kind !== 'syncing')
167
206
  return;
168
207
  let cancelled = false;
169
- const { slug, name, courseCount, outDir } = step;
208
+ const { slug, name, courseCount, outDir, videos, startedAt } = step;
170
209
  controllers
171
210
  .runSync(slug, outDir, (event) => {
172
211
  if (cancelled)
173
212
  return;
174
213
  setStep((previous) => previous.kind === 'syncing' ? { ...previous, progress: applyProgress(previous.progress, event) } : previous);
175
- })
214
+ }, videos)
176
215
  .then((summary) => {
177
216
  if (!cancelled)
178
- setStep({ kind: 'done', slug, name, courseCount, outDir, summary });
217
+ setStep({ kind: 'done', slug, name, courseCount, outDir, elapsedMs: Date.now() - startedAt, summary });
179
218
  })
180
219
  .catch((error) => {
181
220
  if (!cancelled)
@@ -218,68 +257,105 @@ export function App({ controllers }) {
218
257
  });
219
258
  }
220
259
  }, [exit]);
221
- switch (step.kind) {
222
- case 'checking-session':
223
- return _jsx(Text, { children: "Checking your Skool session\u2026" });
224
- case 'login-needed':
225
- return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { color: "yellow", children: "You are not signed in to Skool." }), _jsx(Text, { dimColor: true, children: "Nothing is typed for you \u2014 a browser window opens and you sign in yourself." }), _jsx(ConfirmPrompt, { onConfirm: () => setStep({ kind: 'logging-in' }) })] }));
226
- case 'logging-in':
227
- return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { children: "A browser window is open. Sign in there." }), _jsx(Text, { dimColor: true, children: "Waiting for you to finish (Ctrl+C to cancel)\u2026" })] }));
228
- case 'discovering':
229
- return _jsx(Text, { children: "Looking up the communities you belong to\u2026" });
230
- case 'picking': {
231
- const items = [
232
- ...step.communities.map((community) => ({
233
- label: `${community.name} (skool.com/${community.slug})`,
234
- value: community,
235
- })),
236
- { label: 'Enter a slug manually…', value: null },
237
- ];
238
- return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { children: "Which community?" }), _jsx(SelectList, { items: items, onSelect: (value) => {
239
- if (value === null)
240
- setStep({ kind: 'manual-slug', value: '', error: null });
241
- else
242
- chooseCommunity(value.slug, value.name);
243
- } })] }));
244
- }
245
- case 'manual-slug':
246
- return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { children: "Enter the community slug \u2014 the part after skool.com/" }), _jsxs(Text, { children: ["Slug: ", step.value, _jsx(Text, { color: "gray", children: "\u2588" })] }), step.error && _jsx(Text, { color: "red", children: step.error }), _jsx(ManualSlugInput, { value: step.value, onChange: (value) => setStep({ kind: 'manual-slug', value, error: null }), onSubmit: (value) => {
247
- const slug = normalizeSlug(value);
248
- if (!isValidSlug(slug)) {
249
- setStep({ kind: 'manual-slug', value, error: `"${value}" doesn't look like a valid slug.` });
250
- return;
251
- }
252
- chooseCommunity(slug, slug);
253
- } })] }));
254
- case 'counting':
255
- return _jsxs(Text, { children: ["Looking up courses in ", step.name, "\u2026"] });
256
- case 'confirming':
257
- return (_jsxs(Box, { flexDirection: "column", children: [_jsxs(Text, { children: ["Community: ", step.name, " (skool.com/", step.slug, ")"] }), _jsxs(Text, { children: ["Courses found: ", step.courseCount] }), _jsxs(Text, { children: ["Output will be written to: ", step.outDir] }), _jsx(Text, { children: " " }), _jsx(ConfirmPrompt, { onConfirm: () => setStep({
258
- kind: 'syncing',
259
- slug: step.slug,
260
- name: step.name,
261
- courseCount: step.courseCount,
262
- outDir: step.outDir,
263
- progress: createProgressState(0),
264
- }) })] }));
265
- case 'syncing': {
266
- const { progress } = step;
267
- return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { children: formatProgressBar(progress.done, progress.total) }), progress.current && (_jsxs(Text, { dimColor: true, children: ["current: ", progress.current.course, " / ", progress.current.title] })), _jsxs(Text, { dimColor: true, children: ["ok ", progress.counts.ok, " skipped ", progress.counts.skipped, " no-video ", progress.counts['no-video'], ' ', "no-access ", progress.counts['no-access'], " unavailable ", progress.counts.unavailable, " failed", ' ', progress.counts.failed] })] }));
268
- }
269
- case 'done': {
270
- const lines = formatSummary(step.summary, step.outDir);
271
- const done = {
272
- slug: step.slug,
273
- name: step.name,
274
- courseCount: step.courseCount,
275
- outDir: step.outDir,
276
- summary: step.summary,
277
- };
278
- return (_jsxs(Box, { flexDirection: "column", children: [lines.map((line, index) => (
279
- // eslint-disable-next-line react/no-array-index-key
280
- _jsx(Text, { children: line || ' ' }, index))), _jsx(Text, { children: " " }), _jsx(Text, { children: "What next?" }), _jsx(SelectList, { items: MENU_ITEMS, onSelect: (choice) => handleMenuChoice(done, choice) })] }));
260
+ return (_jsxs(Box, { flexDirection: "column", paddingX: 1, paddingY: 1, children: [_jsx(Header, {}), renderStep()] }));
261
+ function renderStep() {
262
+ switch (step.kind) {
263
+ case 'checking-session':
264
+ return _jsx(Spinner, { label: "Checking your Skool session\u2026" });
265
+ case 'login-needed':
266
+ return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { color: "yellow", children: "You're not signed in to Skool yet." }), _jsx(Text, { dimColor: true, children: "A browser window will open and you sign in yourself. skrape never sees your password." }), _jsx(Text, { children: " " }), _jsx(ConfirmPrompt, { onConfirm: () => setStep({ kind: 'logging-in' }) })] }));
267
+ case 'logging-in':
268
+ return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Spinner, { label: "Waiting for you to sign in in the browser window\u2026" }), _jsx(Text, { dimColor: true, children: "Ctrl+C to cancel" })] }));
269
+ case 'discovering':
270
+ return _jsx(Spinner, { label: "Finding your communities\u2026" });
271
+ case 'picking': {
272
+ const items = [
273
+ ...step.communities.map((community) => ({
274
+ label: community.name,
275
+ hint: `skool.com/${community.slug}`,
276
+ value: community,
277
+ })),
278
+ { label: 'Enter a slug manually…', value: null },
279
+ ];
280
+ return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { bold: true, children: "Which community?" }), _jsx(Text, { children: " " }), _jsx(SelectList, { items: items, onSelect: (value) => {
281
+ if (value === null)
282
+ setStep({ kind: 'manual-slug', value: '', error: null });
283
+ else
284
+ chooseCommunity(value.slug, value.name);
285
+ } })] }));
286
+ }
287
+ case 'manual-slug':
288
+ return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Text, { bold: true, children: "Community slug" }), _jsxs(Text, { dimColor: true, children: ["The part after skool.com/, e.g. skool.com/", _jsx(Text, { color: ACCENT, children: "my-community" })] }), _jsx(Text, { children: " " }), _jsxs(Text, { children: [_jsx(Text, { color: ACCENT, children: "\u276F " }), step.value, _jsx(Text, { color: ACCENT, children: "\u258C" })] }), step.error && _jsx(Text, { color: "red", children: step.error }), _jsx(ManualSlugInput, { value: step.value, onChange: (value) => setStep({ kind: 'manual-slug', value, error: null }), onSubmit: (value) => {
289
+ const slug = normalizeSlug(value);
290
+ if (!isValidSlug(slug)) {
291
+ setStep({ kind: 'manual-slug', value, error: `"${value}" doesn't look like a valid slug.` });
292
+ return;
293
+ }
294
+ chooseCommunity(slug, slug);
295
+ } })] }));
296
+ case 'counting':
297
+ return _jsx(Spinner, { label: `Looking up courses in ${step.name}…` });
298
+ case 'confirming': {
299
+ const { slug, name, courseCount, outDir } = step;
300
+ return (_jsxs(Box, { flexDirection: "column", children: [_jsx(Card, { rows: [
301
+ ['Community', _jsx(Text, { bold: true, children: name })],
302
+ ['URL', `skool.com/${slug}`],
303
+ ['Courses', `${courseCount} you can access`],
304
+ ['Saving to', displayPath(outDir)],
305
+ ] }), _jsx(Text, { children: " " }), _jsx(Text, { bold: true, children: "What should skrape pull?" }), _jsx(Text, { children: " " }), _jsx(SelectList, { items: MODE_ITEMS, onSelect: (mode) => {
306
+ if (mode === 'quit') {
307
+ exit();
308
+ return;
309
+ }
310
+ const start = () => setStep({
311
+ kind: 'syncing', slug, name, courseCount, outDir,
312
+ videos: mode === 'videos',
313
+ startedAt: Date.now(),
314
+ progress: createProgressState(0),
315
+ });
316
+ if (mode !== 'videos') {
317
+ start();
318
+ return;
319
+ }
320
+ controllers
321
+ .hasYtDlp()
322
+ .catch(() => false)
323
+ .then((ok) => {
324
+ if (ok)
325
+ start();
326
+ else
327
+ setStep({ ...step, notice: YT_DLP_MISSING });
328
+ });
329
+ } }), step.notice && (_jsx(Box, { marginTop: 1, children: _jsx(Text, { color: "yellow", children: step.notice }) }))] }));
330
+ }
331
+ case 'syncing': {
332
+ const { progress } = step;
333
+ return (_jsxs(Box, { flexDirection: "column", children: [_jsxs(Text, { children: ["Syncing ", _jsx(Text, { bold: true, children: step.name }), _jsx(Text, { dimColor: true, children: step.videos ? ' · transcripts + videos' : ' · transcripts' })] }), _jsx(Text, { children: " " }), _jsx(Text, { color: ACCENT, children: formatProgressBar(progress.done, progress.total, 32) }), progress.current ? (_jsx(Spinner, { label: `${progress.current.course} › ${progress.current.title}` })) : (_jsx(Spinner, { label: "Reading the course tree\u2026" })), _jsx(Text, { children: " " }), _jsx(OutcomeChips, { counts: progress.counts, always: ['ok'] }), _jsx(Timing, { startedAt: step.startedAt, done: progress.done, total: progress.total })] }));
334
+ }
335
+ case 'done': {
336
+ const { summary } = step;
337
+ const empty = nothingTranscribed(summary);
338
+ const videoLine = formatVideoLine(summary);
339
+ const done = {
340
+ slug: step.slug,
341
+ name: step.name,
342
+ courseCount: step.courseCount,
343
+ outDir: step.outDir,
344
+ summary,
345
+ };
346
+ const rows = [
347
+ ['Words', summary.totalWords.toLocaleString('en-US')],
348
+ ['Lessons', _jsx(OutcomeChips, { counts: summary.counts })],
349
+ ];
350
+ if (videoLine)
351
+ rows.push(['Videos', videoLine]);
352
+ rows.push(['Saved to', _jsx(Text, { color: ACCENT, children: displayPath(step.outDir) })]);
353
+ return (_jsxs(Box, { flexDirection: "column", children: [empty && summary.counts.skipped === 0 ? (_jsx(Text, { color: "yellow", bold: true, children: "\u26A0 Nothing was transcribed." })) : (_jsxs(Text, { color: "green", bold: true, children: ["\u2713 ", empty ? `${step.name} is up to date` : `Synced ${step.name}`, _jsxs(Text, { dimColor: true, bold: false, children: [" in ", formatDuration(step.elapsedMs)] })] })), _jsx(Text, { children: " " }), _jsx(Card, { rows: rows }), summary.problems.length > 0 && (_jsxs(Box, { flexDirection: "column", marginTop: 1, children: [_jsxs(Text, { bold: true, children: ["Needs attention (", summary.problems.length, ")"] }), sortedProblems(summary).map((problem, index) => (
354
+ // eslint-disable-next-line react/no-array-index-key
355
+ _jsxs(Text, { wrap: "truncate-end", children: [_jsx(Text, { color: OUTCOME_STYLE[problem.outcome].color, children: OUTCOME_STYLE[problem.outcome].icon }), ' ', problem.course, " \u203A ", problem.title, " ", _jsx(Text, { dimColor: true, children: problem.reason })] }, index)))] })), _jsx(Text, { children: " " }), _jsx(Text, { bold: true, children: "What next?" }), _jsx(Text, { children: " " }), _jsx(SelectList, { items: MENU_ITEMS, onSelect: (choice) => handleMenuChoice(done, choice) })] }));
356
+ }
357
+ case 'error':
358
+ return (_jsxs(Box, { flexDirection: "column", borderStyle: "round", borderColor: "red", paddingX: 1, alignSelf: "flex-start", children: [_jsx(Text, { color: "red", bold: true, children: "\u2717 Something went wrong" }), _jsx(Text, { children: step.message }), _jsx(Text, { children: " " }), _jsx(ExitPrompt, { label: "Press any key to exit." })] }));
281
359
  }
282
- case 'error':
283
- return (_jsxs(Box, { flexDirection: "column", children: [_jsxs(Text, { color: "red", children: ["Something went wrong: ", step.message] }), _jsx(ExitPrompt, { label: "Press any key to exit." })] }));
284
360
  }
285
361
  }
@@ -2,6 +2,8 @@ import React from 'react';
2
2
  export interface SelectItem<T> {
3
3
  label: string;
4
4
  value: T;
5
+ /** Dim text shown after the label. */
6
+ hint?: string;
5
7
  }
6
8
  interface SelectListProps<T> {
7
9
  items: Array<SelectItem<T>>;
@@ -1,6 +1,7 @@
1
- import { jsxs as _jsxs, jsx as _jsx } from "react/jsx-runtime";
1
+ import { jsx as _jsx, jsxs as _jsxs } from "react/jsx-runtime";
2
2
  import React from 'react';
3
3
  import { Box, Text, useInput } from 'ink';
4
+ import { ACCENT } from './theme.js';
4
5
  /** Minimal arrow-key list, built directly on Ink's useInput rather than pulling
5
6
  * in a select-input dependency. Cheap to render: a handful of Text lines,
6
7
  * re-rendered only on cursor movement or selection. */
@@ -21,5 +22,9 @@ export function SelectList({ items, onSelect }) {
21
22
  onSelect(item.value);
22
23
  }
23
24
  });
24
- return (_jsx(Box, { flexDirection: "column", children: items.map((item, index) => (_jsxs(Text, { color: index === cursor ? 'cyan' : undefined, children: [index === cursor ? '> ' : ' ', item.label] }, item.label))) }));
25
+ const labelWidth = Math.max(...items.map((item) => item.label.length));
26
+ return (_jsxs(Box, { flexDirection: "column", children: [items.map((item, index) => {
27
+ const active = index === cursor;
28
+ return (_jsxs(Text, { children: [_jsx(Text, { color: ACCENT, children: active ? '❯ ' : ' ' }), _jsx(Text, { color: active ? ACCENT : undefined, bold: active, children: item.hint ? item.label.padEnd(labelWidth) : item.label }), item.hint && _jsxs(Text, { dimColor: true, children: [" ", item.hint] })] }, item.label));
29
+ }), _jsxs(Text, { dimColor: true, children: ['\n', "\u2191\u2193 move \u00B7 enter select"] })] }));
25
30
  }
@@ -13,7 +13,7 @@ export function isProfileLockError(error) {
13
13
  */
14
14
  export function translateLaunchError(error) {
15
15
  if (isProfileLockError(error)) {
16
- const friendly = new Error('The Chrome profile is already in use. Another skrape process may be running — ' +
16
+ const friendly = new Error('The Chrome profile is already in use. Another skrape process may be running. ' +
17
17
  'if not, delete ~/.skool-skrape/chrome-profile/SingletonLock and try again.');
18
18
  friendly.cause = error;
19
19
  return friendly;
@@ -1,11 +1,5 @@
1
- /**
2
- * Pure decision logic for the one-time "make sure Chrome is installed" step
3
- * that runs before the session check. Kept free of fs/child_process/Playwright
4
- * details (those live in `../fetch/chromeSetup.ts`) so the branching here is
5
- * unit-testable against plain fakes, the same split used by `flow.ts` and
6
- * `browserLifecycle.ts` elsewhere in this package.
7
- */
8
- const MANUAL_FALLBACK = 'npx playwright install chrome';
1
+ import { INSTALL_TARGET } from '../fetch/chromeSetup.js';
2
+ const MANUAL_FALLBACK = `npx playwright install ${INSTALL_TARGET}`;
9
3
  /** Thrown when the automatic install fails. Its message is already
10
4
  * human-actionable (includes the manual fallback command) — callers can
11
5
  * surface `.message` directly without needing to know install internals. */
@@ -26,6 +26,5 @@ export function formatProgressBar(done, total, width = 24) {
26
26
  const safeTotal = total > 0 ? total : 1;
27
27
  const ratio = Math.min(1, Math.max(0, done / safeTotal));
28
28
  const filled = Math.round(ratio * width);
29
- const bar = '#'.repeat(filled) + '-'.repeat(width - filled);
30
- return `[${bar}] ${done}/${total}`;
29
+ return `${'█'.repeat(filled)}${'░'.repeat(width - filled)} ${done}/${total} ${Math.round(ratio * 100)}%`;
31
30
  }
package/dist/tui/run.js CHANGED
@@ -5,13 +5,13 @@ import { BrowserFetcher } from '../fetch/browser.js';
5
5
  import { isChromeMarkedInstalled, installChromeViaCli, markChromeInstalled, probeChromeLaunchable } from '../fetch/chromeSetup.js';
6
6
  import { openDb } from '../store/db.js';
7
7
  import { syncClassroom } from '../sync.js';
8
- import { chromeMarkerPath, dbPath, ensureRoot, isLoggedIn, login, profileDir } from '../auth/session.js';
8
+ import { hasYtDlp } from '../media/download.js';
9
+ import { chromeMarkerPath, dbPath, defaultOutRoot, ensureRoot, isLoggedIn, login, profileDir } from '../auth/session.js';
9
10
  import { listCourses } from '../discover/skool.js';
10
11
  import { listUserCommunities } from '../discover/communities.js';
11
12
  import { App } from './App.js';
12
13
  import { createBrowserLifecycle } from './browserLifecycle.js';
13
14
  import { ensureChromeReady } from './chromeSetup.js';
14
- const OUT_ROOT = './out';
15
15
  /**
16
16
  * Runs the guided, no-arguments flow: session check, community picker,
17
17
  * confirmation, live sync progress, and a result summary. Owns the db and
@@ -33,13 +33,13 @@ export async function runGuidedFlow() {
33
33
  probeLaunchable: probeChromeLaunchable,
34
34
  markInstalled: (executablePath) => markChromeInstalled(chromeMarkerPath(), executablePath),
35
35
  installChrome: async (onProgress) => {
36
- console.log('\nSetting up the browser skrape needs — this happens once, ~150MB.\n');
36
+ console.log('\nSetting up the browser skrape needs. This happens once, ~150MB.\n');
37
37
  await installChromeViaCli(onProgress);
38
38
  console.log('\nBrowser setup complete.\n');
39
39
  const executablePath = await probeChromeLaunchable();
40
40
  if (!executablePath) {
41
- throw new Error('Chrome installed, but still could not be launched. This usually means a platform-specific ' +
42
- 'dependency is missing — see the Playwright install output above for details.');
41
+ throw new Error('The browser installed, but still could not be launched. This usually means a system library is missing' +
42
+ (process.platform === 'linux' ? '. Fix it with: sudo npx playwright install-deps chromium' : '. See the install output above.'));
43
43
  }
44
44
  return executablePath;
45
45
  },
@@ -73,7 +73,7 @@ export async function runGuidedFlow() {
73
73
  };
74
74
  process.on('SIGINT', onSigint);
75
75
  const controllers = {
76
- outRoot: OUT_ROOT,
76
+ outRoot: defaultOutRoot(),
77
77
  checkLoggedIn: async () => lifecycle.checkLoggedIn(),
78
78
  login: async () => {
79
79
  await lifecycle.login();
@@ -83,7 +83,8 @@ export async function runGuidedFlow() {
83
83
  const courses = await listCourses(slug, lifecycle.getFetcher());
84
84
  return courses.filter((course) => course.hasAccess).length;
85
85
  },
86
- runSync: async (slug, outDir, onProgress) => syncClassroom({ slug, outDir, db, fetcher: lifecycle.getFetcher(), concurrency: 4, onProgress }),
86
+ hasYtDlp,
87
+ runSync: async (slug, outDir, onProgress, videos) => syncClassroom({ slug, outDir, db, fetcher: lifecycle.getFetcher(), concurrency: 4, videos, onProgress }),
87
88
  };
88
89
  const { waitUntilExit } = render(React.createElement(App, { controllers }));
89
90
  try {
@@ -1,7 +1,11 @@
1
1
  import type { SyncSummary } from '../sync.js';
2
- export declare function formatCountLines(summary: SyncSummary): string[];
3
- export declare function formatProblemLines(summary: SyncSummary): string[];
2
+ import { type OutcomeColor } from './theme.js';
3
+ /** Colors a piece of text; identity by default so output stays plain for tests and pipes. */
4
+ export type Paint = (color: OutcomeColor | 'bold' | 'dim', text: string) => string;
5
+ export declare function formatVideoLine(summary: SyncSummary): string | null;
6
+ /** Problems worst-first, so real failures aren't buried under routine "no video" lessons. */
7
+ export declare function sortedProblems(summary: SyncSummary): SyncSummary['problems'];
4
8
  /** True when nothing was transcribed at all — the case that must never be
5
9
  * rendered as a cheerful, empty-looking summary. */
6
10
  export declare function nothingTranscribed(summary: SyncSummary): boolean;
7
- export declare function formatSummary(summary: SyncSummary, outDir: string): string[];
11
+ export declare function formatSummary(summary: SyncSummary, outDir: string, paint?: Paint): string[];
@@ -1,29 +1,49 @@
1
- export function formatCountLines(summary) {
2
- const lines = [];
3
- for (const [outcome, count] of Object.entries(summary.counts)) {
4
- if (count > 0)
5
- lines.push(` ${outcome.padEnd(14)} ${count}`);
6
- }
7
- return lines;
1
+ import { OUTCOME_STYLE } from './theme.js';
2
+ const plain = (_color, text) => text;
3
+ export function formatVideoLine(summary) {
4
+ const v = summary.videos;
5
+ if (!v)
6
+ return null;
7
+ return `${v.downloaded} downloaded · ${v.existing} already on disk · ${v.failed} failed`;
8
8
  }
9
- export function formatProblemLines(summary) {
10
- return summary.problems.map((problem) => ` [${problem.outcome}] ${problem.course} / ${problem.title}: ${problem.reason}`);
9
+ const SEVERITY = ['failed', 'unavailable', 'no-access', 'no-video', 'skipped', 'ok'];
10
+ /** Problems worst-first, so real failures aren't buried under routine "no video" lessons. */
11
+ export function sortedProblems(summary) {
12
+ return [...summary.problems].sort((a, b) => SEVERITY.indexOf(a.outcome) - SEVERITY.indexOf(b.outcome));
11
13
  }
12
14
  /** True when nothing was transcribed at all — the case that must never be
13
15
  * rendered as a cheerful, empty-looking summary. */
14
16
  export function nothingTranscribed(summary) {
15
17
  return summary.counts.ok === 0 && summary.totalWords === 0;
16
18
  }
17
- export function formatSummary(summary, outDir) {
19
+ function headline(summary, paint) {
20
+ if (!nothingTranscribed(summary)) {
21
+ return paint('green', `✓ Transcribed ${summary.totalWords.toLocaleString('en-US')} words`);
22
+ }
23
+ // A re-sync where every lesson was already on disk is a success, not an empty run.
24
+ if (summary.counts.skipped > 0)
25
+ return paint('green', '✓ Up to date, nothing new to transcribe.');
26
+ return paint('yellow', '⚠ Nothing was transcribed.');
27
+ }
28
+ export function formatSummary(summary, outDir, paint = plain) {
18
29
  const lines = [];
19
- if (nothingTranscribed(summary)) {
20
- lines.push('Nothing was transcribed.', '');
30
+ lines.push(headline(summary, paint), '');
31
+ for (const [outcome, count] of Object.entries(summary.counts)) {
32
+ if (count === 0)
33
+ continue;
34
+ const style = OUTCOME_STYLE[outcome];
35
+ lines.push(` ${paint(style.color, style.icon)} ${style.label.padEnd(14)} ${count}`);
21
36
  }
22
- lines.push('--- summary ---', ...formatCountLines(summary));
23
- lines.push(` ${'words'.padEnd(14)} ${summary.totalWords.toLocaleString('en-US')}`);
37
+ const videoLine = formatVideoLine(summary);
38
+ if (videoLine)
39
+ lines.push(` ${paint('dim', '▸')} ${'videos'.padEnd(14)} ${videoLine}`);
24
40
  if (summary.problems.length > 0) {
25
- lines.push('', '--- not transcribed ---', ...formatProblemLines(summary));
41
+ lines.push('', paint('bold', `Needs attention (${summary.problems.length})`));
42
+ for (const problem of sortedProblems(summary)) {
43
+ const style = OUTCOME_STYLE[problem.outcome];
44
+ lines.push(` ${paint(style.color, style.icon)} ${problem.course} › ${problem.title} ${paint('dim', problem.reason)}`);
45
+ }
26
46
  }
27
- lines.push('', `Output: ${outDir}`);
47
+ lines.push('', `${paint('dim', 'Saved to')} ${outDir}`);
28
48
  return lines;
29
49
  }
@@ -0,0 +1,14 @@
1
+ import type { Outcome } from '../sync.js';
2
+ /** Brand accent: peach. Ink takes hex; the plain CLI falls back to a named color. */
3
+ export declare const ACCENT = "#FFB38A";
4
+ export type OutcomeColor = 'green' | 'gray' | 'yellow' | 'red';
5
+ export declare const OUTCOME_STYLE: Record<Outcome, {
6
+ icon: string;
7
+ label: string;
8
+ color: OutcomeColor;
9
+ }>;
10
+ /** A path as a person reads it: absolute, with the home folder shortened to ~. */
11
+ export declare function displayPath(path: string): string;
12
+ export declare function formatDuration(ms: number): string;
13
+ /** Remaining time from a linear rate; null until there's at least one data point. */
14
+ export declare function estimateRemainingMs(elapsedMs: number, done: number, total: number): number | null;
@@ -0,0 +1,31 @@
1
+ import { homedir } from 'node:os';
2
+ /** Brand accent: peach. Ink takes hex; the plain CLI falls back to a named color. */
3
+ export const ACCENT = '#FFB38A';
4
+ export const OUTCOME_STYLE = {
5
+ ok: { icon: '✓', label: 'transcribed', color: 'green' },
6
+ skipped: { icon: '↷', label: 'already done', color: 'gray' },
7
+ 'no-video': { icon: '·', label: 'no video', color: 'gray' },
8
+ 'no-access': { icon: '⊘', label: 'locked', color: 'yellow' },
9
+ unavailable: { icon: '!', label: 'unavailable', color: 'yellow' },
10
+ failed: { icon: '✗', label: 'failed', color: 'red' },
11
+ };
12
+ /** A path as a person reads it: absolute, with the home folder shortened to ~. */
13
+ export function displayPath(path) {
14
+ const home = homedir();
15
+ return path === home || path.startsWith(`${home}/`) ? `~${path.slice(home.length)}` : path;
16
+ }
17
+ export function formatDuration(ms) {
18
+ const seconds = Math.max(0, Math.round(ms / 1000));
19
+ if (seconds < 60)
20
+ return `${seconds}s`;
21
+ const minutes = Math.floor(seconds / 60);
22
+ if (minutes < 60)
23
+ return `${minutes}m ${seconds % 60}s`;
24
+ return `${Math.floor(minutes / 60)}h ${minutes % 60}m`;
25
+ }
26
+ /** Remaining time from a linear rate; null until there's at least one data point. */
27
+ export function estimateRemainingMs(elapsedMs, done, total) {
28
+ if (done <= 0 || done >= total)
29
+ return null;
30
+ return (elapsedMs / done) * (total - done);
31
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@mogulmoretti/skrape",
3
- "version": "0.1.0",
3
+ "version": "0.2.1",
4
4
  "description": "Read Skool course content instead of watching it — pulls the classroom of a community you belong to into clean, readable transcripts.",
5
5
  "keywords": [
6
6
  "skool",