sc-dlp 0.0.0-stage → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +213 -2
  3. package/lib/bundle-node.cjs +1875 -0
  4. package/lib/bundle-node.mjs +1846 -0
  5. package/lib/cli.cjs +3622 -0
  6. package/lib/types/cli/args.d.ts +3 -0
  7. package/lib/types/cli/auth/login.d.ts +1 -0
  8. package/lib/types/cli/auth/logout.d.ts +1 -0
  9. package/lib/types/cli/auth/session.d.ts +12 -0
  10. package/lib/types/cli/auth/status.d.ts +1 -0
  11. package/lib/types/cli/auth.d.ts +4 -0
  12. package/lib/types/cli/config.d.ts +5 -0
  13. package/lib/types/cli/config_command.d.ts +1 -0
  14. package/lib/types/cli/download/conflict.d.ts +15 -0
  15. package/lib/types/cli/download/display.d.ts +15 -0
  16. package/lib/types/cli/download/logger.d.ts +16 -0
  17. package/lib/types/cli/download/persistence.d.ts +4 -0
  18. package/lib/types/cli/download/thumbnail.d.ts +3 -0
  19. package/lib/types/cli/download.d.ts +5 -0
  20. package/lib/types/cli/types.d.ts +47 -0
  21. package/lib/types/cli/utils.d.ts +39 -0
  22. package/lib/types/cli.d.ts +1 -0
  23. package/lib/types/downloader/core.d.ts +3 -0
  24. package/lib/types/downloader/metadata/list.d.ts +13 -0
  25. package/lib/types/downloader/metadata/playlist/delete_account.d.ts +2 -0
  26. package/lib/types/downloader/metadata/playlist/featured.d.ts +2 -0
  27. package/lib/types/downloader/metadata/playlist/mystuff.d.ts +2 -0
  28. package/lib/types/downloader/metadata/playlist/shorthand.d.ts +2 -0
  29. package/lib/types/downloader/metadata/playlist/url.d.ts +2 -0
  30. package/lib/types/downloader/metadata/playlist.d.ts +2 -0
  31. package/lib/types/downloader/metadata/project.d.ts +3 -0
  32. package/lib/types/downloader/metadata/user.d.ts +12 -0
  33. package/lib/types/downloader/metadata.d.ts +4 -0
  34. package/lib/types/downloader/remote.d.ts +13 -0
  35. package/lib/types/downloader/sb2.d.ts +13 -0
  36. package/lib/types/downloader/sb3.d.ts +13 -0
  37. package/lib/types/downloader/turbowarp.d.ts +5 -0
  38. package/lib/types/downloader/types.d.ts +149 -0
  39. package/lib/types/downloader/utils.d.ts +56 -0
  40. package/lib/types/downloader/zip.d.ts +16 -0
  41. package/lib/types/downloader.d.ts +7 -0
  42. package/lib/types/environment.d.ts +6 -0
  43. package/lib/types/errors.d.ts +19 -0
  44. package/lib/types/export-node.d.ts +1 -0
  45. package/lib/types/fetch-asset.d.ts +12 -0
  46. package/lib/types/fetch-with-progress.d.ts +2 -0
  47. package/lib/types/fetch-wrapper.d.ts +5 -0
  48. package/lib/types/urls.d.ts +26 -0
  49. package/package.json +81 -4
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2019-2024 Thomas Weber
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,214 @@
1
- # Temporary Holding Version
1
+ # sc-dlp
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ [![npm version](https://img.shields.io/npm/v/sc-dlp.svg)](https://www.npmjs.com/package/sc-dlp)
4
+ [![license](https://img.shields.io/npm/l/sc-dlp.svg)](https://github.com/rbstxt/sc-dlp/blob/main/LICENSE)
5
+
6
+ A high-performance CLI and Node.js library for downloading Scratch and TurboWarp projects. Supports projects from Scratch 1.4 (`sb`), 2.0 (`sb2`), 3.0 (`sb3`), and TurboWarp Share (`sha256`).
7
+
8
+ `sc-dlp` is a fork of [forkphorus/sb-downloader](https://github.com/forkphorus/sb-downloader), redesigned for robust project archival, analysis, and research.
9
+
10
+ ## Features
11
+
12
+ - **TurboWarp Support**: Full support for `share.turbowarp.org` projects, including SHA-256 asset mapping and internal metadata extraction.
13
+ - **Authentication**: Log in with your Scratch account to access projects in "My Stuff" or recently viewed projects.
14
+ - **Advanced Collection Support**: Download projects from Loves, Favorites, Studios, Search results, Featured projects, Explore categories, Remixes, and "My Stuff".
15
+ - **Automatic Pagination**: Automatically fetches all projects from multi-page collections.
16
+ - **Shorthand Aliases**: Quick access to any collection using simple patterns.
17
+ - **High Performance**: Parallel project downloads (`-c`), asset caching/retries, and automatic MD5 hash verification.
18
+ - **Rich CLI & Library**: Real-time worker progress bars for terminals, clean logs for pipes/CI, and an ESM/CJS/TypeScript Node.js API.
19
+
20
+ ## Installation
21
+
22
+ Requires **Node.js 18.17.0 or later**.
23
+
24
+ ### Global CLI
25
+
26
+ ```bash
27
+ npm install -g sc-dlp
28
+ ```
29
+
30
+ ### Run without installing
31
+
32
+ ```bash
33
+ npx sc-dlp 60917032
34
+ ```
35
+
36
+ ### As a Node.js library
37
+
38
+ ```bash
39
+ npm install sc-dlp
40
+ ```
41
+
42
+ ## CLI Usage
43
+
44
+ Use project IDs, URLs, or shorthand aliases to download projects.
45
+
46
+ ```bash
47
+ # Single project (Scratch)
48
+ sc-dlp 60917032
49
+ sc-dlp https://scratch.mit.edu/projects/60917032/
50
+
51
+ # Single project (TurboWarp Share)
52
+ sc-dlp 4cd76e33-0e7a-468f-85c9-17799748a864
53
+ sc-dlp turbowarp/4cd76e33-0e7a-468f-85c9-17799748a864
54
+ sc-dlp https://share.turbowarp.org/projects/4cd76e33-0e7a-468f-85c9-17799748a864
55
+
56
+ # User collections
57
+ sc-dlp @scratchcat
58
+ sc-dlp @scratchcat/loves
59
+ sc-dlp @scratchcat/favorites
60
+
61
+ # Specialized collections
62
+ sc-dlp mystuff/shared
63
+ sc-dlp featured/curated
64
+ sc-dlp explore/animations
65
+ sc-dlp recently
66
+ sc-dlp 10123/remixes
67
+ sc-dlp studio/1234
68
+ sc-dlp search/platformer
69
+
70
+ # Authentication & configuration
71
+ sc-dlp login
72
+ sc-dlp status
73
+ sc-dlp logout
74
+ sc-dlp config --list
75
+ sc-dlp config --set conflictAction=skip
76
+ ```
77
+
78
+ ### Authentication
79
+
80
+ `sc-dlp login` supports three authentication methods:
81
+
82
+ - **Username & Password**: Full session access (requires solving a captcha if triggered).
83
+ - **Session ID**: Full session access using a `scratchsessionsid` cookie value.
84
+ - **X-Token**: Limited permissions access. Suitable for `recently` and public data, but cannot access `mystuff`.
85
+
86
+ ### Configuration
87
+
88
+ Manage persistent defaults (language, conflict action, proxy, custom API URLs, TurboWarp mirrors) with `sc-dlp config`:
89
+
90
+ ```bash
91
+ sc-dlp config --list
92
+ sc-dlp config --get proxy
93
+ sc-dlp config --set proxy=http://127.0.0.1:8080
94
+ sc-dlp config --set language=ja
95
+ sc-dlp config --reset
96
+ ```
97
+
98
+ ### Collection Aliases
99
+
100
+ | Collection | Alias Pattern | Note |
101
+ | ------------------- | ---------------------------------------------- | ----------------------------------- |
102
+ | **Loves** | `@user/loves`, `@user/lov` | Scrapes HTML for all pages. |
103
+ | **Favorites** | `@user/favorites`, `@user/fav` | Fetches via API. |
104
+ | **User Projects** | `@user` | Fetches all shared projects. |
105
+ | **My Stuff** | `mystuff`, `mystuff/shared`, `mystuff/trash` | Requires login. |
106
+ | **Recently Viewed** | `recently`, `recent` | Requires login. |
107
+ | **Featured** | `featured`, `featured/curated`, `featured/top` | Fetches from Scratch homepage. |
108
+ | **Explore** | `explore/[category]`, `search/[query]` | Fetches results from browse/search. |
109
+ | **Remixes** | `id/remixes`, `id/remix` | Fetches all remixes of a project. |
110
+ | **Studio** | `studio/id` | Fetches projects in a studio. |
111
+ | **TurboWarp** | `turbowarp/[id]`, `[uuid]` | Fetches from TurboWarp Share. |
112
+
113
+ ### Options
114
+
115
+ - `login` / `status` / `logout`: Manage Scratch authentication.
116
+ - `config [options...]`: Manage saved configuration (`sc-dlp config --help` for details).
117
+ - `deleteaccount`: Download projects listed on the Scratch account deletion confirmation page.
118
+ - `-h`, `--help`: Show the help screen.
119
+ - `-c`, `--concurrency NUMBER`: Number of projects to download in parallel (default: `1`).
120
+ - `-l`, `--limit NUMBER`: Maximum number of projects to fetch from a collection.
121
+ - `--page NUMBER`: Fetch a specific page from a collection (e.g., users, favorites, loves, studios, search, mystuff).
122
+ - `--playlist-start NUMBER`: Playlist item to start at (default: `1`).
123
+ - `--playlist-end NUMBER`: Playlist item to end at.
124
+ - `-I`, `--playlist-items ITEMS`: Specific playlist indices (e.g., `1,2,5-10`).
125
+ - `--language CODE`: Override language for search results (default: `en`).
126
+ - `--project-token TOKEN`: Token to access private/unshared projects.
127
+ - `--format EXT`: Output file extension (`sb3`, `sb2`, `sb`, `json`; default: auto-detected).
128
+ - `-j`, `--json-only`: Download only the `project.json` file (implies `--format json`).
129
+ - `-a`, `--no-assets`: Download project code only (skip costumes and sounds).
130
+ - `--conflict ACTION`: Conflict action: `overwrite`, `skip`, or `keep-both`.
131
+ - `--retries NUMBER`: Number of times to retry failed asset downloads (default: `5`).
132
+ - `-m`, `--mtime`: Set file modification time to the project's last modified/shared date.
133
+ - `-t`, `--write-thumbnail`: Download the project thumbnail alongside the project file.
134
+ - `--thumbnail-size WIDTHxHEIGHT`: Thumbnail size (default: `9999x9999`).
135
+ - `-v`, `--verbose`: Show detailed logs and stack traces on error.
136
+
137
+ ### File Conflict Resolution
138
+
139
+ If a project with the same name and ID already exists in the target directory, `sc-dlp` prompts you for an action (unless `--conflict` or `config --set conflictAction=...` is configured). By default, filenames include the project ID (e.g., `Title (12345).sb3`) to avoid collisions between projects with the same title.
140
+
141
+ - **Overwrite**: Replace the existing file with the new download.
142
+ - **Skip**: Skip downloading this project.
143
+ - **Keep both**: Automatically rename the new file by appending a number (e.g., `Project 2.sb3`).
144
+
145
+ Once selected in a session, the choice is applied automatically to remaining conflicts in the same run.
146
+
147
+ ## Programmatic Usage (Node.js API)
148
+
149
+ `sc-dlp` ships ESM, CommonJS, and TypeScript declarations.
150
+
151
+ ```ts
152
+ import fs from "node:fs/promises";
153
+ import { downloadProjectFromID, getProjectsFromPlaylistURL } from "sc-dlp";
154
+
155
+ const project = await downloadProjectFromID("60917032", {
156
+ onProgress: (type, loaded, total) => {
157
+ console.log(`${type}: ${loaded}/${total}`);
158
+ },
159
+ });
160
+
161
+ await fs.writeFile(
162
+ `${project.title}.${project.type}`,
163
+ Buffer.from(project.arrayBuffer)
164
+ );
165
+
166
+ const studioProjects = await getProjectsFromPlaylistURL("studio/1234", {
167
+ limit: 20,
168
+ });
169
+ console.log(`Found ${studioProjects?.length ?? 0} projects`);
170
+ ```
171
+
172
+ CommonJS is also supported:
173
+
174
+ ```js
175
+ const { downloadProjectFromID } = require("sc-dlp");
176
+ ```
177
+
178
+ Main exported functions:
179
+
180
+ - `downloadProjectFromID(id, options?)`
181
+ - `downloadProjectFromURL(url, options?)`
182
+ - `downloadTurboWarpProject(id, options?)`
183
+ - `downloadProjectFromJSON(json, options?)`
184
+ - `downloadProjectFromBuffer(buffer, options?)`
185
+ - `getProjectMetadata(id, options?)`
186
+ - `getTurboWarpMetadata(id, options?)`
187
+ - `getProjectsFromPlaylistURL(input, options?)`
188
+ - `clearAssetCache()`
189
+
190
+ ## Advanced
191
+
192
+ ### Asset Integrity
193
+
194
+ By default, `sc-dlp` calculates the MD5 hash of every downloaded asset. If the calculated hash differs from the hash referenced in `project.json`, the asset is automatically renamed and `project.json` is updated so the resulting project remains valid and loadable.
195
+
196
+ ### TurboWarp Assets
197
+
198
+ TurboWarp Share projects use SHA-256 for asset storage. `sc-dlp` automatically extracts the `md5extsToSha256` mapping from the project page HTML to fetch and remap assets for Scratch compatibility.
199
+
200
+ ## Privacy
201
+
202
+ By default, `sc-dlp` communicates directly with Scratch APIs (`api.scratch.mit.edu`, `projects.scratch.mit.edu`, `assets.scratch.mit.edu`) and TurboWarp APIs (`share.turbowarp.org`). Configuration and session data are stored locally in `~/.sc-dlp/config.json` with `0600` permissions.
203
+
204
+ ## Development
205
+
206
+ To build and test from source:
207
+
208
+ ```bash
209
+ git clone https://github.com/rbstxt/sc-dlp.git
210
+ cd sc-dlp
211
+ npm install
212
+ npm run build
213
+ npm test
214
+ ```