sc-dlp 0.0.0-stage → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +213 -2
- package/lib/bundle-node.cjs +1875 -0
- package/lib/bundle-node.mjs +1846 -0
- package/lib/cli.cjs +3622 -0
- package/lib/types/cli/args.d.ts +3 -0
- package/lib/types/cli/auth/login.d.ts +1 -0
- package/lib/types/cli/auth/logout.d.ts +1 -0
- package/lib/types/cli/auth/session.d.ts +12 -0
- package/lib/types/cli/auth/status.d.ts +1 -0
- package/lib/types/cli/auth.d.ts +4 -0
- package/lib/types/cli/config.d.ts +5 -0
- package/lib/types/cli/config_command.d.ts +1 -0
- package/lib/types/cli/download/conflict.d.ts +15 -0
- package/lib/types/cli/download/display.d.ts +15 -0
- package/lib/types/cli/download/logger.d.ts +16 -0
- package/lib/types/cli/download/persistence.d.ts +4 -0
- package/lib/types/cli/download/thumbnail.d.ts +3 -0
- package/lib/types/cli/download.d.ts +5 -0
- package/lib/types/cli/types.d.ts +47 -0
- package/lib/types/cli/utils.d.ts +39 -0
- package/lib/types/cli.d.ts +1 -0
- package/lib/types/downloader/core.d.ts +3 -0
- package/lib/types/downloader/metadata/list.d.ts +13 -0
- package/lib/types/downloader/metadata/playlist/delete_account.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/featured.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/mystuff.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/shorthand.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/url.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist.d.ts +2 -0
- package/lib/types/downloader/metadata/project.d.ts +3 -0
- package/lib/types/downloader/metadata/user.d.ts +12 -0
- package/lib/types/downloader/metadata.d.ts +4 -0
- package/lib/types/downloader/remote.d.ts +13 -0
- package/lib/types/downloader/sb2.d.ts +13 -0
- package/lib/types/downloader/sb3.d.ts +13 -0
- package/lib/types/downloader/turbowarp.d.ts +5 -0
- package/lib/types/downloader/types.d.ts +149 -0
- package/lib/types/downloader/utils.d.ts +56 -0
- package/lib/types/downloader/zip.d.ts +16 -0
- package/lib/types/downloader.d.ts +7 -0
- package/lib/types/environment.d.ts +6 -0
- package/lib/types/errors.d.ts +19 -0
- package/lib/types/export-node.d.ts +1 -0
- package/lib/types/fetch-asset.d.ts +12 -0
- package/lib/types/fetch-with-progress.d.ts +2 -0
- package/lib/types/fetch-wrapper.d.ts +5 -0
- package/lib/types/urls.d.ts +26 -0
- package/package.json +81 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2019-2024 Thomas Weber
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,214 @@
|
|
|
1
|
-
#
|
|
1
|
+
# sc-dlp
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/sc-dlp)
|
|
4
|
+
[](https://github.com/rbstxt/sc-dlp/blob/main/LICENSE)
|
|
5
|
+
|
|
6
|
+
A high-performance CLI and Node.js library for downloading Scratch and TurboWarp projects. Supports projects from Scratch 1.4 (`sb`), 2.0 (`sb2`), 3.0 (`sb3`), and TurboWarp Share (`sha256`).
|
|
7
|
+
|
|
8
|
+
`sc-dlp` is a fork of [forkphorus/sb-downloader](https://github.com/forkphorus/sb-downloader), redesigned for robust project archival, analysis, and research.
|
|
9
|
+
|
|
10
|
+
## Features
|
|
11
|
+
|
|
12
|
+
- **TurboWarp Support**: Full support for `share.turbowarp.org` projects, including SHA-256 asset mapping and internal metadata extraction.
|
|
13
|
+
- **Authentication**: Log in with your Scratch account to access projects in "My Stuff" or recently viewed projects.
|
|
14
|
+
- **Advanced Collection Support**: Download projects from Loves, Favorites, Studios, Search results, Featured projects, Explore categories, Remixes, and "My Stuff".
|
|
15
|
+
- **Automatic Pagination**: Automatically fetches all projects from multi-page collections.
|
|
16
|
+
- **Shorthand Aliases**: Quick access to any collection using simple patterns.
|
|
17
|
+
- **High Performance**: Parallel project downloads (`-c`), asset caching/retries, and automatic MD5 hash verification.
|
|
18
|
+
- **Rich CLI & Library**: Real-time worker progress bars for terminals, clean logs for pipes/CI, and an ESM/CJS/TypeScript Node.js API.
|
|
19
|
+
|
|
20
|
+
## Installation
|
|
21
|
+
|
|
22
|
+
Requires **Node.js 18.17.0 or later**.
|
|
23
|
+
|
|
24
|
+
### Global CLI
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
npm install -g sc-dlp
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
### Run without installing
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
npx sc-dlp 60917032
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### As a Node.js library
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
npm install sc-dlp
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## CLI Usage
|
|
43
|
+
|
|
44
|
+
Use project IDs, URLs, or shorthand aliases to download projects.
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
# Single project (Scratch)
|
|
48
|
+
sc-dlp 60917032
|
|
49
|
+
sc-dlp https://scratch.mit.edu/projects/60917032/
|
|
50
|
+
|
|
51
|
+
# Single project (TurboWarp Share)
|
|
52
|
+
sc-dlp 4cd76e33-0e7a-468f-85c9-17799748a864
|
|
53
|
+
sc-dlp turbowarp/4cd76e33-0e7a-468f-85c9-17799748a864
|
|
54
|
+
sc-dlp https://share.turbowarp.org/projects/4cd76e33-0e7a-468f-85c9-17799748a864
|
|
55
|
+
|
|
56
|
+
# User collections
|
|
57
|
+
sc-dlp @scratchcat
|
|
58
|
+
sc-dlp @scratchcat/loves
|
|
59
|
+
sc-dlp @scratchcat/favorites
|
|
60
|
+
|
|
61
|
+
# Specialized collections
|
|
62
|
+
sc-dlp mystuff/shared
|
|
63
|
+
sc-dlp featured/curated
|
|
64
|
+
sc-dlp explore/animations
|
|
65
|
+
sc-dlp recently
|
|
66
|
+
sc-dlp 10123/remixes
|
|
67
|
+
sc-dlp studio/1234
|
|
68
|
+
sc-dlp search/platformer
|
|
69
|
+
|
|
70
|
+
# Authentication & configuration
|
|
71
|
+
sc-dlp login
|
|
72
|
+
sc-dlp status
|
|
73
|
+
sc-dlp logout
|
|
74
|
+
sc-dlp config --list
|
|
75
|
+
sc-dlp config --set conflictAction=skip
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### Authentication
|
|
79
|
+
|
|
80
|
+
`sc-dlp login` supports three authentication methods:
|
|
81
|
+
|
|
82
|
+
- **Username & Password**: Full session access (requires solving a captcha if triggered).
|
|
83
|
+
- **Session ID**: Full session access using a `scratchsessionsid` cookie value.
|
|
84
|
+
- **X-Token**: Limited permissions access. Suitable for `recently` and public data, but cannot access `mystuff`.
|
|
85
|
+
|
|
86
|
+
### Configuration
|
|
87
|
+
|
|
88
|
+
Manage persistent defaults (language, conflict action, proxy, custom API URLs, TurboWarp mirrors) with `sc-dlp config`:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
sc-dlp config --list
|
|
92
|
+
sc-dlp config --get proxy
|
|
93
|
+
sc-dlp config --set proxy=http://127.0.0.1:8080
|
|
94
|
+
sc-dlp config --set language=ja
|
|
95
|
+
sc-dlp config --reset
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Collection Aliases
|
|
99
|
+
|
|
100
|
+
| Collection | Alias Pattern | Note |
|
|
101
|
+
| ------------------- | ---------------------------------------------- | ----------------------------------- |
|
|
102
|
+
| **Loves** | `@user/loves`, `@user/lov` | Scrapes HTML for all pages. |
|
|
103
|
+
| **Favorites** | `@user/favorites`, `@user/fav` | Fetches via API. |
|
|
104
|
+
| **User Projects** | `@user` | Fetches all shared projects. |
|
|
105
|
+
| **My Stuff** | `mystuff`, `mystuff/shared`, `mystuff/trash` | Requires login. |
|
|
106
|
+
| **Recently Viewed** | `recently`, `recent` | Requires login. |
|
|
107
|
+
| **Featured** | `featured`, `featured/curated`, `featured/top` | Fetches from Scratch homepage. |
|
|
108
|
+
| **Explore** | `explore/[category]`, `search/[query]` | Fetches results from browse/search. |
|
|
109
|
+
| **Remixes** | `id/remixes`, `id/remix` | Fetches all remixes of a project. |
|
|
110
|
+
| **Studio** | `studio/id` | Fetches projects in a studio. |
|
|
111
|
+
| **TurboWarp** | `turbowarp/[id]`, `[uuid]` | Fetches from TurboWarp Share. |
|
|
112
|
+
|
|
113
|
+
### Options
|
|
114
|
+
|
|
115
|
+
- `login` / `status` / `logout`: Manage Scratch authentication.
|
|
116
|
+
- `config [options...]`: Manage saved configuration (`sc-dlp config --help` for details).
|
|
117
|
+
- `deleteaccount`: Download projects listed on the Scratch account deletion confirmation page.
|
|
118
|
+
- `-h`, `--help`: Show the help screen.
|
|
119
|
+
- `-c`, `--concurrency NUMBER`: Number of projects to download in parallel (default: `1`).
|
|
120
|
+
- `-l`, `--limit NUMBER`: Maximum number of projects to fetch from a collection.
|
|
121
|
+
- `--page NUMBER`: Fetch a specific page from a collection (e.g., users, favorites, loves, studios, search, mystuff).
|
|
122
|
+
- `--playlist-start NUMBER`: Playlist item to start at (default: `1`).
|
|
123
|
+
- `--playlist-end NUMBER`: Playlist item to end at.
|
|
124
|
+
- `-I`, `--playlist-items ITEMS`: Specific playlist indices (e.g., `1,2,5-10`).
|
|
125
|
+
- `--language CODE`: Override language for search results (default: `en`).
|
|
126
|
+
- `--project-token TOKEN`: Token to access private/unshared projects.
|
|
127
|
+
- `--format EXT`: Output file extension (`sb3`, `sb2`, `sb`, `json`; default: auto-detected).
|
|
128
|
+
- `-j`, `--json-only`: Download only the `project.json` file (implies `--format json`).
|
|
129
|
+
- `-a`, `--no-assets`: Download project code only (skip costumes and sounds).
|
|
130
|
+
- `--conflict ACTION`: Conflict action: `overwrite`, `skip`, or `keep-both`.
|
|
131
|
+
- `--retries NUMBER`: Number of times to retry failed asset downloads (default: `5`).
|
|
132
|
+
- `-m`, `--mtime`: Set file modification time to the project's last modified/shared date.
|
|
133
|
+
- `-t`, `--write-thumbnail`: Download the project thumbnail alongside the project file.
|
|
134
|
+
- `--thumbnail-size WIDTHxHEIGHT`: Thumbnail size (default: `9999x9999`).
|
|
135
|
+
- `-v`, `--verbose`: Show detailed logs and stack traces on error.
|
|
136
|
+
|
|
137
|
+
### File Conflict Resolution
|
|
138
|
+
|
|
139
|
+
If a project with the same name and ID already exists in the target directory, `sc-dlp` prompts you for an action (unless `--conflict` or `config --set conflictAction=...` is configured). By default, filenames include the project ID (e.g., `Title (12345).sb3`) to avoid collisions between projects with the same title.
|
|
140
|
+
|
|
141
|
+
- **Overwrite**: Replace the existing file with the new download.
|
|
142
|
+
- **Skip**: Skip downloading this project.
|
|
143
|
+
- **Keep both**: Automatically rename the new file by appending a number (e.g., `Project 2.sb3`).
|
|
144
|
+
|
|
145
|
+
Once selected in a session, the choice is applied automatically to remaining conflicts in the same run.
|
|
146
|
+
|
|
147
|
+
## Programmatic Usage (Node.js API)
|
|
148
|
+
|
|
149
|
+
`sc-dlp` ships ESM, CommonJS, and TypeScript declarations.
|
|
150
|
+
|
|
151
|
+
```ts
|
|
152
|
+
import fs from "node:fs/promises";
|
|
153
|
+
import { downloadProjectFromID, getProjectsFromPlaylistURL } from "sc-dlp";
|
|
154
|
+
|
|
155
|
+
const project = await downloadProjectFromID("60917032", {
|
|
156
|
+
onProgress: (type, loaded, total) => {
|
|
157
|
+
console.log(`${type}: ${loaded}/${total}`);
|
|
158
|
+
},
|
|
159
|
+
});
|
|
160
|
+
|
|
161
|
+
await fs.writeFile(
|
|
162
|
+
`${project.title}.${project.type}`,
|
|
163
|
+
Buffer.from(project.arrayBuffer)
|
|
164
|
+
);
|
|
165
|
+
|
|
166
|
+
const studioProjects = await getProjectsFromPlaylistURL("studio/1234", {
|
|
167
|
+
limit: 20,
|
|
168
|
+
});
|
|
169
|
+
console.log(`Found ${studioProjects?.length ?? 0} projects`);
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
CommonJS is also supported:
|
|
173
|
+
|
|
174
|
+
```js
|
|
175
|
+
const { downloadProjectFromID } = require("sc-dlp");
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Main exported functions:
|
|
179
|
+
|
|
180
|
+
- `downloadProjectFromID(id, options?)`
|
|
181
|
+
- `downloadProjectFromURL(url, options?)`
|
|
182
|
+
- `downloadTurboWarpProject(id, options?)`
|
|
183
|
+
- `downloadProjectFromJSON(json, options?)`
|
|
184
|
+
- `downloadProjectFromBuffer(buffer, options?)`
|
|
185
|
+
- `getProjectMetadata(id, options?)`
|
|
186
|
+
- `getTurboWarpMetadata(id, options?)`
|
|
187
|
+
- `getProjectsFromPlaylistURL(input, options?)`
|
|
188
|
+
- `clearAssetCache()`
|
|
189
|
+
|
|
190
|
+
## Advanced
|
|
191
|
+
|
|
192
|
+
### Asset Integrity
|
|
193
|
+
|
|
194
|
+
By default, `sc-dlp` calculates the MD5 hash of every downloaded asset. If the calculated hash differs from the hash referenced in `project.json`, the asset is automatically renamed and `project.json` is updated so the resulting project remains valid and loadable.
|
|
195
|
+
|
|
196
|
+
### TurboWarp Assets
|
|
197
|
+
|
|
198
|
+
TurboWarp Share projects use SHA-256 for asset storage. `sc-dlp` automatically extracts the `md5extsToSha256` mapping from the project page HTML to fetch and remap assets for Scratch compatibility.
|
|
199
|
+
|
|
200
|
+
## Privacy
|
|
201
|
+
|
|
202
|
+
By default, `sc-dlp` communicates directly with Scratch APIs (`api.scratch.mit.edu`, `projects.scratch.mit.edu`, `assets.scratch.mit.edu`) and TurboWarp APIs (`share.turbowarp.org`). Configuration and session data are stored locally in `~/.sc-dlp/config.json` with `0600` permissions.
|
|
203
|
+
|
|
204
|
+
## Development
|
|
205
|
+
|
|
206
|
+
To build and test from source:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
git clone https://github.com/rbstxt/sc-dlp.git
|
|
210
|
+
cd sc-dlp
|
|
211
|
+
npm install
|
|
212
|
+
npm run build
|
|
213
|
+
npm test
|
|
214
|
+
```
|