sc-dlp 1.0.0 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +116 -40
- package/lib/cli.cjs +1 -1
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -1,44 +1,45 @@
|
|
|
1
1
|
# sc-dlp
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/sc-dlp)
|
|
4
|
+
[](https://github.com/rbstxt/sc-dlp/blob/main/LICENSE)
|
|
5
|
+
|
|
6
|
+
A high-performance CLI and Node.js library for downloading Scratch and TurboWarp projects. Supports projects from Scratch 1.4 (`sb`), 2.0 (`sb2`), 3.0 (`sb3`), and TurboWarp Share (`sha256`).
|
|
4
7
|
|
|
5
8
|
`sc-dlp` is a fork of [forkphorus/sb-downloader](https://github.com/forkphorus/sb-downloader), redesigned for robust project archival, analysis, and research.
|
|
6
9
|
|
|
7
10
|
## Features
|
|
8
11
|
|
|
9
|
-
- **TurboWarp Support**:
|
|
10
|
-
- **Authentication**:
|
|
11
|
-
- **Advanced Collection Support**: Download projects from Loves, Favorites, Studios, Search results, Featured projects, Explore categories, and "My Stuff".
|
|
12
|
+
- **TurboWarp Support**: Full support for `share.turbowarp.org` projects, including SHA-256 asset mapping and internal metadata extraction.
|
|
13
|
+
- **Authentication**: Log in with your Scratch account to access projects in "My Stuff" or recently viewed projects.
|
|
14
|
+
- **Advanced Collection Support**: Download projects from Loves, Favorites, Studios, Search results, Featured projects, Explore categories, Remixes, and "My Stuff".
|
|
12
15
|
- **Automatic Pagination**: Automatically fetches all projects from multi-page collections.
|
|
13
16
|
- **Shorthand Aliases**: Quick access to any collection using simple patterns.
|
|
14
|
-
- **High Performance**: Parallel project downloads, asset
|
|
15
|
-
- **Rich CLI
|
|
17
|
+
- **High Performance**: Parallel project downloads (`-c`), asset caching/retries, and automatic MD5 hash verification.
|
|
18
|
+
- **Rich CLI & Library**: Real-time worker progress bars for terminals, clean logs for pipes/CI, and an ESM/CJS/TypeScript Node.js API.
|
|
16
19
|
|
|
17
20
|
## Installation
|
|
18
21
|
|
|
19
|
-
|
|
22
|
+
Requires **Node.js 18.17.0 or later**.
|
|
23
|
+
|
|
24
|
+
### Global CLI
|
|
20
25
|
|
|
21
26
|
```bash
|
|
22
27
|
npm install -g sc-dlp
|
|
23
28
|
```
|
|
24
29
|
|
|
25
|
-
|
|
30
|
+
### Run without installing
|
|
26
31
|
|
|
27
32
|
```bash
|
|
28
33
|
npx sc-dlp 60917032
|
|
29
34
|
```
|
|
30
35
|
|
|
31
|
-
###
|
|
36
|
+
### As a Node.js library
|
|
32
37
|
|
|
33
38
|
```bash
|
|
34
|
-
|
|
35
|
-
cd sc-dlp
|
|
36
|
-
npm install
|
|
37
|
-
npm run build
|
|
38
|
-
npm link
|
|
39
|
+
npm install sc-dlp
|
|
39
40
|
```
|
|
40
41
|
|
|
41
|
-
## Usage
|
|
42
|
+
## CLI Usage
|
|
42
43
|
|
|
43
44
|
Use project IDs, URLs, or shorthand aliases to download projects.
|
|
44
45
|
|
|
@@ -53,14 +54,9 @@ sc-dlp turbowarp/4cd76e33-0e7a-468f-85c9-17799748a864
|
|
|
53
54
|
sc-dlp https://share.turbowarp.org/projects/4cd76e33-0e7a-468f-85c9-17799748a864
|
|
54
55
|
|
|
55
56
|
# User collections
|
|
57
|
+
sc-dlp @scratchcat
|
|
56
58
|
sc-dlp @scratchcat/loves
|
|
57
59
|
sc-dlp @scratchcat/favorites
|
|
58
|
-
sc-dlp @scratchcat
|
|
59
|
-
|
|
60
|
-
# Authentication
|
|
61
|
-
sc-dlp login
|
|
62
|
-
sc-dlp status
|
|
63
|
-
sc-dlp logout
|
|
64
60
|
|
|
65
61
|
# Specialized collections
|
|
66
62
|
sc-dlp mystuff/shared
|
|
@@ -70,6 +66,13 @@ sc-dlp recently
|
|
|
70
66
|
sc-dlp 10123/remixes
|
|
71
67
|
sc-dlp studio/1234
|
|
72
68
|
sc-dlp search/platformer
|
|
69
|
+
|
|
70
|
+
# Authentication & configuration
|
|
71
|
+
sc-dlp login
|
|
72
|
+
sc-dlp status
|
|
73
|
+
sc-dlp logout
|
|
74
|
+
sc-dlp config --list
|
|
75
|
+
sc-dlp config --set conflictAction=skip
|
|
73
76
|
```
|
|
74
77
|
|
|
75
78
|
### Authentication
|
|
@@ -80,6 +83,18 @@ sc-dlp search/platformer
|
|
|
80
83
|
- **Session ID**: Full session access using a `scratchsessionsid` cookie value.
|
|
81
84
|
- **X-Token**: Limited permissions access. Suitable for `recently` and public data, but cannot access `mystuff`.
|
|
82
85
|
|
|
86
|
+
### Configuration
|
|
87
|
+
|
|
88
|
+
Manage persistent defaults (language, conflict action, proxy, custom API URLs, TurboWarp mirrors) with `sc-dlp config`:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
sc-dlp config --list
|
|
92
|
+
sc-dlp config --get proxy
|
|
93
|
+
sc-dlp config --set proxy=http://127.0.0.1:8080
|
|
94
|
+
sc-dlp config --set language=ja
|
|
95
|
+
sc-dlp config --reset
|
|
96
|
+
```
|
|
97
|
+
|
|
83
98
|
### Collection Aliases
|
|
84
99
|
|
|
85
100
|
| Collection | Alias Pattern | Note |
|
|
@@ -97,42 +112,103 @@ sc-dlp search/platformer
|
|
|
97
112
|
|
|
98
113
|
### Options
|
|
99
114
|
|
|
100
|
-
-
|
|
101
|
-
-
|
|
102
|
-
-
|
|
103
|
-
- `-
|
|
104
|
-
- `--
|
|
105
|
-
- `--
|
|
115
|
+
- `login` / `status` / `logout`: Manage Scratch authentication.
|
|
116
|
+
- `config [options...]`: Manage saved configuration (`sc-dlp config --help` for details).
|
|
117
|
+
- `deleteaccount`: Download projects listed on the Scratch account deletion confirmation page.
|
|
118
|
+
- `-h`, `--help`: Show the help screen.
|
|
119
|
+
- `-c`, `--concurrency NUMBER`: Number of projects to download in parallel (default: `1`).
|
|
120
|
+
- `-l`, `--limit NUMBER`: Maximum number of projects to fetch from a collection.
|
|
121
|
+
- `--page NUMBER`: Fetch a specific page from a collection (e.g., users, favorites, loves, studios, search, mystuff).
|
|
122
|
+
- `--playlist-start NUMBER`: Playlist item to start at (default: `1`).
|
|
106
123
|
- `--playlist-end NUMBER`: Playlist item to end at.
|
|
107
|
-
- `-I`, `--playlist-items ITEMS`: Specific indices (e.g., 1,2,5-10).
|
|
108
|
-
- `--
|
|
109
|
-
- `--
|
|
110
|
-
-
|
|
111
|
-
- `-
|
|
124
|
+
- `-I`, `--playlist-items ITEMS`: Specific playlist indices (e.g., `1,2,5-10`).
|
|
125
|
+
- `--language CODE`: Override language for search results (default: `en`).
|
|
126
|
+
- `--project-token TOKEN`: Token to access private/unshared projects.
|
|
127
|
+
- `--format EXT`: Output file extension (`sb3`, `sb2`, `sb`, `json`; default: auto-detected).
|
|
128
|
+
- `-j`, `--json-only`: Download only the `project.json` file (implies `--format json`).
|
|
129
|
+
- `-a`, `--no-assets`: Download project code only (skip costumes and sounds).
|
|
112
130
|
- `--conflict ACTION`: Conflict action: `overwrite`, `skip`, or `keep-both`.
|
|
113
|
-
- `--retries NUMBER`: Number of times to retry failed asset downloads (default:
|
|
114
|
-
- `-
|
|
131
|
+
- `--retries NUMBER`: Number of times to retry failed asset downloads (default: `5`).
|
|
132
|
+
- `-m`, `--mtime`: Set file modification time to the project's last modified/shared date.
|
|
133
|
+
- `-t`, `--write-thumbnail`: Download the project thumbnail alongside the project file.
|
|
134
|
+
- `--thumbnail-size WIDTHxHEIGHT`: Thumbnail size (default: `9999x9999`).
|
|
135
|
+
- `-v`, `--verbose`: Show detailed logs and stack traces on error.
|
|
115
136
|
|
|
116
137
|
### File Conflict Resolution
|
|
117
138
|
|
|
118
|
-
If a project with the same name and ID already exists in the target directory, `sc-dlp`
|
|
139
|
+
If a project with the same name and ID already exists in the target directory, `sc-dlp` prompts you for an action (unless `--conflict` or `config --set conflictAction=...` is configured). By default, filenames include the project ID (e.g., `Title (12345).sb3`) to avoid collisions between projects with the same title.
|
|
119
140
|
|
|
120
141
|
- **Overwrite**: Replace the existing file with the new download.
|
|
121
|
-
- **Skip**:
|
|
142
|
+
- **Skip**: Skip downloading this project.
|
|
122
143
|
- **Keep both**: Automatically rename the new file by appending a number (e.g., `Project 2.sb3`).
|
|
123
144
|
|
|
124
|
-
Once
|
|
145
|
+
Once selected in a session, the choice is applied automatically to remaining conflicts in the same run.
|
|
146
|
+
|
|
147
|
+
## Programmatic Usage (Node.js API)
|
|
148
|
+
|
|
149
|
+
`sc-dlp` ships ESM, CommonJS, and TypeScript declarations.
|
|
150
|
+
|
|
151
|
+
```ts
|
|
152
|
+
import fs from "node:fs/promises";
|
|
153
|
+
import { downloadProjectFromID, getProjectsFromPlaylistURL } from "sc-dlp";
|
|
154
|
+
|
|
155
|
+
const project = await downloadProjectFromID("60917032", {
|
|
156
|
+
onProgress: (type, loaded, total) => {
|
|
157
|
+
console.log(`${type}: ${loaded}/${total}`);
|
|
158
|
+
},
|
|
159
|
+
});
|
|
160
|
+
|
|
161
|
+
await fs.writeFile(
|
|
162
|
+
`${project.title}.${project.type}`,
|
|
163
|
+
Buffer.from(project.arrayBuffer)
|
|
164
|
+
);
|
|
165
|
+
|
|
166
|
+
const studioProjects = await getProjectsFromPlaylistURL("studio/1234", {
|
|
167
|
+
limit: 20,
|
|
168
|
+
});
|
|
169
|
+
console.log(`Found ${studioProjects?.length ?? 0} projects`);
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
CommonJS is also supported:
|
|
173
|
+
|
|
174
|
+
```js
|
|
175
|
+
const { downloadProjectFromID } = require("sc-dlp");
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Main exported functions:
|
|
179
|
+
|
|
180
|
+
- `downloadProjectFromID(id, options?)`
|
|
181
|
+
- `downloadProjectFromURL(url, options?)`
|
|
182
|
+
- `downloadTurboWarpProject(id, options?)`
|
|
183
|
+
- `downloadProjectFromJSON(json, options?)`
|
|
184
|
+
- `downloadProjectFromBuffer(buffer, options?)`
|
|
185
|
+
- `getProjectMetadata(id, options?)`
|
|
186
|
+
- `getTurboWarpMetadata(id, options?)`
|
|
187
|
+
- `getProjectsFromPlaylistURL(input, options?)`
|
|
188
|
+
- `clearAssetCache()`
|
|
125
189
|
|
|
126
190
|
## Advanced
|
|
127
191
|
|
|
128
192
|
### Asset Integrity
|
|
129
193
|
|
|
130
|
-
By default, `sc-dlp` calculates the MD5 hash of every downloaded asset. If the calculated hash differs from the hash
|
|
194
|
+
By default, `sc-dlp` calculates the MD5 hash of every downloaded asset. If the calculated hash differs from the hash referenced in `project.json`, the asset is automatically renamed and `project.json` is updated so the resulting project remains valid and loadable.
|
|
131
195
|
|
|
132
196
|
### TurboWarp Assets
|
|
133
197
|
|
|
134
|
-
TurboWarp Share projects use SHA-256 for asset storage. `sc-dlp` automatically extracts the `md5extsToSha256` mapping from the project
|
|
198
|
+
TurboWarp Share projects use SHA-256 for asset storage. `sc-dlp` automatically extracts the `md5extsToSha256` mapping from the project page HTML to fetch and remap assets for Scratch compatibility.
|
|
135
199
|
|
|
136
200
|
## Privacy
|
|
137
201
|
|
|
138
|
-
By default, `sc-dlp` communicates directly with Scratch APIs (`api.scratch.mit.edu`, `projects.scratch.mit.edu`, `assets.scratch.mit.edu`) and TurboWarp APIs (`share.turbowarp.org`).
|
|
202
|
+
By default, `sc-dlp` communicates directly with Scratch APIs (`api.scratch.mit.edu`, `projects.scratch.mit.edu`, `assets.scratch.mit.edu`) and TurboWarp APIs (`share.turbowarp.org`). Configuration and session data are stored locally in `~/.sc-dlp/config.json` with `0600` permissions.
|
|
203
|
+
|
|
204
|
+
## Development
|
|
205
|
+
|
|
206
|
+
To build and test from source:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
git clone https://github.com/rbstxt/sc-dlp.git
|
|
210
|
+
cd sc-dlp
|
|
211
|
+
npm install
|
|
212
|
+
npm run build
|
|
213
|
+
npm test
|
|
214
|
+
```
|
package/lib/cli.cjs
CHANGED
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sc-dlp",
|
|
3
|
-
"version": "1.0.
|
|
4
|
-
"description": "A
|
|
3
|
+
"version": "1.0.1",
|
|
4
|
+
"description": "A high-performance CLI and Node.js library for downloading Scratch and TurboWarp projects.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"scratch",
|
|
@@ -80,4 +80,4 @@
|
|
|
80
80
|
"typescript": "^5.9.2",
|
|
81
81
|
"vitest": "^3.2.4"
|
|
82
82
|
}
|
|
83
|
-
}
|
|
83
|
+
}
|