sc-dlp 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +137 -2
- package/lib/bundle-node.cjs +1875 -0
- package/lib/bundle-node.mjs +1846 -0
- package/lib/cli.cjs +3622 -0
- package/lib/types/cli/args.d.ts +3 -0
- package/lib/types/cli/auth/login.d.ts +1 -0
- package/lib/types/cli/auth/logout.d.ts +1 -0
- package/lib/types/cli/auth/session.d.ts +12 -0
- package/lib/types/cli/auth/status.d.ts +1 -0
- package/lib/types/cli/auth.d.ts +4 -0
- package/lib/types/cli/config.d.ts +5 -0
- package/lib/types/cli/config_command.d.ts +1 -0
- package/lib/types/cli/download/conflict.d.ts +15 -0
- package/lib/types/cli/download/display.d.ts +15 -0
- package/lib/types/cli/download/logger.d.ts +16 -0
- package/lib/types/cli/download/persistence.d.ts +4 -0
- package/lib/types/cli/download/thumbnail.d.ts +3 -0
- package/lib/types/cli/download.d.ts +5 -0
- package/lib/types/cli/types.d.ts +47 -0
- package/lib/types/cli/utils.d.ts +39 -0
- package/lib/types/cli.d.ts +1 -0
- package/lib/types/downloader/core.d.ts +3 -0
- package/lib/types/downloader/metadata/list.d.ts +13 -0
- package/lib/types/downloader/metadata/playlist/delete_account.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/featured.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/mystuff.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/shorthand.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist/url.d.ts +2 -0
- package/lib/types/downloader/metadata/playlist.d.ts +2 -0
- package/lib/types/downloader/metadata/project.d.ts +3 -0
- package/lib/types/downloader/metadata/user.d.ts +12 -0
- package/lib/types/downloader/metadata.d.ts +4 -0
- package/lib/types/downloader/remote.d.ts +13 -0
- package/lib/types/downloader/sb2.d.ts +13 -0
- package/lib/types/downloader/sb3.d.ts +13 -0
- package/lib/types/downloader/turbowarp.d.ts +5 -0
- package/lib/types/downloader/types.d.ts +149 -0
- package/lib/types/downloader/utils.d.ts +56 -0
- package/lib/types/downloader/zip.d.ts +16 -0
- package/lib/types/downloader.d.ts +7 -0
- package/lib/types/environment.d.ts +6 -0
- package/lib/types/errors.d.ts +19 -0
- package/lib/types/export-node.d.ts +1 -0
- package/lib/types/fetch-asset.d.ts +12 -0
- package/lib/types/fetch-with-progress.d.ts +2 -0
- package/lib/types/fetch-wrapper.d.ts +5 -0
- package/lib/types/urls.d.ts +26 -0
- package/package.json +80 -3
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2019-2024 Thomas Weber
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,138 @@
|
|
|
1
|
-
#
|
|
1
|
+
# sc-dlp
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A high-performance command-line tool for downloading Scratch and TurboWarp projects. Supports projects from Scratch 1.4 (sb), 2.0 (sb2), 3.0 (sb3), and TurboWarp Share (sha256).
|
|
4
|
+
|
|
5
|
+
`sc-dlp` is a fork of [forkphorus/sb-downloader](https://github.com/forkphorus/sb-downloader), redesigned for robust project archival, analysis, and research.
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
|
|
9
|
+
- **TurboWarp Support**: Comprehensive support for `share.turbowarp.org` projects, including SHA-256 asset mapping and internal metadata extraction.
|
|
10
|
+
- **Authentication**: Login with Scratch account to access projects in "My Stuff" or recently viewed projects.
|
|
11
|
+
- **Advanced Collection Support**: Download projects from Loves, Favorites, Studios, Search results, Featured projects, Explore categories, and "My Stuff".
|
|
12
|
+
- **Automatic Pagination**: Automatically fetches all projects from multi-page collections.
|
|
13
|
+
- **Shorthand Aliases**: Quick access to any collection using simple patterns.
|
|
14
|
+
- **High Performance**: Parallel project downloads, asset fetching, and smart hash verification.
|
|
15
|
+
- **Rich CLI**: Professional logging with distinct prefixes (`[assets]`, `[mapping]`, `[compress]`), interactive prompts, and real-time progress bars.
|
|
16
|
+
|
|
17
|
+
## Installation
|
|
18
|
+
|
|
19
|
+
### From npm
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
npm install -g sc-dlp
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Or run directly with `npx`:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
npx sc-dlp 60917032
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
### From source
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
git clone https://github.com/rbstxt/sc-dlp.git
|
|
35
|
+
cd sc-dlp
|
|
36
|
+
npm install
|
|
37
|
+
npm run build
|
|
38
|
+
npm link
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## Usage
|
|
42
|
+
|
|
43
|
+
Use project IDs, URLs, or shorthand aliases to download projects.
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
# Single project (Scratch)
|
|
47
|
+
sc-dlp 60917032
|
|
48
|
+
sc-dlp https://scratch.mit.edu/projects/60917032/
|
|
49
|
+
|
|
50
|
+
# Single project (TurboWarp Share)
|
|
51
|
+
sc-dlp 4cd76e33-0e7a-468f-85c9-17799748a864
|
|
52
|
+
sc-dlp turbowarp/4cd76e33-0e7a-468f-85c9-17799748a864
|
|
53
|
+
sc-dlp https://share.turbowarp.org/projects/4cd76e33-0e7a-468f-85c9-17799748a864
|
|
54
|
+
|
|
55
|
+
# User collections
|
|
56
|
+
sc-dlp @scratchcat/loves
|
|
57
|
+
sc-dlp @scratchcat/favorites
|
|
58
|
+
sc-dlp @scratchcat
|
|
59
|
+
|
|
60
|
+
# Authentication
|
|
61
|
+
sc-dlp login
|
|
62
|
+
sc-dlp status
|
|
63
|
+
sc-dlp logout
|
|
64
|
+
|
|
65
|
+
# Specialized collections
|
|
66
|
+
sc-dlp mystuff/shared
|
|
67
|
+
sc-dlp featured/curated
|
|
68
|
+
sc-dlp explore/animations
|
|
69
|
+
sc-dlp recently
|
|
70
|
+
sc-dlp 10123/remixes
|
|
71
|
+
sc-dlp studio/1234
|
|
72
|
+
sc-dlp search/platformer
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Authentication
|
|
76
|
+
|
|
77
|
+
`sc-dlp login` supports three authentication methods:
|
|
78
|
+
|
|
79
|
+
- **Username & Password**: Full session access (requires solving a captcha if triggered).
|
|
80
|
+
- **Session ID**: Full session access using a `scratchsessionsid` cookie value.
|
|
81
|
+
- **X-Token**: Limited permissions access. Suitable for `recently` and public data, but cannot access `mystuff`.
|
|
82
|
+
|
|
83
|
+
### Collection Aliases
|
|
84
|
+
|
|
85
|
+
| Collection | Alias Pattern | Note |
|
|
86
|
+
| ------------------- | ---------------------------------------------- | ----------------------------------- |
|
|
87
|
+
| **Loves** | `@user/loves`, `@user/lov` | Scrapes HTML for all pages. |
|
|
88
|
+
| **Favorites** | `@user/favorites`, `@user/fav` | Fetches via API. |
|
|
89
|
+
| **User Projects** | `@user` | Fetches all shared projects. |
|
|
90
|
+
| **My Stuff** | `mystuff`, `mystuff/shared`, `mystuff/trash` | Requires login. |
|
|
91
|
+
| **Recently Viewed** | `recently`, `recent` | Requires login. |
|
|
92
|
+
| **Featured** | `featured`, `featured/curated`, `featured/top` | Fetches from Scratch homepage. |
|
|
93
|
+
| **Explore** | `explore/[category]`, `search/[query]` | Fetches results from browse/search. |
|
|
94
|
+
| **Remixes** | `id/remixes`, `id/remix` | Fetches all remixes of a project. |
|
|
95
|
+
| **Studio** | `studio/id` | Fetches projects in a studio. |
|
|
96
|
+
| **TurboWarp** | `turbowarp/[id]`, `[uuid]` | Fetches from TurboWarp Share. |
|
|
97
|
+
|
|
98
|
+
### Options
|
|
99
|
+
|
|
100
|
+
- `--help`: Shows the help screen.
|
|
101
|
+
- `-c`, `--concurrency NUMBER`: Number of projects to download in parallel (default: 1).
|
|
102
|
+
- `-m`, `--mtime`: Set file modification time to project last shared/modified date.
|
|
103
|
+
- `-t`, `--write-thumbnail`: Download project thumbnail.
|
|
104
|
+
- `--thumbnail-size WIDTHxHEIGHT`: Thumbnail size (default: 9999x9999).
|
|
105
|
+
- `--playlist-start NUMBER`: Playlist item to start at.
|
|
106
|
+
- `--playlist-end NUMBER`: Playlist item to end at.
|
|
107
|
+
- `-I`, `--playlist-items ITEMS`: Specific indices (e.g., 1,2,5-10).
|
|
108
|
+
- `--page NUMBER`: Page number to fetch for collections (e.g., users, favorites, loves, studios, search, mystuff).
|
|
109
|
+
- `--format EXT`: Output extension (sb3, sb2, sb, json).
|
|
110
|
+
- `-j`, `--json-only`: Download only the `project.json` file.
|
|
111
|
+
- `-a`, `--no-assets`: Download code only (no costumes/sounds).
|
|
112
|
+
- `--conflict ACTION`: Conflict action: `overwrite`, `skip`, or `keep-both`.
|
|
113
|
+
- `--retries NUMBER`: Number of times to retry failed asset downloads (default: 3).
|
|
114
|
+
- `-v`, `--verbose`: Show detailed information and stack traces on error.
|
|
115
|
+
|
|
116
|
+
### File Conflict Resolution
|
|
117
|
+
|
|
118
|
+
If a project with the same name and ID already exists in the target directory, `sc-dlp` will prompt you for action. By default, sc-dlp includes the project ID in the filename (e.g., `Title (12345).sb3`) to avoid conflicts between different projects with the same title.
|
|
119
|
+
|
|
120
|
+
- **Overwrite**: Replace the existing file with the new download.
|
|
121
|
+
- **Skip**: Abort the download for this specific project.
|
|
122
|
+
- **Keep both**: Automatically rename the new file by appending a number (e.g., `Project 2.sb3`).
|
|
123
|
+
|
|
124
|
+
Once you make a choice, it will be automatically applied to all remaining conflicts within the same download session (e.g., when downloading a studio or user's projects) to ensure a smooth, unattended experience.
|
|
125
|
+
|
|
126
|
+
## Advanced
|
|
127
|
+
|
|
128
|
+
### Asset Integrity
|
|
129
|
+
|
|
130
|
+
By default, `sc-dlp` calculates the MD5 hash of every downloaded asset. If the calculated hash differs from the hash provided in the project metadata, the tool automatically renames the file and updates the `project.json` to ensure the project remains loadable and consistent.
|
|
131
|
+
|
|
132
|
+
### TurboWarp Assets
|
|
133
|
+
|
|
134
|
+
TurboWarp Share projects use SHA-256 for asset storage. `sc-dlp` automatically extracts the `md5extsToSha256` mapping from the project's page HTML to correctly fetch and rename these assets for Scratch compatibility.
|
|
135
|
+
|
|
136
|
+
## Privacy
|
|
137
|
+
|
|
138
|
+
By default, `sc-dlp` communicates directly with Scratch APIs (`api.scratch.mit.edu`, `projects.scratch.mit.edu`, `assets.scratch.mit.edu`) and TurboWarp APIs (`share.turbowarp.org`). Session data is stored locally in `~/.sc-dlp/config.json`.
|