sc-dlp 0.0.0-stage → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +137 -2
  3. package/lib/bundle-node.cjs +1875 -0
  4. package/lib/bundle-node.mjs +1846 -0
  5. package/lib/cli.cjs +3622 -0
  6. package/lib/types/cli/args.d.ts +3 -0
  7. package/lib/types/cli/auth/login.d.ts +1 -0
  8. package/lib/types/cli/auth/logout.d.ts +1 -0
  9. package/lib/types/cli/auth/session.d.ts +12 -0
  10. package/lib/types/cli/auth/status.d.ts +1 -0
  11. package/lib/types/cli/auth.d.ts +4 -0
  12. package/lib/types/cli/config.d.ts +5 -0
  13. package/lib/types/cli/config_command.d.ts +1 -0
  14. package/lib/types/cli/download/conflict.d.ts +15 -0
  15. package/lib/types/cli/download/display.d.ts +15 -0
  16. package/lib/types/cli/download/logger.d.ts +16 -0
  17. package/lib/types/cli/download/persistence.d.ts +4 -0
  18. package/lib/types/cli/download/thumbnail.d.ts +3 -0
  19. package/lib/types/cli/download.d.ts +5 -0
  20. package/lib/types/cli/types.d.ts +47 -0
  21. package/lib/types/cli/utils.d.ts +39 -0
  22. package/lib/types/cli.d.ts +1 -0
  23. package/lib/types/downloader/core.d.ts +3 -0
  24. package/lib/types/downloader/metadata/list.d.ts +13 -0
  25. package/lib/types/downloader/metadata/playlist/delete_account.d.ts +2 -0
  26. package/lib/types/downloader/metadata/playlist/featured.d.ts +2 -0
  27. package/lib/types/downloader/metadata/playlist/mystuff.d.ts +2 -0
  28. package/lib/types/downloader/metadata/playlist/shorthand.d.ts +2 -0
  29. package/lib/types/downloader/metadata/playlist/url.d.ts +2 -0
  30. package/lib/types/downloader/metadata/playlist.d.ts +2 -0
  31. package/lib/types/downloader/metadata/project.d.ts +3 -0
  32. package/lib/types/downloader/metadata/user.d.ts +12 -0
  33. package/lib/types/downloader/metadata.d.ts +4 -0
  34. package/lib/types/downloader/remote.d.ts +13 -0
  35. package/lib/types/downloader/sb2.d.ts +13 -0
  36. package/lib/types/downloader/sb3.d.ts +13 -0
  37. package/lib/types/downloader/turbowarp.d.ts +5 -0
  38. package/lib/types/downloader/types.d.ts +149 -0
  39. package/lib/types/downloader/utils.d.ts +56 -0
  40. package/lib/types/downloader/zip.d.ts +16 -0
  41. package/lib/types/downloader.d.ts +7 -0
  42. package/lib/types/environment.d.ts +6 -0
  43. package/lib/types/errors.d.ts +19 -0
  44. package/lib/types/export-node.d.ts +1 -0
  45. package/lib/types/fetch-asset.d.ts +12 -0
  46. package/lib/types/fetch-with-progress.d.ts +2 -0
  47. package/lib/types/fetch-wrapper.d.ts +5 -0
  48. package/lib/types/urls.d.ts +26 -0
  49. package/package.json +80 -3
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2019-2024 Thomas Weber
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,138 @@
1
- # Temporary Holding Version
1
+ # sc-dlp
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ A high-performance command-line tool for downloading Scratch and TurboWarp projects. Supports projects from Scratch 1.4 (sb), 2.0 (sb2), 3.0 (sb3), and TurboWarp Share (sha256).
4
+
5
+ `sc-dlp` is a fork of [forkphorus/sb-downloader](https://github.com/forkphorus/sb-downloader), redesigned for robust project archival, analysis, and research.
6
+
7
+ ## Features
8
+
9
+ - **TurboWarp Support**: Comprehensive support for `share.turbowarp.org` projects, including SHA-256 asset mapping and internal metadata extraction.
10
+ - **Authentication**: Login with Scratch account to access projects in "My Stuff" or recently viewed projects.
11
+ - **Advanced Collection Support**: Download projects from Loves, Favorites, Studios, Search results, Featured projects, Explore categories, and "My Stuff".
12
+ - **Automatic Pagination**: Automatically fetches all projects from multi-page collections.
13
+ - **Shorthand Aliases**: Quick access to any collection using simple patterns.
14
+ - **High Performance**: Parallel project downloads, asset fetching, and smart hash verification.
15
+ - **Rich CLI**: Professional logging with distinct prefixes (`[assets]`, `[mapping]`, `[compress]`), interactive prompts, and real-time progress bars.
16
+
17
+ ## Installation
18
+
19
+ ### From npm
20
+
21
+ ```bash
22
+ npm install -g sc-dlp
23
+ ```
24
+
25
+ Or run directly with `npx`:
26
+
27
+ ```bash
28
+ npx sc-dlp 60917032
29
+ ```
30
+
31
+ ### From source
32
+
33
+ ```bash
34
+ git clone https://github.com/rbstxt/sc-dlp.git
35
+ cd sc-dlp
36
+ npm install
37
+ npm run build
38
+ npm link
39
+ ```
40
+
41
+ ## Usage
42
+
43
+ Use project IDs, URLs, or shorthand aliases to download projects.
44
+
45
+ ```bash
46
+ # Single project (Scratch)
47
+ sc-dlp 60917032
48
+ sc-dlp https://scratch.mit.edu/projects/60917032/
49
+
50
+ # Single project (TurboWarp Share)
51
+ sc-dlp 4cd76e33-0e7a-468f-85c9-17799748a864
52
+ sc-dlp turbowarp/4cd76e33-0e7a-468f-85c9-17799748a864
53
+ sc-dlp https://share.turbowarp.org/projects/4cd76e33-0e7a-468f-85c9-17799748a864
54
+
55
+ # User collections
56
+ sc-dlp @scratchcat/loves
57
+ sc-dlp @scratchcat/favorites
58
+ sc-dlp @scratchcat
59
+
60
+ # Authentication
61
+ sc-dlp login
62
+ sc-dlp status
63
+ sc-dlp logout
64
+
65
+ # Specialized collections
66
+ sc-dlp mystuff/shared
67
+ sc-dlp featured/curated
68
+ sc-dlp explore/animations
69
+ sc-dlp recently
70
+ sc-dlp 10123/remixes
71
+ sc-dlp studio/1234
72
+ sc-dlp search/platformer
73
+ ```
74
+
75
+ ### Authentication
76
+
77
+ `sc-dlp login` supports three authentication methods:
78
+
79
+ - **Username & Password**: Full session access (requires solving a captcha if triggered).
80
+ - **Session ID**: Full session access using a `scratchsessionsid` cookie value.
81
+ - **X-Token**: Limited permissions access. Suitable for `recently` and public data, but cannot access `mystuff`.
82
+
83
+ ### Collection Aliases
84
+
85
+ | Collection | Alias Pattern | Note |
86
+ | ------------------- | ---------------------------------------------- | ----------------------------------- |
87
+ | **Loves** | `@user/loves`, `@user/lov` | Scrapes HTML for all pages. |
88
+ | **Favorites** | `@user/favorites`, `@user/fav` | Fetches via API. |
89
+ | **User Projects** | `@user` | Fetches all shared projects. |
90
+ | **My Stuff** | `mystuff`, `mystuff/shared`, `mystuff/trash` | Requires login. |
91
+ | **Recently Viewed** | `recently`, `recent` | Requires login. |
92
+ | **Featured** | `featured`, `featured/curated`, `featured/top` | Fetches from Scratch homepage. |
93
+ | **Explore** | `explore/[category]`, `search/[query]` | Fetches results from browse/search. |
94
+ | **Remixes** | `id/remixes`, `id/remix` | Fetches all remixes of a project. |
95
+ | **Studio** | `studio/id` | Fetches projects in a studio. |
96
+ | **TurboWarp** | `turbowarp/[id]`, `[uuid]` | Fetches from TurboWarp Share. |
97
+
98
+ ### Options
99
+
100
+ - `--help`: Shows the help screen.
101
+ - `-c`, `--concurrency NUMBER`: Number of projects to download in parallel (default: 1).
102
+ - `-m`, `--mtime`: Set file modification time to project last shared/modified date.
103
+ - `-t`, `--write-thumbnail`: Download project thumbnail.
104
+ - `--thumbnail-size WIDTHxHEIGHT`: Thumbnail size (default: 9999x9999).
105
+ - `--playlist-start NUMBER`: Playlist item to start at.
106
+ - `--playlist-end NUMBER`: Playlist item to end at.
107
+ - `-I`, `--playlist-items ITEMS`: Specific indices (e.g., 1,2,5-10).
108
+ - `--page NUMBER`: Page number to fetch for collections (e.g., users, favorites, loves, studios, search, mystuff).
109
+ - `--format EXT`: Output extension (sb3, sb2, sb, json).
110
+ - `-j`, `--json-only`: Download only the `project.json` file.
111
+ - `-a`, `--no-assets`: Download code only (no costumes/sounds).
112
+ - `--conflict ACTION`: Conflict action: `overwrite`, `skip`, or `keep-both`.
113
+ - `--retries NUMBER`: Number of times to retry failed asset downloads (default: 3).
114
+ - `-v`, `--verbose`: Show detailed information and stack traces on error.
115
+
116
+ ### File Conflict Resolution
117
+
118
+ If a project with the same name and ID already exists in the target directory, `sc-dlp` will prompt you for action. By default, sc-dlp includes the project ID in the filename (e.g., `Title (12345).sb3`) to avoid conflicts between different projects with the same title.
119
+
120
+ - **Overwrite**: Replace the existing file with the new download.
121
+ - **Skip**: Abort the download for this specific project.
122
+ - **Keep both**: Automatically rename the new file by appending a number (e.g., `Project 2.sb3`).
123
+
124
+ Once you make a choice, it will be automatically applied to all remaining conflicts within the same download session (e.g., when downloading a studio or user's projects) to ensure a smooth, unattended experience.
125
+
126
+ ## Advanced
127
+
128
+ ### Asset Integrity
129
+
130
+ By default, `sc-dlp` calculates the MD5 hash of every downloaded asset. If the calculated hash differs from the hash provided in the project metadata, the tool automatically renames the file and updates the `project.json` to ensure the project remains loadable and consistent.
131
+
132
+ ### TurboWarp Assets
133
+
134
+ TurboWarp Share projects use SHA-256 for asset storage. `sc-dlp` automatically extracts the `md5extsToSha256` mapping from the project's page HTML to correctly fetch and rename these assets for Scratch compatibility.
135
+
136
+ ## Privacy
137
+
138
+ By default, `sc-dlp` communicates directly with Scratch APIs (`api.scratch.mit.edu`, `projects.scratch.mit.edu`, `assets.scratch.mit.edu`) and TurboWarp APIs (`share.turbowarp.org`). Session data is stored locally in `~/.sc-dlp/config.json`.