git-ripper 1.5.1 → 1.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +20 -20
- package/README.md +340 -340
- package/bin/git-ripper.js +3 -3
- package/package.json +62 -62
- package/src/archiver.js +210 -210
- package/src/downloader.js +904 -904
- package/src/index.js +195 -195
- package/src/parser.js +37 -37
- package/src/resumeManager.js +213 -213
package/README.md
CHANGED
|
@@ -1,340 +1,340 @@
|
|
|
1
|
-
# Git-ripper
|
|
2
|
-
|
|
3
|
-
<div align="center">
|
|
4
|
-
|
|
5
|
-
[](https://www.npmjs.com/package/git-ripper)
|
|
6
|
-
[](https://github.com/sairajB/git-ripper/blob/main/LICENSE)
|
|
7
|
-
[](https://www.npmjs.com/package/git-ripper)
|
|
8
|
-
[](https://www.npmjs.com/package/git-ripper)
|
|
9
|
-
[](https://github.com/sairajB/git-ripper/issues)
|
|
10
|
-
[](https://github.com/sairajB/git-ripper/stargazers)
|
|
11
|
-
[](https://github.com/sairajB/git-ripper/network)
|
|
12
|
-
[](https://github.com/sairajB/git-ripper/commits/master)
|
|
13
|
-
|
|
14
|
-
**Download specific folders from GitHub repositories without cloning the entire codebase**
|
|
15
|
-
|
|
16
|
-
[Installation](#installation) •
|
|
17
|
-
[Usage](#usage) •
|
|
18
|
-
[Features](#features) •
|
|
19
|
-
[Examples](#examples) •
|
|
20
|
-
[Configuration](#configuration) •
|
|
21
|
-
[Troubleshooting](#troubleshooting) •
|
|
22
|
-
[Contributing](#contributing) •
|
|
23
|
-
[License](#license)
|
|
24
|
-
|
|
25
|
-
</div>
|
|
26
|
-
|
|
27
|
-
## Why Git-ripper?
|
|
28
|
-
|
|
29
|
-
Have you ever needed just a single component from a massive repository? Or wanted to reference a specific configuration directory without downloading gigabytes of code? Git-ripper solves this problem by letting you extract and download only the folders you need, saving bandwidth, time, and disk space.
|
|
30
|
-
|
|
31
|
-
## Features
|
|
32
|
-
|
|
33
|
-
- **Selective Downloads**: Fetch specific folders instead of entire repositories
|
|
34
|
-
- **Resume Interrupted Downloads**: Automatically resume downloads that were interrupted or failed
|
|
35
|
-
- **Progress Tracking**: Visual progress indicators with file-by-file download status
|
|
36
|
-
- **File Integrity Verification**: Ensures downloaded files are complete and uncorrupted
|
|
37
|
-
- **Directory Structure**: Preserves complete folder structure
|
|
38
|
-
- **Custom Output**: Specify your preferred output directory
|
|
39
|
-
- **Branch Support**: Works with any branch, not just the default one
|
|
40
|
-
- **Archive Export**: Create ZIP archives of downloaded content
|
|
41
|
-
- **Checkpoint Management**: View and manage saved download progress
|
|
42
|
-
- **Simple Interface**: Clean, intuitive command-line experience
|
|
43
|
-
- **Lightweight**: Minimal dependencies and fast execution
|
|
44
|
-
- **No Authentication**: Works with public repositories without requiring credentials
|
|
45
|
-
|
|
46
|
-
## Installation
|
|
47
|
-
|
|
48
|
-
### Requirements
|
|
49
|
-
|
|
50
|
-
Git-ripper requires Node.js >=16.0.0 due to its use of modern JavaScript features and built-in Node.js modules.
|
|
51
|
-
|
|
52
|
-
### Global Installation (Recommended)
|
|
53
|
-
|
|
54
|
-
```bash
|
|
55
|
-
npm install -g git-ripper
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
This installs Git-ripper as a global command-line tool accessible from anywhere in your terminal.
|
|
59
|
-
|
|
60
|
-
### On-demand Usage
|
|
61
|
-
|
|
62
|
-
```bash
|
|
63
|
-
npx git-ripper <github-folder-url>
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
Run Git-ripper directly without installation using `npx`.
|
|
67
|
-
|
|
68
|
-
## Usage
|
|
69
|
-
|
|
70
|
-
### Basic Command
|
|
71
|
-
|
|
72
|
-
```bash
|
|
73
|
-
git-ripper https://github.com/username/repository/tree/branch/folder
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
### With Custom Output Directory
|
|
77
|
-
|
|
78
|
-
```bash
|
|
79
|
-
git-ripper https://github.com/username/repository/tree/branch/folder -o ./my-output-folder
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
### Creating ZIP Archive
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
git-ripper https://github.com/username/repository/tree/branch/folder --zip
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
### Creating ZIP Archive with Custom Name
|
|
89
|
-
|
|
90
|
-
```bash
|
|
91
|
-
git-ripper https://github.com/username/repository/tree/branch/folder --zip="my-archive.zip"
|
|
92
|
-
```
|
|
93
|
-
|
|
94
|
-
### Command Line Options
|
|
95
|
-
|
|
96
|
-
| Option | Description | Default |
|
|
97
|
-
| -------------------------- | ---------------------------------------- | ----------------- |
|
|
98
|
-
| `-o, --output <directory>` | Specify output directory | Current directory |
|
|
99
|
-
| `--gh-token <token>` | GitHub Personal Access Token | - |
|
|
100
|
-
| `--zip [filename]` | Create ZIP archive of downloaded content | - |
|
|
101
|
-
| `--no-resume` | Disable resume functionality | - |
|
|
102
|
-
| `--force-restart` | Ignore existing checkpoints and restart | - |
|
|
103
|
-
| `--list-checkpoints` | List all saved download checkpoints | - |
|
|
104
|
-
| `-V, --version` | Show version number | - |
|
|
105
|
-
| `-h, --help` | Show help | - |
|
|
106
|
-
|
|
107
|
-
## Authentication (Private Repositories & Rate Limits)
|
|
108
|
-
|
|
109
|
-
To download from private repositories or to increase your API rate limit, you need to provide a GitHub Personal Access Token (PAT).
|
|
110
|
-
|
|
111
|
-
### How to Generate a Token
|
|
112
|
-
|
|
113
|
-
You can use either a **Fine-grained token** (Recommended) or a **Classic token**.
|
|
114
|
-
|
|
115
|
-
#### Option A: Fine-grained Token (Recommended)
|
|
116
|
-
|
|
117
|
-
1. Go to **Settings** > **Developer settings** > **Personal access tokens** > **Fine-grained tokens**.
|
|
118
|
-
2. Click **Generate new token**.
|
|
119
|
-
3. Name it (e.g., "Git-ripper").
|
|
120
|
-
4. **Resource owner**: Select your user.
|
|
121
|
-
5. **Repository access**: Select **Only select repositories** and choose the private repository you want to download from.
|
|
122
|
-
6. **Permissions**:
|
|
123
|
-
- Click on **Repository permissions**.
|
|
124
|
-
- Find **Contents** and change Access to **Read-only**.
|
|
125
|
-
- _Note: Metadata permission is selected automatically._
|
|
126
|
-
7. Click **Generate token**.
|
|
127
|
-
|
|
128
|
-
#### Option B: Classic Token
|
|
129
|
-
|
|
130
|
-
1. Go to **Settings** > **Developer settings** > **Personal access tokens** > **Tokens (classic)**.
|
|
131
|
-
2. Click **Generate new token** > **Generate new token (classic)**.
|
|
132
|
-
3. Give your token a descriptive name.
|
|
133
|
-
4. **Select Scopes:**
|
|
134
|
-
- **For Private Repositories:** Select the **`repo`** scope (Full control of private repositories).
|
|
135
|
-
5. Click **Generate token**.
|
|
136
|
-
|
|
137
|
-
### Using the Token
|
|
138
|
-
|
|
139
|
-
Pass the token using the `--gh-token` flag:
|
|
140
|
-
|
|
141
|
-
```bash
|
|
142
|
-
git-ripper https://github.com/username/private-repo/tree/main/src --gh-token ghp_YourTokenHere
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
> **Security Note:** Be careful not to share your token or commit it to public repositories.
|
|
146
|
-
|
|
147
|
-
## Examples
|
|
148
|
-
|
|
149
|
-
### Extract a Component Library
|
|
150
|
-
|
|
151
|
-
```bash
|
|
152
|
-
# Download React DOM package
|
|
153
|
-
git-ripper https://github.com/facebook/react/tree/main/packages/react-dom
|
|
154
|
-
```
|
|
155
|
-
|
|
156
|
-
### Get Configuration Files
|
|
157
|
-
|
|
158
|
-
```bash
|
|
159
|
-
# Extract VS Code build configuration
|
|
160
|
-
git-ripper https://github.com/microsoft/vscode/tree/main/build -o ./vscode-build-config
|
|
161
|
-
```
|
|
162
|
-
|
|
163
|
-
### Download Documentation
|
|
164
|
-
|
|
165
|
-
```bash
|
|
166
|
-
# Get Node.js documentation
|
|
167
|
-
git-ripper https://github.com/nodejs/node/tree/main/doc -o ./node-docs
|
|
168
|
-
```
|
|
169
|
-
|
|
170
|
-
### Copy UI Templates
|
|
171
|
-
|
|
172
|
-
```bash
|
|
173
|
-
# Extract Tailwind components
|
|
174
|
-
git-ripper https://github.com/tailwindlabs/tailwindcss/tree/master/src/components -o ./tailwind-components
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
### Download from Private Repository
|
|
178
|
-
|
|
179
|
-
```bash
|
|
180
|
-
# Download from a private repository using a token
|
|
181
|
-
git-ripper https://github.com/my-org/private-project/tree/main/src --gh-token ghp_abc123...
|
|
182
|
-
```
|
|
183
|
-
|
|
184
|
-
### Download and Create Archive
|
|
185
|
-
|
|
186
|
-
```bash
|
|
187
|
-
# Download React DOM package and create a ZIP archive
|
|
188
|
-
git-ripper https://github.com/facebook/react/tree/main/packages/react-dom --zip
|
|
189
|
-
|
|
190
|
-
# Extract VS Code build configuration with custom archive name
|
|
191
|
-
git-ripper https://github.com/microsoft/vscode/tree/main/build --zip="vscode-build.zip"
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
## Resume Downloads
|
|
195
|
-
|
|
196
|
-
Git-ripper now supports resuming interrupted downloads, making it perfect for large folders or unstable network connections.
|
|
197
|
-
|
|
198
|
-
### Automatic Resume (Default Behavior)
|
|
199
|
-
|
|
200
|
-
```bash
|
|
201
|
-
# Start a download
|
|
202
|
-
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench
|
|
203
|
-
|
|
204
|
-
# If interrupted (Ctrl+C, network issues, etc.), simply run the same command again
|
|
205
|
-
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench
|
|
206
|
-
# It will automatically resume from where it left off
|
|
207
|
-
```
|
|
208
|
-
|
|
209
|
-
### Force Restart
|
|
210
|
-
|
|
211
|
-
```bash
|
|
212
|
-
# Ignore any existing progress and start fresh
|
|
213
|
-
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench --force-restart
|
|
214
|
-
```
|
|
215
|
-
|
|
216
|
-
### Disable Resume
|
|
217
|
-
|
|
218
|
-
```bash
|
|
219
|
-
# Use traditional behavior without resume functionality
|
|
220
|
-
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench --no-resume
|
|
221
|
-
```
|
|
222
|
-
|
|
223
|
-
### Manage Checkpoints
|
|
224
|
-
|
|
225
|
-
```bash
|
|
226
|
-
# List all saved download progress
|
|
227
|
-
git-ripper --list-checkpoints
|
|
228
|
-
|
|
229
|
-
# Output shows:
|
|
230
|
-
# 1. ID: a1b2c3d4
|
|
231
|
-
# URL: https://github.com/microsoft/vscode/tree/main/src/vs/workbench
|
|
232
|
-
# Progress: 45/120 files
|
|
233
|
-
# Last Updated: 2025-06-04T10:30:00Z
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
### Resume Features
|
|
237
|
-
|
|
238
|
-
- **Automatic Progress Saving**: Downloads are checkpointed every few files
|
|
239
|
-
- **File Integrity Verification**: Ensures existing files are complete and valid
|
|
240
|
-
- **Smart Recovery**: Detects corrupted or incomplete files and re-downloads them
|
|
241
|
-
- **Multi-Download Support**: Manage multiple concurrent download projects
|
|
242
|
-
- **Progress Indicators**: Visual feedback showing completed vs remaining files
|
|
243
|
-
|
|
244
|
-
## How It Works
|
|
245
|
-
|
|
246
|
-
Git-ripper operates in five stages:
|
|
247
|
-
|
|
248
|
-
1. **URL Parsing**: Extracts repository owner, name, branch, and target folder path
|
|
249
|
-
2. **Resume Check**: Looks for existing download progress and validates already downloaded files
|
|
250
|
-
3. **API Request**: Uses GitHub's API to fetch the folder structure
|
|
251
|
-
4. **Content Download**: Retrieves each file individually while maintaining directory structure and saving progress
|
|
252
|
-
5. **Local Storage or Archiving**: Saves files to your specified output directory or creates an archive
|
|
253
|
-
|
|
254
|
-
The resume functionality uses checkpoint files stored in `.git_ripper_checkpoints/` to track download progress, file integrity hashes, and metadata for each download session.
|
|
255
|
-
|
|
256
|
-
## Configuration
|
|
257
|
-
|
|
258
|
-
Git-ripper works out of the box without configuration. For rate-limited GitHub API usage or private repositories, use the `--gh-token` option as described in the [Authentication](#authentication-private-repositories--rate-limits) section.
|
|
259
|
-
|
|
260
|
-
## Troubleshooting
|
|
261
|
-
|
|
262
|
-
### Common Issues
|
|
263
|
-
|
|
264
|
-
#### Rate Limit Exceeded
|
|
265
|
-
|
|
266
|
-
```
|
|
267
|
-
Error: Request failed with status code 403
|
|
268
|
-
```
|
|
269
|
-
|
|
270
|
-
**Solution**: GitHub limits unauthenticated API requests. You can either:
|
|
271
|
-
|
|
272
|
-
1. Wait a few minutes and try again
|
|
273
|
-
2. Use the `--gh-token` option with a Personal Access Token to significantly increase your rate limit
|
|
274
|
-
|
|
275
|
-
#### Invalid URL Format
|
|
276
|
-
|
|
277
|
-
```
|
|
278
|
-
Error: Invalid GitHub URL format
|
|
279
|
-
```
|
|
280
|
-
|
|
281
|
-
**Solution**: Ensure your URL follows the pattern: `https://github.com/owner/repo/tree/branch/folder`
|
|
282
|
-
|
|
283
|
-
#### Folder Not Found
|
|
284
|
-
|
|
285
|
-
```
|
|
286
|
-
Error: Path not found in repository
|
|
287
|
-
```
|
|
288
|
-
|
|
289
|
-
**Solution**: Verify the folder path exists in the specified branch and repository.
|
|
290
|
-
|
|
291
|
-
#### Resume Issues
|
|
292
|
-
|
|
293
|
-
If you encounter problems with resume functionality:
|
|
294
|
-
|
|
295
|
-
```bash
|
|
296
|
-
# Clear all checkpoints and start fresh
|
|
297
|
-
git-ripper https://github.com/owner/repo/tree/branch/folder --force-restart
|
|
298
|
-
|
|
299
|
-
# Or disable resume entirely
|
|
300
|
-
git-ripper https://github.com/owner/repo/tree/branch/folder --no-resume
|
|
301
|
-
```
|
|
302
|
-
|
|
303
|
-
#### Corrupted Download
|
|
304
|
-
|
|
305
|
-
If files appear corrupted after resume:
|
|
306
|
-
|
|
307
|
-
```bash
|
|
308
|
-
# Force restart will re-download everything
|
|
309
|
-
git-ripper https://github.com/owner/repo/tree/branch/folder --force-restart
|
|
310
|
-
```
|
|
311
|
-
|
|
312
|
-
The resume feature automatically detects and re-downloads corrupted files, but `--force-restart` ensures a completely clean download.
|
|
313
|
-
|
|
314
|
-
## Contributing
|
|
315
|
-
|
|
316
|
-
Contributions make the open-source community an amazing place to learn, inspire, and create. Any contributions to Git-ripper are **greatly appreciated**.
|
|
317
|
-
|
|
318
|
-
1. Fork the repository
|
|
319
|
-
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
|
|
320
|
-
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
|
|
321
|
-
4. Push to the branch (`git push origin feature/amazing-feature`)
|
|
322
|
-
5. Open a Pull Request
|
|
323
|
-
|
|
324
|
-
See the [open issues](https://github.com/sairajB/git-ripper/issues) for a list of proposed features and known issues.
|
|
325
|
-
|
|
326
|
-
## License
|
|
327
|
-
|
|
328
|
-
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
329
|
-
|
|
330
|
-
## Acknowledgments
|
|
331
|
-
|
|
332
|
-
- GitHub API for making this tool possible
|
|
333
|
-
- Everyone who has provided feedback and suggestions
|
|
334
|
-
|
|
335
|
-
---
|
|
336
|
-
|
|
337
|
-
<div align="center">
|
|
338
|
-
<p>Made with ❤️ by <a href="https://github.com/sairajB">sairajB</a></p>
|
|
339
|
-
<p>If you find this tool useful, consider <a href="https://github.com/sponsors/sairajB">sponsoring</a> its development</p>
|
|
340
|
-
</div>
|
|
1
|
+
# Git-ripper
|
|
2
|
+
|
|
3
|
+
<div align="center">
|
|
4
|
+
|
|
5
|
+
[](https://www.npmjs.com/package/git-ripper)
|
|
6
|
+
[](https://github.com/sairajB/git-ripper/blob/main/LICENSE)
|
|
7
|
+
[](https://www.npmjs.com/package/git-ripper)
|
|
8
|
+
[](https://www.npmjs.com/package/git-ripper)
|
|
9
|
+
[](https://github.com/sairajB/git-ripper/issues)
|
|
10
|
+
[](https://github.com/sairajB/git-ripper/stargazers)
|
|
11
|
+
[](https://github.com/sairajB/git-ripper/network)
|
|
12
|
+
[](https://github.com/sairajB/git-ripper/commits/master)
|
|
13
|
+
|
|
14
|
+
**Download specific folders from GitHub repositories without cloning the entire codebase**
|
|
15
|
+
|
|
16
|
+
[Installation](#installation) •
|
|
17
|
+
[Usage](#usage) •
|
|
18
|
+
[Features](#features) •
|
|
19
|
+
[Examples](#examples) •
|
|
20
|
+
[Configuration](#configuration) •
|
|
21
|
+
[Troubleshooting](#troubleshooting) •
|
|
22
|
+
[Contributing](#contributing) •
|
|
23
|
+
[License](#license)
|
|
24
|
+
|
|
25
|
+
</div>
|
|
26
|
+
|
|
27
|
+
## Why Git-ripper?
|
|
28
|
+
|
|
29
|
+
Have you ever needed just a single component from a massive repository? Or wanted to reference a specific configuration directory without downloading gigabytes of code? Git-ripper solves this problem by letting you extract and download only the folders you need, saving bandwidth, time, and disk space.
|
|
30
|
+
|
|
31
|
+
## Features
|
|
32
|
+
|
|
33
|
+
- **Selective Downloads**: Fetch specific folders instead of entire repositories
|
|
34
|
+
- **Resume Interrupted Downloads**: Automatically resume downloads that were interrupted or failed
|
|
35
|
+
- **Progress Tracking**: Visual progress indicators with file-by-file download status
|
|
36
|
+
- **File Integrity Verification**: Ensures downloaded files are complete and uncorrupted
|
|
37
|
+
- **Directory Structure**: Preserves complete folder structure
|
|
38
|
+
- **Custom Output**: Specify your preferred output directory
|
|
39
|
+
- **Branch Support**: Works with any branch, not just the default one
|
|
40
|
+
- **Archive Export**: Create ZIP archives of downloaded content
|
|
41
|
+
- **Checkpoint Management**: View and manage saved download progress
|
|
42
|
+
- **Simple Interface**: Clean, intuitive command-line experience
|
|
43
|
+
- **Lightweight**: Minimal dependencies and fast execution
|
|
44
|
+
- **No Authentication**: Works with public repositories without requiring credentials
|
|
45
|
+
|
|
46
|
+
## Installation
|
|
47
|
+
|
|
48
|
+
### Requirements
|
|
49
|
+
|
|
50
|
+
Git-ripper requires Node.js >=16.0.0 due to its use of modern JavaScript features and built-in Node.js modules.
|
|
51
|
+
|
|
52
|
+
### Global Installation (Recommended)
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
npm install -g git-ripper
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
This installs Git-ripper as a global command-line tool accessible from anywhere in your terminal.
|
|
59
|
+
|
|
60
|
+
### On-demand Usage
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
npx git-ripper <github-folder-url>
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Run Git-ripper directly without installation using `npx`.
|
|
67
|
+
|
|
68
|
+
## Usage
|
|
69
|
+
|
|
70
|
+
### Basic Command
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
git-ripper https://github.com/username/repository/tree/branch/folder
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
### With Custom Output Directory
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
git-ripper https://github.com/username/repository/tree/branch/folder -o ./my-output-folder
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
### Creating ZIP Archive
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
git-ripper https://github.com/username/repository/tree/branch/folder --zip
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
### Creating ZIP Archive with Custom Name
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
git-ripper https://github.com/username/repository/tree/branch/folder --zip="my-archive.zip"
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
### Command Line Options
|
|
95
|
+
|
|
96
|
+
| Option | Description | Default |
|
|
97
|
+
| -------------------------- | ---------------------------------------- | ----------------- |
|
|
98
|
+
| `-o, --output <directory>` | Specify output directory | Current directory |
|
|
99
|
+
| `--gh-token <token>` | GitHub Personal Access Token | - |
|
|
100
|
+
| `--zip [filename]` | Create ZIP archive of downloaded content | - |
|
|
101
|
+
| `--no-resume` | Disable resume functionality | - |
|
|
102
|
+
| `--force-restart` | Ignore existing checkpoints and restart | - |
|
|
103
|
+
| `--list-checkpoints` | List all saved download checkpoints | - |
|
|
104
|
+
| `-V, --version` | Show version number | - |
|
|
105
|
+
| `-h, --help` | Show help | - |
|
|
106
|
+
|
|
107
|
+
## Authentication (Private Repositories & Rate Limits)
|
|
108
|
+
|
|
109
|
+
To download from private repositories or to increase your API rate limit, you need to provide a GitHub Personal Access Token (PAT).
|
|
110
|
+
|
|
111
|
+
### How to Generate a Token
|
|
112
|
+
|
|
113
|
+
You can use either a **Fine-grained token** (Recommended) or a **Classic token**.
|
|
114
|
+
|
|
115
|
+
#### Option A: Fine-grained Token (Recommended)
|
|
116
|
+
|
|
117
|
+
1. Go to **Settings** > **Developer settings** > **Personal access tokens** > **Fine-grained tokens**.
|
|
118
|
+
2. Click **Generate new token**.
|
|
119
|
+
3. Name it (e.g., "Git-ripper").
|
|
120
|
+
4. **Resource owner**: Select your user.
|
|
121
|
+
5. **Repository access**: Select **Only select repositories** and choose the private repository you want to download from.
|
|
122
|
+
6. **Permissions**:
|
|
123
|
+
- Click on **Repository permissions**.
|
|
124
|
+
- Find **Contents** and change Access to **Read-only**.
|
|
125
|
+
- _Note: Metadata permission is selected automatically._
|
|
126
|
+
7. Click **Generate token**.
|
|
127
|
+
|
|
128
|
+
#### Option B: Classic Token
|
|
129
|
+
|
|
130
|
+
1. Go to **Settings** > **Developer settings** > **Personal access tokens** > **Tokens (classic)**.
|
|
131
|
+
2. Click **Generate new token** > **Generate new token (classic)**.
|
|
132
|
+
3. Give your token a descriptive name.
|
|
133
|
+
4. **Select Scopes:**
|
|
134
|
+
- **For Private Repositories:** Select the **`repo`** scope (Full control of private repositories).
|
|
135
|
+
5. Click **Generate token**.
|
|
136
|
+
|
|
137
|
+
### Using the Token
|
|
138
|
+
|
|
139
|
+
Pass the token using the `--gh-token` flag:
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
git-ripper https://github.com/username/private-repo/tree/main/src --gh-token ghp_YourTokenHere
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
> **Security Note:** Be careful not to share your token or commit it to public repositories.
|
|
146
|
+
|
|
147
|
+
## Examples
|
|
148
|
+
|
|
149
|
+
### Extract a Component Library
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
# Download React DOM package
|
|
153
|
+
git-ripper https://github.com/facebook/react/tree/main/packages/react-dom
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
### Get Configuration Files
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
# Extract VS Code build configuration
|
|
160
|
+
git-ripper https://github.com/microsoft/vscode/tree/main/build -o ./vscode-build-config
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
### Download Documentation
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
# Get Node.js documentation
|
|
167
|
+
git-ripper https://github.com/nodejs/node/tree/main/doc -o ./node-docs
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
### Copy UI Templates
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
# Extract Tailwind components
|
|
174
|
+
git-ripper https://github.com/tailwindlabs/tailwindcss/tree/master/src/components -o ./tailwind-components
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Download from Private Repository
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
# Download from a private repository using a token
|
|
181
|
+
git-ripper https://github.com/my-org/private-project/tree/main/src --gh-token ghp_abc123...
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
### Download and Create Archive
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
# Download React DOM package and create a ZIP archive
|
|
188
|
+
git-ripper https://github.com/facebook/react/tree/main/packages/react-dom --zip
|
|
189
|
+
|
|
190
|
+
# Extract VS Code build configuration with custom archive name
|
|
191
|
+
git-ripper https://github.com/microsoft/vscode/tree/main/build --zip="vscode-build.zip"
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
## Resume Downloads
|
|
195
|
+
|
|
196
|
+
Git-ripper now supports resuming interrupted downloads, making it perfect for large folders or unstable network connections.
|
|
197
|
+
|
|
198
|
+
### Automatic Resume (Default Behavior)
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
# Start a download
|
|
202
|
+
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench
|
|
203
|
+
|
|
204
|
+
# If interrupted (Ctrl+C, network issues, etc.), simply run the same command again
|
|
205
|
+
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench
|
|
206
|
+
# It will automatically resume from where it left off
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
### Force Restart
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
# Ignore any existing progress and start fresh
|
|
213
|
+
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench --force-restart
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
### Disable Resume
|
|
217
|
+
|
|
218
|
+
```bash
|
|
219
|
+
# Use traditional behavior without resume functionality
|
|
220
|
+
git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench --no-resume
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
### Manage Checkpoints
|
|
224
|
+
|
|
225
|
+
```bash
|
|
226
|
+
# List all saved download progress
|
|
227
|
+
git-ripper --list-checkpoints
|
|
228
|
+
|
|
229
|
+
# Output shows:
|
|
230
|
+
# 1. ID: a1b2c3d4
|
|
231
|
+
# URL: https://github.com/microsoft/vscode/tree/main/src/vs/workbench
|
|
232
|
+
# Progress: 45/120 files
|
|
233
|
+
# Last Updated: 2025-06-04T10:30:00Z
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
### Resume Features
|
|
237
|
+
|
|
238
|
+
- **Automatic Progress Saving**: Downloads are checkpointed every few files
|
|
239
|
+
- **File Integrity Verification**: Ensures existing files are complete and valid
|
|
240
|
+
- **Smart Recovery**: Detects corrupted or incomplete files and re-downloads them
|
|
241
|
+
- **Multi-Download Support**: Manage multiple concurrent download projects
|
|
242
|
+
- **Progress Indicators**: Visual feedback showing completed vs remaining files
|
|
243
|
+
|
|
244
|
+
## How It Works
|
|
245
|
+
|
|
246
|
+
Git-ripper operates in five stages:
|
|
247
|
+
|
|
248
|
+
1. **URL Parsing**: Extracts repository owner, name, branch, and target folder path
|
|
249
|
+
2. **Resume Check**: Looks for existing download progress and validates already downloaded files
|
|
250
|
+
3. **API Request**: Uses GitHub's API to fetch the folder structure
|
|
251
|
+
4. **Content Download**: Retrieves each file individually while maintaining directory structure and saving progress
|
|
252
|
+
5. **Local Storage or Archiving**: Saves files to your specified output directory or creates an archive
|
|
253
|
+
|
|
254
|
+
The resume functionality uses checkpoint files stored in `.git_ripper_checkpoints/` to track download progress, file integrity hashes, and metadata for each download session.
|
|
255
|
+
|
|
256
|
+
## Configuration
|
|
257
|
+
|
|
258
|
+
Git-ripper works out of the box without configuration. For rate-limited GitHub API usage or private repositories, use the `--gh-token` option as described in the [Authentication](#authentication-private-repositories--rate-limits) section.
|
|
259
|
+
|
|
260
|
+
## Troubleshooting
|
|
261
|
+
|
|
262
|
+
### Common Issues
|
|
263
|
+
|
|
264
|
+
#### Rate Limit Exceeded
|
|
265
|
+
|
|
266
|
+
```
|
|
267
|
+
Error: Request failed with status code 403
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
**Solution**: GitHub limits unauthenticated API requests. You can either:
|
|
271
|
+
|
|
272
|
+
1. Wait a few minutes and try again
|
|
273
|
+
2. Use the `--gh-token` option with a Personal Access Token to significantly increase your rate limit
|
|
274
|
+
|
|
275
|
+
#### Invalid URL Format
|
|
276
|
+
|
|
277
|
+
```
|
|
278
|
+
Error: Invalid GitHub URL format
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
**Solution**: Ensure your URL follows the pattern: `https://github.com/owner/repo/tree/branch/folder`
|
|
282
|
+
|
|
283
|
+
#### Folder Not Found
|
|
284
|
+
|
|
285
|
+
```
|
|
286
|
+
Error: Path not found in repository
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
**Solution**: Verify the folder path exists in the specified branch and repository.
|
|
290
|
+
|
|
291
|
+
#### Resume Issues
|
|
292
|
+
|
|
293
|
+
If you encounter problems with resume functionality:
|
|
294
|
+
|
|
295
|
+
```bash
|
|
296
|
+
# Clear all checkpoints and start fresh
|
|
297
|
+
git-ripper https://github.com/owner/repo/tree/branch/folder --force-restart
|
|
298
|
+
|
|
299
|
+
# Or disable resume entirely
|
|
300
|
+
git-ripper https://github.com/owner/repo/tree/branch/folder --no-resume
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
#### Corrupted Download
|
|
304
|
+
|
|
305
|
+
If files appear corrupted after resume:
|
|
306
|
+
|
|
307
|
+
```bash
|
|
308
|
+
# Force restart will re-download everything
|
|
309
|
+
git-ripper https://github.com/owner/repo/tree/branch/folder --force-restart
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
The resume feature automatically detects and re-downloads corrupted files, but `--force-restart` ensures a completely clean download.
|
|
313
|
+
|
|
314
|
+
## Contributing
|
|
315
|
+
|
|
316
|
+
Contributions make the open-source community an amazing place to learn, inspire, and create. Any contributions to Git-ripper are **greatly appreciated**.
|
|
317
|
+
|
|
318
|
+
1. Fork the repository
|
|
319
|
+
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
|
|
320
|
+
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
|
|
321
|
+
4. Push to the branch (`git push origin feature/amazing-feature`)
|
|
322
|
+
5. Open a Pull Request
|
|
323
|
+
|
|
324
|
+
See the [open issues](https://github.com/sairajB/git-ripper/issues) for a list of proposed features and known issues.
|
|
325
|
+
|
|
326
|
+
## License
|
|
327
|
+
|
|
328
|
+
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
329
|
+
|
|
330
|
+
## Acknowledgments
|
|
331
|
+
|
|
332
|
+
- GitHub API for making this tool possible
|
|
333
|
+
- Everyone who has provided feedback and suggestions
|
|
334
|
+
|
|
335
|
+
---
|
|
336
|
+
|
|
337
|
+
<div align="center">
|
|
338
|
+
<p>Made with ❤️ by <a href="https://github.com/sairajB">sairajB</a></p>
|
|
339
|
+
<p>If you find this tool useful, consider <a href="https://github.com/sponsors/sairajB">sponsoring</a> its development</p>
|
|
340
|
+
</div>
|