git-ripper 1.4.4 → 1.4.6

package/README.md

@@ -32,10 +32,14 @@ Have you ever needed just a single component from a massive repository? Or wante
 ## Features
 
 - **Selective Downloads**: Fetch specific folders instead of entire repositories
+- **Resume Interrupted Downloads**: Automatically resume downloads that were interrupted or failed
+- **Progress Tracking**: Visual progress indicators with file-by-file download status
+- **File Integrity Verification**: Ensures downloaded files are complete and uncorrupted
 - **Directory Structure**: Preserves complete folder structure
 - **Custom Output**: Specify your preferred output directory
 - **Branch Support**: Works with any branch, not just the default one
 - **Archive Export**: Create ZIP archives of downloaded content
+- **Checkpoint Management**: View and manage saved download progress
 - **Simple Interface**: Clean, intuitive command-line experience
 - **Lightweight**: Minimal dependencies and fast execution
 - **No Authentication**: Works with public repositories without requiring credentials
@@ -94,6 +98,9 @@ git-ripper https://github.com/username/repository/tree/branch/folder --zip="my-a
 | -------------------------- | ---------------------------------------- | ----------------- |
 | `-o, --output <directory>` | Specify output directory                 | Current directory |
 | `--zip [filename]`         | Create ZIP archive of downloaded content | -                 |
+| `--no-resume`              | Disable resume functionality             | -                 |
+| `--force-restart`          | Ignore existing checkpoints and restart  | -                 |
+| `--list-checkpoints`       | List all saved download checkpoints      | -                 |
 | `-V, --version`            | Show version number                      | -                 |
 | `-h, --help`               | Show help                                | -                 |
 
@@ -137,14 +144,67 @@ git-ripper https://github.com/facebook/react/tree/main/packages/react-dom --zip
 git-ripper https://github.com/microsoft/vscode/tree/main/build --zip="vscode-build.zip"
 ```
 
+## Resume Downloads
+
+Git-ripper now supports resuming interrupted downloads, making it perfect for large folders or unstable network connections.
+
+### Automatic Resume (Default Behavior)
+
+```bash
+# Start a download
+git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench
+
+# If interrupted (Ctrl+C, network issues, etc.), simply run the same command again
+git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench
+# It will automatically resume from where it left off
+```
+
+### Force Restart
+
+```bash
+# Ignore any existing progress and start fresh
+git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench --force-restart
+```
+
+### Disable Resume
+
+```bash
+# Use traditional behavior without resume functionality
+git-ripper https://github.com/microsoft/vscode/tree/main/src/vs/workbench --no-resume
+```
+
+### Manage Checkpoints
+
+```bash
+# List all saved download progress
+git-ripper --list-checkpoints
+
+# Output shows:
+# 1. ID: a1b2c3d4
+#    URL: https://github.com/microsoft/vscode/tree/main/src/vs/workbench
+#    Progress: 45/120 files
+#    Last Updated: 2025-06-04T10:30:00Z
+```
+
+### Resume Features
+
+- **Automatic Progress Saving**: Downloads are checkpointed every few files
+- **File Integrity Verification**: Ensures existing files are complete and valid
+- **Smart Recovery**: Detects corrupted or incomplete files and re-downloads them
+- **Multi-Download Support**: Manage multiple concurrent download projects
+- **Progress Indicators**: Visual feedback showing completed vs remaining files
+
 ## How It Works
 
-Git-ripper operates in four stages:
+Git-ripper operates in five stages:
 
 1. **URL Parsing**: Extracts repository owner, name, branch, and target folder path
-2. **API Request**: Uses GitHub's API to fetch the folder structure
-3. **Content Download**: Retrieves each file individually while maintaining directory structure
-4. **Local Storage or Archiving**: Saves files to your specified output directory or creates an archive
+2. **Resume Check**: Looks for existing download progress and validates already downloaded files
+3. **API Request**: Uses GitHub's API to fetch the folder structure
+4. **Content Download**: Retrieves each file individually while maintaining directory structure and saving progress
+5. **Local Storage or Archiving**: Saves files to your specified output directory or creates an archive
+
+The resume functionality uses checkpoint files stored in `.git_ripper_checkpoints/` to track download progress, file integrity hashes, and metadata for each download session.
 
 ## Configuration
 
@@ -178,6 +238,29 @@ Error: Path not found in repository
 
 **Solution**: Verify the folder path exists in the specified branch and repository.
 
+#### Resume Issues
+
+If you encounter problems with resume functionality:
+
+```bash
+# Clear all checkpoints and start fresh
+git-ripper https://github.com/owner/repo/tree/branch/folder --force-restart
+
+# Or disable resume entirely
+git-ripper https://github.com/owner/repo/tree/branch/folder --no-resume
+```
+
+#### Corrupted Download
+
+If files appear corrupted after resume:
+
+```bash
+# Force restart will re-download everything
+git-ripper https://github.com/owner/repo/tree/branch/folder --force-restart
+```
+
+The resume feature automatically detects and re-downloads corrupted files, but `--force-restart` ensures a completely clean download.
+
 ## Contributing
 
 Contributions make the open-source community an amazing place to learn, inspire, and create. Any contributions to Git-ripper are **greatly appreciated**.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "git-ripper",
-  "version": "1.4.4",
+  "version": "1.4.6",
   "description": "CLI tool that lets you download specific folders from GitHub repositories without cloning the entire repo.",
   "main": "src/index.js",
   "type": "module",
@@ -29,15 +29,13 @@
   "author": "sairajb",
   "license": "MIT",
   "dependencies": {
-    "ansi-styles": "^6.2.1",
     "archiver": "^6.0.1",
     "axios": "^1.6.7",
     "chalk": "^5.3.0",
     "cli-progress": "^3.12.0",
     "commander": "^12.0.0",
     "p-limit": "^6.2.0",
-    "pretty-bytes": "^6.1.1",
-    "supports-color": "^9.4.0"
+    "pretty-bytes": "^6.1.1"
   },
   "repository": {
     "type": "git",

package/src/archiver.js

@@ -144,10 +144,32 @@ export const downloadAndArchive = async (
   // Create a temporary directory for the download
   const tempDir = path.join(outputDir, `.temp-${Date.now()}`);
   fs.mkdirSync(tempDir, { recursive: true });
-
   try {
     // Download the folder contents
-    await downloadFolder(repoInfo, tempDir);
+    const downloadResult = await downloadFolder(repoInfo, tempDir);
+
+    // Check if download failed
+    if (downloadResult && !downloadResult.success) {
+      throw new Error(
+        `Download failed: ${
+          downloadResult.failedFiles || 0
+        } files could not be downloaded`
+      );
+    }
+
+    // Check if there's anything to archive
+    const files = fs
+      .readdirSync(tempDir, { recursive: true })
+      .filter((file) => {
+        const fullPath = path.join(tempDir, file);
+        return fs.statSync(fullPath).isFile();
+      });
+
+    if (files.length === 0) {
+      throw new Error(
+        `No files to archive - download may have failed or repository is empty`
+      );
+    }
 
     // Determine archive filename
     let archiveFileName = archiveName;

package/src/downloader.js

@@ -45,6 +45,7 @@ const getSpinnerFrame = () => {
  * @param {string} branch - Branch name
  * @param {string} folderPath - Path to the folder
  * @returns {Promise<Array>} - Promise resolving to an array of file objects
+ * @throws {Error} - Throws error on API failures instead of returning empty array
  */
 const fetchFolderContents = async (owner, repo, branch, folderPath) => {
   let effectiveBranch = branch;
@@ -55,12 +56,9 @@ const fetchFolderContents = async (owner, repo, branch, folderPath) => {
       const repoInfoResponse = await axios.get(repoInfoUrl);
       effectiveBranch = repoInfoResponse.data.default_branch;
       if (!effectiveBranch) {
-        console.error(
-          chalk.red(
-            `Could not determine default branch for ${owner}/${repo}. Please specify a branch in the URL.`
-          )
+        throw new Error(
+          `Could not determine default branch for ${owner}/${repo}. Please specify a branch in the URL.`
         );
-        return [];
       }
       console.log(
         chalk.blue(
@@ -68,12 +66,12 @@ const fetchFolderContents = async (owner, repo, branch, folderPath) => {
         )
       );
     } catch (error) {
-      console.error(
-        chalk.red(
-          `Failed to fetch default branch for ${owner}/${repo}: ${error.message}`
-        )
+      if (error.message.includes("Could not determine default branch")) {
+        throw error;
+      }
+      throw new Error(
+        `Failed to fetch default branch for ${owner}/${repo}: ${error.message}`
       );
-      return [];
     }
   }
 
@@ -109,54 +107,44 @@ const fetchFolderContents = async (owner, repo, branch, folderPath) => {
       return response.data.tree.filter((item) => item.path.startsWith(prefix));
     }
   } catch (error) {
+    let errorMessage = "";
+    let isRateLimit = false;
+
     if (error.response) {
       // Handle specific HTTP error codes
       switch (error.response.status) {
         case 403:
           if (error.response.headers["x-ratelimit-remaining"] === "0") {
-            console.error(
-              chalk.red(
-                `GitHub API rate limit exceeded. Please wait until ${new Date(
-                  parseInt(error.response.headers["x-ratelimit-reset"]) * 1000
-                ).toLocaleTimeString()} or add a GitHub token (feature coming soon).`
-              )
-            );
+            isRateLimit = true;
+            errorMessage = `GitHub API rate limit exceeded. Please wait until ${new Date(
+              parseInt(error.response.headers["x-ratelimit-reset"]) * 1000
+            ).toLocaleTimeString()} or add a GitHub token (feature coming soon).`;
           } else {
-            console.error(
-              chalk.red(
-                `Access forbidden: ${
-                  error.response.data.message || "Unknown reason"
-                }`
-              )
-            );
+            errorMessage = `Access forbidden: ${
+              error.response.data.message ||
+              "Repository may be private or you may not have access"
+            }`;
           }
           break;
         case 404:
-          console.error(
-            chalk.red(
-              `Repository, branch, or folder not found: ${owner}/${repo}/${branch}/${folderPath}`
-            )
-          );
+          errorMessage = `Repository, branch, or folder not found: ${owner}/${repo}/${branch}/${folderPath}`;
           break;
         default:
-          console.error(
-            chalk.red(
-              `API error (${error.response.status}): ${
-                error.response.data.message || error.message
-              }`
-            )
-          );
+          errorMessage = `API error (${error.response.status}): ${
+            error.response.data.message || error.message
+          }`;
       }
     } else if (error.request) {
-      console.error(
-        chalk.red(
-          `Network error: No response received from GitHub. Please check your internet connection.`
-        )
-      );
+      errorMessage = `Network error: No response received from GitHub. Please check your internet connection.`;
     } else {
-      console.error(chalk.red(`Error preparing request: ${error.message}`));
+      errorMessage = `Error preparing request: ${error.message}`;
     }
-    return [];
+
+    // Always throw the error instead of returning empty array
+    const enrichedError = new Error(errorMessage);
+    enrichedError.isRateLimit = isRateLimit;
+    enrichedError.statusCode = error.response?.status;
+    throw enrichedError;
   }
 };
 
@@ -349,11 +337,15 @@ const downloadFolder = async (
     const contents = await fetchFolderContents(owner, repo, branch, folderPath);
 
     if (!contents || contents.length === 0) {
-      console.log(
-        chalk.yellow(`No files found in ${folderPath || "repository root"}`)
-      );
-      console.log(chalk.green(`Folder cloned successfully!`));
-      return;
+      const message = `No files found in ${folderPath || "repository root"}`;
+      console.log(chalk.yellow(message));
+      // Don't print success message when no files are found - this might indicate an error
+      return {
+        success: true,
+        filesDownloaded: 0,
+        failedFiles: 0,
+        isEmpty: true,
+      };
     }
 
     // Filter for blob type (files)
@@ -361,15 +353,18 @@ const downloadFolder = async (
     const totalFiles = files.length;
 
     if (totalFiles === 0) {
-      console.log(
-        chalk.yellow(
-          `No files found in ${
-            folderPath || "repository root"
-          } (only directories)`
-        )
-      );
-      console.log(chalk.green(`Folder cloned successfully!`));
-      return;
+      const message = `No files found in ${
+        folderPath || "repository root"
+      } (only directories)`;
+      console.log(chalk.yellow(message));
+      // This is a legitimate case - directory exists but contains only subdirectories
+      console.log(chalk.green(`Directory structure downloaded successfully!`));
+      return {
+        success: true,
+        filesDownloaded: 0,
+        failedFiles: 0,
+        isEmpty: true,
+      };
     }
 
     console.log(
@@ -464,7 +459,6 @@ const downloadFolder = async (
     // Count successful and failed downloads
     const succeeded = results.filter((r) => r.success).length;
     const failed = failedFiles.length;
-
     if (failed > 0) {
       console.log(
         chalk.yellow(
@@ -485,15 +479,45 @@ const downloadFolder = async (
           )
         );
       }
+
+      // Don't claim success if files failed to download
+      if (succeeded === 0) {
+        console.log(
+          chalk.red(`❌ Download failed: No files were downloaded successfully`)
+        );
+        return {
+          success: false,
+          filesDownloaded: succeeded,
+          failedFiles: failed,
+          isEmpty: false,
+        };
+      } else {
+        console.log(chalk.yellow(`⚠️  Download completed with errors`));
+        return {
+          success: false,
+          filesDownloaded: succeeded,
+          failedFiles: failed,
+          isEmpty: false,
+        };
+      }
     } else {
       console.log(
-        chalk.green(` All ${succeeded} files downloaded successfully!`)
+        chalk.green(`✅ All ${succeeded} files downloaded successfully!`)
       );
+      console.log(chalk.green(`Folder cloned successfully!`));
+      return {
+        success: true,
+        filesDownloaded: succeeded,
+        failedFiles: failed,
+        isEmpty: false,
+      };
     }
-
-    console.log(chalk.green(`Folder cloned successfully!`));
   } catch (error) {
-    console.error(chalk.red(`Error downloading folder: ${error.message}`));
+    // Log the specific error details
+    console.error(chalk.red(`❌ Error downloading folder: ${error.message}`));
+
+    // Re-throw the error so the main CLI can exit with proper error code
+    throw error;
   }
 };
 
@@ -576,13 +600,16 @@ const downloadFolderWithResume = async (
 
   try {
     const contents = await fetchFolderContents(owner, repo, branch, folderPath);
-
     if (!contents || contents.length === 0) {
-      console.log(
-        chalk.yellow(`No files found in ${folderPath || "repository root"}`)
-      );
-      console.log(chalk.green(`Folder cloned successfully!`));
-      return;
+      const message = `No files found in ${folderPath || "repository root"}`;
+      console.log(chalk.yellow(message));
+      // Don't print success message when no files are found - this might indicate an error
+      return {
+        success: true,
+        filesDownloaded: 0,
+        failedFiles: 0,
+        isEmpty: true,
+      };
     }
 
     // Filter for blob type (files)
@@ -590,15 +617,18 @@ const downloadFolderWithResume = async (
     const totalFiles = files.length;
 
     if (totalFiles === 0) {
-      console.log(
-        chalk.yellow(
-          `No files found in ${
-            folderPath || "repository root"
-          } (only directories)`
-        )
-      );
-      console.log(chalk.green(`Folder cloned successfully!`));
-      return;
+      const message = `No files found in ${
+        folderPath || "repository root"
+      } (only directories)`;
+      console.log(chalk.yellow(message));
+      // This is a legitimate case - directory exists but contains only subdirectories
+      console.log(chalk.green(`Directory structure downloaded successfully!`));
+      return {
+        success: true,
+        filesDownloaded: 0,
+        failedFiles: 0,
+        isEmpty: true,
+      };
     }
 
     // Create new checkpoint if none exists
@@ -748,7 +778,6 @@ const downloadFolderWithResume = async (
     // Count results
     const succeeded = checkpoint.downloadedFiles.length;
     const failed = failedFiles.length;
-
     if (failed > 0) {
       console.log(
         chalk.yellow(
@@ -766,21 +795,47 @@ const downloadFolderWithResume = async (
       console.log(
         chalk.blue(`💡 Run the same command again to retry failed downloads`)
       );
+
+      // Don't claim success if files failed to download
+      if (succeeded === 0) {
+        console.log(
+          chalk.red(`❌ Download failed: No files were downloaded successfully`)
+        );
+        return {
+          success: false,
+          filesDownloaded: succeeded,
+          failedFiles: failed,
+          isEmpty: false,
+        };
+      } else {
+        console.log(chalk.yellow(`⚠️  Download completed with errors`));
+        return {
+          success: false,
+          filesDownloaded: succeeded,
+          failedFiles: failed,
+          isEmpty: false,
+        };
+      }
     } else {
       console.log(
         chalk.green(`🎉 All ${succeeded} files downloaded successfully!`)
       );
       resumeManager.cleanupCheckpoint(url, outputDir);
+      console.log(chalk.green(`Folder cloned successfully!`));
+      return {
+        success: true,
+        filesDownloaded: succeeded,
+        failedFiles: failed,
+        isEmpty: false,
+      };
     }
-
-    console.log(chalk.green(`Folder cloned successfully!`));
   } catch (error) {
     // Save checkpoint on any error
     if (checkpoint) {
       resumeManager.saveCheckpoint(checkpoint);
     }
 
-    console.error(chalk.red(`Error downloading folder: ${error.message}`));
+    console.error(chalk.red(`❌ Error downloading folder: ${error.message}`));
     throw error;
   }
 };

package/src/index.js

@@ -123,24 +123,46 @@ const initializeCLI = () => {
           forceRestart: options.forceRestart || false,
         };
 
-        if (createArchive) {
-          console.log(`Creating ZIP archive...`);
-          await downloadAndArchive(parsedUrl, options.output, archiveName);
-        } else {
-          console.log(`Downloading folder to: ${options.output}`);
+        let operationType = createArchive ? "archive" : "download";
+        let result = null;
+        let error = null;
 
-          if (downloadOptions.resume) {
-            await downloadFolderWithResume(
-              parsedUrl,
-              options.output,
-              downloadOptions
-            );
+        try {
+          if (createArchive) {
+            console.log(`Creating ZIP archive...`);
+            await downloadAndArchive(parsedUrl, options.output, archiveName);
           } else {
-            await downloadFolder(parsedUrl, options.output);
+            console.log(`Downloading folder to: ${options.output}`);
+            if (downloadOptions.resume) {
+              result = await downloadFolderWithResume(
+                parsedUrl,
+                options.output,
+                downloadOptions
+              );
+            } else {
+              result = await downloadFolder(parsedUrl, options.output);
+            }
           }
+        } catch (opError) {
+          error = opError;
         }
 
-        console.log("Operation completed successfully!");
+        // Consolidated result and error handling
+        if (error) {
+          const failMsg =
+            operationType === "archive"
+              ? `❌ Archive creation failed: ${error.message}`
+              : `❌ Download failed: ${error.message}`;
+          console.error(chalk.red(failMsg));
+          process.exit(1);
+        } else if (!createArchive && result && !result.success) {
+          console.error(chalk.red(`❌ Download failed`));
+          process.exit(1);
+        } else if (!createArchive && result && result.isEmpty) {
+          console.log("Operation completed - no files to download!");
+        } else {
+          console.log("Operation completed successfully!");
+        }
       } catch (error) {
         console.error("Error:", error.message);
         process.exit(1);