replicate-predictions-downloader 1.0.0 → 2.0.1

package/README.md

@@ -1,13 +1,26 @@
-# Replicate Predictions Downloader
+# Replicate Predictions Downloader 
 
 A Node.js script to download and organize all your Replicate predictions, including images, metadata, and other outputs before they expire.
 
-![Replicate Downloader Banner](https://replicate.com/static/favicon.png)
+![Replicate Downloader Banner](replicate-prediction-banner.png)
+
+<div align="center">
+
+[![npm version](https://badge.fury.io/js/replicate-predictions-downloader.svg)](https://badge.fury.io/js/replicate-predictions-downloader)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Node.js](https://img.shields.io/badge/Node.js-18%2B-green.svg)](https://nodejs.org/)
+[![GitHub stars](https://img.shields.io/github/stars/closestfriend/replicate-predictions-downloader.svg)](https://github.com/closestfriend/replicate-predictions-downloader/stargazers)
+
+*A Node.js utility to preserve your Replicate AI predictions before they expire*
+
+</div>
 
 ## Features
 
 - 📥 Downloads all predictions from your Replicate account
 - 🗂️ Organizes files by model and date
+- 📅 **NEW: Date filtering** to avoid duplicate downloads
+- 🔄 **NEW: Incremental downloads** since last successful run
 - 🗜️ Creates ZIP archives for each model (optional)
 - 📊 Saves enhanced metadata for all predictions
 - 📈 Shows detailed download statistics
@@ -37,7 +50,7 @@ cd replicate-predictions-downloader
 npm install
 ```
 
-#### Option 2: Install via npm (coming soon)
+#### Option 2: Install via npm
 
 ```bash
 npm install -g replicate-predictions-downloader
@@ -61,6 +74,8 @@ Set your API token using one of these methods:
 
 ## Usage
 
+### Basic Usage
+
 Run the script:
 ```bash
 # If installed locally
@@ -69,19 +84,50 @@ npm start
 # Or directly
 node index.js
 
-# If installed globally (coming soon)
+# If installed globally
 replicate-downloader
 ```
 
-The script will:
-1. Fetch all your predictions from Replicate
-2. Download all outputs
-3. Organize them by model
-4. Create ZIP archives (if enabled)
-5. Save detailed metadata
+### Date Filtering Options
+
+Avoid duplicate downloads with intelligent date filtering:
+
+```bash
+# Download predictions since a specific date
+node index.js --since "2024-01-15"
+
+# Download predictions in a date range
+node index.js --since "2024-01-01" --until "2024-01-31"
+
+# Download only new predictions since last run (recommended for regular use)
+node index.js --last-run
+
+# Download predictions from the last 7 days
+node index.js --since "7 days ago"
+
+# Download all predictions (default behavior)
+node index.js --all
+```
+
+### Date Format Examples
+
+The tool accepts various date formats:
+- ISO dates: `"2024-01-15"`, `"2024-01-15T10:30:00Z"`
+- Relative dates: `"2 days ago"`, `"1 week ago"`, `"yesterday"`
+- Natural language: `"January 15, 2024"`
+
+### What the Script Does
+
+1. **Fetches predictions** from Replicate (with optional date filtering)
+2. **Downloads all outputs** for successful predictions
+3. **Organizes files** by model and date
+4. **Creates ZIP archives** for each model (if enabled)
+5. **Saves detailed metadata** and tracks state for future runs
 
 ## Configuration
 
+### Basic Settings
+
 You can adjust these settings in the CONFIG object:
 - `requestDelay`: Delay between API requests (ms)
 - `downloadDelay`: Delay between downloads (ms)
@@ -89,6 +135,15 @@ You can adjust these settings in the CONFIG object:
 - `createZips`: Whether to create ZIP archives
 - `enhancedMetadata`: Whether to save enhanced metadata
 
+### State Tracking
+
+The tool automatically creates a `.replicate-downloader-state.json` file to track:
+- Last successful run timestamp
+- Total predictions processed
+- Successful predictions count
+
+This enables the `--last-run` option for incremental downloads without duplicates.
+
 ## Output Structure
 
 The tool creates a directory structure like this:
@@ -113,6 +168,27 @@ replicate_metadata_YYYY-MM-DD.json
 
 This is a tool I created for personal use. I'm sharing it in case others find it helpful, but I may not be able to provide extensive support. Pull requests are welcome!
 
+## Command-Line Interface
+
+The tool now includes a comprehensive CLI with help and version information:
+
+```bash
+# Show help
+node index.js --help
+
+# Show version
+node index.js --version
+
+# Available options
+Options:
+  -s, --since <date>     Download predictions created since this date
+  -u, --until <date>     Download predictions created until this date
+  -l, --last-run         Download only predictions since last successful run
+  --all                  Download all predictions (default behavior)
+  -h, --help             Display help for command
+  -V, --version          Display version for command
+```
+
 ## Contributing
 
 Contributions are welcome! Please feel free to submit a Pull Request.
@@ -128,4 +204,98 @@ This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md
 
 ## Background
 
-This tool was created while working on LLM behavior/personality research to preserve valuable predictions before they expired.
+Replicate is a powerful platform for running AI models, but predictions don't last forever - they expire after a period of time. This tool was originally created while working on LLM behavior/personality research when I realized I was losing valuable generated content.
+
+After generating hundreds of images with Stable Diffusion, video clips, and other AI outputs, I needed a way to systematically download and organize everything before it disappeared. The tool has since evolved into a comprehensive solution for anyone who wants to preserve their Replicate predictions with intelligent organization and duplicate prevention.
+
+## Quick Start
+
+Get up and running in 2 minutes:
+
+1. **Get your API token**: Visit [replicate.com/account/api-tokens](https://replicate.com/account/api-tokens)
+
+2. **Set your token**:
+   ```bash
+   export REPLICATE_API_TOKEN=your_token_here
+   ```
+
+3. **Install and run**:
+   ```bash
+   # Clone and install
+   git clone https://github.com/closestfriend/replicate-predictions-downloader.git
+   cd replicate-predictions-downloader
+   npm install
+   
+   # Download all your predictions
+   npm start
+   
+   # Or just new ones since last run (recommended for regular use)
+   node index.js --last-run
+   ```
+
+That's it! Your predictions will be downloaded and organized by model and date.
+
+## What You'll See
+
+When you run the downloader, here's what happens:
+
+```
+Starting Replicate Predictions Downloader v2.0.0
+Using date filter: since 2024-01-01
+Fetching predictions from Replicate API...
+Found 47 predictions to process
+
+PROCESSING STATISTICS:
+├── Total predictions: 47
+├── Successful: 43 (91.5%)
+├── Failed: 4 (8.5%)
+└── Models found: 8
+
+DOWNLOADING OUTPUTS:
+├── stable-diffusion-xl: 25 images
+├── midjourney-v6: 12 images  
+├── whisper-large-v3: 8 audio files
+└── llama-2-70b-chat: 6 text files
+
+SAVING TO: replicate_outputs_2024-01-15/
+├── by-model/
+│   ├── stable-diffusion-xl/
+│   ├── midjourney-v6/
+│   └── ...
+├── stable-diffusion-xl.zip
+├── midjourney-v6.zip
+└── replicate_metadata_2024-01-15.json
+
+Complete! Downloaded 51 files (127.3 MB)
+Elapsed time: 2m 34s
+Next time, use --last-run to download only new predictions
+```
+
+## Common Issues
+
+**"Invalid API token" or "Authentication failed"**
+- Double-check your token at [replicate.com/account/api-tokens](https://replicate.com/account/api-tokens)
+- Make sure you've set `REPLICATE_API_TOKEN` in your environment or `.env` file
+- Try: `echo $REPLICATE_API_TOKEN` to verify it's set
+
+**"Network timeout" or connection errors**
+- Check your internet connection
+- The tool automatically retries failed downloads
+- For persistent issues, try running with fewer concurrent requests by editing the CONFIG object
+
+**"Permission denied" or file errors**
+- Ensure you have write permissions in the current directory
+- On Windows, try running as administrator if needed
+- Make sure you have enough disk space for your downloads
+
+**"No new predictions found"**
+- If using `--last-run`, this means you're up to date!
+- Try `--since "1 week ago"` to see recent predictions
+- Use `--all` to redownload everything (may create duplicates)
+
+**Files have weird names or special characters**
+- The tool automatically sanitizes filenames
+- Prompts longer than the limit (default: 50 chars) are truncated
+- Adjust `maxPromptLength` in CONFIG if needed
+
+**Need help?** Check existing [GitHub issues](https://github.com/closestfriend/replicate-predictions-downloader/issues) or create a new one.

package/example.js

@@ -1,4 +1,4 @@
-// Example of using ReplicateDownloader as a module
+// Example of using ReplicateDownloader as a module with date filtering
 import ReplicateDownloader from './index.js';
 import dotenv from 'dotenv';
 
@@ -15,25 +15,45 @@ async function example() {
   
   console.log('Starting Replicate Downloader example...');
   
-  // Create a new instance with your API token
-  const downloader = new ReplicateDownloader(process.env.REPLICATE_API_TOKEN);
+  // Example 1: Download all predictions (default behavior)
+  console.log('\n📥 Example 1: Download all predictions');
+  const downloader1 = new ReplicateDownloader(process.env.REPLICATE_API_TOKEN);
+  // await downloader1.run(); // Uncomment to run
   
-  // Optional: Override default configuration
-  // downloader.config = {
-  //   requestDelay: 500,     // Increase delay between API requests
-  //   downloadDelay: 200,    // Increase delay between downloads
-  //   maxPromptLength: 30,   // Shorter prompt in filenames
-  //   createZips: false,     // Disable ZIP creation
-  //   enhancedMetadata: true // Keep enhanced metadata
-  // };
+  // Example 2: Download predictions since a specific date
+  console.log('\n📅 Example 2: Download predictions since 2024-01-01');
+  const downloader2 = new ReplicateDownloader(process.env.REPLICATE_API_TOKEN, {
+    since: '2024-01-01'
+  });
+  // await downloader2.run(); // Uncomment to run
   
-  try {
-    // Run the downloader
-    await downloader.run();
-    console.log('Example completed successfully!');
-  } catch (error) {
-    console.error('Example failed:', error.message);
-  }
+  // Example 3: Download predictions in a date range
+  console.log('\n📅 Example 3: Download predictions between 2024-01-01 and 2024-01-31');
+  const downloader3 = new ReplicateDownloader(process.env.REPLICATE_API_TOKEN, {
+    since: '2024-01-01',
+    until: '2024-01-31'
+  });
+  // await downloader3.run(); // Uncomment to run
+  
+  // Example 4: Incremental download (since last run)
+  console.log('\n🔄 Example 4: Incremental download since last successful run');
+  const downloader4 = new ReplicateDownloader(process.env.REPLICATE_API_TOKEN, {
+    lastRun: true
+  });
+  // await downloader4.run(); // Uncomment to run
+  
+  // Example 5: Download predictions from the last 7 days
+  console.log('\n📅 Example 5: Download predictions from the last 7 days');
+  const sevenDaysAgo = new Date();
+  sevenDaysAgo.setDate(sevenDaysAgo.getDate() - 7);
+  const downloader5 = new ReplicateDownloader(process.env.REPLICATE_API_TOKEN, {
+    since: sevenDaysAgo.toISOString()
+  });
+  // await downloader5.run(); // Uncomment to run
+  
+  console.log('\n💡 To run any example, uncomment the corresponding await line');
+  console.log('💡 The tool automatically saves state for incremental downloads');
+  console.log('💡 Use --help for command-line options when running directly');
 }
 
 // Run the example

package/index.js

@@ -2,6 +2,11 @@ import fs from 'fs';
 import path from 'path';
 import https from 'https';
 import archiver from 'archiver';
+import { Command } from 'commander';
+import dotenv from 'dotenv';
+
+// Load environment variables
+dotenv.config();
 
 // Get API token from environment variable
 const REPLICATE_API_TOKEN = process.env.REPLICATE_API_TOKEN;
@@ -16,7 +21,7 @@ const CONFIG = {
 };
 
 class ReplicateDownloader {
-    constructor(apiToken) {
+    constructor(apiToken, options = {}) {
         this.apiToken = apiToken;
         this.baseUrl = 'https://api.replicate.com/v1';
         this.allPredictions = [];
@@ -25,6 +30,82 @@ class ReplicateDownloader {
             totalFiles: 0,
             totalSize: 0
         };
+        
+        // Date filtering options
+        this.dateFilters = {
+            since: options.since || null,
+            until: options.until || null,
+            lastRun: options.lastRun || false
+        };
+        
+        this.stateFile = '.replicate-downloader-state.json';
+    }
+
+    // Load state from previous runs
+    loadState() {
+        try {
+            if (fs.existsSync(this.stateFile)) {
+                const state = JSON.parse(fs.readFileSync(this.stateFile, 'utf8'));
+                return state;
+            }
+        } catch (error) {
+            console.warn('Warning: Could not load state file:', error.message);
+        }
+        return { lastSuccessfulRun: null };
+    }
+
+    // Save state for future runs
+    saveState(state) {
+        try {
+            fs.writeFileSync(this.stateFile, JSON.stringify(state, null, 2));
+        } catch (error) {
+            console.warn('Warning: Could not save state file:', error.message);
+        }
+    }
+
+    // Parse date string to ISO format
+    parseDate(dateStr) {
+        if (!dateStr) return null;
+        
+        // Handle various date formats
+        const date = new Date(dateStr);
+        if (isNaN(date.getTime())) {
+            throw new Error(`Invalid date format: ${dateStr}. Use formats like "2024-01-15", "2024-01-15T10:30:00Z", or "2 days ago"`);
+        }
+        
+        return date.toISOString();
+    }
+
+    // Get date filter parameters for API
+    getDateFilterParams() {
+        const params = new URLSearchParams();
+        
+        let sinceDate = this.dateFilters.since;
+        
+        // Handle --last-run option
+        if (this.dateFilters.lastRun) {
+            const state = this.loadState();
+            if (state.lastSuccessfulRun) {
+                sinceDate = state.lastSuccessfulRun;
+                console.log(`Using last run date: ${sinceDate}`);
+            } else {
+                console.log('No previous run found, downloading all predictions');
+            }
+        }
+        
+        if (sinceDate) {
+            const parsedSince = this.parseDate(sinceDate);
+            // Add a small buffer to avoid edge cases
+            const sinceBuffer = new Date(new Date(parsedSince).getTime() - 1000);
+            params.append('created_at.gte', sinceBuffer.toISOString());
+        }
+        
+        if (this.dateFilters.until) {
+            const parsedUntil = this.parseDate(this.dateFilters.until);
+            params.append('created_at.lte', parsedUntil);
+        }
+        
+        return params.toString();
     }
 
     // Make authenticated request to Replicate API
@@ -133,11 +214,18 @@ class ReplicateDownloader {
         });
     }
 
-    // Fetch all predictions with pagination
+    // Fetch all predictions with pagination and date filtering
     async fetchAllPredictions() {
         let nextUrl = `${this.baseUrl}/predictions`;
         let page = 1;
 
+        // Apply date filtering
+        const dateParams = this.getDateFilterParams();
+        if (dateParams) {
+            nextUrl += `?${dateParams}`;
+            console.log('Applying date filters:', dateParams);
+        }
+
         console.log('Starting to fetch predictions...');
 
         while (nextUrl) {
@@ -152,7 +240,19 @@ class ReplicateDownloader {
                     console.log(`Total so far: ${this.allPredictions.length}`);
                 }
 
-                nextUrl = response.next;
+                // Handle pagination with date filters
+                if (response.next) {
+                    // Ensure date filters are preserved in next URL
+                    if (dateParams && !response.next.includes('created_at')) {
+                        const separator = response.next.includes('?') ? '&' : '?';
+                        nextUrl = `${response.next}${separator}${dateParams}`;
+                    } else {
+                        nextUrl = response.next;
+                    }
+                } else {
+                    nextUrl = null;
+                }
+                
                 page++;
 
                 // Custom delay
@@ -399,12 +499,38 @@ class ReplicateDownloader {
             console.log(`  ❌ Failed: ${failed}`);
             console.log(`  ⏹️  Canceled: ${canceled}`);
 
+            // Save state for future runs
+            if (successful > 0) {
+                const state = {
+                    lastSuccessfulRun: new Date().toISOString(),
+                    totalPredictions: this.allPredictions.length,
+                    successfulPredictions: successful
+                };
+                this.saveState(state);
+                console.log('\n💾 State saved for future incremental downloads');
+            }
+
         } catch (error) {
             console.error('💥 Fatal error:', error.message);
         }
     }
 }
 
+// Setup command-line interface
+const program = new Command();
+
+program
+    .name('replicate-downloader')
+    .description('Download and organize Replicate predictions with date filtering')
+    .version('2.0.0')
+    .option('-s, --since <date>', 'Download predictions created since this date (ISO format, e.g., "2024-01-15" or "2 days ago")')
+    .option('-u, --until <date>', 'Download predictions created until this date (ISO format)')
+    .option('-l, --last-run', 'Download only predictions created since the last successful run')
+    .option('--all', 'Download all predictions (default behavior)')
+    .parse();
+
+const options = program.opts();
+
 // Usage
 async function main() {
     if (!REPLICATE_API_TOKEN) {
@@ -418,7 +544,35 @@ async function main() {
         return;
     }
 
-    const downloader = new ReplicateDownloader(REPLICATE_API_TOKEN);
+    // Validate date options
+    if (options.since && options.lastRun) {
+        console.error('❌ Cannot use both --since and --last-run options together');
+        return;
+    }
+
+    if (options.until && options.lastRun) {
+        console.error('❌ Cannot use both --until and --last-run options together');
+        return;
+    }
+
+    const downloaderOptions = {};
+    
+    if (options.since) {
+        downloaderOptions.since = options.since;
+        console.log(`📅 Filtering predictions since: ${options.since}`);
+    }
+    
+    if (options.until) {
+        downloaderOptions.until = options.until;
+        console.log(`📅 Filtering predictions until: ${options.until}`);
+    }
+    
+    if (options.lastRun) {
+        downloaderOptions.lastRun = true;
+        console.log('🔄 Using incremental download (since last run)');
+    }
+
+    const downloader = new ReplicateDownloader(REPLICATE_API_TOKEN, downloaderOptions);
     await downloader.run();
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "replicate-predictions-downloader",
-  "version": "1.0.0",
+  "version": "2.0.1",
   "description": "A Node.js script to download and organize all your Replicate predictions, including images, metadata, and other outputs",
   "main": "index.js",
   "type": "module",
@@ -17,7 +17,14 @@
     "predictions",
     "organization",
     "replicate-api",
-    "model-outputs"
+    "model-outputs",
+    "stable-diffusion",
+    "midjourney-alternative",
+    "ai-art",
+    "batch-download",
+    "backup",
+    "archive",
+    "cli"
   ],
   "author": "closestfriend",
   "license": "MIT",
@@ -34,7 +41,8 @@
   },
   "dependencies": {
     "archiver": "^5.3.1",
-    "dotenv": "^16.3.1"
+    "dotenv": "^16.3.1",
+    "commander": "^11.0.0"
   },
   "files": [
     "index.js",