parallel_sftp 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '0899099ab70f5bc7db699b0efb103fc2670fa1519b7aff996a095abeef41caed'
+  data.tar.gz: e83babc2dd7d491cb3a32f7699a23b9366f243973ca6d6c4ae6b4b4b1075a627
+SHA512:
+  metadata.gz: 33c36ce67292ba5fc79873d4db509bbd665e1a21968cad089ea530e0cf053f51f3cf8f17b5521c25a3696ebf9df392067c558176fcefde279667049796d35e35
+  data.tar.gz: 5a947dffa20be7cb1969166c4231b0588db67f21e1d070a3e3a8fdd5eb3beea628a95717f36e8107e1b60780311b6d309114692dd1bf5742be8f4b35538d626a

data/.envrc

@@ -0,0 +1 @@
+use nix

data/.gitignore

@@ -0,0 +1,18 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/.gems/
+
+# rspec failure tracking
+.rspec_status
+
+# Gem build artifacts
+*.gem
+
+# Bundler lockfile (gems should test against fresh dependency resolution)
+Gemfile.lock

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/CHANGELOG.md

@@ -0,0 +1,45 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] - 2026-02-05
+
+### Added
+- `ZipIntegrityError` class for zip file corruption detection
+- Automatic zip integrity verification using `unzip -t` after download
+- Auto-retry on zip corruption with parallel-first strategy
+- New `retry_on_corruption` option (default: true) to enable/disable auto-retry
+- New `parallel_retries` option (default: 2) - retries with same segment count before reducing
+
+### Changed
+- Downloads now verify zip integrity before returning success
+- Corrupted downloads are automatically cleaned up before retry
+
+## [0.2.0] - 2026-02-04
+
+### Added
+- Per-segment progress tracking via `on_segment_progress` callback
+- `SegmentProgressParser` class for parsing `.lftp-pget-status` files
+- `TimeEstimator` class for calculating download speed and ETA with moving window
+- Calculated time estimates independent of lftp's reported ETA
+- Elapsed time tracking since download start
+- Average speed calculation from download start
+
+## [0.1.0] - 2026-02-04
+
+### Added
+- Initial release
+- `ParallelSftp.download` one-liner for simple downloads
+- `ParallelSftp::Client` for multiple downloads with shared connection settings
+- Parallel/segmented downloads via lftp's `pget` command
+- Resume support for interrupted downloads
+- Progress callbacks with percent, speed, ETA
+- Global configuration with `ParallelSftp.configure`
+- `optimize_for_large_files!` preset for 20GB+ files
+- Error classes: `LftpNotFoundError`, `ConnectionError`, `DownloadError`, `IntegrityError`
+- lftp availability check with `lftp_available?` and `lftp_version`

data/CLAUDE.md

@@ -0,0 +1,178 @@
+# parallel_sftp Gem Guidelines
+
+> **Purpose**: Fast parallel SFTP downloads using lftp's segmented transfer (`pget` command)
+
+## Project Overview
+
+- **Test Framework**: RSpec
+- **Ruby Version**: >= 2.5.0
+- **External Dependency**: lftp (must be installed on system)
+- **License**: MIT
+
+## Quick Commands
+
+```bash
+# Run all tests
+bundle exec rspec
+
+# Run specific test file
+bundle exec rspec spec/parallel_sftp/segment_progress_parser_spec.rb
+
+# Run with documentation format
+bundle exec rspec --format documentation
+
+# Integration tests (require SFTP server)
+SFTP_TEST_HOST=host SFTP_TEST_USER=user SFTP_TEST_PASSWORD=pass bundle exec rspec spec/integration
+```
+
+## Architecture
+
+```
+lib/parallel_sftp/
+├── client.rb                  # High-level API, creates LftpCommand and Download
+├── configuration.rb           # Global settings (segments, timeout, retries)
+├── download.rb                # Executes lftp, handles progress callbacks
+├── errors.rb                  # Custom error classes
+├── lftp_command.rb            # Builds lftp script with pget command
+├── progress_parser.rb         # Parses lftp stdout for progress info
+├── segment_progress_parser.rb # Parses .lftp-pget-status file for per-segment progress
+├── time_estimator.rb          # Calculates speed/ETA with moving window
+└── version.rb                 # Gem version
+```
+
+## Key Patterns
+
+### Adding New Options
+
+When adding a new option that flows through the API:
+
+1. Add to `Configuration` class (with default)
+2. Add to `LftpCommand#initialize` parameters
+3. Add to `Client#download` options handling
+4. Add to `ParallelSftp.download` module method
+5. Update specs for all affected classes
+
+### Progress Callbacks
+
+Two callback types exist:
+
+- **`on_progress`**: Parsed from lftp stdout (percent, speed, eta)
+- **`on_segment_progress`**: Polled from `.lftp-pget-status` file (per-segment detail)
+
+### Thread Safety
+
+The `Download` class spawns a background thread for segment progress polling. Key patterns:
+
+```ruby
+# Start polling
+@stop_polling = false
+@polling_thread = Thread.new { poll_segment_progress(status_file) }
+
+# Stop polling (with timeout)
+@stop_polling = true
+@polling_thread.join(2)
+@polling_thread.kill if @polling_thread.alive?
+```
+
+### lftp Status File Format
+
+lftp creates `{filename}.lftp-pget-status` during pget downloads:
+
+```
+size=20955686931
+0.pos=57442304
+0.limit=2619460869
+1.pos=2670611717
+1.limit=5238921735
+```
+
+- `size`: Total file size (-2 indicates unknown/error)
+- `N.pos`: Current byte position for segment N
+- `N.limit`: End byte position for segment N
+- Segment start = previous segment's limit (or 0 for segment 0)
+
+### Password Escaping
+
+Special characters in passwords must be URL-encoded for lftp:
+
+```ruby
+# In LftpCommand
+def escaped_password
+  CGI.escape(password)
+end
+```
+
+### Legacy SSH Server Support
+
+Some servers only offer `ssh-rsa` host keys. Configure via:
+
+```ruby
+ParallelSftp.configure do |config|
+  config.sftp_connect_program = 'ssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa'
+end
+
+# Or pass explicitly (recommended for rake tasks)
+ParallelSftp.download(
+  sftp_connect_program: 'ssh -o HostKeyAlgorithms=+ssh-rsa',
+  ...
+)
+```
+
+## Testing Patterns
+
+### Mocking lftp Availability
+
+```ruby
+before do
+  allow(ParallelSftp).to receive(:lftp_available?).and_return(true)
+end
+```
+
+### Mocking Download Execution
+
+```ruby
+download_mock = instance_double(ParallelSftp::Download, execute: local_path)
+allow(ParallelSftp::Download).to receive(:new).and_return(download_mock)
+```
+
+### Testing Progress Callbacks
+
+```ruby
+it "passes progress callback to Download" do
+  progress_callback = ->(info) { puts info }
+
+  expect(ParallelSftp::Download).to receive(:new)
+    .with(anything, on_progress: progress_callback, on_segment_progress: nil)
+    .and_return(download_mock)
+
+  client.download(remote_path, local_path, on_progress: progress_callback)
+end
+```
+
+### Using Tempfile for Status File Tests
+
+```ruby
+Tempfile.create("status") do |f|
+  f.write("size=1000\n0.pos=500\n0.limit=1000\n")
+  f.rewind
+  expect(parser.parse(f.path)).to be true
+end
+```
+
+## Error Handling
+
+Custom errors inherit from `ParallelSftp::Error`:
+
+- `LftpNotFoundError` - lftp not installed
+- `ConnectionError` - SFTP connection failed (includes host, exit_status)
+- `DownloadError` - Transfer failed (includes remote_path, exit_status, output)
+- `IntegrityError` - File size mismatch (includes expected_size, actual_size)
+
+## Code Style
+
+- Frozen string literals in all files
+- RSpec `described_class` for subject
+- Keyword arguments for complex methods
+- Guard clauses for early returns
+- Struct for simple data objects (e.g., `Segment`, `Sample`)
+

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in parallel_sftp.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nestor G Pestelos Jr
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,186 @@
+# ParallelSftp
+
+Fast parallel SFTP downloads using lftp's segmented transfer. This gem wraps lftp to enable multi-connection parallel downloads for large files.
+
+## Features
+
+- Parallel/segmented downloads using multiple connections
+- Resume interrupted downloads
+- Progress callbacks for monitoring
+- Configurable retry and timeout settings
+- Optimized presets for large files (20GB+)
+
+## Requirements
+
+- **lftp** must be installed on the system
+
+### Installing lftp
+
+**macOS:**
+```bash
+brew install lftp
+```
+
+**Ubuntu/Debian:**
+```bash
+apt install lftp
+```
+
+**Heroku:**
+```bash
+heroku buildpacks:add --index 1 https://github.com/heroku/heroku-buildpack-apt
+echo "lftp" > Aptfile
+git add Aptfile && git commit -m "Add lftp via apt buildpack"
+git push heroku master
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'parallel_sftp'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+## Usage
+
+### Simple one-liner
+
+```ruby
+require 'parallel_sftp'
+
+ParallelSftp.download(
+  host: 'sftp.example.com',
+  user: 'username',
+  password: 'secret',
+  remote_path: '/path/to/large_file.zip',
+  local_path: '/tmp/large_file.zip'
+)
+```
+
+### Using the Client
+
+```ruby
+client = ParallelSftp::Client.new(
+  host: 'sftp.example.com',
+  user: 'username',
+  password: 'secret',
+  port: 22
+)
+
+# Basic download
+client.download('/remote/file.zip', '/local/file.zip')
+
+# With options
+client.download('/remote/file.zip', '/local/file.zip',
+  segments: 8,           # parallel connections (default: 4)
+  resume: true,          # continue interrupted downloads (default: true)
+  timeout: 60,           # connection timeout seconds (default: 30)
+  max_retries: 15,       # retry attempts (default: 10)
+  on_progress: ->(info) { puts "#{info[:percent]}%" }
+)
+```
+
+### Progress Callback
+
+The progress callback receives a hash with:
+
+```ruby
+{
+  percent: 45,                    # percentage complete
+  bytes_transferred: 1073741824,  # bytes downloaded
+  total_bytes: 21474836480,       # total file size
+  speed: 10485760,                # bytes per second
+  eta: "30m"                      # estimated time remaining
+}
+```
+
+### Global Configuration
+
+```ruby
+ParallelSftp.configure do |config|
+  config.default_segments = 8      # parallel connections
+  config.timeout = 60              # connection timeout
+  config.max_retries = 15          # retry attempts
+  config.reconnect_interval = 10   # seconds between retries
+  config.default_port = 22         # SFTP port
+end
+```
+
+### Large File Optimization
+
+For files 20GB+, use the optimized settings:
+
+```ruby
+ParallelSftp.configuration.optimize_for_large_files!
+# Sets: segments=8, timeout=60, max_retries=15, reconnect_interval=10
+```
+
+## Tuned Settings
+
+| Setting | Default | Large File (20GB+) | Purpose |
+|---------|---------|-------------------|---------|
+| `segments` | 4 | 8 | More parallel connections |
+| `timeout` | 30 | 60 | Longer timeout for slow starts |
+| `max_retries` | 10 | 15 | More retries for flaky connections |
+| `reconnect_interval` | 5 | 10 | Wait longer between retries |
+
+## Error Handling
+
+```ruby
+begin
+  ParallelSftp.download(...)
+rescue ParallelSftp::LftpNotFoundError
+  # lftp is not installed
+rescue ParallelSftp::ConnectionError => e
+  # SFTP connection failed
+  puts e.host
+  puts e.exit_status
+rescue ParallelSftp::DownloadError => e
+  # Transfer failed
+  puts e.remote_path
+  puts e.exit_status
+  puts e.output
+rescue ParallelSftp::IntegrityError => e
+  # File size mismatch
+  puts e.expected_size
+  puts e.actual_size
+end
+```
+
+## Checking lftp Availability
+
+```ruby
+if ParallelSftp.lftp_available?
+  puts "lftp version: #{ParallelSftp.lftp_version}"
+else
+  puts "lftp is not installed"
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt.
+
+### Running Integration Tests
+
+Integration tests require a running SFTP server. Set these environment variables:
+
+```bash
+export SFTP_TEST_HOST=your-sftp-server.com
+export SFTP_TEST_USER=username
+export SFTP_TEST_PASSWORD=password
+export SFTP_TEST_FILE=/path/to/test/file.txt
+
+rspec spec/integration
+```
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "parallel_sftp"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/parallel_sftp/client.rb

@@ -0,0 +1,108 @@
+# frozen_string_literal: true
+
+require "fileutils"
+
+module ParallelSftp
+  # SFTP client for parallel downloads
+  class Client
+    attr_reader :host, :user, :password, :port
+
+    # Default number of times to retry with same segment count before reducing
+    DEFAULT_PARALLEL_RETRIES = 2
+
+    def initialize(options = {})
+      @host = options.fetch(:host)
+      @user = options.fetch(:user)
+      @password = options.fetch(:password)
+      @port = options.fetch(:port, ParallelSftp.configuration.default_port)
+    end
+
+    # Download a file from the remote server
+    #
+    # @param remote_path [String] Path to the file on the remote server
+    # @param local_path [String] Local path to save the file
+    # @param options [Hash] Download options
+    # @option options [Integer] :segments Number of parallel connections (default: 4)
+    # @option options [Boolean] :resume Continue interrupted downloads (default: true)
+    # @option options [Integer] :timeout Connection timeout in seconds (default: 30)
+    # @option options [Integer] :max_retries Maximum retry attempts (default: 10)
+    # @option options [Integer] :reconnect_interval Seconds between retries (default: 5)
+    # @option options [Proc] :on_progress Progress callback receiving hash with :percent, :speed, etc.
+    # @option options [Proc] :on_segment_progress Per-segment progress callback receiving hash with
+    #   :total_size, :segments, :total_downloaded, :overall_percent, :speed, :eta, :elapsed
+    # @option options [Boolean] :retry_on_corruption Auto-retry on zip corruption (default: true)
+    # @option options [Integer] :parallel_retries Times to retry with same segments before reducing (default: 2)
+    #
+    # @return [String] Local path to the downloaded file
+    # @raise [DownloadError] if download fails
+    # @raise [ZipIntegrityError] if zip corruption persists after all retries
+    def download(remote_path, local_path, options = {})
+      segments = options.fetch(:segments, ParallelSftp.configuration.default_segments)
+      retry_on_corruption = options.fetch(:retry_on_corruption, true)
+      parallel_retries = options.fetch(:parallel_retries, DEFAULT_PARALLEL_RETRIES)
+
+      return execute_download(remote_path, local_path, options.merge(segments: segments)) unless retry_on_corruption
+
+      download_with_retry(remote_path, local_path, options, segments, parallel_retries)
+    end
+
+    private
+
+    def download_with_retry(remote_path, local_path, options, segments, parallel_retries)
+      current_segments = segments
+      retries_at_current_segments = 0
+
+      loop do
+        begin
+          return execute_download(remote_path, local_path, options.merge(segments: current_segments, resume: false))
+        rescue ZipIntegrityError => e
+          cleanup_corrupted_download(local_path)
+
+          retries_at_current_segments += 1
+
+          if retries_at_current_segments < parallel_retries
+            warn "[parallel_sftp] Zip corruption detected, retrying with #{current_segments} segments " \
+                 "(attempt #{retries_at_current_segments + 1}/#{parallel_retries})..."
+          elsif current_segments > 1
+            # Reduce segments and reset retry counter
+            current_segments = (current_segments / 2).clamp(1, current_segments - 1)
+            retries_at_current_segments = 0
+            warn "[parallel_sftp] Zip corruption persists, reducing to #{current_segments} segments..."
+          else
+            # segments = 1 and still failing, give up
+            raise e
+          end
+        end
+      end
+    end
+
+    def cleanup_corrupted_download(local_path)
+      FileUtils.rm_f(local_path)
+      FileUtils.rm_f("#{local_path}.lftp-pget-status")
+    end
+
+    def execute_download(remote_path, local_path, options)
+      lftp_command = LftpCommand.new(
+        host: host,
+        user: user,
+        password: password,
+        port: port,
+        remote_path: remote_path,
+        local_path: local_path,
+        segments: options.fetch(:segments, ParallelSftp.configuration.default_segments),
+        resume: options.fetch(:resume, true),
+        timeout: options.fetch(:timeout, ParallelSftp.configuration.timeout),
+        max_retries: options.fetch(:max_retries, ParallelSftp.configuration.max_retries),
+        reconnect_interval: options.fetch(:reconnect_interval, ParallelSftp.configuration.reconnect_interval),
+        sftp_connect_program: options[:sftp_connect_program]
+      )
+
+      download = Download.new(
+        lftp_command,
+        on_progress: options[:on_progress],
+        on_segment_progress: options[:on_segment_progress]
+      )
+      download.execute
+    end
+  end
+end

data/lib/parallel_sftp/configuration.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module ParallelSftp
+  class Configuration
+    # Number of parallel connections for segmented download
+    attr_accessor :default_segments
+
+    # Connection timeout in seconds
+    attr_accessor :timeout
+
+    # Maximum retry attempts
+    attr_accessor :max_retries
+
+    # Seconds to wait between reconnection attempts
+    attr_accessor :reconnect_interval
+
+    # Default SFTP port
+    attr_accessor :default_port
+
+    # Custom SSH connect program for lftp (e.g., for legacy host key algorithms)
+    # Example: "ssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa"
+    attr_accessor :sftp_connect_program
+
+    def initialize
+      @default_segments = 4
+      @timeout = 30
+      @max_retries = 10
+      @reconnect_interval = 5
+      @default_port = 22
+      @sftp_connect_program = nil
+    end
+
+    # Apply large file optimizations (20GB+)
+    def optimize_for_large_files!
+      @default_segments = 8
+      @timeout = 60
+      @max_retries = 15
+      @reconnect_interval = 10
+    end
+  end
+end