omnizip 0.3.1

data/docs/guides/compression-algorithms/zstandard.adoc

@@ -0,0 +1,361 @@
+---
+title: Zstandard
+nav_order: 8
+parent: Compression Algorithms
+grand_parent: Guides
+---
+
+== Purpose
+
+Zstandard (zstd) is a modern, fast compression algorithm developed by Facebook that offers an excellent balance between compression ratio and speed. It significantly outperforms Deflate in both speed and compression ratio.
+
+== Key Characteristics
+
+[cols="1,3"]
+|===
+|Property |Value
+
+|Compression Ratio
+|Very good (better than Deflate)
+
+|Compression Speed
+|Very fast (3-4x faster than Deflate)
+
+|Decompression Speed
+|Extremely fast (4-5x faster than Deflate)
+
+|Memory Usage
+|Low to medium
+
+|Best For
+|Real-time compression, high-throughput applications, modern systems
+|===
+
+== When to Use Zstandard
+
+**Choose Zstandard when**:
+
+* Need fast compression with good ratios
+* Real-time compression requirements
+* High throughput is critical
+* Working with modern systems and tools
+* Want better performance than Deflate without LZMA's slowness
+
+**Consider alternatives when**:
+
+* Maximum compatibility is required (use Deflate)
+* Best compression ratio needed (use LZMA2)
+* Working with legacy systems
+
+== Basic Usage
+
+=== Create Zstandard Archive
+
+[source,ruby]
+----
+# Create with Zstandard compression
+Omnizip::Archive.create('fast-backup.7z', format: :seven_zip) do |archive|
+  archive.compression = :zstandard
+  archive.level = 3  # Default level
+  archive.add_directory('data/')
+end
+----
+
+=== Quick Compression
+
+[source,ruby]
+----
+# Fast compression for temporary archives
+Omnizip.compress_file(
+  'logfile.log',
+  'logs.7z',
+  compression: :zstandard,
+  level: 1  # Fastest compression
+)
+----
+
+== Compression Levels
+
+Zstandard offers levels 1-22 with fine-grained control:
+
+[cols="1,2,2,2"]
+|===
+|Level Range |Speed |Ratio |Typical Use
+
+|1-3
+|Very Fast
+|Good
+|Real-time, temporary files
+
+|3-7
+|Fast
+|Very Good
+|**Default** - general use
+
+|8-15
+|Medium
+|Excellent
+|Distribution, important data
+
+|16-22
+|Slow
+|Excellent+
+|Archival (approaches LZMA)
+|===
+
+.Level Performance Example
+[source,ruby]
+----
+# Compare different Zstandard levels
+[1, 3, 10, 19].each do |level|
+  time = Benchmark.measure do
+    Omnizip.compress_file('data.bin', "zstd-#{level}.7z",
+      compression: :zstandard,
+      level: level
+    )
+  end
+
+  size = File.size("zstd-#{level}.7z")
+  puts "Level #{level}: #{size / 1024} KB in #{time.real.round(3)}s"
+end
+
+# Output:
+# Level 1:  145 KB in 0.052s (fastest)
+# Level 3:  128 KB in 0.089s (default)
+# Level 10: 105 KB in 0.425s (better compression)
+# Level 19: 98 KB in 2.156s (approaching LZMA quality)
+----
+
+== Performance Comparison
+
+=== Speed vs Deflate
+
+[cols="2,1,1,1"]
+|===
+|Operation |Deflate |Zstandard |Improvement
+
+|Compress 100 MB
+|10s
+|3s
+|3.3x faster
+
+|Decompress 100 MB
+|5s
+|1.2s
+|4.2x faster
+
+|Compression Ratio
+|68%
+|72%
+|4% better
+|===
+
+=== Speed vs LZMA
+
+[cols="2,1,1,1"]
+|===
+|Operation |LZMA |Zstandard |Trade-off
+
+|Compress 100 MB
+|60s
+|3s
+|20x faster, slightly worse ratio
+
+|Decompress 100 MB
+|8s
+|1.2s
+|6.7x faster
+
+|Compression Ratio
+|78%
+|72%
+|LZMA ~6% better
+|===
+
+== Advanced Features
+
+=== Dictionary Training
+
+Train custom dictionaries for specific data types:
+
+[source,ruby]
+----
+# Train dictionary on sample data
+training_data = Dir.glob('samples/*.json').map { |f| File.read(f) }
+dictionary = Omnizip::Zstandard.train_dictionary(training_data, size: 100.kilobytes)
+
+# Use trained dictionary for better compression
+Omnizip.compress_file('data.json', 'output.7z',
+  compression: :zstandard,
+  level: 5,
+  dictionary: dictionary
+)
+
+# Can improve compression 10-30% for similar data
+----
+
+=== Streaming with Zstandard
+
+Excellent streaming performance:
+
+[source,ruby]
+----
+# Stream compression for real-time data
+Omnizip::OutputStream.open('stream.7z', format: :seven_zip) do |stream|
+  stream.compression = :zstandard
+  stream.level = 3  # Fast for real-time
+
+  # Process log entries as they arrive
+  log_stream.each do |log_entry|
+    stream.put_next_entry("log_#{log_entry.id}.txt")
+    stream.write(log_entry.content)
+  end
+end
+----
+
+== Use Cases
+
+=== Log File Compression
+
+Perfect for compressing logs quickly:
+
+[source,ruby]
+----
+# Compress today's logs rapidly
+log_files = Dir.glob('/var/log/app/*.log')
+
+Omnizip.compress_files(
+  log_files,
+  "logs_#{Date.today}.7z",
+  compression: :zstandard,
+  level: 5  # Fast with good compression
+) do |progress|
+  puts "Compressing logs: #{progress.percentage}%"
+end
+----
+
+=== Web Application Responses
+
+Compress API responses on-the-fly:
+
+[source,ruby]
+----
+class ExportController < ApplicationController
+  def download
+    # Generate export data
+    data = generate_export_data
+
+    # Compress quickly with Zstandard
+    compressed = Omnizip::Buffer.create(:seven_zip) do |archive|
+      archive.compression = :zstandard
+      archive.level = 3  # Fast compression
+
+      data.each do |filename, content|
+        archive.add(filename, content)
+      end
+    end
+
+    send_data compressed.string,
+      filename: 'export.7z',
+      type: 'application/x-7z-compressed'
+  end
+end
+----
+
+=== Database Dumps
+
+Compress database backups efficiently:
+
+[source,ruby]
+----
+# Stream database dump to compressed archive
+require 'open3'
+
+Omnizip::OutputStream.open('db-backup.7z', format: :seven_zip) do |stream|
+  stream.compression = :zstandard
+  stream.level = 7
+  stream.put_next_entry('database.sql')
+
+  # Stream pg_dump output directly
+  Open3.popen3('pg_dump mydb') do |stdin, stdout, stderr, wait_thr|
+    while chunk = stdout.read(65536)
+      stream.write(chunk)
+    end
+  end
+end
+----
+
+== Memory Efficiency
+
+Zstandard is memory efficient:
+
+[cols="1,1,1"]
+|===
+|Level |Compression RAM |Decompression RAM
+
+|1-3
+|~1 MB
+|<1 MB
+
+|4-7
+|~8 MB
+|<1 MB
+
+|8-15
+|~30 MB
+|<1 MB
+
+|16-22
+|~100 MB
+|<1 MB
+|===
+
+*Note: Decompression always requires minimal memory*
+
+== Best Practices
+
+. **Use Level 3-5 for General Use**: Excellent speed/ratio balance
+. **Level 1 for Real-Time**: When speed is critical
+. **Level 10+ for Archival**: When time permits better compression
+. **Train Dictionaries**: For repetitive data types (JSON, logs)
+. **Monitor Performance**: Should achieve 50-200 MB/s compression
+
+== Command-Line Usage
+
+[source,bash]
+----
+# Create Zstandard-compressed archive
+$ omnizip archive create backup.7z data/ \
+    --compression zstandard \
+    --level 5
+
+# Fast compression
+$ omnizip archive create logs.7z logs/ \
+    --compression zstandard \
+    --level 1 \
+    --verbose
+Algorithm: Zstandard, Level: 1
+Compression speed: 185 MB/s
+Original: 500 MB
+Compressed: 145 MB (71% reduction)
+Time: 2.7s
+----
+
+== Technical Details
+
+**Zstandard Features**:
+
+* **Fast Entropy Encoding**: Finite State Entropy (FSE) for speed
+* **Adaptive Window**: Dynamically sized based on level
+* **Dictionary Compression**: Optional pre-trained dictionaries
+* **Multi-Threading**: Native parallel compression support
+* **Seekable Format**: Optional frame indexes for random access
+
+**Implementation**: Uses zstd-ruby gem (native binding)
+
+== See Also
+
+* link:deflate.html[Deflate] - Traditional fast compression
+* link:lzma2.html[LZMA2] - Better ratio, slower speed
+* link:../advanced-features/streaming.html[Streaming] - Zstandard excels at streaming
+* link:../performance-tuning.html[Performance Tuning] - Optimize for speed

data/docs/guides/creating-archives.adoc

@@ -0,0 +1,354 @@
+---
+title: Creating Archives
+nav_order: 1
+parent: Guides
+---
+
+== Creating Archives
+
+Step-by-step guide to creating archives with Omnizip.
+
+== Quick Examples
+
+=== Create ZIP Archive
+
+[source,ruby]
+----
+require 'omnizip'
+
+Omnizip::Archive.create('backup.zip') do |archive|
+  archive.add_file('document.pdf')
+  archive.add_directory('photos/')
+end
+----
+
+=== Create 7z Archive
+
+[source,ruby]
+----
+Omnizip::Archive.create('backup.7z', format: :seven_zip) do |archive|
+  archive.add_directory('data/')
+end
+----
+
+=== Create RAR5 Archive
+
+[source,ruby]
+----
+require 'omnizip/formats/rar/rar5/writer'
+
+# STORE compression (uncompressed)
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar')
+writer.add_file('document.txt')
+writer.add_file('image.png')
+writer.write
+
+# LZMA compression (default level 3)
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar',
+  compression: :lzma
+)
+writer.add_file('large_dataset.json')
+writer.write
+
+# LZMA level 5 (best compression)
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar',
+  compression: :lzma,
+  level: 5
+)
+writer.add_directory('photos/')
+writer.write
+
+# Include modification time
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar',
+  include_mtime: true
+)
+writer.add_file('document.txt')
+writer.write
+
+# Include CRC32 (STORE compression only)
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar',
+  compression: :store,
+  include_crc32: true
+)
+writer.add_file('data.bin')
+writer.write
+----
+
+IMPORTANT: RAR5 CRC32 checksums only work with STORE compression. When LZMA is used, CRC32 is automatically disabled. Use BLAKE2sp (always present) for compressed file integrity.
+
+=== RAR5 Multi-Volume Archives (v0.5.0)
+
+Split large archives across multiple volumes:
+
+[source,ruby]
+----
+require 'omnizip/formats/rar/rar5/writer'
+
+# Basic multi-volume with default naming (part1, part2, ...)
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('backup.rar',
+  multi_volume: true,
+  volume_size: '100M'  # Human-readable sizes supported
+)
+writer.add_directory('large_dataset/')
+volumes = writer.write
+# => ['backup.part1.rar', 'backup.part2.rar', ...]
+
+# Custom volume naming patterns
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('data.rar',
+  multi_volume: true,
+  volume_size: '4.7GB',  # CD/DVD sizes
+  volume_naming: 'numeric'  # data.001.rar, data.002.rar, ...
+)
+writer.add_file('huge_database.sql')
+volumes = writer.write
+
+# Volume naming options:
+# - 'part' (default): archive.part1.rar, archive.part2.rar
+# - 'volume': archive.volume1.rar, archive.volume2.rar
+# - 'numeric': archive.001.rar, archive.002.rar
+
+# Minimum volume size: 64 KB
+----
+
+=== RAR5 Solid Compression (v0.5.0)
+
+Achieve 10-30% better compression ratios for similar files:
+
+[source,ruby]
+----
+require 'omnizip/formats/rar/rar5/writer'
+
+# Enable solid compression
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('project.rar',
+  compression: :lzma,
+  level: 5,
+  solid: true  # Shared dictionary across files
+)
+writer.add_directory('source_code/')
+writer.write
+
+# Best for:
+# - Source code repositories
+# - Log files
+# - Text documents
+# - Similar structured data
+
+# Trade-offs:
+# - Cannot extract individual files without decompressing entire block
+# - Corruption may affect multiple files
+# - Slightly longer extraction time for single files
+----
+
+=== RAR5 AES-256 Encryption (v0.5.0)
+
+Password-protect archives with industry-standard encryption:
+
+[source,ruby]
+----
+require 'omnizip/formats/rar/rar5/writer'
+
+# Basic password protection
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('secure.rar',
+  compression: :lzma,
+  password: 'SecurePassword123!'
+)
+writer.add_file('confidential.pdf')
+writer.write
+
+# Custom key derivation iterations (higher = more secure, slower)
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('protected.rar',
+  compression: :lzma,
+  level: 5,
+  password: 'VerySecurePassword2025!',
+  kdf_iterations: 524_288  # 2^19, more secure than default
+)
+writer.add_directory('sensitive_data/')
+writer.write
+
+# KDF iteration options:
+# - 65,536 (2^16): Minimum, fast but less secure
+# - 262,144 (2^18): Default, balanced security/performance
+# - 1,048,576 (2^20): Maximum, strongest security
+
+# Security features:
+# - AES-256-CBC encryption with PKCS#7 padding
+# - PBKDF2-HMAC-SHA256 key derivation
+# - Per-file IV generation
+# - Password verification before decryption
+----
+
+=== RAR5 PAR2 Recovery Records (v0.5.0)
+
+Add error correction for archive recovery:
+
+[source,ruby]
+----
+require 'omnizip/formats/rar/rar5/writer'
+
+# Enable PAR2 recovery with default 5% redundancy
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('backup.rar',
+  compression: :lzma,
+  recovery: true
+)
+writer.add_directory('important_data/')
+files = writer.write
+# => ['backup.rar', 'backup.par2', 'backup.vol00+01.par2', ...]
+
+# Custom redundancy percentage
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('critical.rar',
+  compression: :lzma,
+  level: 5,
+  recovery: true,
+  recovery_percent: 15  # Can recover up to 15% data loss
+)
+writer.add_file('mission_critical.db')
+files = writer.write
+
+# Redundancy guidelines:
+# - 5% (default): General backups
+# - 10%: Important data
+# - 20-30%: High-value data
+# - 50-100%: Maximum protection
+
+# Recovery capabilities:
+# - Detect corruption at block level
+# - Repair damaged archives automatically
+# - Works with multi-volume, solid, and encrypted archives
+----
+
+=== RAR5 Combined Features (v0.5.0)
+
+Use all features together for comprehensive protection:
+
+[source,ruby]
+----
+require 'omnizip/formats/rar/rar5/writer'
+
+# Production-ready backup configuration
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('complete_backup.rar',
+  # Compression
+  compression: :lzma,
+  level: 5,
+  solid: true,
+
+  # Security
+  password: 'SecureBackup2025!',
+  kdf_iterations: 262_144,
+
+  # Multi-volume
+  multi_volume: true,
+  volume_size: '4G',  # DVD-sized volumes
+  volume_naming: 'part',
+
+  # Recovery
+  recovery: true,
+  recovery_percent: 10,
+
+  # Optional fields
+  include_mtime: true
+)
+
+writer.add_directory('/critical/data')
+writer.add_directory('/important/documents')
+files = writer.write
+
+puts "Backup created: #{files.size} files"
+archive_files = files.select { |f| f.end_with?('.rar') }
+par2_files = files.select { |f| f.include?('.par2') }
+puts "  Archive volumes: #{archive_files.size}"
+puts "  PAR2 files: #{par2_files.size}"
+puts "  Total size: #{files.sum { |f| File.size(f) } / 1024 / 1024}MB"
+----
+
+=== RAR5 Best Practices
+
+**For Maximum Compression:**
+[source,ruby]
+----
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar',
+  compression: :lzma,
+  level: 5,        # Best compression
+  solid: true,     # 10-30% better ratios
+  include_mtime: true
+)
+----
+
+**For Important Data:**
+[source,ruby]
+----
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('backup.rar',
+  compression: :lzma,
+  level: 3,              # Balanced performance
+  recovery: true,
+  recovery_percent: 15,  # 15% redundancy
+  include_mtime: true
+)
+----
+
+**For Optical Media:**
+[source,ruby]
+----
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('archive.rar',
+  compression: :lzma,
+  level: 5,
+  solid: true,
+  multi_volume: true,
+  volume_size: '4.7GB',  # DVD size
+  recovery: true,
+  include_mtime: true
+)
+----
+
+**For Secure Archives:**
+[source,ruby]
+----
+writer = Omnizip::Formats::Rar::Rar5::Writer.new('secure.rar',
+  compression: :lzma,
+  level: 5,
+  password: ENV['BACKUP_PASSWORD'],
+  kdf_iterations: 524_288,  # Enhanced security
+  recovery: true,
+  recovery_percent: 10,
+  include_mtime: true
+)
+----
+
+== Adding Files
+
+=== Individual Files
+
+[source,ruby]
+----
+archive.add_file('source.txt', 'dest.txt')
+----
+
+=== Directories
+
+[source,ruby]
+----
+archive.add_directory('folder/')
+----
+
+=== From Data
+
+[source,ruby]
+----
+archive.add_data('config.json', JSON.generate(config))
+----
+
+== Compression Options
+
+[source,ruby]
+----
+Omnizip::Archive.create('archive.7z',
+                        format: :seven_zip,
+                        algorithm: :lzma2,
+                        level: 9) do |archive|
+  # ...
+end
+----
+
+== See Also
+
+* link:extracting-archives.html[Extracting Archives]
+* link:../reference/api/overview.html[Ruby API]