lopace 0.1.0__tar.gz → 0.1.2__tar.gz

{lopace-0.1.0/lopace.egg-info → lopace-0.1.2}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: lopace
-Version: 0.1.0
+Version: 0.1.2
 Summary: Lossless Optimized Prompt Accurate Compression Engine
 Home-page: https://github.com/connectaman/LoPace
 Author: Aman Ulla
 License: MIT
-Project-URL: Homepage, https://github.com/amanulla/lopace
-Project-URL: Repository, https://github.com/amanulla/lopace
-Project-URL: Issues, https://github.com/amanulla/lopace/issues
+Project-URL: Homepage, https://github.com/connectaman/LoPace
+Project-URL: Repository, https://github.com/connectaman/LoPace
+Project-URL: Issues, https://github.com/connectaman/LoPace/issues
 Keywords: prompt,compression,tokenization,zstd,bpe,nlp
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
@@ -38,6 +38,32 @@ A professional, open-source Python package for compressing and decompressing pro
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
 
+## The Problem: Storage Challenges with Large Prompts
+
+When building LLM applications, storing prompts efficiently becomes a critical challenge, especially as you scale:
+
+- **💾 Massive Storage Overhead**: Large system prompts, context windows, and conversation histories consume significant database space. For applications serving thousands of users with multiple LLM calls, this translates to gigabytes or terabytes of storage requirements.
+
+- **🚀 Performance Bottlenecks**: Storing uncompressed prompts increases database size, slows down queries, and increases I/O operations. As your user base grows, retrieval and storage operations become progressively slower.
+
+- **💰 Cost Implications**: Larger databases mean higher cloud storage costs, increased backup times, and more expensive infrastructure. With LLM applications handling millions of prompts, these costs compound rapidly.
+
+- **⚡ Latency Issues**: Loading large prompts from storage adds latency to your application. Multiple LLM calls per user session multiply this problem, creating noticeable delays in response times.
+
+## The Solution: LoPace Compression Engine
+
+LoPace solves these challenges by providing **lossless compression** that dramatically reduces storage requirements while maintaining fast compression and decompression speeds:
+
+- **📉 Up to 80% Space Reduction**: The hybrid compression method can reduce prompt storage by 70-80% on average, meaning you store 5x less data while maintaining perfect fidelity.
+
+- **⚡ Fast Processing**: Achieve 50-200 MB/s compression throughput with sub-linear scaling. Decompression is even faster (100-500 MB/s), ensuring minimal impact on application latency.
+
+- **✅ 100% Lossless**: Perfect reconstruction guarantees your prompts are identical to the original - no data loss, no corruption, no compromises.
+
+- **🎯 Production-Ready**: Optimized for database storage with minimal memory footprint (under 10 MB for typical use cases) and excellent scalability for millions of prompts.
+
+Whether you're storing system prompts for thousands of users, maintaining conversation histories, or caching LLM interactions, LoPace helps you optimize storage costs and improve performance without sacrificing data integrity.
+
 ## Features
 
 - 🚀 **Three Compression Methods**:
@@ -229,17 +255,7 @@ Enumeration of available compression methods:
 
 ### Compression Pipeline (Hybrid Method)
 
-```
-Input: Raw System Prompt String (100%)
-  ↓
-Tokenization: Convert to Tiktoken IDs (~70% reduced)
-  ↓
-Binary Packing: Convert IDs to uint16 (~50% of above)
-  ↓
-Zstd: Final compression (~30% further reduction)
-  ↓
-Output: Compressed Binary Blob
-```
+![Compression Pipeline](screenshots/compression-pipeline.png)
 
 ### Why Hybrid is Best for Databases
 
@@ -257,6 +273,88 @@ Output: Compressed Binary Blob
 #   Hybrid: 120 bytes (76% space saved) ← Best!
 ```
 
+## Benchmarks & Performance Analysis
+
+Comprehensive benchmarks were conducted on 10 diverse prompts across three size categories (small, medium, and large) to evaluate LoPace's compression performance. The following visualizations present detailed analysis of compression metrics, storage efficiency, speed, and memory usage.
+
+### Compression Ratio Analysis
+
+![Compression Ratio](screenshots/compression_ratio.svg)
+
+**Key Insights:**
+- **Hybrid method consistently achieves the highest compression ratios** across all prompt sizes
+- Compression effectiveness increases with prompt size, with large prompts showing 4-6x compression ratios
+- Box plots show the distribution of compression ratios, demonstrating consistent performance
+- Token-based compression provides moderate compression, while Zstd alone offers good baseline performance
+
+### Space Savings Performance
+
+![Space Savings](screenshots/space_savings.svg)
+
+**Key Insights:**
+- **Hybrid method achieves 70-80% space savings** on average across all prompt categories
+- Space savings improve significantly with larger prompts (up to 85% for very large prompts)
+- Error bars indicate consistent performance with low variance
+- All three methods show substantial space reduction compared to uncompressed storage
+
+### Disk Size Comparison
+
+![Disk Size Comparison](screenshots/disk_size_comparison.svg)
+
+**Key Insights:**
+- **Dramatic reduction in storage requirements** - compressed data is 3-6x smaller than original
+- Log-scale visualization shows the magnitude of space savings across different prompt sizes
+- Hybrid method provides the best storage efficiency, especially for large prompts
+- Size reduction percentage increases linearly with prompt complexity
+
+### Speed & Throughput Metrics
+
+![Speed Metrics](screenshots/speed_metrics.svg)
+
+**Key Insights:**
+- **Compression speeds range from 50-200 MB/s** depending on method and prompt size
+- Decompression is consistently faster than compression (100-500 MB/s)
+- Hybrid method maintains excellent throughput despite additional processing steps
+- Processing time scales sub-linearly with prompt size, demonstrating efficient algorithms
+
+### Memory Usage Analysis
+
+![Memory Usage](screenshots/memory_usage.svg)
+
+**Key Insights:**
+- **Memory footprint is minimal** - typically under 10 MB even for large prompts
+- Memory usage scales gracefully with input size
+- Compression and decompression show similar memory requirements
+- All methods demonstrate efficient memory utilization suitable for production environments
+
+### Comprehensive Method Comparison
+
+![Comprehensive Comparison](screenshots/comprehensive_comparison.svg)
+
+**Key Insights:**
+- **Heatmaps provide at-a-glance comparison** of all metrics across methods and prompt sizes
+- Hybrid method consistently ranks highest in compression ratio and space savings
+- Throughput remains competitive across all methods
+- Memory usage is well-balanced, with no method showing excessive requirements
+
+### Scalability Analysis
+
+![Scalability Analysis](screenshots/scalability_analysis.svg)
+
+**Key Insights:**
+- **Performance scales efficiently** with prompt size across all metrics
+- Compression ratio improves with larger inputs (better pattern recognition)
+- Processing time increases sub-linearly, demonstrating algorithmic efficiency
+- Memory usage grows modestly, making LoPace suitable for very large prompts
+
+### Key Findings Summary
+
+1. **Hybrid method is optimal** for maximum compression (70-80% space savings)
+2. **All methods are lossless** - 100% fidelity verified across all test cases
+3. **Speed is production-ready** - 50-200 MB/s compression throughput
+4. **Memory efficient** - Under 10 MB for typical use cases
+5. **Scales excellently** - Performance improves with larger prompts
+
 ## Running the Example
 
 ```bash
@@ -334,6 +432,8 @@ See [.github/workflows/README.md](.github/workflows/README.md) for detailed setu
 
 LoPace uses the following compression techniques:
 
+![Compression techniques](screenshots/lopace-compression-technique.png)
+
 1. **LZ77 (Sliding Window)**: Used **indirectly** through Zstandard
    - Zstandard internally uses LZ77-style algorithms to find repeated patterns
    - Instead of storing "assistant" again, it stores a tuple: (distance_back, length)

{lopace-0.1.0 → lopace-0.1.2}/README.md

@@ -7,6 +7,32 @@ A professional, open-source Python package for compressing and decompressing pro
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
 
+## The Problem: Storage Challenges with Large Prompts
+
+When building LLM applications, storing prompts efficiently becomes a critical challenge, especially as you scale:
+
+- **💾 Massive Storage Overhead**: Large system prompts, context windows, and conversation histories consume significant database space. For applications serving thousands of users with multiple LLM calls, this translates to gigabytes or terabytes of storage requirements.
+
+- **🚀 Performance Bottlenecks**: Storing uncompressed prompts increases database size, slows down queries, and increases I/O operations. As your user base grows, retrieval and storage operations become progressively slower.
+
+- **💰 Cost Implications**: Larger databases mean higher cloud storage costs, increased backup times, and more expensive infrastructure. With LLM applications handling millions of prompts, these costs compound rapidly.
+
+- **⚡ Latency Issues**: Loading large prompts from storage adds latency to your application. Multiple LLM calls per user session multiply this problem, creating noticeable delays in response times.
+
+## The Solution: LoPace Compression Engine
+
+LoPace solves these challenges by providing **lossless compression** that dramatically reduces storage requirements while maintaining fast compression and decompression speeds:
+
+- **📉 Up to 80% Space Reduction**: The hybrid compression method can reduce prompt storage by 70-80% on average, meaning you store 5x less data while maintaining perfect fidelity.
+
+- **⚡ Fast Processing**: Achieve 50-200 MB/s compression throughput with sub-linear scaling. Decompression is even faster (100-500 MB/s), ensuring minimal impact on application latency.
+
+- **✅ 100% Lossless**: Perfect reconstruction guarantees your prompts are identical to the original - no data loss, no corruption, no compromises.
+
+- **🎯 Production-Ready**: Optimized for database storage with minimal memory footprint (under 10 MB for typical use cases) and excellent scalability for millions of prompts.
+
+Whether you're storing system prompts for thousands of users, maintaining conversation histories, or caching LLM interactions, LoPace helps you optimize storage costs and improve performance without sacrificing data integrity.
+
 ## Features
 
 - 🚀 **Three Compression Methods**:
@@ -198,17 +224,7 @@ Enumeration of available compression methods:
 
 ### Compression Pipeline (Hybrid Method)
 
-```
-Input: Raw System Prompt String (100%)
-  ↓
-Tokenization: Convert to Tiktoken IDs (~70% reduced)
-  ↓
-Binary Packing: Convert IDs to uint16 (~50% of above)
-  ↓
-Zstd: Final compression (~30% further reduction)
-  ↓
-Output: Compressed Binary Blob
-```
+![Compression Pipeline](screenshots/compression-pipeline.png)
 
 ### Why Hybrid is Best for Databases
 
@@ -226,6 +242,88 @@ Output: Compressed Binary Blob
 #   Hybrid: 120 bytes (76% space saved) ← Best!
 ```
 
+## Benchmarks & Performance Analysis
+
+Comprehensive benchmarks were conducted on 10 diverse prompts across three size categories (small, medium, and large) to evaluate LoPace's compression performance. The following visualizations present detailed analysis of compression metrics, storage efficiency, speed, and memory usage.
+
+### Compression Ratio Analysis
+
+![Compression Ratio](screenshots/compression_ratio.svg)
+
+**Key Insights:**
+- **Hybrid method consistently achieves the highest compression ratios** across all prompt sizes
+- Compression effectiveness increases with prompt size, with large prompts showing 4-6x compression ratios
+- Box plots show the distribution of compression ratios, demonstrating consistent performance
+- Token-based compression provides moderate compression, while Zstd alone offers good baseline performance
+
+### Space Savings Performance
+
+![Space Savings](screenshots/space_savings.svg)
+
+**Key Insights:**
+- **Hybrid method achieves 70-80% space savings** on average across all prompt categories
+- Space savings improve significantly with larger prompts (up to 85% for very large prompts)
+- Error bars indicate consistent performance with low variance
+- All three methods show substantial space reduction compared to uncompressed storage
+
+### Disk Size Comparison
+
+![Disk Size Comparison](screenshots/disk_size_comparison.svg)
+
+**Key Insights:**
+- **Dramatic reduction in storage requirements** - compressed data is 3-6x smaller than original
+- Log-scale visualization shows the magnitude of space savings across different prompt sizes
+- Hybrid method provides the best storage efficiency, especially for large prompts
+- Size reduction percentage increases linearly with prompt complexity
+
+### Speed & Throughput Metrics
+
+![Speed Metrics](screenshots/speed_metrics.svg)
+
+**Key Insights:**
+- **Compression speeds range from 50-200 MB/s** depending on method and prompt size
+- Decompression is consistently faster than compression (100-500 MB/s)
+- Hybrid method maintains excellent throughput despite additional processing steps
+- Processing time scales sub-linearly with prompt size, demonstrating efficient algorithms
+
+### Memory Usage Analysis
+
+![Memory Usage](screenshots/memory_usage.svg)
+
+**Key Insights:**
+- **Memory footprint is minimal** - typically under 10 MB even for large prompts
+- Memory usage scales gracefully with input size
+- Compression and decompression show similar memory requirements
+- All methods demonstrate efficient memory utilization suitable for production environments
+
+### Comprehensive Method Comparison
+
+![Comprehensive Comparison](screenshots/comprehensive_comparison.svg)
+
+**Key Insights:**
+- **Heatmaps provide at-a-glance comparison** of all metrics across methods and prompt sizes
+- Hybrid method consistently ranks highest in compression ratio and space savings
+- Throughput remains competitive across all methods
+- Memory usage is well-balanced, with no method showing excessive requirements
+
+### Scalability Analysis
+
+![Scalability Analysis](screenshots/scalability_analysis.svg)
+
+**Key Insights:**
+- **Performance scales efficiently** with prompt size across all metrics
+- Compression ratio improves with larger inputs (better pattern recognition)
+- Processing time increases sub-linearly, demonstrating algorithmic efficiency
+- Memory usage grows modestly, making LoPace suitable for very large prompts
+
+### Key Findings Summary
+
+1. **Hybrid method is optimal** for maximum compression (70-80% space savings)
+2. **All methods are lossless** - 100% fidelity verified across all test cases
+3. **Speed is production-ready** - 50-200 MB/s compression throughput
+4. **Memory efficient** - Under 10 MB for typical use cases
+5. **Scales excellently** - Performance improves with larger prompts
+
 ## Running the Example
 
 ```bash
@@ -303,6 +401,8 @@ See [.github/workflows/README.md](.github/workflows/README.md) for detailed setu
 
 LoPace uses the following compression techniques:
 
+![Compression techniques](screenshots/lopace-compression-technique.png)
+
 1. **LZ77 (Sliding Window)**: Used **indirectly** through Zstandard
    - Zstandard internally uses LZ77-style algorithms to find repeated patterns
    - Instead of storing "assistant" again, it stores a tuple: (distance_back, length)

{lopace-0.1.0 → lopace-0.1.2/lopace.egg-info}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: lopace
-Version: 0.1.0
+Version: 0.1.2
 Summary: Lossless Optimized Prompt Accurate Compression Engine
 Home-page: https://github.com/connectaman/LoPace
 Author: Aman Ulla
 License: MIT
-Project-URL: Homepage, https://github.com/amanulla/lopace
-Project-URL: Repository, https://github.com/amanulla/lopace
-Project-URL: Issues, https://github.com/amanulla/lopace/issues
+Project-URL: Homepage, https://github.com/connectaman/LoPace
+Project-URL: Repository, https://github.com/connectaman/LoPace
+Project-URL: Issues, https://github.com/connectaman/LoPace/issues
 Keywords: prompt,compression,tokenization,zstd,bpe,nlp
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
@@ -38,6 +38,32 @@ A professional, open-source Python package for compressing and decompressing pro
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
 
+## The Problem: Storage Challenges with Large Prompts
+
+When building LLM applications, storing prompts efficiently becomes a critical challenge, especially as you scale:
+
+- **💾 Massive Storage Overhead**: Large system prompts, context windows, and conversation histories consume significant database space. For applications serving thousands of users with multiple LLM calls, this translates to gigabytes or terabytes of storage requirements.
+
+- **🚀 Performance Bottlenecks**: Storing uncompressed prompts increases database size, slows down queries, and increases I/O operations. As your user base grows, retrieval and storage operations become progressively slower.
+
+- **💰 Cost Implications**: Larger databases mean higher cloud storage costs, increased backup times, and more expensive infrastructure. With LLM applications handling millions of prompts, these costs compound rapidly.
+
+- **⚡ Latency Issues**: Loading large prompts from storage adds latency to your application. Multiple LLM calls per user session multiply this problem, creating noticeable delays in response times.
+
+## The Solution: LoPace Compression Engine
+
+LoPace solves these challenges by providing **lossless compression** that dramatically reduces storage requirements while maintaining fast compression and decompression speeds:
+
+- **📉 Up to 80% Space Reduction**: The hybrid compression method can reduce prompt storage by 70-80% on average, meaning you store 5x less data while maintaining perfect fidelity.
+
+- **⚡ Fast Processing**: Achieve 50-200 MB/s compression throughput with sub-linear scaling. Decompression is even faster (100-500 MB/s), ensuring minimal impact on application latency.
+
+- **✅ 100% Lossless**: Perfect reconstruction guarantees your prompts are identical to the original - no data loss, no corruption, no compromises.
+
+- **🎯 Production-Ready**: Optimized for database storage with minimal memory footprint (under 10 MB for typical use cases) and excellent scalability for millions of prompts.
+
+Whether you're storing system prompts for thousands of users, maintaining conversation histories, or caching LLM interactions, LoPace helps you optimize storage costs and improve performance without sacrificing data integrity.
+
 ## Features
 
 - 🚀 **Three Compression Methods**:
@@ -229,17 +255,7 @@ Enumeration of available compression methods:
 
 ### Compression Pipeline (Hybrid Method)
 
-```
-Input: Raw System Prompt String (100%)
-  ↓
-Tokenization: Convert to Tiktoken IDs (~70% reduced)
-  ↓
-Binary Packing: Convert IDs to uint16 (~50% of above)
-  ↓
-Zstd: Final compression (~30% further reduction)
-  ↓
-Output: Compressed Binary Blob
-```
+![Compression Pipeline](screenshots/compression-pipeline.png)
 
 ### Why Hybrid is Best for Databases
 
@@ -257,6 +273,88 @@ Output: Compressed Binary Blob
 #   Hybrid: 120 bytes (76% space saved) ← Best!
 ```
 
+## Benchmarks & Performance Analysis
+
+Comprehensive benchmarks were conducted on 10 diverse prompts across three size categories (small, medium, and large) to evaluate LoPace's compression performance. The following visualizations present detailed analysis of compression metrics, storage efficiency, speed, and memory usage.
+
+### Compression Ratio Analysis
+
+![Compression Ratio](screenshots/compression_ratio.svg)
+
+**Key Insights:**
+- **Hybrid method consistently achieves the highest compression ratios** across all prompt sizes
+- Compression effectiveness increases with prompt size, with large prompts showing 4-6x compression ratios
+- Box plots show the distribution of compression ratios, demonstrating consistent performance
+- Token-based compression provides moderate compression, while Zstd alone offers good baseline performance
+
+### Space Savings Performance
+
+![Space Savings](screenshots/space_savings.svg)
+
+**Key Insights:**
+- **Hybrid method achieves 70-80% space savings** on average across all prompt categories
+- Space savings improve significantly with larger prompts (up to 85% for very large prompts)
+- Error bars indicate consistent performance with low variance
+- All three methods show substantial space reduction compared to uncompressed storage
+
+### Disk Size Comparison
+
+![Disk Size Comparison](screenshots/disk_size_comparison.svg)
+
+**Key Insights:**
+- **Dramatic reduction in storage requirements** - compressed data is 3-6x smaller than original
+- Log-scale visualization shows the magnitude of space savings across different prompt sizes
+- Hybrid method provides the best storage efficiency, especially for large prompts
+- Size reduction percentage increases linearly with prompt complexity
+
+### Speed & Throughput Metrics
+
+![Speed Metrics](screenshots/speed_metrics.svg)
+
+**Key Insights:**
+- **Compression speeds range from 50-200 MB/s** depending on method and prompt size
+- Decompression is consistently faster than compression (100-500 MB/s)
+- Hybrid method maintains excellent throughput despite additional processing steps
+- Processing time scales sub-linearly with prompt size, demonstrating efficient algorithms
+
+### Memory Usage Analysis
+
+![Memory Usage](screenshots/memory_usage.svg)
+
+**Key Insights:**
+- **Memory footprint is minimal** - typically under 10 MB even for large prompts
+- Memory usage scales gracefully with input size
+- Compression and decompression show similar memory requirements
+- All methods demonstrate efficient memory utilization suitable for production environments
+
+### Comprehensive Method Comparison
+
+![Comprehensive Comparison](screenshots/comprehensive_comparison.svg)
+
+**Key Insights:**
+- **Heatmaps provide at-a-glance comparison** of all metrics across methods and prompt sizes
+- Hybrid method consistently ranks highest in compression ratio and space savings
+- Throughput remains competitive across all methods
+- Memory usage is well-balanced, with no method showing excessive requirements
+
+### Scalability Analysis
+
+![Scalability Analysis](screenshots/scalability_analysis.svg)
+
+**Key Insights:**
+- **Performance scales efficiently** with prompt size across all metrics
+- Compression ratio improves with larger inputs (better pattern recognition)
+- Processing time increases sub-linearly, demonstrating algorithmic efficiency
+- Memory usage grows modestly, making LoPace suitable for very large prompts
+
+### Key Findings Summary
+
+1. **Hybrid method is optimal** for maximum compression (70-80% space savings)
+2. **All methods are lossless** - 100% fidelity verified across all test cases
+3. **Speed is production-ready** - 50-200 MB/s compression throughput
+4. **Memory efficient** - Under 10 MB for typical use cases
+5. **Scales excellently** - Performance improves with larger prompts
+
 ## Running the Example
 
 ```bash
@@ -334,6 +432,8 @@ See [.github/workflows/README.md](.github/workflows/README.md) for detailed setu
 
 LoPace uses the following compression techniques:
 
+![Compression techniques](screenshots/lopace-compression-technique.png)
+
 1. **LZ77 (Sliding Window)**: Used **indirectly** through Zstandard
    - Zstandard internally uses LZ77-style algorithms to find repeated patterns
    - Instead of storing "assistant" again, it stores a tuple: (distance_back, length)

{lopace-0.1.0 → lopace-0.1.2}/lopace.egg-info/SOURCES.txt

@@ -11,5 +11,7 @@ lopace.egg-info/SOURCES.txt
 lopace.egg-info/dependency_links.txt
 lopace.egg-info/requires.txt
 lopace.egg-info/top_level.txt
+scripts/__init__.py
+scripts/generate_visualizations.py
 tests/__init__.py
 tests/test_compressor.py

{lopace-0.1.0 → lopace-0.1.2}/lopace.egg-info/top_level.txt

@@ -1,2 +1,3 @@
 lopace
+scripts
 tests

{lopace-0.1.0 → lopace-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "lopace"
-version = "0.1.0"
+version = "0.1.2"
 description = "Lossless Optimized Prompt Accurate Compression Engine"
 readme = "README.md"
 requires-python = ">=3.8"
@@ -32,6 +32,6 @@ dependencies = [
 ]
 
 [project.urls]
-Homepage = "https://github.com/amanulla/lopace"
-Repository = "https://github.com/amanulla/lopace"
-Issues = "https://github.com/amanulla/lopace/issues"
+Homepage = "https://github.com/connectaman/LoPace"
+Repository = "https://github.com/connectaman/LoPace"
+Issues = "https://github.com/connectaman/LoPace/issues"

lopace-0.1.2/scripts/__init__.py

@@ -0,0 +1 @@
+"""Scripts for generating visualizations and benchmarks."""