lopace 0.1.3__tar.gz → 0.1.5__tar.gz

lopace-0.1.5/.dockerignore

@@ -0,0 +1,57 @@
+# Git
+.git
+.gitignore
+.github
+
+# Python
+__pycache__
+*.py[cod]
+*$py.class
+*.so
+.Python
+*.egg-info
+dist
+build
+.eggs
+
+# Virtual environments
+venv/
+ENV/
+env/
+.venv
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+
+# Documentation
+*.md
+!README.md
+
+# CI/CD
+.github/
+
+# Development files
+tests/
+scripts/
+example.py
+
+# Logs
+*.log
+
+# Cache
+.mypy_cache/
+.dmypy.json
+dmypy.json

lopace-0.1.5/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,59 @@
+---
+name: Bug Report
+about: Create a report to help us improve
+title: '[BUG] '
+labels: bug
+assignees: ''
+---
+
+## Bug Description
+
+A clear and concise description of what the bug is.
+
+## Steps to Reproduce
+
+1. Import the library: `...`
+2. Call the function: `...`
+3. See error: `...`
+
+## Expected Behavior
+
+A clear and concise description of what you expected to happen.
+
+## Actual Behavior
+
+A clear and concise description of what actually happened.
+
+## Code Example
+
+```python
+from lopace import PromptCompressor, CompressionMethod
+
+compressor = PromptCompressor()
+# Your code here
+```
+
+## Error Message
+
+```
+Paste the full error traceback here
+```
+
+## Environment
+
+- **OS**: [e.g., Windows 10, macOS 13, Ubuntu 22.04]
+- **Python Version**: [e.g., 3.11.0]
+- **LoPace Version**: [e.g., 0.1.0]
+- **Installed Dependencies**:
+  ```
+  zstandard==...
+  tiktoken==...
+  ```
+
+## Additional Context
+
+Add any other context about the problem here.
+
+## Possible Solution
+
+If you have suggestions on how to fix this, please describe them here.

lopace-0.1.5/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,44 @@
+---
+name: Feature Request
+about: Suggest an idea for this project
+title: '[FEATURE] '
+labels: enhancement
+assignees: ''
+---
+
+## Feature Description
+
+A clear and concise description of the feature you'd like to see.
+
+## Motivation
+
+Why is this feature needed? What problem does it solve?
+
+## Proposed Solution
+
+A clear and concise description of what you want to happen.
+
+## Example Usage
+
+```python
+from lopace import PromptCompressor
+
+compressor = PromptCompressor()
+# Show how you would use the new feature
+```
+
+## Alternatives Considered
+
+A clear and concise description of any alternative solutions or features you've considered.
+
+## Additional Context
+
+Add any other context, mockups, or examples about the feature request here.
+
+## Implementation Ideas
+
+If you have ideas on how to implement this feature, please share them here.
+
+## Related Issues
+
+Link to any related issues or discussions.

lopace-0.1.5/.github/dependabot.yml

@@ -0,0 +1,34 @@
+version: 2
+updates:
+  # Enable version updates for pip
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "09:00"
+    open-pull-requests-limit: 5
+    labels:
+      - "dependencies"
+      - "python"
+    commit-message:
+      prefix: "chore"
+      include: "scope"
+    ignore:
+      # Ignore major version updates for critical dependencies
+      - dependency-name: "zstandard"
+        update-types: ["version-update:semver-major"]
+      - dependency-name: "tiktoken"
+        update-types: ["version-update:semver-major"]
+
+  # Enable version updates for GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "monthly"
+    labels:
+      - "dependencies"
+      - "github-actions"
+    commit-message:
+      prefix: "chore"
+      include: "scope"

lopace-0.1.5/.github/pull_request_template.md

@@ -0,0 +1,70 @@
+## Description
+
+<!-- Provide a brief description of your changes -->
+
+## Type of Change
+
+<!-- Mark the relevant option with an 'x' -->
+
+- [ ] Bug fix (non-breaking change which fixes an issue)
+- [ ] New feature (non-breaking change which adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
+- [ ] Documentation update
+- [ ] Performance improvement
+- [ ] Code refactoring
+- [ ] Test addition/update
+- [ ] Other (please describe):
+
+## Related Issues
+
+<!-- Link to related issues using #issue_number -->
+
+Fixes #
+Related to #
+
+## Changes Made
+
+<!-- Describe the changes you made in detail -->
+
+## Testing
+
+<!-- Describe the tests you ran and how to verify your changes -->
+
+- [ ] All existing tests pass
+- [ ] New tests added for new functionality
+- [ ] Manual testing completed
+
+### Test Commands
+
+```bash
+# Add any test commands you ran
+pytest tests/ -v
+```
+
+## Screenshots/Demo
+
+<!-- If applicable, add screenshots or a demo -->
+
+## Checklist
+
+<!-- Mark completed items with an 'x' -->
+
+- [ ] My code follows the style guidelines of this project
+- [ ] I have performed a self-review of my own code
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding changes to the documentation
+- [ ] My changes generate no new warnings
+- [ ] I have added tests that prove my fix is effective or that my feature works
+- [ ] New and existing unit tests pass locally with my changes
+- [ ] Any dependent changes have been merged and published
+- [ ] I have updated the CHANGELOG.md if needed
+- [ ] I have read and followed the [Contributing Guidelines](CONTRIBUTING.md)
+- [ ] I have read and agree to the [Code of Conduct](CODE_OF_CONDUCT.md)
+
+## Additional Notes
+
+<!-- Add any additional notes, context, or information for reviewers -->
+
+## Review Request
+
+<!-- Tag specific reviewers or leave blank for general review -->

lopace-0.1.5/.github/workflows/ci-cd.yml

@@ -0,0 +1,142 @@
+name: CI/CD - Test and Publish to PyPI
+
+on:
+  push:
+    branches:
+      - main
+      - master
+    tags:
+      - 'v*'  # Trigger on version tags like v0.1.0
+  pull_request:
+    branches:
+      - main
+      - master
+
+jobs:
+  test:
+    name: Run Tests
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+    
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+      
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+          pip install -e .
+      
+      - name: Run linting
+        run: |
+          flake8 lopace tests --count --select=E9,F63,F7,F82 --show-source --statistics
+          flake8 lopace tests --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+        continue-on-error: true
+      
+      - name: Run tests with pytest
+        run: |
+          pytest tests/ -v --cov=lopace --cov-report=xml --cov-report=term
+      
+      - name: Upload coverage reports
+        uses: codecov/codecov-action@v5
+        with:
+          file: ./coverage.xml
+          flags: unittests
+          name: codecov-umbrella
+          fail_ci_if_error: false
+        continue-on-error: true
+
+  build-and-publish:
+    name: Build and Publish to PyPI
+    needs: test
+    runs-on: ubuntu-latest
+    if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/v'))
+    
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+      
+      - name: Install build dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build twine
+      
+      - name: Install package dependencies
+        run: |
+          pip install -r requirements-dev.txt
+      
+      - name: Run tests before build
+        run: |
+          pytest tests/ -v
+      
+      - name: Build package
+        run: |
+          python -m build
+      
+      - name: Check package
+        run: |
+          twine check dist/*
+      
+      - name: Publish to PyPI
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: |
+          twine upload dist/*
+        if: success()
+
+  publish-testpypi:
+    name: Publish to TestPyPI
+    needs: test
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+    
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+      
+      - name: Install build dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build twine
+      
+      - name: Run tests before build
+        run: |
+          pip install -r requirements-dev.txt
+          pytest tests/ -v
+      
+      - name: Build package
+        run: |
+          python -m build
+      
+      - name: Publish to TestPyPI
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
+        run: |
+          twine upload --repository testpypi dist/*
+        continue-on-error: true

lopace-0.1.5/.github/workflows/test-only.yml

@@ -0,0 +1,45 @@
+name: Test Only
+
+on:
+  push:
+    branches-ignore:
+      - main
+      - master
+  pull_request:
+    branches:
+      - main
+      - master
+
+jobs:
+  test:
+    name: Run Tests
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python-version: ['3.9', '3.11', '3.12']
+    
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+      
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+          pip install -e .
+      
+      - name: Run tests with pytest
+        run: |
+          pytest tests/ -v --cov=lopace --cov-report=term
+      
+      - name: Run linting
+        run: |
+          flake8 lopace tests --count --select=E9,F63,F7,F82 --show-source --statistics
+        continue-on-error: true

lopace-0.1.5/.gitignore

@@ -0,0 +1,79 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+scripts/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Virtual environments
+venv/
+ENV/
+env/
+.venv
+
+# IDEs
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Project specific
+*.log
+.pytest_cache/
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# setuptools-scm generated (version from git tags)
+lopace/_version.py
+
+# Generated visualizations (keep directory structure but not generated files)
+# screenshots/*.svg
+# screenshots/*.csv
+!screenshots/.gitkeep

lopace-0.1.5/CHANGELOG.md

@@ -0,0 +1,87 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- Initial project structure
+- GitHub Actions CI/CD workflow
+- Contributing guidelines
+- Code of Conduct
+
+## [0.1.0] - 2024-12-XX
+
+### Added
+- **Core Features**:
+  - Zstd compression method
+  - Token-based (BPE) compression method
+  - Hybrid compression method (Token + Zstd)
+  - Lossless compression/decompression
+
+- **Compression Methods**:
+  - `compress_zstd()` / `decompress_zstd()` - Zstandard compression
+  - `compress_token()` / `decompress_token()` - BPE tokenization with binary packing
+  - `compress_hybrid()` / `decompress_hybrid()` - Combined tokenization and Zstd
+  - Generic `compress()` / `decompress()` methods
+
+- **Evaluation Metrics**:
+  - Compression Ratio (CR)
+  - Space Savings (SS)
+  - Bits Per Character (BPC)
+  - Throughput (MB/s)
+  - SHA-256 hash verification
+  - Exact match verification
+  - Reconstruction error calculation
+  - Shannon Entropy calculation
+  - Theoretical compression limits
+
+- **API**:
+  - `PromptCompressor` class with configurable tokenizer and Zstd level
+  - `CompressionMethod` enum for method selection
+  - `compress_and_return_both()` method
+  - `get_compression_stats()` method
+  - `calculate_shannon_entropy()` method
+  - `get_theoretical_compression_limit()` method
+
+- **Streamlit Web App**:
+  - Interactive compression interface
+  - Real-time metrics calculation
+  - Side-by-side method comparison
+  - Comprehensive evaluation dashboard
+
+- **Documentation**:
+  - Comprehensive README with usage examples
+  - API reference documentation
+  - Mathematical background explanation
+  - Installation instructions
+
+- **Testing**:
+  - Complete test suite with pytest
+  - Test coverage for all compression methods
+  - Edge case testing
+  - Lossless verification tests
+
+- **DevOps**:
+  - GitHub Actions CI/CD pipeline
+  - Automated testing on multiple Python versions
+  - Automated PyPI publishing
+
+### Technical Details
+
+- **Supported Tokenizers**: cl100k_base, p50k_base, r50k_base, gpt2
+- **Zstd Levels**: 1-22 (default: 15)
+- **Python Versions**: 3.8, 3.9, 3.10, 3.11, 3.12
+- **Dependencies**: zstandard>=0.22.0, tiktoken>=0.5.0
+- **Smart Format Detection**: Automatically uses uint16 or uint32 based on token ID ranges
+
+### Fixed
+- Token ID overflow handling for tokenizers with vocab > 65535
+- Format byte handling for backward compatibility
+- Error handling for edge cases
+
+[Unreleased]: https://github.com/amanulla/lopace/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/amanulla/lopace/releases/tag/v0.1.0