rust-cave-001 0.2.1__tar.gz

rust_cave_001-0.2.1/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,24 @@
+## Bug Report
+
+**Description**
+A clear description of the bug.
+
+**Steps to Reproduce**
+1.
+2.
+3.
+
+**Expected Behavior**
+What you expected to happen.
+
+**Actual Behavior**
+What actually happened.
+
+**Environment**
+- OS:
+- Python version:
+- Rust version:
+- rust-cave-001 version:
+
+**Additional Context**
+Any other relevant information.

rust_cave_001-0.2.1/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,13 @@
+## Feature Request
+
+**Problem / Use Case**
+What problem are you trying to solve?
+
+**Proposed Solution**
+Describe the feature you would like.
+
+**Alternatives Considered**
+Any alternative approaches you considered.
+
+**Additional Context**
+Any other relevant information or examples.

rust_cave_001-0.2.1/.github/workflows/ci.yml

@@ -0,0 +1,90 @@
+name: CI
+
+on:
+  push:
+    branches: [master, main]
+  pull_request:
+    branches: [master, main]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt, clippy
+
+      - name: Cache cargo registry
+        uses: actions/cache@v4
+        with:
+          path: ~/.cargo/registry
+          key: cargo-registry-${{ hashFiles('Cargo.lock') }}
+          restore-keys: |
+            cargo-registry-
+
+      - name: Cache pip
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: pip-${{ hashFiles('pyproject.toml') }}
+          restore-keys: |
+            pip-
+
+      - name: Install Python dependencies
+        run: |
+          sudo apt-get update -qq && sudo apt-get install -y -qq python3-dev
+          pip install --quiet maturin pytest
+
+      - name: Check Rust formatting
+        run: cargo fmt --check
+
+      - name: Check Rust clippy
+        run: cargo clippy -- -D warnings
+
+      - name: Rust tests
+        run: cargo test
+
+      - name: Build and install
+        env:
+          PYO3_PYTHON: python3.10
+          PYTHON: python3.10
+        run: maturin build --release --auditwheel skip -o dist/
+
+      - name: Install built wheel
+        run: pip install dist/*.whl --force-reinstall --no-deps
+
+      - name: Run Python tests
+        run: pytest tests/ -v
+
+  audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Cache cargo registry
+        uses: actions/cache@v4
+        with:
+          path: ~/.cargo/registry
+          key: cargo-registry-${{ hashFiles('Cargo.lock') }}
+          restore-keys: |
+            cargo-registry-
+
+      - name: Install cargo-audit
+        run: cargo install cargo-audit --locked
+
+      - name: Run cargo audit
+        run: cargo audit

rust_cave_001-0.2.1/.gitignore

@@ -0,0 +1,55 @@
+# Rust
+/target/
+**/*.rs.bk
+*.pdb
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# PyO3
+pyo3-0.*/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Testing
+.pytest_cache/
+/test_remove_articles/
+
+# Local config
+.env
+.env.local
+
+# Internal docs
+PUBLIC_READINESS_PLAN.md
+
+# Local Cargo build config (target-dir on USB)
+.cargo/

rust_cave_001-0.2.1/ATTRIBUTION.md

@@ -0,0 +1,42 @@
+# Attribution
+
+## Upstream Sources
+
+### Caveman Compression SPEC & Original Implementation
+
+- **Repository:** [wilpel/caveman-compression](https://github.com/wilpel/caveman-compression)
+- **License:** MIT
+- **Role:** Defines the Caveman Compression specification (SPEC.md) that rust-cave-001 implements. The 9 compression rules (sentence atomicity, word limit, connective elimination, active voice & present tense, preserve specifics, intensifier removal, article omission, pronoun handling, logical completeness) originate from this specification.
+- **Files derived from SPEC:**
+  - `src/lib.rs` — all 7 compression rules (adapted from SPEC rules 1-9)
+  - `README.md` — "The 7 Compression Rules" section
+  - `BENCHMARKS.md` — methodology inspired by wilpel's benchmarks
+
+### Token Reduction Concept
+
+- **Repository:** [JuliusBrussee/caveman](https://github.com/JuliusBrussee/caveman)
+- **License:** MIT
+- **Role:** Market validation for output-side token compression (~37K GitHub stars). Proves massive demand for token reduction in AI agent contexts. rust-cave-001 addresses the complementary input-side problem.
+
+### Hook-Based Compression Architecture (Reference)
+
+- **Repository:** [claudioemmanuel/squeez](https://github.com/claudioemmanuel/squeez)
+- **License:** MIT
+- **Role:** Reference for Rust-based MCP server architecture and cross-host hook integration patterns.
+
+## Third-Party Rust Crates
+
+| Crate | Version | License | Purpose |
+|-------|---------|---------|---------|
+| [pyo3](https://crates.io/crates/pyo3) | 0.24.2 | Apache-2.0 | Python bindings |
+| [lz4](https://crates.io/crates/lz4) | 1.0 | MIT | LZ4 block compression |
+| [regex](https://crates.io/crates/regex) | 1.10 | MIT/Apache-2.0 | Text pattern matching |
+
+## Build Tools
+
+- [Maturin](https://github.com/PyO3/maturin) — Python wheel builder for Rust projects
+- [PyO3](https://github.com/PyO3/pyo3) — Rust↔Python interoperability framework
+
+---
+
+*This project is MIT-licensed. All upstream works referenced above are also MIT-licensed where noted. Attribution is provided as a courtesy and in the spirit of open source collaboration.*

rust_cave_001-0.2.1/BENCHMARKS.md

@@ -0,0 +1,53 @@
+# Benchmarks
+
+Benchmark results for rust-cave-001 v0.2.1.
+
+```bash
+python3 benchmarks/benchmark.py
+```
+
+## NLP Compression (`compress()`)
+
+The Caveman Rules pipeline across 9 text types. Measured on aarch64 (Raspberry Pi 5).
+
+| Text Type | Chars | → | Compressed | Reduction | Tokens | → | Compressed | Reduction | Time |
+|---|---|---|---|---|---|---|---|---|---|
+| Technical (short) | 61 | → | 32 | 47.5% | 11 | → | 5 | **54.5%** | 2.79ms |
+| Technical (paragraph) | 369 | → | 147 | 60.2% | 54 | → | 22 | **59.3%** | 11.11ms |
+| Conversational | 260 | → | 78 | 70.0% | 43 | → | 14 | **67.4%** | 8.58ms |
+| Academic/Dense | 436 | → | 134 | 69.3% | 55 | → | 15 | **72.7%** | 8.37ms |
+| Dialogue/Chat | 196 | → | 35 | 82.1% | 39 | → | 9 | **76.9%** | 5.57ms |
+| Mixed (code+numbers) | 260 | → | 99 | 61.9% | 43 | → | 17 | **60.5%** | 8.32ms |
+| Already minimal | 21 | → | 21 | 0.0% | 4 | → | 4 | 0.0% | 2.68ms |
+| Repetitive | 179 | → | 107 | 40.2% | 36 | → | 20 | 44.4% | 11.04ms |
+| Short sentences | 41 | → | 41 | 0.0% | 9 | → | 9 | 0.0% | 8.03ms |
+
+**Summary:** Average token reduction **48.4%** (54.5% on texts with >4 tokens).
+Average call time **7.4ms** on RPi 5 (aarch64).
+
+## LZ4 Binary Compression (`my_compress`)
+
+| Data Type | Size (B) | Compressed (B) | Ratio | Saved | Time |
+|---|---|---|---|---|---|
+| Repeated text (2KB) | 1,560 | 32 | **48.75x** | 97.9% | 7.6µs |
+| Repeated text (10KB) | 7,800 | 57 | **136.84x** | 99.3% | 8.4µs |
+| Random bytes (1KB) | 1,024 | 1,033 | 0.99x | -0.9% | 13.7µs |
+| JSON-like (2KB) | 912 | 235 | 3.88x | 74.2% | 9.7µs |
+| UTF-8 text (1KB) | 1,350 | 65 | 20.77x | 95.2% | 7.5µs |
+| Small data (100B) | 100 | 15 | 6.67x | 85.0% | 7.2µs |
+
+## Combined Pipeline: `compress()` → `my_compress()`
+
+Applying NLP compression then LZ4 in sequence, measured end-to-end.
+
+| Text | Original | After NLP | After LZ4 | Total Ratio | Total Time |
+|---|---|---|---|---|---|
+| Technical (short) | 61B | 32B | 38B | 1.61x | 2.9ms |
+| Technical (paragraph) | 369B | 147B | 150B | 2.46x | 11.2ms |
+| Conversational | 262B | 80B | 86B | **3.05x** | 8.8ms |
+
+## Notes
+
+- LZ4 has block overhead (~15B per block), so small blocks may expand on first call but decompress correctly.
+- Random/incompressible data may have a tiny overhead from LZ4 block header.
+- Timings are on a Raspberry Pi 5 (aarch64, Cortex-A76). Expect ~2-4x faster on x86_64.

rust_cave_001-0.2.1/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.1] - 2026-05-15
+
+### Fixed
+- `normalize_present_tense` ed-stripping: off-by-one in vowel detection (included→includ, sorted→sorte)
+- Added 21 e-drop verb entries to map (agreed→agree, merged→merge, cached→cache, etc.)
+- Tightened eeed guard to allow "agreed" → "agree" while still blocking "speed" → "spe"
+- Added vowel-before-ded/ted check to prevent false e-drop on consonant patterns
+
+## [0.2.0] - 2026-05-15
+
+### Added
+- **Present tense normalization** — `normalize_present_tense()` (SPEC Rule 4) converts past-tense verbs to present tense using a reverse conjugation map of 100+ verbs. Un-dead-coded from `#[allow(dead_code)]`.
+- **Pronoun resolution** — `resolve_pronouns()` (SPEC Rule 8) replaces ambiguous pronouns (`it`, `they`, `them`) with preceding noun when 2+ candidates exist.
+- **ATTRIBUTION.md** — Properly credits wilpel/caveman-compression SPEC, JuliusBrussee/caveman, claudioemmanuel/squeez, and all third-party crate dependencies.
+- **`normalize_present_tense()` exposed** as a Python-callable function.
+
+### Changed
+- Pipeline now applies 9 rules (was 7): pronoun resolution → active voice → present tense → articles → intensifiers → connectives → word limit → completeness check.
+- Cargo.toml: v0.2.0, added authors and description.
+- pyproject.toml: v0.2.0.
+- README: Updated rules from 7 → 9, added SPEC attribution link.
+
+### Fixed
+- Benchmark sanity check: accepts both past and present tense passive voice output.
+
+## [0.1.1] - 2026-05-15
+
+### Added
+- Benchmark suite (benchmarks/benchmark.py) — measures NLP compression ratios, LZ4 binary compression, and combined pipeline performance across 9 text types
+- BENCHMARKS.md with detailed results and performance characteristics
+- Sample texts dataset (benchmarks/sample_texts.py) for reproducible benchmarking
+
+### Changed
+- README: Added benchmarks section, removed "No benchmark suite" known limitation
+
+## [0.1.0] - 2026-05-12
+
+### Added
+- Initial public release
+- LZ4 compression and decompression (`my_compress`, `decompress`)
+- Token estimation via word-boundary regex (`estimate_tokens`)
+- Compression statistics (`get_stats`)
+- Bincode serialization pipeline (`serialize_compressed`, `deserialize_compressed`)
+- Passive-to-active voice transformation (`preprocess_text`, `transform_active_voice`)
+- Full Caveman Compression pipeline (`compress`) applying 7 rules:
+  1. Sentence splitting
+  2. Word limit (2-5 words per sentence)
+  3. Connective elimination
+  4. Active voice transformation
+  5. Intensifier removal
+  6. Article removal
+  7. Logical completeness check
+- Verb conjugation map with 60+ irregular verbs
+- Python package via PyO3/Maturin
+- Pytest test suite (53 tests)
+- MIT License
+- CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md
+- GitHub Actions CI workflow
+- GitHub issue templates (bug report, feature request)
+
+### Fixed
+- Case-insensitive connective removal
+- "this" article removal
+- Trailing period stripping in active-voice agent position
+- Word-merging prevention after connective elimination

rust_cave_001-0.2.1/CODE_OF_CONDUCT.md

@@ -0,0 +1,33 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior:
+
+- The use of sexualized language or imagery and unwelcome sexual attention
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information without explicit permission
+- Other conduct which could reasonably be considered inappropriate
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project maintainers. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.
+
+[homepage]: https://www.contributor-covenant.org

rust_cave_001-0.2.1/CONTINUE_HERE.md

@@ -0,0 +1,61 @@
+# rust-cave-001 — Session Reference (May 16, 2026)
+
+## Resume Here
+
+Code audit complete and fixes applied (v0.2.1). **Next: build the self-learning framework** — strategy selector that picks compression rules per input type.
+
+### Quickstart
+```
+git clone https://github.com/ether-btc/rust-cave-001
+cd rust-cave-001
+source .venv/bin/activate
+maturin develop --release
+pytest tests/ -v        # 66 tests, all pass
+python3 benchmarks/benchmark.py
+```
+
+### Key References
+| What | Value |
+|------|-------|
+| Repo | `github.com/ether-btc/rust-cave-001` (master) |
+| Latest commit | `ce9088c` — "feat: add input text classifier for adaptive compression strategies" |
+| Tags | v0.1.0, v0.1.1, v0.2.0, [v0.2.1](https://github.com/ether-btc/rust-cave-001/releases/tag/v0.2.1) |
+| CI | Green — 66/66 tests, benchmarks 48.4% avg token reduction |
+| Current version | **v0.2.1** (deduplicated maps, clean docs, fresh wheel, input classifier built) |
+| Ceiling | 48.4% static — self-learning framework component (classifier) built, strategy selector next |
+### What Exists
+- **9 compression rules** in pipeline (`src/lib.rs`, 813 lines)
+- **Input classifier** (`src/classifier.rs`, 353 lines) — 13-heuristic text type detection
+- **Benchmark suite** (`benchmarks/benchmark.py`) — 9 text types, LZ4, combined pipeline
+- **66 passing tests** (`tests/test_rust_cave_001.py`) — includes direct `normalize_present_tense` tests (26 new)
+- **ATTRIBUTION.md** — credits upstream SPEC repos
+- **Verb maps**: 94 unique entries (transform_active_voice) + 147 unique entries (normalize_present_tense)
+
+### Latest Audit Findings Addressed (v0.2.1)
+- ✅ Deduplicated verb conjugation maps (removed 27 duplicate entries)
+- ✅ Bumped Cargo.toml/pyproject.toml v0.2.0 → v0.2.1
+- ✅ De-hermesified pyproject.toml description for public consumption
+- ✅ Fixed build.rs DEP_PYO3_ABI3 check (PY312 → PY310)
+- ✅ Updated BENCHMARKS.md version reference (v0.1.0 → v0.2.1)
+- ✅ Updated README.md verb count (~60 → ~100)
+- ✅ Added 26 direct tests for normalize_present_tense (irregular, regular, same-form, capitalization)
+- ✅ Removed dead code comment in preprocess_text
+- ✅ CI formatting fixed (`cargo fmt` pass on build.rs + lib.rs)
+
+### Latest Feature (v0.2.1+)
+- ✅ **Input text classifier** (`src/classifier.rs`) — detects technical, conversational, academic, dialogue, minimal, mixed text types using 13 heuristic dimensions. Exposed as Python-callable `classify_text()` and `recommended_strategy_for_text()`.
+
+### Next Steps (priority order)
+1. **Wire classifier into compress()** — use `recommended_strategy()` to select rule subset per input type
+2. **Auto-benchmark oracle** — measure which strategy wins for each text domain
+3. **Publish to PyPI** — needs API token from pypi.org/manage/account/token/
+4. **MCP server integration** — expose as MCP tool (reference: squeez architecture)
+5. **Add Python tests for classifier** — test classify_text() and recommended_strategy_for_text()
+6. **Expand verb map** — 94 → 150+ unique entries in transform_active_voice
+
+### Upstream Repos to Watch
+| Repo | Why | Absorb When |
+|------|-----|-------------|
+| wilpel/caveman-compression | Direct SPEC upstream, MLM/LLM modes | Framework phase |
+| JuliusBrussee/caveman (37K ⭐) | Market validation, /caveman-compress | Post-framework |
+| claudioemmanuel/squeez (~2K ⭐) | MCP server architecture, Rust hooks | Framework phase |

rust_cave_001-0.2.1/CONTRIBUTING.md

@@ -0,0 +1,56 @@
+# Contributing Guide
+
+Thank you for your interest in contributing to rust-cave-001! This document will help you get started.
+
+## Getting Started
+
+1. Fork the repository on GitHub
+2. Clone your fork:
+   ```bash
+   git clone https://github.com/YOUR_USERNAME/rust-cave-001.git
+   cd rust-cave-001
+   ```
+3. Set up a Python virtual environment (recommended):
+   ```bash
+   python3 -m venv .venv
+   source .venv/bin/activate
+   pip install maturin pytest
+   ```
+
+## Building
+
+```bash
+# Development build
+maturin develop
+
+# Release build
+maturin develop --release
+```
+
+## Running Tests
+
+```bash
+pytest tests/ -v
+```
+
+## Code Quality
+
+- Run `cargo fmt` and `cargo clippy` before committing
+- Ensure all tests pass before submitting a PR
+- Add tests for new features
+
+## Pull Request Process
+
+1. Create a feature branch: `git checkout -b feature/my-feature`
+2. Make your changes and add tests
+3. Run the full test suite
+4. Commit with a clear message
+5. Push and open a Pull Request
+
+## Code of Conduct
+
+Be respectful and constructive. We welcome contributors of all skill levels.
+
+## Questions?
+
+Open a GitHub Discussion or Issue. We are happy to help.