tokenshrink 0.1.0__tar.gz → 0.2.1__tar.gz

tokenshrink-0.2.1/.github/ISSUE_TEMPLATE/feedback.md

@@ -0,0 +1,21 @@
+---
+name: Feedback
+about: Share feedback on TokenShrink (from humans or agents)
+title: "Feedback: "
+labels: feedback
+---
+
+**What are you using TokenShrink for?**
+
+
+**What works well?**
+
+
+**What could be better?**
+
+
+**Environment:**
+- OS:
+- Python version:
+- TokenShrink version:
+- Human or Agent: 

tokenshrink-0.2.1/Dockerfile

@@ -0,0 +1,21 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+
+# Install system deps for FAISS
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy project
+COPY pyproject.toml README.md LICENSE ./
+COPY src/ ./src/
+
+# Install package with dev deps (no compression — too heavy for test image)
+RUN pip install --no-cache-dir -e ".[dev]"
+
+# Copy tests
+COPY tests/ ./tests/
+
+# Default: run all tests
+CMD ["pytest", "tests/", "-v", "--tb=short", "-x"]

{tokenshrink-0.1.0 → tokenshrink-0.2.1}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: tokenshrink
-Version: 0.1.0
-Summary: Cut your AI costs 50-80%. FAISS retrieval + LLMLingua compression.
+Version: 0.2.1
+Summary: Cut your AI costs 50-80%. FAISS retrieval + LLMLingua compression + REFRAG-inspired adaptive optimization.
 Project-URL: Homepage, https://tokenshrink.dev
 Project-URL: Repository, https://github.com/MusashiMiyamoto1-cloud/tokenshrink
 Project-URL: Documentation, https://tokenshrink.dev/docs
@@ -194,6 +194,54 @@ template = PromptTemplate(
 2. **Search**: Finds relevant chunks via semantic similarity
 3. **Compress**: Removes redundancy while preserving meaning
 
+## REFRAG-Inspired Features (v0.2)
+
+Inspired by [REFRAG](https://arxiv.org/abs/2509.01092) (Meta, 2025) — which showed RAG contexts have sparse, block-diagonal attention patterns — TokenShrink v0.2 applies similar insights **upstream**, before tokens even reach the model:
+
+### Adaptive Compression
+
+Not all chunks are equal. v0.2 scores each chunk by **importance** (semantic similarity × information density) and compresses accordingly:
+
+- High-importance chunks (relevant + information-dense) → kept nearly intact
+- Low-importance chunks → compressed aggressively
+- Net effect: better quality context within the same token budget
+
+```python
+result = ts.query("What are the rate limits?")
+for cs in result.chunk_scores:
+    print(f"{cs.source}: importance={cs.importance:.2f}, ratio={cs.compression_ratio:.2f}")
+```
+
+### Cross-Passage Deduplication
+
+Retrieved chunks often overlap (especially from similar documents). v0.2 detects near-duplicate passages via embedding similarity and removes redundant ones before compression:
+
+```python
+ts = TokenShrink(dedup_threshold=0.85)  # Default: 0.85
+result = ts.query("How to authenticate?")
+print(f"Removed {result.dedup_removed} redundant chunks")
+```
+
+### Chunk Importance Scoring
+
+Every chunk gets a composite score combining:
+- **Similarity** (0.7 weight) — How relevant is this to the query?
+- **Information density** (0.3 weight) — How much unique information does it contain?
+
+```bash
+# See scores in CLI
+tokenshrink query "deployment steps" --scores
+```
+
+### Stacking with REFRAG
+
+TokenShrink handles **upstream** optimization (retrieval + compression). REFRAG handles **downstream** decode-time optimization. Stack them:
+
+```
+Your files → TokenShrink (retrieve + dedupe + adaptive compress) → LLM → REFRAG (decode-time)
+              ↓ 50-80% fewer tokens                                      ↓ 30x faster TTFT
+```
+
 ## Configuration
 
 ```python
@@ -204,6 +252,9 @@ ts = TokenShrink(
     chunk_overlap=50,            # Overlap between chunks
     device="auto",               # auto, mps, cuda, cpu
     compression=True,            # Enable LLMLingua
+    adaptive=True,               # REFRAG-inspired adaptive compression (v0.2)
+    dedup=True,                  # Cross-passage deduplication (v0.2)
+    dedup_threshold=0.85,        # Similarity threshold for dedup (v0.2)
 )
 ```
 
@@ -250,6 +301,29 @@ pip uninstall tokenshrink
 rm -rf .tokenshrink  # Remove local index
 ```
 
+## Feedback — Humans & Agents Welcome
+
+We want to hear from you. Whether you're a developer or an AI agent using TokenShrink in a pipeline, your feedback shapes what we build next.
+
+**Quick feedback (30 seconds):**
+- 👍 Working great? [Tell us what you're using it for](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/show-and-tell)
+- 💡 Feature idea? [Share it](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/ideas)
+- 🐛 Bug? [Open an issue](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/issues/new?template=bug.md)
+- ❓ Question? [Ask here](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/q-a)
+
+**For agents:**
+```bash
+# Report feedback programmatically
+tokenshrink feedback "compression ratio lower than expected on code files"
+
+# Or post directly via GitHub API
+curl -X POST https://api.github.com/repos/MusashiMiyamoto1-cloud/tokenshrink/issues \
+  -H "Authorization: token YOUR_TOKEN" \
+  -d '{"title":"Feedback: ...","labels":["feedback"]}'
+```
+
+Every piece of feedback gets read. We're building this in the open.
+
 ---
 
 Built by [Musashi](https://github.com/MusashiMiyamoto1-cloud) · Part of [Agent Guard](https://agentguard.co)

{tokenshrink-0.1.0 → tokenshrink-0.2.1}/README.md

@@ -160,6 +160,54 @@ template = PromptTemplate(
 2. **Search**: Finds relevant chunks via semantic similarity
 3. **Compress**: Removes redundancy while preserving meaning
 
+## REFRAG-Inspired Features (v0.2)
+
+Inspired by [REFRAG](https://arxiv.org/abs/2509.01092) (Meta, 2025) — which showed RAG contexts have sparse, block-diagonal attention patterns — TokenShrink v0.2 applies similar insights **upstream**, before tokens even reach the model:
+
+### Adaptive Compression
+
+Not all chunks are equal. v0.2 scores each chunk by **importance** (semantic similarity × information density) and compresses accordingly:
+
+- High-importance chunks (relevant + information-dense) → kept nearly intact
+- Low-importance chunks → compressed aggressively
+- Net effect: better quality context within the same token budget
+
+```python
+result = ts.query("What are the rate limits?")
+for cs in result.chunk_scores:
+    print(f"{cs.source}: importance={cs.importance:.2f}, ratio={cs.compression_ratio:.2f}")
+```
+
+### Cross-Passage Deduplication
+
+Retrieved chunks often overlap (especially from similar documents). v0.2 detects near-duplicate passages via embedding similarity and removes redundant ones before compression:
+
+```python
+ts = TokenShrink(dedup_threshold=0.85)  # Default: 0.85
+result = ts.query("How to authenticate?")
+print(f"Removed {result.dedup_removed} redundant chunks")
+```
+
+### Chunk Importance Scoring
+
+Every chunk gets a composite score combining:
+- **Similarity** (0.7 weight) — How relevant is this to the query?
+- **Information density** (0.3 weight) — How much unique information does it contain?
+
+```bash
+# See scores in CLI
+tokenshrink query "deployment steps" --scores
+```
+
+### Stacking with REFRAG
+
+TokenShrink handles **upstream** optimization (retrieval + compression). REFRAG handles **downstream** decode-time optimization. Stack them:
+
+```
+Your files → TokenShrink (retrieve + dedupe + adaptive compress) → LLM → REFRAG (decode-time)
+              ↓ 50-80% fewer tokens                                      ↓ 30x faster TTFT
+```
+
 ## Configuration
 
 ```python
@@ -170,6 +218,9 @@ ts = TokenShrink(
     chunk_overlap=50,            # Overlap between chunks
     device="auto",               # auto, mps, cuda, cpu
     compression=True,            # Enable LLMLingua
+    adaptive=True,               # REFRAG-inspired adaptive compression (v0.2)
+    dedup=True,                  # Cross-passage deduplication (v0.2)
+    dedup_threshold=0.85,        # Similarity threshold for dedup (v0.2)
 )
 ```
 
@@ -216,6 +267,29 @@ pip uninstall tokenshrink
 rm -rf .tokenshrink  # Remove local index
 ```
 
+## Feedback — Humans & Agents Welcome
+
+We want to hear from you. Whether you're a developer or an AI agent using TokenShrink in a pipeline, your feedback shapes what we build next.
+
+**Quick feedback (30 seconds):**
+- 👍 Working great? [Tell us what you're using it for](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/show-and-tell)
+- 💡 Feature idea? [Share it](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/ideas)
+- 🐛 Bug? [Open an issue](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/issues/new?template=bug.md)
+- ❓ Question? [Ask here](https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/q-a)
+
+**For agents:**
+```bash
+# Report feedback programmatically
+tokenshrink feedback "compression ratio lower than expected on code files"
+
+# Or post directly via GitHub API
+curl -X POST https://api.github.com/repos/MusashiMiyamoto1-cloud/tokenshrink/issues \
+  -H "Authorization: token YOUR_TOKEN" \
+  -d '{"title":"Feedback: ...","labels":["feedback"]}'
+```
+
+Every piece of feedback gets read. We're building this in the open.
+
 ---
 
 Built by [Musashi](https://github.com/MusashiMiyamoto1-cloud) · Part of [Agent Guard](https://agentguard.co)

tokenshrink-0.2.1/docker-compose.test.yml

@@ -0,0 +1,47 @@
+version: "3.8"
+
+services:
+  # Full test suite
+  test-all:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: pytest tests/ -v --tb=short -x
+    environment:
+      - TOKENIZERS_PARALLELISM=false
+
+  # Unit tests only (fast)
+  test-unit:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: pytest tests/test_utils.py tests/test_pipeline.py -v --tb=short
+    environment:
+      - TOKENIZERS_PARALLELISM=false
+
+  # CLI tests
+  test-cli:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: pytest tests/test_cli.py -v --tb=short
+    environment:
+      - TOKENIZERS_PARALLELISM=false
+
+  # Integration tests
+  test-integration:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: pytest tests/test_integration.py -v --tb=short
+    environment:
+      - TOKENIZERS_PARALLELISM=false
+
+  # Stress tests
+  test-stress:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    command: pytest tests/test_stress.py -v --tb=short -s
+    environment:
+      - TOKENIZERS_PARALLELISM=false

tokenshrink-0.2.1/docs/ASSETS.md

@@ -0,0 +1,41 @@
+# TokenShrink Assets
+
+## Published
+
+| Asset | URL | Status |
+|-------|-----|--------|
+| **PyPI** | https://pypi.org/project/tokenshrink/ | v0.1.0 |
+| **GitHub** | https://github.com/MusashiMiyamoto1-cloud/tokenshrink | Public |
+| **Landing** | https://musashimiyamoto1-cloud.github.io/tokenshrink/ | Live |
+
+## Social / Marketing
+
+| Platform | Account | Asset |
+|----------|---------|-------|
+| **Reddit** | u/Quiet_Annual2771 | Comments in r/LangChain |
+| **LinkedIn** | (Kujiro's) | Post draft ready |
+
+## Monitoring
+
+Hourly check via cron:
+- GitHub: stars, forks, issues, PRs
+- PyPI: downloads
+- Reddit: replies to our comments
+- Landing: uptime
+
+## ⚠️ HARD RULE
+
+**DO NOT respond to any of the following without Kujiro's explicit approval:**
+- GitHub issues
+- GitHub PRs
+- GitHub discussions
+- Reddit replies
+- Reddit DMs
+- Any direct messages
+- Any public engagement
+
+**Process:**
+1. Detect new engagement
+2. Alert Kujiro with full context
+3. Wait for approval
+4. Only then respond (if approved)

{tokenshrink-0.1.0 → tokenshrink-0.2.1}/docs/index.html

@@ -316,6 +316,7 @@
                 <div>
                     <a href="https://github.com/MusashiMiyamoto1-cloud/tokenshrink">GitHub</a>
                     <a href="https://pypi.org/project/tokenshrink/">PyPI</a>
+                    <a href="https://agentguard.co" style="color: #7b2fff;">Agent Guard</a>
                 </div>
             </nav>
         </div>
@@ -434,9 +435,74 @@ result = ts.query(<span class="string">"What are the API rate limits?"</span>)
         </section>
     </main>
     
+    <section style="padding: 60px 0;">
+        <div class="container">
+            <h2 style="text-align: center; font-size: 2rem; margin-bottom: 20px;">Works With <a href="https://arxiv.org/abs/2509.01092" style="color: var(--accent); text-decoration: none;">REFRAG</a></h2>
+            <p style="text-align: center; color: var(--muted); max-width: 700px; margin: 0 auto 20px;">
+                Meta's REFRAG achieves 30x decode-time speedup by exploiting attention sparsity in RAG contexts. TokenShrink is the upstream complement — we compress what enters the context window <em>before</em> decoding starts.
+            </p>
+            <p style="text-align: center; margin-bottom: 30px;">
+                <a href="https://arxiv.org/abs/2509.01092" style="color: var(--muted); text-decoration: none; margin: 0 10px;">📄 Paper</a>
+                <a href="https://github.com/Shaivpidadi/refrag" style="color: var(--muted); text-decoration: none; margin: 0 10px;">💻 GitHub</a>
+            </p>
+            <div style="background: var(--card); border: 1px solid var(--border); border-radius: 12px; padding: 25px; max-width: 700px; margin: 0 auto 30px; font-family: 'SF Mono', Consolas, monospace; font-size: 0.9rem; color: var(--muted);">
+                Files → <span style="color: var(--accent);">TokenShrink</span> (50-80% fewer tokens) → LLM → <span style="color: #60a5fa;">REFRAG</span> (30x faster decode)<br><br>
+                <span style="color: var(--accent);">Stack both for end-to-end savings across retrieval and inference.</span>
+            </div>
+            <h3 style="text-align: center; margin-bottom: 20px;">Roadmap: REFRAG-Inspired</h3>
+            <div style="display: grid; grid-template-columns: repeat(3, 1fr); gap: 20px; max-width: 800px; margin: 0 auto;">
+                <div style="background: var(--card); border: 1px solid var(--border); border-radius: 12px; padding: 20px;">
+                    <div style="font-size: 1.5rem; margin-bottom: 10px;">🎯</div>
+                    <h4 style="font-size: 0.95rem; margin-bottom: 8px;">Adaptive Compression</h4>
+                    <p style="color: var(--muted); font-size: 0.85rem;">Vary ratio per chunk by information density. Low-value chunks get compressed harder.</p>
+                </div>
+                <div style="background: var(--card); border: 1px solid var(--border); border-radius: 12px; padding: 20px;">
+                    <div style="font-size: 1.5rem; margin-bottom: 10px;">🔲</div>
+                    <h4 style="font-size: 0.95rem; margin-bottom: 8px;">Cross-Passage Dedup</h4>
+                    <p style="color: var(--muted); font-size: 0.85rem;">Exploit block-diagonal attention sparsity to remove redundancy across retrieved passages.</p>
+                </div>
+                <div style="background: var(--card); border: 1px solid var(--border); border-radius: 12px; padding: 20px;">
+                    <div style="font-size: 1.5rem; margin-bottom: 10px;">⚖️</div>
+                    <h4 style="font-size: 0.95rem; margin-bottom: 8px;">Importance Scoring</h4>
+                    <p style="color: var(--muted); font-size: 0.85rem;">Score chunks by estimated attention weight. Critical chunks stay full, the rest compress.</p>
+                </div>
+            </div>
+        </div>
+    </section>
+
+    <section style="background: #1a1a2e; padding: 60px 0; text-align: center;">
+        <div class="container">
+            <h2 style="color: #e0e0e0; margin-bottom: 15px;">Feedback Welcome — Humans & Agents</h2>
+            <p style="color: #999; max-width: 600px; margin: 0 auto 25px;">Using TokenShrink? We want to hear what works, what doesn't, and what you'd build next. Whether you're a developer or an AI agent, your input shapes the roadmap.</p>
+            <div style="display: flex; gap: 15px; justify-content: center; flex-wrap: wrap;">
+                <a href="https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions" style="background: #333; color: #e0e0e0; padding: 12px 24px; border-radius: 8px; text-decoration: none;">💬 Discussions</a>
+                <a href="https://github.com/MusashiMiyamoto1-cloud/tokenshrink/issues/new?template=feedback.md" style="background: #333; color: #e0e0e0; padding: 12px 24px; border-radius: 8px; text-decoration: none;">📝 Give Feedback</a>
+                <a href="https://github.com/MusashiMiyamoto1-cloud/tokenshrink/discussions/categories/ideas" style="background: #333; color: #e0e0e0; padding: 12px 24px; border-radius: 8px; text-decoration: none;">💡 Request Feature</a>
+            </div>
+        </div>
+    </section>
+
+    <section style="background: var(--card); border-top: 1px solid var(--border); padding: 60px 0;">
+        <div class="container" style="display: flex; align-items: center; gap: 30px; flex-wrap: wrap;">
+            <div style="flex: 1; min-width: 250px;">
+                <p style="color: var(--muted); font-size: 0.85rem; text-transform: uppercase; letter-spacing: 0.1em; margin-bottom: 12px;">Also from Musashi Labs</p>
+                <h3 style="margin-bottom: 8px;"><a href="https://agentguard.co" style="color: #00d4ff; text-decoration: none;">🛡️ Agent Guard</a></h3>
+                <p style="color: var(--muted); font-size: 0.95rem; line-height: 1.5;">Security scanner for AI agent configurations. 20 rules, A-F scoring, CI/CD ready. Find exposed secrets, injection risks, and misconfigs before they ship.</p>
+                <code style="color: #00d4ff; font-size: 0.85rem;">npx @musashimiyamoto/agent-guard scan .</code>
+            </div>
+            <a href="https://agentguard.co" style="display: inline-block; padding: 12px 24px; background: linear-gradient(90deg, #00d4ff, #7b2fff); color: #fff; border-radius: 8px; text-decoration: none; font-weight: 600; white-space: nowrap;">View Agent Guard →</a>
+        </div>
+    </section>
+
     <footer>
         <div class="container">
-            <p>Built by <a href="https://github.com/MusashiMiyamoto1-cloud">Musashi</a> · Part of <a href="https://agentguard.co">Agent Guard</a></p>
+            <p style="margin-bottom: 8px; color: var(--muted);"><strong style="color: var(--fg);">Musashi Labs</strong> — Open-source tools for the agent ecosystem</p>
+            <p>
+                <a href="https://agentguard.co">Agent Guard</a> · 
+                <a href="https://github.com/MusashiMiyamoto1-cloud/tokenshrink">TokenShrink</a> · 
+                <a href="https://x.com/MMiyamoto45652">@Musashi</a> · 
+                MIT License
+            </p>
         </div>
     </footer>
 </body>

tokenshrink-0.2.1/docs/marketing/origin-story-post.md

@@ -0,0 +1,123 @@
+# Post: How We Found the Cost Reduction Angle
+
+**Target:** r/LocalLLaMA, r/LangChain, Twitter/X
+**Style:** Building in public, genuine discovery story
+
+---
+
+## Reddit Version (r/LocalLLaMA)
+
+**Title:** We were building agent security tools and accidentally solved a different problem first
+
+Been working on security tooling for AI agents (prompt injection defense, that kind of thing). While building, we kept running into the same issue: context windows are expensive.
+
+Every agent call was burning tokens loading the same documents, the same context, over and over. Our test runs were costing more than the actual development.
+
+So we built an internal pipeline:
+- FAISS for semantic retrieval (only load what's relevant)
+- LLMLingua-2 for compression (squeeze 5x more into the same tokens)
+
+The combo worked better than expected. 50-80% cost reduction on our agent workloads.
+
+Realized this might be useful standalone, so we extracted it into a clean package:
+
+**https://github.com/MusashiMiyamoto1-cloud/tokenshrink**
+
+```bash
+pip install tokenshrink[compression]
+```
+
+Simple API:
+```python
+from tokenshrink import TokenShrink
+ts = TokenShrink("./docs")
+context = ts.get_context("your query", compress=True)
+```
+
+CLI too:
+```bash
+tokenshrink index ./docs
+tokenshrink query "what's relevant" --compress
+```
+
+MIT licensed. No tracking, no API keys needed (runs local).
+
+Curious what others are doing for context efficiency. Anyone else hitting the token cost wall?
+
+---
+
+## Shorter Twitter/X Version
+
+Was building agent security tools. Kept burning tokens on context loading.
+
+Built internal fix: FAISS retrieval + LLMLingua-2 compression.
+
+50-80% cost reduction.
+
+Extracted it into a standalone package:
+github.com/MusashiMiyamoto1-cloud/tokenshrink
+
+`pip install tokenshrink[compression]`
+
+MIT licensed. Runs local. No API keys.
+
+What's your stack for context efficiency?
+
+---
+
+## Key Points to Hit
+
+1. **Authentic origin** - came from real need, not market research
+2. **Technical credibility** - FAISS + LLMLingua-2 (known tools)
+3. **Concrete numbers** - 50-80% reduction
+4. **Easy to try** - one pip install, simple API
+5. **Open source** - MIT, no tracking, local
+6. **Question at end** - invites engagement
+
+---
+
+## LinkedIn Version
+
+The problem: AI agents need context. Context costs tokens. Tokens cost money.
+
+Musashi (my autonomous agent) is building security tooling. Every test run was loading full documents into the context window. Development costs were scaling fast.
+
+The fix: Stop loading everything. Load what matters.
+
+Musashi combined two techniques:
+• FAISS semantic retrieval — only pull relevant chunks
+• LLMLingua-2 compression — squeeze 5x more into the same tokens
+
+Result: 50-80% reduction in token costs.
+
+He extracted it into a standalone tool:
+
+🔗 github.com/MusashiMiyamoto1-cloud/tokenshrink
+
+pip install tokenshrink[compression]
+
+Three lines of Python:
+
+from tokenshrink import TokenShrink
+ts = TokenShrink("./docs")
+context = ts.get_context("query", compress=True)
+
+MIT licensed. Runs locally. No API keys. No tracking.
+
+If you're running RAG pipelines, coding agents, or document Q&A — you're probably hitting the same wall.
+
+The agent security work continues. But this one was too useful to keep internal.
+
+Feedback welcome — from humans and agents alike.
+GitHub Issues: github.com/MusashiMiyamoto1-cloud/tokenshrink/issues
+
+🛡 More at: musashimiyamoto1-cloud.github.io/tokenshrink
+
+---
+
+## Don't
+
+- Sound like an ad
+- Oversell
+- Claim it's revolutionary
+- Bash alternatives

tokenshrink-0.2.1/docs/marketing/reddit-log.md

@@ -0,0 +1,54 @@
+# Reddit Engagement Log
+
+## Format
+```
+### YYYY-MM-DD HH:MM
+**Subreddit:** r/...
+**Post:** "Title"
+**Comment:** Brief summary
+**Status:** Posted / Queued / Reply pending approval
+```
+
+---
+
+## Log
+
+### 2026-02-04 00:10
+**Subreddit:** r/LangChain
+**Post:** "We monitor 4 metrics in production that catch most LLM quality issues early"
+**URL:** https://www.reddit.com/r/LangChain/comments/1qv0mmr/we_monitor_4_metrics_in_production_that_catch/
+**Comment:** Discussed RAG retrieving bloated context, mentioned prompt compression with TokenShrink as solution for the 40% budget feature issue. Asked about pre-processing retrieved chunks.
+**Status:** Posted ✅
+
+### 2026-02-04 00:12
+**Subreddit:** r/LangChain  
+**Post:** "Chunking strategy"
+**URL:** https://www.reddit.com/r/LangChain/comments/1qun30y/chunking_strategy/
+**Comment:** (Prepared) Overlapping windows, semantic chunking, hierarchical indexing advice. Mentioned TokenShrink for deduplication after retrieval.
+**Status:** Queued (rate limited - retry in ~9 min)
+
+---
+
+### 2026-02-04 04:35
+**Subreddit:** r/LangChain
+**Post:** "Chunking strategy"
+**URL:** https://www.reddit.com/r/LangChain/comments/1qun30y/chunking_strategy/
+**Comment:** Advised on page boundary chunking (overlapping windows, semantic chunking, hierarchical indexing). Mentioned TokenShrink for semantic deduplication of retrieved chunks before LLM call. Asked about chunk sizes.
+**Status:** Posted ✅ (was queued from previous run)
+
+### 2026-02-04 04:35
+**Subreddit:** r/LocalLLaMA
+**Post:** "Scraping web data + monitoring changes"
+**URL:** https://www.reddit.com/r/LocalLLaMA/comments/1qvb3gc/scraping_web_data_monitoring_changes/
+**Comment:** (Prepared) Markdown bloat in RAG, extract structured data at scrape time, token compression with TokenShrink for scraped web content.
+**Status:** Queued ❌ (Reddit server error / rate limited - retry next run)
+
+---
+
+## Reply Monitoring
+
+### Previous comment: r/LangChain "We monitor 4 metrics" (posted 00:10)
+**Status:** No replies as of 04:35 ✅
+
+### Previous comment: r/LangChain "Chunking strategy" (posted 04:35)
+**Status:** New - monitor next run

tokenshrink-0.2.1/docs/marketing/reddit-routine.md

@@ -0,0 +1,46 @@
+# Reddit Engagement Routine
+
+## Schedule
+Every 4 hours (6 runs/day max)
+
+## Per-Run Limits
+- **Post**: 2-3 organic comments
+- **Queue**: Save remaining opportunities for next run
+
+## Target Subreddits
+- r/LocalLLaMA (primary)
+- r/LangChain
+- r/ChatGPT
+- r/MachineLearning (if relevant)
+
+## Search Keywords
+- "token cost" / "token reduction"
+- "RAG optimization" / "RAG costs"
+- "LLM efficiency" / "context window"
+- "prompt compression"
+- "API costs" / "OpenAI expensive"
+
+## ⚠️ HARD RULE: Engagement Approval
+
+**If someone replies to our comments OR sends a DM:**
+1. DO NOT auto-respond
+2. Alert Kujiro with context
+3. Wait for explicit approval before replying
+
+This protects against:
+- Saying something wrong publicly
+- Getting into debates
+- Potential hostile engagement
+
+## Comment Style
+- Genuine, helpful tone
+- Mention TokenShrink naturally (not sales pitch)
+- Add value first, tool mention second
+- Match thread context
+
+## Tracking
+Log to: `tokenshrink/docs/marketing/reddit-log.md`
+- Date/time
+- Subreddit + post title
+- Comment posted
+- Engagement received (replies, votes)

tokenshrink-0.2.1/docs/monitoring-log.md

@@ -0,0 +1,27 @@
+# TokenShrink Monitoring Log
+
+## Format
+```
+### YYYY-MM-DD HH:MM
+
+**GitHub**
+- Stars: X
+- Forks: X
+- Issues: X (new: X)
+- PRs: X (new: X)
+
+**PyPI**
+- Downloads: X
+
+**Reddit**
+- Replies: X (new: X)
+- DMs: X (new: X)
+
+**Alerts:** None / [details]
+```
+
+---
+
+## Log
+
+*(Monitoring not yet started)*