agent-os-kernel 1.1.0__py3-none-any.whl → 1.3.0__py3-none-any.whl

modules/caas/docs/RELATED_WORK.md

@@ -0,0 +1,312 @@
+# Related Work and Citations
+
+## Overview
+
+Context-as-a-Service builds upon and addresses limitations in existing research on Retrieval-Augmented Generation (RAG), context management, and information retrieval systems. This document provides citations to related work and positions CaaS within the broader research landscape.
+
+## Foundational Work
+
+### Retrieval-Augmented Generation (RAG)
+
+1. **Lewis, P., et al. (2020).** "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." *NeurIPS 2020*.
+   - **Contribution**: Introduced RAG combining retrieval with generation
+   - **Relevance**: CaaS extends RAG with structure-aware, time-aware context serving
+   - **Paper**: https://arxiv.org/abs/2005.11401
+
+2. **Guu, K., et al. (2020).** "REALM: Retrieval-Augmented Language Model Pre-Training." *ICML 2020*.
+   - **Contribution**: Pre-training with retrieved knowledge
+   - **Relevance**: CaaS focuses on serving-time retrieval optimization
+   - **Paper**: https://arxiv.org/abs/2002.08909
+
+3. **Izacard, G., & Grave, E. (2021).** "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering." *EACL 2021*.
+   - **Contribution**: Fusion-in-Decoder architecture
+   - **Relevance**: CaaS provides the retrieval component for such architectures
+   - **Paper**: https://arxiv.org/abs/2007.01282
+
+### Document Structure and Hierarchical Indexing
+
+4. **Cohan, A., et al. (2018).** "A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents." *NAACL 2018*.
+   - **Contribution**: Hierarchical document structure for summarization
+   - **Relevance**: Inspired CaaS's structure-aware indexing approach
+   - **Paper**: https://arxiv.org/abs/1804.05685
+
+5. **Liu, Y., & Lapata, M. (2019).** "Hierarchical Transformers for Multi-Document Summarization." *ACL 2019*.
+   - **Contribution**: Multi-level document representation
+   - **Relevance**: Similar to CaaS's High/Medium/Low value tiers
+   - **Paper**: https://arxiv.org/abs/1905.13164
+
+6. **Xiao, W., & Carenini, G. (2019).** "Extractive Summarization of Long Documents by Combining Global and Local Context." *EMNLP 2019*.
+   - **Contribution**: Global vs. local context distinction
+   - **Relevance**: Parallels CaaS's metadata injection for context preservation
+   - **Paper**: https://arxiv.org/abs/1909.08089
+
+## Temporal and Dynamic Information Retrieval
+
+### Time-Aware Retrieval
+
+7. **Dai, Z., & Callan, J. (2019).** "Deeper Text Understanding for IR with Contextual Neural Language Modeling." *SIGIR 2019*.
+   - **Contribution**: Context-aware ranking models
+   - **Relevance**: CaaS adds explicit temporal decay to ranking
+   - **Paper**: https://arxiv.org/abs/1905.09217
+
+8. **Nguyen, T., et al. (2016).** "A Neural Network Approach to Context-Sensitive Generation of Conversational Responses." *NAACL 2016*.
+   - **Contribution**: Time-aware conversation modeling
+   - **Relevance**: Influenced CaaS's sliding window approach
+   - **Paper**: https://arxiv.org/abs/1506.06714
+
+9. **Campos, R., et al. (2014).** "Survey of Temporal Information Retrieval and Scoping Methods." *WWW Journal*.
+   - **Contribution**: Comprehensive survey of temporal IR
+   - **Relevance**: Theoretical foundation for CaaS's time decay
+   - **Citation**: DOI: 10.1007/s11280-013-0230-y
+
+### Knowledge Freshness
+
+10. **Kasai, J., et al. (2022).** "RealTime QA: What's the Answer Right Now?" *NeurIPS 2022*.
+    - **Contribution**: Benchmark for time-sensitive questions
+    - **Relevance**: Motivation for CaaS's temporal awareness
+    - **Paper**: https://arxiv.org/abs/2207.13332
+
+11. **Lazaridou, A., et al. (2021).** "Mind the Gap: Assessing Temporal Generalization in Neural Language Models." *NeurIPS 2021*.
+    - **Contribution**: Showed LLMs struggle with temporal knowledge
+    - **Relevance**: Justifies CaaS's explicit time-based retrieval
+    - **Paper**: https://arxiv.org/abs/2102.01951
+
+## Source Attribution and Provenance
+
+### Citation and Source Tracking
+
+12. **Gao, L., et al. (2022).** "Rarr: Researching and Revising What Language Models Say, Using Language Models." *ACL 2023*.
+    - **Contribution**: Using LLMs to research and cite sources
+    - **Relevance**: CaaS provides the source tracking infrastructure
+    - **Paper**: https://arxiv.org/abs/2210.08726
+
+13. **Menick, J., et al. (2022).** "Teaching Language Models to Support Answers with Verified Quotes." *NeurIPS 2022*.
+    - **Contribution**: Attribution of generated text to sources
+    - **Relevance**: CaaS's pragmatic truth and source citations
+    - **Paper**: https://arxiv.org/abs/2203.11147
+
+14. **Rashkin, H., et al. (2021).** "Measuring Attribution in Natural Language Generation Models." *CL 2021*.
+    - **Contribution**: Metrics for source attribution quality
+    - **Relevance**: Potential evaluation framework for CaaS
+    - **Paper**: https://arxiv.org/abs/2112.12870
+
+### Information Credibility
+
+15. **Thorne, J., & Vlachos, A. (2018).** "Automated Fact Checking: Task Formulations, Methods and Future Directions." *COLING 2018*.
+    - **Contribution**: Framework for fact verification
+    - **Relevance**: Inspired CaaS's conflict detection between sources
+    - **Paper**: https://arxiv.org/abs/1806.07687
+
+## The Accumulation Paradox and Long-Context Degradation
+
+A growing body of work reveals a counterintuitive phenomenon we term the **Accumulation Paradox**: adding more context to LLMs can paradoxically *degrade* rather than improve performance.
+
+### Lost in the Middle Phenomenon
+
+34. **Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023).** "Lost in the Middle: How Language Models Use Long Contexts." *Transactions of the Association for Computational Linguistics (TACL)*.
+    - **Contribution**: Landmark study showing U-shaped performance curve where models ignore information in the middle of long contexts
+    - **Relevance**: Directly motivates CaaS's context prioritization and decay mechanisms
+    - **Key Finding**: "Performance can degrade significantly when changing the position of relevant information"
+    - **Paper**: https://arxiv.org/abs/2307.03172
+
+### Streaming and Attention Degradation
+
+35. **Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2024).** "Efficient Streaming Language Models with Attention Sinks." *ICLR 2024*.
+    - **Contribution**: Showed window attention fails when context exceeds cache size; introduced "attention sink" phenomenon
+    - **Relevance**: Motivates CaaS's sliding window and context decay approach for streaming/agentic settings
+    - **Key Finding**: Disproportionate attention to initial tokens regardless of semantic relevance
+    - **Paper**: https://arxiv.org/abs/2309.17453
+
+### Long In-Context Learning Struggles
+
+36. **Li, T., Zhang, G., Do, Q. D., Yue, X., & Chen, W. (2024).** "Long-context LLMs Struggle with Long In-context Learning." *arXiv preprint*.
+    - **Contribution**: Demonstrated that even purpose-built long-context LLMs degrade with accumulated context
+    - **Relevance**: Validates CaaS's position that more context is not always better
+    - **Key Finding**: Bias toward later-presented labels and degraded reasoning over multiple context pieces
+    - **Paper**: https://arxiv.org/abs/2404.02060
+
+### Agentic Memory Management
+
+37. **Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023).** "MemGPT: Towards LLMs as Operating Systems." *arXiv preprint*.
+    - **Contribution**: Showed raw context accumulation cannot sustain long-running agents; proposed OS-inspired memory management
+    - **Relevance**: Directly motivates CaaS's hierarchical context management for multi-turn and agentic settings
+    - **Key Finding**: Virtual context management enables stable operation with millions of tokens
+    - **Paper**: https://arxiv.org/abs/2310.08560
+
+**CaaS addresses the Accumulation Paradox through**: (1) time-based decay that deprioritizes older context, (2) the Context Triad ensuring relevant context occupies attention-friendly positions, and (3) structure-aware indexing preventing low-value content from diluting the context window.
+
+## Context Window Management
+
+### Conversation History Management
+
+16. **Dinan, E., et al. (2019).** "Wizard of Wikipedia: Knowledge-Powered Conversational Agents." *ICLR 2019*.
+    - **Contribution**: Managing knowledge in conversations
+    - **Relevance**: Informed CaaS's conversation management approach
+    - **Paper**: https://arxiv.org/abs/1811.01241
+
+17. **Zhang, S., et al. (2020).** "DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation." *ACL 2020*.
+    - **Contribution**: Multi-turn conversation modeling
+    - **Relevance**: Context window limitations motivate CaaS's FIFO approach
+    - **Paper**: https://arxiv.org/abs/1911.00536
+
+### Compression and Summarization
+
+18. **Gekhman, Z., et al. (2023).** "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" *arXiv*.
+    - **Contribution**: Showed summarization can introduce errors
+    - **Relevance**: Justifies CaaS's "chopping over summarizing" philosophy
+    - **Paper**: https://arxiv.org/abs/2405.05904
+
+19. **Chevalier, A., et al. (2023).** "Adapting Language Models to Compress Contexts." *EMNLP 2023*.
+    - **Contribution**: Context compression techniques
+    - **Relevance**: Alternative approach to CaaS's sliding window
+    - **Paper**: https://arxiv.org/abs/2305.14788
+
+## Efficient RAG and Retrieval
+
+### Chunking and Segmentation
+
+20. **Jansen, B. J., & Rieh, S. Y. (2010).** "The Seventeen Theoretical Constructs of Information Searching and Information Retrieval." *JASIST*.
+    - **Contribution**: Theoretical foundations of IR
+    - **Relevance**: Theoretical basis for CaaS's chunking strategy
+    - **Citation**: DOI: 10.1002/asi.21358
+
+21. **Winata, G. I., et al. (2023).** "Retrieval Augmented Generation for Information Retrieval: A Survey." *arXiv*.
+    - **Contribution**: Comprehensive RAG survey
+    - **Relevance**: Positions CaaS in the RAG ecosystem
+    - **Paper**: https://arxiv.org/abs/2312.10997
+
+### Optimization and Efficiency
+
+22. **Wang, L., et al. (2023).** "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." *arXiv*.
+    - **Contribution**: Adaptive retrieval strategies
+    - **Relevance**: CaaS uses deterministic heuristics instead of learned routing
+    - **Paper**: https://arxiv.org/abs/2310.11511
+
+23. **Khattab, O., et al. (2021).** "Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval." *NeurIPS 2021*.
+    - **Contribution**: Efficient multi-hop retrieval
+    - **Relevance**: Future direction for CaaS query routing
+    - **Paper**: https://arxiv.org/abs/2101.00436
+
+## Enterprise and Production Systems
+
+### Trust and Security
+
+24. **Carlini, N., et al. (2023).** "Extracting Training Data from Large Language Models." *USENIX Security 2021*.
+    - **Contribution**: Privacy risks in LLMs
+    - **Relevance**: Motivates CaaS's Trust Gateway and on-prem deployment
+    - **Paper**: https://arxiv.org/abs/2012.07805
+
+25. **Brown, H., et al. (2022).** "What Does It Mean for a Language Model to Preserve Privacy?" *FAccT 2022*.
+    - **Contribution**: Privacy definitions for NLP systems
+    - **Relevance**: Framework for CaaS's privacy guarantees
+    - **Paper**: https://arxiv.org/abs/2202.05520
+
+### API and Gateway Patterns
+
+26. **Richardson, C., & Smith, F. (2016).** *Microservices Patterns*. Manning Publications.
+    - **Contribution**: API Gateway pattern
+    - **Relevance**: Architectural inspiration for Trust Gateway
+    - **Book**: ISBN 978-1617294549
+
+27. **Newman, S. (2021).** *Building Microservices: Designing Fine-Grained Systems*. O'Reilly Media.
+    - **Contribution**: Service decomposition and boundaries
+    - **Relevance**: Modular architecture of CaaS components
+    - **Book**: ISBN 978-1492034025
+
+## Multi-Agent Systems
+
+### Agent Collaboration
+
+28. **Wu, Q., et al. (2023).** "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation." *arXiv*.
+    - **Contribution**: Framework for multi-agent LLM systems
+    - **Relevance**: Integration target for CaaS multi-agent examples
+    - **Paper**: https://arxiv.org/abs/2308.08155
+
+29. **Hong, S., et al. (2023).** "MetaGPT: Meta Programming for Multi-Agent Collaborative Framework." *arXiv*.
+    - **Contribution**: Multi-agent software development
+    - **Relevance**: Use case for shared CaaS context across agents
+    - **Paper**: https://arxiv.org/abs/2308.00352
+
+30. **Chase, H. (2022).** "LangChain: Building Applications with LLMs through Composability."
+    - **Contribution**: Composable LLM application framework
+    - **Relevance**: Integration target for CaaS context serving
+    - **GitHub**: https://github.com/langchain-ai/langchain
+
+## Benchmarks and Evaluation
+
+### RAG Evaluation
+
+31. **Es, S., et al. (2023).** "RAGAS: Automated Evaluation of Retrieval Augmented Generation." *arXiv*.
+    - **Contribution**: Metrics for RAG quality evaluation
+    - **Relevance**: Evaluation framework for CaaS context quality
+    - **Paper**: https://arxiv.org/abs/2309.15217
+
+32. **Chen, J., et al. (2023).** "Dense X Retrieval: What Retrieval Granularity Should We Use?" *arXiv*.
+    - **Contribution**: Evaluation of retrieval granularities
+    - **Relevance**: Validates CaaS's hierarchical chunking approach
+    - **Paper**: https://arxiv.org/abs/2312.06648
+
+### Information Retrieval Benchmarks
+
+33. **Thakur, N., et al. (2021).** "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models." *NeurIPS 2021*.
+    - **Contribution**: Comprehensive IR evaluation benchmark
+    - **Relevance**: Potential benchmark for CaaS retrieval quality
+    - **Paper**: https://arxiv.org/abs/2104.08663
+
+## Key Differences: CaaS vs. Prior Work
+
+### What CaaS Adds
+
+1. **Structure-Aware Indexing**: Explicit hierarchical value tiers (High/Medium/Low) rather than flat chunking
+2. **Pragmatic Truth**: Dual-source tracking (official vs. practical) with conflict detection
+3. **Zero-Overhead Routing**: Heuristic-based routing (0ms) instead of LLM-based (100ms+)
+4. **Chopping over Summarizing**: FIFO sliding window instead of lossy summarization
+5. **Enterprise Trust**: On-premises/air-gapped deployment as first-class citizen
+6. **Integrated Pipeline**: End-to-end system addressing multiple RAG fallacies
+
+### What CaaS Does NOT Claim
+
+- **Not claiming state-of-the-art retrieval accuracy**: Trade-off efficiency and transparency for some accuracy
+- **Not a new ML model**: Uses existing techniques (embeddings, decay functions) in a novel pipeline
+- **Not replacing vector databases**: Can integrate with them, but provides alternative architecture
+- **Not a research contribution in algorithms**: Contribution is in system design and integration
+
+## Future Research Directions
+
+Based on related work, promising directions for CaaS include:
+
+1. **Learned Weight Tuning**: Using ML to optimize weights instead of heuristics (similar to [22])
+2. **Multi-Hop Reasoning**: Extending heuristic router for complex queries (inspired by [23])
+3. **Adaptive Context Compression**: Dynamic compression based on query needs (from [19])
+4. **Federated RAG**: Privacy-preserving corpus analysis across organizations (inspired by [24, 25])
+5. **Formal Evaluation**: Rigorous benchmarking against baselines (using frameworks from [31, 32, 33])
+
+## Contributing Citations
+
+If you use CaaS in your research or extend it with novel techniques, we encourage you to:
+
+1. Cite relevant prior work from this document
+2. Add your own citations to this list via pull request
+3. Document how your work relates to existing research
+4. Share evaluation results and benchmarks
+
+## BibTeX (for CaaS itself)
+
+```bibtex
+@software{context_as_a_service_2026,
+  title = {Context-as-a-Service: A Managed Pipeline for Intelligent Context Extraction and Serving},
+  author = {Siddique, Imran},
+  year = {2026},
+  url = {https://github.com/imran-siddique/context-as-a-service},
+  version = {0.1.0}
+}
+```
+
+## Acknowledgments
+
+CaaS builds on decades of research in information retrieval, natural language processing, and distributed systems. We are grateful to the researchers whose work made this possible.
+
+---
+
+*Last Updated: January 2026*
+*Note: This is a living document. Contributions and corrections welcome via pull request.*

modules/caas/docs/RELEASE_CHECKLIST.md

@@ -0,0 +1,219 @@
+# Release Checklist for PyPI and GitHub
+
+This checklist ensures a smooth release process for Context-as-a-Service.
+
+## Pre-Release Checklist
+
+### 1. Version Updates
+- [ ] Update version in `caas/__init__.py`
+- [ ] Update version in `pyproject.toml`
+- [ ] Update version in `setup.py`
+- [ ] Update `CHANGELOG.md` with release notes
+
+### 2. Documentation
+- [ ] README badges are up-to-date
+- [ ] All documentation is current
+- [ ] Examples in README work correctly
+- [ ] API documentation is generated and accessible
+
+### 3. Code Quality
+- [ ] All tests pass: `pytest tests/ -v`
+- [ ] Linting passes: `ruff check caas/ tests/`
+- [ ] Code formatting passes: `black --check caas/ tests/`
+- [ ] Type checking passes (if enabled): `mypy caas/`
+- [ ] CI/CD pipeline is green
+
+### 4. Package Building
+- [ ] Clean build artifacts: `rm -rf dist/ build/ *.egg-info`
+- [ ] Build package: `python -m build`
+- [ ] Validate package: `twine check dist/*`
+- [ ] Test installation in fresh venv:
+  ```bash
+  python -m venv test_env
+  source test_env/bin/activate
+  pip install dist/*.whl
+  python -c "import caas; print(caas.__version__)"
+  deactivate
+  rm -rf test_env
+  ```
+
+### 5. Docker
+- [ ] Docker build succeeds: `docker build -t context-as-a-service:latest .`
+- [ ] Docker compose works: `docker-compose up`
+- [ ] Health check passes: `curl http://localhost:8000/health`
+
+## Release Process
+
+### Option A: GitHub Release (Recommended - Triggers Auto-Publish)
+
+1. **Create and Push Git Tag**
+   ```bash
+   git tag v0.1.0
+   git push origin v0.1.0
+   ```
+
+2. **Create GitHub Release**
+   - Go to: https://github.com/imran-siddique/context-as-a-service/releases/new
+   - Select tag: `v0.1.0`
+   - Release title: `v0.1.0 - Initial Release`
+   - Description: Copy from CHANGELOG.md
+   - Click "Publish release"
+
+3. **Automated Actions**
+   - GitHub Actions will automatically:
+     - Build the package
+     - Run tests
+     - Publish to PyPI
+     - Upload signed artifacts to GitHub Release
+
+4. **Verify Release**
+   - Check PyPI: https://pypi.org/project/context-as-a-service/
+   - Test install: `pip install context-as-a-service`
+   - Verify version: `python -c "import caas; print(caas.__version__)"`
+
+### Option B: Manual PyPI Upload (Backup Method)
+
+If automated publishing fails, use this manual process:
+
+1. **Set up PyPI credentials**
+   ```bash
+   # Create ~/.pypirc file
+   cat > ~/.pypirc << EOF
+   [pypi]
+   username = __token__
+   password = pypi-YOUR-API-TOKEN-HERE
+   EOF
+   chmod 600 ~/.pypirc
+   ```
+
+2. **Build and Upload**
+   ```bash
+   # Clean and build
+   rm -rf dist/ build/ *.egg-info
+   python -m build
+   
+   # Check
+   twine check dist/*
+   
+   # Upload to PyPI
+   twine upload dist/*
+   ```
+
+3. **Verify on PyPI**
+   - Visit: https://pypi.org/project/context-as-a-service/
+   - Test install: `pip install context-as-a-service`
+
+### Option C: Test on TestPyPI First
+
+For testing before publishing to main PyPI:
+
+1. **Trigger GitHub Action**
+   - Go to: Actions → "Publish to PyPI"
+   - Click "Run workflow"
+   - Check "Publish to TestPyPI"
+   - Click "Run workflow"
+
+2. **Test Installation from TestPyPI**
+   ```bash
+   pip install -i https://test.pypi.org/simple/ \
+     --extra-index-url https://pypi.org/simple/ \
+     context-as-a-service
+   ```
+
+3. **Verify**
+   ```bash
+   python -c "import caas; print(caas.__version__)"
+   caas --help
+   ```
+
+4. **If successful, proceed with main PyPI release**
+
+## Post-Release Checklist
+
+### 1. Verify Installation
+- [ ] Install works: `pip install context-as-a-service`
+- [ ] CLI works: `caas --help`
+- [ ] Python import works: `import caas`
+- [ ] Version is correct: `caas.__version__`
+
+### 2. Update Documentation
+- [ ] PyPI page looks correct
+- [ ] README badges show correct version
+- [ ] Documentation links work
+- [ ] Docker image builds with new version
+
+### 3. Announce Release
+- [ ] Update README with installation instructions
+- [ ] Post in project channels (Slack, Discord, etc.)
+- [ ] Tweet/social media announcement (if applicable)
+- [ ] Update project website (if applicable)
+
+### 4. Monitor
+- [ ] Watch PyPI download stats
+- [ ] Monitor GitHub issues for release-related problems
+- [ ] Check CI/CD is still green
+- [ ] Verify Docker Hub build (if configured)
+
+## Troubleshooting
+
+### Build Fails
+```bash
+# Clean everything
+rm -rf dist/ build/ *.egg-info __pycache__ .pytest_cache .ruff_cache .mypy_cache
+
+# Reinstall build tools
+pip install --upgrade build twine
+
+# Try again
+python -m build
+```
+
+### Upload Fails
+- Check PyPI credentials
+- Verify package name isn't taken
+- Ensure version number is unique (can't re-upload same version)
+- Check file sizes (PyPI has limits)
+
+### Tests Fail After Release
+- Ensure all dependencies are specified correctly
+- Check version constraints aren't too strict
+- Test in fresh environment
+- Verify optional dependencies are truly optional
+
+## Version Numbering
+
+We follow [Semantic Versioning](https://semver.org/):
+
+- **MAJOR** version (1.0.0): Breaking changes
+- **MINOR** version (0.1.0): New features (backwards compatible)
+- **PATCH** version (0.0.1): Bug fixes
+
+Current version: **0.1.0** (Initial release)
+
+## Rollback Procedure
+
+If a release has critical issues:
+
+1. **Yank the release on PyPI** (doesn't delete, but marks as unusable)
+   - Go to PyPI project page
+   - Click "Manage" → "Releases"
+   - Click "Options" → "Yank release"
+
+2. **Fix the issue**
+   - Create hotfix branch
+   - Fix bug
+   - Bump patch version
+
+3. **Release new version**
+   - Follow release process above
+   - Document what was fixed
+
+## Contacts
+
+- **PyPI Package**: https://pypi.org/project/context-as-a-service/
+- **GitHub Releases**: https://github.com/imran-siddique/context-as-a-service/releases
+- **Issues**: https://github.com/imran-siddique/context-as-a-service/issues
+
+---
+
+*Last Updated: January 21, 2026*