agent-os-kernel 1.1.0__py3-none-any.whl → 1.2.0__py3-none-any.whl

modules/caas/docs/CONTRIBUTING.md

@@ -0,0 +1,346 @@
+# Contributing to Context-as-a-Service
+
+Thank you for your interest in contributing! This document provides guidelines for contributing to the project.
+
+## Code of Conduct
+
+Be respectful, inclusive, and collaborative. We're here to build something great together.
+
+## Getting Started
+
+### 1. Fork and Clone
+
+```bash
+git clone https://github.com/<your-username>/context-as-a-service.git
+cd context-as-a-service
+```
+
+### 2. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### 3. Run Tests
+
+```bash
+python run_tests.py
+```
+
+## Project Structure
+
+```
+context-as-a-service/
+├── caas/                    # Main package
+│   ├── __init__.py
+│   ├── models.py           # Data models
+│   ├── cli.py              # CLI tool
+│   ├── api/                # REST API
+│   ├── ingestion/          # Document processors
+│   ├── detection/          # Type detection
+│   ├── tuning/             # Weight tuning
+│   ├── storage/            # Document storage
+│   ├── enrichment.py       # Metadata enrichment
+│   ├── decay.py            # Time-based decay
+│   ├── triad.py            # Context Triad (Hot/Warm/Cold)
+│   ├── pragmatic_truth.py  # Source tracking
+│   ├── routing/            # Heuristic routing
+│   ├── conversation.py     # Conversation management
+│   └── gateway/            # Trust Gateway
+├── tests/                  # Test suite
+├── examples/               # Example usage
+│   ├── agents/            # Sample agent implementations
+│   └── *.py               # Demo scripts
+├── docs/                   # Documentation (markdown files)
+├── run_tests.py           # Test runner
+├── TESTING.md             # Testing guide
+├── CONTRIBUTING.md        # This file
+└── README.md              # Project overview
+```
+
+## Development Workflow
+
+### 1. Create a Branch
+
+```bash
+git checkout -b feature/your-feature-name
+```
+
+### 2. Make Changes
+
+- Write clear, documented code
+- Follow existing code style
+- Add docstrings to functions and classes
+- Use type hints where appropriate
+
+### 3. Add Tests
+
+For any new functionality, add tests in `tests/`:
+
+```python
+"""
+Test description.
+"""
+import sys
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from caas.module import NewFeature
+
+
+def test_new_feature():
+    """Test the new feature."""
+    print("\n=== Testing New Feature ===")
+    
+    feature = NewFeature()
+    result = feature.do_something()
+    
+    assert result is not None
+    print("✓ Feature working correctly")
+```
+
+### 4. Run Tests
+
+```bash
+python run_tests.py
+```
+
+### 5. Commit Changes
+
+Use clear, descriptive commit messages:
+
+```bash
+git add .
+git commit -m "Add feature: brief description
+
+- Detailed point 1
+- Detailed point 2
+- Closes #123"
+```
+
+### 6. Push and Create PR
+
+```bash
+git push origin feature/your-feature-name
+```
+
+Then create a Pull Request on GitHub.
+
+## Coding Standards
+
+### Python Style
+
+- Follow PEP 8 style guide
+- Use 4 spaces for indentation
+- Maximum line length: 100 characters
+- Use descriptive variable names
+
+### Docstrings
+
+Use Google-style docstrings:
+
+```python
+def extract_context(document_id: str, query: str, max_tokens: int = 2000) -> Tuple[str, Dict]:
+    """
+    Extract context from a document.
+    
+    Args:
+        document_id: ID of document to extract from
+        query: Search query
+        max_tokens: Maximum tokens to extract
+        
+    Returns:
+        Tuple of (context_string, metadata_dict)
+        
+    Raises:
+        ValueError: If document not found
+        
+    Example:
+        >>> context, metadata = extractor.extract_context("doc-123", "authentication")
+        >>> print(len(context))
+        1847
+    """
+    # Implementation
+```
+
+### Type Hints
+
+Use type hints for function signatures:
+
+```python
+from typing import List, Dict, Optional
+
+def process_documents(
+    docs: List[Document],
+    max_count: Optional[int] = None
+) -> Dict[str, Any]:
+    ...
+```
+
+## Testing Guidelines
+
+### Test Coverage
+
+- Write tests for all new features
+- Test edge cases and error conditions
+- Aim for clear, readable test code
+
+### Test Structure
+
+```python
+def test_feature_name():
+    """Test description."""
+    print("\n=== Testing Feature ===")
+    
+    # Setup
+    component = Component()
+    
+    # Execute
+    result = component.method()
+    
+    # Assert
+    assert result == expected_value
+    print("✓ Test passed")
+```
+
+### Running Tests
+
+```bash
+# All tests
+python run_tests.py
+
+# Specific test
+python -m tests.test_module_name
+```
+
+## Areas for Contribution
+
+### High Priority
+
+1. **Additional Document Processors**
+   - Support for more file formats (DOCX, Markdown, etc.)
+   - Better code language support
+   - Improved structure detection
+
+2. **Enhanced Detection**
+   - Better document type classification
+   - More sophisticated pattern matching
+   - Machine learning-based detection
+
+3. **Performance Optimization**
+   - Faster document processing
+   - Efficient storage mechanisms
+   - Caching strategies
+
+### Medium Priority
+
+4. **API Enhancements**
+   - Additional endpoints
+   - WebSocket support for real-time updates
+   - GraphQL API option
+
+5. **CLI Improvements**
+   - Interactive mode
+   - Better output formatting
+   - Progress indicators
+
+6. **Documentation**
+   - More examples
+   - Tutorials
+   - Video guides
+
+### Lower Priority
+
+7. **UI/Dashboard**
+   - Web-based interface
+   - Visualization of document structures
+   - Analytics dashboard
+
+8. **Integrations**
+   - Slack bot
+   - VS Code extension
+   - Zapier integration
+
+## Module-Specific Guidelines
+
+### Ingestion Module (`caas/ingestion/`)
+
+When adding new processors:
+- Inherit from base `Processor` class
+- Implement `process()` method
+- Track document hierarchy
+- Extract meaningful sections
+
+### Detection Module (`caas/detection/`)
+
+When adding detection patterns:
+- Add patterns to appropriate category
+- Test with diverse documents
+- Consider false positives/negatives
+
+### Tuning Module (`caas/tuning/`)
+
+When adding tuning rules:
+- Add to `TYPE_SPECIFIC_WEIGHTS`
+- Document reasoning
+- Test with real documents
+
+## Documentation
+
+### Inline Documentation
+
+- Add docstrings to all public functions/classes
+- Include examples in docstrings
+- Explain complex algorithms
+
+### README Updates
+
+Update README.md when:
+- Adding new features
+- Changing API
+- Updating installation process
+
+### Architecture Documentation
+
+Document architectural decisions in:
+- Code comments for complex logic
+- Separate docs for major changes
+- Examples for new patterns
+
+## Pull Request Process
+
+1. **Before Submitting**
+   - Run all tests
+   - Update documentation
+   - Add examples if needed
+   - Rebase on latest main
+
+2. **PR Description**
+   - Describe what and why
+   - Link related issues
+   - Show before/after examples
+   - List breaking changes
+
+3. **Review Process**
+   - Address review comments
+   - Keep discussions focused
+   - Be open to suggestions
+
+4. **After Merge**
+   - Delete your branch
+   - Update your fork
+   - Celebrate! 🎉
+
+## Questions or Issues?
+
+- **Bug reports**: Open an issue with reproduction steps
+- **Feature requests**: Open an issue with use case description
+- **Questions**: Start a discussion or open an issue
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the MIT License.
+
+---
+
+Thank you for contributing to Context-as-a-Service! 🚀

modules/caas/docs/ETHICS_AND_LIMITATIONS.md

@@ -0,0 +1,336 @@
+# Ethics and Limitations
+
+## Overview
+
+This document addresses the ethical considerations, known limitations, potential biases, and responsible use guidelines for Context-as-a-Service (CaaS). We believe in transparent communication about system capabilities and constraints.
+
+## Ethical Considerations
+
+### 1. Data Privacy and Consent
+
+#### User Consent
+- **Principle**: Users must explicitly consent to document ingestion and processing
+- **Implementation**: CaaS processes only explicitly provided documents
+- **Consideration**: Organizations must ensure they have rights to process all ingested content
+- **Risk**: Inadvertent processing of personal or confidential data without proper consent
+
+#### Data Minimization
+- **Principle**: Collect and retain only necessary data
+- **Implementation**: Configurable retention policies and data purging capabilities
+- **Consideration**: Balance between context quality and privacy
+- **Risk**: Over-retention of data beyond its useful lifetime
+
+### 2. Transparency and Explainability
+
+#### Source Attribution
+- **Strength**: CaaS provides transparent source citations for all context
+- **Benefit**: Users can verify information origin and trustworthiness
+- **Limitation**: Source tracking may reveal organizational patterns
+- **Mitigation**: Configurable source anonymization options
+
+#### Decision Transparency
+- **Strength**: Heuristic routing provides deterministic, auditable decisions
+- **Benefit**: No "black box" AI routing decisions
+- **Limitation**: Simple heuristics may miss nuanced query intent
+- **Trade-off**: Transparency over potentially higher accuracy
+
+### 3. Bias and Fairness
+
+#### Temporal Bias
+- **Issue**: Time decay inherently biases toward recent information
+- **Impact**: Older but still-valid information may be deprioritized
+- **Use Case**: Appropriate for fast-moving domains (software, news)
+- **Inappropriate**: Historical research, legal precedents, foundational knowledge
+- **Mitigation**: Configurable decay rates, ability to disable time-based ranking
+
+#### Source Bias
+- **Issue**: "Pragmatic Truth" may elevate unofficial sources (Slack, forums) over official documentation
+- **Impact**: Unofficial but practical knowledge gets visibility
+- **Risk**: Unofficial sources may contain incorrect information
+- **Mitigation**: Conflict detection highlights discrepancies between sources
+
+#### Structural Bias
+- **Issue**: Auto-tuning weights based on detected patterns
+- **Impact**: Content types appearing frequently get higher weights
+- **Risk**: Minority document types may be underrepresented
+- **Mitigation**: Manual weight overrides, minimum weight thresholds
+
+#### Language and Cultural Bias
+- **Issue**: Structure detection and metadata enrichment tuned for English content
+- **Impact**: Non-English documents may have suboptimal structure detection
+- **Risk**: Reduced quality for international users
+- **Limitation**: Current version is English-centric
+- **Future Work**: Multi-language support, cultural context awareness
+
+### 4. Dual Use and Misuse Potential
+
+#### Surveillance and Monitoring
+- **Risk**: CaaS could be used to monitor employee communications
+- **Example**: Ingesting internal Slack/email to answer "What is the team saying about X?"
+- **Ethics**: Employee surveillance without consent is unethical
+- **Guidance**: Organizations must have clear policies and employee notification
+
+#### Competitive Intelligence
+- **Risk**: Aggressive scraping of competitor websites/documentation
+- **Ethics**: Respecting robots.txt, terms of service, and legal boundaries
+- **Guidance**: Only ingest publicly available, legally accessible content
+
+#### Disinformation and Manipulation
+- **Risk**: Selectively ingesting biased sources to manipulate context
+- **Example**: Only ingesting one side of a debate to bias AI responses
+- **Mitigation**: Encourage diverse source ingestion
+- **Responsibility**: Ultimately on the deploying organization
+
+### 5. Environmental Impact
+
+#### Carbon Footprint
+- **Processing Cost**: Document ingestion and structure analysis require computation
+- **Embedding Cost**: If vector embeddings are used (optional future feature)
+- **Inference Cost**: Context extraction and serving
+- **Mitigation**: 
+  - Efficient algorithms (no unnecessary re-processing)
+  - Local deployment reduces network costs
+  - Heuristic routing (no LLM calls for routing)
+  - Sliding window (no summarization LLM calls)
+
+#### Resource Efficiency
+- **Strength**: CaaS optimizes for efficiency over maximum accuracy
+- **Examples**:
+  - Chopping (FIFO) instead of expensive summarization
+  - Heuristic routing instead of LLM-based routing
+  - Local processing instead of API calls
+- **Philosophy**: "Good enough" solutions that minimize resource waste
+
+## Known Limitations
+
+### 1. Context Quality Limitations
+
+#### Flat Embedding Limitations (if using vector search)
+- **Issue**: Even with structure-aware indexing, semantic search has inherent limitations
+- **Example**: Cannot distinguish between "This is good" (positive) and "This is not good" (negative)
+- **Impact**: Some nuanced queries may retrieve suboptimal context
+- **Mitigation**: Hybrid search combining keywords and semantics (future work)
+
+#### Metadata Incompleteness
+- **Issue**: Metadata enrichment depends on structure detection accuracy
+- **Example**: Unstructured documents may have minimal metadata
+- **Impact**: Less effective chunk disambiguation
+- **Mitigation**: Manual metadata addition capabilities
+
+#### Cold Start Problem
+- **Issue**: Auto-tuning requires sufficient corpus to learn patterns
+- **Example**: First few documents have generic weights
+- **Impact**: Suboptimal context quality initially
+- **Mitigation**: Sensible defaults, manual tuning option
+
+### 2. Temporal Limitations
+
+#### Truth Stability Assumption
+- **Issue**: Time decay assumes older content is less relevant
+- **Problem**: Some domains have stable truths (mathematics, history)
+- **Example**: A 10-year-old explanation of quicksort is still valid
+- **Impact**: Inappropriate for domains with stable knowledge
+- **Mitigation**: Configurable decay rates, domain-specific policies
+
+#### Timestamp Reliability
+- **Issue**: Relies on file modification times or explicit timestamps
+- **Problem**: Copied/migrated files may have incorrect timestamps
+- **Impact**: Incorrect recency judgments
+- **Mitigation**: Manual timestamp overrides, ingestion date tracking
+
+### 3. Scale Limitations
+
+#### Single-Node Architecture (Current)
+- **Issue**: Current implementation assumes single-server deployment
+- **Limitation**: Limited to documents that fit in available storage
+- **Impact**: May not scale to massive corporate corpora
+- **Future Work**: Distributed storage and processing
+
+#### Query Performance
+- **Issue**: Linear search over all chunks as corpus grows
+- **Impact**: Slower response times with large corpora
+- **Mitigation**: Indexing, caching strategies (future work)
+
+### 4. Language and Format Limitations
+
+#### Supported Formats
+- **Current**: PDF, HTML, Python/JavaScript source code
+- **Limitation**: No support for DOCX, PowerPoint, images, videos
+- **Impact**: Cannot ingest all document types
+- **Workaround**: Convert to supported formats
+- **Future Work**: Additional format processors
+
+#### Language Support
+- **Current**: Optimized for English text
+- **Limitation**: Non-English text may have suboptimal processing
+- **Impact**: Reduced effectiveness for international deployments
+- **Future Work**: Multi-language support, language detection
+
+### 5. Integration Limitations
+
+#### No Native Vector Database
+- **Current**: Simple in-memory or file-based storage
+- **Limitation**: No optimized vector similarity search
+- **Impact**: May be slower than specialized solutions at scale
+- **Future Work**: Optional integrations with Qdrant, Pinecone, Weaviate
+
+#### No Built-in LLM Integration
+- **Current**: CaaS is context-serving only, not a complete RAG system
+- **Benefit**: Modular, bring-your-own-LLM
+- **Limitation**: Requires separate LLM infrastructure
+- **Philosophy**: Separation of concerns (context ≠ generation)
+
+## Failure Modes and Edge Cases
+
+### 1. Heuristic Router Failures
+
+#### Ambiguous Queries
+- **Scenario**: Query matches multiple heuristic patterns
+- **Failure Mode**: Falls back to default strategy
+- **Impact**: May not route optimally
+- **Frequency**: Low-medium
+- **Mitigation**: More specific patterns, user feedback loop
+
+#### Unseen Query Types
+- **Scenario**: Query type not covered by any heuristic
+- **Failure Mode**: Generic fallback routing
+- **Impact**: Suboptimal results
+- **Frequency**: Medium
+- **Mitigation**: Extensible pattern system, analytics to identify gaps
+
+### 2. Pragmatic Truth Conflicts
+
+#### Irreconcilable Conflicts
+- **Scenario**: Official docs say X, team says Y, both plausible
+- **Failure Mode**: System highlights conflict but cannot resolve
+- **Impact**: User must manually adjudicate
+- **Frequency**: Low
+- **Philosophy**: Transparent uncertainty is better than false confidence
+
+#### Stale Unofficial Information
+- **Scenario**: Slack message from 6 months ago contradicts current docs
+- **Failure Mode**: Time decay may not fully resolve which is current
+- **Impact**: Potentially outdated information surfaced
+- **Frequency**: Low
+- **Mitigation**: Source-specific decay rates
+
+### 3. Time Decay Side Effects
+
+#### Recent Errors Amplified
+- **Scenario**: Recently ingested document contains errors
+- **Failure Mode**: Error gets high weight due to recency
+- **Impact**: Bad information prioritized
+- **Frequency**: Low
+- **Mitigation**: Document review processes, explicit corrections
+
+#### Historical Knowledge Lost
+- **Scenario**: Foundational documents decay over time
+- **Failure Mode**: Core knowledge deprioritized
+- **Impact**: Important background information missing
+- **Frequency**: Medium (in domains with stable knowledge)
+- **Mitigation**: Pin important documents, disable decay for foundations
+
+## Hallucination and Accuracy
+
+### Important Distinction
+- **CaaS is NOT a generative AI system**
+- **CaaS does NOT generate text or make claims**
+- **CaaS retrieves and ranks existing content**
+
+### What CaaS Does
+- Extracts actual text from real documents
+- Ranks and prioritizes based on structure, time, and source
+- Provides transparent citations for all content
+
+### What CaaS Cannot Do
+- Synthesize new information not in the corpus
+- Answer questions about events outside the ingested documents
+- Generate creative content
+
+### Accuracy Depends On
+1. **Source Quality**: Garbage in, garbage out
+2. **Structure Detection**: Better detection = better ranking
+3. **Weight Tuning**: Appropriate weights for your use case
+4. **Query Matching**: Heuristics must match your query patterns
+
+## Responsible Use Guidelines
+
+### For Organizations Deploying CaaS
+
+1. **Obtain Proper Consents**
+   - Ensure rights to process all ingested content
+   - Notify employees if processing internal communications
+   - Comply with data protection regulations (GDPR, CCPA, etc.)
+
+2. **Implement Access Controls**
+   - Not all documents should be accessible to all users
+   - Implement role-based access controls
+   - Audit access to sensitive contexts
+
+3. **Monitor for Bias**
+   - Regularly review source distribution
+   - Check for underrepresented content types
+   - Validate time decay appropriateness for your domain
+
+4. **Establish Governance**
+   - Clear policies on what can be ingested
+   - Review processes for document quality
+   - Incident response for inaccurate information
+
+5. **Provide User Training**
+   - Explain system capabilities and limitations
+   - Teach users to verify critical information
+   - Encourage feedback on result quality
+
+### For Developers Extending CaaS
+
+1. **Preserve Privacy Protections**
+   - Maintain on-premises deployment capability
+   - Avoid mandatory external API calls
+   - Respect data minimization principles
+
+2. **Maintain Transparency**
+   - Keep heuristic routing deterministic
+   - Provide clear source attribution
+   - Document all algorithmic decisions
+
+3. **Test for Bias**
+   - Evaluate on diverse document sets
+   - Check for language/format discrimination
+   - Validate across different domains
+
+4. **Document Limitations**
+   - Be clear about what your extension can/cannot do
+   - Provide guidance on appropriate use cases
+   - Warn about potential failure modes
+
+## Future Ethical Considerations
+
+As CaaS evolves, we will continue to address:
+
+1. **AI-Generated Content Detection**: How to handle documents that are themselves AI-generated
+2. **Federated Learning**: Privacy-preserving corpus analysis across organizations
+3. **Differential Privacy**: Formal privacy guarantees for sensitive documents
+4. **Fairness Metrics**: Quantitative evaluation of bias in context serving
+5. **Explainable AI**: Even better explanations of ranking decisions
+6. **Red Teaming**: Adversarial testing for misuse scenarios
+
+## Reporting Issues
+
+If you identify ethical concerns, biases, or limitations not covered here:
+
+1. Open a GitHub issue with label `ethics` or `bias`
+2. Provide specific examples and reproduction steps
+3. Suggest potential mitigations if you have ideas
+4. We commit to addressing reports within 7 days
+
+## Conclusion
+
+Context-as-a-Service is a tool, and like all tools, it can be used responsibly or irresponsibly. We've designed CaaS with transparency, privacy, and efficiency as core principles. However, the ultimate responsibility for ethical deployment lies with the organizations and individuals using the system.
+
+**Use CaaS to empower, not surveil. To inform, not manipulate. To augment human intelligence, not replace human judgment.**
+
+---
+
+*Last Updated: January 2026*
+*Version: 0.1.0*