ai-pipeline-core 0.1.1__tar.gz

ai_pipeline_core-0.1.1/.gitignore

@@ -0,0 +1,158 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+Pipfile.lock
+
+# poetry
+poetry.lock
+
+# pdm
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+.idea/
+
+# VS Code
+.vscode/
+
+# macOS
+.DS_Store
+
+# Windows
+Thumbs.db
+ehthumbs.db
+
+# Prefect
+.prefect/
+
+# Test artifacts
+tests/test_data/
+test_output/

ai_pipeline_core-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 bbarwik@gmail.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ai_pipeline_core-0.1.1/PKG-INFO

@@ -0,0 +1,477 @@
+Metadata-Version: 2.4
+Name: ai-pipeline-core
+Version: 0.1.1
+Summary: Core utilities for AI-powered processing pipelines using prefect
+Project-URL: Homepage, https://github.com/bbarwik/ai-pipeline-core
+Project-URL: Repository, https://github.com/bbarwik/ai-pipeline-core
+Project-URL: Issues, https://github.com/bbarwik/ai-pipeline-core/issues
+Author-email: bbarwik <bbarwik@gmail.com>
+License: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: jinja2>=3.1.6
+Requires-Dist: lmnr>=0.7.4
+Requires-Dist: openai>=1.99.9
+Requires-Dist: prefect>=3.4.13
+Requires-Dist: pydantic-settings>=2.10.1
+Requires-Dist: pydantic>=2.11.7
+Requires-Dist: python-magic>=0.4.27
+Requires-Dist: ruamel-yaml>=0.18.14
+Requires-Dist: tiktoken>=0.11.0
+Provides-Extra: dev
+Requires-Dist: basedpyright>=1.31.2; extra == 'dev'
+Requires-Dist: bump2version>=1.0.1; extra == 'dev'
+Requires-Dist: pre-commit>=4.3.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=1.1.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
+Requires-Dist: pytest-mock>=3.14.0; extra == 'dev'
+Requires-Dist: pytest-xdist>=3.8.0; extra == 'dev'
+Requires-Dist: pytest>=8.4.1; extra == 'dev'
+Requires-Dist: ruff>=0.12.9; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# AI Pipeline Core
+
+A high-performance, type-safe Python library for building AI-powered data processing pipelines with Prefect orchestration and LMNR observability.
+
+[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
+[![Type Checked: Basedpyright](https://img.shields.io/badge/type%20checked-basedpyright-blue)](https://github.com/DetachHead/basedpyright)
+[![Test Coverage](https://img.shields.io/badge/coverage-80%25-green)](https://github.com/bbarwik/ai-pipeline-core)
+[![Status: Beta](https://img.shields.io/badge/status-beta-yellow)](https://github.com/bbarwik/ai-pipeline-core)
+[![PyPI version](https://img.shields.io/pypi/v/ai-pipeline-core.svg)](https://pypi.org/project/ai-pipeline-core/)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/ai-pipeline-core.svg)](https://pypi.org/project/ai-pipeline-core/)
+
+> [!NOTE]
+> **Beta Release**
+>
+> This library is in beta. While actively used in production systems, the API may still evolve. We follow semantic versioning for releases.
+
+## Overview
+
+AI Pipeline Core provides a robust foundation for building production-grade AI pipelines with a focus on:
+
+- **100% Async Architecture** - Built for high-throughput, non-blocking operations
+- **Type Safety** - Comprehensive type hints with Pydantic models throughout
+- **Minimal Design** - Every line of code justified, no unnecessary abstractions
+- **Production Ready** - Built-in retry logic, caching, monitoring, and error handling
+- **LLM Optimization** - Smart context/message splitting for efficient token usage
+
+## Key Features
+
+### 🚀 Performance First
+- Fully asynchronous I/O operations
+- Intelligent caching for LLM context
+- Streaming support for large documents
+- Automatic retry with exponential backoff
+
+### 🔒 Type Safety
+- Pydantic models for all data structures
+- Strict type checking with basedpyright
+- Runtime validation for all inputs
+- Immutable configurations by default
+
+### 📊 Observability
+- LMNR (Laminar) tracing integration
+- Structured logging with Prefect
+- Cost tracking for LLM operations
+- Performance metrics out of the box
+
+### 🎯 Developer Experience
+- Self-documenting code for experienced developers
+- Consistent patterns throughout
+- Comprehensive error messages
+- Smart defaults with override capabilities
+
+### 🤖 Advanced LLM Features
+- Search-enabled models (Perplexity Sonar, Gemini Flash Search)
+- Reasoning models support (O1 series)
+- Structured output with Pydantic models
+- Dynamic model selection based on task
+
+## Installation
+
+```bash
+pip install ai-pipeline-core
+```
+
+### Development Installation
+
+For contributors and development:
+
+```bash
+git clone https://github.com/bbarwik/ai-pipeline-core.git
+cd ai-pipeline-core
+pip install -e ".[dev]"
+make install-dev  # Installs pre-commit hooks
+```
+
+### Requirements
+- Python 3.12 or higher
+- Linux/macOS (Windows via WSL2)
+
+## Quick Start
+
+### Basic Document Processing
+```python
+from ai_pipeline_core.documents import Document, FlowDocument
+from ai_pipeline_core.llm import generate_structured, AIMessages, ModelOptions
+from pydantic import BaseModel
+
+class InputDocument(FlowDocument):
+    """Custom document type for your flow"""
+    def get_type(self) -> str:
+        return "input"
+
+class AnalysisResult(BaseModel):
+    """Example Pydantic model for structured output"""
+    summary: str
+    key_points: list[str]
+
+async def process_document(doc: Document):
+    # Generate AI response with structured output
+    response = await generate_structured(
+        model="gemini-2.5-pro",  # Model is required first parameter
+        response_format=AnalysisResult,  # Pydantic model class
+        context=AIMessages([doc]),  # Cached context
+        messages=AIMessages(["Analyze this document"]),  # Dynamic messages
+        options=ModelOptions(max_completion_tokens=5000)  # Optional options
+    )
+    return response.parsed
+```
+
+### Prefect Flow Integration
+```python
+from prefect import flow, task
+from ai_pipeline_core.documents import Document, DocumentList, FlowDocument
+from ai_pipeline_core.flow import FlowConfig
+from ai_pipeline_core.tracing import trace
+
+class OutputDocument(FlowDocument):
+    """Custom output document type"""
+    def get_type(self) -> str:
+        return "output"
+
+class MyFlowConfig(FlowConfig):
+    INPUT_DOCUMENT_TYPES = [InputDocument]
+    OUTPUT_DOCUMENT_TYPE = OutputDocument
+
+@task
+@trace
+async def process_task(doc: Document) -> Document:
+    # Task-level processing with automatic tracing
+    result = await process_document(doc)
+    # Convert result to JSON string for document content
+    import json
+    return OutputDocument(name="result", content=json.dumps(result.model_dump()).encode())
+
+@flow
+async def my_pipeline(documents: DocumentList):
+    config = MyFlowConfig()
+    input_docs = config.get_input_documents(documents)
+
+    results = await process_task.map(input_docs)
+
+    config.validate_output_documents(results)
+    return results
+```
+
+## Core Modules
+
+### Documents System
+The foundation for all data handling. Documents are immutable, type-safe wrappers around content with automatic MIME type detection.
+
+```python
+from ai_pipeline_core.documents import Document, DocumentList
+
+# Documents handle encoding/decoding automatically
+doc = MyDocument(
+    name="report.pdf",
+    content=pdf_bytes,
+    description="Q3 Financial Report"
+)
+
+# Type-safe document collections
+docs = DocumentList([doc1, doc2])
+```
+
+### LLM Module
+Managed AI interactions with built-in retry logic, cost tracking, and structured outputs.
+
+**Supported Models** (via LiteLLM proxy):
+- OpenAI: GPT-4, GPT-5 series
+- Anthropic: Claude 3 series
+- Google: Gemini 2.5 series
+- xAI: Grok models
+- Perplexity: Sonar models (with search capabilities)
+- And many more through LiteLLM compatibility
+
+```python
+from ai_pipeline_core.llm import generate_structured, AIMessages, ModelOptions
+from pydantic import BaseModel
+
+class YourPydanticModel(BaseModel):
+    field1: str
+    field2: int
+
+# Get structured Pydantic model responses
+result = await generate_structured(
+    model="gemini-2.5-pro",  # Model is required first parameter
+    response_format=YourPydanticModel,  # Pydantic model class for structured output
+    context=AIMessages(),  # Optional context (cached)
+    messages=AIMessages(["Your prompt here"]),  # Required messages
+    options=ModelOptions(
+        retries=3,
+        timeout=30,
+        max_completion_tokens=10000
+    )
+)
+# Access the parsed result
+model_instance = result.parsed  # Type: YourPydanticModel
+```
+
+### Prompt Management
+Flexible Jinja2-based prompt system with smart path resolution.
+
+```python
+from ai_pipeline_core import PromptManager
+
+pm = PromptManager(__file__)
+prompt = pm.get("analyze_document.jinja2",
+                 document=doc,
+                 instructions=instructions)
+```
+
+### Tracing & Monitoring
+Automatic observability with LMNR integration.
+
+```python
+from ai_pipeline_core.tracing import trace
+
+@trace(metadata={"workflow": "analysis"})
+async def analyze_data(data: InputData) -> OutputData:
+    # Automatic tracing with performance metrics
+    ...
+```
+
+## Architecture Principles
+
+### 1. Async-First Design
+Every I/O operation is asynchronous. No blocking calls, no synchronous fallbacks.
+
+### 2. Type Safety Throughout
+Complete type annotations with runtime validation. If it compiles, it works.
+
+### 3. Minimal Surface Area
+Less code is better code. Every line must justify its existence.
+
+### 4. Configuration as Code
+All configurations are Pydantic models - validated, typed, and immutable.
+
+## Project Structure
+
+```
+ai_pipeline_core/
+├── documents/          # Document handling system
+│   ├── document.py     # Base document class
+│   ├── flow_document.py # Prefect flow documents
+│   └── task_document.py # Prefect task documents
+├── llm/               # LLM interaction layer
+│   ├── client.py      # Async client implementation
+│   └── model_options.py # Configuration models
+├── flow/              # Prefect flow utilities
+│   └── config.py      # Type-safe flow configuration
+├── logging/           # Structured logging
+├── tracing.py         # Observability decorators
+└── settings.py        # Centralized configuration
+```
+
+## Development
+
+### Running Tests
+```bash
+make test           # Run all tests
+make test-cov      # Run with coverage report
+pytest tests/test_documents.py::TestDocument::test_creation  # Single test
+```
+
+### Code Quality
+```bash
+make lint          # Run linting checks
+make format        # Auto-format code
+make typecheck     # Run type checking
+make pre-commit    # Run all pre-commit hooks
+```
+
+### Development Workflow
+1. Create feature branch
+2. Write tests first (TDD)
+3. Implement minimal solution
+4. Run `make format` and `make typecheck`
+5. Ensure >80% test coverage
+6. Submit PR with clear description
+
+## Best Practices
+
+### DO ✅
+- Use async/await for all I/O operations
+- Define Pydantic models for all data structures
+- Keep functions under 20 lines
+- Use type hints for everything
+- Let Documents handle serialization
+
+### DON'T ❌
+- Import `logging` directly (use pipeline logger)
+- Use raw dictionaries for configuration
+- Write defensive code for unlikely scenarios
+- Add comments explaining what (code should be clear)
+- Use `requests` or other blocking libraries
+
+## Configuration
+
+### Environment Variables
+```bash
+# Required for LLM operations
+OPENAI_API_KEY=sk-...  # Your OpenAI or LiteLLM proxy key
+OPENAI_BASE_URL=http://your-proxy:8000  # LiteLLM proxy endpoint
+
+# Optional - for observability
+LMNR_PROJECT_API_KEY=lmnr_...  # LMNR tracing
+
+# Optional - for orchestration
+PREFECT_API_URL=http://localhost:4200/api
+AI_PIPELINE_LOG_LEVEL=INFO
+```
+
+### Settings Management
+```python
+from ai_pipeline_core.settings import settings
+
+# All settings are validated Pydantic models
+api_key = settings.openai_api_key
+base_url = settings.openai_base_url  # LiteLLM proxy endpoint
+```
+
+## Integration Examples
+
+### With Prefect Cloud
+```python
+from prefect import flow
+from ai_pipeline_core.flow import FlowConfig
+
+@flow(name="document-processor")
+async def process_documents(docs: DocumentList):
+    # Automatic Prefect Cloud integration
+    ...
+```
+
+### With Custom LLM Providers
+```python
+from ai_pipeline_core.settings import settings
+
+# Configure LiteLLM proxy endpoint via environment variables
+# OPENAI_BASE_URL=http://your-litellm-proxy:8000
+# OPENAI_API_KEY=your-proxy-key
+
+# Access in code (settings are immutable)
+base_url = settings.openai_base_url
+```
+
+## Performance Considerations
+
+- **Context Caching**: The LLM module automatically caches context to reduce token usage
+- **Document Streaming**: Large documents are streamed rather than loaded entirely into memory
+- **Batch Processing**: Use Prefect's `.map()` for parallel task execution
+- **Connection Pooling**: HTTP clients use connection pooling by default
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Import Errors**: Ensure Python 3.12+ is installed
+2. **Async Warnings**: All I/O operations must use `await`
+3. **Type Errors**: Run `make typecheck` to identify issues
+4. **MIME Detection**: Install `python-magic` system dependencies
+
+### Debug Mode
+```python
+from ai_pipeline_core.logging import setup_logging, LoggingConfig
+
+# Setup logging with DEBUG level
+setup_logging(LoggingConfig(level="DEBUG"))
+```
+
+## Release Process
+
+See [RELEASE.md](RELEASE.md) for detailed release procedures.
+
+**Important**: All releases require:
+- ✅ Zero errors from `make typecheck`
+- ✅ All unit tests passing with >80% coverage
+- ✅ **Integration tests passing** (with configured API keys)
+
+## Contributing
+
+> [!NOTE]
+> As this is a preview repository used internally, we are not actively accepting external contributions. The codebase may change significantly without notice.
+>
+> **Recommended approach:**
+> 1. Fork the repository
+> 2. Make changes in your fork
+> 3. Share your improvements with the community through your fork
+
+If you've found a critical security issue, please report it via the GitHub Security tab.
+
+For learning purposes, see [CLAUDE.md](CLAUDE.md) for our comprehensive coding standards and architecture guide.
+
+## Documentation
+
+- [CLAUDE.md](CLAUDE.md) - Detailed coding standards and architecture guide
+- [Prefect Integration](docs/prefect.md) - Prefect patterns and best practices
+- [Deployment Guide](docs/prefect_deployment.md) - Production deployment
+- [Prefect Logging](docs/prefect_logging.md) - Logging configuration guide
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Support
+
+> [!CAUTION]
+> This is a preview repository with no guaranteed support. Issues and discussions may not be actively monitored.
+
+- **For Learning**: Review the code, documentation, and examples
+- **For Usage**: Fork the repository and maintain your own version
+- **Security Issues**: Report via GitHub Security tab
+
+## Acknowledgments
+
+Built with:
+- [Prefect](https://www.prefect.io/) - Workflow orchestration
+- [LMNR](https://www.lmnr.ai/) - LLM observability
+- [LiteLLM](https://litellm.ai/) - LLM proxy
+- [Pydantic](https://pydantic-docs.helpmanual.io/) - Data validation
+
+## Stability Notice
+
+**Current Version**: 0.1.1
+**Status**: Internal Preview
+**API Stability**: Unstable - Breaking changes expected
+**Recommended Use**: Learning and reference only
+
+For production use, please fork this repository and maintain your own stable version.
+
+---
+
+**Remember**: The best code is no code. The second best is minimal, typed, async code that does exactly what's needed.