ragtrace 0.2.0__tar.gz

ragtrace-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 RAG Debugger Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ragtrace-0.2.0/PKG-INFO

@@ -0,0 +1,403 @@
+Metadata-Version: 2.4
+Name: ragtrace
+Version: 0.2.0
+Summary: Observability and tracing for Retrieval-Augmented Generation pipelines
+License-File: LICENSE
+Author: Sabyasachi Ghosh
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: click (>=8.1.0,<9.0.0)
+Requires-Dist: fastapi (>=0.109.0,<0.110.0)
+Requires-Dist: langchain (>=0.1.0,<0.2.0)
+Requires-Dist: langchain-community (>=0.0.20,<0.0.21)
+Requires-Dist: langchain-openai (>=0.0.5,<0.0.6)
+Requires-Dist: openai (>=1.0.0,<2.0.0)
+Requires-Dist: pydantic (>=2.5.0,<3.0.0)
+Requires-Dist: rich (>=13.7.0,<14.0.0)
+Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0)
+Requires-Dist: tiktoken (>=0.5.0,<0.6.0)
+Requires-Dist: uvicorn[standard] (>=0.27.0,<0.28.0)
+Description-Content-Type: text/markdown
+
+# RAGTrace 📊
+
+> **Observability for RAG pipelines** - Trace, inspect, and optimize Retrieval-Augmented Generation systems with ease.
+
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+## ✨ What is RAGTrace?
+
+RAGTrace is a lightweight observability layer for RAG (Retrieval-Augmented Generation) systems that captures and visualizes every step of your pipeline:
+
+- 🔍 **Event Capture** - Automatically intercepts retrieval, prompt, and generation events
+- 💰 **Cost Tracking** - Accurate token counting and cost estimation per query
+- 📊 **Interactive Web UI** - Modern timeline view with charts, filters, and event inspection
+- 🔧 **CLI Tool** - Developer-friendly command-line interface
+- 🌐 **REST API** - Query and analyze sessions programmatically
+- 🧪 **Regression Testing** - Snapshot and compare RAG outputs
+
+**Think of it as OpenTelemetry, but specifically for RAG pipelines.**
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/ragtrace.git
+cd ragtrace
+
+# Install dependencies (using pip)
+pip install -e .
+
+# Or using Poetry
+poetry install
+poetry shell
+
+# Initialize database
+ragtrace init
+```
+
+### Basic Usage
+
+```python
+from langchain.vectorstores import FAISS
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.chat_models import ChatOpenAI
+from langchain.chains import RetrievalQA
+from ragtrace import RagTracer
+
+# Your existing RAG setup
+embeddings = OpenAIEmbeddings()
+vectorstore = FAISS.from_texts(["Your documents here..."], embeddings)
+llm = ChatOpenAI(model="gpt-3.5-turbo")
+
+# Add RAGTrace - just one line!
+tracer = RagTracer(auto_save=True)
+
+# Create chain with tracer
+chain = RetrievalQA.from_chain_type(
+    llm=llm,
+    retriever=vectorstore.as_retriever(),
+    callbacks=[tracer]  # ← Automatic capture!
+)
+
+# Run your query (automatically captured)
+result = chain.run("What is RAG?")
+```
+
+### View Results
+
+```bash
+# View latest session in CLI
+ragtrace show last
+
+# List all sessions
+ragtrace list
+
+# Export to JSON
+ragtrace export <session-id> > session.json
+
+# Start API server (for Web UI)
+ragtrace run
+
+# Or start both servers for full Web UI experience
+uvicorn api.main:app --port 8000 &  # API server
+python ui/serve.py                   # UI server → http://localhost:3000
+```
+
+## 🌐 Web UI
+
+RAGTrace includes a modern web interface for visualizing and analyzing your RAG pipelines:
+
+### Start the Servers
+
+```bash
+# Terminal 1: Start API server
+cd ragtrace
+source venv/bin/activate
+uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload
+
+# Terminal 2: Start UI server
+cd ragtrace
+python ui/serve.py
+```
+
+Then open **http://localhost:3000** in your browser.
+
+### Web UI Features
+
+- **📋 Sessions View** - Browse all captured RAG sessions with search and filtering
+- **📊 Timeline View** - Interactive timeline showing retrieval → prompt → generation flow
+- **📈 Performance Charts** - Waterfall chart for event durations, cost breakdown by component
+- **🔍 Event Inspector** - Click any event to see full details including tokens, costs, and data
+- **🎯 Smart Filters** - Filter events by type, duration, and cost
+- **📤 Export Tools** - Export session data as JSON or CSV, copy to clipboard
+
+### Quick UI Guide
+
+1. **Sessions Tab**: View all RAG sessions, sorted by recent activity
+2. **Timeline Tab**: Click on any session to see detailed event timeline
+3. **Click Events**: Click timeline events to inspect full details
+4. **Apply Filters**: Use dropdowns to filter by event type or performance metrics
+5. **Export Data**: Use export buttons to download session data
+
+
+## 📊 Example Output
+
+```
+╭─ Session: d4f3a8b2-... ─────────────────────────────────╮
+│ Query: What is RAG?                                      │
+│ Model: gpt-3.5-turbo                                     │
+│ Cost: $0.00360                                           │
+│ Duration: 1,850ms                                        │
+╰──────────────────────────────────────────────────────────╯
+
+┏━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
+┃ Event       ┃ Duration   ┃ Cost       ┃
+┡━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
+│ Retrieval   │ 150ms      │ $0.00001   │
+│ Prompt      │ 0ms        │ $0.00000   │
+│ Generation  │ 1,700ms    │ $0.00359   │
+└─────────────┴────────────┴────────────┘
+```
+
+## 🎯 Features
+
+### ✅ Core Features
+
+- **Automatic Event Capture** - Works with LangChain callbacks
+- **Cost Tracking** - Uses tiktoken for accurate token counting
+- **Timeline Visualization** - See your RAG pipeline in action
+- **Session Management** - Store and retrieve debugging sessions
+- **CLI Tool** - Rich formatted terminal output
+- **REST API** - FastAPI server with OpenAPI docs
+- **Web UI** - Interactive timeline with charts and event inspection
+- **JSON/CSV Export** - Export sessions for analysis
+- **Snapshot Testing** - Save and compare pipeline outputs
+
+### 🎨 Web UI Features
+
+- **Sessions Dashboard** - View all sessions with search and sort
+- **Interactive Timeline** - Click events to inspect details
+- **Performance Waterfall** - Visualize event durations side-by-side
+- **Cost Breakdown Chart** - See spending by component (embeddings, prompts, generation)
+- **Smart Filters** - Filter by event type, duration, or cost
+- **Real-time Updates** - WebSocket support for live session monitoring
+- **Export Tools** - Download as JSON, CSV, or copy to clipboard
+
+### 🎨 CLI Commands
+
+```bash
+ragtrace init              # Initialize database
+ragtrace list              # List recent sessions
+ragtrace show [id]         # View session details
+ragtrace show last         # View latest  session
+ragtrace export <id>       # Export to JSON
+ragtrace clear             # Clear all data
+ragtrace snapshot save     # Save snapshot
+ragtrace snapshot list     # List snapshots
+ragtrace run               # Start API server
+```
+
+### 🌐 API Endpoints
+
+```
+POST   /api/sessions                      # Create session
+GET    /api/sessions                      # List sessions
+GET    /api/sessions/{id}                 # Get session
+PATCH  /api/sessions/{id}                 # Update session
+DELETE /api/sessions/{id}                 # Delete session
+POST   /api/sessions/{id}/events          # Add event
+GET    /api/sessions/{id}/costs           # Get costs
+POST   /api/snapshots                     # Create snapshot
+GET    /api/snapshots                     # List snapshots
+GET    /api/snapshots/{id1}/compare/{id2} # Compare snapshots
+```
+
+Visit `http://localhost:8000/docs` after running `ragtrace run` for interactive API documentation.
+
+## 🏗️ Architecture
+
+```
+ragtrace/
+├── core/              # Core business logic
+│   ├── models.py      # Pydantic data models
+│   ├── storage.py     # SQLite database layer
+│   ├── cost.py        # Token counting & cost calculation
+│   └── capture.py     # Event aggregation
+├── langchain/         # LangChain integration
+│   └── middleware.py  # Callback handler
+├── api/               # REST API (FastAPI)
+│   ├── main.py        # FastAPI application
+│   └── routes.py      # API endpoints
+├── ui/                # Web UI (HTML/CSS/JS)
+│   ├── index.html     # Main UI page
+│   ├── app.js         # Frontend application logic
+│   ├── styles.css     # UI styling
+│   └── serve.py       # Development server
+├── cli/               # Command-line interface
+│   └── main.py        # Click CLI commands
+├── examples/          # Usage examples
+│   └── simple_rag.py  # Complete working example
+└── tests/             # Test suite
+    ├── test_cost.py   # Cost calculation tests
+    ├── test_storage.py # Database tests
+    └── test_capture.py # Capture logic tests
+```
+
+## 📋 Requirements
+
+- **Python**: 3.11+ (3.12 recommended)
+- **Dependencies**: FastAPI, LangChain, tiktoken, Rich, Click
+- **OpenAI API Key**: Required for examples (not for core functionality)
+
+## 🛠️ Development
+
+### Setup Development Environment
+
+```bash
+# Clone repository
+git clone https://github.com/yourusername/ragtrace.git
+cd ragtrace
+
+# Install with development dependencies
+poetry install
+poetry shell
+
+# Run tests
+pytest
+
+# Run with coverage
+pytest --cov=core --cov=langchain --cov=api --cov=cli
+```
+
+### Running Tests
+
+```bash
+# All tests
+pytest
+
+# Specific test file
+pytest tests/test_cost.py -v
+
+# With coverage report
+pytest --cov=core tests/
+```
+
+### Code Quality
+
+```bash
+# Format code
+black .
+
+# Lint
+ruff check .
+
+# Type checking
+mypy .
+```
+
+## 📖 Examples
+
+Check out the `examples/` directory for complete working examples:
+
+- **`simple_rag.py`** - Basic RAG pipeline with debugger
+- **`with_sources.py`** - RAG with source tracking (coming soon)
+
+## 🔬 Use Cases
+
+### 1. Debug Failed Queries
+```bash
+# See exactly why your RAG pipeline failed
+ragtrace show last
+```
+
+### 2. Track Costs
+```bash
+# Monitor spending per query
+ragtrace list --sort-by cost
+```
+
+### 3. Identify Retrieval Issues
+```python
+# Check which documents were retrieved
+session = tracer.get_latest_session()
+print(session.retrieval_event.chunks)
+```
+
+### 4. Regression Testing
+```bash
+# Save baseline
+ragtrace snapshot save "v1-baseline"
+
+# Compare after changes
+ragtrace snapshot compare <snapshot-id-1> <snapshot-id-2>
+```
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+## 📝 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## 🙏 Acknowledgments
+
+- Built with [FastAPI](https://fastapi.tiangolo.com/)
+- Integrated with [LangChain](https://python.langchain.com/)
+- Token counting via [tiktoken](https://github.com/openai/tiktoken)
+- Beautiful CLI with [Rich](https://rich.readthedocs.io/)
+
+## 📞 Support
+
+- **Issues**: [GitHub Issues](https://github.com/yourusername/ragtrace/issues)
+- **Documentation**: See [examples/](examples/) directory
+- **API Docs**: Run `ragtrace run` and visit `http://localhost:8000/docs`
+
+## 🗺️ Roadmap
+
+### v0.2.0 (Current)
+- [x] Web UI for timeline visualization
+- [x] Interactive event inspection
+- [x] Performance charts (waterfall, cost breakdown)
+- [x] Export tools (JSON, CSV)
+- [ ] Advanced regression testing
+- [ ] LlamaIndex integration
+- [ ] Prompt versioning
+
+### v0.3.0 (Planned)
+- [ ] Agent tracing support
+- [ ] Cost optimization suggestions
+- [ ] Quality scoring
+- [ ] Team collaboration features
+
+### v1.0.0
+- [ ] Cloud mode
+- [ ] Advanced analytics
+- [ ] Alert system
+- [ ] Multi-framework support
+
+## ⭐ Star History
+
+If you find RAGTrace useful, please consider giving it a star! ⭐
+
+---
+
+**Built with ❤️ for RAG developers**
+