remdb 0.3.13__tar.gz

remdb-0.3.13/.claude/archive/PUBLISH.md

@@ -0,0 +1,158 @@
+# Publishing remdb to PyPI
+
+## Prerequisites
+
+```bash
+# Install build tools
+pip install build twine
+
+# Set up PyPI credentials (use API token)
+# Create ~/.pypirc with your credentials
+```
+
+## Publishing Process
+
+### 1. Update Version
+
+Edit `pyproject.toml`:
+```toml
+version = "0.1.1"  # Increment version
+```
+
+### 2. Build Distribution
+
+```bash
+cd rem/
+
+# Clean old builds
+rm -rf dist/ build/ *.egg-info/
+
+# Build wheel and source distribution
+python -m build
+```
+
+This creates:
+- `dist/remdb-0.1.1-py3-none-any.whl`
+- `dist/remdb-0.1.1.tar.gz`
+
+### 3. Test on TestPyPI (Optional but Recommended)
+
+```bash
+# Upload to TestPyPI
+python -m twine upload --repository testpypi dist/*
+
+# Test installation from TestPyPI
+pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ remdb[all]
+```
+
+### 4. Publish to PyPI
+
+```bash
+# Upload to PyPI (uses credentials from ~/.pypirc or environment)
+python -m twine upload dist/*
+
+# Or with explicit token
+python -m twine upload dist/* --username __token__ --password $PYPI_API_KEY
+```
+
+### 5. Verify Installation
+
+```bash
+# In a fresh environment
+pip install remdb[all]
+
+# Test CLI
+rem --help
+```
+
+## Using PyPI API Token
+
+Recommended approach - use an API token instead of username/password:
+
+1. Go to https://pypi.org/manage/account/token/
+2. Create a new API token with scope for the `remdb` project
+3. Set environment variable:
+   ```bash
+   export PYPI_API_KEY="pypi-..."
+   ```
+
+4. Publish:
+   ```bash
+   python -m twine upload dist/* --username __token__ --password $PYPI_API_KEY
+   ```
+
+## Automation with GitHub Actions
+
+Create `.github/workflows/publish.yml`:
+
+```yaml
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install build tools
+        run: pip install build twine
+
+      - name: Build package
+        run: python -m build
+        working-directory: rem
+
+      - name: Publish to PyPI
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: python -m twine upload dist/*
+        working-directory: rem
+```
+
+Add `PYPI_API_TOKEN` to GitHub repository secrets.
+
+## Version Naming Convention
+
+- **0.1.x** - Alpha releases (breaking changes expected)
+- **0.2.x** - Beta releases (stabilizing API)
+- **1.0.0** - First stable release
+- **1.x.y** - Stable releases (semantic versioning)
+
+## Quick Publish Script
+
+Save as `publish.sh`:
+
+```bash
+#!/bin/bash
+set -e
+
+# Load PyPI token from environment
+if [ -z "$PYPI_API_KEY" ]; then
+    echo "Error: PYPI_API_KEY not set"
+    exit 1
+fi
+
+# Clean and build
+rm -rf dist/ build/ *.egg-info/
+python -m build
+
+# Upload to PyPI
+python -m twine upload dist/* --username __token__ --password "$PYPI_API_KEY"
+
+echo "✓ Published successfully!"
+```
+
+Make executable and run:
+```bash
+chmod +x publish.sh
+./publish.sh
+```

remdb-0.3.13/.claude/archive/QUICKSTART.md

@@ -0,0 +1,205 @@
+# REM Quickstart Guide
+
+## 🚀 Published to PyPI!
+
+**Package:** `remdb` version 0.1.9
+**PyPI:** https://pypi.org/project/remdb/
+**CLI Command:** `rem`
+
+## Installation Options
+
+### Option 1: Standalone Docker (Zero Installation)
+
+Run everything in containers - no Python installation needed.
+
+```bash
+# Clone repository
+git clone https://github.com/mr-saoirse/remstack.git
+cd remstack/rem
+
+# Set API keys
+export ANTHROPIC_API_KEY="your-key"
+export OPENAI_API_KEY="your-key"  # Optional
+
+# Start all services
+docker compose up -d
+
+# Use CLI via docker exec
+docker exec rem-api rem --help
+docker exec rem-api rem schema generate
+docker exec rem-api rem db migrate --sql-dir sql/migrations
+```
+
+**Access Points:**
+- API: http://localhost:8000
+- Docs: http://localhost:8000/docs
+- MCP: http://localhost:8000/api/v1/mcp
+- PostgreSQL: localhost:5050
+
+### Option 2: Hybrid (Recommended for Development)
+
+Docker for infrastructure, pip install for CLI/library.
+
+```bash
+# Clone repository
+git clone https://github.com/mr-saoirse/remstack.git
+cd remstack/rem
+
+# Start PostgreSQL only
+docker compose up postgres -d
+
+# Install from PyPI
+pip install remdb[all]
+
+# Set environment variables
+export ANTHROPIC_API_KEY="your-key"
+export POSTGRES__CONNECTION_STRING="postgresql://rem:rem@localhost:5050/rem"
+export AUTH__ENABLED="false"
+export OTEL__ENABLED="false"
+
+# Use CLI directly (no docker exec!)
+rem --help
+rem schema validate
+rem db migrate --sql-dir sql/migrations
+rem ask "What is REM?"
+
+# Run API locally with hot reload
+uvicorn rem.api.main:app --reload --port 8000
+```
+
+**Why Hybrid?**
+- ⚡ Faster CLI (no container overhead)
+- 🔥 Hot reload for development
+- 🐛 Native debugging
+- 🐘 Isolated PostgreSQL
+
+### Option 3: Library Usage
+
+Use REM as a Python library in your projects.
+
+```bash
+# Install in your project
+pip install remdb[all]
+```
+
+```python
+from rem.services.rem.service import RemService
+from rem.agentic.context import AgentContext
+
+# Connect to Docker PostgreSQL
+service = RemService()
+context = AgentContext(
+    user_id="demo-user",
+    tenant_id="demo-tenant"
+)
+
+# Ask questions
+result = await service.ask_rem(
+    query="What resources do we have?",
+    context=context
+)
+print(result)
+```
+
+## Testing the Installation
+
+### Verify CLI Installation
+```bash
+rem --help
+rem schema --help
+rem db --help
+```
+
+### Verify PostgreSQL Connection
+```bash
+docker exec rem-postgres psql -U rem -d rem -c "SELECT version();"
+```
+
+### Verify API Health
+```bash
+curl http://localhost:8000/health
+```
+
+## Environment Configuration
+
+All configuration via environment variables:
+
+```bash
+# LLM Configuration (Required)
+export LLM__ANTHROPIC_API_KEY="sk-ant-..."
+export LLM__OPENAI_API_KEY="sk-..."
+export LLM__DEFAULT_MODEL="anthropic:claude-sonnet-4-5-20250929"
+
+# Database (Auto-configured in docker-compose)
+export POSTGRES__CONNECTION_STRING="postgresql://rem:rem@localhost:5050/rem"
+
+# Auth & Observability (Disabled for local dev)
+export AUTH__ENABLED="false"
+export OTEL__ENABLED="false"
+
+# S3 Storage (Optional)
+export S3__BUCKET_NAME="rem-storage"
+export S3__ENDPOINT_URL="http://minio:9000"
+```
+
+## Next Steps
+
+1. **Explore API Docs:** http://localhost:8000/docs
+2. **Upload Files:** Use file processing endpoints
+3. **Create Agent Schemas:** Define custom extractors
+4. **Connect MCP Clients:** http://localhost:8000/api/v1/mcp
+5. **Run Evaluations:** `rem experiments --help`
+
+## Troubleshooting
+
+### CLI Not Found
+```bash
+# Make sure package is installed
+pip list | grep remdb
+
+# Check if CLI is in PATH
+which rem
+
+# If using venv, make sure it's activated
+source venv/bin/activate
+```
+
+### PostgreSQL Connection Failed
+```bash
+# Check if container is running
+docker compose ps
+
+# Check logs
+docker logs rem-postgres
+
+# Verify connection string
+echo $POSTGRES__CONNECTION_STRING
+```
+
+### API Not Starting
+```bash
+# Check API logs
+docker logs rem-api
+
+# Verify environment variables
+docker exec rem-api env | grep LLM
+```
+
+## Uninstall
+
+```bash
+# Stop Docker services
+docker compose down -v
+
+# Remove Python package
+pip uninstall remdb
+
+# Remove test data
+rm -rf /tmp/test-remdb
+```
+
+## Support
+
+- **Issues:** https://github.com/mr-saoirse/remstack/issues
+- **Docs:** https://github.com/mr-saoirse/remstack/blob/main/README.md
+- **PyPI:** https://pypi.org/project/remdb/

remdb-0.3.13/.claude/archive/publish.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+set -e
+
+echo "🚀 Publishing remdb to PyPI"
+echo "============================="
+
+# Check for PyPI token
+if [ -z "$PYPI_API_KEY" ]; then
+    echo "❌ Error: PYPI_API_KEY environment variable not set"
+    echo "   Export your PyPI API token:"
+    echo "   export PYPI_API_KEY='pypi-...'"
+    exit 1
+fi
+
+# Get current version from pyproject.toml
+VERSION=$(grep '^version = ' pyproject.toml | cut -d'"' -f2)
+echo "📦 Building version: $VERSION"
+
+# Clean previous builds
+echo "🧹 Cleaning old builds..."
+rm -rf dist/ build/
+
+# Build package
+echo "🔨 Building package..."
+uv run python -m build
+
+# Show built files
+echo ""
+echo "📁 Built files:"
+ls -lh dist/
+
+# Confirm before publishing
+echo ""
+read -p "⚠️  Publish remdb $VERSION to PyPI? (y/N) " -n 1 -r
+echo
+if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+    echo "❌ Publishing cancelled"
+    exit 1
+fi
+
+# Publish to PyPI
+echo "📤 Publishing to PyPI..."
+uv run python -m twine upload dist/* --username __token__ --password "$PYPI_API_KEY"
+
+echo ""
+echo "✅ Successfully published remdb $VERSION to PyPI!"
+echo "📦 Install with: pip install remdb[all]"

remdb-0.3.13/.dockerignore

@@ -0,0 +1,71 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+*.egg-info/
+dist/
+build/
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+tests/
+
+# Environment variables (should be passed via docker-compose)
+.env
+.env.*
+
+# Docker
+Dockerfile
+docker-compose.yml
+.dockerignore
+
+# Git
+.git/
+.gitignore
+
+# Documentation (keep README.md for package metadata)
+CLAUDE.md
+FILE_PROCESSING.md
+docs/
+
+# Manifests (separate from application)
+manifests/
+
+# SQL migrations (already mounted as volume)
+sql/
+
+# Temporary files
+*.log
+*.tmp
+tmp/
+temp/
+
+# macOS
+.DS_Store
+
+# CI/CD
+.github/
+.gitlab-ci.yml
+
+# Development tools
+.mypy_cache/
+.ruff_cache/

remdb-0.3.13/.env.example

@@ -0,0 +1,165 @@
+# REM Environment Variables
+# Copy this file to .env and update with your values
+#
+# Quick Start (Minimal Configuration):
+# 1. Copy: cp .env.example .env
+# 2. Set LLM API key: LLM__ANTHROPIC_API_KEY or LLM__OPENAI_API_KEY
+# 3. Database is pre-configured for Docker Compose (port 5050)
+# 4. Run: docker compose up -d
+#
+# Optional Features (Enable as needed):
+# - Vision LLM: Set CONTENT__IMAGE_VLLM_SAMPLE_RATE > 0.0
+# - CLIP Embeddings: Set CONTENT__JINA_API_KEY (free at jina.ai/embeddings)
+# - Authentication: Set AUTH__ENABLED=true + OAuth credentials
+# - Observability: Set OTEL__ENABLED=true / PHOENIX__ENABLED=true
+
+# =============================================================================
+# Development Settings
+# =============================================================================
+# Skip loading ~/.rem/config.yaml (use .env only for development)
+# This prevents conflicts between user config and development .env
+REM_SKIP_CONFIG_FILE=true
+
+# =============================================================================
+# API Server Settings
+# =============================================================================
+API__HOST=0.0.0.0
+API__PORT=8000
+API__RELOAD=true
+API__LOG_LEVEL=info
+
+# =============================================================================
+# LLM Provider Settings
+# =============================================================================
+LLM__DEFAULT_MODEL=anthropic:claude-sonnet-4-5-20250929
+LLM__DEFAULT_TEMPERATURE=0.5
+LLM__MAX_RETRIES=10
+
+# Optional: Override models for specific tasks
+# LLM__EVALUATOR_MODEL=gpt-4.1
+# LLM__QUERY_AGENT_MODEL=gpt-4o-mini
+
+# API Keys (required for LLM providers)
+LLM__OPENAI_API_KEY=
+LLM__ANTHROPIC_API_KEY=
+
+# =============================================================================
+# Database Settings (PostgreSQL 18 with pgvector)
+# =============================================================================
+# Docker Compose uses port 5050, change to 5432 for CloudNativePG/RDS
+POSTGRES__CONNECTION_STRING=postgresql://rem:rem@localhost:5050/rem
+POSTGRES__POOL_MIN_SIZE=5
+POSTGRES__POOL_MAX_SIZE=20
+POSTGRES__POOL_TIMEOUT=30
+POSTGRES__STATEMENT_TIMEOUT=30000
+
+# =============================================================================
+# Authentication Settings (OAuth 2.1 / OIDC)
+# =============================================================================
+AUTH__ENABLED=false
+AUTH__SESSION_SECRET=
+
+# --- Google OAuth ---
+AUTH__GOOGLE__CLIENT_ID=
+AUTH__GOOGLE__CLIENT_SECRET=
+AUTH__GOOGLE__REDIRECT_URI=http://localhost:8000/api/auth/google/callback
+# AUTH__GOOGLE__HOSTED_DOMAIN=example.com
+
+# --- Microsoft Entra ID OAuth ---
+AUTH__MICROSOFT__CLIENT_ID=
+AUTH__MICROSOFT__CLIENT_SECRET=
+AUTH__MICROSOFT__REDIRECT_URI=http://localhost:8000/api/auth/microsoft/callback
+AUTH__MICROSOFT__TENANT=common
+
+# =============================================================================
+# OpenTelemetry Settings
+# =============================================================================
+OTEL__ENABLED=false
+OTEL__SERVICE_NAME=rem-api
+OTEL__COLLECTOR_ENDPOINT=http://localhost:4318
+OTEL__PROTOCOL=http
+OTEL__EXPORT_TIMEOUT=10000
+
+# =============================================================================
+# Arize Phoenix Settings (LLM Observability)
+# =============================================================================
+PHOENIX__ENABLED=false
+PHOENIX__API_KEY=
+PHOENIX__COLLECTOR_ENDPOINT=http://localhost:6006/v1/traces
+PHOENIX__PROJECT_NAME=rem
+
+# =============================================================================
+# Filesystem Settings
+# =============================================================================
+# REM_HOME is the base directory for local filesystem storage
+# Defaults to ~/.rem if not set
+# Local paths: $REM_HOME/fs/rem/v1/uploads/...
+REM_HOME=~/.rem
+
+# =============================================================================
+# S3 Storage Settings
+# =============================================================================
+# For AWS S3 (production with IRSA)
+S3__BUCKET_NAME=rem-io-staging
+S3__REGION=us-east-1
+
+# Path structure version and prefixes
+S3__VERSION=v1
+S3__UPLOADS_PREFIX=uploads
+S3__PARSED_PREFIX=parsed
+
+# Convention:
+# Uploads: s3://{bucket}/{version}/uploads/user-123/2025/01/19/file.pdf
+# Parsed:  s3://{bucket}/{version}/parsed/user-123/2025/01/19/file.pdf/content.md
+
+# For MinIO (local development)
+# S3__ENDPOINT_URL=http://localhost:9000
+# S3__ACCESS_KEY_ID=minioadmin
+# S3__SECRET_ACCESS_KEY=minioadmin
+# S3__USE_SSL=false
+
+# =============================================================================
+# Content Processing Settings
+# =============================================================================
+
+# --- Document Chunking ---
+CHUNKING__CHUNK_SIZE=1500
+CHUNKING__OVERLAP=200
+CHUNKING__MIN_CHUNK_SIZE=100
+CHUNKING__MAX_CHUNK_SIZE=2500
+
+# --- Vision LLM (Image Analysis) ---
+# Expensive - tier-gated or sampled
+CONTENT__IMAGE_VLLM_PROVIDER=anthropic
+CONTENT__IMAGE_VLLM_MODEL=
+CONTENT__IMAGE_VLLM_SAMPLE_RATE=0.0
+
+# --- CLIP Embeddings (Image Search) ---
+# Cheap - always-on when key provided (~$0.00008 per image)
+CONTENT__CLIP_PROVIDER=jina
+CONTENT__CLIP_MODEL=jina-clip-v2
+CONTENT__JINA_API_KEY=
+
+# =============================================================================
+# SQS Queue Settings (File Processing)
+# =============================================================================
+# Queue URL from Pulumi output (manifests/infra/file-queue)
+SQS__QUEUE_URL=https://sqs.us-east-1.amazonaws.com/ACCOUNT_ID/rem-file-processing
+SQS__REGION=us-east-1
+SQS__MAX_MESSAGES=10
+SQS__WAIT_TIME_SECONDS=20
+SQS__VISIBILITY_TIMEOUT=300
+
+# =============================================================================
+# MCP Server Settings
+# =============================================================================
+# MCP_SERVER_TEST=http://localhost:8000/api/v1/mcp
+# MCP_SERVER_PROD=https://api.example.com/api/v1/mcp
+
+# =============================================================================
+# Application Settings
+# =============================================================================
+ENVIRONMENT=development
+TEAM=rem
+# DOMAIN=https://api.example.com
+# ROOT_PATH=/rem