clio-agent 0.5.0__tar.gz

clio_agent-0.5.0/.claude/CLAUDE.md

@@ -0,0 +1,392 @@
+# CLIO Agent Development Rules for AI Agents
+
+Essential rules and guidelines for AI agents implementing CLIO Agent.
+
+---
+
+## Core Principles
+
+**CLIO Agent IS**: Self-improving autonomous agent for scientific data (not a framework)
+**Architecture**: 3-Tier Orchestration + ARC Memory + Optimizer Layer + IOWarp Integration
+**Internal Engine**: DSPy 3.x (signatures, modules, optimizers) + FastMCP 3.x (tool servers)
+
+---
+
+## Locked-Down Stack
+
+### Required Dependencies
+
+```python
+# Core
+"dspy-ai>=3.1.0"             # Agent patterns + optimizers (INTERNAL)
+"fastmcp>=3.0.0"             # MCP protocol + gateway composition
+
+# UI
+"rich>=14.2.0"               # Terminal UI
+
+# Memory Layer
+"sortedcontainers>=2.4.0"    # B-tree index
+"lru-dict>=1.3.0"            # LRU cache
+"msgspec>=0.18.0"            # Serialization
+
+# Optimizer Layer (Phase 3+)
+"scipy>=1.11.0"              # Statistical tests
+"numpy>=1.24.0"              # Numerical ops
+
+# Tools (Phase 1+)
+"h5py>=3.10.0"               # HDF5 MCP server
+"pyarrow>=14.0.0"            # Parquet MCP server (Phase 2+)
+```
+
+### Forbidden
+
+**DO NOT add** without approval:
+- LangChain, CrewAI, AutoGen as core dependencies (external integration only via A2A)
+- Heavy ML frameworks (TensorFlow, PyTorch)
+- Database ORMs (SQLAlchemy)
+- Custom async/sync bridge code (use native DSPy/FastMCP instead)
+
+**Python Version**: >= 3.12 (LOCKED)
+
+---
+
+## Critical Rules (NON-NEGOTIABLE)
+
+### RULE 1: Follow PLAN.md Phase Order
+- Implement phases sequentially: Phase 1 -> Phase 2 -> ... -> Phase 6
+- Complete tasks within a phase before moving to next
+- Never skip phases or cherry-pick tasks from later phases
+
+### RULE 2: Never Break Baseline
+- Main agent, DataExpert, CLI must always work
+- Test before committing: `uv run src/clio_agent/ui/cli.py`
+- If a change breaks baseline, revert first, then fix
+
+### RULE 3: DSPy is Internal Implementation Detail
+- Use DSPy internally for signatures, modules, optimizers
+- **DO NOT expose** DSPy in user-facing docs, APIs, or error messages
+- **Exception**: CLAUDE.md and code comments only
+
+### RULE 4: All Data Goes Through ARC
+- Store conversations, invocations, metrics in ARC
+- **DO NOT create** separate storage systems, databases, or caches
+
+### RULE 5: Tool Curation (Max 5-7 Per Expert)
+- Each expert gets 5-7 high-level curated tools, not every atomic operation
+- Hide implementation complexity behind composite tools
+- Document each tool with an "agent story" (when/why an agent would use it)
+- **DO NOT auto-generate** tools from OpenAPI specs or file system scans
+
+### RULE 6: Context is Compiled, Not Concatenated
+- Never dump raw conversation history into prompts
+- Use context compilation pipeline: filter -> compact -> enrich -> assemble
+- Set context budgets per tier (T1: 2K tokens, T2: 4K tokens)
+- **DO NOT concatenate** all ARC data into a single string
+
+### RULE 7: Performance Targets
+- Cache hit rate > 85%
+- ARC retrieval < 10ms (O(log N))
+- Tool cache hit rate > 50%
+
+### RULE 8: Test Coverage > 80% (Phase 4+)
+- Unit tests for all new code
+- Integration tests for critical paths
+- Use `Client(server)` for in-memory MCP server testing
+- Run `pytest tests/` before committing
+
+### RULE 9: Type Hints + Docstrings
+- Type hints on all functions
+- Google-style docstrings
+- Use `Literal` types for routing decisions
+
+---
+
+## DSPy 3.x Patterns (Use These)
+
+### ChatAdapter for Local Models
+```python
+# Makes ReAct work with LM Studio, Ollama, any chat model
+import dspy
+lm = dspy.LM("openai/model-name", api_base="http://127.0.0.1:1234/v1")
+dspy.configure(lm=lm, adapter=dspy.ChatAdapter())
+```
+
+### Tool.from_mcp_tool() for MCP Bridge
+```python
+# Native bridge: MCP tool -> DSPy tool (replaces mcp_connector.py)
+from fastmcp import Client
+client = Client(server)
+async with client:
+    mcp_tools = await client.list_tools()
+    dspy_tools = [dspy.Tool.from_mcp_tool(t) for t in mcp_tools]
+    agent = dspy.ReAct(signature, tools=dspy_tools)
+```
+
+### Per-Request Model Selection
+```python
+# Use dspy.context() instead of global dspy.configure()
+with dspy.context(lm=expert_model):
+    result = expert(question=query)
+```
+
+### SIMBA for Agentic Optimization
+```python
+# Designed for multi-step agent tasks (not just single predictions)
+optimizer = dspy.SIMBA(metric=success_metric, max_steps=50)
+optimized_agent = optimizer.compile(agent, trainset=examples)
+```
+
+### Typed Outputs
+```python
+# Use Literal for routing decisions (optimizable by DSPy)
+class RoutingSignature(dspy.Signature):
+    question: str = dspy.InputField()
+    selected_expert: Literal["data", "hpc", "none"] = dspy.OutputField()
+```
+
+---
+
+## FastMCP 3.x Patterns (Use These)
+
+### Gateway with mount()
+```python
+from fastmcp import FastMCP
+gateway = FastMCP("clio-gateway")
+gateway.mount("/hdf5", hdf5_server)
+gateway.mount("/parquet", parquet_server)
+# Tools namespaced: hdf5_list_datasets, parquet_analyze_schema
+```
+
+### In-Memory Testing
+```python
+# Test MCP servers without subprocess or network
+from fastmcp import Client
+async def test_hdf5_analyze():
+    async with Client(hdf5_server) as client:
+        result = await client.call_tool("analyze_dataset", {"filepath": "test.h5"})
+        assert result is not None
+```
+
+### Dependency Injection
+```python
+from fastmcp import FastMCP, Depends
+
+def get_arc_memory():
+    return ARCMemory()
+
+@mcp.tool()
+def analyze(filepath: str, arc: ARCMemory = Depends(get_arc_memory)) -> dict:
+    # arc is injected, hidden from LLM tool schema
+    cached = arc.get_cached_tool_result("hdf5", "analyze", {"filepath": filepath})
+    if cached:
+        return cached
+    # ... actual analysis
+```
+
+### Transforms for Access Control
+```python
+from fastmcp.transforms import Enabled
+# Only expose tools matching a condition
+gateway.mount("/admin", admin_server, transforms=[Enabled(lambda t: user.is_admin)])
+```
+
+---
+
+## Architecture DOs and DONTs
+
+### 3-Tier Hierarchy
+- **DO**: Tier 1 (Main) -> Tier 2 (Experts) -> Tier 3 (Nanoagents via dspy.Parallel)
+- **DON'T**: Skip tiers, mix responsibilities, or have experts call other experts directly
+
+### Agent Registry
+- **DO**: Use registry for capability-based routing with typed outputs
+- **DON'T**: Hardcode if/else routing logic or keyword-match routing
+
+### ARC Memory
+- **DO**: Check cache before expensive operations; compile context before injection
+- **DON'T**: Skip caching, use O(N) algorithms, or concatenate raw history
+
+### MCP Tools
+- **DO**: Use FastMCP mount() gateway pattern; test with Client(server) in-memory
+- **DON'T**: Write custom async/sync bridges, spawn subprocess per tool call
+- **DO**: Curate 5-7 tools per expert with clear agent stories
+- **DON'T**: Auto-generate tools, expose 10+ tools, or duplicate tool functionality
+
+### Optimizers
+- **DO**: Validate with statistical significance before deploying optimized variants
+- **DON'T**: Deploy without testing; optimize without sufficient training data (min 50 examples)
+
+### System Prompts
+- **DO**: Write 500+ word domain-specific prompts for each expert signature
+- **DON'T**: Use generic "helpful assistant" prompts or share prompts across experts
+
+### Model Selection
+- **DO**: Use `dspy.context(lm=...)` for per-request model selection
+- **DON'T**: Mutate global `dspy.configure()` from within expert code
+
+---
+
+## Development Workflow
+
+### Environment setup
+
+- Python >= 3.12; [`uv`](https://astral.sh/uv) drives everything (deps + execution).
+- Install deps: `uv sync --extra dev --extra api --extra optimizers`
+- `CLIO_ALLOWED_ROOTS` gates tool file access. The tool-server tests
+  write fixtures into the pytest temp dir, so that dir must be on the
+  allow list or those tests fail with `outside_allowed_roots`. Unset,
+  the file policy defaults to the current working directory, which
+  excludes temp paths. Before running the tool tests:
+  `CLIO_ALLOWED_ROOTS="$TMPDIR:$PWD" uv run pytest tests/`
+
+### Workflow
+
+1. Read PLAN.md for current phase and task
+2. Read relevant architecture docs (CLIO_AGENT_ARCHITECTURE, etc.)
+3. Implement with type hints + docstrings
+4. Write tests (using `Client(server)` for MCP, mocks for LM calls)
+5. Test + lint: `uv run pytest tests/ -m "not integration"` then `ruff check src/`
+6. Verify baseline still works: `uv run src/clio_agent/ui/cli.py`
+7. Commit with clear message: `<type>: <description>`
+
+### Running CLIO
+
+- **Dev (from source):** `uv run src/clio_agent/ui/cli.py`
+- **Deployed:** the `install/` system installs a `clio` launcher CLI
+  (`clio start|stop|restart|status|logs|doctor`) alongside the gact
+  TUI. See `install/README.md`.
+
+---
+
+## File Organization
+
+### Current Working Files
+```
+src/clio_agent/
+├── config.py                 # Multi-provider LM configuration
+├── agent.py                  # Main agent — planner loop (Tier 1)
+├── harness.py                # RunTrace, RouteDecision, tool-result normalization
+├── conversation_manager.py   # Session conversation state
+├── errors.py                 # Structured error types
+├── signatures/               # DSPy signatures (planner, expert, chat)
+├── experts/                  # Domain experts (Tier 2): data, analysis, visualization
+├── registry/                 # Capability-based agent registry + matching
+├── arc/                      # ARC memory: cache, B-tree index, LSM, storage, retrieval
+├── optimizer/                # Instrumentation, training, variants, runner
+├── runtime/                  # Doctor / runtime status + nanoagent spawn primitive (Tier 3)
+├── providers/                # Provider-specific auth (e.g. Argonne / ALCF)
+├── tools/                    # FastMCP gateway, file policy, execution boundary
+│   └── servers/              # HDF5 + Parquet MCP servers
+├── gact/                     # GACT v0.2 server — the API surface gact-tui talks to
+└── ui/
+    ├── cli.py                # Interactive CLI + doctor
+    └── api.py                # REST API
+```
+
+---
+
+## Common Patterns
+
+### Pattern 1: ARC Cache-First
+```python
+cached = arc.get_cached_tool_result(tool, params)
+if cached:
+    return cached
+result = execute_tool(tool, params)
+arc.cache_tool_result(tool, params, result)
+```
+
+### Pattern 2: Store Metrics
+```python
+start = time.time()
+result = expert.forward(query)
+arc.store_invocation({
+    "duration_ms": (time.time() - start) * 1000,
+    "success": True,
+    ...
+})
+```
+
+### Pattern 3: Registry Routing (Typed)
+```python
+class RoutingSignature(dspy.Signature):
+    question: str = dspy.InputField()
+    selected_expert: Literal["data", "hpc", "none"] = dspy.OutputField()
+
+router = dspy.ChainOfThought(RoutingSignature)
+routing = router(question=query)
+expert = registry.get_agent(routing.selected_expert)
+```
+
+### Pattern 4: Context Compilation
+```python
+# DON'T: raw_context = "\n".join(all_messages)
+# DO:
+compiled = context_compiler.compile(
+    query=question,
+    session_id=session_id,
+    budget_tokens=2000,
+    include_procedural=True  # what worked/failed before
+)
+```
+
+---
+
+## Testing Requirements
+
+- Unit tests: `tests/test_<module>/`
+- Integration tests: `tests/test_integration/`
+- MCP server tests: use `Client(server)` in-memory (no subprocess)
+- LM tests: mock dspy.LM responses
+- Coverage gate: Phase 1 (50%), Phase 2 (60%), Phase 3 (70%), Phase 4+ (80%)
+- Run before commit: `pytest tests/`
+
+---
+
+## Error Handling
+
+**Graceful Degradation Chain**:
+- IOWarp unavailable -> file-based ARC storage
+- ARC unavailable -> continue without memory (warn user)
+- MCP server down -> pure reasoning mode (no tool calls)
+- Optimizer fails -> keep current variant (no rollback)
+- LM timeout -> retry once, then return partial answer
+
+---
+
+## Git Workflow
+
+**Commit Format**: `<type>: <subject>`
+- `feat`: New feature
+- `fix`: Bug fix
+- `refactor`: Code refactoring
+- `test`: Tests
+- `docs`: Documentation
+
+**Before Commit**:
+- [ ] Tests passing: `pytest tests/`
+- [ ] Lint clean: `ruff check src/`
+- [ ] Baseline works: `uv run src/clio_agent/ui/cli.py`
+
+---
+
+## Quick Reference
+
+**Read First**:
+- `PLAN.md` - What to build (current phase)
+- `docs/CLIO_AGENT_ARCHITECTURE.md` - How it all fits together
+
+**Test**:
+```bash
+pytest tests/
+ruff check src/
+```
+
+**Run**:
+```bash
+uv run src/clio_agent/ui/cli.py
+```
+
+---
+
+**THIS IS YOUR REFERENCE. FOLLOW PLAN.MD. USE NATIVE DSPy/FastMCP PATTERNS.**

clio_agent-0.5.0/.gitattributes

@@ -0,0 +1,7 @@
+# Shell scripts must keep LF endings or they break on Linux/macOS
+# (a CRLF shebang line is not a valid interpreter path).
+install/clio   text eol=lf
+*.sh           text eol=lf
+
+# Batch files are Windows-only; keep CRLF.
+*.cmd          text eol=crlf

clio_agent-0.5.0/.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: CI
+
+on:
+  push:
+    branches: [main, develop, "v*"]
+  pull_request:
+    branches: [main, develop]
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          version: "latest"
+      - run: uv sync --extra dev --extra api --extra optimizers
+      - run: uv run ruff check src/ tests/ scripts/create_demo_data.py
+      - run: bash -n scripts/homelab-env.sh
+      - run: uv run mypy src/
+      # Coverage floor lowered to 70 as the v0.5 merge brought in
+      # large under-tested gact code; tracked in iowarp/clio-agent#31
+      # to ratchet back up to 80% as the new code gets tests.
+      - run: uv run pytest tests/ -m "not integration" --cov-fail-under=70

clio_agent-0.5.0/.github/workflows/release.yml

@@ -0,0 +1,47 @@
+name: Release
+
+# Publish clio-agent to PyPI when a version tag is pushed.
+#
+# Uses PyPI Trusted Publishing (OIDC) -- no API token secret needed.
+# One-time setup on PyPI before the first release:
+#   PyPI project "clio-agent" -> Settings -> Publishing -> add a
+#   trusted publisher:
+#     owner:        iowarp
+#     repository:   clio-agent
+#     workflow:     release.yml
+#     environment:  pypi
+# If the project doesn't exist on PyPI yet, create it (or do one manual
+# `uv publish --token ...` for the first upload, then switch to OIDC).
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # required for PyPI Trusted Publishing (OIDC)
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v5
+        with:
+          version: "latest"
+
+      - name: Verify tag matches package version
+        run: |
+          tag="${GITHUB_REF_NAME#v}"
+          pkg="$(uv version --short)"
+          if [ "$tag" != "$pkg" ]; then
+            echo "tag $GITHUB_REF_NAME does not match pyproject version $pkg" >&2
+            exit 1
+          fi
+
+      - name: Build sdist + wheel
+        run: uv build
+
+      - name: Publish to PyPI
+        run: uv publish

clio_agent-0.5.0/.gitignore

@@ -0,0 +1,161 @@
+# CLIO Agent .gitignore
+# Comprehensive ignore patterns for DSPy agent + UV + HPC scientific computing project
+
+# ============================================================================
+# Python & Virtual Environments
+# ============================================================================
+
+# Virtual environments
+venv/
+env/
+ENV/
+.venv
+.env
+*.env
+
+# Python cache
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+*.egg-info/
+dist/
+build/
+
+# pytest
+.pytest_cache/
+.coverage
+htmlcov/
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+
+# ============================================================================
+# MCP & Tool Artifacts
+# ============================================================================
+
+# MCP tool outputs
+mcp_output/
+mcp_logs/
+tool_cache/
+
+# Tool-generated files
+tool_output/
+*.tool.log
+mcp_*.log
+
+# ============================================================================
+# Development & Testing
+# ============================================================================
+
+# IDE files
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+.env.local
+.env.*.local
+
+# Testing
+test_output/
+test_*.json
+.pytest_cache/
+
+# Temporary files
+tmp/
+temp/
+*.tmp
+*.bak
+
+
+# ============================================================================
+# Configuration & Secrets
+# ============================================================================
+
+# Environment variables with secrets
+.env
+.env.local
+.env.*.local
+*.env.local
+
+# API keys and credentials (CRITICAL - never commit these)
+secrets/
+.secrets/
+credentials.json
+api_keys.json
+*.key
+*.pem
+*.pfx
+
+# Configuration that varies per environment
+config.local.py
+settings.local.json
+local_config.yaml
+
+# ============================================================================
+# FastAPI/Server Artifacts
+# ============================================================================
+
+# FastAPI/Uvicorn
+.uvicorn_socket
+uvicorn.sock
+
+# Server logs
+server_*.log
+api_*.log
+
+# ============================================================================
+# Build & Packaging
+# ============================================================================
+
+# UV lockfile (optional - include if you want reproducible deps)
+# uv.lock
+
+# Build artifacts
+build/
+dist/
+*.egg-info/
+*.whl
+
+# ============================================================================
+# OS Specific
+# ============================================================================
+
+# macOS
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Windows
+Thumbs.db
+ehthumbs.db
+Desktop.ini
+
+# Linux
+.directory
+
+# ============================================================================
+# Project Specific - CLIO Agent
+# ============================================================================
+
+# AI agents
+.planning/
+.clio_agent/
+.codex
+
+ai-docs/_archived/
+ANTHONY/
+.research/
+
+
+
+# Test artefacts (relative-path file_diff tests sometimes leak)
+/b.py
+/x.py
+scratch/

clio_agent-0.5.0/.pre-commit-config.yaml

@@ -0,0 +1,7 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.9.6
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format

clio_agent-0.5.0/AGENTS.md

@@ -0,0 +1,43 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+`src/clio_agent/` contains production code, organized by responsibility:
+- `ui/` for CLI and FastAPI entry points
+- `experts/`, `registry/`, and `signatures/` for routing and expert behavior
+- `arc/` for memory/indexing/storage components
+- `tools/servers/` for FastMCP-backed tool servers
+- `optimizer/` for tuning and instrumentation workflows
+- `runtime/` for doctor and integration status reporting
+
+Tests mirror the runtime layout in `tests/` (`test_core/`, `test_arc/`, `test_experts/`, `test_tools/`, `test_integration/`). Architecture and design docs live in `docs/`, and reference material is in `ai-docs/`.
+Helper scripts for local demos and homelab setup live in `scripts/`.
+
+## Build, Test, and Development Commands
+Use `uv` for environment and command execution:
+- `uv sync --extra dev --extra api --extra optimizers` installs contributor dependencies.
+- `uv run ruff check src/ tests/ scripts/create_demo_data.py` runs lint checks.
+- `uv run ruff format src/ tests/ scripts/create_demo_data.py` formats code.
+- `uv run pytest tests/` runs the full test suite with coverage output.
+- `uv run src/clio_agent/ui/cli.py doctor` reports runtime truth for LM, tools, file policy, API, and `clio-core`.
+- `uv run src/clio_agent/ui/cli.py` starts the interactive CLI.
+- `uv run src/clio_agent/ui/api.py --port 8000` starts the REST API locally.
+
+## Coding Style & Naming Conventions
+Target Python 3.12 with 4-space indentation and type hints on public interfaces. Keep lines readable within the configured `line-length = 100`. Follow existing naming:
+- modules/functions: `snake_case`
+- classes: `PascalCase`
+- constants: `UPPER_SNAKE_CASE`
+
+Ruff is the authoritative linter/formatter (`ruff`, `ruff-format` in pre-commit). Run lint/format before opening a PR.
+
+## Testing Guidelines
+Pytest is the test framework (`test_*.py`, `Test*`, `test_*`). Coverage is enabled by default via `pyproject.toml` (`--cov=clio_agent --cov-report=term-missing --cov-report=html`). Add tests in the matching domain folder and prefer deterministic fixtures from `tests/conftest.py`. Integration tests that need LM Studio are already guarded with skip conditions.
+
+## Commit & Pull Request Guidelines
+The history follows Conventional Commit style, e.g. `feat(api): ...`, `fix(ci): ...`, `test(core): ...`, `docs(04-03): ...`. Keep subject lines imperative and scoped.
+
+PRs should include:
+- a short problem/solution summary
+- linked issue(s) when applicable
+- test/lint evidence (commands + result)
+- API/CLI example output when behavior changes