contextos-vault 1.5.0__tar.gz

contextos_vault-1.5.0/.continue/config.json

@@ -0,0 +1,13 @@
+{
+  "models": [],
+  "contextProviders": [
+    {
+      "name": "http",
+      "params": {
+        "url": "http://127.0.0.1:8080/context",
+        "title": "ContextOS",
+        "description": "Local project knowledge vault"
+      }
+    }
+  ]
+}

contextos_vault-1.5.0/.cursorrules

@@ -0,0 +1,9 @@
+# ContextOS Integration
+Before every task, retrieve context from ContextOS:
+
+curl -s -X POST http://127.0.0.1:8080/context \
+  -H "Authorization: Bearer $CONTEXTOS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "<your task>", "project": "<project-name>"}'
+
+After completing a task: run `context index` to update the knowledge graph.

contextos_vault-1.5.0/.kiro/hooks/contextos-prefetch.json

@@ -0,0 +1,12 @@
+{
+  "name": "ContextOS Prefetch",
+  "version": "1.0.0",
+  "description": "Fetches structured project context from ContextOS before every Kiro task. Injects architecture decisions, domain models, and current sprint context automatically.",
+  "when": {
+    "type": "preTaskExecution"
+  },
+  "then": {
+    "type": "askAgent",
+    "prompt": "Before starting this task, retrieve relevant project context from ContextOS. Call: curl -s -X POST http://127.0.0.1:8080/context -H \"Authorization: Bearer $CONTEXTOS_TOKEN\" -H \"Content-Type: application/json\" -d \"{\\\"query\\\": \\\"{{task}}\\\", \\\"project\\\": \\\"{{project}}\\\", \\\"max_tokens\\\": 4000}\". Use the returned context block to inform your implementation. If the server is not running, proceed without it."
+  }
+}

contextos_vault-1.5.0/AGENTS.md

@@ -0,0 +1,355 @@
+# ContextOS — Agent Rulebook & Operating Manual
+
+## What is ContextOS?
+
+ContextOS is a 100% local, filesystem-native knowledge operating system for software projects. It is both:
+1. **A tool you USE** — to give yourself persistent project memory across sessions
+2. **A codebase you MAINTAIN** — this repo is the ContextOS source code
+
+As an AI coding agent working in this repo, you must use ContextOS itself while you build it (dogfooding).
+
+---
+
+## Quick Start for Agents (Read First)
+
+### Step 1 — Install
+```bash
+pip install -e .
+```
+
+### Step 2 — Initialize in this repo
+```bash
+context init
+```
+
+### Step 3 — Import the knowledge vault
+```bash
+context import ./docs/vault
+```
+
+### Step 4 — Index (first run downloads ~130MB model, then fully offline)
+```bash
+context index
+```
+
+### Step 5 — Create your session token
+```bash
+context token create agent-session
+# Copy the printed ctx_ token — shown once only
+export CONTEXTOS_TOKEN=ctx_<your-token>
+```
+
+### Step 6 — Start the server (keep running in background)
+```bash
+context serve --port 8765
+```
+
+### Step 7 — Verify
+```bash
+curl http://127.0.0.1:8765/health
+```
+
+---
+
+## Using ContextOS FROM Other Project Folders
+
+You do NOT need to be in the ContextOS repo directory to use it. From any project folder:
+
+```bash
+# The server is already running at 127.0.0.1:8765
+# Just call the API with your token
+
+# Get context before starting a task
+curl -s -X POST http://127.0.0.1:8765/context \
+  -H "Authorization: Bearer $CONTEXTOS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "YOUR TASK DESCRIPTION", "project": "contextos"}'
+
+# Search for specific knowledge
+curl -s -X POST http://127.0.0.1:8765/search \
+  -H "Authorization: Bearer $CONTEXTOS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "how does chunking work", "project": "contextos", "limit": 5}'
+```
+
+**The graph updates automatically** when you run `context index` from the ContextOS repo directory. Run it after significant code changes to keep the knowledge graph current.
+
+---
+
+## Agent Workflow Per Session
+
+Follow this workflow EVERY session without exception:
+
+### Before Starting Any Task
+```
+1. Check server: curl http://127.0.0.1:8765/health
+2. If not running: cd /path/to/contextos && context serve --port 8765 &
+3. Fetch context: POST /context with your task description
+4. Read the returned context block before writing any code
+```
+
+### During a Task
+```
+- Use context grep "<pattern>" to find relevant code fast (faster than reading files)
+- Use context read <file> --lines X:Y to read specific sections
+- Use POST /search for semantic lookups ("how does X work")
+- Never re-explain architecture you can retrieve — ask the server
+```
+
+### After Completing a Task
+```
+1. Run: context index  (update the knowledge graph with your changes)
+2. Update relevant vault docs if architecture changed
+3. If you added a new module: add a corresponding doc to docs/vault/architecture/
+```
+
+---
+
+## Architecture Rules (Non-Negotiable)
+
+These rules govern every line of code in this repo. Violating them breaks the system.
+
+### 1. API Server MUST bind to 127.0.0.1 only
+```python
+# CORRECT
+uvicorn.run(app, host="127.0.0.1", port=port)
+
+# FORBIDDEN — will be rejected in code review
+uvicorn.run(app, host="0.0.0.0", port=port)
+```
+
+### 2. No external network calls at runtime
+```python
+# FORBIDDEN at runtime
+import requests
+requests.get("https://api.openai.com/...")  # Never
+
+# The embedding model must use local_files_only=True after first download
+SentenceTransformer(model_name, local_files_only=True)  # Correct
+```
+
+### 3. All paths use pathlib.Path
+```python
+# CORRECT
+from pathlib import Path
+config_path = Path.cwd() / ".contextos" / "config.yaml"
+
+# FORBIDDEN
+import os
+config_path = os.path.join(os.getcwd(), ".contextos", "config.yaml")
+```
+
+### 4. Vault documents are read-only
+```python
+# ContextOS NEVER writes to vault documents
+# All writes go to .contextos/ only
+# Never: open(vault_path / "file.md", "w")
+```
+
+### 5. Token raw values are never stored
+```python
+# Only SHA-256 hash is persisted
+# Raw token printed once at creation, never again
+# Never: json.dump({"token": raw_token}, f)
+# Always: json.dump({"hash": sha256(raw_token)}, f)
+```
+
+### 6. All Pydantic models live in schema.py
+```python
+# New data models MUST be added to contextos/schema.py
+# Never create ad-hoc dicts where a Pydantic model should exist
+# Import from contextos.schema — never duplicate model definitions
+```
+
+---
+
+## File Generation Order (Strict — Avoids Import Errors)
+
+When adding new modules, always respect this dependency order:
+
+```
+1. contextos/schema.py      ← Pydantic models (no dependencies)
+2. contextos/config.py      ← Config (depends on schema)
+3. contextos/vault.py       ← Filesystem scanner (depends on schema, config)
+4. contextos/chunker.py     ← Chunker (depends on schema)
+5. contextos/embedder.py    ← Embedder (depends on schema)
+6. contextos/store.py       ← LanceDB (depends on schema)
+7. contextos/graph.py       ← NetworkX graph (depends on schema)
+8. contextos/memory.py      ← Memory manager (depends on config, store, graph)
+9. contextos/retrieval.py   ← Retrieval pipeline (depends on all above)
+10. contextos/auth.py       ← Token auth (depends on schema)
+11. contextos/api.py        ← FastAPI (depends on retrieval, auth)
+12. contextos/cli.py        ← CLI (depends on everything)
+```
+
+---
+
+## Project Vault Structure
+
+The knowledge vault lives in `docs/vault/`. This is what gets indexed.
+
+```
+docs/vault/
+  architecture/
+    overview.md          ← System architecture (3 layers)
+    api.md               ← API endpoint reference
+    retrieval.md         ← Retrieval pipeline internals
+    storage.md           ← LanceDB + vector store design
+  domain/
+    document.md          ← Document model
+    chunk.md             ← Chunk model
+    token.md             ← Token model
+    graph.md             ← Graph node/edge models
+  decisions/
+    ADR-001-lancedb.md
+    ADR-002-bge-small.md
+    ADR-003-local-only.md
+  workflows/
+    indexing-flow.md     ← How context index works end-to-end
+    search-flow.md       ← How POST /search works
+    context-flow.md      ← How POST /context assembles output
+  context/
+    current.md           ← Updated each sprint with current goals
+```
+
+---
+
+## Tech Stack Reference
+
+| Component | Package | Version |
+|---|---|---|
+| Language | Python | 3.11+ |
+| CLI | Typer | 0.12+ |
+| UI | Rich | 13.7+ |
+| API | FastAPI | 0.111+ |
+| Embeddings | sentence-transformers | 3.x |
+| Embedding model | BAAI/bge-small-en-v1.5 | local, 384-dim |
+| Vector store | LanceDB | 0.6+ |
+| Graph | NetworkX | 3.x |
+| Config | pydantic-settings | 2.x |
+| Models | Pydantic | v2 |
+
+---
+
+## API Quick Reference
+
+All endpoints at `http://127.0.0.1:8765` (or configured port).
+
+```bash
+# No auth required
+GET  /health
+
+# All others require: Authorization: Bearer ctx_<token>
+POST /search          # Semantic document search
+POST /context         # Assembled context block for agent
+GET  /documents       # List indexed documents
+GET  /graph           # Full knowledge graph
+POST /grep            # Fast regex codebase search (v1.1)
+POST /files/read      # Read file with metadata (v1.1)
+GET  /tree            # Project structure (v1.1)
+GET  /changelog       # Recent git commits (v1.1)
+```
+
+### POST /search request shape
+```json
+{
+  "query": "how does payment retry work",
+  "project": "contextos",
+  "type": "architecture",
+  "domain": null,
+  "limit": 5,
+  "include_graph": false
+}
+```
+
+### POST /context request shape
+```json
+{
+  "query": "implement the memory manager",
+  "project": "contextos",
+  "max_tokens": 4000,
+  "priority_order": ["context", "adr", "architecture", "domain", "workflow", "product"]
+}
+```
+
+---
+
+## Memory Management
+
+When disk space is tight or a project is complete, use the memory manager:
+
+```bash
+# See disk usage breakdown
+context memory status
+
+# List all projects with sizes
+context memory projects
+
+# Archive a project (compresses index, keeps vault)
+context memory archive <project-name>
+
+# Purge a project's index (keeps vault, forces re-index on next use)
+context memory purge <project-name>
+
+# Clear embedding cache (re-downloads on next index)
+context memory clear-embeddings
+
+# Full reset of .contextos/ (keeps vault, wipes all indexes)
+context memory reset
+```
+
+---
+
+## Common Patterns
+
+### Add a new CLI command
+1. Add function to `contextos/cli.py` with `@app.command("name")`
+2. Import dependencies lazily inside the function (not at module top)
+3. Follow the UI spec: banner → status → rich panel result → next action hint
+4. Add the command to `AGENTS.md` API reference
+
+### Add a new API endpoint
+1. Add route to `contextos/api.py`
+2. Add request/response models to `contextos/schema.py`
+3. Implement logic in the appropriate module (not inline in api.py)
+4. Add to API reference in `AGENTS.md`
+
+### Update the knowledge graph
+After any significant change:
+```bash
+context index    # rebuilds chunks, embeddings, graph
+context status   # verify document count increased
+```
+
+---
+
+## Environment Variables
+
+```bash
+CONTEXTOS_TOKEN=ctx_<token>    # Bearer token for API auth
+CONTEXTOS_PORT=8765            # Port override (default: 8080)
+CONTEXTOS_LOG_LEVEL=info       # debug | info | warning
+```
+
+Add to your shell profile (`~/.bashrc`, `~/.zshrc`, `~/.profile`):
+```bash
+export CONTEXTOS_TOKEN=ctx_<your-token>
+export CONTEXTOS_PORT=8765
+```
+
+---
+
+## What Makes ContextOS Different
+
+| Problem | ContextOS Solution |
+|---|---|
+| Agent forgets architecture between sessions | Persistent vault indexed and searchable |
+| Context window bloat from re-explaining | `/context` endpoint returns only relevant chunks |
+| Can't search codebase fast | `context grep` with ripgrep, sub-100ms |
+| Need to read file sections | `context read file.py --lines 40:80` |
+| Token waste on unchanged context | Content-hash change detection — only re-indexes changed files |
+| Cloud lock-in | 100% local, zero network at runtime |
+
+---
+
+*ContextOS AGENTS.md — v1.0.0-rc1 — Keep this file updated as the system evolves.*

contextos_vault-1.5.0/CHANGELOG.md

@@ -0,0 +1,132 @@
+# Changelog
+
+All notable changes to ContextOS are documented here.
+This project follows [Semantic Versioning](https://semver.org/).
+
+---
+
+## [1.5.0] — 2026-06-04 — First Stable Release 🎉
+
+**Published to PyPI as [`contextos-vault`](https://pypi.org/project/contextos-vault/)**
+
+```bash
+pip install contextos-vault
+context start
+```
+
+This is the first stable release. All features from v1.0 through v1.5.0rc1 are included.
+No code changes from rc1 — version bump only after verifying end-to-end PyPI installation.
+
+---
+
+## [1.5.0rc1] — 2026-06-04
+
+### Added — Production Maturity
+
+- **E2E test suite** (`tests/test_e2e.py`) — 29 tests covering full pipeline, incremental index, API auth flow, CLI commands, BM25 cache, batch delete, watch mode
+- **BM25 disk cache** — index built at `context index` time, pickled to `.contextos/cache/bm25.pkl`, loaded on first query. Eliminates per-query corpus rebuild
+- **Batch delete** — re-index uses `doc_id IN (...)` instead of N individual deletes
+- **Configurable embedding dimension** — `embedding_dim: 384` in config; auto-detected from loaded model
+- **Path containment check** — `context read` validates `is_relative_to()` against vault roots
+
+### Fixed
+
+- `--watch` mode: `cmd_serve` now uses `try/finally` to stop watcher on shutdown
+- `embedder.py`: `get_embedding_dimension()` FutureWarning fixed with proper fallback
+- Bare `except: pass` eliminated — all silent failures now log at `debug` level
+
+---
+
+## [1.4.1] — 2026-06-04
+
+### Fixed — Audit-Driven Bug Fixes
+
+- **CRITICAL**: `api.py require_scope()` infinite recursion at import time — inner `_check` now depends on `require_token`, not `require_scope`
+- **HIGH**: `retrieval.py assemble_context()` now uses `_rrf_score` (hybrid) correctly, not stale `_distance`
+- **HIGH**: `vault._ingest_document()` injects `project_name` from caller — PDF/DOCX/PPTX no longer get `project: unknown`
+- **MEDIUM**: `auth.check_rate_limit()` merge-write — no longer overwrites `request_count`
+- **MEDIUM**: `memory.get_projects_breakdown()` single table scan — eliminated N+1 LanceDB queries
+- **MEDIUM**: `watcher.py` now watches `.pdf`, `.docx`, `.pptx` in addition to `.md`
+- **MEDIUM**: `mcp_server._mcp_cfg` cached at module level — not reloaded per tool call
+- **MEDIUM**: `store.hybrid_search()` returns vector results when BM25 returns empty
+
+### Added
+
+- `schema.py SearchRequest/ContextRequest`: `use_hybrid` and `hybrid_alpha` fields exposed on API
+- `config.py`: `hybrid_search` and `hybrid_alpha` configurable in `config.yaml`
+- `eval/contextos-questions.json` — 8 golden questions for ContextOS self-evaluation
+
+---
+
+## [1.4.0rc1] — 2026-06-04
+
+### Added — Hybrid Search, Multi-format Ingestors, Evaluator, One-command Bootstrap
+
+- **Hybrid Search** — BM25 + vector + Reciprocal Rank Fusion. `alpha=0.7` default. 30-50% better retrieval for exact terms
+- **PDF ingestor** — `pymupdf` page-by-page extraction
+- **DOCX ingestor** — `python-docx` headings/tables/bold/italic → Markdown
+- **PPTX ingestor** — `python-pptx` slides + speaker notes
+- **Retrieval evaluator** — Hit Rate @K, MRR, avg score, latency. `context eval` CLI
+- **`context start`** — one-command bootstrap: init + scaffold + import + index + token + serve
+- **MCP tool scope enforcement** — `CONTEXTOS_TOKEN` validated per tool call
+- ADR-004 (hybrid search), ADR-005 (ingestors)
+
+---
+
+## [1.3.0rc1] — 2026-06-03
+
+### Added — Auth Scopes, Logging, Cache, Plugins, Scaffolding, CI
+
+- **Token scopes**: `read | write | admin` with hierarchical enforcement
+- **Token expiry** and **rate limiting** (1000 req/min sliding window)
+- **Structured logging**: `app.jsonl`, `slow.jsonl`, `audit.jsonl` with rotation
+- **Context response cache**: LRU, TTL=5min, invalidated on index
+- **Plugin system**: `~/.contextos/plugins/`, PyPI `entry_points`, `context plugin install`
+- **Vault scaffolder**: `default`, `microservice`, `api-first` templates
+- **CI commands**: `context ci check`, `context ci index`
+- API: `/metrics`, `/audit`, `X-Request-ID` headers, `require_scope` middleware
+
+---
+
+## [1.2.0rc1] — 2026-06-03
+
+### Added — Session Memory, Connectors, TUI Dashboard
+
+- **Session manager**: create, event, end, summary, vault export
+- **Connectors**: GitHub Issues/Wiki, OpenAPI spec, JSON/YAML/TOML
+- **Textual TUI dashboard**: `context dashboard` — live 4-panel system monitor
+- **`context export`** — vault to single Markdown or JSON
+- API: `/session/*`, `/pull`
+
+---
+
+## [1.1.0rc1] — 2026-06-03
+
+### Added — Performance + Integration
+
+- **Incremental index** — content-hash change detection, skip unchanged docs
+- **MCP server** — 6 native tools (stdio transport)
+- **Symbol index** — Python AST + JS/TS regex extraction
+- **Smart compression** — sumy TF-IDF extractive, no LLM
+- **Live watch mode** — watchfiles per-file re-index
+- `context context`, `context diff`, `context projects`, `context symbols`, `context setup`
+- Agent templates: `.cursorrules`, `mcp.json`, `.continue/config.json`, copilot instructions
+
+---
+
+## [1.0.0rc1] — 2026-06-03
+
+### Initial Release
+
+- Core 3-layer architecture: Vault → Index → API
+- FastAPI server bound exclusively to `127.0.0.1`
+- BAAI/bge-small-en-v1.5 embeddings (384-dim, local CPU)
+- LanceDB vector store
+- NetworkX knowledge graph
+- 15 CLI commands with Rich UI
+- Token authentication (SHA-256 hash, never plaintext)
+- Memory management (purge, archive, reset)
+- Kiro pre-task hook
+- Self-documenting vault (`docs/vault/`)
+- Example project vault (`examples/my-project/`)
+- 15 smoke tests

contextos_vault-1.5.0/CLAUDE.md

@@ -0,0 +1,34 @@
+# ContextOS Project Memory
+
+This project uses ContextOS itself for persistent knowledge management.
+
+## Before Starting Any Task
+
+Retrieve context from the local ContextOS server:
+
+```bash
+curl -s -X POST http://127.0.0.1:8765/context \
+  -H "Authorization: Bearer $CONTEXTOS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "<YOUR TASK HERE>", "project": "contextos", "max_tokens": 4000}'
+```
+
+If server is not running:
+```bash
+cd /path/to/contextos && context serve --port 8765 &
+```
+
+## Architecture Rules
+See AGENTS.md for complete rules. Critical constraints:
+- API server MUST bind to 127.0.0.1 only (never 0.0.0.0)
+- No external network calls at runtime
+- All paths use pathlib.Path
+- Vault documents are read-only
+- Token raw values are never stored — SHA-256 hash only
+- All Pydantic models live in schema.py
+
+## After Task Completion
+```bash
+context index    # update knowledge graph
+context status   # verify
+```

contextos_vault-1.5.0/CONTRIBUTING.md

@@ -0,0 +1,139 @@
+# Contributing to ContextOS
+
+Thank you for your interest in contributing. ContextOS is an enterprise open-source project. All contributions must maintain the core design principles — especially the zero-network-at-runtime guarantee.
+
+---
+
+## Quick Start for Contributors
+
+```bash
+git clone https://github.com/AbhayankarBellur/ContextOS.git
+cd ContextOS
+pip install -e ".[dev]"
+context init
+context import docs/vault
+context index
+```
+
+---
+
+## Development Setup
+
+### Requirements
+- Python 3.11+
+- pip
+
+### Install with dev dependencies
+```bash
+pip install -e ".[dev]"
+```
+
+### Run the CLI
+```bash
+context --help
+context doctor    # validate your setup
+```
+
+### Run tests
+```bash
+pytest tests/ -v
+```
+
+---
+
+## Architecture Rules (Non-Negotiable)
+
+These rules apply to every pull request. PRs that violate them will not be merged.
+
+| Rule | Requirement |
+|---|---|
+| API binding | `host='127.0.0.1'` only — never `0.0.0.0` |
+| Network calls | Zero external calls at runtime |
+| Paths | `pathlib.Path` only — never `os.path` |
+| Vault files | Read-only — ContextOS never modifies source documents |
+| Token storage | SHA-256 hash only — raw value never persisted |
+| Data models | All Pydantic v2 models in `contextos/schema.py` |
+| UI output | Rich only — no manual ANSI, no `print()` outside `ui.py` |
+
+---
+
+## File Structure
+
+```
+contextos/
+  schema.py      ← All Pydantic models (source of truth)
+  config.py      ← pydantic-settings Config
+  vault.py       ← Filesystem scanner + frontmatter parser
+  chunker.py     ← MarkdownHeaderTextSplitter
+  embedder.py    ← sentence-transformers wrapper
+  store.py       ← LanceDB vector store
+  graph.py       ← NetworkX knowledge graph
+  memory.py      ← Disk management engine
+  retrieval.py   ← Full retrieval pipeline
+  auth.py        ← Token generation + validation
+  api.py         ← FastAPI server (127.0.0.1 only)
+  cli.py         ← Typer CLI (all commands)
+  ui.py          ← Rich UI theme + shared components
+
+docs/vault/      ← ContextOS's own knowledge vault
+examples/        ← Example vaults for testing
+tests/           ← Test suite
+```
+
+**Dependency order must be respected.** Never import `cli.py` from `api.py`. Never import `api.py` from `retrieval.py`. See `AGENTS.md` for the full dependency graph.
+
+---
+
+## Pull Request Checklist
+
+- [ ] `pip install -e .` succeeds cleanly
+- [ ] `context doctor` reports all checks passing
+- [ ] `context index` runs fully offline (after first model download)
+- [ ] `context search "test"` returns results
+- [ ] No new external network calls introduced
+- [ ] All new data models added to `schema.py`
+- [ ] All new CLI commands follow the UI spec in `ui.py`
+- [ ] `AGENTS.md` updated if API surface changed
+
+---
+
+## Commit Message Format
+
+```
+type: short description
+
+Types: feat | fix | docs | refactor | test | chore
+```
+
+Examples:
+```
+feat: add POST /grep endpoint for codebase search
+fix: handle missing frontmatter on vault documents
+docs: update AGENTS.md with memory management commands
+```
+
+---
+
+## Versioning
+
+ContextOS follows semantic versioning. Version is set in `pyproject.toml` and `contextos/__init__.py`.
+
+| Version | Scope |
+|---|---|
+| v1.0.x | Bug fixes |
+| v1.1.x | New endpoints, new CLI commands |
+| v1.2.x | New retrieval modes (GraphRAG, symbol search) |
+| v2.0.0 | Breaking changes (new schema, protocol changes) |
+
+---
+
+## Reporting Issues
+
+Use GitHub Issues. Include:
+- Output of `context doctor`
+- OS and Python version
+- Exact command and error output
+
+---
+
+*ContextOS — Local-first. Agent-native. Enterprise-grade.*