codebatch 0.1.0__tar.gz

codebatch-0.1.0/.gitignore

@@ -0,0 +1,49 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+.env
+.venv
+env/
+venv/
+ENV/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+.nox/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# CodeBatch stores (example stores, not the code)
+*.store/

codebatch-0.1.0/.spec_baseline_hash

@@ -0,0 +1 @@
+8578b4ba5bda9a45a2868f13a9a19b5e2f46ee4fed63803e8e85c25553a818ac

codebatch-0.1.0/CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+All notable changes to the CodeBatch specification and implementation.
+
+## [Unreleased]
+
+### Added
+- Initial repository structure
+- Core specification document
+
+---
+
+## [spec-v1.0-draft] - 2025-02-02
+
+### Added
+- Complete storage and execution specification (SPEC.md)
+- Content-addressed object store layout
+- Snapshot immutability contract
+- Batch, task, and shard execution model
+- Output record indexing semantics
+- Query model guarantees
+- Large output chunking rules
+- Versioning requirements for all records
+
+### Defined
+- 14 specification sections covering full execution lifecycle
+- 6 global invariants for system correctness
+- Compliance criteria for implementations
+
+---
+
+[Unreleased]: https://github.com/mcp-tool-shop-org/code-batch/compare/spec-v1.0-draft...HEAD
+[spec-v1.0-draft]: https://github.com/mcp-tool-shop-org/code-batch/releases/tag/spec-v1.0-draft

codebatch-0.1.0/PKG-INFO

@@ -0,0 +1,66 @@
+Metadata-Version: 2.4
+Name: codebatch
+Version: 0.1.0
+Summary: Content-addressed batch execution engine
+Author-email: mcp-tool-shop <64996768+mcp-tool-shop@users.noreply.github.com>
+License-Expression: MIT
+Keywords: batch,content-addressed,deterministic,execution
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# CodeBatch
+
+Content-addressed batch execution engine with deterministic sharding and queryable outputs.
+
+## Overview
+
+CodeBatch provides a filesystem-based execution substrate for running deterministic transformations over codebases. It captures inputs as immutable snapshots, executes work in isolated shards, and indexes all semantic outputs for efficient querying—without requiring a database.
+
+## Documentation
+
+- **[SPEC.md](./SPEC.md)** — Full storage and execution specification
+- **[CHANGELOG.md](./CHANGELOG.md)** — Version history
+
+## Spec Versioning
+
+The specification uses semantic versioning with draft/stable markers. Each version is tagged in git (e.g., `spec-v1.0-draft`). Breaking changes increment the major version. Implementations should declare which spec version they target and tolerate unknown fields for forward compatibility.
+
+## Project Structure
+
+```
+schemas/      JSON Schema definitions for all record types
+src/          Core implementation
+tests/        Test suites and fixtures
+examples/     Usage examples
+.github/      CI/CD workflows
+```
+
+## Quick Start
+
+```bash
+# Create a snapshot of a directory
+codebatch snapshot ./my-project --store ./store
+
+# Initialize a batch with a pipeline
+codebatch batch init --snapshot <id> --pipeline parse
+
+# Run a shard
+codebatch run-shard --batch <id> --task 01_parse --shard ab
+
+# Query results
+codebatch query diagnostics --batch <id> --task 01_parse
+```
+
+## License
+
+MIT

codebatch-0.1.0/README.md

@@ -0,0 +1,46 @@
+# CodeBatch
+
+Content-addressed batch execution engine with deterministic sharding and queryable outputs.
+
+## Overview
+
+CodeBatch provides a filesystem-based execution substrate for running deterministic transformations over codebases. It captures inputs as immutable snapshots, executes work in isolated shards, and indexes all semantic outputs for efficient querying—without requiring a database.
+
+## Documentation
+
+- **[SPEC.md](./SPEC.md)** — Full storage and execution specification
+- **[CHANGELOG.md](./CHANGELOG.md)** — Version history
+
+## Spec Versioning
+
+The specification uses semantic versioning with draft/stable markers. Each version is tagged in git (e.g., `spec-v1.0-draft`). Breaking changes increment the major version. Implementations should declare which spec version they target and tolerate unknown fields for forward compatibility.
+
+## Project Structure
+
+```
+schemas/      JSON Schema definitions for all record types
+src/          Core implementation
+tests/        Test suites and fixtures
+examples/     Usage examples
+.github/      CI/CD workflows
+```
+
+## Quick Start
+
+```bash
+# Create a snapshot of a directory
+codebatch snapshot ./my-project --store ./store
+
+# Initialize a batch with a pipeline
+codebatch batch init --snapshot <id> --pipeline parse
+
+# Run a shard
+codebatch run-shard --batch <id> --task 01_parse --shard ab
+
+# Query results
+codebatch query diagnostics --batch <id> --task 01_parse
+```
+
+## License
+
+MIT

codebatch-0.1.0/SPEC.md

@@ -0,0 +1,342 @@
+# CodeBatch Storage & Execution Specification
+
+**Specification Version: 1.0 (Draft)**
+**Schema Version: 1** (`schema_version` field in all records)
+
+> **Note**: The specification version tracks the document itself. The schema version
+> is the integer value written to `schema_version` fields in JSON records and is
+> incremented when record formats change in backward-incompatible ways.
+
+---
+
+## 1. Scope and Non-Goals
+
+This specification defines:
+
+- The on-disk storage layout
+- Content-addressed object rules
+- Snapshot, batch, task, and shard contracts
+- Execution and output indexing semantics
+- Queryability guarantees
+
+This specification does **not** define:
+
+- Scheduling policies
+- UI concerns
+- Cloud-specific integrations
+- Programming language requirements
+
+All behaviors are defined in terms of filesystem structure and serialized records.
+
+---
+
+<!-- SPEC_PROTECTED_BEGIN -->
+<!-- WARNING: Changes to sections 2-8 require Phase 3+ and schema version bump -->
+<!-- Allowed changes: adding output kinds (§9), clarifying plan deps (§7), new schemas -->
+
+## 2. Global Invariants
+
+The system SHALL maintain the following invariants:
+
+1. Content-addressed objects are immutable and add-only.
+2. Snapshots represent frozen input state and never change after creation.
+3. Batches represent execution attempts and may be repeated or discarded.
+4. Semantic results are discoverable without reading execution logs.
+5. Partial execution SHALL NOT corrupt previously completed work.
+6. All indexes are append-only and rebuildable from authoritative data.
+
+---
+
+## 3. Store Root Layout
+
+A CodeBatch store has the following top-level layout:
+
+```
+<store_root>/
+  store.json
+  objects/
+  snapshots/
+  batches/
+  indexes/        # optional acceleration only
+```
+
+`store.json` describes store-level configuration and versioning.
+
+---
+
+## 4. Object Store
+
+### 4.1 Object Identity
+
+- Each object is identified by `sha256(raw_bytes)`.
+- Object identity is independent of filename, origin, or usage.
+- Object hashes are stable across platforms.
+
+### 4.2 Storage Layout
+
+Objects SHALL be stored at:
+
+```
+objects/sha256/<aa>/<bb>/<full_hash>
+```
+
+Where `<aa>` and `<bb>` are the first two byte pairs of the hex hash.
+
+### 4.3 Object Metadata (Optional)
+
+An object MAY have an adjacent metadata file:
+
+```
+<full_hash>.meta.json
+```
+
+Metadata is advisory and SHALL NOT be required for correctness.
+
+---
+
+## 5. Snapshots
+
+### 5.1 Snapshot Definition
+
+- A snapshot represents a frozen view of an input source at a specific point in time.
+- Snapshots are immutable once written.
+
+### 5.2 Snapshot Layout
+
+```
+snapshots/<snapshot_id>/
+  snapshot.json
+  files.index.jsonl
+```
+
+### 5.3 Path Canonicalization
+
+All file paths in a snapshot SHALL be canonicalized:
+
+- UTF-8 encoded
+- `/` as separator
+- No `.` or `..` segments
+- No trailing slash
+- Stable casing preserved
+
+A `path_key` field SHALL be included for normalized comparison.
+
+### 5.4 File Index Records
+
+Each line in `files.index.jsonl` describes exactly one file.
+
+**Required fields:**
+
+| Field | Description |
+|-------|-------------|
+| `schema_version` | Record schema version |
+| `path` | Original file path |
+| `path_key` | Normalized path for comparison |
+| `object` | SHA-256 hash of file content |
+| `size` | File size in bytes |
+
+**Optional fields MAY include:**
+
+- `text_hash`
+- `lang_hint`
+- `mode`
+- `mtime`
+
+---
+
+## 6. Batches
+
+### 6.1 Batch Definition
+
+- A batch represents one execution attempt over a snapshot.
+- Batches are isolated, repeatable, and discardable.
+
+### 6.2 Batch Layout
+
+```
+batches/<batch_id>/
+  batch.json
+  plan.json
+  events.jsonl
+  tasks/
+```
+
+### 6.3 Batch Events
+
+- `events.jsonl` records execution facts only.
+- Events SHALL NOT be required to answer semantic questions about outputs.
+
+---
+
+## 7. Tasks
+
+### 7.1 Task Definition
+
+- A task performs a deterministic transformation over snapshot inputs or prior task outputs.
+- Tasks SHALL be idempotent per shard.
+
+### 7.2 Task Layout
+
+```
+tasks/<task_id>/
+  task.json
+  events.jsonl
+  shards/
+```
+
+### 7.3 Task Configuration
+
+`task.json` SHALL fully describe:
+
+- Task identity
+- Input requirements
+- Sharding strategy
+- Resolved configuration parameters
+
+Task configuration SHALL be treated as immutable.
+
+---
+
+## 8. Shards
+
+### 8.1 Shard Identity
+
+- Shards are deterministic partitions of task input space.
+- Shard identifiers SHALL be stable across executions.
+
+### 8.2 Shard Layout
+
+```
+shards/<shard_id>/
+  state.json
+  outputs.index.jsonl
+```
+
+### 8.3 Shard State
+
+- `state.json` tracks shard execution status.
+- Shard state transitions SHALL be monotonic.
+
+### 8.4 Shard Execution Rules
+
+A shard:
+
+1. Reads only snapshot and prior task outputs
+2. Writes only within its own shard directory
+3. Adds objects to the object store
+4. Appends records to its outputs index
+5. Emits completion events only after outputs are committed
+
+<!-- SPEC_PROTECTED_END -->
+
+---
+
+## 9. Outputs
+
+### 9.1 Output Records
+
+- All semantic results SHALL be represented as output records.
+- Output records are append-only.
+
+### 9.2 Output Index
+
+Each shard SHALL maintain an `outputs.index.jsonl`.
+
+Each record SHALL include:
+
+| Field | Description |
+|-------|-------------|
+| `schema_version` | Record schema version |
+| `snapshot_id` | Source snapshot |
+| `batch_id` | Execution batch |
+| `task_id` | Owning task |
+| `shard_id` | Owning shard |
+| `path` | Source file path |
+| `kind` | Output type |
+| `ts` | Timestamp |
+
+Records MAY include:
+
+- `object` — Content hash for stored outputs
+- `format` — Output format identifier
+- Diagnostic fields (`severity`, `code`, `message`)
+
+### 9.3 Diagnostics
+
+- Diagnostics SHALL be represented as output records with `kind = diagnostic`.
+- Diagnostics SHALL NOT be inferred from execution events.
+
+---
+
+## 10. Large Outputs
+
+### 10.1 Chunking Requirement
+
+- Outputs exceeding a configured size threshold SHALL be chunked.
+- Chunked outputs SHALL be represented by a manifest object.
+
+### 10.2 Chunk Manifest
+
+A chunk manifest SHALL include:
+
+| Field | Description |
+|-------|-------------|
+| `schema_name` | Manifest schema identifier |
+| `schema_version` | Manifest schema version |
+| `kind` | Output kind |
+| `format` | Content format |
+| `chunks` | Array of chunk object references |
+| `total_bytes` | Total size across all chunks |
+
+Output records SHALL reference the manifest object.
+
+---
+
+## 11. Execution Semantics
+
+### 11.1 Determinism
+
+Given identical snapshot, task configuration, and shard identifier:
+
+- The same outputs SHALL be produced
+- Duplicate objects SHALL deduplicate naturally
+
+### 11.2 Failure Handling
+
+- Shard failure SHALL NOT invalidate other shards.
+- Restarting a shard SHALL NOT require cleanup.
+
+---
+
+## 12. Query Model
+
+The following questions SHALL be answerable without reading execution logs:
+
+1. Which files produced diagnostics?
+2. Which outputs exist for a given task?
+3. Which files failed a given task?
+4. Aggregate counts by kind, severity, or language
+
+Indexes MAY be accelerated but SHALL remain rebuildable.
+
+---
+
+## 13. Versioning
+
+All structured records SHALL include:
+
+- `schema_name`
+- `schema_version`
+
+Readers SHALL tolerate unknown fields.
+
+---
+
+## 14. Compliance
+
+An implementation is compliant if:
+
+1. All required structures are present
+2. All invariants are preserved
+3. Semantic state is discoverable from indexes alone
+4. Partial execution does not corrupt prior results

codebatch-0.1.0/docs/PHASE2_ACCEPTANCE.md

@@ -0,0 +1,158 @@
+# Phase 2 Acceptance Checklist
+
+**Human-readable checklist mirroring automated gates.**
+
+Use this when debugging CI failures.
+
+---
+
+## Pre-Flight Checks
+
+- [ ] All Phase 1 tests still pass (121 baseline)
+- [ ] No uncommitted changes to protected SPEC regions
+- [ ] No network imports in new code
+
+---
+
+## Gate 1: Multi-Task Pipeline End-to-End
+
+**What it tests**: Complete pipeline with deps works.
+
+**Manual verification**:
+```bash
+# Init and snapshot
+codebatch init ./test-store
+codebatch snapshot ./fixtures/corpus --store ./test-store
+
+# Create batch with full pipeline
+codebatch batch init --snapshot <id> --pipeline full --store ./test-store
+
+# Run tasks in order
+codebatch run-task --batch <id> --task 01_parse --store ./test-store
+codebatch run-task --batch <id> --task 02_analyze --store ./test-store
+codebatch run-task --batch <id> --task 03_symbols --store ./test-store
+codebatch run-task --batch <id> --task 04_lint --store ./test-store
+
+# Verify all shards done
+find ./test-store/batches/*/tasks/*/shards/*/state.json -exec grep -l '"status": "done"' {} \;
+```
+
+**Pass condition**: All shard states are "done", each task has outputs.
+
+---
+
+## Gate 2: Log Independence
+
+**What it tests**: Semantic queries don't depend on events.
+
+**Manual verification**:
+```bash
+# Run queries, save output
+codebatch query diagnostics --batch <id> --task 04_lint --store ./test-store > before.json
+codebatch query outputs --batch <id> --task 03_symbols --kind symbol --store ./test-store >> before.json
+
+# Delete all events
+find ./test-store -name "events.jsonl" -delete
+
+# Rerun queries
+codebatch query diagnostics --batch <id> --task 04_lint --store ./test-store > after.json
+codebatch query outputs --batch <id> --task 03_symbols --kind symbol --store ./test-store >> after.json
+
+# Compare (ignoring timestamps)
+diff <(jq -S 'del(.ts)' before.json) <(jq -S 'del(.ts)' after.json)
+```
+
+**Pass condition**: Identical semantic results.
+
+---
+
+## Gate 3: Cache Deletion Equivalence
+
+**What it tests**: indexes/ is truly optional.
+
+**Manual verification**:
+```bash
+# Run queries with cache
+codebatch query stats --batch <id> --task 03_symbols --store ./test-store > with_cache.json
+
+# Delete cache
+rm -rf ./test-store/indexes/
+
+# Rerun queries
+codebatch query stats --batch <id> --task 03_symbols --store ./test-store > without_cache.json
+
+# Compare
+diff with_cache.json without_cache.json
+```
+
+**Pass condition**: Identical results.
+
+---
+
+## Gate 4: Retry Determinism
+
+**What it tests**: Per-shard replacement produces same outputs on retry.
+
+**Manual verification**:
+```bash
+# Run shard, capture outputs
+codebatch run-shard --batch <id> --task 03_symbols --shard ab --store ./test-store
+cp ./test-store/batches/<id>/tasks/03_symbols/shards/ab/outputs.index.jsonl run1.jsonl
+
+# Reset shard (delete state and outputs)
+rm ./test-store/batches/<id>/tasks/03_symbols/shards/ab/state.json
+rm ./test-store/batches/<id>/tasks/03_symbols/shards/ab/outputs.index.jsonl
+
+# Rerun
+codebatch run-shard --batch <id> --task 03_symbols --shard ab --store ./test-store
+cp ./test-store/batches/<id>/tasks/03_symbols/shards/ab/outputs.index.jsonl run2.jsonl
+
+# Compare (ignoring ts)
+diff <(jq -S 'del(.ts)' run1.jsonl) <(jq -S 'del(.ts)' run2.jsonl)
+```
+
+**Pass condition**: Same semantic records (objects may differ if content-equivalent).
+
+---
+
+## Gate 5: SPEC Stability
+
+**What it tests**: Protected SPEC regions unchanged.
+
+**Manual verification**:
+```bash
+# Check protected region
+git diff origin/main -- SPEC.md | grep -A5 -B5 "SPEC_PROTECTED"
+```
+
+**Pass condition**: No changes between SPEC_PROTECTED markers.
+
+---
+
+## Task Output Verification
+
+### 02_analyze
+- [ ] Emits `kind=metric` records
+- [ ] Metrics include: `loc`, `size`, `complexity` (optional)
+
+### 03_symbols
+- [ ] Emits `kind=symbol` records with `name`, `symbol_type`, `line`
+- [ ] Emits `kind=edge` records with `edge_type`, `target`
+- [ ] Symbol types include: `function`, `class`, `variable`
+- [ ] Edge types include: `imports`, `calls` (optional)
+
+### 04_lint
+- [ ] Emits `kind=diagnostic` records
+- [ ] Diagnostics have `severity`, `code`, `message`
+- [ ] Line/col positions when available
+
+---
+
+## Final Checklist
+
+- [ ] 150+ tests passing
+- [ ] All 5 gates pass in CI
+- [ ] No network imports
+- [ ] No writes outside allowed paths
+- [ ] Plan deps enforced
+- [ ] Documentation updated