sorkit 0.1.0__tar.gz

sorkit-0.1.0/.claude/skills/experiment-loop.md

@@ -0,0 +1,84 @@
+# Skill: Autonomous Experiment Loop
+
+## When to Use
+When implementing any layer of this project autonomously.
+This skill defines the experiment protocol — how to iterate, evaluate, and ratchet.
+
+## Protocol
+
+### Before You Start
+
+1. Read `CLAUDE.md` to confirm which layer you're working on
+2. Read the oracle tests for your layer (see the contracts/scored_tests in sor.yaml)
+3. Read the current mutation surface files to understand the starting state
+4. Check `results.tsv` for previous experiment history (if any)
+
+### Experiment Loop
+
+Each iteration follows this exact sequence. Do not deviate.
+
+#### Step 1: Plan the Change
+
+Before editing any code, write a one-line hypothesis:
+
+```
+HYPOTHESIS: [what you're changing] should [expected effect] because [reasoning]
+```
+
+#### Step 2: Implement
+
+Edit ONLY files in the current layer's mutation surface (see CLAUDE.md).
+Do NOT touch frozen files. Do NOT touch files from other layers.
+Keep changes atomic — one idea per iteration.
+
+#### Step 3: Run the Ratchet
+
+```bash
+./scripts/ratchet.sh <layer_number> "brief hypothesis description"
+```
+
+The ratchet will:
+- Run the oracle
+- Compare scores to previous best
+- Git commit if improved, git reset if not
+- Check all stopping conditions
+- Notify if a stopping condition is hit
+
+#### Step 4: Parse the Output
+
+The ratchet prints exactly one of:
+- `KEEP score={X} prev={Y}` — improvement, committed
+- `DISCARD score={X} best={Y}` — no improvement, reverted
+- `DISCARD FAIL` — tests failed, reverted
+- `STOP:{reason} score={X} attempts={N} kept={K}` — stopping condition hit
+
+#### Step 5: Decide Next Experiment
+
+Review `results.tsv` to see what you've tried. Pick a different approach.
+Do NOT repeat a failed hypothesis with minor variations more than once.
+
+If you have 3+ consecutive failures, read the test output more carefully:
+```bash
+tail -n 50 run.log
+```
+
+### Stopping Conditions
+
+Stop the loop and report to the human if you see any `STOP:` output:
+- `TARGET_MET` — scored layer reached its target composite score
+- `ALL_PASS` — pass/fail layer succeeded
+- `PLATEAU` — too many consecutive non-improvements
+- `DIMINISHING` — improvements too small to matter
+- `MAX_ATTEMPTS` — hard ceiling reached
+- `CONSECUTIVE_FAILURES` — too many crashes in a row
+- `ORACLE_ERROR` — the oracle itself is broken (needs human fix)
+
+### Results TSV Format
+
+Tab-separated, appended by the ratchet:
+
+```
+timestamp	layer	hypothesis	score	outcome
+2026-03-12T10:30:00	0	hybrid BM25+cosine 0.6/0.4	0.72	KEEP
+2026-03-12T10:35:00	0	pure cosine similarity	0.65	DISCARD
+```

sorkit-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,30 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Test
+        run: pytest -v

sorkit-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,30 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  id-token: write
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

sorkit-0.1.0/.gitignore

@@ -0,0 +1,31 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+*.egg
+dist/
+build/
+*.whl
+
+# Virtual environments
+.venv/
+venv/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# SOR runtime
+run.log
+reports/

sorkit-0.1.0/CLAUDE.md

@@ -0,0 +1,75 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What This Is
+
+sorkit is a Python MCP server (`pip install sorkit`) implementing the Surface-Oracle-Ratchet pattern for autonomous code optimization. An AI agent edits designated files (surfaces), is evaluated by frozen tests (oracles), and advances via git commit on improvement / git reset on failure (ratchet).
+
+## Commands
+
+```bash
+# Install (editable with dev deps)
+pip install -e ".[dev]"
+
+# Run all tests
+python -m pytest
+
+# Run a single test file
+python -m pytest tests/test_config.py
+
+# Run a single test by name
+python -m pytest -k test_loads_layers
+
+# Run with verbose output
+python -m pytest -v
+
+# Run the MCP server
+sorkit
+
+# Build for distribution
+python -m build
+```
+
+## Architecture
+
+The package lives in `src/sorkit/` with this data flow:
+
+```
+sor.yaml → config.py → server.py (9 MCP tools)
+                          ↓
+                  oracle.py (run tests, extract metrics)
+                          ↓
+                  ratchet.py (compare → git commit/reset → check stops)
+                          ↓
+                  results.py (append to results.tsv)
+                          ↓
+                  notify.py (file/Slack/email/desktop)
+```
+
+**config.py** — Dataclass model for `sor.yaml`. `load_config()` parses YAML into `SorConfig` with layers, each containing `OracleConfig`, `MetricConfig`, `ThresholdConfig`. `resolve_threshold()` cascades layer overrides to defaults. `resolve_layer_index()` accepts name or numeric index.
+
+**server.py** — FastMCP server (`from fastmcp import FastMCP`). Nine `@mcp.tool()` functions. `sor_init` uses a two-call pattern: no config returns a template, with config saves and generates artifacts. `sor_ratchet` is the core loop tool.
+
+**oracle.py** — Async subprocess runner. Contracts run first (`-x --tb=short -q`), scored tests add `-s` to capture `print()` output. `_extract_metric()` uses regex `^{PATTERN}:\s+(\S+)` to pull floats from stdout. Returns `OracleResult` with composite score.
+
+**ratchet.py** — `ratchet_once()` is the convergence engine. Checks 7 stopping conditions: TARGET_MET, ALL_PASS, PLATEAU, DIMINISHING, MAX_ATTEMPTS, CONSECUTIVE_FAILURES, ORACLE_ERROR. Git operations use `asyncio.create_subprocess_exec`.
+
+**init.py** — Generates `CLAUDE.md`, `.claude/skills/experiment-loop.md`, and `results.tsv` from config. `config_from_dict()` handles both template format (with `_value`/`_description` keys) and plain dicts.
+
+**results.py** — `ResultsStore` reads/writes a TSV file. Methods like `get_best_score()`, `get_consecutive_failures()`, `get_consecutive_non_improvements()` drive stopping condition checks.
+
+**frozen.py** — `get_frozen_paths()` computes the full frozen set: `always_frozen` + surfaces from all layers below the current one.
+
+**audit.py** — Analysis tools over `ResultsStore`: score progression with running best, hypothesis grouping with keep/discard/fail counts, full audit reports with convergence estimates.
+
+## Testing Patterns
+
+- All tests are async-compatible (`asyncio_mode = "auto"` in pyproject.toml)
+- Shared fixture `sor_project` (in `tests/conftest.py`) creates a tmp dir with a valid `sor.yaml`
+- Server tests use `async with Client(mcp) as client` then `result = await client.call_tool(name, args)` — access output via `result.content[0].text`
+- Oracle/ratchet tests mock `asyncio.create_subprocess_exec` to avoid real subprocess calls
+
+## Example
+
+`examples/sentiment/` is a self-contained demo: a naive sentiment classifier (~40% accuracy) that an agent optimizes against 50 labeled examples. Layer 1 is scored (classifier), Layer 2 is pass/fail (API stub). Run from that directory with `python -m pytest tests/ -s`.

sorkit-0.1.0/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 2Lines
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.