feedloop 0.1.0__tar.gz

feedloop-0.1.0/.claude/settings.local.json

@@ -0,0 +1,9 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(pip3 install:*)",
+      "Bash(python3 -c \":*)",
+      "Bash(source /Users/rammuthiah/Documents/Coding/hitl/.venv/bin/activate)"
+    ]
+  }
+}

feedloop-0.1.0/.gitignore

@@ -0,0 +1,19 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.venv/
+venv/
+.env
+*.db
+*.db-wal
+*.db-shm
+node_modules/
+src/feedloop/_static/*
+!src/feedloop/_static/.gitkeep
+.pytest_cache/
+.mypy_cache/

feedloop-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ram Muthiah
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

feedloop-0.1.0/Makefile

@@ -0,0 +1,18 @@
+.PHONY: dev-frontend dev-backend build test clean
+
+dev-frontend:
+	cd frontend && npm run dev
+
+dev-backend:
+	uvicorn feedloop._server:app --reload
+
+build:
+	cd frontend && npm run build
+	python -m build
+
+test:
+	pytest tests/ -v
+
+clean:
+	rm -rf dist/ build/ *.egg-info
+	find . -type d -name __pycache__ -exec rm -rf {} +

feedloop-0.1.0/PKG-INFO

@@ -0,0 +1,41 @@
+Metadata-Version: 2.4
+Name: feedloop
+Version: 0.1.0
+Summary: The fastest way to collect human preference data for LLMs
+Project-URL: Homepage, https://github.com/rammuthiah/feedloop
+Project-URL: Repository, https://github.com/rammuthiah/feedloop
+Author: Ram Muthiah
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dpo,human-feedback,llm,preference-data,rlhf
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: fastapi>=0.100
+Requires-Dist: pydantic>=2.0
+Requires-Dist: uvicorn[standard]>=0.20
+Description-Content-Type: text/markdown
+
+# feedloop
+
+The fastest way to collect human preference data for LLMs.
+
+```python
+import feedloop
+
+feedloop.start()
+
+feedloop.compare(
+    prompt="Explain quantum computing",
+    outputs=["Response A", "Response B"],
+)
+
+# Rate in the browser, then export
+feedloop.export("preferences.jsonl")
+```

feedloop-0.1.0/PLAN.md

@@ -0,0 +1,265 @@
+# HITL Microservice — Implementation Plan
+
+## Context
+
+AI developers lack a frictionless way to collect human preference data during model iteration. Existing tools are either cloud-heavy (LangSmith, Humanloop), enterprise-scale (Labelbox, Scale AI), or platform-y (Argilla). This project fills the gap: a local-first, pip-installable tool that lets a solo developer go from "bad model output" to "DPO training example" in under a minute.
+
+**Positioning:** "The SQLite of human feedback" — zero config, instant value.
+
+---
+
+## Distribution Strategy
+
+**Both pip package + open-source GitHub repo.** pip for frictionless install, GitHub for trust and contributions.
+
+- Package name: `feedloop` (appears available on PyPI — register early)
+- Import name: `feedloop` regardless of pip name
+- License: MIT
+- Versioning: SemVer, start at `0.1.0`
+
+---
+
+## Architecture
+
+```
+Developer's Python script
+       │
+       │  feedloop.start()       → spawns FastAPI in a daemon thread
+       │  feedloop.compare(...)  → inserts row into SQLite, returns immediately
+       │  feedloop.export(...)   → reads SQLite, writes JSONL
+       │
+  ┌────▼──────────────────────────────────┐
+  │  FastAPI Server (background thread)    │
+  │                                        │
+  │  GET  /api/pending   → next comparison │
+  │  POST /api/feedback  → record choice   │
+  │  GET  /api/stats     → progress        │
+  │  GET  /api/export    → download JSONL  │
+  │  GET  /*             → React static    │
+  │                                        │
+  │  SQLite (~/.feedloop/feedloop.db, WAL mode)    │
+  └────┬──────────────────────────────────┘
+       │
+  ┌────▼──────────────────────────────────┐
+  │  React UI (static files)               │
+  │  Polls /api/pending every 1s           │
+  │  Side-by-side comparison + shortcuts   │
+  └────────────────────────────────────────┘
+```
+
+**Key design decisions:**
+- Background daemon thread (not subprocess) — shares SQLite, dies when script exits
+- SQLite WAL mode for concurrent reads across threads
+- `atexit` handler for graceful shutdown
+- Position randomization (shuffle A/B) to reduce bias — stored per row for correct export
+
+---
+
+## Project Structure
+
+```
+feedloop/
+├── pyproject.toml
+├── LICENSE (MIT)
+├── README.md
+├── Makefile
+├── src/
+│   └── feedloop/
+│       ├── __init__.py        # Public API: start(), compare(), export(), wait(), stop(), status()
+│       ├── __main__.py        # python -m feedloop CLI entry
+│       ├── _server.py         # FastAPI app + uvicorn background thread
+│       ├── _db.py             # SQLite schema, CRUD, WAL mode
+│       ├── _models.py         # Pydantic models
+│       ├── _routes.py         # API endpoints
+│       ├── _export.py         # SQLite → JSONL
+│       ├── _config.py         # Defaults (port, db path)
+│       ├── _static/           # React build output (bundled in wheel)
+│       └── py.typed
+├── frontend/
+│   ├── package.json
+│   ├── vite.config.ts         # outputs to ../src/feedloop/_static/
+│   ├── tsconfig.json
+│   ├── index.html
+│   └── src/
+│       ├── main.tsx
+│       ├── App.tsx
+│       ├── api.ts
+│       ├── components/
+│       │   ├── ComparisonView.tsx
+│       │   ├── ResponseCard.tsx
+│       │   ├── PromptHeader.tsx
+│       │   ├── FeedbackBar.tsx
+│       │   ├── ProgressBar.tsx
+│       │   ├── ExportPanel.tsx
+│       │   └── EmptyState.tsx
+│       └── hooks/
+│           └── useComparison.ts
+└── tests/
+    ├── test_db.py
+    ├── test_routes.py
+    ├── test_sdk.py
+    └── test_export.py
+```
+
+---
+
+## SDK API Design
+
+```python
+import feedloop
+
+# Launch server + open browser
+feedloop.start(port=7856, db_path=None, open_browser=True)
+
+# Submit comparisons (non-blocking)
+cid = feedloop.compare(
+    prompt="Explain quantum computing",
+    outputs=["Response A text", "Response B text"],
+    metadata={"model_a": "gpt-4", "model_b": "claude-3"}
+)
+
+# Optionally block until rated
+result = feedloop.wait(cid, timeout=60)
+
+# Check progress
+feedloop.status()  # {"pending": 5, "completed": 10, "total": 15}
+
+# Export DPO-formatted data
+feedloop.export("preferences.jsonl", session_id=None, format="dpo")
+
+# Cleanup
+feedloop.stop()
+```
+
+**Example end-to-end usage:**
+```python
+import feedloop
+
+feedloop.start()
+
+for prompt in test_prompts:
+    a = model_a.generate(prompt)
+    b = model_b.generate(prompt)
+    feedloop.compare(prompt=prompt, outputs=[a, b])
+
+input("Rate all comparisons in the browser, then press Enter...")
+feedloop.export("my_preferences.jsonl")
+```
+
+---
+
+## Data Model (SQLite)
+
+```sql
+CREATE TABLE comparisons (
+    id            TEXT PRIMARY KEY,       -- UUID
+    session_id    TEXT NOT NULL,
+    prompt        TEXT NOT NULL,
+    output_a      TEXT NOT NULL,
+    output_b      TEXT NOT NULL,
+    display_order TEXT NOT NULL,          -- 'ab' or 'ba' (position randomization)
+    status        TEXT NOT NULL DEFAULT 'pending',  -- pending | completed | skipped
+    chosen        TEXT,                  -- 'a' | 'b' | NULL
+    created_at    TEXT NOT NULL,
+    completed_at  TEXT,
+    metadata      TEXT                   -- JSON blob
+);
+```
+
+**DPO export format:**
+```json
+{"prompt": "...", "chosen": "...", "rejected": "..."}
+```
+
+---
+
+## Frontend Design
+
+Three states:
+1. **Empty** — "Waiting for comparisons... Run `feedloop.compare()` in your script." (polls every 1s)
+2. **Comparison** — Side-by-side A/B with keyboard shortcuts (`1` = A, `2` = B, `S` = skip), progress bar, markdown rendering
+3. **Done** — Summary stats + download JSONL button
+
+Tech: Vite + React + TypeScript, minimal CSS (no Tailwind), `react-markdown` for output rendering.
+
+---
+
+## Build & Packaging
+
+- **Build backend:** Hatchling (handles `src/` layout, includes `_static/` in wheel)
+- **Frontend build:** `cd frontend && npm run build` → outputs to `src/feedloop/_static/`
+- **Python build:** `python -m build` → wheel includes static files
+- **Dependencies:** `fastapi>=0.100`, `uvicorn[standard]>=0.20`, `pydantic>=2.0`
+- **Python target:** `>=3.10` (for `X | Y` union syntax)
+
+---
+
+## Phased Build Order
+
+### Phase 1: Foundation
+- [x] `pyproject.toml` + directory structure
+- [x] `_db.py` — SQLite schema, WAL mode, CRUD functions
+- [x] `_models.py` — Pydantic models
+- [x] `tests/test_db.py`
+
+### Phase 2: Backend API
+- [x] `_routes.py` — API endpoints
+- [x] `_server.py` — FastAPI app, CORS, static mount
+- [x] `_export.py` — JSONL export
+- [x] `tests/test_routes.py`
+
+### Phase 3: SDK
+- [x] `__init__.py` — public API (start, compare, export, wait, stop, status)
+- [x] Background thread server launch
+- [x] `__main__.py` — CLI entry
+- [ ] `tests/test_sdk.py`
+
+### Phase 4: Frontend
+- [x] Scaffold Vite + React + TS, configure output path
+- [x] `api.ts` + `useComparison` polling hook
+- [x] ComparisonView, ResponseCard, FeedbackBar components
+- [x] ProgressBar, EmptyState, ExportPanel
+- [x] Keyboard shortcuts + transitions
+- [x] Styling
+
+### Phase 5: Integration & Polish
+- [x] Build frontend into `_static/`, verify in wheel
+- [x] End-to-end test: start → compare → rate → export
+- [x] Position randomization (A/B shuffle)
+- [x] README with quickstart
+
+### Phase 6: Distribution
+- [ ] TestPyPI upload + fresh venv validation
+- [ ] GitHub repo + MIT license + CI
+- [ ] PyPI publish
+
+---
+
+## Future Roadmap
+
+### v1.1 — Uncertainty-based triggering
+- SDK accepts an `uncertainty` score per comparison
+- Only surfaces comparisons to the human when score exceeds a threshold
+- Aligns with active learning / cost-efficient RLHF
+
+### v1.5 — Training loop helpers
+- "Generate training script" button in UI
+- "Re-run prompts with new model" helper
+- Light guidance toward DPO fine-tuning without owning compute
+
+### v2.0 — Extended formats
+- Support for >2 outputs (ranking, not just pairwise)
+- Single-output Likert-scale rating (reward model data)
+- Export formats: `anthropic`, `openai`, `trl`
+- WebSocket for real-time updates (replace polling)
+
+---
+
+## Verification
+
+1. `pip install -e .` in a fresh venv
+2. Run `python demo.py` — browser opens, comparisons appear
+3. Rate all comparisons via keyboard shortcuts (`1`, `2`, `S`)
+4. Press Enter — exports to `preferences.jsonl` in DPO format
+5. `python -m build` — verify wheel contains `_static/index.html`
+6. `pip install dist/feedloop-0.1.0-py3-none-any.whl` in clean venv — verify it works

feedloop-0.1.0/README.md

@@ -0,0 +1,17 @@
+# feedloop
+
+The fastest way to collect human preference data for LLMs.
+
+```python
+import feedloop
+
+feedloop.start()
+
+feedloop.compare(
+    prompt="Explain quantum computing",
+    outputs=["Response A", "Response B"],
+)
+
+# Rate in the browser, then export
+feedloop.export("preferences.jsonl")
+```

feedloop-0.1.0/demo.py

@@ -0,0 +1,40 @@
+"""Demo script — run this to try feedloop."""
+
+import feedloop
+
+feedloop.start()
+
+# Add some sample comparisons
+feedloop.compare(
+    prompt="Explain quantum computing to a 5-year-old",
+    outputs=[
+        "Quantum computing uses qubits that can be in superposition, meaning they can represent both 0 and 1 simultaneously. This allows quantum computers to process many possibilities at once, making them powerful for certain types of calculations.",
+        "Imagine you have a magic coin. A normal coin is either heads or tails. But a magic coin can be both at the same time! A quantum computer uses millions of these magic coins to solve really hard puzzles super fast.",
+    ],
+    metadata={"model_a": "gpt-4", "model_b": "claude-3"},
+)
+
+feedloop.compare(
+    prompt="Write a haiku about programming",
+    outputs=[
+        "Lines of code cascade\nSilicon dreams come alive\nBugs hide in the night",
+        "Semicolons lost\nThe compiler screams in pain\nStack overflow saves",
+    ],
+)
+
+feedloop.compare(
+    prompt="What is the meaning of life?",
+    outputs=[
+        "The meaning of life is a deeply philosophical question that has been debated for centuries. Different traditions offer different answers — from religious purpose to existentialist self-creation.",
+        "42. But seriously, the meaning of life is whatever you decide to make it. Find what brings you joy, help others when you can, and try to leave things better than you found them.",
+    ],
+)
+
+print("\n3 comparisons ready. Rate them in the browser!")
+print("When done, press Enter to export.\n")
+
+input()
+
+count = feedloop.export("preferences.jsonl")
+print(f"\nExported {count} preferences to preferences.jsonl")
+feedloop.stop()

feedloop-0.1.0/frontend/index.html

@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>feedloop</title>
+  </head>
+  <body>
+    <div id="root"></div>
+    <script type="module" src="/src/main.tsx"></script>
+  </body>
+</html>