rlhf-feedback-loop 0.5.0

package/CHANGELOG.md

@@ -0,0 +1,26 @@
+# Changelog
+
+## 0.5.0 - 2026-03-03
+
+- Added autonomous GitOps workflows: agent auto-merge, Dependabot auto-merge, self-healing monitor, and merge-branch fallback.
+- Enabled CI proof artifact uploads and strengthened CI concurrency/branch scoping.
+- Added self-healing command layer (`scripts/self-healing-check.js`, `scripts/self-heal.js`) with unit tests.
+- Added semantic cache for ContextFS context-pack construction with TTL + similarity gating and provenance events.
+- Added secret-sync helper (`scripts/sync-gh-secrets-from-env.sh`) and docs for required repo settings/secrets.
+
+## 0.4.0 - 2026-03-03
+
+- Added rubric-based RLHF scoring with configurable criteria and weighted evaluation.
+- Added anti-reward-hacking safeguards: guardrail checks and multi-judge disagreement detection.
+- Added rubric-aware memory promotion gates for positive feedback.
+- Added rubric-aware context evaluation, prevention-rule dimensions, and DPO export metadata.
+- Extended API/MCP/Gemini contracts for rubric scores and guardrails.
+- Added automated proof harness for rubric + intent + API/MCP end-to-end validation (`proof/automation/*`).
+
+## 0.3.0 - 2026-03-03
+
+- Added production API server with secure auth defaults and safe-path checks.
+- Added local MCP server for Claude/Codex integrations.
+- Added ChatGPT, Gemini, Codex, Claude, and Amp adapter bundles.
+- Added budget guard and PaperBanana generation workflow.
+- Added platform research, packaging plan, and verification artifacts.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Igor Ganapolsky
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,308 @@
+# RLHF Feedback Loop
+
+[![CI](https://github.com/IgorGanapolsky/rlhf-feedback-loop/actions/workflows/ci.yml/badge.svg)](https://github.com/IgorGanapolsky/rlhf-feedback-loop/actions/workflows/ci.yml)
+[![Self-Healing](https://github.com/IgorGanapolsky/rlhf-feedback-loop/actions/workflows/self-healing-monitor.yml/badge.svg)](https://github.com/IgorGanapolsky/rlhf-feedback-loop/actions/workflows/self-healing-monitor.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![MCP Ready](https://img.shields.io/badge/MCP-ready-black)](adapters/mcp/server-stdio.js)
+[![DPO Ready](https://img.shields.io/badge/DPO-ready-blue)](scripts/export-dpo-pairs.js)
+
+Production-grade RLHF operations for AI agents across ChatGPT, Claude, Gemini, Codex, and Amp.
+
+## Quick Install
+
+Install on any platform with a single command. Be capturing feedback in under 5 minutes.
+
+### Universal (any platform)
+
+```bash
+npx rlhf-feedback-loop init
+node .rlhf/capture-feedback.js --feedback=up --context="test"
+```
+
+### Claude Code
+
+```bash
+cp plugins/claude-skill/SKILL.md .claude/skills/rlhf-feedback.md
+```
+
+Full guide: [plugins/claude-skill/INSTALL.md](plugins/claude-skill/INSTALL.md)
+
+### Codex
+
+```bash
+cat adapters/codex/config.toml >> ~/.codex/config.toml
+```
+
+Full guide: [plugins/codex-profile/INSTALL.md](plugins/codex-profile/INSTALL.md)
+
+### Gemini
+
+```bash
+cp adapters/gemini/function-declarations.json .gemini/rlhf-tools.json
+```
+
+Full guide: [plugins/gemini-extension/INSTALL.md](plugins/gemini-extension/INSTALL.md)
+
+### Amp
+
+```bash
+cp plugins/amp-skill/SKILL.md .amp/skills/rlhf-feedback.md
+```
+
+Full guide: [plugins/amp-skill/INSTALL.md](plugins/amp-skill/INSTALL.md)
+
+### ChatGPT (GPT Actions)
+
+Import `adapters/chatgpt/openapi.yaml` in the GPT Builder Actions editor.
+
+Full guide: [adapters/chatgpt/INSTALL.md](adapters/chatgpt/INSTALL.md)
+
+---
+
+## Value Proposition
+
+Most teams collect feedback but do not convert it into reliable behavior change.
+This project gives you a working loop:
+
+1. Capture thumbs up/down with context.
+2. Score outcomes with weighted rubrics and objective guardrails.
+3. Promote only schema-valid, rubric-eligible memories.
+4. Generate prevention rules from repeated mistakes and failed rubric dimensions.
+5. Export DPO-ready preference pairs with rubric deltas.
+6. Construct bounded context packs (constructor/loader/evaluator).
+7. Reuse the same core through API + MCP wrappers.
+8. Route intents through policy bundles with human checkpoints on high-risk actions.
+
+## Pricing
+
+| Plan | Price | What you get |
+|------|-------|-------------|
+| **Open Source** | $0 forever | Full source, self-hosted, MIT license, 314+ tests, 5-platform plugins |
+| **Cloud Pro** | $49/mo | Hosted HTTPS API on Railway, provisioned API key on payment, usage metering, email support |
+
+Get Cloud Pro: see the [landing page](docs/landing-page.html) or go straight to Stripe Checkout.
+
+---
+
+## Quick Start
+
+```bash
+cp .env.example .env
+npm test
+npm run prove:adapters
+npm run prove:automation
+npm run start:api
+```
+
+Set `RLHF_API_KEY` before running the API (or explicitly set `RLHF_ALLOW_INSECURE=true` for isolated local testing only).
+
+Capture feedback:
+
+```bash
+node .claude/scripts/feedback/capture-feedback.js \
+  --feedback=down \
+  --context="Claimed done without test evidence" \
+  --what-went-wrong="No proof attached" \
+  --what-to-change="Always run tests and include output" \
+  --tags="verification,testing"
+```
+
+## Integration Adapters
+
+- ChatGPT Actions: `adapters/chatgpt/openapi.yaml`
+- Claude MCP: `adapters/claude/.mcp.json`
+- Codex MCP: `adapters/codex/config.toml`
+- Gemini tools: `adapters/gemini/function-declarations.json`
+- Amp skill: `adapters/amp/skills/rlhf-feedback/SKILL.md`
+
+## API Surface
+
+- `POST /v1/feedback/capture`
+- `GET /v1/feedback/stats`
+- `GET /v1/intents/catalog`
+- `POST /v1/intents/plan`
+- `GET /v1/feedback/summary`
+- `POST /v1/feedback/rules`
+- `POST /v1/dpo/export`
+- `POST /v1/context/construct`
+- `POST /v1/context/evaluate`
+- `GET /v1/context/provenance`
+
+Spec: `openapi/openapi.yaml`
+
+## Versioning
+
+- Package/runtime release version: `package.json`
+- API contract version: `openapi/openapi.yaml`
+- MCP server protocol version: `adapters/mcp/server-stdio.js` `serverInfo.version`
+
+## ContextFS
+
+The repo includes a file-system context substrate for multi-agent memory orchestration:
+- Constructor: relevance-ranked context pack assembly
+- Loader: strict `maxItems` + `maxChars` budgeting
+- Evaluator: outcome/provenance logging for improvement loops
+
+Docs: [docs/CONTEXTFS.md](docs/CONTEXTFS.md)
+
+## MCP Policy Profiles
+
+Use least-privilege MCP profiles based on runtime risk:
+
+- `default`: full local toolset
+- `readonly`: read-heavy operations
+- `locked`: summary-only constrained mode
+
+Config: [config/mcp-allowlists.json](config/mcp-allowlists.json)
+
+## Rubric Engine
+
+Rubric config: `config/rubrics/default-v1.json`
+
+- Weighted criteria scoring (`1-5`)
+- Multi-judge disagreement detection
+- Objective guardrail checks (`testsPassed`, `pathSafety`, `budgetCompliant`)
+- Promotion gate blocks positive memory writes on unsafe/high-disagreement signals
+
+## Intent Router
+
+Versioned orchestration bundles define intent-to-action plans and checkpoint policy:
+
+- Bundle configs: `config/policy-bundles/*.json`
+- CLI list: `npm run intents:list`
+- CLI plan: `npm run intents:plan`
+
+The router marks high-risk intents as `checkpoint_required` unless explicitly approved.
+Details: [docs/INTENT_ROUTER.md](docs/INTENT_ROUTER.md)
+
+## Autonomous GitOps
+
+The repo now ships with PR-gated autonomous operations:
+
+- `CI` (`.github/workflows/ci.yml`): required quality gate (`npm test`, adapter proof, automation proof)
+- `Agent PR Auto-Merge` (`.github/workflows/agent-automerge.yml`): auto-merges eligible agent branches (`claude/*`, `codex/*`, `auto/*`, `agent/*`) after required checks pass
+- `Dependabot Auto-Merge` (`.github/workflows/dependabot-automerge.yml`): auto-approves and merges safe dependency updates after required checks pass
+- `Self-Healing Monitor` (`.github/workflows/self-healing-monitor.yml`): scheduled health checks, auto-created alert issue on failure, remediation PR generation when fixable
+- `Self-Healing Auto-Fix` (`.github/workflows/self-healing-auto-fix.yml`): scheduled safe-fix attempts that open remediation PRs
+- `Merge Branch to Main` (`.github/workflows/merge-branch.yml`): manual fallback that still uses PR flow and branch protections
+
+Required repo settings:
+
+- `main` protected + required check(s)
+- auto-merge enabled
+- branch deletion on merge enabled
+
+Secrets:
+
+- Required: `GH_PAT` (or rely on `GITHUB_TOKEN` where permitted)
+- Optional: `SENTRY_AUTH_TOKEN`, `SENTRY_DSN`
+- Optional (LLM router): `LLM_GATEWAY_BASE_URL`, `LLM_GATEWAY_API_KEY`, `TETRATE_API_KEY`
+
+Sync helper:
+
+```bash
+bash scripts/sync-gh-secrets-from-env.sh IgorGanapolsky/rlhf-feedback-loop
+```
+
+## Architecture
+
+### RLHF Feedback Loop
+
+```mermaid
+flowchart TD
+    A["👍/👎 User Feedback"] --> B["Capture Layer\n(context + tags)"]
+    B --> C{"Action Resolver"}
+    C -->|store-learning| D["Schema Validator"]
+    C -->|store-mistake| D
+    C -->|no-action| X["Discard"]
+    D -->|valid| E["Memory Store\n(learning / error)"]
+    D -->|invalid| X
+    E --> F["Analytics\n(trends + recurrence)"]
+    F --> G["Prevention Rules Engine"]
+    F --> H["DPO Export\n(prompt/chosen/rejected)"]
+    E --> I["Rubric Engine\n(weighted scoring + guardrails)"]
+    I -->|promotion gate| E
+```
+
+### Plugin Topology
+
+```mermaid
+flowchart LR
+    subgraph Adapters
+        GPT["ChatGPT\n(GPT Actions)"]
+        CL["Claude\n(MCP Server)"]
+        CX["Codex\n(MCP Config)"]
+        GEM["Gemini\n(Function Calling)"]
+        AMP["Amp\n(Skills Template)"]
+    end
+
+    subgraph Core["RLHF Feedback API"]
+        SV["Schema Validation"]
+        PR["Prevention Rules"]
+        DPO["DPO Export"]
+        BG["Budget Guard\n($10/mo cap)"]
+    end
+
+    GPT <--> Core
+    CL <--> Core
+    CX <--> Core
+    GEM <--> Core
+    AMP <--> Core
+```
+
+### PaperBanana (high-fidelity PNG)
+
+Generate richer architecture visuals with a budget guard:
+
+```bash
+npm run diagrams:paperbanana
+npm run budget:status
+```
+
+Docs: [docs/PAPERBANANA.md](docs/PAPERBANANA.md)
+Verification evidence: [docs/VERIFICATION_EVIDENCE.md](docs/VERIFICATION_EVIDENCE.md)
+Compatibility proof artifacts: [proof/compatibility/report.md](proof/compatibility/report.md), [proof/compatibility/report.json](proof/compatibility/report.json)
+Automation proof artifacts: [proof/automation/report.md](proof/automation/report.md), [proof/automation/report.json](proof/automation/report.json)
+
+## Budget Guardrail
+
+Default monthly cap is `$10` for paid external operations.
+The local budget ledger blocks additional spend if cap would be exceeded.
+
+## Semantic Cache (Cost + Latency)
+
+Context pack construction now supports semantic cache reuse for similar queries:
+
+- token-overlap (Jaccard) similarity gate
+- TTL-bound cache entries
+- full provenance (`context_pack_cache_hit`)
+
+Environment toggles:
+
+- `RLHF_SEMANTIC_CACHE_ENABLED=true|false` (default `true`)
+- `RLHF_SEMANTIC_CACHE_THRESHOLD=0.7`
+- `RLHF_SEMANTIC_CACHE_TTL_SECONDS=86400`
+
+This directly reduces repeated retrieval/LLM context assembly work and improves response latency under budget constraints.
+
+## Optional Tetrate Router
+
+Not required for core local RLHF logic.
+Recommended only when routing paid LLM calls (PaperBanana, external judges, hosted control-plane features):
+
+- centralized provider routing
+- price/fallback control
+- unified usage observability
+
+## Commercialization
+
+- OSS core for adoption
+- Hosted control plane for teams
+- Enterprise support and compliance features
+
+See:
+
+- [docs/PACKAGING_AND_SALES_PLAN.md](docs/PACKAGING_AND_SALES_PLAN.md)
+- [docs/PLATFORM_RESEARCH_2026-03-03.md](docs/PLATFORM_RESEARCH_2026-03-03.md)
+- [docs/PLUGIN_DISTRIBUTION.md](docs/PLUGIN_DISTRIBUTION.md)
+- [docs/AUTONOMOUS_GITOPS.md](docs/AUTONOMOUS_GITOPS.md)

package/adapters/README.md

@@ -0,0 +1,8 @@
+# Adapter Bundles
+
+- `chatgpt/openapi.yaml`: import into GPT Actions.
+- `gemini/function-declarations.json`: Gemini function-calling definitions.
+- `mcp/server-stdio.js`: local MCP server for Claude/Codex.
+- `claude/.mcp.json`: example Claude Code MCP config.
+- `codex/config.toml`: example Codex MCP profile section.
+- `amp/skills/rlhf-feedback/SKILL.md`: Amp skill template.

package/adapters/amp/skills/rlhf-feedback/SKILL.md

@@ -0,0 +1,20 @@
+---
+name: rlhf-feedback
+description: Capture thumbs feedback and apply prevention rules before coding
+---
+
+# Amp RLHF Skill
+
+On explicit user feedback:
+
+```bash
+node .claude/scripts/feedback/capture-feedback.js --feedback=up --context="..." --tags="..."
+node .claude/scripts/feedback/capture-feedback.js --feedback=down --context="..." --tags="..."
+```
+
+Before major implementation:
+
+```bash
+npm run feedback:summary
+npm run feedback:rules
+```

package/adapters/chatgpt/INSTALL.md

@@ -0,0 +1,80 @@
+# ChatGPT GPT Actions: RLHF Feedback Loop Install
+
+Import the OpenAPI spec into a Custom GPT in under 5 minutes. No coding required.
+
+## Prerequisites
+
+- A ChatGPT Plus or Team account (Custom GPTs require a paid plan)
+- RLHF API running at a public HTTPS URL (see [Deployment docs](../../docs/deployment.md))
+
+## Step 1 — Open GPT Builder
+
+1. Go to [https://chat.openai.com/gpts/editor](https://chat.openai.com/gpts/editor)
+2. Click **Create a GPT**
+3. Switch to the **Configure** tab
+
+## Step 2 — Add Actions
+
+1. Scroll to the **Actions** section
+2. Click **Create new action**
+3. Click **Import from URL** — paste your hosted spec URL:
+   ```
+   https://<your-railway-domain>/openapi.yaml
+   ```
+   Or click **Upload file** and select:
+   ```
+   adapters/chatgpt/openapi.yaml
+   ```
+
+## Step 3 — Set Authentication
+
+In the Actions panel:
+
+1. Select **Authentication type: API Key**
+2. **Auth type**: Bearer
+3. **API Key**: paste your `RLHF_API_KEY` value
+
+## Step 4 — Update the Server URL
+
+In the imported spec, confirm the `servers.url` points to your deployed API:
+
+```yaml
+servers:
+  - url: https://<your-railway-domain>
+```
+
+If you uploaded the file, edit the server URL in the GPT Actions editor.
+
+## Step 5 — Verify
+
+Click **Test** on the `captureFeedback` action:
+
+```json
+{
+  "signal": "up",
+  "context": "GPT Actions install verified"
+}
+```
+
+Expected response: `200 OK` with `{ "id": "fb-...", "status": "captured" }`.
+
+## Available Actions
+
+| Action | Method | Path | Description |
+|---|---|---|---|
+| `captureFeedback` | POST | `/v1/feedback/capture` | Capture thumbs up/down signal |
+| `getFeedbackStats` | GET | `/v1/feedback/stats` | Aggregated feedback statistics |
+| `getFeedbackSummary` | GET | `/v1/feedback/summary` | Recent feedback summary |
+| `generatePreventionRules` | POST | `/v1/feedback/rules` | Generate prevention rules |
+| `exportDpoPairs` | POST | `/v1/dpo/export` | Export DPO preference pairs |
+| `listIntentCatalog` | GET | `/v1/intents/catalog` | List available intents |
+| `planIntent` | POST | `/v1/intents/plan` | Generate policy-scoped plan |
+| `constructContextPack` | POST | `/v1/context/construct` | Build context pack |
+
+Full spec: `adapters/chatgpt/openapi.yaml`
+
+## Troubleshooting
+
+- **401 Unauthorized**: Verify `RLHF_API_KEY` is set and matches the Bearer token
+- **Connection refused**: Confirm Railway deployment is live (`curl https://<domain>/health`)
+- **Schema errors**: Ensure you are using the latest `openapi.yaml` (version 1.1.0+)