cannyforge 0.2.0__tar.gz → 0.3.0__tar.gz

cannyforge-0.3.0/PKG-INFO

@@ -0,0 +1,282 @@
+Metadata-Version: 2.4
+Name: cannyforge
+Version: 0.3.0
+Summary: Self-improving agents with closed-loop learning — agents that learn to get it right
+License: BSL-1.1
+Project-URL: Homepage, https://github.com/cannyforge/cannyforge
+Project-URL: Documentation, https://github.com/cannyforge/cannyforge#readme
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Provides-Extra: claude
+Requires-Dist: anthropic>=0.39.0; extra == "claude"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == "openai"
+Provides-Extra: deepseek
+Requires-Dist: openai>=1.0.0; extra == "deepseek"
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.2.0; extra == "langgraph"
+Provides-Extra: mcp
+Requires-Dist: mcp[cli]>=1.0.0; extra == "mcp"
+Provides-Extra: dashboard
+Requires-Dist: streamlit>=1.30.0; extra == "dashboard"
+Provides-Extra: all
+Requires-Dist: anthropic>=0.39.0; extra == "all"
+Requires-Dist: openai>=1.0.0; extra == "all"
+Requires-Dist: langgraph>=0.2.0; extra == "all"
+Requires-Dist: mcp[cli]>=1.0.0; extra == "all"
+Requires-Dist: streamlit>=1.30.0; extra == "all"
+Dynamic: license-file
+
+# CannyForge
+
+**Reliability memory for tool-using LLM agents.**
+
+CannyForge watches your agent make mistakes, learns corrections, and injects them as SystemMessages before each LLM call. Your agent gets better over time — no retraining required.
+
+```
+Agent makes errors → CannyForge learns corrections → Agent stops repeating them
+```
+
+![CannyForge demo: 80% → 100% on real LLM](docs/demo.gif)
+
+## Quick Start (LangGraph)
+
+```python
+from cannyforge import CannyForge
+from cannyforge.adapters.langgraph import CannyForgeMiddleware
+from langgraph.prebuilt import create_react_agent
+
+forge = CannyForge()
+middleware = CannyForgeMiddleware(forge)
+agent = create_react_agent(model, tools,
+    pre_model_hook=middleware.before_model,
+    post_model_hook=middleware.after_model)
+
+# Just run tasks. CannyForge records errors via after_model.
+# After learning, before_model injects corrections as SystemMessages.
+```
+
+## v0.3 Direction
+
+The benchmark branch is focused on three adoption-oriented improvements:
+
+- better reliability loops for tool-using agents
+- broader benchmark coverage for arguments, multi-step execution, and recovery
+- portable learned skills for assistant and workflow reuse
+
+This is aimed at modern agent stacks, including LangGraph, LangChain-style tool agents,
+CrewAI-style orchestration, MCP-connected assistants, and personal assistant workflows.
+
+See [docs/v0.3-public.md](/home/xiwei/pg/cannyforge/docs/v0.3-public.md) for the public summary.
+
+## How It Works
+
+1. **Record errors** — `after_model` detects tool failures and records them
+2. **Learn corrections** — `run_learning_cycle()` clusters errors and generates specific correction text (template or LLM-generated)
+3. **Inject corrections** — `before_model` prepends a SystemMessage with all active corrections before each LLM call
+4. **Track effectiveness** — corrections that prevent recurrence are kept; ineffective ones can be regenerated
+
+The correction is specific and actionable:
+```
+[CANNYFORGE] Learned rules for this request:
+- When the task involves report, summary, sales, use `generate_report`, NOT `get_data`.
+  Example: "Create a summary of Q4 sales performance"
+```
+
+## Demo: 60% → 100% on Real LLM
+
+```bash
+pip install langgraph langchain-openai
+# Set LLM_API_KEY in .env
+python scenarios/demo_cannyforge.py
+```
+
+This runs 15 ambiguous tool-selection tasks twice:
+- **Phase 1**: baseline without corrections — records errors
+- **Learning**: generates corrections from observed errors
+- **Phase 2**: same tasks with correction injection — accuracy improves
+
+Real output with DeepSeek:
+```
+Phase 1 accuracy: 9/15 (60%)
+Phase 2 accuracy: 15/15 (100%)
+Tasks fixed:
+  - Restart the staging server -> execute_action
+  - Send an alert to the on-call team -> execute_action
+  - Deploy the latest build to production -> execute_action
+  - Create a summary of Q4 sales performance -> generate_report
+  - Write up a status report for this sprint -> generate_report
+  - Generate a monthly uptime report -> generate_report
+```
+
+No simulated errors. No hand-crafted rules. Real LLM decisions, real corrections from the pipeline.
+
+## Install
+
+```bash
+pip install cannyforge           # from PyPI
+```
+
+Or from source:
+```bash
+git clone https://github.com/cannyforge/cannyforge.git
+cd cannyforge
+pip install -e .
+```
+
+## Scenarios
+
+| Script | Purpose |
+|--------|---------|
+| `scenarios/demo_cannyforge.py` | **Canonical demo** — full pipeline: baseline → learn → improve |
+| `scenarios/demo_langgraph_tool_use.py` | Minimal quickstart — 3-line integration |
+| `scenarios/demo.py` | Animated terminal demo (internal skill system) |
+
+Older demo scripts are in `scenarios/archive/` for reference.
+
+## Framework Coverage
+
+| Surface | Current repo path |
+|--------|-------------------|
+| LangGraph middleware | `cannyforge/adapters/langgraph.py` |
+| LangChain adapter | `cannyforge/adapters/langchain.py` |
+| CrewAI adapter | `cannyforge/adapters/crewai.py` |
+| MCP server | `cannyforge/mcp_server.py` |
+| Bundled assistant skills | `cannyforge/bundled_skills/` |
+
+CannyForge is designed to sit on top of existing agent frameworks rather than replace them.
+The public focus for v0.3 is improved tool routing, argument quality, multi-step execution, and
+reusable skills.
+
+## Core Architecture
+
+### Corrections Pipeline (LangGraph integration)
+
+```
+cannyforge/corrections.py    — Correction dataclass + CorrectionGenerator
+cannyforge/adapters/langgraph.py — CannyForgeMiddleware (pre/post model hooks)
+cannyforge/knowledge.py      — KnowledgeBase stores corrections + rules
+cannyforge/learning.py       — PatternDetector + LearningEngine
+cannyforge/core.py           — CannyForge orchestrator
+```
+
+**CorrectionGenerator** turns error clusters into actionable text:
+- **Template mode** (no LLM): groups by `(wrong_tool, right_tool)`, extracts keywords, formats guidance
+- **LLM mode**: sends error cluster to LLM asking for a generalized rule covering unseen tasks
+
+**CannyForgeMiddleware** hooks into LangGraph's `create_react_agent`:
+- `before_model`: injects always-on corrections + conditional rules as a SystemMessage
+- `after_model`: records tool failures, tracks correction effectiveness
+
+### Internal Skill System
+
+CannyForge also includes a declarative skill system for standalone use (without LangGraph):
+
+- Skills defined via `SKILL.md` files ([AgentSkills.io](https://agentskills.io/specification) spec)
+- Three-tier execution: custom handler → LLM multi-step → template fallback
+- PATTERN_LIBRARY with condition-based rules for internal context signals
+- Rule lifecycle: ACTIVE → PROBATION → DORMANT → resurrection
+
+```python
+from cannyforge import CannyForge
+forge = CannyForge()
+result = forge.execute("Write an email about the 3 PM meeting")
+```
+
+See `scenarios/demo.py` for the animated terminal demo of this path.
+
+## How Learning Works
+
+### 1. Error Recording
+
+```python
+# Via middleware (automatic):
+agent = create_react_agent(llm, tools, post_model_hook=middleware.after_model)
+
+# Or manual:
+forge.learning_engine.record_error(
+    skill_name="tool_use",
+    task_description="Create a Q4 summary",
+    error_type="WrongToolError",
+    error_message="Called get_data instead of generate_report",
+    context_snapshot={...},
+)
+```
+
+### 2. Learning Cycle
+
+```python
+metrics = forge.run_learning_cycle(min_frequency=2, min_confidence=0.3)
+# Produces:
+#   - Condition-based rules (for internal skill system)
+#   - Corrections (for LangGraph injection)
+```
+
+### 3. Correction Injection
+
+```python
+corrections = forge.knowledge_base.get_corrections("tool_use")
+# [Correction(content="When task involves report, summary... use generate_report, NOT get_data")]
+
+# Automatically injected by middleware.before_model() as a SystemMessage
+```
+
+## Run Tests
+
+```bash
+pytest tests/ -v
+```
+
+## Project Structure
+
+```
+cannyforge/
+├── cannyforge/
+│   ├── core.py              # CannyForge orchestrator
+│   ├── corrections.py       # Correction + CorrectionGenerator
+│   ├── knowledge.py         # KnowledgeBase, Rules, Conditions, Actions
+│   ├── learning.py          # PatternDetector, LearningEngine
+│   ├── skills.py            # Declarative skill system
+│   ├── llm.py               # LLM providers (Claude, OpenAI, DeepSeek)
+│   ├── tools.py             # Tool definitions and execution
+│   ├── storage.py           # Storage backends (JSON, SQLite)
+│   └── adapters/
+│       └── langgraph.py     # LangGraph middleware (pre/post model hooks)
+│
+├── scenarios/
+│   ├── demo_cannyforge.py   # Canonical demo (corrections pipeline)
+│   ├── demo_langgraph_tool_use.py  # Minimal quickstart
+│   └── demo.py              # Animated demo (internal skill system)
+│
+├── tests/                   # Test suite
+└── skills/                  # Built-in skill definitions (SKILL.md)
+```
+
+## Further Reading
+
+- Blog post: [From Prompt Tweaks to Learning Machines: The Agent Skill Primitive](https://medium.com/@xiweizhou/from-prompt-tweaks-to-learning-machines-the-agent-skill-primitive-93c8fa9dec8c?sk=ac888430da699bce7b635456ae2b1166)
+
+## License
+
+Licensed under [BSL 1.1](LICENSE). Free to use in production, but you may not offer CannyForge as a competing hosted service. Converts to Apache 2.0 on 2030-03-01. See LICENSE for full terms.
+
+For commercial licensing inquiries: cannyforge@gmail.com
+
+---
+
+**CannyForge** — Your agent makes fewer repeated mistakes over time, with measurable evidence.

cannyforge-0.3.0/README.md

@@ -0,0 +1,238 @@
+# CannyForge
+
+**Reliability memory for tool-using LLM agents.**
+
+CannyForge watches your agent make mistakes, learns corrections, and injects them as SystemMessages before each LLM call. Your agent gets better over time — no retraining required.
+
+```
+Agent makes errors → CannyForge learns corrections → Agent stops repeating them
+```
+
+![CannyForge demo: 80% → 100% on real LLM](docs/demo.gif)
+
+## Quick Start (LangGraph)
+
+```python
+from cannyforge import CannyForge
+from cannyforge.adapters.langgraph import CannyForgeMiddleware
+from langgraph.prebuilt import create_react_agent
+
+forge = CannyForge()
+middleware = CannyForgeMiddleware(forge)
+agent = create_react_agent(model, tools,
+    pre_model_hook=middleware.before_model,
+    post_model_hook=middleware.after_model)
+
+# Just run tasks. CannyForge records errors via after_model.
+# After learning, before_model injects corrections as SystemMessages.
+```
+
+## v0.3 Direction
+
+The benchmark branch is focused on three adoption-oriented improvements:
+
+- better reliability loops for tool-using agents
+- broader benchmark coverage for arguments, multi-step execution, and recovery
+- portable learned skills for assistant and workflow reuse
+
+This is aimed at modern agent stacks, including LangGraph, LangChain-style tool agents,
+CrewAI-style orchestration, MCP-connected assistants, and personal assistant workflows.
+
+See [docs/v0.3-public.md](/home/xiwei/pg/cannyforge/docs/v0.3-public.md) for the public summary.
+
+## How It Works
+
+1. **Record errors** — `after_model` detects tool failures and records them
+2. **Learn corrections** — `run_learning_cycle()` clusters errors and generates specific correction text (template or LLM-generated)
+3. **Inject corrections** — `before_model` prepends a SystemMessage with all active corrections before each LLM call
+4. **Track effectiveness** — corrections that prevent recurrence are kept; ineffective ones can be regenerated
+
+The correction is specific and actionable:
+```
+[CANNYFORGE] Learned rules for this request:
+- When the task involves report, summary, sales, use `generate_report`, NOT `get_data`.
+  Example: "Create a summary of Q4 sales performance"
+```
+
+## Demo: 60% → 100% on Real LLM
+
+```bash
+pip install langgraph langchain-openai
+# Set LLM_API_KEY in .env
+python scenarios/demo_cannyforge.py
+```
+
+This runs 15 ambiguous tool-selection tasks twice:
+- **Phase 1**: baseline without corrections — records errors
+- **Learning**: generates corrections from observed errors
+- **Phase 2**: same tasks with correction injection — accuracy improves
+
+Real output with DeepSeek:
+```
+Phase 1 accuracy: 9/15 (60%)
+Phase 2 accuracy: 15/15 (100%)
+Tasks fixed:
+  - Restart the staging server -> execute_action
+  - Send an alert to the on-call team -> execute_action
+  - Deploy the latest build to production -> execute_action
+  - Create a summary of Q4 sales performance -> generate_report
+  - Write up a status report for this sprint -> generate_report
+  - Generate a monthly uptime report -> generate_report
+```
+
+No simulated errors. No hand-crafted rules. Real LLM decisions, real corrections from the pipeline.
+
+## Install
+
+```bash
+pip install cannyforge           # from PyPI
+```
+
+Or from source:
+```bash
+git clone https://github.com/cannyforge/cannyforge.git
+cd cannyforge
+pip install -e .
+```
+
+## Scenarios
+
+| Script | Purpose |
+|--------|---------|
+| `scenarios/demo_cannyforge.py` | **Canonical demo** — full pipeline: baseline → learn → improve |
+| `scenarios/demo_langgraph_tool_use.py` | Minimal quickstart — 3-line integration |
+| `scenarios/demo.py` | Animated terminal demo (internal skill system) |
+
+Older demo scripts are in `scenarios/archive/` for reference.
+
+## Framework Coverage
+
+| Surface | Current repo path |
+|--------|-------------------|
+| LangGraph middleware | `cannyforge/adapters/langgraph.py` |
+| LangChain adapter | `cannyforge/adapters/langchain.py` |
+| CrewAI adapter | `cannyforge/adapters/crewai.py` |
+| MCP server | `cannyforge/mcp_server.py` |
+| Bundled assistant skills | `cannyforge/bundled_skills/` |
+
+CannyForge is designed to sit on top of existing agent frameworks rather than replace them.
+The public focus for v0.3 is improved tool routing, argument quality, multi-step execution, and
+reusable skills.
+
+## Core Architecture
+
+### Corrections Pipeline (LangGraph integration)
+
+```
+cannyforge/corrections.py    — Correction dataclass + CorrectionGenerator
+cannyforge/adapters/langgraph.py — CannyForgeMiddleware (pre/post model hooks)
+cannyforge/knowledge.py      — KnowledgeBase stores corrections + rules
+cannyforge/learning.py       — PatternDetector + LearningEngine
+cannyforge/core.py           — CannyForge orchestrator
+```
+
+**CorrectionGenerator** turns error clusters into actionable text:
+- **Template mode** (no LLM): groups by `(wrong_tool, right_tool)`, extracts keywords, formats guidance
+- **LLM mode**: sends error cluster to LLM asking for a generalized rule covering unseen tasks
+
+**CannyForgeMiddleware** hooks into LangGraph's `create_react_agent`:
+- `before_model`: injects always-on corrections + conditional rules as a SystemMessage
+- `after_model`: records tool failures, tracks correction effectiveness
+
+### Internal Skill System
+
+CannyForge also includes a declarative skill system for standalone use (without LangGraph):
+
+- Skills defined via `SKILL.md` files ([AgentSkills.io](https://agentskills.io/specification) spec)
+- Three-tier execution: custom handler → LLM multi-step → template fallback
+- PATTERN_LIBRARY with condition-based rules for internal context signals
+- Rule lifecycle: ACTIVE → PROBATION → DORMANT → resurrection
+
+```python
+from cannyforge import CannyForge
+forge = CannyForge()
+result = forge.execute("Write an email about the 3 PM meeting")
+```
+
+See `scenarios/demo.py` for the animated terminal demo of this path.
+
+## How Learning Works
+
+### 1. Error Recording
+
+```python
+# Via middleware (automatic):
+agent = create_react_agent(llm, tools, post_model_hook=middleware.after_model)
+
+# Or manual:
+forge.learning_engine.record_error(
+    skill_name="tool_use",
+    task_description="Create a Q4 summary",
+    error_type="WrongToolError",
+    error_message="Called get_data instead of generate_report",
+    context_snapshot={...},
+)
+```
+
+### 2. Learning Cycle
+
+```python
+metrics = forge.run_learning_cycle(min_frequency=2, min_confidence=0.3)
+# Produces:
+#   - Condition-based rules (for internal skill system)
+#   - Corrections (for LangGraph injection)
+```
+
+### 3. Correction Injection
+
+```python
+corrections = forge.knowledge_base.get_corrections("tool_use")
+# [Correction(content="When task involves report, summary... use generate_report, NOT get_data")]
+
+# Automatically injected by middleware.before_model() as a SystemMessage
+```
+
+## Run Tests
+
+```bash
+pytest tests/ -v
+```
+
+## Project Structure
+
+```
+cannyforge/
+├── cannyforge/
+│   ├── core.py              # CannyForge orchestrator
+│   ├── corrections.py       # Correction + CorrectionGenerator
+│   ├── knowledge.py         # KnowledgeBase, Rules, Conditions, Actions
+│   ├── learning.py          # PatternDetector, LearningEngine
+│   ├── skills.py            # Declarative skill system
+│   ├── llm.py               # LLM providers (Claude, OpenAI, DeepSeek)
+│   ├── tools.py             # Tool definitions and execution
+│   ├── storage.py           # Storage backends (JSON, SQLite)
+│   └── adapters/
+│       └── langgraph.py     # LangGraph middleware (pre/post model hooks)
+│
+├── scenarios/
+│   ├── demo_cannyforge.py   # Canonical demo (corrections pipeline)
+│   ├── demo_langgraph_tool_use.py  # Minimal quickstart
+│   └── demo.py              # Animated demo (internal skill system)
+│
+├── tests/                   # Test suite
+└── skills/                  # Built-in skill definitions (SKILL.md)
+```
+
+## Further Reading
+
+- Blog post: [From Prompt Tweaks to Learning Machines: The Agent Skill Primitive](https://medium.com/@xiweizhou/from-prompt-tweaks-to-learning-machines-the-agent-skill-primitive-93c8fa9dec8c?sk=ac888430da699bce7b635456ae2b1166)
+
+## License
+
+Licensed under [BSL 1.1](LICENSE). Free to use in production, but you may not offer CannyForge as a competing hosted service. Converts to Apache 2.0 on 2030-03-01. See LICENSE for full terms.
+
+For commercial licensing inquiries: cannyforge@gmail.com
+
+---
+
+**CannyForge** — Your agent makes fewer repeated mistakes over time, with measurable evidence.

{cannyforge-0.2.0 → cannyforge-0.3.0}/cannyforge/__init__.py

@@ -48,8 +48,10 @@ from cannyforge.learning import (
     ErrorRepository,
 )
 from cannyforge.tools import ToolDefinition, ToolExecutor, ToolRegistry
+from cannyforge.adapters.langgraph import CannyForgeMiddleware
+from cannyforge.corrections import Correction, CorrectionGenerator
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 __all__ = [
     # Core
@@ -91,4 +93,9 @@ __all__ = [
     "ToolDefinition",
     "ToolExecutor",
     "ToolRegistry",
+    # Adapters
+    "CannyForgeMiddleware",
+    # Corrections
+    "Correction",
+    "CorrectionGenerator",
 ]

{cannyforge-0.2.0 → cannyforge-0.3.0}/cannyforge/adapters/__init__.py

@@ -6,5 +6,6 @@ AI/agent frameworks. Each adapter is ~50-80 lines.
 
 Available adapters:
     - langchain: CannyForgeTool for LangChain
+    - langgraph: CannyForgeMiddleware for LangGraph agents
     - crewai: CannyForgeCrewTool for CrewAI
 """