fasticrl 1.0.0__tar.gz

fasticrl-1.0.0/PKG-INFO

@@ -0,0 +1,215 @@
+Metadata-Version: 2.3
+Name: fasticrl
+Version: 1.0.0
+Summary: In-Context Reinforcement Learning framework for LLMs — no fine-tuning required.
+Keywords: reinforcement-learning,in-context-learning,icrl,llm,ai,agents,prompt-engineering
+Author: Maximilian König
+Author-email: Maximilian König <maximilian.koenig@mein.gmx>
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: agno>=2.6.5
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: tqdm>=4.67.3
+Requires-Dist: ollama>=0.6.2 ; extra == 'ollama'
+Requires-Dist: openai>=2.32.0 ; extra == 'openai'
+Requires-Python: >=3.13
+Project-URL: Homepage, https://github.com/makoeta/FastICRL
+Project-URL: Issues, https://github.com/makoeta/FastICRL/issues
+Project-URL: Source, https://github.com/makoeta/FastICRL
+Provides-Extra: ollama
+Provides-Extra: openai
+Description-Content-Type: text/markdown
+
+# FastICRL
+
+<p align="center">
+  <a href="https://pypi.org/project/fasticrl/"><img src="https://img.shields.io/pypi/v/fasticrl.svg" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/fasticrl/"><img src="https://img.shields.io/pypi/pyversions/fasticrl.svg" alt="Python"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
+  <img src="https://img.shields.io/badge/no%20fine--tuning-required-brightgreen" alt="No fine-tuning required">
+  <img src="https://img.shields.io/badge/no%20GPU-required-blue" alt="No GPU required">
+  <a href="https://github.com/agno-agi/agno"><img src="https://img.shields.io/badge/powered%20by-agno-8A2BE2" alt="Powered by agno"></a>
+</p>
+
+**In-Context Reinforcement Learning for LLMs — no fine-tuning, no gradient updates, no GPU.**
+
+FastICRL implements the ICRL paradigm from [*Reward Is Enough: LLMs Are In-Context Reinforcement Learners*](https://arxiv.org/abs/2506.06303) (Song et al., 2025). A learner LLM improves its outputs purely by reading its own history of attempts and rewards inside the context window — guided by a meta-cognitive strategist. No training, no infrastructure, just inference.
+
+---
+
+## How it works
+
+Three LLM agents collaborate in a feedback loop:
+
+```text
+┌──────────────────────────────────────────────────┐
+│                   ICRLLearner                    │
+│                                                  │
+│  Task ──► Learner ──► Output ──► Reward Agent    │
+│             ▲                         │          │
+│             │        Attempt          │          │
+│             │  (task, output, score)  │          │
+│             └─────────────────────────┘          │
+│                          │                       │
+│                  (every N episodes)              │
+│                          ▼                       │
+│                      Strategist                  │
+│                (refines the strategy)            │
+└──────────────────────────────────────────────────┘
+```
+
+| Agent | Role |
+| --- | --- |
+| **Learner** | Generates task outputs; balances exploration vs. exploitation based on reward history |
+| **Reward** | Scores each output on a 0–10 scale (acts as the reward function) |
+| **Strategist** | Analyzes past attempts to synthesize actionable strategies for future episodes |
+
+Each agent can be backed by a different model — e.g. a cheap model for reward, a powerful one for the learner.
+
+---
+
+## Installation
+
+```bash
+pip install fasticrl
+```
+
+Or with [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv add fasticrl
+```
+
+Model provider extras (install whichever you use):
+
+```bash
+pip install "fasticrl[openai]"   # OpenAI
+pip install "fasticrl[ollama]"   # Ollama (local models)
+```
+
+Requires Python ≥ 3.13.
+
+---
+
+## Quick start
+
+```python
+from fasticrl import ICRLLearner
+from agno.models.openai import OpenAIChat
+
+model = OpenAIChat(id="gpt-4o-mini")
+
+learner = ICRLLearner(
+    learner_model=model,
+    reward_model=model,
+    strategy_model=model,
+    task_description="Write a concise, compelling product description for an e-commerce listing.",
+    tasks=[
+        "Wireless noise-cancelling headphones",
+        "Ergonomic standing desk",
+        "Portable espresso maker",
+    ],
+)
+
+# Run 3 episodes, update strategy every 2 steps, show progress bar
+learner.auto_learn(episodes=3, batch_size=2, cli_mode=True, strategy_update_interval=2)
+
+# Inspect what the agent learned
+print(learner.strategy)
+```
+
+---
+
+## API
+
+### `ICRLLearner`
+
+```python
+ICRLLearner(
+    learner_model,        # agno Model for the learner agent
+    reward_model,         # agno Model for the reward agent
+    strategy_model,       # agno Model for the strategist agent
+    task_description,     # describes the overall task domain (required)
+    tasks,                # list of concrete task instances to cycle through
+    buffer,               # optional: pre-loaded list of Attempt objects
+    strategy,             # optional: pre-loaded strategy string
+)
+```
+
+#### Key methods
+
+| Method | Description |
+| --- | --- |
+| `auto_learn(episodes, batch_size, cli_mode, strategy_update_interval)` | Run N episodes. `batch_size > 1` parallelizes tasks with a thread pool. `cli_mode=True` shows a progress bar. `strategy_update_interval=K` refreshes the strategy every K episodes. |
+| `generate_action(task)` | Run the learner on a single task and return its output |
+| `generate_reward(task, action)` | Score a learner output with the reward agent |
+| `generate_attempt_by_present_task()` | Single step: generate + score the current task |
+| `update_strategy()` | Ask the strategist to refine the strategy from the current buffer |
+| `to_yaml(path)` | Persist the full agent state (buffer + strategy) to a YAML file |
+| `ICRLLearner.from_yaml(path, ...)` | Resume from a saved state |
+
+---
+
+## Saving and resuming
+
+```python
+# Save
+learner.to_yaml("my_agent.yaml")
+
+# Resume later
+learner = ICRLLearner.from_yaml(
+    "my_agent.yaml",
+    learner_model=model,
+    reward_model=model,
+    strategy_model=model,
+)
+learner.auto_learn(episodes=5)
+```
+
+---
+
+## Using Ollama (local models)
+
+```python
+from agno.models.ollama import Ollama
+
+learner = ICRLLearner(
+    learner_model=Ollama(id="llama3.2"),
+    reward_model=Ollama(id="llama3.2"),
+    strategy_model=Ollama(id="llama3.2"),
+    task_description="...",
+    tasks=[...],
+)
+```
+
+Any [agno](https://github.com/agno-agi/agno)-compatible model works.
+
+---
+
+## Citation
+
+This project is based on and inspired by the following papers:
+
+*Reward Is Enough: LLMs Are In-Context Reinforcement Learners*  
+Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Shangtong Zhang, Yanjun Qi  
+arXiv:2506.06303 — [https://arxiv.org/abs/2506.06303](https://arxiv.org/abs/2506.06303)
+
+*Large Language Models as Optimizers*  
+Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen  
+arXiv:2309.03409 — [https://arxiv.org/abs/2309.03409](https://arxiv.org/abs/2309.03409)
+
+*Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs*  
+Yifan Zhou, Sachin Grover, Mohamed El Mistiri, Kamalesh Kalirathinam, Pratyush Kerhalkar, Swaroop Mishra, Neelesh Kumar, Sanket Gaurav, Oya Aran, Heni Ben Amor  
+NeurIPS 2025 — [https://openreview.net/forum?id=95plu1Mo20](https://openreview.net/forum?id=95plu1Mo20)
+
+---
+
+## License
+
+MIT

fasticrl-1.0.0/README.md

@@ -0,0 +1,187 @@
+# FastICRL
+
+<p align="center">
+  <a href="https://pypi.org/project/fasticrl/"><img src="https://img.shields.io/pypi/v/fasticrl.svg" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/fasticrl/"><img src="https://img.shields.io/pypi/pyversions/fasticrl.svg" alt="Python"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
+  <img src="https://img.shields.io/badge/no%20fine--tuning-required-brightgreen" alt="No fine-tuning required">
+  <img src="https://img.shields.io/badge/no%20GPU-required-blue" alt="No GPU required">
+  <a href="https://github.com/agno-agi/agno"><img src="https://img.shields.io/badge/powered%20by-agno-8A2BE2" alt="Powered by agno"></a>
+</p>
+
+**In-Context Reinforcement Learning for LLMs — no fine-tuning, no gradient updates, no GPU.**
+
+FastICRL implements the ICRL paradigm from [*Reward Is Enough: LLMs Are In-Context Reinforcement Learners*](https://arxiv.org/abs/2506.06303) (Song et al., 2025). A learner LLM improves its outputs purely by reading its own history of attempts and rewards inside the context window — guided by a meta-cognitive strategist. No training, no infrastructure, just inference.
+
+---
+
+## How it works
+
+Three LLM agents collaborate in a feedback loop:
+
+```text
+┌──────────────────────────────────────────────────┐
+│                   ICRLLearner                    │
+│                                                  │
+│  Task ──► Learner ──► Output ──► Reward Agent    │
+│             ▲                         │          │
+│             │        Attempt          │          │
+│             │  (task, output, score)  │          │
+│             └─────────────────────────┘          │
+│                          │                       │
+│                  (every N episodes)              │
+│                          ▼                       │
+│                      Strategist                  │
+│                (refines the strategy)            │
+└──────────────────────────────────────────────────┘
+```
+
+| Agent | Role |
+| --- | --- |
+| **Learner** | Generates task outputs; balances exploration vs. exploitation based on reward history |
+| **Reward** | Scores each output on a 0–10 scale (acts as the reward function) |
+| **Strategist** | Analyzes past attempts to synthesize actionable strategies for future episodes |
+
+Each agent can be backed by a different model — e.g. a cheap model for reward, a powerful one for the learner.
+
+---
+
+## Installation
+
+```bash
+pip install fasticrl
+```
+
+Or with [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv add fasticrl
+```
+
+Model provider extras (install whichever you use):
+
+```bash
+pip install "fasticrl[openai]"   # OpenAI
+pip install "fasticrl[ollama]"   # Ollama (local models)
+```
+
+Requires Python ≥ 3.13.
+
+---
+
+## Quick start
+
+```python
+from fasticrl import ICRLLearner
+from agno.models.openai import OpenAIChat
+
+model = OpenAIChat(id="gpt-4o-mini")
+
+learner = ICRLLearner(
+    learner_model=model,
+    reward_model=model,
+    strategy_model=model,
+    task_description="Write a concise, compelling product description for an e-commerce listing.",
+    tasks=[
+        "Wireless noise-cancelling headphones",
+        "Ergonomic standing desk",
+        "Portable espresso maker",
+    ],
+)
+
+# Run 3 episodes, update strategy every 2 steps, show progress bar
+learner.auto_learn(episodes=3, batch_size=2, cli_mode=True, strategy_update_interval=2)
+
+# Inspect what the agent learned
+print(learner.strategy)
+```
+
+---
+
+## API
+
+### `ICRLLearner`
+
+```python
+ICRLLearner(
+    learner_model,        # agno Model for the learner agent
+    reward_model,         # agno Model for the reward agent
+    strategy_model,       # agno Model for the strategist agent
+    task_description,     # describes the overall task domain (required)
+    tasks,                # list of concrete task instances to cycle through
+    buffer,               # optional: pre-loaded list of Attempt objects
+    strategy,             # optional: pre-loaded strategy string
+)
+```
+
+#### Key methods
+
+| Method | Description |
+| --- | --- |
+| `auto_learn(episodes, batch_size, cli_mode, strategy_update_interval)` | Run N episodes. `batch_size > 1` parallelizes tasks with a thread pool. `cli_mode=True` shows a progress bar. `strategy_update_interval=K` refreshes the strategy every K episodes. |
+| `generate_action(task)` | Run the learner on a single task and return its output |
+| `generate_reward(task, action)` | Score a learner output with the reward agent |
+| `generate_attempt_by_present_task()` | Single step: generate + score the current task |
+| `update_strategy()` | Ask the strategist to refine the strategy from the current buffer |
+| `to_yaml(path)` | Persist the full agent state (buffer + strategy) to a YAML file |
+| `ICRLLearner.from_yaml(path, ...)` | Resume from a saved state |
+
+---
+
+## Saving and resuming
+
+```python
+# Save
+learner.to_yaml("my_agent.yaml")
+
+# Resume later
+learner = ICRLLearner.from_yaml(
+    "my_agent.yaml",
+    learner_model=model,
+    reward_model=model,
+    strategy_model=model,
+)
+learner.auto_learn(episodes=5)
+```
+
+---
+
+## Using Ollama (local models)
+
+```python
+from agno.models.ollama import Ollama
+
+learner = ICRLLearner(
+    learner_model=Ollama(id="llama3.2"),
+    reward_model=Ollama(id="llama3.2"),
+    strategy_model=Ollama(id="llama3.2"),
+    task_description="...",
+    tasks=[...],
+)
+```
+
+Any [agno](https://github.com/agno-agi/agno)-compatible model works.
+
+---
+
+## Citation
+
+This project is based on and inspired by the following papers:
+
+*Reward Is Enough: LLMs Are In-Context Reinforcement Learners*  
+Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Shangtong Zhang, Yanjun Qi  
+arXiv:2506.06303 — [https://arxiv.org/abs/2506.06303](https://arxiv.org/abs/2506.06303)
+
+*Large Language Models as Optimizers*  
+Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen  
+arXiv:2309.03409 — [https://arxiv.org/abs/2309.03409](https://arxiv.org/abs/2309.03409)
+
+*Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs*  
+Yifan Zhou, Sachin Grover, Mohamed El Mistiri, Kamalesh Kalirathinam, Pratyush Kerhalkar, Swaroop Mishra, Neelesh Kumar, Sanket Gaurav, Oya Aran, Heni Ben Amor  
+NeurIPS 2025 — [https://openreview.net/forum?id=95plu1Mo20](https://openreview.net/forum?id=95plu1Mo20)
+
+---
+
+## License
+
+MIT

fasticrl-1.0.0/pyproject.toml

@@ -0,0 +1,49 @@
+[project]
+name = "fasticrl"
+version = "1.0.0"
+description = "In-Context Reinforcement Learning framework for LLMs — no fine-tuning required."
+keywords = [
+    "reinforcement-learning",
+    "in-context-learning",
+    "icrl",
+    "llm",
+    "ai",
+    "agents",
+    "prompt-engineering",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.13",
+]
+readme = "README.md"
+authors = [
+    { name = "Maximilian König", email = "maximilian.koenig@mein.gmx" }
+]
+requires-python = ">=3.13"
+dependencies = [
+    "agno>=2.6.5",
+    "pyyaml>=6.0.3",
+    "tqdm>=4.67.3",
+]
+
+[project.optional-dependencies]
+openai = ["openai>=2.32.0"]
+ollama = ["ollama>=0.6.2"]
+
+[project.urls]
+Homepage = "https://github.com/makoeta/FastICRL"
+Source = "https://github.com/makoeta/FastICRL"
+Issues = "https://github.com/makoeta/FastICRL/issues"
+
+[dependency-groups]
+dev = ["pytest>=8.0.0"]
+
+[build-system]
+requires = ["uv_build>=0.8.15,<0.9.0"]
+build-backend = "uv_build"

fasticrl-1.0.0/src/fasticrl/__init__.py

@@ -0,0 +1,4 @@
+from fasticrl.icrl_learner import ICRLLearner
+from fasticrl.models.attempt import Attempt
+
+__all__ = ["ICRLLearner", "Attempt"]

fasticrl-1.0.0/src/fasticrl/icrl_learner.py

@@ -0,0 +1,166 @@
+from fasticrl.strategist.models.strategy_output import StrategyOutput
+
+import math
+import tqdm
+import yaml
+from agno.models.base import Model
+from concurrent.futures import ThreadPoolExecutor
+from functools import partial
+
+import fasticrl.models.sentinel as sentinel
+from fasticrl.learner.core import LearnerAgent
+from fasticrl.learner.models.learn_output import LearnerOutput
+from fasticrl.models.agent_save_state import AgentSaveState
+from fasticrl.models.attempt import Attempt
+from fasticrl.reward.core import RewardAgent
+from fasticrl.reward.models.reward_output import RewardOutput
+from fasticrl.strategist.core import StrategistAgent
+
+
+class ICRLLearner:
+    def __init__(
+        self,
+        learner_model: Model,
+        reward_model: Model,
+        strategy_model: Model,
+        task_description: str = sentinel._sentinel,
+        tasks: list[str] = None,
+        buffer: list[Attempt] = None,
+        strategy: str = "",
+    ):
+        if task_description is sentinel._sentinel:
+            raise ValueError("task_description must be provided")
+        if tasks is not None and len(tasks) == 0:
+            raise ValueError("tasks list must not be empty")
+
+        self.buffer = buffer if buffer is not None else []
+        self.task_description = task_description
+        self.tasks = tasks if tasks is not None else []
+        self.strategy = strategy
+
+        self.learner_agent = LearnerAgent(model=learner_model)
+        self.reward_agent = RewardAgent(model=reward_model)
+        self.strategist_agent = StrategistAgent(model=strategy_model)
+
+        self.episode = len(self.buffer)
+
+    def auto_learn(
+        self,
+        episodes: int = None,
+        batch_size: int = 1,
+        cli_mode: bool = False,
+        strategy_update_interval: int = None,
+    ):
+        if not self.tasks:
+            raise ValueError("tasks list is empty — nothing to learn from")
+
+        if episodes is None:
+            episodes = math.ceil(len(self.tasks) / batch_size)
+
+        iterator = tqdm.tqdm(range(episodes)) if cli_mode else range(episodes)
+
+        for i in iterator:
+            batch_tasks = [
+                str(self.tasks[(self.episode + j) % len(self.tasks)])
+                for j in range(batch_size)
+            ]
+
+            buffer_snapshot = self.__attempts_as_xml()
+
+            if batch_size == 1:
+                attempts = [self._generate_attempt_for_task(batch_tasks[0], buffer_snapshot)]
+            else:
+                with ThreadPoolExecutor(max_workers=batch_size) as pool:
+                    attempts = list(pool.map(
+                        partial(self._generate_attempt_for_task, buffer_as_xml=buffer_snapshot),
+                        batch_tasks,
+                    ))
+
+            self.buffer.extend(attempts)
+            self.episode += batch_size
+
+            if strategy_update_interval is not None and strategy_update_interval > 0 and (i + 1) % strategy_update_interval == 0:
+                self.update_strategy()
+
+    def __attempts_as_xml(self) -> str:
+        attempts = ""
+        for attempt in self.buffer:
+            attempts += (
+                "<attempt>\n"
+                + f"\tTask: {attempt.task}\n\tOutput: {attempt.output}\n\tReward: {str(attempt.reward)}"
+                + "\n</attempt>\n"
+            )
+        return attempts.strip()
+
+    def generate_action(self, task: str, buffer_as_xml: str = None) -> LearnerOutput:
+        return self.learner_agent.generate_learning_output(
+            task=task,
+            task_description=self.task_description,
+            strategy=self.strategy,
+            buffer_as_xml=buffer_as_xml if buffer_as_xml is not None else self.__attempts_as_xml(),
+        )
+
+    def generate_reward(self, task: str, action: LearnerOutput) -> RewardOutput:
+        return self.reward_agent.generate_reward_output(
+            task=task, output=action.learning_output
+        )
+
+    def _generate_attempt_for_task(self, task: str, buffer_as_xml: str = None) -> Attempt:
+        learner_output: LearnerOutput = self.generate_action(task, buffer_as_xml=buffer_as_xml)
+        reward_output: RewardOutput = self.generate_reward(task, learner_output)
+        return Attempt(
+            task=task,
+            reward=reward_output.reward,
+            output=learner_output.learning_output,
+        )
+
+    def generate_attempt_by_present_task(self) -> Attempt:
+        return self._generate_attempt_for_task(self._present_learning_task)
+
+    def update_strategy(self):
+        strategist_out: StrategyOutput = self.strategist_agent.generate_new_strategy(
+            task_description=self.task_description,
+            existing_strategy=self.strategy,
+            buffer_as_xml=self.__attempts_as_xml(),
+        )
+        self.strategy = strategist_out.strategy
+
+    @property
+    def _present_learning_task(self) -> str:
+        return str(self.tasks[self.episode % len(self.tasks)])
+
+    @property
+    def agent_save_state(self) -> AgentSaveState:
+        return AgentSaveState(
+            task_description=self.task_description,
+            strategy=self.strategy,
+            buffer=[dict(b.model_dump(mode="json")) for b in self.buffer],
+        )
+
+    def to_yaml(self, path: str):
+        if not path.endswith(".yaml"):
+            path += ".yaml"
+
+        with open(path, "w") as outfile:
+            yaml.dump(
+                self.agent_save_state.model_dump(mode="json"),
+                outfile,
+                default_flow_style=False,
+            )
+
+    @classmethod
+    def from_yaml(
+        cls, path: str, learner_model: Model, reward_model: Model, strategy_model: Model
+    ):
+        with open(path, "r") as f:
+            save_state = yaml.load(f, Loader=yaml.SafeLoader)
+            agent_state: AgentSaveState = AgentSaveState.model_validate(save_state)
+
+            return ICRLLearner(
+                task_description=agent_state.task_description,
+                buffer=agent_state.buffer,
+                strategy=agent_state.strategy,
+                learner_model=learner_model,
+                reward_model=reward_model,
+                strategy_model=strategy_model,
+            )

fasticrl-1.0.0/src/fasticrl/learner/core.py

@@ -0,0 +1,31 @@
+from fasticrl.learner.prompts import ICRL_LEARNING_SYSTEM_PROMPT
+from agno.agent import Agent
+from agno.models.base import Model
+from agno.run.agent import RunOutput
+
+from fasticrl.learner.models.learn_output import LearnerOutput
+from fasticrl.learner.prompts import ICRL_LEARNING_INPUT_PROMPT
+
+
+class LearnerAgent(Agent):
+
+    def __init__(self, model: Model, **kwargs):
+        super().__init__(**kwargs)
+
+        self.model = model
+        self.output_schema = LearnerOutput
+        self.system_message = ICRL_LEARNING_SYSTEM_PROMPT
+
+    def generate_learning_output(
+        self, buffer_as_xml: str, task: str, task_description: str, strategy: str
+    ) -> LearnerOutput:
+        run_out: RunOutput = super().run(
+            ICRL_LEARNING_INPUT_PROMPT.format(
+                attempts=buffer_as_xml,
+                task=task,
+                task_description=task_description,
+                strategy=strategy,
+            )
+        )
+
+        return run_out.content

fasticrl-1.0.0/src/fasticrl/learner/models/learn_output.py

@@ -0,0 +1,17 @@
+from typing import Annotated, Literal
+from pydantic import BaseModel, Field
+
+
+class LearnerOutput(BaseModel):
+    decision: Annotated[
+        Literal["EXPLOIT", "EXPLORE"],
+        Field(description="The explore/exploit decision for this step."),
+    ] # Not used in the code again - just to let the model think about the strategies
+    reflection: Annotated[
+        str,
+        Field(description="Brief justification for the decision."),
+    ]
+    learning_output: Annotated[
+        str,
+        Field(description="The actual response to the task. No meta-commentary — only the answer."),
+    ]

fasticrl-1.0.0/src/fasticrl/learner/prompts.py

@@ -0,0 +1,47 @@
+ICRL_LEARNING_SYSTEM_PROMPT = """
+You are an In-Context Reinforcement Learning (ICRL) agent. Your objective is to maximize cumulative reward over time by learning from feedback and dynamically balancing exploration and exploitation.
+
+### Exploration vs. Exploitation Decision
+
+Use this decision framework:
+
+| Situation | Decision |
+|---|---|
+| Recent rewards are high and stable | EXPLOIT |
+| Recent rewards are declining or stagnant | EXPLORE |
+| Few attempts exist (< 3) | EXPLORE |
+| High reward variance across attempts | EXPLORE |
+| Clear winning strategy identified | EXPLOIT |
+
+**Exploit:** Apply the best-known strategy with minor refinements.  
+**Explore:** Deliberately try a different approach — vary structure, reasoning style, or output format.
+
+### Input Structure (XML):
+- <attempts>: Historical (input, output, reward) triples. Your learning signal.
+- <taskdescription>: The specific sub-task to solve this turn.
+- <strategy>: Accumulated strategy derived from past attempts. Empty on first run.
+
+### Core Directives:
+1. Analyze <attempts> to identify what drove high vs. low rewards.
+2. Set `decision` to EXPLOIT or EXPLORE based on the framework above.
+3. Set `reflection` to a brief justification for that decision.
+4. Set `learning_output` to your answer to the task — no meta-commentary, no decision labels, just the answer.
+""".strip()
+
+ICRL_LEARNING_INPUT_PROMPT = """
+<attempts>
+    {attempts}
+</attempts>
+
+<task>
+    {task}
+</task>
+
+<taskdescription>
+    {task_description}
+</taskdescription>
+
+<strategy>
+    {strategy}
+</strategy>
+""".strip()

fasticrl-1.0.0/src/fasticrl/models/agent_save_state.py

@@ -0,0 +1,11 @@
+from typing import Annotated
+
+from pydantic import BaseModel, Field
+
+from fasticrl.models.attempt import Attempt
+
+
+class AgentSaveState(BaseModel):
+    task_description: Annotated[str, Field()]
+    strategy: Annotated[str, Field(default="")]
+    buffer: Annotated[list[Attempt], Field(default_factory=list)]

fasticrl-1.0.0/src/fasticrl/models/attempt.py

@@ -0,0 +1,8 @@
+from pydantic import Field
+from pydantic import BaseModel
+
+
+class Attempt(BaseModel):
+    task: str = Field()
+    output: str = Field()
+    reward: int = Field()

fasticrl-1.0.0/src/fasticrl/models/sentinel.py

@@ -0,0 +1 @@
+_sentinel = object()

fasticrl-1.0.0/src/fasticrl/reward/core.py

@@ -0,0 +1,24 @@
+from agno.agent import Agent
+from agno.models.base import Model
+from agno.run.agent import RunOutput
+
+from fasticrl.reward.models.reward_output import RewardOutput
+from fasticrl.reward.prompts import ICRL_REWARD_INPUT_PROMPT, ICRL_REWARD_SYSTEM_PROMPT
+
+
+class RewardAgent(Agent):
+
+    def __init__(self, model: Model, **kwargs):
+        super().__init__(**kwargs)
+
+        self.model = model
+        self.output_schema = RewardOutput
+        self.system_message = ICRL_REWARD_SYSTEM_PROMPT
+
+    def generate_reward_output(self, task: str, output: str) -> RewardOutput:
+
+        run_out: RunOutput = super().run(
+            ICRL_REWARD_INPUT_PROMPT.format(task=task, output=output)
+        )
+
+        return run_out.content

fasticrl-1.0.0/src/fasticrl/reward/models/reward_output.py

@@ -0,0 +1,14 @@
+from pydantic import Field
+from typing import Annotated
+from pydantic import BaseModel
+
+
+class RewardOutput(BaseModel):
+    reward: Annotated[
+        int,
+        Field(
+            description="Reward to the given output. Must be in range from 1 - 10.",
+            ge=1,
+            le=10,
+        ),
+    ]

fasticrl-1.0.0/src/fasticrl/reward/prompts.py

@@ -0,0 +1,28 @@
+ICRL_REWARD_SYSTEM_PROMPT = """
+You are an expert evaluator acting as the reward function for an In-Context Reinforcement Learning (ICRL) system. Your objective is to critically and objectively score the quality of an AI-generated output based strictly on the provided task.
+
+# Evaluation Criteria
+When evaluating, consider the following dimensions:
+- Accuracy & Relevance: Does the output correctly address the core task?
+- Constraint Satisfaction: Were all specific instructions and formats followed?
+- Quality & Clarity: Is the output coherent, well-structured, and logically sound?
+
+# Scoring Rubric (0 - 10)
+Use the entire 0-10 scale. Avoid score inflation; do not default to extreme scores unless completely warranted.
+- 0-2 (Fail): Completely incorrect, irrelevant, or fails to address the core task.
+- 3-4 (Poor): Partially addresses the task but contains major omissions, errors, or hallucinations.
+- 5-6 (Average): Acceptable. Meets the basic requirements but lacks polish, nuance, or is merely mediocre. Use this baseline frequently.
+- 7-8 (Good): Accurate, well-crafted, and follows instructions closely with only minor flaws.
+- 9-10 (Exceptional): Flawless execution. Perfectly hits all constraints with high-quality reasoning and formatting.
+""".strip()
+
+ICRL_REWARD_INPUT_PROMPT = """
+# Input
+<task>
+    {task}
+</task>
+
+<output>
+    {output}
+</output>
+""".strip()

fasticrl-1.0.0/src/fasticrl/strategist/core.py

@@ -0,0 +1,32 @@
+from agno.agent import Agent
+from agno.models.base import Model
+from agno.run.agent import RunOutput
+
+from fasticrl.strategist.models.strategy_output import StrategyOutput
+from fasticrl.strategist.prompts import (
+    ICRL_STRATEGY_INPUT_PROMPT,
+    ICRL_STRATEGY_SYSTEM_PROMPT,
+)
+
+
+class StrategistAgent(Agent):
+
+    def __init__(self, model: Model, **kwargs):
+        super().__init__(**kwargs)
+
+        self.model = model
+        self.system_message = ICRL_STRATEGY_SYSTEM_PROMPT
+        self.output_schema = StrategyOutput
+
+    def generate_new_strategy(
+        self, task_description: str, existing_strategy: str, buffer_as_xml: str
+    ) -> StrategyOutput:
+        run_out: RunOutput = super().run(
+            ICRL_STRATEGY_INPUT_PROMPT.format(
+                task_description=task_description,
+                strategy=existing_strategy,
+                attempts=buffer_as_xml,
+            )
+        )
+
+        return run_out.content

fasticrl-1.0.0/src/fasticrl/strategist/models/strategy_output.py

@@ -0,0 +1,10 @@
+from pydantic import Field
+from typing import Annotated
+from pydantic import BaseModel
+
+
+class StrategyOutput(BaseModel):
+
+    strategy: Annotated[
+        str, Field(description="The strategy extracted from prior action and rewards.")
+    ]

fasticrl-1.0.0/src/fasticrl/strategist/prompts.py

@@ -0,0 +1,32 @@
+ICRL_STRATEGY_SYSTEM_PROMPT = """
+You are an expert Meta-Cognitive Analyst and Prompt Engineer specializing in In-Context Reinforcement Learning. Your sole objective is to analyze execution history and synthesize actionable strategies that maximize future rewards.
+
+### Role
+You do NOT solve tasks. You analyze attempts, identify reward-driving patterns, and produce refined strategies for the agent that does.
+
+### Analysis Framework
+When evaluating attempts, apply these lenses in order:
+
+1. **Pattern Recognition:** What specific behaviors in high-reward (8-10) outputs differ from low-reward (0-4) outputs? Look for structure, tone, detail level, constraint adherence.
+2. **Failure Mode Classification:** Categorize low-reward causes:
+   - Formatting violations
+   - Insufficient detail or hallucination
+   - Constraint/instruction ignored
+   - Wrong reasoning approach
+3. **Incremental Refinement:** Never discard an existing strategy. Extend it with new evidence — add concrete Do's and Don'ts.
+4. **Actionability:** Every strategy element must be a directive a model can directly follow, not a vague observation.
+""".strip()
+
+ICRL_STRATEGY_INPUT_PROMPT = """
+<task_description>
+    {task_description}
+</task_description>
+
+<existing_strategy>
+    {strategy}
+</existing_strategy>
+
+<attempts>
+    {attempts}
+</attempts>
+""".strip()