@r1z4x/tezgah 1.0.0

package/skills/rl-env/generate-ors-env/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: generate-ors-env
+description: Builds an Open Reward Standard (ORS) variant of an RL environment using the official `openreward` Python package. Use whenever someone asks to scaffold an ORS env, port to OpenReward, add per-tool-call rewards, deploy to OpenReward.ai, or wrap an existing env in the ORS protocol. ORS is the right framework when the user wants HTTP+REST+SSE, rewards arriving inline with each tool call (not post-episode), task-spec-driven sessions, splits (train/val/test), or deployment to OpenReward.ai or HF Spaces. Output is a runnable `<env_dir>/ors/` folder with `server.py`, `tasks.py`, `pyproject.toml`, `Dockerfile.spaces`, and `rollout.py`. Use for prompts like "wrap my env in ORS", "make an OpenReward env for X", or "add per-call reward to my env".
+---
+
+# generate-ors-env
+
+Build the ORS variant of an env using the official **`openreward >= 0.1.33`** package (the `ors-sdk` name is a common mistake — it does not exist on PyPI).
+
+## Concept
+
+ORS is the Open Reward Standard ([openrewardstandard.io](https://openrewardstandard.io)) — an HTTP REST + Server-Sent Events protocol for agent envs. Reward arrives **inline** with every `ToolOutput`, which is the framework's defining feature compared to OpenEnv (external/post-hoc reward) and NeMo Gym (post-episode `/verify`).
+
+When the user has a shared domain module (`<domain>.py`) and wants an ORS variant, **never duplicate domain logic into the framework folder** — wrap it.
+
+## Archetypes
+
+| Archetype | Hallmarks |
+|---|---|
+| **Pure-Python game** | Single `@tool`, `tasks.py` with N task dicts forming the `train` split, terminal reward via `finished=True`. |
+| **Stateful sandbox** | `setup()` allocates resources from `task_spec`; `teardown()` frees them; per-tool reward stubs. |
+| **Vision / computer-use** | `ImageBlock(data=<base64>, mimeType="image/png")` returns; `terminate(status)` tool emits the terminal reward. |
+
+## Imports — exactly these
+
+Server side:
+```python
+from openreward.environments import (
+    Environment, Server, tool, ToolOutput, TextBlock, Split, ImageBlock,
+)
+```
+
+Client side (rollouts):
+```python
+from openreward import EnvironmentsAPI
+api = EnvironmentsAPI(base_url=URL, api_key="")
+env = api.get(ENV_NAME)
+```
+
+> **Don't use `OpenReward(api_key=..., base_url=...)`** even though it's the high-level client. It prepends `matrix.` / `api.` / `construct.` subdomains to the base URL — that breaks HF Space URLs. `EnvironmentsAPI` talks to `base_url` verbatim.
+
+## Architecture
+
+```
+<env_dir>/ors/
+├── pyproject.toml         # openreward>=0.1.33 + e2b-* (if needed) + pydantic
+├── __init__.py
+├── Dockerfile             # local dev image
+├── Dockerfile.spaces      # HF Space (port 7860, single-stage pip install)
+├── README.spaces.md       # HF Space frontmatter
+├── server.py              # the Environment subclass + main()
+├── tasks.py               # list of dicts (task_spec for each task)
+├── rollout.py             # or rollout_openai.py + rollout_qwen.py
+└── README.md              # one-page dev README
+```
+
+## Implementation order
+
+### 1. Tasks file — `tasks.py`
+
+A list of plain dicts. Each dict becomes a `task_spec` per session. ORS auto-wraps these into `Task` objects on `list_tasks()`.
+
+```python
+TASKS = [
+    {"answer": "apple", "task": "Guess the 5-letter word."},
+    # ...
+]
+```
+
+### 2. The Environment subclass — `server.py`
+
+```python
+from pydantic import BaseModel
+from openreward.environments import Environment, Server, tool, ToolOutput, TextBlock, Split
+
+class GuessInput(BaseModel):
+    word: str
+
+class WordleORS(Environment):
+    def __init__(self, task_spec=None, secrets=None, **kw):
+        super().__init__(task_spec=task_spec or {}, secrets=secrets or {})
+        self._game = None
+
+    def setup(self):                      # called on first tool invocation
+        self._game = WordleGame(self.task_spec.get("answer"))
+
+    def teardown(self):                   # called on session delete
+        self._game = None
+
+    @classmethod
+    def list_splits(cls): return [Split(name="train", type="train")]
+
+    @classmethod
+    def list_tasks(cls, split): return TASKS
+
+    def get_prompt(self):
+        return [TextBlock(text="Play Wordle. Guess the 5-letter word.")]
+
+    @tool
+    def guess(self, params: GuessInput) -> ToolOutput:
+        feedback = self._game.guess(params.word)
+        return ToolOutput(
+            blocks=[TextBlock(text=feedback)],
+            reward=self._game.reward,
+            finished=self._game.done,
+        )
+```
+
+Key contracts:
+- **Tools take a `params: PydanticModel`** as the second arg. ORS uses the model's JSON schema as the tool's `input_schema`.
+- **Empty inputs** still need a Pydantic model (`class _Empty(BaseModel): pass`). Don't omit the param.
+- **`ToolOutput.blocks`** is `[TextBlock | ImageBlock]`. For images: `ImageBlock(data=<base64>, mimeType="image/png")`. **Vision models actually see this.**
+- **`reward`** is `float | None`. `None` means "no reward this step"; `0.0` means "stepped, scored zero". For pure terminal reward, return `None` everywhere except in the last `ToolOutput`.
+- **`finished=True`** ends the session. Pair with `reward=1.0` (or whatever) to give the rollout a clean stop.
+- **`task_spec`** is a `dict` you read from `self.task_spec` — no schema validation. If you want validation, do it in `setup()`.
+
+### 3. Server entry point — `server.py` main
+
+```python
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--port", type=int, default=8080)
+    parser.add_argument("--host", type=str, default="0.0.0.0")
+    args = parser.parse_args()
+    Server([WordleORS]).run(host=args.host, port=args.port)
+```
+
+The endpoint name is auto-derived from the class name lowercased — `WordleORS` → `wordleors`. Tell the user this so they know what `ENV_NAME` to pass.
+
+### 4. Rollout
+
+Always discover tools and tasks from the env. Don't hardcode names:
+
+```python
+api = EnvironmentsAPI(base_url=ENV_URL, api_key="")
+env = api.get("wordleors")
+tasks = env.list_tasks("train")
+tools = env.list_tools(format="openai")     # built-in OpenAI tool-schema converter
+with env.session(task=tasks[0]) as session:
+    prompt = session.get_prompt()
+    result = session.call_tool("guess", {"word": "crane"})
+    # result.blocks, result.reward, result.finished
+```
+
+For **vision envs**, the screenshot tool returns an `ImageBlock` — read it as `b.data` (already base64). Pass that into the model's image content.
+
+### 5. Dockerfiles
+
+`Dockerfile.spaces` is the HF Space deploy image. Keep it minimal:
+
+```dockerfile
+FROM python:3.11-slim
+RUN useradd -m -u 1000 user
+RUN pip install --no-cache-dir openreward pydantic <other-deps>
+USER user
+ENV HOME=/home/user PATH=/home/user/.local/bin:$PATH
+WORKDIR $HOME/app
+COPY --chown=user . $HOME/app
+EXPOSE 7860
+CMD ["python", "server.py", "--host", "0.0.0.0", "--port", "7860"]
+```
+
+`README.spaces.md`:
+```yaml
+---
+title: My Env ORS
+emoji: 🎯
+colorFrom: pink
+colorTo: indigo
+sdk: docker
+app_port: 7860
+tags: [ors, openreward]
+---
+```
+
+## Pushing to HF Spaces
+
+Create a Space named `<owner>/<env_name>-ors`. Set `E2B_API_KEY` (and any other secrets) as **Space secrets**, not environment variables — they survive rebuilds. The local `.env` file should not be uploaded.
+
+```python
+api.add_space_secret(repo_id="<owner>/<env>-ors", key="E2B_API_KEY", value="...")
+api.upload_file(path_or_fileobj="Dockerfile.spaces", path_in_repo="Dockerfile", repo_id=...)
+api.upload_file(path_or_fileobj="README.spaces.md", path_in_repo="README.md", repo_id=...)
+# upload server.py, tasks.py, __init__.py, pyproject.toml
+```
+
+## Validation gates
+
+1. **Local server** — `uv run python server.py --port 8772` then `curl http://localhost:8772/list_environments` returns `["<envname>"]`.
+2. **Tool discovery** — `curl http://localhost:8772/<envname>/tools | jq '.tools | length'` matches the number of `@tool` methods.
+3. **End-to-end** — `MAX_TURNS=3 uv run python rollout.py` drives the model through at least one tool call without errors.
+
+## Gotchas (from real-world ORS work)
+
+- **`from openreward.environments.types import Task`** — wrong; `Task` is in `openreward.api.environments.types` and you usually don't import it. `list_tasks` can return plain dicts; ORS wraps them.
+- **`OpenReward(base_url=URL)` rewrites the URL** — prepends `matrix.` / `api.` / `construct.` subdomains. For HF Spaces, use `EnvironmentsAPI(base_url=URL, api_key="")` directly.
+- **`e2b-desktop` without `e2b`** — `e2b-desktop` imports from `e2b`, but doesn't pin it. Add both to `dependencies`.
+- **Endpoint name is the lowercased class name** — `MyEnvORS` becomes `myenvors`. Tell users this explicitly so their `ENV_NAME` env var is right.
+
+## Reference
+
+- `references/architecture.md` — protocol shape + Server / Environment / Session lifecycle
+
+## Official documentation
+
+- [openrewardstandard.io](https://openrewardstandard.io) — protocol specification
+- [docs.openreward.ai](https://docs.openreward.ai) — Python SDK + platform docs
+- [openreward on PyPI](https://pypi.org/project/openreward/) — current package (latest 0.1.81+)
+- [Talc-AI/OpenReward on GitHub](https://github.com/Talc-AI/OpenReward) — source

package/skills/rl-env/generate-ors-env/references/architecture.md

@@ -0,0 +1,113 @@
+# ORS architecture (deep)
+
+## Wire protocol (REST + SSE)
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET`  | `/list_environments` | List env names served by this server |
+| `GET`  | `/<env>/tools` | List tools (returns `{"tools": [{"name", "description", "input_schema"}, ...]}`) |
+| `GET`  | `/<env>/splits` | List splits |
+| `POST` | `/<env>/tasks` | Body `{"split": "train"}` → list of `Task` |
+| `GET`  | `/<env>/prompt` | Returns the prompt blocks for the current task |
+| `POST` | `/<env>/sessions` | Body `{"task_spec": {...}, "secrets": {...}}` → SSE stream returns the `session_id` |
+| `POST` | `/<env>/sessions/<sid>/tool` | Body `{"name": "x", "input": {...}}` → SSE stream of the `ToolOutput` |
+| `DELETE` | `/<env>/sessions/<sid>` | Tear down |
+
+Endpoint name = `<EnvironmentClassName>.lower()`. So `class WordleORS(Environment)` → `/wordleors/...`.
+
+The `EnvironmentsAPI` Python client wraps this via `aiohttp`. `OpenReward` wraps `EnvironmentsAPI` and rewrites `base_url` to `matrix.<host>` — that's why HF Space targets need `EnvironmentsAPI` direct.
+
+## `Environment` lifecycle
+
+1. **`__init__(task_spec, secrets, **kw)`** — called once per session. Store `task_spec` (already on `self.task_spec` after `super().__init__`); validate or stub.
+2. **`setup()`** — called on first tool invocation. Allocate sandbox, init game state.
+3. **`teardown()`** — called on `DELETE /sessions/<sid>`. Kill the sandbox, free resources.
+4. **`get_prompt() -> [TextBlock | ImageBlock]`** — called when the client asks for the prompt. Reads from `self.task_spec`.
+5. **`@classmethod list_splits()`** and **`@classmethod list_tasks(split)`** — class-level (no `self`). Splits are static metadata. Tasks can be plain dicts; ORS wraps them.
+6. **Tool methods** — decorated `@tool`, signature `(self, params: BaseModel) -> ToolOutput`.
+
+## ToolOutput shape
+
+```python
+ToolOutput(
+    blocks=[TextBlock(text="..."), ImageBlock(data=<b64>, mimeType="image/png")],
+    metadata={"any": "json"},
+    reward=0.5,                  # float | None
+    finished=False,              # bool — True ends the session
+)
+```
+
+Per-step rewards add up across the trajectory; `finished=True` is the only way for the env to signal terminal state.
+
+## Splits & tasks
+
+`Split.type` is `"train" | "validation" | "test"`. The split name and type can differ. List N tasks per split, each a dict that becomes the per-session `task_spec`.
+
+A common pattern:
+```python
+TASKS = [{"answer": w, "task": "Guess the word"} for w in WORDS[:50]]
+
+@classmethod
+def list_tasks(cls, split): return TASKS
+```
+
+ORS wraps each dict into a `Task(server_name=cls.__name__, environment_name=..., task_spec=dict)` automatically.
+
+## Secrets
+
+`Environment.__init__` takes `secrets`. The client passes `secrets` per-session in the `POST /sessions` body. Use this for per-rollout API keys (E2B sandbox, etc.) — they don't appear in the server's environment.
+
+## Sync vs async clients
+
+Both `EnvironmentsAPI` and `AsyncEnvironmentsAPI` exist. The sync wrapper runs an async loop under the hood. For multi-rollout scenarios prefer async:
+
+```python
+async with AsyncOpenReward(api_key="").environments as api:
+    env = api.get(name, base_url=URL)
+    async with env.session(task=task) as session:
+        ...
+```
+
+## Image handling
+
+`ImageBlock.data` is **base64**, not raw bytes. The server-side `ImageBlock(data=base64.b64encode(png).decode(), mimeType="image/png")` matches the client-side `b.data` (still base64). Don't re-encode.
+
+Canonical helper:
+
+```python
+def _shot_block(sandbox) -> ImageBlock:
+    data = sandbox.screenshot()
+    return ImageBlock(data=base64.b64encode(data).decode("utf-8"), mimeType="image/png")
+```
+
+## Production deployment
+
+The wordle and desktop ORS variants both deploy to HF Spaces using a minimal `Dockerfile.spaces`:
+
+```dockerfile
+FROM python:3.11-slim
+RUN useradd -m -u 1000 user
+RUN pip install --no-cache-dir openreward pydantic <env-specific>
+USER user
+WORKDIR /home/user/app
+COPY --chown=user . .
+EXPOSE 7860
+CMD ["python", "server.py", "--host", "0.0.0.0", "--port", "7860"]
+```
+
+Plus `README.spaces.md` with the HF frontmatter. Push via:
+
+```python
+api.add_space_secret(repo_id, "E2B_API_KEY", value)
+api.upload_file(path_or_fileobj="Dockerfile.spaces", path_in_repo="Dockerfile", ...)
+api.upload_file(path_or_fileobj="README.spaces.md", path_in_repo="README.md", ...)
+```
+
+For **OpenReward.ai** deployment (the platform), the same files work — see [docs.openreward.ai](https://docs.openreward.ai/) for the GitHub-integration flow.
+
+## What can go wrong
+
+- `pip install ors-sdk` — package doesn't exist on PyPI.
+- Endpoint name mismatch — class `MyEnv` is served at `/myenv`, not `/my_env`.
+- `OpenReward(base_url="https://X.hf.space")` ends up calling `https://matrix.X.hf.space` — broken DNS. Use `EnvironmentsAPI`.
+- `setup()` not called — happens silently when an `__init__` raises before tools are registered. Check the server log.

package/skills/rl-env/generate-verifiers-env/SKILL.md

@@ -0,0 +1,188 @@
+---
+name: generate-verifiers-env
+description: Builds a Verifiers (PrimeIntellect) variant of an RL environment. Use whenever someone asks to scaffold a Verifiers env, port to Verifiers, build an in-process toolkit, set up a `vf.ToolEnv` with a Rubric, or wire up a TRL `GRPOTrainer` rollout. Verifiers is the right framework when the user wants in-process tools (no HTTP server), structured tool calling driven by plain Python functions, composable reward rubrics with multiple grader functions, fast iteration with no Docker, or the cleanest path from prototype to TRL training. Output is a runnable `<env_dir>/verifiers/` folder with `env.py` (toolkit + standalone tool functions + `create_verifiers_env`), `rollout.py`, and `pyproject.toml`. Use for prompts like "make a verifiers env for X", "wrap my game in verifiers", or "set up a vf.ToolEnv".
+---
+
+# generate-verifiers-env
+
+Build the Verifiers variant of an env. Verifiers is **in-process** — no HTTP server, no Docker, no HF Space. The trainer (or a manual rollout) imports tool functions directly from `env.py`.
+
+## Concept
+
+[PrimeIntellect Verifiers](https://github.com/PrimeIntellect-ai/verifiers) is a Python library — not a server framework. It provides `vf.ToolEnv` (multi-turn rollout), `vf.Rubric` (composable async graders), and adapters into TRL `GRPOTrainer`. The trainer or rollout owns the LLM client; the env owns the tools and the grader.
+
+When the user has a shared domain module (`<domain>.py`) and wants a Verifiers variant, wrap it as a toolkit class plus standalone tool functions. Don't duplicate domain logic.
+
+## Archetypes
+
+| Archetype | Hallmarks |
+|---|---|
+| **Pure-Python game** | One `@tool`-style function, terminal reward via rubric checking the trajectory. |
+| **Stateful sandbox in-process** | Toolkit owns the sandbox (E2B, browser); `initialize()` is lazy; `cleanup()` is mandatory in `finally`. |
+| **Vision env** | Drive the toolkit manually (skip `vf.ToolEnv` since vision content blocks aren't first-class in verifiers' rollout). Send the screenshot in the user message each turn. |
+
+## Two consumption paths (always provide both)
+
+### Path A — `DesktopToolkit`-style class (used by TRL adapter + manual rollout)
+
+```python
+class WordleToolkit:
+    def __init__(self): ...
+    def initialize(self): ...     # lazy E2B / state init
+    def cleanup(self): ...        # kill sandbox
+    def reset(self): ...          # new episode
+    def guess(self, word: str) -> str:
+        """Submit a 5-letter word guess. Returns colored feedback."""
+        ...
+```
+
+Public methods are introspected as tools by the TRL adapter. Docstrings become tool descriptions.
+
+### Path B — `vf.ToolEnv` for native verifiers `env.evaluate(client, model)`
+
+```python
+def create_verifiers_env():
+    import verifiers as vf
+    from datasets import Dataset
+    dataset = Dataset.from_list([{"question": t["task"], "answer": t["expected_output"]} for t in TASKS])
+    async def correctness(completion, answer, **kwargs) -> float:
+        # read from the completion trajectory; return 0.0–1.0
+        ...
+    rubric = vf.Rubric(funcs=[correctness])
+    return vf.ToolEnv(tools=TOOL_FUNCTIONS, max_turns=8, dataset=dataset, rubric=rubric, system_prompt="...")
+```
+
+`TOOL_FUNCTIONS` is a list of **plain Python functions** (not bound methods). They can share state via a module-level toolkit instance.
+
+## Recommended file layout
+
+The user picks the actual paths. The canonical shape:
+
+```
+<env_dir>/verifiers/
+├── pyproject.toml      # verifiers + e2b-* + datasets + python-dotenv + openai
+├── __init__.py
+├── env.py              # Toolkit class + standalone tool fns + create_verifiers_env()
+├── rollout.py          # Drives the toolkit manually with the openai client
+└── README.md
+```
+
+## Implementation order
+
+### 1. The toolkit class
+
+- `__init__` takes config (`api_key=""`, `app="firefox"`, etc.). **Don't** create the sandbox here — too eager.
+- `initialize()` is the lazy creation hook. Always call it from each tool method.
+- `cleanup()` kills the sandbox. Always call it from `finally` in the rollout.
+- `reset()` calls `cleanup()` + reinitializes. Used between episodes by the TRL adapter.
+- Each tool method:
+  - takes typed args (used for OpenAI tool-schema generation via `inspect`)
+  - has a docstring (becomes the tool description — first paragraph only)
+  - calls `self.initialize()` first, mutates state, returns a string
+
+### 2. Standalone tool functions for `vf.ToolEnv`
+
+Module-level shared toolkit, plus thin wrappers:
+
+```python
+_shared: Optional[WordleToolkit] = None
+def _kit():
+    global _shared
+    if _shared is None:
+        _shared = WordleToolkit()
+    return _shared
+
+def guess(word: str) -> str:
+    """Submit a 5-letter word guess."""
+    return _kit().guess(word)
+
+TOOL_FUNCTIONS = [guess]
+```
+
+Why both? The TRL adapter wants the toolkit class (per-rollout instance, isolated state). `vf.ToolEnv` wants free functions. Don't pick one — provide both.
+
+### 3. The rubric
+
+Rubrics are composable graders. Each grader is `async def func(completion, answer, **kwargs) -> float`. Combine multiple in a `vf.Rubric(funcs=[...])` and they're averaged (or weighted, see verifiers docs).
+
+For a single-criterion env, one grader suffices:
+
+```python
+async def correctness(completion, answer, **kwargs) -> float:
+    if not completion: return 0.0
+    last = completion[-1].get("content", "") if isinstance(completion[-1], dict) else str(completion[-1])
+    return 1.0 if answer.strip() in last.strip() else 0.0
+```
+
+For multi-criterion (e.g. computer-use envs that need both `terminate(success)` AND a state check):
+
+```python
+async def correctness(completion, answer, **kwargs) -> float:
+    seen_success = any("terminated: success" in str(m) for m in completion)
+    seen_expected = any(answer in str(m) for m in completion)
+    return 1.0 if (seen_success and seen_expected) else (0.5 if seen_success else 0.0)
+```
+
+### 4. Rollout — `rollout.py`
+
+Build OpenAI tool schemas from the function signatures + docstrings via `inspect`:
+
+```python
+def func_to_openai_tool(fn):
+    sig = inspect.signature(fn)
+    hints = get_type_hints(fn)
+    doc = (fn.__doc__ or "").strip().split("\n\n")[0]
+    properties, required = {}, []
+    for name, p in sig.parameters.items():
+        ann = hints.get(name, str)
+        origin = get_origin(ann)
+        if origin in (list, "list"):
+            inner = get_args(ann)
+            properties[name] = {"type": "array", "items": {"type": "integer" if (inner and inner[0] is int) else "string"}}
+        elif ann is int:    properties[name] = {"type": "integer"}
+        elif ann is float:  properties[name] = {"type": "number"}
+        elif ann is bool:   properties[name] = {"type": "boolean"}
+        else:               properties[name] = {"type": "string"}
+        if p.default is inspect.Parameter.empty:
+            required.append(name)
+    return {"type": "function", "function": {
+        "name": fn.__name__, "description": doc,
+        "parameters": {"type": "object", "properties": properties, "required": required},
+    }}
+```
+
+This pattern works for any toolkit. Use it as the standard adapter from Python signatures to OpenAI tool schemas.
+
+For multimodal envs, drive the toolkit manually (don't use `vf.ToolEnv` since vision-content blocks aren't first-class in verifiers' rollout). Send the latest screenshot in the user message every turn:
+
+```python
+text, b64 = kit._ctrl.screenshot()      # if you exposed _ctrl
+messages.append({"role": "user", "content": [
+    {"type": "text", "text": "Latest screenshot:"},
+    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
+]})
+```
+
+## Validation gates
+
+1. **Toolkit imports cleanly** — `uv run python -c "from env import DesktopToolkit, TOOL_FUNCTIONS"`
+2. **`vf.ToolEnv` builds** — `uv run python -c "from env import create_verifiers_env; env = create_verifiers_env(); print(env)"`
+3. **Manual rollout** — `MAX_TURNS=3 uv run python rollout.py` runs end-to-end. Hits a real backend (E2B or whatever the env uses).
+
+## Gotchas
+
+- **`ModuleNotFoundError: attrs`** — `e2b-desktop` transitively needs `attrs` but doesn't pin it. Add `attrs>=23.0` to `dependencies`.
+- **TypedDict vs dataclass for verifiers data structures** — most are TypedDicts. Access by key, not attribute. (Same trap exists in skyrl-gym; we hit it during the desktop_env port.)
+- **Tool-schema `**kwargs` is forbidden** — vLLM (used by some trainers) can't introspect `**kwargs` for JSON schema generation. Define explicit params, even if empty.
+- **Don't return huge strings** — verifiers passes the result through to the model verbatim. A 100KB log dump will blow your context. Truncate / summarize in the tool method.
+
+## Reference
+
+- `references/architecture.md` — `vf.ToolEnv` internals + Rubric composition + TRL adapter shape
+
+## Official documentation
+
+- [PrimeIntellect-ai/verifiers](https://github.com/PrimeIntellect-ai/verifiers) — source repo
+- [Prime Intellect Verifiers docs](https://docs.primeintellect.ai/verifiers/overview)
+- [verifiers/docs/environments.md](https://github.com/PrimeIntellect-ai/verifiers/blob/main/docs/environments.md) — `ToolEnv` / `StatefulToolEnv` / `MultiTurnEnv` reference
+- [verifiers on PyPI](https://pypi.org/project/verifiers/) — latest is 0.1.9+ (Jan 2026)

package/skills/rl-env/generate-verifiers-env/references/architecture.md

@@ -0,0 +1,119 @@
+# Verifiers architecture (deep)
+
+## Conceptual model
+
+Verifiers is **not** a server framework. It's a Python library that gives you:
+
+1. A `vf.ToolEnv` class that runs a multi-turn tool-calling rollout against any OpenAI-compatible client.
+2. A `vf.Rubric` class that aggregates async grader functions into a single reward.
+3. Adapters into TRL (`GRPOTrainer`) so the env can be a plain Python object passed to the trainer.
+
+The trainer or rollout owns the LLM client. The env owns the tools and the grader. There's no HTTP layer.
+
+## `vf.ToolEnv` shape
+
+```python
+env = vf.ToolEnv(
+    tools: list[Callable],          # plain Python functions, signatures = OpenAI tool schemas
+    max_turns: int,                  # hard cap per rollout
+    dataset: Dataset,                # HF Dataset with `question`, `answer` columns
+    rubric: vf.Rubric,               # grader composition
+    system_prompt: str = "",
+    parser: Optional[Parser] = None, # optional output parsing
+)
+```
+
+Tool functions are introspected via `inspect.signature` + `get_type_hints` + the docstring. The first paragraph of the docstring becomes the tool description.
+
+## Rubric
+
+```python
+async def correctness(completion, answer, **kwargs) -> float: ...
+async def efficiency(completion, answer, **kwargs) -> float: ...
+
+rubric = vf.Rubric(funcs=[correctness, efficiency])
+```
+
+`completion` is the message trajectory (list of dicts in OpenAI message format). `answer` is the ground truth from the dataset row. Each grader returns a float; the rubric averages them by default. Per-grader weights via `Rubric(funcs=[...], weights=[...])`.
+
+The rubric runs **after** the rollout completes. There's no per-step reward — that's an ORS feature, not a Verifiers one.
+
+## TRL adapter
+
+For training with `GRPOTrainer`, Verifiers exposes an adapter that wraps a toolkit class:
+
+```python
+from trl import GRPOTrainer
+trainer = GRPOTrainer(
+    model=...,
+    environment_factory=DesktopToolkit,    # the class, not an instance
+    environment_config={"app": "firefox"},
+    reward_funcs=[correctness],            # the rubric's funcs
+    ...
+)
+```
+
+The factory is invoked once per rollout. The toolkit's public methods (those without leading `_`) become the tools available to the model.
+
+## Toolkit class contract
+
+```python
+class DesktopToolkit:
+    def __init__(self, **config): ...
+    def initialize(self): ...     # lazy backend init (E2B, browser, etc.)
+    def cleanup(self): ...        # release backend
+    def reset(self): ...          # cleanup + new episode
+    def my_tool(self, x: int) -> str:
+        """One-line description (becomes the tool description).
+
+        Optional longer block (ignored).
+        """
+        ...
+```
+
+Public methods → tools. Private (leading `_`) → not exposed. Use this for shared helpers.
+
+## Two consumption paths in one file
+
+Provide **both** in `env.py`:
+
+```python
+# Path A — for the TRL adapter:
+class DesktopToolkit: ...
+
+# Path B — for native vf.ToolEnv:
+_shared = None
+def _kit(): ...
+def my_tool(x: int) -> str: return _kit().my_tool(x)
+TOOL_FUNCTIONS = [my_tool, ...]
+
+def create_verifiers_env() -> vf.ToolEnv: ...
+```
+
+Why both: the TRL adapter expects a class with per-rollout instances (state isolation between concurrent rollouts in a batch). `vf.ToolEnv` expects free functions. The shared `_kit()` lazy-loads a single toolkit when used the second way.
+
+## Tool-schema introspection rules
+
+- Type hints become JSON-schema types: `int → integer`, `float → number`, `bool → boolean`, `str → string`, `List[int] → {type: array, items: {type: integer}}`.
+- Default values mark parameters as optional.
+- The first paragraph of the docstring is the description. Subsequent text is dropped.
+- `**kwargs` is **forbidden** by some downstream trainers (vLLM-based) — JSON schema generation fails. Always use explicit params.
+
+## Grader patterns
+
+| Pattern | Use when |
+|---|---|
+| **Substring match** — `answer in completion[-1]["content"]` | Deterministic answer (math, code output). |
+| **Multi-criterion** — return 0.0 / 0.5 / 1.0 based on combinations | Tasks needing both an action AND a state check (e.g. computer-use envs). |
+| **LLM judge** — call another model in the grader | Subjective tasks (writing quality, creative). |
+| **Unit tests** — execute test code in a sandbox | Coding tasks. |
+
+Always return floats in `[0.0, 1.0]` unless you're explicitly using a different scale.
+
+## What can go wrong
+
+- `**kwargs in tool` — vLLM JSON-schema fail. Use explicit params.
+- TypedDicts everywhere — `BaseTextEnvStepOutput` from skyrl-gym, similar in verifiers internals. Access by key.
+- Returning huge strings from tools — fills the context. Truncate / summarize at the tool boundary.
+- Forgetting `cleanup()` in the rollout `finally` — leaks sandboxes; cost adds up.
+- Module-level shared state without lazy init — instantiates the backend at import time, which breaks `python -c "from env import ..."` smoke tests.