agentcore-rl-toolkit 0.1.3__tar.gz → 0.1.4__tar.gz

{agentcore_rl_toolkit-0.1.3 → agentcore_rl_toolkit-0.1.4}/.gitignore

@@ -5,6 +5,7 @@
 # Python cache
 __pycache__/
 .ruff_cache/
+.pytest_cache/
 
 # Build artifacts
 *.egg-info/
@@ -15,8 +16,23 @@ build/
 .env
 .wandb.env
 
+# Logs and experiment tracking
+wandb/
+*.log
+
 # Local slime backend config (contains ARN/bucket; commit config.yaml.example only)
 src/agentcore_rl_toolkit/backends/slime/examples/**/config.yaml
+src/agentcore_rl_toolkit/backends/experimental/slime/examples/**/config.yaml
+
+# Training/example run artifacts, wherever generated. Convention-level patterns:
+# hydra run dirs, verl checkpoint dirs (scripts use CKPTS_DIR=exp_*), wandb run
+# dirs, and generated datasets (reproducible via the example preprocess scripts).
+outputs/
+checkpoints/
+exp_*/
+wandb/
+*.parquet
+local/
 
 # Config files
 config.toml
@@ -27,5 +43,7 @@ config.toml
 !.claude/skills/
 .claude/skills/*
 !.claude/skills/check-cloudwatch-session-logs/
+# Personal per-project Claude instructions (loaded alongside CLAUDE.md, never committed)
+CLAUDE.local.md
 
 .bedrock_agentcore.yaml

agentcore_rl_toolkit-0.1.4/NOTICE

@@ -0,0 +1,29 @@
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+------------------------------------------------------------------------------
+
+This product includes software developed as part of the slime project
+(https://github.com/THUDM/slime), licensed under the Apache License, Version 2.0.
+
+The rollout gateway (src/agentcore_rl_toolkit/rollout_gateway/) adapts code from
+slime, notably:
+  - trajectory.py  — TrajectoryManager, TurnRecord, MessageNode, and the
+                     CLEAN/FORK drift classification, adapted from
+                     slime/agent/trajectory.py (made torch-free).
+  - adapters/      — OpenAIAdapter, AnthropicAdapter, and the shared BaseAdapter
+                     pipeline, adapted from slime/agent/adapters/.
+  - parsing.py     — output-parsing helpers, adapted from slime/agent/parsing.py.
+  - server.py      — background-thread serving mechanics and FilteredAccessLogger,
+                     adapted from slime/agent/aiohttp_threaded.py.
+
+------------------------------------------------------------------------------
+
+This product includes data developed as part of the TRL project
+(https://github.com/huggingface/trl), licensed under the Apache License,
+Version 2.0.
+
+The rollout gateway's response-schema registry
+(src/agentcore_rl_toolkit/rollout_gateway/response_schemas.py) vendors the
+response-schema dicts from trl/chat_template_utils.py and derives its
+chat-template hash table from trl/chat_templates/*.jinja (baseline commit
+7073af94).

{agentcore_rl_toolkit-0.1.3 → agentcore_rl_toolkit-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: agentcore-rl-toolkit
-Version: 0.1.3
+Version: 0.1.4
 Summary: Toolkit for Seamlessly Enabling RL Training with Bedrock AgentCore.
 Project-URL: Homepage, https://github.com/awslabs/agentcore-rl-toolkit
 Author-email: Amazon Web Services - Labs <opensource@amazon.com>
@@ -10,35 +10,70 @@ License-File: NOTICE
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Requires-Python: >=3.10
+Requires-Python: >=3.11
 Requires-Dist: bedrock-agentcore-starter-toolkit>=0.2.0
-Requires-Dist: bedrock-agentcore>=1.0.3
+Requires-Dist: bedrock-agentcore>=1.23.1
 Requires-Dist: boto3>=1.40.55
 Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: a2a
+Requires-Dist: a2a-sdk[http-server]<2.0,>=1.0.1; extra == 'a2a'
+Requires-Dist: aioboto3>=15.5.0; extra == 'a2a'
+Requires-Dist: aiohttp>=3.9; extra == 'a2a'
+Requires-Dist: backoff; extra == 'a2a'
+Requires-Dist: httpx>=0.27.0; extra == 'a2a'
+Requires-Dist: pydantic; extra == 'a2a'
+Provides-Extra: a2a-server
+Requires-Dist: a2a-sdk[http-server]<2.0,>=1.0.1; extra == 'a2a-server'
 Provides-Extra: dev
+Requires-Dist: a2a-sdk[http-server]<2.0,>=1.0.1; extra == 'dev'
+Requires-Dist: aioboto3>=15.5.0; extra == 'dev'
+Requires-Dist: aiohttp>=3.9; extra == 'dev'
+Requires-Dist: backoff; extra == 'dev'
+Requires-Dist: httpx>=0.27.0; extra == 'dev'
 Requires-Dist: mypy>=1.0; extra == 'dev'
+Requires-Dist: openai>=1.0; extra == 'dev'
 Requires-Dist: pre-commit>=3.0; extra == 'dev'
+Requires-Dist: pydantic; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
 Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: tokenizers>=0.22.2; extra == 'dev'
+Requires-Dist: transformers>=5.0; extra == 'dev'
+Provides-Extra: gateway
+Requires-Dist: aiohttp>=3.9; extra == 'gateway'
+Requires-Dist: jmespath>=1.0; extra == 'gateway'
+Requires-Dist: tokenizers>=0.22.2; extra == 'gateway'
+Requires-Dist: transformers>=5.0; extra == 'gateway'
 Provides-Extra: rllm
 Requires-Dist: rllm; extra == 'rllm'
+Provides-Extra: rollout
+Requires-Dist: aioboto3>=15.5.0; extra == 'rollout'
+Requires-Dist: aiohttp>=3.9; extra == 'rollout'
+Requires-Dist: backoff; extra == 'rollout'
+Requires-Dist: httpx>=0.27.0; extra == 'rollout'
+Requires-Dist: pydantic; extra == 'rollout'
 Provides-Extra: slime
 Requires-Dist: rllm-model-gateway>=0.1.0; extra == 'slime'
+Provides-Extra: tinker-api
+Requires-Dist: tinker==0.30.1; extra == 'tinker-api'
+Provides-Extra: tinker-skyrl
+Requires-Dist: tinker==0.24.1; extra == 'tinker-skyrl'
 Provides-Extra: verl
+Requires-Dist: aiohttp>=3.9; extra == 'verl'
 Requires-Dist: flash-attn==2.8.3; extra == 'verl'
+Requires-Dist: flash-linear-attention>=0.5.2; extra == 'verl'
 Requires-Dist: numpy; extra == 'verl'
+Requires-Dist: orjson; extra == 'verl'
 Requires-Dist: qwen-vl-utils; extra == 'verl'
 Requires-Dist: ray; extra == 'verl'
-Requires-Dist: rllm-model-gateway>=0.1.0; extra == 'verl'
 Requires-Dist: torch>=2.10.0; extra == 'verl'
 Requires-Dist: torchvision>=0.23.0; extra == 'verl'
-Requires-Dist: transformers>=5.5.3; extra == 'verl'
-Requires-Dist: verl==0.8.0; extra == 'verl'
-Requires-Dist: vllm==0.21.0; extra == 'verl'
+Requires-Dist: transferqueue==0.1.8; extra == 'verl'
+Requires-Dist: transformers==5.12.1; extra == 'verl'
+Requires-Dist: verl==0.9.0; extra == 'verl'
+Requires-Dist: vllm==0.24.0; extra == 'verl'
 Description-Content-Type: text/markdown
 
 
@@ -88,6 +123,7 @@ While session routing is valuable for multi-turn production agents, RL rollouts
 - **Async end-to-end**: Fire-and-forget server + future-based client — no long-lived TCP connections needed for long-running agent rollouts
 - **Simplified rollout collection**: `RolloutClient` manages concurrency, rate limiting, and S3 result polling out of the box for both training loops and batch evaluation
 - **Production parity**: Direct code reuse from production agents ensures training behavior matches real deployment
+- **Sandbox on ACR** (`SandboxClient` + `agentcore-sandboxd`): Run shell commands in arbitrary Docker images (e.g. SWE-bench-style coding environments) as isolated AgentCore runtime sessions — no agent server required in the image
 
 ## Agent-Side: Adapting Your Agent for RL
 
@@ -313,6 +349,10 @@ This architecture enables parallel and highly efficient rollouts with secure exe
 - [rLLM](https://github.com/rllm-org/rllm) — supports multiple backends (veRL, Tinker, and more)
   - [Math Agent](examples/strands_math_agent/): [Tinker](https://github.com/rllm-org/rllm/blob/main/examples/agentcore_math/train_agentcore_math_tinker.sh)
 
+For a self-hosted Tinker-compatible training and sampling endpoint, see the
+[SkyRL EC2 deployment example](src/agentcore_rl_toolkit/backends/tinker_api/skyrl/).
+It uses SkyPilot to deploy Qwen3.5-4B on one P4d.
+
 ### Prepare Your Agent Container
 
 ACR deploys agents as Docker containers. Most Dockerfiles can be auto-generated with the [AgentCore CLI](https://aws.github.io/bedrock-agentcore-starter-toolkit/api-reference/cli.html). Example command:
@@ -352,12 +392,29 @@ chmod +x scripts/build_docker_image_and_push_to_ecr.sh
   --context=examples/strands_math_agent
 ```
 
+## Sandbox: Command Execution in Arbitrary Images
+
+Coding-agent evaluation and RL training typically start from prebuilt environment images (repository checkout, dependencies, test runner) that are not HTTP agent servers. The sandbox SDK bridges them onto AgentCore Runtime: a tiny health shim (`agentcore-sandboxd`, built from [`sandboxd/`](./sandboxd/)) is added to the image to satisfy the ACR container contract, and commands run through ACR's native `InvokeAgentRuntimeCommand` API.
+
+```python
+from agentcore_rl_toolkit.sandbox import SandboxClient
+
+client = SandboxClient(runtime_arn="arn:aws:bedrock-agentcore:...:runtime/...")
+with client.start() as sb:
+    result = sb.exec("cd /app && pytest -q", timeout=900)
+    print(result.exit_code, result.stdout)
+```
+
+Failing commands and timeouts are results, not exceptions — exactly what reward functions need. See [`examples/sandbox_quickstart/`](./examples/sandbox_quickstart/) for the full walkthrough (build the shim binary, wrap and push an image, create a runtime, run commands).
+
 ## Contributing
 
 We welcome contributions!
 
 - **Adding examples**: Create a new folder under `examples/` with its own `pyproject.toml`
 - **Improving the core library**: See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup
+- **Proposing significant features or architectural changes**: See the
+  [design document process](designs/README.md)
 
 For bug reports, feature requests, and pull request guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).
 

{agentcore_rl_toolkit-0.1.3 → agentcore_rl_toolkit-0.1.4}/README.md

@@ -45,6 +45,7 @@ While session routing is valuable for multi-turn production agents, RL rollouts
 - **Async end-to-end**: Fire-and-forget server + future-based client — no long-lived TCP connections needed for long-running agent rollouts
 - **Simplified rollout collection**: `RolloutClient` manages concurrency, rate limiting, and S3 result polling out of the box for both training loops and batch evaluation
 - **Production parity**: Direct code reuse from production agents ensures training behavior matches real deployment
+- **Sandbox on ACR** (`SandboxClient` + `agentcore-sandboxd`): Run shell commands in arbitrary Docker images (e.g. SWE-bench-style coding environments) as isolated AgentCore runtime sessions — no agent server required in the image
 
 ## Agent-Side: Adapting Your Agent for RL
 
@@ -270,6 +271,10 @@ This architecture enables parallel and highly efficient rollouts with secure exe
 - [rLLM](https://github.com/rllm-org/rllm) — supports multiple backends (veRL, Tinker, and more)
   - [Math Agent](examples/strands_math_agent/): [Tinker](https://github.com/rllm-org/rllm/blob/main/examples/agentcore_math/train_agentcore_math_tinker.sh)
 
+For a self-hosted Tinker-compatible training and sampling endpoint, see the
+[SkyRL EC2 deployment example](src/agentcore_rl_toolkit/backends/tinker_api/skyrl/).
+It uses SkyPilot to deploy Qwen3.5-4B on one P4d.
+
 ### Prepare Your Agent Container
 
 ACR deploys agents as Docker containers. Most Dockerfiles can be auto-generated with the [AgentCore CLI](https://aws.github.io/bedrock-agentcore-starter-toolkit/api-reference/cli.html). Example command:
@@ -309,12 +314,29 @@ chmod +x scripts/build_docker_image_and_push_to_ecr.sh
   --context=examples/strands_math_agent
 ```
 
+## Sandbox: Command Execution in Arbitrary Images
+
+Coding-agent evaluation and RL training typically start from prebuilt environment images (repository checkout, dependencies, test runner) that are not HTTP agent servers. The sandbox SDK bridges them onto AgentCore Runtime: a tiny health shim (`agentcore-sandboxd`, built from [`sandboxd/`](./sandboxd/)) is added to the image to satisfy the ACR container contract, and commands run through ACR's native `InvokeAgentRuntimeCommand` API.
+
+```python
+from agentcore_rl_toolkit.sandbox import SandboxClient
+
+client = SandboxClient(runtime_arn="arn:aws:bedrock-agentcore:...:runtime/...")
+with client.start() as sb:
+    result = sb.exec("cd /app && pytest -q", timeout=900)
+    print(result.exit_code, result.stdout)
+```
+
+Failing commands and timeouts are results, not exceptions — exactly what reward functions need. See [`examples/sandbox_quickstart/`](./examples/sandbox_quickstart/) for the full walkthrough (build the shim binary, wrap and push an image, create a runtime, run commands).
+
 ## Contributing
 
 We welcome contributions!
 
 - **Adding examples**: Create a new folder under `examples/` with its own `pyproject.toml`
 - **Improving the core library**: See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup
+- **Proposing significant features or architectural changes**: See the
+  [design document process](designs/README.md)
 
 For bug reports, feature requests, and pull request guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).
 

{agentcore_rl_toolkit-0.1.3 → agentcore_rl_toolkit-0.1.4}/docs/site/src/content/docs/api/backends/slime/runner.md

@@ -41,13 +41,22 @@ SlimeRunner(
     slime_dir: str = '/root/slime',
     megatron_dir: str = '/root/Megatron-LM',
 
+    # --- Optional: CUDA toolchain pinning ---
+    cuda_home: str | None = None,
+
     # --- Optional: ACR / toolkit (forwarded to slime via custom-config yaml) ---
     model_id: str = 'default',
     acr_timeout: int = 900,
     acr_tps_limit: int = 25,
     max_concurrent: int = 100,
+    max_pool_connections: int = 100,
     gateway_port: int = 9090,
     reward_postprocessing: str = 'grpo',
+    sglang_tool_call_parser: str = 'qwen',
+    sglang_reasoning_parser: str | None = None,
+    cumulative_token_mode: bool = False,
+    renderer_family: str = 'auto',
+    gateway_log_level: str = 'warning',
 
     # --- Optional: training hyperparameters ---
     rollout_batch_size: int = 32,
@@ -60,6 +69,7 @@ SlimeRunner(
     weight_decay: float = 0.1,
     adam_beta2: float = 0.98,
     sglang_mem_fraction_static: float = 0.7,
+    sglang_context_length: int | None = None,
     max_tokens_per_gpu: int = 9216,
 
     # --- Wandb (opt-in; no defaults injected if unset) ---
@@ -95,6 +105,10 @@ via ``ray job submit``. Streams stdout/stderr to the parent process.
 
 - `agent_runtime_arn` *(str)*
 
+- `cuda_home` *(str | None)*
+
+- `cumulative_token_mode` *(bool)*
+
 - `data_path` *(str)*
 
 - `eps_clip` *(float)*
@@ -105,12 +119,16 @@ via ``ray job submit``. Streams stdout/stderr to the parent process.
 
 - `extra_flags` *(list[str])*
 
+- `gateway_log_level` *(str)*
+
 - `gateway_port` *(int)*
 
 - `lr` *(float)*
 
 - `max_concurrent` *(int)*
 
+- `max_pool_connections` *(int)*
+
 - `max_tokens_per_gpu` *(int)*
 
 - `megatron_dir` *(str)*
@@ -125,6 +143,8 @@ via ``ray job submit``. Streams stdout/stderr to the parent process.
 
 - `num_gpus` *(int)*
 
+- `renderer_family` *(str)*
+
 - `reward_postprocessing` *(str)*
 
 - `rollout_batch_size` *(int)*
@@ -137,8 +157,14 @@ via ``ray job submit``. Streams stdout/stderr to the parent process.
 
 - `s3_bucket` *(str)*
 
+- `sglang_context_length` *(int | None)*
+
 - `sglang_mem_fraction_static` *(float)*
 
+- `sglang_reasoning_parser` *(str | None)*
+
+- `sglang_tool_call_parser` *(str)*
+
 - `slime_dir` *(str)*
 
 - `tp_size` *(int)*

{agentcore_rl_toolkit-0.1.3 → agentcore_rl_toolkit-0.1.4}/docs/site/src/content/docs/api/core/client.md

@@ -198,7 +198,7 @@ RolloutFuture(
     input_id: str = None,
     agentcore_client = None,
     agent_runtime_arn: str = None,
-    rate_limiter: ACRRateLimiter = None,
+    rate_limiter: LocalRateLimiter = None,
 )
 ```
 

agentcore_rl_toolkit-0.1.4/docs/site/src/content/docs/blog/async-rl-performance-optimization.md

@@ -0,0 +1,111 @@
+---
+title: Async RL Performance Optimization
+description: Improving training throughput with asynchronous reinforcement learning
+---
+
+_By [Danylo Vashchilenko](https://github.com/hellodanylo) · October 1, 2026_
+
+:::note[TL;DR]
+* 🚀 While training [SWE Gym](../../examples/openhands-swegym-agent) agent, we show that asynchronous RL enables higher learning throughput, leading to ~22% higher reward improvement per GPU-hour of training.
+* ℹ️ We show that the efficiency increase is explained by higher LLM inference concurrency and throughput.
+* ⏭️ We observe that KV cache remains underutilized (time-average at ~25%), which is an opportunity for further optimization.
+:::
+
+## RL Throughput and Concurrency of Inference and Training
+
+All other things being equal, we prefer an RL system with higher throughput per GPU, as measured in processed trajectories per GPU-hour.
+In highly parallel systems, **throughput and concurrency are positively correlated** up to a certain level of concurrency (the saturation point).
+Measuring and maximizing concurrency is the primary method of optimizing the RL training system.
+
+For the inference servers, KV cache capacity of GPU memory is the bottleneck in SWE and other long-context tasks.
+For example, for Qwen3 Coder 30B on p6 node, there is ~5M tokens of capacity, which is only 78 trajectories at 64K context.
+However, the actual concurrency can be significantly lower than the capacity.
+For example, inference concurrency typically peaks right after a rollout batch is dispatched
+and reaches a low right before the next rollout batch is dispatched. When the distribution
+of the trajectory-level latency has high variance, the tail can significantly
+reduce the actual concurrency and therefore inference throughput. In this scenario,
+increasing rollout batch size through off-policy training can improve inference throughput.
+
+For example, the following plots show the inference concurrency (x-axis)
+and throughput (y-axis) over the span of a training run. We see
+that the variance of throughput is directly explained by the changes in concurrency.
+![Throughput vs Concurrency](../../../assets/blog/async-rl-performance-optimization/inference_throughput_and_concurrency.png)
+
+In this report, we will specifically compare (1) 1 mini-batch per step (less off-policy), and (2) 2 mini-batches per step (more off-policy).
+We expect that **increasing mini-batch count per step will increase rollout throughput**, because higher inference concurrency will bring the hardware closer to the utilization saturation point.
+
+![Sync RL](../../../assets/blog/async-rl-performance-optimization/sync_rl.png)
+
+For the training servers, GPU memory (activations, gradients, optimizer state) is also the bottleneck, but the token batch size is typically significantly higher than the saturation point, so the latency of the update step is roughly linear in the count of tokens. For long trajectories, the training server typically needs [memory saving methods](https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-guide.html) like Model/Context Parallelism, Activation Recomputation and Offloading, which reduce throughput. Quantization can significantly improve throughput if it allows to turn off the memory saving methods without hitting OOMs.
+Notably, context compression during rollout can improve the training server's efficiency,
+because it only needs enough GPU memory to handle the longest *sequence*, not the longest trajectory.
+
+It's previously been widely discussed that **RL is rollout-bound, but that's not always true for multi-turn RL** like SWE. The inference server encodes the new context tokens (e.g. prompt and tool responses) in parallel and decodes the agent response tokens sequentially. The training server encodes all tokens in parallel, but also does the backward pass for each token. Therefore, the rollout throughput is positively correlated with the non-agent fraction of tokens in the context, while the trainer throughput does not depend on this ratio.
+
+| Server | Agent Tokens | Environment Tokens |
+| --- | --- | --- |
+| Training | parallel fwd+bwd | parallel fwd+bwd |
+| Inference | sequential fwd (slower) | parallel fwd (faster) |
+
+What we observe in this report's training scenario (Qwen3 Coder 30B on SWE Gym) is that in an average trajectory only ~20% of tokens are from the agent.
+In this regime, the **inference and training servers can have roughly the same throughput per GPU**.
+
+In short, the bottleneck is an async RL pipeline depends on many characteristics of the workload, but it's certainly not always rollout-bound.
+This observation is highly important for prioritizing training efficiency research, and for allocating hardware resources between training and inference servers in experiments.
+
+Using the framework of this section, we expect the disaggregated cluster to have higher throughput than a colocated cluster, because reducing inference servers count increases the effective inference concurrency. And we expect the async RL pipeline to have higher throughput than sync RL pipeline, because rollout batches can overlap, leading to higher inference concurrency.
+
+![Sync RL](../../../assets/blog/async-rl-performance-optimization/async_rl.png)
+
+With the knowledge that the inference and trainer throughput are roughly the same, we choose to have 1:1 ratio of GPUs allocated to trainer and rollout in the disaggregated cluster. We will specifically compare (1) a colocated cluster with 2 hybrid model replicas and (2) a disaggregated cluster with 1 training replica and 1 inference replica.
+
+## Training Results
+
+First, let's review the trainer and rollout throughput measurements of these configurations. As expected, we observe a significant throughput speedup (up to ~1.42x) from using off-policy and async RL, which is explained by the increase of average inference request concurrency. In the sync-1 baseline, the average concurrency of ~11 does not saturate the GPU throughput. Additionally, we observe that average KV cache utilization in all cases is <25%, which means that there is 4x headroom for higher concurrency, which is an opportunity for further optimization.
+
+| **Training Cluster Config** | **Throughput (kilo-tokens/minute/GPU)** | **Throughput Speedup** | **Inference Concurrency per Replica** |
+| --- | --- | --- | --- |
+| Sync 1 step | 27.82 | 1 | 11.03 |
+| Sync 2 steps | 34.47 | 1.24x | 15.20 |
+| Async 1 step | 38.49 | 1.38x | 33.97 |
+| Async 2 steps | 39.39 | 1.42x | 38.22 |
+
+We know from the past RL research that off-policy learning can be less efficient per optimizer step, so next we measure whether reward
+has the same rate of change per 100 optimizer steps. We observe that Async-1 and Sync-2 configurations reduce the rate of reward improvement, while Async-2 configuration increases it.
+
+| **Training Cluster Config** | **Reward Improvement Rate (per 100 optimizer steps)** | **Reward Improvement Speedup** |
+| --- | --- | --- |
+| Sync 1 step | 5.98pp | 1 |
+| Sync 2 steps | 5.71pp | 0.96x |
+| Async 1 step | 4.55pp | 0.76x |
+| Async 2 steps | 7.79pp | 1.30x |
+
+Finally, we would like to combine the two parts of this analysis -- throughput per GPU and reward improvement rate -- giving us a measurement of the reward change per GPU-hour. In the following table, we calculate the reward's rate of change per 640 GPU-hours. We observe that Async-2 configuration learns 1.22x faster and cheaper than the Sync-1 baseline.
+
+| **Training Cluster Config** | **Reward Improvement Rate (per 640 GPU-hours)** | **Reward Improvement Speedup** |
+| --- | --- | --- |
+| Sync 1 step | 7.17pp | 1 |
+| Sync 2 steps | 7.86pp | 1.10x |
+| Async 1 step | 7.05pp | 0.98x |
+| Async 2 steps | 8.73pp | 1.22x |
+
+
+We plot the learning curve of the Sync-1 and Async-2 training runs against the GPU-hours used to observe that Async-2 configuration has been ahead of the Sync-1 run during most of the training runtime.
+
+![Reward Curve](../../../assets/blog/async-rl-performance-optimization/reward_curve.png)
+
+## Rollout Correction
+
+In this report, we used Verl's implementation of [rollout correction](https://verl.readthedocs.io/en/latest/algo/rollout_corr.html)
+with the following parameters: token-level importance ratio, masking threshold of 2, decoupled mode.
+
+In order to understand the off-policy dynamics of these training runs,
+we plot the effective non-masked ratio of tokens in each training step, as reported by Verl.
+We observe that the ratio of masked tokens grows faster for 2-step configurations
+than for 1-step configuration. Notably, up to 3% of tokens are masked towards
+the end of the training runs for Sync-2 and Async-2 configurations.
+
+![Rollout Correction](../../../assets/blog/async-rl-performance-optimization/rollout_corr_eff_sample_size.png)
+
+In conclusion, training stability remains an important measurement
+in asynchronous and off-policy RL, and further counter-measures might be needed in longer runs.