trinity-rft 0.1.0__tar.gz

trinity_rft-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

trinity_rft-0.1.0/PKG-INFO

@@ -0,0 +1,396 @@
+Metadata-Version: 2.4
+Name: trinity-rft
+Version: 0.1.0
+Summary: Trinity-RFT: A Framework for Training Large Language Models with Reinforcement Fine-Tuning
+Author-email: Trinity-RFT Team <trinity-rft@outlook.com>
+Project-URL: Homepage, https://github.com/modelscope/Trinity-RFT
+Project-URL: Documentation, https://modelscope.github.io/Trinity-RFT/
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: verl==0.3.0.post1
+Requires-Dist: ray[default]>=2.45.0
+Requires-Dist: vllm>=0.8.5
+Requires-Dist: tensordict==0.6.2
+Requires-Dist: wandb
+Requires-Dist: omegaconf
+Requires-Dist: sqlalchemy
+Requires-Dist: psycopg2-binary
+Requires-Dist: networkx
+Requires-Dist: latex2sympy2_extended
+Requires-Dist: math_verify
+Requires-Dist: ninja
+Requires-Dist: fire
+Requires-Dist: streamlit
+Requires-Dist: flask
+Requires-Dist: requests
+Requires-Dist: tensorboard
+Requires-Dist: openai
+Provides-Extra: data
+Requires-Dist: py-data-juicer; extra == "data"
+Provides-Extra: agent
+Requires-Dist: agentscope; extra == "agent"
+Provides-Extra: dev
+Requires-Dist: pre-commit>=2.17.0; extra == "dev"
+Requires-Dist: black>=23.7.0; extra == "dev"
+Requires-Dist: flake8>=6.1.0; extra == "dev"
+Requires-Dist: flake8-docstrings>=1.6.0; extra == "dev"
+Requires-Dist: isort>=5.12.0; extra == "dev"
+Requires-Dist: mypy>=1.7.0; extra == "dev"
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-json-ctrf; extra == "dev"
+Provides-Extra: doc
+Requires-Dist: sphinx; extra == "doc"
+Requires-Dist: sphinx-autobuild; extra == "doc"
+Requires-Dist: sphinx_rtd_theme; extra == "doc"
+Requires-Dist: myst-parser; extra == "doc"
+Dynamic: license-file
+
+
+
+<!-- ![trinity-rft](./docs/sphinx_doc/assets/trinity-title.png) -->
+
+<div align="center">
+  <img src="https://img.alicdn.com/imgextra/i1/O1CN01lvLpfw25Pl4ohGZnU_!!6000000007519-2-tps-1628-490.png" alt="Trinity-RFT" style="height: 120px;">
+</div>
+
+
+&nbsp;
+
+
+
+**Trinity-RFT is a general-purpose, flexible, scalable and user-friendly framework designed for reinforcement fine-tuning (RFT) of large language models (LLM).**
+
+
+Built with a decoupled design, seamless integration for agent-environment interaction, and systematic data processing pipelines, Trinity-RFT can be easily adapted for diverse application scenarios, and serve as a unified platform for exploring advanced reinforcement learning (RL) paradigms.
+
+
+
+
+
+## Vision of this project
+
+
+Current RFT approaches, such as RLHF (Reinforcement Learning from Human Feedback) with proxy reward models or training long-CoT reasoning models with rule-based rewards, are limited in their ability to handle dynamic, real-world, and continuous learning.
+
+Trinity-RFT envisions a future where AI agents learn by interacting directly with environments, collecting delayed or complex reward signals, and continuously refining their behavior through RL.
+
+
+For example, imagine an AI scientist that designs an experiment, executes it, waits for feedback (while working on other tasks concurrently), and iteratively updates itself based on true environmental rewards when the experiment is finally finished.
+
+
+Trinity-RFT offers a path into this future by providing various useful features.
+
+
+
+
+
+## Key features
+
+
+
++ **Unified RFT modes & algorithm support.**
+Trinity-RFT unifies and generalizes existing RFT methodologies into a flexible and configurable framework, supporting synchronous/asynchronous, on-policy/off-policy, and online/offline training, as well as hybrid modes that combine them seamlessly into a single learning process.
+
+
++ **Agent-environment interaction as a first-class citizen.**
+Trinity-RFT allows delayed rewards in multi-step/time-lagged feedback loops, handles long-tailed latencies and environment/agent failures gracefully, and supports distributed deployment where explorers and trainers can operate across separate devices and scale up independently.
+
+
+
++ **Data processing pipelines optimized for RFT with diverse/messy data.**
+These include converting raw datasets to task sets for RL, cleaning/filtering/prioritizing experiences stored in the replay buffer, synthesizing data for tasks and experiences, offering user interfaces for human in the loop, etc.
+
+
+
+## The design of Trinity-RFT
+
+
+<!-- ![design](./docs/sphinx_doc/assets/trinity-design.png) -->
+
+<div align="center">
+  <img src="https://img.alicdn.com/imgextra/i2/O1CN01X5jFm81peNsADtRt2_!!6000000005385-2-tps-3298-1498.png" alt="Trinity-RFT">
+</div>
+
+
+
+
+
+The overall design of Trinity-RFT exhibits a trinity:
++ RFT-core;
++ agent-environment interaction;
++ data processing pipelines;
+
+and the design of RFT-core also exhibits a trinity:
++ explorer;
++ trainer;
++ buffer.
+
+
+
+The *explorer*, powered by the rollout model, interacts with the environment and generates rollout trajectories to be stored in the experience buffer.
+
+The *trainer*, powered by the policy model, samples batches of experiences from the buffer and updates the policy model via RL algorithms.
+
+These two can be completely decoupled and act asynchronously on separate machines, except that they share the same experience buffer, and their model weights are synchronized once in a while.
+Such a decoupled design is crucial for making the aforementioned features of Trinity-RFT possible.
+
+<!-- e.g., flexible and configurable RFT modes (on-policy/off-policy, synchronous/asynchronous, immediate/lagged rewards),
+fault tolerance for failures of explorer (agent/environment) or trainer,
+high efficiency in the presence of long-tailed rollout latencies,
+data processing pipelines and human in the loop of RFT (e.g., via acting on the experience buffer, which is implemented as a persistent database),
+among others. -->
+
+
+
+Meanwhile, Trinity-RFT has done a lot of work to ensure high efficiency and robustness in every component of the framework,
+e.g., utilizing NCCL (when feasible) for model weight synchronization, sequence concatenation with proper masking for multi-turn conversations and ReAct-style workflows, pipeline parallelism for the synchronous RFT mode,
+asynchronous and concurrent LLM inference for rollout,
+fault tolerance for agent/environment failures,
+among many others.
+
+
+
+## Getting started
+
+
+> [!NOTE]
+> This project is currently under active development. Comments and suggestions are welcome!
+
+
+
+
+### Step 1: preparations
+
+
+
+
+Installation from source (recommended):
+
+```shell
+# Pull the source code from GitHub
+git clone https://github.com/modelscope/Trinity-RFT
+cd Trinity-RFT
+
+# Create a new environment using Conda or venv
+# Option 1: Conda
+conda create -n trinity python=3.10
+conda activate trinity
+
+# Option 2: venv
+python3.10 -m venv .venv
+source .venv/bin/activate
+
+# Install the package in editable mode
+# for bash
+pip install -e .[dev]
+# for zsh
+pip install -e .\[dev\]
+
+# Install flash-attn after all dependencies are installed
+# Note: flash-attn will take a long time to compile, please be patient.
+pip install flash-attn -v
+# Try the following command if you encounter errors during installation
+# pip install flash-attn -v --no-build-isolation
+```
+
+Installation from docker:
+we have provided a dockerfile for Trinity-RFT (trinity)
+
+```shell
+git clone https://github.com/modelscope/Trinity-RFT
+cd Trinity-RFT
+
+# build the docker image
+# Note: you can edit the dockerfile to customize the environment
+# e.g., use pip mirrors or set api key
+docker build -f scripts/docker/Dockerfile -t trinity-rft:latest .
+
+# run the docker image
+docker run -it --gpus all --shm-size="64g" --rm -v $PWD:/workspace -v <root_path_of_data_and_checkpoints>:/data trinity-rft:latest
+```
+
+
+Trinity-RFT requires
+Python version >= 3.10,
+CUDA version >= 12.4,
+and at least 2 GPUs.
+
+
+### Step 2: prepare dataset and model
+
+
+Trinity-RFT supports most datasets and models from Huggingface and ModelScope.
+
+
+**Prepare the model** in the local directory `$MODEL_PATH/{model_name}`:
+
+```bash
+# Using Huggingface
+huggingface-cli download {model_name} --local-dir $MODEL_PATH/{model_name}
+
+# Using Modelscope
+modelscope download {model_name} --local_dir $MODEL_PATH/{model_name}
+```
+
+For more details about model downloading, please refer to [Huggingface](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli) or  [ModelScope](https://modelscope.cn/docs/models/download).
+
+
+
+**Prepare the dataset** in the local directory `$DATASET_PATH/{dataset_name}`:
+
+```bash
+# Using Huggingface
+huggingface-cli download {dataset_name} --repo-type dataset --local-dir $DATASET_PATH/{dataset_name}
+
+# Using Modelscope
+modelscope download --dataset {dataset_name} --local_dir $DATASET_PATH/{dataset_name}
+```
+
+For more details about dataset downloading, please refer to [Huggingface](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli#download-a-dataset-or-a-space) or [ModelScope](https://modelscope.cn/docs/datasets/download).
+
+
+
+### Step 3: configurations
+
+
+For convenience, Trinity-RFT provides a web interface for configuring your RFT process.
+
+> [!NOTE]
+> This is an experimental feature, and we will continue to improve it.
+
+```bash
+trinity studio --port 8080
+```
+
+
+Then you can configure your RFT process in the web page and generate a config file.
+You can save the config for later use or run it directly as described in the following section.
+
+
+Advanced users can also configure the RFT process by editing the config file directly.
+We provide a set of example config files in [`examples`](examples/).
+
+
+### Step 4: run the RFT process
+
+
+First, start a ray cluster with the following command:
+
+```shell
+# On master node
+ray start --head
+
+# On worker nodes
+ray start --address=<master_address>
+```
+
+Optionally, we can login into [wandb](https://docs.wandb.ai/quickstart/) to better monitor the RFT process:
+
+```shell
+export WANDB_API_KEY=<your_api_key>
+wandb login
+```
+
+Then, for command-line users, run the RFT process with the following command:
+
+```shell
+trinity run --config <config_path>
+```
+
+> For example, below is the command for fine-tuning Qwen-2.5-1.5B-Instruct on GSM8k dataset using GRPO algorithm:
+> ```shell
+> trinity run --config examples/grpo_gsm8k/gsm8k.yaml
+> ```
+
+For studio users, just click the "Run" button in the web page.
+
+
+For more detailed examples about how to use Trinity-RFT, please refer to the following tutorials:
++ [A quick example with GSM8k](./docs/sphinx_doc/source/tutorial/example_reasoning_basic.md)
++ [Off-policy mode of RFT](./docs/sphinx_doc/source/tutorial/example_reasoning_advanced.md)
++ [Asynchronous mode of RFT](./docs/sphinx_doc/source/tutorial/example_async_mode.md)
++ [Multi-turn tasks](./docs/sphinx_doc/source/tutorial/example_multi_turn.md)
++ [Offline learning by DPO](./docs/sphinx_doc/source/tutorial/example_dpo.md)
++ [Advanced data processing / human-in-the-loop](./docs/sphinx_doc/source/tutorial/example_data_functionalities.md)
+
+
+
+
+
+## Advanced usage and full configurations
+
+
+Please refer to [this document](./docs/sphinx_doc/source/tutorial/trinity_configs.md).
+
+
+
+
+
+## Programming guide for developers
+
+
+Please refer to [this document](./docs/sphinx_doc/source/tutorial/trinity_programming_guide.md).
+
+
+## Upcoming features
+
+A tentative roadmap: https://github.com/modelscope/Trinity-RFT/issues/51
+
+
+
+## Contribution guide
+
+
+This project is currently under active development, and we welcome contributions from the community!
+
+
+Code style check:
+
+```shell
+pre-commit run --all-files
+```
+
+
+
+Unit tests:
+
+```shell
+python -m pytest tests
+```
+
+
+
+## Acknowledgements
+
+
+This project is built upon many excellent open-source projects, including:
+
++ [verl](https://github.com/volcengine/verl) and [PyTorch's FSDP](https://pytorch.org/docs/stable/fsdp.html) for LLM training;
++ [vLLM](https://github.com/vllm-project/vllm) for LLM inference;
++ [Data-Juicer](https://github.com/modelscope/data-juicer?tab=readme-ov-file) for data processing pipelines;
++ [AgentScope](https://github.com/modelscope/agentscope) for agentic workflow;
++ [Ray](https://github.com/ray-project/ray) for distributed systems;
++ we have also drawn inspirations from RL frameworks like [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF), [TRL](https://github.com/huggingface/trl) and [ChatLearn](https://github.com/alibaba/ChatLearn);
++ ......
+
+
+
+
+
+## Citation
+```plain
+@misc{Trinity-RFT,
+  title={Trinity-RFT},
+  author={{Trinity-RFT Team}},
+  url={https://github.com/modelscope/trinity-rft},
+  year={2025},
+}
+```