starforge-cli 0.1.6__py3-none-any.whl

starforge_cli/scaffold/experiment-template/README.md

@@ -0,0 +1,36 @@
+# <method>_<model>_<dataset>_<tag>
+
+> 复制本模板新建实验：`sf new <新实验名> --method <framework>/<method>`
+> 实验名遵循 `docs/naming-convention.md`。
+
+## 目标
+
+一句话说明这个实验要验证 / 达成什么。
+
+## 配置（NeMo-RL 0.6.0，配置继承）
+
+- `config.yaml` 通过 `defaults` 继承基底（`configs/base/`）+ 模型片段（`configs/models/`），
+  **只写本实验差异**；不断调参就改 `config.yaml` 的「本实验差异」部分。
+- 训练入口、指标与产物契约由 SDK 中的版本化 recipe 声明；实验目录不写 `framework`
+  标记，也不按 `run.py`/`train.sh` 是否存在猜测框架。
+- 硬件与资源：提交时 `sf submit --profile 名称[:总卡数]` 一个参数说清（如 `h200`、`h200:4`）；
+  卡型、默认形状、env/overrides 均由 Console 服务端注册表下发。
+
+## 监控
+
+- project：`<实验名>`
+- run：`<超参组合>`
+- 链接：<贴上监控面板链接>
+
+## 运行
+
+```bash
+uv run sf submit <实验名> --profile h100
+```
+
+产物（checkpoint / 日志）落到本目录 `outputs/`（已 .gitignore）。
+
+## 结果与结论
+
+- 关键指标：
+- 结论 / 下一步：

starforge_cli/scaffold/experiment-template/config.yaml

@@ -0,0 +1,44 @@
+# 实验配置：继承基底 + 模型片段，只写本实验差异（NeMo-RL 0.6.0 原生 defaults 机制）。
+# 调参就改下面「本实验差异」部分；基底不动。
+#
+# 切换方法：
+#   - SFT       : 使用 sft recipe 模板；入口由 recipe 固定为 examples/run_sft.py
+#   - 多轮 Agent: 换成 ../../configs/base/grpo_sliding_puzzle.yaml（已含 max_rollout_turns/env 范例）
+defaults:
+  - ../../configs/base/grpo_math_1B.yaml
+  - ../../configs/models/qwen3.5-4b.yaml
+
+# ===================== 本实验差异 =====================
+policy:
+  max_total_sequence_length: 4096
+  optimizer:
+    kwargs:
+      lr: 1.0e-6
+
+grpo:
+  num_prompts_per_step: 32
+  num_generations_per_prompt: 16
+  max_rollout_turns: 1            # 多轮 Agent 设 >1
+
+loss_fn:
+  reference_policy_kl_penalty: 0.01
+
+# 数据集（按实际填写，见 common/data/README.md 与官方 docs/guides/grpo.md）
+# data:
+#   train:
+#     data_path: /abs/path/train.jsonl
+#     input_key: question
+#   default:
+#     dataset_name: ResponseDataset
+#     env_name: math
+
+logger:
+  swanlab_enabled: true
+  swanlab:
+    project: "<method>_<model>_<dataset>_<tag>"   # 对齐实验目录名
+    name: "<超参组合>"                            # 如 lr1e6-g16-kl0.01
+  monitor_gpus: true
+# =====================================================
+# 注：checkpoint 目录与 log 目录由统一 launcher 指到本实验 outputs/（已 .gitignore），
+#     集群/硬件相关（节点数、并行度）由服务端 profile 注册表在运行时叠加（FORGE_PROFILE_OVERRIDES）。
+#     提交时 sf submit --profile 指定目标集群；下列数值应按该卡显存调。

starforge_cli/scaffold/project/common/README.md

@@ -0,0 +1,12 @@
+# common/ — 项目共享代码
+
+实验通过 `common.*` 引用的自定义代码放这里，随作业包上传到集群：
+
+| 目录 | 放什么 | 何时需要 |
+| --- | --- | --- |
+| `common/data/` | 数据预处理脚本 `prepare_*.py`（`sf dataset prepare` 自动发现） | 需要预处理内部数据时 |
+| `common/environments/` | 多轮 Agent 环境（工具调用 / 可验证任务） | Agent RL 实验 |
+| `common/rewards/` | 自定义奖励函数（关键词 / LLM 裁判） | RL 实验需要自定义奖励时 |
+
+纯 SFT / 数学 GRPO 实验用官方基底即可，不需要写 common 代码。
+Agent 环境与裁判/沙箱的写法见平台文档「Agent 环境套件」。

starforge_cli/scaffold/project/configs/README.md

@@ -0,0 +1,103 @@
+# configs/ — 配置继承体系
+
+NeMo-RL 0.7.0 **原生支持配置继承**（`nemo_rl/utils/config.py` 的 `load_config`）：配置里写
+`defaults: parent.yaml` 即可继承，支持多继承、嵌套继承、`${}` 插值、`_override_` 整段覆盖，
+再叠加命令行 Hydra override。官方自己就这么用（`grpo_sliding_puzzle.yaml` 继承 `grpo_math_1B.yaml`）。
+
+所以每个模型 / 每个实验都有自己的配置、不断调参，但**只写差异**，公共部分继承。
+
+## 三层结构
+
+```
+configs/
+├── base/      祖父：官方 v0.7.0 example 原样副本（version-locked，升级时从 NeMo-RL 源码目录整体拷入，勿手改）
+│   ├── grpo_math_1B.yaml          # GRPO 基底
+│   ├── sft.yaml                   # SFT 基底
+│   ├── grpo_sliding_puzzle.yaml   # 多轮 Agent 基底（已继承 grpo_math_1B）
+│   ├── grpo_megatron.yaml         # ★自定义 overlay：Megatron 后端 + 低显存实测显存/性能调优
+│   ├── grpo_lora.yaml             # ★自定义 overlay：LoRA/PEFT（低显存机型上让 9B 能跑起来）
+│   └── grpo_noncolocated.yaml     # ★自定义 overlay：非 colocated 生成（1 卡生成 / 1 卡训练）
+└── models/    父：各基础模型的公共片段（model_name / tokenizer / 显存策略…）
+    ├── qwen3.5-4b.yaml
+    └── qwen3.5-9b.yaml
+```
+
+## 训练后端 + LoRA（两个 overlay）
+
+本仓库 GRPO 实验默认 **Megatron-Core + LoRA**，靠两个 overlay 叠加（都来自低显存机型上跑通的配置）：
+
+```yaml
+defaults:
+  - ../../configs/base/grpo_math_1B.yaml
+  - ../../configs/models/qwen3.5-9b.yaml
+  - ../../configs/base/grpo_megatron.yaml   # ① Megatron 后端 + 低显存显存/性能调优
+  - ../../configs/base/grpo_lora.yaml       # ② LoRA（lr 1e-4 / wd 0 / cosine）
+policy:
+  max_total_sequence_length: 1250
+  train_global_batch_size: 32               # = num_prompts_per_step * num_generations_per_prompt
+grpo:
+  num_prompts_per_step: 4
+  num_generations_per_prompt: 8
+```
+
+- `grpo_megatron.yaml`：关 DTensor、开 Megatron，置空 FSDP 的 `policy.optimizer/scheduler`，并带上低显存实测显存项（`activation_checkpointing` / `empty_unused_memory_level=2` / `apply_rope_fusion=false` / `defer_fp32_logits` / `enforce_eager` / 关 sequence packing 等）。
+- `grpo_lora.yaml`：开 `megatron_cfg.peft`，LoRA 学习率写在 `megatron_cfg.optimizer.lr`（1e-4，比全参数高 2 个量级）。
+- `grpo_noncolocated.yaml`（可选第三层）：生成与训练各占独立 GPU（实测 9B 用法）。2 卡场景 = 1 卡生成 / 1 卡训练 → 训练侧 PP=1。删此行即回 colocated（共用 GPU）。
+
+切回方式：
+- **回全参数**：删 `grpo_lora.yaml` 一行，并把 lr 写到 `policy.megatron_cfg.optimizer.lr`（如 1e-6）。
+- **回 DTensor/FSDP**：删 `grpo_megatron.yaml` 一行，并把 lr 写回 `policy.optimizer.kwargs.lr`。
+
+> 并行度（TP/PP/CP）放 `cluster/<profile>/overrides.conf`；9B 实测 PP=1，4B 实测 PP=2。
+> `converter_type` 无需按模型改：Megatron 用 AutoBridge 按 HF 架构自动识别。
+> 显存类调优放在 overlay（merge）而非 `overrides.conf`：CLI override 是 struct 模式且对 SFT 也生效，而 SFT 没有 `policy.generation` 等键。
+
+实验目录里的 `config.yaml` 是**子**：
+
+```yaml
+defaults:
+  - ../../configs/base/grpo_math_1B.yaml   # 方法基底
+  - ../../configs/models/qwen3.5-9b.yaml   # 模型片段
+# 下面只写本实验差异：数据集 / lr / kl / swanlab / 步数 ...
+grpo:
+  num_generations_per_prompt: 16
+loss_fn:
+  reference_policy_kl_penalty: 0.01
+logger:
+  swanlab_enabled: true
+  swanlab: { project: "grpo_qwen3.5-9b_gsm8k_v1", name: "lr1e6-g16-kl0.01" }
+```
+
+> 合并规则：多继承中**后面覆盖前面**，实验自身再覆盖基底/模型。需要整段替换（不合并）时，
+> 在该段加 `_override_: true`。
+
+## 方法 → 入口 → 基底
+
+| 方法 | `ENTRY`（run.sh） | 基底（defaults 第一项） |
+| --- | --- | --- |
+| SFT | `examples/run_sft.py` | `../../configs/base/sft.yaml` |
+| GRPO | `examples/run_grpo.py` | `../../configs/base/grpo_math_1B.yaml` |
+| 多轮 Agent | `examples/run_grpo.py` | `../../configs/base/grpo_sliding_puzzle.yaml` |
+
+> 更大模型可加 `grpo_math_8B.yaml` 等官方基底：把文件名加进 `scripts/sync_base_configs.sh` 同步。
+
+## 集群 / 硬件
+
+不进 `config.yaml`，由 `cluster/<profile>/overrides.conf` 在运行时以 CLI override 叠加（见 `cluster/README.md`），
+这样切硬件（H100 ↔ H200）不动实验配置。
+
+## 常用字段（0.7.0）
+
+| 作用 | key |
+| --- | --- |
+| 基础模型 | `policy.model_name` |
+| 序列长度 | `policy.max_total_sequence_length` |
+| 学习率（Megatron/LoRA，默认） | `policy.megatron_cfg.optimizer.lr`（LoRA 用 1e-4） |
+| 学习率（DTensor 后端） | `policy.optimizer.kwargs.lr` |
+| LoRA 开关 / 秩 | `policy.megatron_cfg.peft.enabled` / `.dim` / `.alpha` |
+| 全局 batch（须整除 prompts×gen） | `policy.train_global_batch_size` |
+| KL 惩罚 | `loss_fn.reference_policy_kl_penalty` |
+| 每步 prompt 数 / 每 prompt 采样数 | `grpo.num_prompts_per_step` / `grpo.num_generations_per_prompt` |
+| 多轮轮数 | `grpo.max_rollout_turns` |
+| 节点 / 卡数 | `cluster.num_nodes` / `cluster.gpus_per_node` |
+| 启用 SwanLab | `logger.swanlab_enabled` + `logger.swanlab.{project,name}` |

starforge_cli/scaffold/project/configs/base/README.md

@@ -0,0 +1,24 @@
+# configs/base — 供继承的基底配置
+
+这里是 **NeMo-RL v0.7.0 官方 example 配置的原样副本**（version-locked），作为所有实验
+`defaults` 继承的「祖父配置」。实验只写差异，调参不动这里。
+
+| 文件 | 来源（v0.7.0） | 用途 |
+| --- | --- | --- |
+| `grpo_math_1B.yaml` | `examples/configs/grpo_math_1B.yaml` | GRPO 基底（最常用的祖父） |
+| `sft.yaml` | `examples/configs/sft.yaml` | SFT 基底 |
+| `grpo_sliding_puzzle.yaml` | `examples/configs/grpo_sliding_puzzle.yaml` | 多轮 Agent 基底（本身 `defaults: grpo_math_1B.yaml`） |
+
+> 官方就是这么用继承的：`grpo_sliding_puzzle.yaml` 第一行即 `defaults: "grpo_math_1B.yaml"`，
+> 只覆盖 `grpo.max_rollout_turns`、`env` 等差异。
+
+## 更新基底（升级 NeMo-RL 版本时）
+
+```bash
+# 从本地 NeMo-RL 源码把 example 配置原样拷入（升级是低频运维操作，直接 cp，不做 CLI 包装）
+cp /path/to/NeMo-RL/examples/configs/grpo_math_1B.yaml configs/base/
+cp /path/to/NeMo-RL/examples/configs/sft.yaml configs/base/
+cp /path/to/NeMo-RL/examples/configs/grpo_sliding_puzzle.yaml configs/base/
+```
+
+**不要手改这些文件**——要调参请在实验的 `config.yaml` 里覆盖。

starforge_cli/scaffold/project/configs/base/distillation_math.yaml

@@ -0,0 +1,284 @@
+# Distillation Algorithm Configuration
+distillation:
+    num_prompts_per_step: 128
+    num_generations_per_prompt: 1
+    max_rollout_turns: 1 # for multi-turn rollouts. Math Environments just have 1 turn (answering the question)
+    max_num_steps: 1000
+    max_num_epochs: 10
+    val_batch_size: 64
+    val_period: 20
+    val_at_start: false
+    val_at_end: false
+    max_val_samples: 512
+    topk_logits_k: 64
+    seed: 42
+
+loss_fn:
+    kl_type: "mixed" # forward, reverse, mixed
+    mixed_kl_weight: 0.5 # when kl_type is "mixed", this is the weight of the forward KL
+    zero_outside_topk: false # zero out the teacher logits outside the top k when calculate forward KL loss
+
+checkpointing:
+    enabled: true
+    checkpoint_dir: "checkpoints/distillation-${policy.model_name}"
+    metric_name: "val:accuracy" # one of "val:" or "train:" followed by the metric name
+    higher_is_better: true
+    keep_top_k: 3
+    save_period: 10
+    checkpoint_must_save_by: null
+    model_save_format: "safetensors"
+    save_consolidated: false
+    save_optimizer: true
+
+policy: &POLICY_BASE
+    model_name: "Qwen/Qwen3-1.7B-Base"
+    tokenizer:
+        name: ${..model_name} ## specify if you'd like to use a tokenizer different from the model's default
+        chat_template_kwargs: null # can be used to pass kwargs to the chat template, e.g., enable_thinking=true
+    train_global_batch_size: 64
+    train_micro_batch_size: 1
+    generation_batch_size: 64
+    logprob_batch_size: 1
+    max_total_sequence_length: 8192
+    precision: "bfloat16"
+    logprob_chunk_size: null
+
+    offload_optimizer_for_logprob: false
+
+    dtensor_cfg: &DTENSOR_BASE
+        enabled: true
+        _v2: true
+        cpu_offload: False
+        sequence_parallel: false
+        activation_checkpointing: true
+        tensor_parallel_size: 2
+        context_parallel_size: 2
+        custom_parallel_plan: null
+
+    dynamic_batching:
+        enabled: true
+        train_mb_tokens: ${mul:${..max_total_sequence_length}, ${..train_micro_batch_size}}
+        logprob_mb_tokens: ${mul:${..max_total_sequence_length}, ${..logprob_batch_size}}
+        sequence_length_round: 64
+
+    sequence_packing:
+        enabled: false
+        train_mb_tokens: ${mul:${..max_total_sequence_length}, ${..train_micro_batch_size}}
+        logprob_mb_tokens: ${mul:${..max_total_sequence_length}, ${..logprob_batch_size}}
+        algorithm: "modified_first_fit_decreasing"
+        sequence_length_round: 64
+
+    max_grad_norm: 1.0
+    # makes the training sequence length divisible by the tensor parallel size
+    # this is useful for sequence parallel training
+    # must be divisible by 2*cp
+    make_sequence_length_divisible_by: ${mul:${mul:${.dtensor_cfg.tensor_parallel_size}, ${.dtensor_cfg.context_parallel_size}}, 2}
+    optimizer:
+        name: "torch.optim.AdamW"
+        kwargs:
+            lr: 2.0e-5
+            weight_decay: 0.01
+            betas: [0.9, 0.999]
+            eps: 1e-8
+            # when using Dtensor, we need to set foreach
+            # and fused to False
+            foreach: False
+            fused: False
+
+    megatron_cfg: &MEGATRON_BASE
+        enabled: false
+        force_reconvert_from_hf: False # Set to True to force reconvert of the model from Hugging Face
+        empty_unused_memory_level: 0
+        activation_checkpointing: false
+        # recompute_granularity controls activation checkpointing depth.
+        # "full": recompute all activations (default, max memory savings).
+        # "selective": recompute only specific modules (see recompute_modules).
+        # Only takes effect when activation_checkpointing: true.
+        recompute_granularity: "full"
+        # Modules to selectively recompute when recompute_granularity="selective".
+        # MCore options: ["core_attn", "moe_act", "layernorm", "mla_up_proj", "mlp", "moe", "shared_experts"].
+        # Null defaults to ["core_attn"]. Full list and per-module constraints:
+        # https://github.com/NVIDIA/Megatron-LM/blob/d30c3ae5469fe3f6a64d4fd2e63b6e7f7844ea81/megatron/core/transformer/transformer_config.py#L483
+        # Use ["moe"] for MoE models to recompute expert activations only.
+        recompute_modules: null
+        tensor_model_parallel_size: 2
+        expert_tensor_parallel_size: 1
+        expert_model_parallel_size: 1
+        pipeline_model_parallel_size: 2
+        num_layers_in_first_pipeline_stage: null
+        num_layers_in_last_pipeline_stage: null
+        context_parallel_size: 2
+        pipeline_dtype: ${policy.precision}
+        sequence_parallel: false
+        freeze_moe_router: true
+        moe_router_dtype: "fp64"
+        moe_router_load_balancing_type: "none" # "seq_aux_loss" causes logprob error divergence for grpo
+        moe_router_bias_update_rate: 0.0 # by default, disable bias updates for grpo
+        moe_permute_fusion: true
+        #gives ~20% training perf speedup with sequence packing 
+        apply_rope_fusion: True
+        bias_activation_fusion: True
+        defer_fp32_logits: False
+        moe_per_layer_logging: False
+        moe_enable_deepep: false
+        moe_token_dispatcher_type: "alltoall"
+        moe_shared_expert_overlap: false
+        gradient_accumulation_fusion: false
+        use_fused_weighted_squared_relu: false
+        optimizer:
+            optimizer: "adam"
+            lr: 2.00001e-5
+            min_lr: 2.0e-5
+            weight_decay: 0.01
+            bf16: true
+            fp16: false
+            params_dtype: "float32"
+
+            #adam
+            adam_beta1: 0.9
+            adam_beta2: 0.999
+            adam_eps: 1e-8
+
+            #sgd
+            sgd_momentum: 0.9
+
+            #distributed optimizer
+            use_distributed_optimizer: true
+            use_precision_aware_optimizer: true
+
+            # optimizer cpu offload
+            optimizer_cpu_offload: false
+            optimizer_offload_fraction: 0.0
+
+            clip_grad: ${policy.max_grad_norm}
+
+        scheduler:
+            start_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay}
+            end_weight_decay: ${policy.megatron_cfg.optimizer.weight_decay}
+            weight_decay_incr_style: "constant"
+            lr_decay_style: "constant"
+            lr_decay_iters: 1000
+            lr_warmup_iters: 10
+            lr_warmup_init: 2.0e-6
+
+        distributed_data_parallel_config:
+            grad_reduce_in_fp32: false
+            overlap_grad_reduce: true
+            overlap_param_gather: true
+            use_custom_fsdp: false
+            data_parallel_sharding_strategy: "optim_grads_params"
+
+        fp8_cfg:
+            enabled: false
+            fp8: "e4m3"
+            fp8_recipe: "blockwise"
+            fp8_param: false
+
+    scheduler:
+        - name: "torch.optim.lr_scheduler.LinearLR"
+          kwargs:
+              start_factor: 0.1
+              end_factor: 1.0
+              total_iters: 10
+        - name: "torch.optim.lr_scheduler.ConstantLR"
+          kwargs:
+              factor: 1.0
+              total_iters: 10000000000
+        - milestones: [10]
+
+    generation:
+        # Port range for vLLM HTTP servers, kept below the OS ephemeral range.
+        # See ray.sub for the full port layout.
+        port_range_low: 11001
+        port_range_high: 15000
+        backend: "vllm"
+        max_new_tokens: ${..max_total_sequence_length} # refer to local policy/teacher config
+        temperature: 1.0
+        top_p: 1.0
+        top_k: null
+        stop_token_ids: null
+        stop_strings: null
+        vllm_cfg:
+            async_engine: false
+            precision: ${...precision}
+            kv_cache_dtype: "auto"
+            tensor_parallel_size: 1
+            pipeline_parallel_size: 1
+            expert_parallel_size: 1  # When EP > 1, EP must be a multiple of TP since vLLM's EP = DP * TP
+            gpu_memory_utilization: 0.6
+            max_model_len: ${...max_total_sequence_length} # refer to local policy/teacher config
+            enforce_eager: False
+            use_deep_gemm: False
+            num_last_layers_in_bf16: 0
+            num_first_layers_in_bf16: 0
+            distributed_executor_backend: null
+        vllm_kwargs: {}
+
+        colocated:
+            # true: generation shares training GPUs
+            # false: uses dedicated generation resources
+            enabled: true
+            # only relevant when enabled is false
+            resources:
+                gpus_per_node: null # Decides num gpus to be dedicated to generation when there is one node in the cluster i.e cluster.num_nodes == 1
+                num_nodes: null # Decides number of nodes to be dedicated to generation
+
+
+teacher:
+    <<: *POLICY_BASE
+    model_name: "Qwen/Qwen3-4B"
+    dtensor_cfg:
+        <<: *DTENSOR_BASE
+        context_parallel_size: 2 
+        tensor_parallel_size: 4
+
+data:
+    max_input_seq_length: ${policy.max_total_sequence_length} # upper bound, real truncation occurs at vllm.max_model_len
+    shuffle: true
+
+    # dataset
+    train:
+        dataset_name: DeepScaler
+    validation:
+        dataset_name: AIME2024
+        repeat: 16
+    # default settings for all datasets
+    default:
+        prompt_file: "examples/prompts/cot.txt"
+        system_prompt_file: null
+        env_name: "math"
+
+env:
+    math:
+        num_workers: 8
+
+logger:
+    log_dir: "logs/distillation"
+    num_val_samples_to_print: 5
+    wandb_enabled: true
+    tensorboard_enabled: true
+    mlflow_enabled: false
+    swanlab_enabled: false
+    monitor_gpus: true
+    wandb:
+        project: "nemo-distillation"
+        name: "distillation-${data.train.dataset_name}-${teacher.model_name}-${policy.model_name}-${loss_fn.kl_type}-${distillation.topk_logits_k}"
+    swanlab:
+        project: "nemo-distillation"
+        name: "distillation-${data.train.dataset_name}-${teacher.model_name}-${policy.model_name}-${loss_fn.kl_type}-${distillation.topk_logits_k}"
+    tensorboard:
+        log_dir: "tb_logs-distillation-${data.train.dataset_name}"
+    mlflow:
+        experiment_name: "distillation-dev"
+        run_name: "distillation-math-cl-logger"
+        tracking_uri: "http://localhost:5000"
+    gpu_monitoring:
+        collection_interval: 10
+        flush_interval: 10
+
+cluster:
+    gpus_per_node: 8
+    num_nodes: 1
+    master_port_range_low: 25000
+    master_port_range_high: 28000
+    segment_size: null  # Nodes per NVLink domain segment for topology-aware alignment; null to disable

starforge_cli/scaffold/project/configs/base/grpo_lora.yaml

@@ -0,0 +1,30 @@
+# LoRA / PEFT overlay（Megatron 后端）——实测在低显存机型上让 9B GRPO 能跑起来。
+# 用法：在实验 defaults 里放在 grpo_megatron.yaml「之后」。删掉这一行即回到全参数微调。
+#   defaults:
+#     - ../../configs/base/grpo_math_1B.yaml
+#     - ../../configs/models/qwen3.5-9b.yaml
+#     - ../../configs/base/grpo_megatron.yaml   # Megatron 后端 + 低显存调优
+#     - ../../configs/base/grpo_lora.yaml       # ← LoRA（删此行即全参数）
+#
+# 要点（来自跑通的 grpo_math_9B_megatron）：
+#   - LoRA 学习率比全参数高 1~2 个量级（这里 1e-4；4B 实测用 2e-4）
+#   - weight_decay=0，cosine 衰减，warmup 比全参数多
+#   - target_modules 用 Megatron 的命名（linear_qkv/linear_proj/linear_fc1/linear_fc2）
+policy:
+  megatron_cfg:
+    peft:
+      enabled: true
+      target_modules: ["linear_qkv", "linear_proj", "linear_fc1", "linear_fc2"]
+      exclude_modules: []
+      dim: 8
+      alpha: 16
+      dropout: 0.0
+      lora_dtype: "bfloat16"
+    optimizer:
+      lr: 1.0e-4
+      min_lr: 1.0e-5
+      weight_decay: 0.0
+    scheduler:
+      lr_decay_style: "cosine"
+      lr_warmup_iters: 50
+      lr_decay_iters: 1000