@researai/deepscientist 1.5.17 → 1.6.0

package/AISB/catalog/aisb.t3.021_tokenrecycling.zh.yaml

@@ -0,0 +1,129 @@
+schema_version: 1
+id: aisb.t3.021_tokenrecycling
+name: '变废为宝：利用 Token 循环加速大语言模型推理'
+version: 0.1.0
+one_line: 评估 Token 循环，这是一种无需训练的字验证解码方法，通过一个 <2 MB 的邻接矩阵重用候选 token，在 SpecBench 和 MBPP 上实现约 2 倍的 LLM 推理加速。
+task_description: '此打包基准测试用于评估 Token 循环（TR），这是一种即插即用、无需训练的字验证解码方法，用于加速 LLM 推理。TR 将每个解码步骤中的 top-k 候选 token 存储在一个词汇表大小的邻接矩阵中（<2 MB），然后通过类 BFS 检索构建草稿树，并使用树注意力机制进行验证。评估流程运行 eval.sh 脚本，对 Vicuna 7B/13B/33B（在 SpecBench 上）和 Code Llama 7B/13B/34B（在 MBPP 上）进行迭代测试，测量平均接受 token 数（MAT）、token/秒以及相对于 HuggingFace 自回归解码的加速比。模型需要预先下载到项目根目录的 ../models/ 目录下。该代码库基于 SpecBench 评估框架，使用贪心解码（temperature=0，batch size=1）。论文报告在 7B 模型上实现约 2 倍加速，比之前的无训练方法提升 30% 以上，比 Medusa（需要训练）提升 25%。所有原始实验均使用单块 A100-80GB GPU，搭配 PyTorch 2.3 和 CUDA 12.2。
+
+  '
+capability_tags:
+- research_code_optimization
+- large_language_models
+- speculative_decoding
+- inference_acceleration
+- systems_efficiency
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 6-24h
+difficulty: hard
+data_access: public
+primary_outputs:
+- mean_accepted_tokens
+- spec_bench_speedup
+- throughput_report
+launch_profiles:
+- id: quick_check
+  label: 快速检查
+  description: '使用单个模型（例如 vicuna-7b-v1.3）在 spec_bench 上运行 eval.sh。需要在 ../models/ 中放置模型权重。在 A100-80GB 上预计需要 1-3 小时。
+
+    '
+- id: specbench_eval
+  label: 完整 SpecBench + MBPP 评估
+  description: '在 SpecBench 上对所有三种 Vicuna 规模，在 MBPP 上对所有三种 Code Llama 规模，运行完整的 eval.sh 循环。根据评估的模型规模，在单块 A100-80GB 上预计需要 6-24 小时。
+
+    '
+dataset_download:
+  primary_method: bundled_plus_models
+  sources:
+  - kind: bundled
+    url: null
+    access: public
+    note: SpecBench 问题文件和 MBPP 数据已打包在 data/ 目录中。
+  - kind: huggingface
+    url: https://huggingface.co/lmsys/vicuna-7b-v1.3
+    access: public
+    note: Vicuna 7B/13B/33B 权重需单独下载到 ../models/。
+  - kind: huggingface
+    url: https://huggingface.co/codellama/CodeLlama-7b-hf
+    access: public
+    note: Code Llama 7B/13B/34B 权重需单独下载到 ../models/。
+  notes:
+  - 模型权重总计 60-130 GB，取决于使用的规模。
+  - eval.sh 期望模型位于 ../models/vicuna-7b-v1.3、../models/vicuna-13b-v1.3 等位置。
+credential_requirements:
+  mode: none
+  items: []
+  notes:
+  - 某些门控模型变体可能需要 HuggingFace 登录，但 Vicuna 和 Code Llama 权重是公开可用的。
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 32
+    disk_gb: 100
+    gpu_count: 1
+    gpu_vram_gb: 24
+  recommended:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 200
+    gpu_count: 1
+    gpu_vram_gb: 80
+environment:
+  python: '3.10'
+  cuda: '12.2'
+  pytorch: '2.3'
+  key_packages:
+  - transformers==4.37.1
+  - fschat
+  - shortuuid
+  notes:
+  - 论文实验使用 PyTorch 2.3、CUDA 12.2、单块 A100-80GB、128 核 CPU。
+  - 打包的 requirements.txt 继承自 SpecBench；通过 pip install -r requirements.txt 安装。
+  - 现有的 YAML 列出了 PyTorch 2.1.1，但论文和代码参考的是 PyTorch 2.3。
+risk_flags:
+- model_download_required
+- gpu_intensive
+- currently_vicuna_only
+risk_notes:
+- 必须提前从外部下载模型（Vicuna、Code Llama）才能进行评估；总计约 60-130 GB。
+- 最小的 24 GB 显存只能以 float16 运行 7B 模型；33B/34B 模型需要 ≥48 GB 显存。
+- 代码库目前仅适配 LLaMA/Vicuna 架构（modeling_llama_kv.py）；适配更新模型已列为待办事项。
+- 打包过程中未执行基准测试；指标值尚未验证。
+recommended_when: '当您需要对 7B+ LLM 进行推理加速评估任务，且无需草稿模型训练、无需大型检索数据存储、额外内存占用极小（<2 MB）时，可使用此基准测试。非常适合在受控的贪心解码、batch size 为 1 的条件下比较无训练的字验证解码方法。
+
+  '
+not_recommended_when: '如果您没有 ≥24 GB 显存的 GPU 来服务开放权重 LLM，或者需要以训练为中心的基准测试，或者需要评估 LLaMA/Vicuna 系列之外的模型（代码目前仅支持基于 LLaMA 的架构），则不建议使用此基准测试。
+
+  '
+paper:
+  title: 'Turning Trash into Treasure: Accelerating Inference of Large Language Models
+    with Token Recycling'
+  venue: ACL 2025 Oral
+  year: 2025
+  url: https://arxiv.org/abs/2408.08696
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.021_tokenrecycling.zip
+  archive_type: zip
+  local_dir_name: paper-21-TokenRecycling
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.021_tokenrecycling.zip
+  sha256: d38519836e52e4c5c5e1fccd9a4befa5b9a3f20a5c8fa787941a3b2773bd1ebd
+  size_bytes: 55377
+commercial:
+  annual_fee: null
+display:
+  palette_seed: lime-slate-recycle
+  art_style: systems-diagram
+  accent_priority: high
+image_path: ../image/021_aisb.t3.021_tokenrecycling.jpg

package/AISB/catalog/aisb.t3.022_chainofreasoning.yaml

@@ -0,0 +1,204 @@
+schema_version: 1
+id: aisb.t3.022_chainofreasoning
+name: Chain-of-Reasoning (CoR)
+version: 0.1.0
+one_line: 'Train and evaluate CoR-Math-7B, a multi-paradigm mathematical reasoning
+  model that chains Natural Language, Algorithmic (Python), and Symbolic (Lean 4)
+  reasoning to solve arithmetic and theorem-proving benchmarks (GSM8K, MATH, miniF2F,
+  AMC 2023, AIME 2024).
+
+  '
+task_description: 'This benchmark covers the full Chain-of-Reasoning (CoR) pipeline:
+  (1) constructing the Multi-Paradigm Mathematical (MPM) dataset (~167k reasoning
+  paths) by extending Lean-Workbook and NuminaMath-TIR with GPT-4o-generated cross-paradigm
+  solutions and Lean 4 proof verification; (2) Progressive Paradigm Training (PPT)
+  over three stages—NLR-only, NLR+AR, NLR+AR+SR—using DeepSpeed ZeRO Stage 3 SFT on
+  a 7B base model; and (3) evaluating the resulting CoR-Math-7B on five math benchmarks
+  (GSM8K, MATH, miniF2F, AMC 2023, AIME 2024) via vLLM-based inference with Sequential
+  Multi-Paradigm Sampling (SMPS). The evaluation script (evaluation/eval_CoR.py) computes
+  accuracy, collects reasoning traces, and optionally verifies Lean 4 proofs via a
+  bundled Lean server scheduler. The dataset construction step requires an OpenAI
+  API key and calls GPT-4o; the training and eval steps are self-contained once data
+  is prepared.
+
+  '
+capability_tags:
+- research_code_optimization
+- mathematical_reasoning
+- formal_methods
+- large_language_models
+- theorem_proving
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 1d+
+difficulty: hard
+data_access: public
+primary_outputs:
+- accuracy
+- reasoning_traces
+- lean_artifacts
+launch_profiles:
+- id: quick_eval
+  label: Quick Eval
+  description: 'Run evaluation only on one benchmark (e.g. MATH or GSM8K) using a
+    pre-trained checkpoint and the bundled eval_CoR.py script with vLLM inference.
+    Skips dataset construction and training entirely. Requires a model checkpoint
+    and 1–8 GPUs.
+
+    '
+- id: full_train_eval
+  label: Full Train + Eval
+  description: 'Build the MPM dataset from Lean-Workbook and NuminaMath-TIR (requires
+    OpenAI API key for GPT-4o augmentation), run 3-stage Progressive Paradigm Training
+    with DeepSpeed ZeRO-3, then evaluate on all five benchmarks (GSM8K, MATH, miniF2F,
+    AMC 2023, AIME 2024). This is the paper-faithful route and requires multi-GPU
+    infrastructure and Lean 4/Mathlib build.
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/internlm/Lean-Workbook
+    access: public
+    note: 'lean_workbook.json — seed data for symbolic reasoning paradigm. Used in
+      utils/generate_informalproof.py to create informal proofs.
+
+      '
+  - kind: huggingface
+    url: https://huggingface.co/datasets/AI-MO/NuminaMath-TIR
+    access: public
+    note: 'train-00000-of-00001.parquet — seed data for algorithmic reasoning paradigm.
+      Must be converted to JSON for utils/generate_lean4.py.
+
+      '
+  notes:
+  - 'The MPM dataset (~167k paths from ~83k problems) is constructed by the user at
+    runtime via GPT-4o augmentation scripts; it is not directly downloadable.
+
+    '
+  - 'Raw seed datasets are modest in size (tens of GB); the synthesized MPM dataset
+    and model checkpoints will require additional storage.
+
+    '
+credential_requirements:
+  mode: optional
+  items:
+  - OpenAI API key (required only for MPM dataset construction via GPT-4o)
+  - Weights & Biases API key (optional, for training logging)
+  notes:
+  - 'The OpenAI API key is used in script/leanwb_informalproof.sh and script/numinatir_lean4.sh
+    to call GPT-4o for generating cross-paradigm reasoning paths. If you use a pre-built
+    MPM dataset or a pre-trained checkpoint, no API key is needed.
+
+    '
+  - 'WandB credentials are set in script/train_CoR.sh but can be set to offline mode.
+
+    '
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 250
+    gpu_count: 1
+    gpu_vram_gb: 48
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 500
+    gpu_count: 8
+    gpu_vram_gb: 80
+environment:
+  python: '3.10'
+  cuda: '12.1'
+  pytorch: 2.5.1
+  flash_attn: 2.7.0.post2
+  key_packages:
+  - flash-attn==2.7.0.post2
+  - deepspeed==0.15.4
+  - vllm==0.6.4.post1
+  - transformers==4.46.3
+  - accelerate
+  - datasets
+  notes:
+  - See requirements.txt for the full dependency set.
+  - 'Lean 4 must be installed separately via elan (leanprover-community.github.io).
+    Mathlib4 must be built locally with `lake build` before running theorem proving
+    evaluation or MPM dataset construction.
+
+    '
+  - 'CUDA version is not explicitly pinned in the repo; 12.1 is recommended based
+    on vLLM and flash-attn compatibility. 11.8 may also work.
+
+    '
+risk_flags:
+- external_api_dependency
+- lean_toolchain_setup
+- large_training_pipeline
+- openai_cost
+risk_notes:
+- 'MPM dataset construction calls the OpenAI API (GPT-4o) extensively across ~285k
+  samples, which can incur significant API costs. Budget several hundred dollars for
+  full construction.
+
+  '
+- 'Lean 4 and Mathlib4 must be compiled locally. The Mathlib build alone can take
+  1–2 hours and requires substantial RAM. Build failures are common on misconfigured
+  systems.
+
+  '
+- 'Training uses DeepSpeed ZeRO Stage 3 across up to 8 GPUs with 4 epochs. The training
+  script has placeholder paths that must be edited before launch.
+
+  '
+- 'The evaluation script uses vLLM with gpu_memory_utilization=0.95, which may OOM
+  on GPUs with less than 48 GB VRAM depending on batch size.
+
+  '
+recommended_when: 'Use this benchmark when you want a heavyweight, multi-stage math-reasoning
+  task that combines LLM fine-tuning, Lean 4 formal verification, Python code execution,
+  and natural language chain-of-thought. Ideal for exploring how multi-paradigm cooperation
+  improves mathematical reasoning across both arithmetic computation (GSM8K, MATH,
+  AIME) and theorem proving (miniF2F).
+
+  '
+not_recommended_when: 'Do not use this if you cannot provision multi-GPU infrastructure
+  (≥48 GB VRAM), cannot install and build Lean 4 + Mathlib4, or want to avoid OpenAI
+  API costs for dataset construction. Also not suitable if you need a quick, self-contained
+  eval-only benchmark—the full pipeline requires significant setup and compute time.
+
+  '
+paper:
+  title: 'Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language
+    Models via a Multi-Paradigm Perspective
+
+    '
+  venue: arXiv preprint
+  year: 2025
+  url: https://arxiv.org/abs/2501.11110
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.022_chainofreasoning.zip
+  archive_type: zip
+  local_dir_name: paper-22-ChainOfReasoning
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.022_chainofreasoning.zip
+  sha256: 10a4cef8d810a0806780f11b2059b970a3ac4152b614f97a1e9e47294629764b
+  size_bytes: 986391
+commercial:
+  annual_fee: null
+display:
+  palette_seed: cobalt-ivory-proof
+  art_style: formal-reasoning
+  accent_priority: high
+image_path: ../image/022_aisb.t3.022_chainofreasoning.jpg

package/AISB/catalog/aisb.t3.022_chainofreasoning.zh.yaml

@@ -0,0 +1,161 @@
+schema_version: 1
+id: aisb.t3.022_chainofreasoning
+name: Chain-of-Reasoning (CoR)
+version: 0.1.0
+one_line: '训练并评估 CoR-Math-7B，一个多范式数学推理模型，通过链式整合自然语言、算法（Python）和符号（Lean 4）推理来求解算术和定理证明基准测试（GSM8K、MATH、miniF2F、AMC 2023、AIME 2024）。
+
+  '
+task_description: '本基准测试涵盖完整的链式推理（CoR）流程：(1) 通过扩展 Lean-Workbook 和 NuminaMath-TIR，利用 GPT-4o 生成跨范式解答和 Lean 4 证明验证，构建多范式数学（MPM）数据集（约 167k 条推理路径）；(2) 在 7B 基座模型上通过 DeepSpeed ZeRO Stage 3 SFT 进行三个阶段的渐进范式训练（PPT）——仅 NLR、NLR+AR、NLR+AR+SR；(3) 使用 vLLM 推理和顺序多范式采样（SMPS）在五个数学基准测试（GSM8K、MATH、miniF2F、AMC 2023、AIME 2024）上评估最终的 CoR-Math-7B 模型。评估脚本（evaluation/eval_CoR.py）计算准确率、收集推理轨迹，并可选地通过捆绑的 Lean 服务器调度器验证 Lean 4 证明。数据集构建步骤需要 OpenAI API 密钥并调用 GPT-4o；训练和评估步骤在数据准备完成后可独立运行。
+
+  '
+capability_tags:
+- research_code_optimization
+- mathematical_reasoning
+- formal_methods
+- large_language_models
+- theorem_proving
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 1d+
+difficulty: hard
+data_access: public
+primary_outputs:
+- accuracy
+- reasoning_traces
+- lean_artifacts
+launch_profiles:
+- id: quick_eval
+  label: 快速评估
+  description: '仅在单个基准测试（如 MATH 或 GSM8K）上使用预训练检查点和捆绑的 eval_CoR.py 脚本进行 vLLM 推理评估。完全跳过数据集构建和训练步骤。需要模型检查点和 1-8 块 GPU。
+
+    '
+- id: full_train_eval
+  label: 完整训练+评估
+  description: '从 Lean-Workbook 和 NuminaMath-TIR 构建 MPM 数据集（需要 OpenAI API 密钥用于 GPT-4o 增强）、运行三阶段渐进范式训练的 DeepSpeed ZeRO-3，然后在所有五个基准测试（GSM8K、MATH、miniF2F、AMC 2023、AIME 2024）上进行评估。这是论文忠实路线，需要多 GPU 基础设施和 Lean 4/Mathlib 构建。
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/internlm/Lean-Workbook
+    access: public
+    note: 'lean_workbook.json — 符号推理范式的种子数据。在 utils/generate_informalproof.py 中用于创建非形式证明。
+
+      '
+  - kind: huggingface
+    url: https://huggingface.co/datasets/AI-MO/NuminaMath-TIR
+    access: public
+    note: 'train-00000-of-00001.parquet — 算法推理范式的种子数据。需转换为 JSON 供 utils/generate_lean4.py 使用。
+
+      '
+  notes:
+  - 'MPM 数据集（约 167k 条路径来自约 83k 道题目）由用户在运行时通过 GPT-4o 增强脚本构建，并非直接可下载。
+
+    '
+  - '原始种子数据集规模适中（数十 GB）；合成后的 MPM 数据集和模型检查点将需要额外的存储空间。
+
+    '
+credential_requirements:
+  mode: optional
+  items:
+  - OpenAI API 密钥（仅在通过 GPT-4o 构建 MPM 数据集时需要）
+  - Weights & Biases API 密钥（可选，用于训练日志记录）
+  notes:
+  - 'OpenAI API 密钥用于 script/leanwb_informalproof.sh 和 script/numinatir_lean4.sh 调用 GPT-4o 生成跨范式推理路径。如果您使用预构建的 MPM 数据集或预训练检查点，则不需要 API 密钥。
+
+    '
+  - 'WandB 凭据在 script/train_CoR.sh 中设置，但可设为离线模式。
+
+    '
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 250
+    gpu_count: 1
+    gpu_vram_gb: 48
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 500
+    gpu_count: 8
+    gpu_vram_gb: 80
+environment:
+  python: '3.10'
+  cuda: '12.1'
+  pytorch: 2.5.1
+  flash_attn: 2.7.0.post2
+  key_packages:
+  - flash-attn==2.7.0.post2
+  - deepspeed==0.15.4
+  - vllm==0.6.4.post1
+  - transformers==4.46.3
+  - accelerate
+  - datasets
+  notes:
+  - 完整的依赖项请参见 requirements.txt。
+  - 'Lean 4 必须通过 elan（leanprover-community.github.io）单独安装。在运行定理证明评估或 MPM 数据集构建之前，必须使用 `lake build` 在本地构建 Mathlib4。
+
+    '
+  - '仓库中未明确固定 CUDA 版本；根据 vLLM 和 flash-attn 的兼容性，推荐 12.1 版本。11.8 也可能可用。
+
+    '
+risk_flags:
+- external_api_dependency
+- lean_toolchain_setup
+- large_training_pipeline
+- openai_cost
+risk_notes:
+- 'MPM 数据集构建需要在约 285k 样本上大量调用 OpenAI API（GPT-4o），可能产生可观的 API 费用。完整构建需预算数百美元。
+
+  '
+- 'Lean 4 和 Mathlib4 必须在本地编译。Mathlib 构建本身可能需要 1-2 小时，并需要大量 RAM。在配置不当的系统上构建失败很常见。
+
+  '
+- '训练在最多 8 块 GPU 上使用 DeepSpeed ZeRO Stage 3 运行 4 个 epoch。训练脚本有占位符路径，启动前必须编辑。
+
+  '
+- '评估脚本使用 vLLM 且 gpu_memory_utilization=0.95，根据批次大小不同可能在显存小于 48 GB 的 GPU 上导致 OOM。
+
+  '
+recommended_when: '当您需要一个重量级、多阶段的数学推理任务，将 LLM 微调、Lean 4 形式验证、Python 代码执行和自然语言思维链相结合时使用此基准测试。非常适合探索多范式协作如何提升数学推理能力，涵盖算术计算（GSM8K、MATH、AIME）和定理证明（miniF2F）。
+
+  '
+not_recommended_when: '如果无法配置多 GPU 基础设施（≥48 GB 显存）、无法安装和构建 Lean 4 + Mathlib4，或希望避免 OpenAI API 数据集构建费用，请勿使用此基准测试。同样不适合需要快速、独立运行的纯评估基准测试——完整流程需要大量设置和计算时间。
+
+  '
+paper:
+  title: 'Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language
+    Models via a Multi-Paradigm Perspective
+
+    '
+  venue: arXiv preprint
+  year: 2025
+  url: https://arxiv.org/abs/2501.11110
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.022_chainofreasoning.zip
+  archive_type: zip
+  local_dir_name: paper-22-ChainOfReasoning
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.022_chainofreasoning.zip
+  sha256: 10a4cef8d810a0806780f11b2059b970a3ac4152b614f97a1e9e47294629764b
+  size_bytes: 986391
+commercial:
+  annual_fee: null
+display:
+  palette_seed: cobalt-ivory-proof
+  art_style: formal-reasoning
+  accent_priority: high
+image_path: ../image/022_aisb.t3.022_chainofreasoning.jpg