@researai/deepscientist 1.5.16 → 1.6.0

package/AISB/catalog/aisb.t3.019_dyscaleut.zh.yaml

@@ -0,0 +1,148 @@
+schema_version: 1
+id: aisb.t3.019_dyscaleut
+name: 面向代码奖励建模的单元测试动态缩放
+version: 0.1.0
+one_line: '将LLM生成的单元测试作为奖励信号进行动态缩放，用于最佳-of-N代码解决方案选择，在HumanEval Plus、MBPP Plus和LiveCodeBench上通过pass@1进行评估。
+
+  '
+task_description: '本基准测试复现了CodeRM流程，通过缩放单元测试奖励建模来改进代码生成。核心工作流程为：(1) 策略LLM为每个编程问题生成N个候选代码解决方案，(2) 奖励LLM（或微调后的CodeRM-8B单元测试生成器）为每个问题生成M个单元测试，(3) 在Docker沙箱内对解决方案执行单元测试，(4) 通过执行结果的方差加权多数投票选择最佳解决方案。动态缩放机制使用训练好的难度分类器为更难的问题分配更多单元测试。主要指标为pass@1（最佳-of-N准确率），通过evaluation/calculate_result.py计算。本地快照包含评估/执行代码、打包的基准数据、预计算的推理结果，以及用于沙箱代码执行的Docker镜像规格。推理和预处理步骤（从头开始使用策略/奖励LLM生成新的解决方案/单元测试）需要外部模型访问，已由打包脚本部分覆盖，但可能需要适配。可用Google Drive上的预计算output.tar.gz替代推理+执行步骤。
+
+  '
+capability_tags:
+- research_code_optimization
+- code_generation
+- reward_modeling
+- unit_test_generation
+- evaluation
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: partial
+support_level: advanced
+cost_band: high
+time_band: 1d+
+difficulty: hard
+data_access: public
+primary_outputs:
+- pass_at_1
+- scaled_unit_tests
+- reward_scores
+launch_profiles:
+- id: quick_check
+  label: 快速检查
+  description: '在打包或下载的预计算执行结果上运行evaluation/calculate_result.py，验证单一基准/模型组合的pass@1。无需GPU或Docker。
+
+    '
+- id: code_reward_eval
+  label: 代码奖励评估
+  description: '拉取Docker沙箱镜像，使用evaluation/evaluate.py对候选解决方案执行单元测试，然后通过evaluation/calculate_result.py计算pass@1。需要Docker和中等算力。使用打包的基准数据和预生成的解决方案/单元测试，无需LLM推理。
+
+    '
+- id: full_pipeline
+  label: 全流程（推理+评估）
+  description: '端到端运行：多进程LLM推理生成解决方案和单元测试、预处理/合并、基于Docker的执行，以及最终pass@1计算。需要GPU访问（用于LLM推理，CodeRM-8B或更大模型）和Docker执行环境。
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/KAKA22/CodeRM-UnitTest
+    access: public
+    note: 用于训练CodeRM-8B的60k个合成Python单元测试。
+  - kind: huggingface
+    url: https://huggingface.co/KAKA22/CodeRM-8B
+    access: public
+    note: 微调后的8B单元测试生成器模型权重。
+  - kind: google_drive
+    url: https://drive.google.com/drive/folders/1-wUvy9Ox49V5CY38TMjCr5RlLysapyyj?usp=sharing
+    access: public
+    note: 预计算的执行输出（output.tar.gz），可替代管道的第1-3步。
+
+  - kind: bundled
+    url: null
+    access: local
+    note: 基准数据（HumanEval Plus、MBPP Plus、LiveCodeBench）和预生成的推理结果包含在快照的data/benchmark/和data/result/目录下。
+  notes:
+  - 训练数据集约60k样本；模型权重约16GB；预计算输出因基准不同而异。
+  - 基准测试本身（HumanEval Plus、MBPP Plus、LiveCodeBench）是公开的。
+credential_requirements:
+  mode: none
+  items: []
+  notes:
+  - 使用打包数据和CodeRM-8B进行推理无需API密钥。
+  - 若需复现GPT-4o-mini或GPT-3.5的策略/奖励实验，则需要OpenAI API密钥。
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 150
+    gpu_count: 1
+    gpu_vram_gb: 24
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 300
+    gpu_count: 2
+    gpu_vram_gb: 48
+environment:
+  python: null
+  cuda: null
+  pytorch: null
+  flash_attn: null
+  key_packages:
+  - vllm
+  - transformers
+  - docker
+  notes:
+  - 快照包含专用Docker执行环境（kaka0605/exec_unit_test:24.12.30），用于沙箱化大规模代码执行生成的单元测试。
+  - 沙箱依赖见docker_source/Dockerfile和docker_source/requirements.txt。
+  - 主机端依赖集见打包的README和requirements。
+  - 推理使用多进程Python（inference/inference_mp.py），可能需要vLLM或类似工具实现高效服务。
+risk_flags:
+- docker_required
+- partial_snapshot
+- external_model_for_full_replication
+- code_execution_sandbox
+risk_notes:
+  - 在运行evaluation/evaluate.py之前必须本地拉取或构建Docker沙箱镜像（kaka0605/exec_unit_test:24.12.30）。没有Docker时，只能在预计算输出上运行最后的calculate_result.py步骤。
+  - 预处理脚本（preprocess/）存在，但完整推理管道需要服务策略LLM和奖励LLM，快照中未完全自动化。
+  - exec_main.py通过signal.SIGALRM超时机制执行任意生成的Python代码；应仅在提供的Docker沙箱或等效隔离环境中运行。
+  - 打包过程中未执行基准测试；指标值尚未验证。
+recommended_when: 当您想研究缩放LLM生成的单元测试如何改进代码奖励信号质量和最佳-of-N代码选择时，或当您需要一个在评估循环中包含真实单元测试执行的代码生成任务时使用此基准。也可用于评估轻量级单元测试生成器与更大教师模型的性能。
+not_recommended_when: 当您无法提供基于Docker的容器化代码执行时不要使用；当您需要一个没有代码执行的纯文本奖励模型基准时不要使用；或者当您缺乏LLM推理的GPU资源且仅需快速指标检查时不要使用（改用预计算输出的quick_check配置文件）。
+paper:
+  title: Dynamic Scaling of Unit Tests for Code Reward Modeling
+  authors:
+  - Zeyao Ma
+  - Xiaokang Zhang
+  - Jing Zhang
+  - Jifan Yu
+  - Sijia Luo
+  - Jie Tang
+  venue: ACL 2025
+  year: 2025
+  url: https://arxiv.org/abs/2501.01054
+  homepage: https://code-reward-model.github.io/
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.019_dyscaleut.zip
+  archive_type: zip
+  local_dir_name: paper-19-DyScaleUT
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.019_dyscaleut.zip
+  sha256: 2eee5573353ade5e13c254f7372a3294b71459ee7c668205f27f2852347c141f
+  size_bytes: 60766
+commercial:
+  annual_fee: null
+display:
+  palette_seed: olive-ink-runtime
+  art_style: code-lab
+  accent_priority: high
+image_path: ../image/019_aisb.t3.019_dyscaleut.jpg

package/AISB/catalog/aisb.t3.020_aristotle.yaml

@@ -0,0 +1,173 @@
+schema_version: 1
+id: aisb.t3.020_aristotle
+name: 'Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve
+  Framework'
+version: 0.1.0
+one_line: 'Multi-stage LLM-driven logical reasoning benchmark using symbolic decomposition,
+  dual-path proof-by-contradiction search, and resolution over ProntoQA, ProofWriter,
+  and LogicNLI datasets.
+
+  '
+task_description: 'This benchmark evaluates and optimizes the Aristotle framework,
+  a logic-complete reasoning pipeline for LLMs. The workflow has four stages: (1)
+  translate_decompose.py converts natural-language premises into Prolog-style symbolic
+  expressions and decomposes them into Conjunctive Normal Form; (2) negate.py initialises
+  dual search paths by negating the target conjecture; (3) search_resolve.py runs
+  proof-by-contradiction search and clause resolution twice (once with negation=True,
+  once with negation=False), producing two result files per dataset; (4) evaluate.py
+  aggregates dual-path answers using the paper''s Eq. (1) and computes accuracy against
+  gold labels. All LLM calls go through an OpenAI-compatible API (GPT-4, GPT-4o, Claude,
+  or LLaMA via base_url). Three bundled datasets of increasing difficulty are supported:
+  ProntoQA (basic deductive), ProofWriter (and/or structures), and LogicNLI (either/or,
+  biconditional). The primary metric is accuracy, code-backed via evaluate.py. No
+  GPU is required; the main cost driver is API token usage.
+
+  '
+capability_tags:
+- research_code_optimization
+- logical_reasoning
+- llm_tooling
+- evaluation
+- symbolic_reasoning
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+cost_band: low
+time_band: 1-2h
+difficulty: medium
+data_access: public
+snapshot_status: runnable
+support_level: turnkey
+primary_outputs:
+- accuracy
+- reasoning_traces
+- evaluation_report
+launch_profiles:
+- id: quick_check
+  label: Quick Check
+  description: 'Run the full four-stage pipeline (translate_decompose → negate → search_resolve
+    × 2 → evaluate) on a single dataset (e.g., ProntoQA dev split) to verify end-to-end
+    connectivity and API access.
+
+    '
+- id: benchmark_eval
+  label: Full Benchmark Evaluation
+  description: 'Run the Aristotle pipeline on all three datasets (ProntoQA, ProofWriter,
+    LogicNLI) with the target model (GPT-4 or GPT-4o) and collect accuracy scores
+    for comparison with paper baselines.
+
+    '
+- id: single_dataset
+  label: Single Dataset Run
+  description: 'Run on one selected dataset to iterate on framework improvements before
+    committing to the full suite.
+
+    '
+dataset_download:
+  primary_method: bundled
+  sources:
+  - kind: local
+    url: ./data
+    access: public
+    note: 'Data for ProntoQA, ProofWriter, and LogicNLI is expected under ./data/{dataset_name}/.
+      Prompt templates are under ./prompts/{dataset_name}/. Some data-error corrections
+      are stored in ./data_errors/LogicNLI/.
+
+      '
+  notes:
+  - Dataset files appear to be bundled in the snapshot under ./data; verify completeness
+    after download.
+  - Total data footprint is small (well under 1 GB).
+credential_requirements:
+  mode: api_key
+  items:
+  - OpenAI API key (or compatible endpoint key for GPT-4/GPT-4o/Claude/LLaMA serving)
+  notes:
+  - The --api_key flag is required by translate_decompose.py and search_resolve.py.
+  - An optional --base_url flag allows pointing to non-OpenAI endpoints.
+  - API costs depend on dataset size and model choice; GPT-4 is significantly more
+    expensive per token than GPT-4o.
+resources:
+  minimum:
+    cpu_cores: 4
+    ram_gb: 8
+    disk_gb: 10
+    gpu_count: 0
+    gpu_vram_gb: 0
+  recommended:
+    cpu_cores: 8
+    ram_gb: 16
+    disk_gb: 20
+    gpu_count: 0
+    gpu_vram_gb: 0
+environment:
+  python: '3.10'
+  cuda: null
+  pytorch: null
+  flash_attn: null
+  key_packages:
+  - openai==0.27.9
+  notes:
+  - CPU-only execution; no GPU required.
+  - All heavy computation is offloaded to the LLM API; local compute handles I/O and
+    symbolic search routing.
+  - negate.py is pure Python with no API calls.
+  - See requirements.txt for the full dependency set.
+  - The openai package version (0.27.9) uses the legacy API interface; verify compatibility
+    if using a newer SDK.
+risk_flags:
+- api_dependency
+- api_cost_variable
+- legacy_sdk_version
+risk_notes:
+- All translation, decomposition, search, and resolution stages except negate.py require
+  live API access; the benchmark cannot run offline.
+- API costs scale with dataset size and number of search iterations (controlled by
+  --search_round and --batch_num).
+- openai==0.27.9 is a legacy SDK; newer OpenAI endpoints may require migration to
+  openai>=1.0.
+- No benchmark execution was performed during the packaging pass; metric values are
+  not yet validated.
+- Concurrent batch execution via threading may hit API rate limits depending on the
+  provider.
+recommended_when: 'Use this benchmark when you want a structured logical-reasoning
+  evaluation driven by API-backed decomposition, symbolic search routing, and resolution
+  stages across datasets of increasing logical complexity. Good fit for studying how
+  LLMs handle formal symbolic reasoning, proof by contradiction, and CNF-based clause
+  resolution without requiring local GPU resources.
+
+  '
+not_recommended_when: 'Do not use this if you need a fully offline or GPU-training-centric
+  benchmark. Not suitable if you cannot provision an OpenAI-compatible API key or
+  if you need to avoid per-query API costs. Also not ideal if you require deterministic
+  reproducibility, as LLM API outputs are stochastic.
+
+  '
+paper:
+  title: 'Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve
+    Framework'
+  venue: ACL 2025
+  year: 2025
+  url: https://arxiv.org/abs/2412.16953
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.020_aristotle.zip
+  archive_type: zip
+  local_dir_name: paper-20-Aristotle
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.020_aristotle.zip
+  sha256: f6c04512a751c42a8cd33286a62c46b34019cf91eac8d1245cead4de475b84e2
+  size_bytes: 695245
+commercial:
+  annual_fee: null
+display:
+  palette_seed: parchment-navy-logic
+  art_style: structured-proof
+  accent_priority: medium
+image_path: ../image/020_aisb.t3.020_aristotle.jpg

package/AISB/catalog/aisb.t3.020_aristotle.zh.yaml

@@ -0,0 +1,119 @@
+schema_version: 1
+id: aisb.t3.020_aristotle
+name: 'Aristotle：使用逻辑完备的分解-搜索-解决框架掌握逻辑推理'
+version: 0.1.0
+one_line: '多阶段LLM驱动的逻辑推理基准测试，采用符号分解、双路径反证法搜索和针对ProntoQA、ProofWriter及LogicNLI数据集的归结技术。'
+task_description: '本基准测试用于评估和优化Aristotle框架——一个面向LLM的逻辑完备推理流程。该工作流包含四个阶段：(1) translate_decompose.py将自然语言前提转换为Prolog风格的符号表达式，并将其分解为合取范式；(2) negate.py通过对目标猜想取反来初始化双搜索路径；(3) search_resolve.py执行反证法搜索和子句归结，运行两次（一次negation=True，一次negation=False），为每个数据集生成两个结果文件；(4) evaluate.py使用论文中的公式(1)聚合双路径答案，并计算相对于黄金标签的准确率。所有LLM调用都通过OpenAI兼容API进行（GPT-4、GPT-4o、Claude或LLaMA，通过base_url配置）。支持三个难度递增的捆绑数据集：ProntoQA（基础演绎推理）、ProofWriter（与或结构）和LogicNLI（异或、双条件）。主要评估指标是准确率，通过evaluate.py计算。无需GPU，主要成本驱动因素是API token使用量。'
+capability_tags:
+- research_code_optimization
+- logical_reasoning
+- llm_tooling
+- evaluation
+- symbolic_reasoning
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+cost_band: low
+time_band: 1-2h
+difficulty: medium
+data_access: public
+snapshot_status: runnable
+support_level: turnkey
+primary_outputs:
+- accuracy
+- reasoning_traces
+- evaluation_report
+launch_profiles:
+- id: quick_check
+  label: 快速检查
+  description: '在单个数据集（如ProntoQA开发集）上运行完整的四阶段流程（translate_decompose → negate → search_resolve × 2 → evaluate），以验证端到端连通性和API访问。'
+- id: benchmark_eval
+  label: 完整基准测试评估
+  description: '使用目标模型（GPT-4或GPT-4o）在所有三个数据集（ProntoQA、ProofWriter、LogicNLI）上运行Aristotle流程，并收集准确率分数以与论文基准进行对比。'
+- id: single_dataset
+  label: 单数据集运行
+  description: '在选定的单个数据集上运行，以便在提交完整测试套件之前迭代框架改进。'
+dataset_download:
+  primary_method: bundled
+  sources:
+  - kind: local
+    url: ./data
+    access: public
+    note: 'ProntoQA、ProofWriter和LogicNLI的数据应位于./data/{dataset_name}/目录下。提示模板位于./prompts/{dataset_name}/目录下。部分数据错误修正存储在./data_errors/LogicNLI/中。'
+  notes:
+  - 数据集文件似乎以捆绑形式包含在快照的./data目录下；下载后请验证完整性。
+  - 总体数据占用很小（远低于1 GB）。
+credential_requirements:
+  mode: api_key
+  items:
+  - OpenAI API密钥（或用于GPT-4/GPT-4o/Claude/LLaMA服务的兼容端点密钥）
+  notes:
+  - translate_decompose.py和search_resolve.py需要--api_key参数。
+  - 可选的--base_url参数允许指向非OpenAI端点。
+  - API成本取决于数据集大小和模型选择；GPT-4每个token的成本显著高于GPT-4o。
+resources:
+  minimum:
+    cpu_cores: 4
+    ram_gb: 8
+    disk_gb: 10
+    gpu_count: 0
+    gpu_vram_gb: 0
+  recommended:
+    cpu_cores: 8
+    ram_gb: 16
+    disk_gb: 20
+    gpu_count: 0
+    gpu_vram_gb: 0
+environment:
+  python: '3.10'
+  cuda: null
+  pytorch: null
+  flash_attn: null
+  key_packages:
+  - openai==0.27.9
+  notes:
+  - 仅CPU执行；无需GPU。
+  - 所有重型计算都卸载到LLM API；本地计算负责I/O和符号搜索路由。
+  - negate.py是纯Python，无需API调用。
+  - 完整依赖项请参见requirements.txt。
+  - openai包版本（0.27.9）使用旧版API接口；如使用新版SDK请验证兼容性。
+risk_flags:
+- api_dependency
+- api_cost_variable
+- legacy_sdk_version
+risk_notes:
+  - 除negate.py外，所有翻译、分解、搜索和归结阶段都需要实时API访问；基准测试无法离线运行。
+  - API成本随数据集大小和搜索迭代次数（由--search_round和--batch_num控制）而增加。
+  - openai==0.27.9是旧版SDK；较新的OpenAI端点可能需要迁移到openai>=1.0。
+  - 打包过程中未执行基准测试；尚未验证指标值。
+  - 通过线程进行的并发批处理执行可能会触及提供商的API速率限制。
+recommended_when: '当您需要进行由API支持的分解、符号搜索路由和归结阶段驱动的结构化逻辑推理评估，且覆盖逻辑复杂度递增的数据集时使用此基准测试。非常适合研究LLM如何处理形式符号推理、反证法证明和基于CNF的子句归结，同时无需本地GPU资源。'
+not_recommended_when: '如果需要完全离线或以GPU训练为中心的基准测试，请勿使用。如果无法配置OpenAI兼容的API密钥或需要避免按查询计费的API成本，也不宜使用。此外，如果需要确定性的可复现性，此基准测试也不理想，因为LLM API输出具有随机性。'
+paper:
+  title: 'Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve
+    Framework'
+  venue: ACL 2025
+  year: 2025
+  url: https://arxiv.org/abs/2412.16953
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.020_aristotle.zip
+  archive_type: zip
+  local_dir_name: paper-20-Aristotle
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.020_aristotle.zip
+  sha256: f6c04512a751c42a8cd33286a62c46b34019cf91eac8d1245cead4de475b84e2
+  size_bytes: 695245
+commercial:
+  annual_fee: null
+display:
+  palette_seed: parchment-navy-logic
+  art_style: structured-proof
+  accent_priority: medium
+image_path: ../image/020_aisb.t3.020_aristotle.jpg

package/AISB/catalog/aisb.t3.021_tokenrecycling.yaml

@@ -0,0 +1,160 @@
+schema_version: 1
+id: aisb.t3.021_tokenrecycling
+name: 'Turning Trash into Treasure: Accelerating Inference of Large Language Models
+  with Token Recycling'
+version: 0.1.0
+one_line: Evaluate Token Recycling, a training-free speculative decoding method that
+  reuses candidate tokens via a <2 MB adjacency matrix to achieve ~2× LLM inference
+  speedup on SpecBench and MBPP.
+task_description: 'This packaged benchmark evaluates Token Recycling (TR), a plug-and-play,
+  training-free speculative decoding method for accelerating LLM inference. TR stores
+  top-k candidate tokens from each decoding step in a vocabulary-sized adjacency matrix
+  (<2 MB), then constructs draft trees via BFS-like retrieval and verifies them with
+  tree attention. The evaluation route runs eval.sh, which iterates over Vicuna 7B/13B/33B
+  (for SpecBench) and Code Llama 7B/13B/34B (for MBPP), measuring Mean Accepted Tokens
+  (MAT), tokens/second, and speedup ratio relative to HuggingFace autoregressive decoding.
+  Models must be pre-downloaded to ../models/ relative to the project root. The codebase
+  is built on the SpecBench evaluation harness and uses greedy decoding (temperature=0,
+  batch size=1). Paper reports ~2× speedup on 7B models, 30%+ over prior train-free
+  methods, and 25% over Medusa (which requires training). All original experiments
+  used a single A100-80GB GPU with PyTorch 2.3 and CUDA 12.2.
+
+  '
+capability_tags:
+- research_code_optimization
+- large_language_models
+- speculative_decoding
+- inference_acceleration
+- systems_efficiency
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 6-24h
+difficulty: hard
+data_access: public
+primary_outputs:
+- mean_accepted_tokens
+- spec_bench_speedup
+- throughput_report
+launch_profiles:
+- id: quick_check
+  label: Quick Check
+  description: 'Run eval.sh with a single model (e.g., vicuna-7b-v1.3) on spec_bench
+    only. Requires model weights in ../models/. Expect 1-3 hours on an A100-80GB.
+
+    '
+- id: specbench_eval
+  label: Full SpecBench + MBPP Eval
+  description: 'Run the complete eval.sh loop over all three Vicuna sizes on SpecBench
+    and all three Code Llama sizes on MBPP. Expect 6-24 hours on a single A100-80GB
+    depending on model sizes evaluated.
+
+    '
+dataset_download:
+  primary_method: bundled_plus_models
+  sources:
+  - kind: bundled
+    url: null
+    access: public
+    note: SpecBench question files and MBPP data are bundled in the data/ directory.
+  - kind: huggingface
+    url: https://huggingface.co/lmsys/vicuna-7b-v1.3
+    access: public
+    note: Vicuna 7B/13B/33B weights must be downloaded separately to ../models/.
+  - kind: huggingface
+    url: https://huggingface.co/codellama/CodeLlama-7b-hf
+    access: public
+    note: Code Llama 7B/13B/34B weights must be downloaded separately to ../models/.
+  notes:
+  - Model weights total 60-130 GB depending on which sizes are used.
+  - eval.sh expects models at ../models/vicuna-7b-v1.3, ../models/vicuna-13b-v1.3,
+    etc.
+credential_requirements:
+  mode: none
+  items: []
+  notes:
+  - HuggingFace login may be needed for some gated model variants, but Vicuna and
+    Code Llama weights are publicly available.
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 32
+    disk_gb: 100
+    gpu_count: 1
+    gpu_vram_gb: 24
+  recommended:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 200
+    gpu_count: 1
+    gpu_vram_gb: 80
+environment:
+  python: '3.10'
+  cuda: '12.2'
+  pytorch: '2.3'
+  key_packages:
+  - transformers==4.37.1
+  - fschat
+  - shortuuid
+  notes:
+  - Paper experiments used PyTorch 2.3, CUDA 12.2, single A100-80GB, 128 CPUs.
+  - The bundled requirements.txt inherits from SpecBench; install via pip install
+    -r requirements.txt.
+  - The existing YAML listed pytorch 2.1.1 but the paper and code reference PyTorch
+    2.3.
+risk_flags:
+- model_download_required
+- gpu_intensive
+- currently_vicuna_only
+risk_notes:
+- Models (Vicuna, Code Llama) must be downloaded externally before evaluation; ~60-130
+  GB total.
+- The minimum 24 GB VRAM can only run the 7B model in float16; 33B/34B models require
+  ≥48 GB VRAM.
+- The codebase is currently adapted to LLaMA/Vicuna architectures only (modeling_llama_kv.py);
+  adapting to newer models is listed as a TODO.
+- No benchmark execution was performed during the packaging pass; metric values are
+  not yet validated.
+recommended_when: 'Use this benchmark when you want an inference-acceleration evaluation
+  task for 7B+ LLMs that requires no draft model training, no large retrieval datastore,
+  and minimal additional memory (<2 MB). Ideal for comparing train-free speculative
+  decoding methods under controlled greedy-decoding, batch-size-one conditions.
+
+  '
+not_recommended_when: 'Do not use this if you lack a GPU with ≥24 GB VRAM for serving
+  open-weight LLMs, if you need a training-centric benchmark, or if you need to evaluate
+  models outside the LLaMA/Vicuna family (the code currently only supports LLaMA-based
+  architectures).
+
+  '
+paper:
+  title: 'Turning Trash into Treasure: Accelerating Inference of Large Language Models
+    with Token Recycling'
+  venue: ACL 2025 Oral
+  year: 2025
+  url: https://arxiv.org/abs/2408.08696
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.021_tokenrecycling.zip
+  archive_type: zip
+  local_dir_name: paper-21-TokenRecycling
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.021_tokenrecycling.zip
+  sha256: d38519836e52e4c5c5e1fccd9a4befa5b9a3f20a5c8fa787941a3b2773bd1ebd
+  size_bytes: 55377
+commercial:
+  annual_fee: null
+display:
+  palette_seed: lime-slate-recycle
+  art_style: systems-diagram
+  accent_priority: high
+image_path: ../image/021_aisb.t3.021_tokenrecycling.jpg