@researai/deepscientist 1.5.17 → 1.6.0

package/AISB/catalog/aisb.t3.030_processrm.zh.yaml

@@ -0,0 +1,166 @@
+schema_version: 1
+id: aisb.t3.030_processrm
+name: EpicPRM – 过程监督奖励模型数据构建
+version: 0.1.0
+one_line: '通过基于困惑度的蒙特卡洛估计和自适应二分搜索构建高质量过程监督数据（Epic50k），在Qwen2-math-1.5B上训练PRM，并在ProcessBench（GSM8K、MATH、OlympiadBench、OmniMATH）上评估步骤级F1分数。
+
+  '
+task_description: '此基准测试复现EpicPRM框架，用于构建数学推理的过程监督奖励模型。流水线包含三个阶段：（1）从多个LLM（LLaMA3-8B-Instruct、LLaMA3.1-8B-Instruct、Qwen2-7B-Instruct）在MATH数据集问题上生成链式思维推理链；（2）使用基于困惑度加权的蒙特卡洛估计和自适应二分搜索算法对中间推理步骤进行标注，该算法根据问题难度动态调整起始位置和采样数量；（3）在生成的Epic50k数据集（50k个标注步骤）上使用Qwen2-math-1.5B-base和二分类头训练过程奖励模型（PRM），然后通过ProcessBench严格协议评估步骤级正确性预测。评估脚本（eval_strict.py）计算F1作为acc_correct（正确识别的正确解决方案比例）和acc_error（预测的首个错误与真实值匹配的错误解决方案比例）的调和均值，阈值在GSM8K上调整后应用于MATH、OlympiadBench和OmniMATH。数据构建阶段（generate_train_data.py）需要运行多个7-8B参数LLM作为补全器进行rollout采样，这是计算最密集的阶段。PRM训练本身使用DeepSpeed ZeRO（提供stage 1/2/3配置）。快照包含所有阶段的可执行代码，但不含预生成的数据或预训练检查点；Epic50k和模型可在HuggingFace上获取。
+
+  '
+capability_tags:
+- research_code_optimization
+- process_reward_modeling
+- mathematical_reasoning
+- large_language_models
+- data_construction
+- monte_carlo_estimation
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 1d+
+difficulty: hard
+data_access: public
+primary_outputs:
+- f1_gsm8k
+- f1_math
+- f1_olympiadbench
+- f1_omnimath
+- process_reward_dataset
+- verifier_checkpoint
+launch_profiles:
+- id: quick_eval
+  label: 快速评估
+  description: '从HuggingFace下载预训练的PRM检查点和Epic50k，然后在ProcessBench数据集（GSM8K、MATH、OlympiadBench、OmniMATH）上运行eval_strict.py以复现F1指标。推理1.5B模型需要至少24GB显存的单块GPU。
+
+    '
+- id: train_and_eval
+  label: 训练+评估
+  description: '使用DeepSpeed在Epic50k上从头训练PRM，然后通过eval_strict.py进行评估。通过使用已发布的数据集跳过数据构建阶段。
+
+    '
+- id: full_pipeline
+  label: 完整流水线
+  description: '运行所有三个阶段：通过generate_train_data.py生成CoT链并标注步骤（需要通过vLLM运行多个7-8B LLM）、使用DeepSpeed训练PRM，然后进行评估。这是计算最密集的配置，需要多GPU进行rollout采样。
+
+    '
+dataset_download:
+  primary_method: huggingface
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/SunW7777/EpicPRM
+    access: public
+    note: 'Epic50k数据集（50k个标注的中间推理步骤）及相关模型。也链接自https://github.com/xiaolizh1/EpicPRM。
+
+      '
+  - kind: external
+    url: https://github.com/openai/prm800k
+    access: public
+    note: 'PRM800k数据集，用于论文中的对比和阈值分析。
+
+      '
+  - kind: external
+    url: https://github.com/peiyi9979/Math-Shepherd
+    access: public
+    note: 'Math-Shepherd数据集，用于基线对比。
+
+      '
+  notes:
+  - Epic50k相对较小（~50k步骤）；下载量不大。
+  - '如果运行完整流水线，需要MATH数据集（Hendrycks等人，2021）用于CoT生成。eval_strict.py需要ProcessBench评估数据。
+
+    '
+  - '训练和数据构建需要从HuggingFace下载基础模型（Qwen2-math-1.5B-base、LLaMA3-8B-Instruct、LLaMA3.1-8B-Instruct、Qwen2-7B-Instruct）。
+
+    '
+credential_requirements:
+  mode: none
+  items: []
+  notes:
+  - 下载LLaMA3系列等 gated 模型可能需要HuggingFace账号。
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 150
+    gpu_count: 1
+    gpu_vram_gb: 24
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 300
+    gpu_count: 2
+    gpu_vram_gb: 48
+environment:
+  python: '3.10'
+  cuda: '11.8'
+  pytorch: 2.1.0
+  key_packages:
+  - deepspeed==0.15.4
+  - transformers
+  - vllm
+  - scikit-learn
+  - numpy
+  - torch
+  notes:
+  - 'DeepSpeed ZeRO配置（stage 1/2/3，bf16）捆绑在deepspeed_config/中。典型2-GPU训练使用ds_config_bf16_zero2.json。
+
+    '
+  - 'vLLM用于加速数据构建阶段（generate_train_data.py）的rollout采样。评估或仅训练配置文件不需要。
+
+    '
+  - 参见捆绑的requirements和scripts/获取具体启动命令。
+risk_flags:
+- large_model_downloads
+- multi_model_dependency
+- compute_intensive_data_construction
+risk_notes:
+- '完整数据构建流水线需要同时运行多个7-8B参数LLM作为补全器，每个问题采样大量rollout。此阶段可能需要多个GPU天。
+
+  '
+- '打包期间未执行基准测试；论文中的指标值尚未对此快照进行独立验证。
+
+  '
+- 'eval_strict.py脚本实现了ProcessBench评估协议；如果未捆绑则需单独获取ProcessBench数据文件。
+
+  '
+- '训练需要基础模型权重（Qwen2-math-1.5B-base）。LLaMA3模型可能需要在HuggingFace上接受许可协议。
+
+  '
+recommended_when: '当您想研究高效数据构建的过程监督奖励模型训练、比较标注策略（基于困惑度与基于计数的蒙特卡洛估计）、或评估PRM在数学推理步骤正确性预测上的质量时使用此基准测试。对对数据高效PRM训练感兴趣的研究人员特别有帮助——Epic50k不到PRM800k的10%但能达到相当或更优的结果。
+
+  '
+not_recommended_when: '如果您无法支持基于DeepSpeed的奖励模型训练、仅需要提示级评估而无需步骤级标注、或无法访问至少24GB GPU，请勿使用。完整数据构建流水线需要更多计算资源（多个7-8B LLM进行rollout采样）。
+
+  '
+paper:
+  title: 'An Efficient and Precise Training Data Construction Framework for Process-supervised
+    Reward Model in Mathematical Reasoning
+
+    '
+  venue: ACL 2025
+  year: 2025
+  url: https://aclanthology.org/2025.acl-long.216/
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.030_processrm.zip
+  archive_type: zip
+  local_dir_name: paper-30-ProcessRM
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.030_processrm.zip
+  sha256: e34d67b264044c51d0bcdaa9d6dc4d7b9cb59f9c2285b4a629b3ac02af8c725a
+  size_bytes: 54360
+display:
+  palette_seed: bronze-indigo-verifier
+  art_style: math-lab
+  accent_priority: high
+image_path: ../image/030_aisb.t3.030_processrm.jpg

package/AISB/catalog/aisb.t3.031_circuitstability.yaml

@@ -0,0 +1,172 @@
+schema_version: 1
+id: aisb.t3.031_circuitstability
+name: Circuit Stability Characterizes Language Model Generalization
+version: 0.1.0
+one_line: 'Extract soft circuits from language models (gemma-2-2b and others) via
+  edge-ablation attribution, measure circuit stability across arithmetic/Boolean/sports
+  subtasks, and correlate stability with generalization performance.
+
+  '
+task_description: 'This benchmark evaluates whether the internal circuits a language
+  model uses remain stable across structured subtask partitions, and whether that
+  stability predicts generalization. The workflow has three phases: (1) run baseline
+  evaluation of gemma-2-2b on arithmetic subtasks (digit-pair combinations from 1×1
+  to 8×8) using few-shot prompting, producing per-subtask exact-match accuracy; (2)
+  perform soft-circuit discovery via noisy-to-clean edge-ablation (EAP-IG) over the
+  model''s computational graph for each subtask, yielding real-valued importance scores
+  for every edge; (3) compute Spearman ρ between soft circuits of different subtasks,
+  apply α-equivalence clustering, and correlate circuit stability with accuracy fluctuations.
+  The paper extends this to Boolean expression evaluation and sports understanding.
+  The primary executable entry point is src/eval_arith_baseline.sh, which evaluates
+  four key digit combinations (1-1, 1-8, 8-1, 8-8) and computes accuracy from pickled
+  outputs. Circuit discovery is implemented in src/experiments/circuit_discovery.py
+  and explored interactively in notebooks/circuit-discovery.ipynb. The benchmark is
+  self-contained: data is generated procedurally (no external dataset download), models
+  are loaded via HuggingFace transformers/TransformerLens, and evaluation is local.
+
+  '
+capability_tags:
+- research_code_optimization
+- mechanistic_interpretability
+- large_language_models
+- circuit_analysis
+- evaluation
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 1d+
+difficulty: hard
+data_access: public
+primary_outputs:
+- accuracy_11
+- accuracy_18
+- accuracy_81
+- accuracy_88
+- circuit_stability_report
+- generalization_analysis
+launch_profiles:
+- id: quick_eval
+  label: Quick Eval (Arithmetic Baseline)
+  description: 'Run src/eval_arith_baseline.sh to evaluate gemma-2-2b on four arithmetic
+    subtask digit combinations (1-1, 1-8, 8-1, 8-8) with 1000 samples each. Requires
+    one A100 80 GB GPU. Produces per-subtask accuracy scores from pickled model outputs.
+
+    '
+- id: full_analysis
+  label: Full Circuit Stability Analysis
+  description: 'Run the complete workflow: baseline evaluation across all 64 subtask
+    pairs, soft-circuit discovery via EAP-IG for each subtask, α-equivalence clustering,
+    Spearman ρ computation, t-SNE visualization, and generalization correlation analysis.
+    Case Studies I and II need one A100 80 GB; Case Study III requires two A100 80
+    GB GPUs. Expect 12-24+ hours depending on GPU count and subtask coverage.
+
+    '
+dataset_download:
+  primary_method: procedural
+  sources: []
+  notes:
+  - All evaluation data (arithmetic, Boolean, sports) is generated procedurally at
+    runtime.
+  - No external dataset download is required.
+  - Model weights (gemma-2-2b) are fetched from HuggingFace Hub or loaded from a local
+    path via --model_path.
+credential_requirements:
+  mode: optional
+  items:
+  - HuggingFace token (only if gemma-2-2b requires gated access)
+  notes:
+  - If model weights are pre-downloaded to a local path, no credentials are needed.
+  - Set --model_path in eval scripts to use local weights.
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 150
+    gpu_count: 1
+    gpu_vram_gb: 80
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 300
+    gpu_count: 2
+    gpu_vram_gb: 80
+environment:
+  python: 3.10.16
+  cuda: null
+  pytorch: 2.4.1
+  key_packages:
+  - jax==0.5.0
+  - jaxlib==0.5.0
+  - transformers==4.44.2
+  - transformer-lens==2.11.0
+  - pygraphviz==1.14
+  - accelerate==0.34.2
+  - einops==0.8.0
+  - datasets==3.0.0
+  notes:
+  - Graphviz system package is a hard prerequisite for circuit visualization (pygraphviz).
+  - Install via install.sh or manually (brew install graphviz / apt-get install graphviz).
+  - Full dependency list in environment.yml (conda) or pip fallback documented in
+    README.
+  - CUDA version is not pinned; JAX and PyTorch will select appropriate CUDA backend.
+risk_flags:
+- gpu_intensive
+- long_running
+- model_download_required
+risk_notes:
+- Case Study III (sports understanding with larger models) requires two A100 80 GB
+  GPUs; single-GPU setups cannot run it.
+- Full 64-subtask circuit discovery is computationally expensive (edge ablation over
+  the entire computational graph per subtask).
+- No benchmark execution was performed during the packaging pass; metric values are
+  code-backed but unverified at runtime.
+- The --model_path flag expects pre-downloaded weights; without it, the script attempts
+  HuggingFace Hub download which may fail for gated models.
+- pygraphviz installation can fail if Graphviz headers are not available on the system.
+recommended_when: 'Use this benchmark when you want a mechanistic-interpretability
+  task that extracts soft circuits via edge-ablation, clusters subtasks by circuit
+  similarity, and quantitatively links circuit stability to generalization. It is
+  well-suited for studying arithmetic, Boolean, or compositional reasoning in transformer
+  models and for testing whether optimization improvements preserve circuit-level
+  structure.
+
+  '
+not_recommended_when: 'Do not use this if you cannot provision at least one A100 80
+  GB GPU, if you need a lightweight fine-tuning benchmark, or if you are looking for
+  a task with externally provided static evaluation datasets rather than procedurally
+  generated inputs.
+
+  '
+paper:
+  title: Circuit Stability Characterizes Language Model Generalization
+  authors:
+  - Alan Sun
+  venue: ACL 2025
+  year: 2025
+  url: https://openreview.net/forum?id=2914j8175g
+  arxiv: https://arxiv.org/abs/2505.24731
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.031_circuitstability.zip
+  archive_type: zip
+  local_dir_name: paper-31-CircuitStability
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.031_circuitstability.zip
+  sha256: 549af47824f5e0af3f66e650ca1b91d28b02dc534b23ac71ddd4c84a70a1ee7e
+  size_bytes: 2108204
+display:
+  palette_seed: graphite-gold-circuit
+  art_style: interpretability-lab
+  accent_priority: high
+image_path: ../image/031_aisb.t3.031_circuitstability.jpg
+commercial:
+  annual_fee: null

package/AISB/catalog/aisb.t3.031_circuitstability.zh.yaml

@@ -0,0 +1,134 @@
+schema_version: 1
+id: aisb.t3.031_circuitstability
+name: 电路稳定性刻画语言模型泛化能力
+version: 0.1.0
+one_line: '从语言模型（gemma-2-2b及其他）中通过边消融归因提取软电路，测量算术/布尔/体育子任务的电路稳定性，并将稳定性与泛化性能相关联。
+
+  '
+task_description: '该基准测试评估语言模型内部电路在结构化子任务划分下是否保持稳定，以及该稳定性是否可预测泛化能力。工作流程分为三个阶段：（1）对算术子任务（1×1到8×8的数字对组合）运行gemma-2-2b基线评估，使用少样本提示，产生每个子任务的精确匹配准确率；（2）通过噪声到清晰的边消融（EAP-IG）对每个子任务在模型的计算图上进行软电路发现，为每条边产生实值重要性分数；（3）计算不同子任务软电路之间的Spearman ρ，应用α等价聚类，并将电路稳定性与准确率波动进行相关性分析。该论文将此方法扩展到布尔表达式评估和体育理解。主要可执行入口是src/eval_arith_baseline.sh，评估四个关键数字组合（1-1、1-8、8-1、8-8）并从pickle输出中计算准确率。电路发现在src/experiments/circuit_discovery.py中实现，并在notebooks/circuit-discovery.ipynb中进行交互式探索。该基准测试是自包含的：数据在运行时程序化生成（无需外部数据集下载），模型通过HuggingFace transformers/TransformerLens加载，评估在本地进行。
+
+  '
+capability_tags:
+- research_code_optimization
+- mechanistic_interpretability
+- large_language_models
+- circuit_analysis
+- evaluation
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 1d+
+difficulty: hard
+data_access: public
+primary_outputs:
+- accuracy_11
+- accuracy_18
+- accuracy_81
+- accuracy_88
+- circuit_stability_report
+- generalization_analysis
+launch_profiles:
+- id: quick_eval
+  label: 快速评估（算术基线）
+  description: '运行src/eval_arith_baseline.sh对四个算术子任务数字组合（1-1、1-8、8-1、8-8）评估gemma-2-2b，每个组合1000个样本。需要一块A100 80 GB GPU。从pickle模型输出中产生每个子任务的准确率分数。
+
+    '
+- id: full_analysis
+  label: 完整电路稳定性分析
+  description: '运行完整工作流程：所有64个子任务对的基线评估、每个子任务的EAP-IG软电路发现、α等价聚类、Spearman ρ计算、t-SNE可视化和泛化相关性分析。案例研究I和II需要一块A100 80 GB；案例研究III需要两块A100 80 GB GPU。根据GPU数量和子任务覆盖范围预计需要12-24小时以上。
+
+    '
+dataset_download:
+  primary_method: procedural
+  sources: []
+  notes:
+  - 所有评估数据（算术、布尔、体育）在运行时程序化生成。
+  - 无需外部数据集下载。
+  - 模型权重（gemma-2-2b）从HuggingFace Hub获取，或通过--model_path从本地路径加载。
+credential_requirements:
+  mode: optional
+  items:
+  - HuggingFace令牌（仅当gemma-2-2b需要门控访问时）
+  notes:
+  - 如果模型权重已预下载到本地路径，则无需凭据。
+  - 在评估脚本中设置--model_path以使用本地权重。
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 150
+    gpu_count: 1
+    gpu_vram_gb: 80
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 300
+    gpu_count: 2
+    gpu_vram_gb: 80
+environment:
+  python: 3.10.16
+  cuda: null
+  pytorch: 2.4.1
+  key_packages:
+  - jax==0.5.0
+  - jaxlib==0.5.0
+  - transformers==4.44.2
+  - transformer-lens==2.11.0
+  - pygraphviz==1.14
+  - accelerate==0.34.2
+  - einops==0.8.0
+  - datasets==3.0.0
+  notes:
+  - Graphviz系统包是电路可视化的硬性前提条件（pygraphviz）。
+  - 通过install.sh或手动安装（brew install graphviz / apt-get install graphviz）。
+  - 完整依赖列表在environment.yml（conda）或README中记录的pip备选方案中。
+  - CUDA版本未固定；JAX和PyTorch将选择合适的CUDA后端。
+risk_flags:
+- gpu_intensive
+- long_running
+- model_download_required
+risk_notes:
+- 案例研究III（使用更大模型的体育理解）需要两块A100 80 GB GPU；单GPU设置无法运行。
+- 完整的64子任务电路发现计算成本高昂（每个子任务在完整计算图上进行边消融）。
+- 打包过程中未执行基准测试；指标值有代码支持但未在运行时验证。
+- --model_path标志期望预下载的权重；没有该标志时，脚本尝试从HuggingFace Hub下载，可能对门控模型失败。
+- 如果系统上缺少Graphviz头文件，pygraphviz安装可能失败。
+recommended_when: '当您需要进行机械可解释性任务，通过边消融提取软电路、按电路相似性对子任务进行聚类，并定量地将电路稳定性与泛化能力关联时，使用此基准测试。它非常适合研究Transformer模型中的算术、布尔或组合推理，以及测试优化改进是否保留电路级结构。
+
+  '
+not_recommended_when: '如果无法配置至少一块A100 80 GB GPU、需要轻量级微调基准测试，或正在寻找具有外部提供的静态评估数据集而非程序化生成输入的任务，请勿使用此基准测试。
+
+  '
+paper:
+  title: Circuit Stability Characterizes Language Model Generalization
+  authors:
+  - Alan Sun
+  venue: ACL 2025
+  year: 2025
+  url: https://openreview.net/forum?id=2914j8175g
+  arxiv: https://arxiv.org/abs/2505.24731
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.031_circuitstability.zip
+  archive_type: zip
+  local_dir_name: paper-31-CircuitStability
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.031_circuitstability.zip
+  sha256: 549af47824f5e0af3f66e650ca1b91d28b02dc534b23ac71ddd4c84a70a1ee7e
+  size_bytes: 2108204
+display:
+  palette_seed: graphite-gold-circuit
+  art_style: interpretability-lab
+  accent_priority: high
+image_path: ../image/031_aisb.t3.031_circuitstability.jpg
+commercial:
+  annual_fee: null

package/AISB/catalog/aisb.t3.032_ptsolver.yaml

@@ -0,0 +1,169 @@
+schema_version: 1
+id: aisb.t3.032_ptsolver
+name: 'Personal Travel Solver: RealTravel Data Foundation'
+version: 0.1.0
+one_line: 'Data-only snapshot of the RealTravel dataset (1,000 test + 155 validation
+  samples across 77 US cities) with LLM-driven user-preference extraction scripts;
+  the full PTS planner and SCIP solver are not bundled.
+
+  '
+task_description: 'This benchmark packages the RealTravel dataset—an extension of
+  the TravelPlanner benchmark that incorporates real user reviews and POI metadata
+  from Google Local for 77 US cities. The snapshot contains structured travel queries,
+  user review histories, POI databases (restaurants, attractions, accommodations),
+  and four Python data-processing scripts that use LLM APIs (GPT-4o / DeepSeek) to
+  extract user preferences, summarize profiles, standardize preference tags, and generate
+  POI pro/con descriptions. The full Personal Travel Solver (PTS) system—including
+  the Translator, Search, Re-rank (SASRec), and Planning (SCIP solver) modules—is
+  NOT included in this snapshot. The primary metric (pass_rate) is blocked because
+  no executable evaluation or planning code is present. To use this benchmark as a
+  runnable evaluation, the downstream PTS planner must be restored from the paper''s
+  external code release.
+
+  '
+capability_tags:
+- research_code_optimization
+- travel_planning
+- llm_tooling
+- constraint_solving
+- personalization
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: data_only
+support_level: recovery
+cost_band: low
+time_band: 1-2h
+difficulty: medium
+data_access: public
+primary_outputs:
+- user_profiles
+- poi_summaries
+- structured_travel_data
+- preference_tags
+launch_profiles:
+- id: data_prep
+  label: Data Preparation
+  description: 'Run the four bundled data-processing scripts to extract user preferences,
+    summarize profiles, standardize tags, and generate POI pro/con descriptions. Requires
+    LLM API credentials (OpenAI or DeepSeek). No GPU needed.
+
+    '
+- id: planner_restore
+  label: Planner Restore
+  description: 'Restore the full PTS pipeline (Translator, Search, Re-rank with SASRec,
+    Planning with SCIP solver) from the paper''s external code release before treating
+    this benchmark as a runnable end-to-end evaluation.
+
+    '
+dataset_download:
+  primary_method: bundled
+  sources:
+  - kind: archive
+    url: https://deepscientist.cc/AISB/032_ptsolver
+    access: public
+    note: 'ZIP archive containing RealTravel dataset (1,000 test / 155 validation
+      samples), POI databases for 77 cities, user review data, and data-processing
+      scripts.
+
+      '
+  notes:
+  - The dataset is derived from Google Local (Yan et al., 2023) and TravelPlanner
+    (Xie et al., 2024).
+  - Database subdirectories contain accommodations, attractions, restaurants, and
+    background data.
+  - Total disk footprint is modest (under 20 GB uncompressed).
+credential_requirements:
+  mode: api_keys
+  items:
+  - OpenAI API key (GPT-4o) or DeepSeek API key for running data-processing scripts
+  - Optional ChatGLM / SiliconFlow / Yi API keys referenced in attraction-pro-con.py
+  notes:
+  - Scripts read keys from environment variables (Gpt_API_KEY, DEEPSEEK_API_KEY, CHATGLM_API_KEY).
+  - Some hardcoded API keys appear in the source code but may be expired or invalid.
+  - No credentials needed if you only inspect the pre-existing dataset files.
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 16
+    disk_gb: 20
+    gpu_count: 0
+    gpu_vram_gb: 0
+  recommended:
+    cpu_cores: 16
+    ram_gb: 32
+    disk_gb: 50
+    gpu_count: 0
+    gpu_vram_gb: 0
+environment:
+  python: '3.10'
+  cuda: null
+  pytorch: null
+  flash_attn: null
+  key_packages:
+  - langchain
+  - openai
+  - tqdm
+  notes:
+  - CPU-only execution is sufficient for all bundled scripts.
+  - The data-processing scripts use langchain ChatOpenAI and openai client libraries.
+  - Full PTS system (not bundled) would additionally require PySCIPOpt, SASRec, BGE
+    embeddings, and scikit-learn for PCA.
+  - See bundled requirements files for the complete dependency set.
+risk_flags:
+- blocked_metric
+- incomplete_pipeline
+- api_key_exposure
+- external_code_dependency
+risk_notes:
+- The primary metric (pass_rate) is blocked—no evaluation or planning code is in the
+  snapshot.
+- The full PTS system (5 modules) is not bundled; only data-processing scripts are
+  present.
+- Hardcoded API keys appear in attraction-pro-con.py; these may be leaked credentials
+  and should not be reused.
+- Running data-processing scripts will incur LLM API costs proportional to the number
+  of users/POIs processed.
+- The Re-rank module (SASRec + BGE embeddings) and Planning module (SCIP solver) must
+  be restored externally.
+recommended_when: 'Use this benchmark when you want a realistic travel-planning dataset
+  grounded in real user reviews and POI data, for tasks that mix symbolic constraint
+  satisfaction with user preference modeling, without requiring heavyweight GPU training.
+  Suitable for evaluating LLM-based preference extraction, user profiling, or as a
+  data foundation for building constraint-based travel planners.
+
+  '
+not_recommended_when: 'Do not use this if you need a fully self-contained, end-to-end
+  runnable benchmark. The planning and evaluation pipeline is not bundled. Also not
+  suitable if you need a benchmark focused on large-scale model fine-tuning, multimodal
+  data, or non-US travel destinations.
+
+  '
+paper:
+  title: 'Personal Travel Solver: A Preference-Driven LLM-Solver System for Travel
+    Planning'
+  venue: ACL 2025
+  year: 2025
+  url: https://aclanthology.org/2025.acl-long.1339/
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.032_ptsolver.zip
+  archive_type: zip
+  local_dir_name: paper-32-PTSolver
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.032_ptsolver.zip
+  sha256: 26f7f39e12eb28552ada092809b289f77090a229d0b745e9dfbcbb7b7b4f9d5c
+  size_bytes: 38141816
+commercial:
+  annual_fee: null
+display:
+  palette_seed: sand-teal-itinerary
+  art_style: trip-planner
+  accent_priority: medium
+image_path: ../image/032_aisb.t3.032_ptsolver.jpg