@researai/deepscientist 1.5.16 → 1.6.0

package/AISB/catalog/aisb.t3.014_piguard.zh.yaml

@@ -0,0 +1,164 @@
+schema_version: 1
+id: aisb.t3.014_piguard
+name: 'PIGuard: 通过缓解过度防御实现提示词注入防护栏'
+version: 0.1.0
+one_line: '训练或微调基于DeBERTa的提示词注入防护模型（PIGuard），并在四个基准数据集（NotInject、PINT、BIPIA、WildGuard-Benign）上进行评估，通过测量恶意检测准确率、良性准确率和过度防御准确率来评估不同触发词难度级别下的表现。
+
+  '
+task_description: '本基准测试整合了完整的PIGuard流程：一个轻量级的基于DeBERTa的二分类器，用于检测提示词注入攻击，同时最大限度减少过度防御（对包含触发词（如"ignore"）的良性输入的误报）。优化任务是提高注入召回率而不增加误报率。训练数据来自20个开源数据集及LLM增强样本，已打包在快照中。评估涵盖四个测试集：NotInject（339个良性样本，含1/2/3个触发词，分为三个难度子集，各113个）、BIPIA（恶意样本）、WildGuard-Benign和PINT。PINT基准需要单独的访问申请。提供了三个评估脚本：eval.py（通过PIGuard.py加载本地检查点）、eval_piguard.py（从/tmp/PIGuard_weights加载HuggingFace格式权重）和run_official_eval.py（与eval_piguard.py相同但使用自定义良性分数阈值0.10）。主要指标包括NotInject上的过度防御准确率（整体及各触发词数量）、PINT/BIPIA上的恶意准确率以及PINT/WildGuard-Benign上的良性准确率。模型权重可从HuggingFace（leolee99/PIGuard）或Google Drive检查点加载。
+
+  '
+capability_tags:
+- research_code_optimization
+- llm_security
+- classification
+- prompt_injection
+- robustness
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: turnkey
+cost_band: medium
+time_band: 6-24h
+difficulty: medium
+data_access: restricted
+primary_outputs:
+- over_defense_accuracy
+- one_trigger_accuracy
+- two_trigger_accuracy
+- three_trigger_accuracy
+- malicious_accuracy
+- benign_accuracy
+- trigger_accuracy_breakdown
+- evaluation_report
+launch_profiles:
+- id: quick_check
+  label: 快速检查（HuggingFace权重）
+  description: '运行eval_hf.py或eval_piguard.py，在所有打包的测试集（NotInject、BIPIA、WildGuard-Benign）上评估来自HuggingFace的预训练PIGuard模型。无需训练。除非单独获取访问权限，否则跳过PINT评估。
+
+    '
+- id: full_eval
+  label: 完整评估（全部四个基准）
+  description: '在NotInject、BIPIA、WildGuard-Benign和PINT上运行eval.py --resume <checkpoint>或run_official_eval.py。需要PINT访问权限并通过util.py进行YAML到JSON的转换。
+
+    '
+- id: train_and_eval
+  label: 训练 + 评估
+  description: '运行train.py，使用MOF策略在打包的20源训练集上重新训练PIGuard，然后在所有四个基准上评估生成的检查点。根据VRAM和批大小，在单GPU上预计需要6-24小时。
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/leolee99/NotInject
+    access: public
+    note: 'NotInject过度防御评估数据集（339个样本，三个子集各113个）。也已打包在快照的datasets/目录下。
+
+      '
+  - kind: huggingface
+    url: https://huggingface.co/leolee99/PIGuard
+    access: public
+    note: 预训练PIGuard模型权重（基于DeBERTa）。
+  - kind: google_drive
+    url: https://drive.google.com/file/d/1JpiVb_wtnbBLNEjIx1KS7PHuvmARQKTu/view?usp=sharing
+    access: public
+    note: 备用检查点下载。
+  - kind: external
+    url: https://share-eu1.hsforms.com/1TwiBEvLXRrCjJSdnbnHpLwfdfs3
+    access: request_required
+    note: '来自Lakera AI的PINT基准。非公开，需要填写访问申请表。下载后必须通过util.py从YAML转换为JSON。
+
+      '
+  - kind: bundled
+    url: null
+    access: public
+    note: '训练数据（20个开源数据集+LLM增强）、验证集（144个样本）和测试集（NotInject、BIPIA、WildGuard-Benign）全部打包在PIGuard/datasets/目录下。
+
+      '
+  notes:
+  - 打包的数据总量较小（几十MB）。大多数配置不需要大规模下载。
+  - PINT是唯一需要单独申请访问权限的外部数据集。
+credential_requirements:
+  mode: none
+  items: []
+  notes:
+  - PINT基准访问需要向Lakera AI提交表单，但不需要API密钥。
+  - HuggingFace模型下载是公开的，无需令牌。
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 32
+    disk_gb: 50
+    gpu_count: 1
+    gpu_vram_gb: 12
+  recommended:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 120
+    gpu_count: 1
+    gpu_vram_gb: 24
+environment:
+  python: '3.10'
+  cuda: '11.8'
+  pytorch: 2.4.0
+  flash_attn: 2.6.1
+  key_packages:
+  - flash-attn==2.6.1
+  - vllm==0.5.4
+  - transformers==4.44.0
+  - ptflops
+  notes:
+  - 完整的依赖项列表请参见打包的requirements.txt。
+  - 模型基于DeBERTa（非LLM）；vllm仅在某些评估模式下使用。
+  - flash-attn需要兼容的CUDA和GPU架构。
+risk_flags:
+- restricted_dataset_component
+- external_model_weights
+risk_notes:
+- 'PINT基准非公开且未打包。完整的四基准评估需要通过网页表单向Lakera AI申请访问权限。没有PINT的情况下，仍可使用四个测试集中的三个。
+
+  '
+- '模型权重必须在运行时从HuggingFace或Google Drive下载，除非已预置在/tmp/PIGuard_weights。
+
+  '
+- 'eval_piguard.py和run_official_eval.py硬编码MODEL_PATH为/tmp/PIGuard_weights，DATASET_ROOT为/repo/datasets。这些路径可能需要调整。
+
+  '
+- 'run_official_eval.py使用自定义的BENIGN_THRESHOLD为0.10（从0.50降低），这与eval.py中的默认argmax方法相比改变了分类边界。
+
+  '
+recommended_when: '当您需要一个安全导向的文本分类任务，具有双重压力：在提高提示词注入检测召回率的同时控制过度防御（对包含类攻击触发词的良性输入的误报）。非常适合研究捷径学习、DeBERTa微调策略以及轻量级防护模型的阈值校准。
+
+  '
+not_recommended_when: '如果无法提供GPU（模型需要CUDA来进行实际训练和批量推理）、无法获取PINT完整评估访问权限，或者您寻找的是生成式LLM基准而非二分类任务，则不应使用此基准。
+
+  '
+paper:
+  title: 'PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free'
+  venue: ACL 2025
+  year: 2025
+  url: https://aclanthology.org/2025.acl-long.1468/
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.014_piguard.zip
+  archive_type: zip
+  local_dir_name: paper-14-PIGuard
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.014_piguard.zip
+  sha256: 9a7996fc3b40709caa357b763d7a51e1f1f2ab449480e403ec34a7c248eb4c81
+  size_bytes: 1621668
+commercial:
+  annual_fee: null
+display:
+  palette_seed: rust-sand-guard
+  art_style: safety-dashboard
+  accent_priority: high
+image_path: ../image/014_aisb.t3.014_piguard.jpg

package/AISB/catalog/aisb.t3.015_frspec.yaml

@@ -0,0 +1,209 @@
+schema_version: 1
+id: aisb.t3.015_frspec
+name: 'FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked
+  Speculative Sampling'
+version: 0.1.0
+one_line: Benchmark frequency-ranked speculative decoding for large-vocabulary LLMs
+  by building custom C/CUDA kernels and measuring tokens/s speedup over EAGLE-2 baselines
+  on Spec-Bench, HumanEval, and GSM8K.
+task_description: 'This benchmark evaluates FR-Spec, a frequency-ranked speculative
+  sampling framework that accelerates large-vocabulary LLM inference by restricting
+  the draft model''s LM Head to a high-frequency token subset (e.g., top 25% of vocabulary),
+  reducing LM Head computation by ~75% while preserving output-distribution equivalence
+  during verification. The task requires: (1) compiling a custom C/CUDA extension
+  (llamacu) with nvcc targeting the correct GPU compute capability, (2) downloading
+  target LLM weights (e.g., Llama-3-8B-Instruct, Qwen2-7B-Instruct) and corresponding
+  EAGLE-2 draft model weights, (3) obtaining or generating token-frequency statistics
+  from SlimPajama-627B via fr/fr.py, (4) running baseline (autoregressive), EAGLE-2,
+  and FR-Spec inference on three benchmark suites (spec_bench, human_eval, gsm8k)
+  using bundled shell scripts, and (5) measuring tokens-per-second throughput and
+  optionally checking correctness on human_eval and gsm8k. The primary metrics are
+  throughput (tokens/s) across seven Spec-Bench subtasks (MT, conversation, RAG, math,
+  QA, summarization, code) plus HumanEval and GSM8K speed. The code includes a modified
+  FlashAttention with tree-attention bitmask support and preallocated memory management.
+  No external evaluation service is needed; all scoring is local.
+
+  '
+capability_tags:
+- research_code_optimization
+- large_language_models
+- speculative_decoding
+- cuda_kernels
+- inference_acceleration
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 6-24h
+difficulty: hard
+data_access: public
+primary_outputs:
+- mt_bench_tokens_per_second
+- translation_tokens_per_second
+- summarization_tokens_per_second
+- qa_tokens_per_second
+- math_reasoning_tokens_per_second
+- rag_tokens_per_second
+- task_throughput_report
+- speed_benchmark
+launch_profiles:
+- id: quick_check
+  label: Quick Check
+  description: 'Run a single packaged speed benchmark (e.g., spec_bench with Llama-3-8B-Instruct)
+    to verify FR-Spec throughput gains versus EAGLE-2 baseline. Requires model weights
+    already downloaded and CUDA extension compiled.
+
+    '
+- id: full_speed_suite
+  label: Full Speed Suite
+  description: 'Run all three benchmark suites (spec_bench, human_eval, gsm8k) for
+    baseline, EAGLE-2, and FR-Spec configurations. Includes speed measurement and
+    optional correctness checks for human_eval and gsm8k. Covers all seven Spec-Bench
+    subtasks.
+
+    '
+- id: freq_stats_generation
+  label: Frequency Statistics Generation
+  description: 'Generate custom token-frequency statistics from SlimPajama-627B using
+    fr/fr.py before running evaluation. Useful if pre-computed stats are unavailable
+    or a different model is targeted.
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec
+    access: public
+    note: Pre-computed token-frequency statistics for Llama-3-8B-Instruct.
+  - kind: huggingface
+    url: https://huggingface.co/thunlp/LLaMA3.2-Instruct-1B-FR-Spec
+    access: public
+    note: Pre-computed token-frequency statistics for Llama-3.2-1B-Instruct.
+  - kind: huggingface
+    url: https://huggingface.co/thunlp/Qwen2-7B-Instruct-FR-Spec
+    access: public
+    note: Pre-computed token-frequency statistics for Qwen2-7B-Instruct.
+  - kind: huggingface
+    url: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
+    access: gated
+    note: Target LLM weights. Requires acceptance of Meta license on HuggingFace.
+  - kind: huggingface
+    url: https://huggingface.co/cerebras/SlimPajama-627B
+    access: public
+    note: Streaming corpus used by fr/fr.py for token-frequency stat generation. Not
+      needed if using pre-computed stats.
+  notes:
+  - EAGLE-2 draft model weights must also be downloaded; see EAGLE repository for
+    URLs.
+  - Target LLM weights are 15-30 GB depending on model; plan 50-100 GB total for models
+    + draft weights + stats.
+  - Spec-Bench evaluation datasets are bundled or fetched by the evaluation scripts.
+credential_requirements:
+  mode: optional
+  items:
+  - HuggingFace token (required only for gated model weights like Llama-3)
+  notes:
+  - If using only Qwen2 or pre-downloaded weights, no credentials are needed.
+  - OpenAI/Anthropic API keys listed in setup.py install_requires (fschat, openai,
+    anthropic) are for MT-Bench judging and may not be exercised in throughput-only
+    runs.
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 32
+    disk_gb: 100
+    gpu_count: 1
+    gpu_vram_gb: 24
+  recommended:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 200
+    gpu_count: 1
+    gpu_vram_gb: 48
+  notes:
+  - CUDA compilation (nvcc) requires ~8-9 GB RAM per parallel job; setup.py auto-limits
+    MAX_JOBS based on free memory.
+  - 24 GB VRAM is sufficient for Llama-3-8B in FP16; 48 GB recommended for comfortable
+    headroom and larger batch experiments.
+  - Compilation itself can take 10-30 minutes depending on core count.
+environment:
+  python: '3.11'
+  cuda: 11.8+
+  pytorch: 2.0+
+  key_packages:
+  - transformers==4.46.2
+  - accelerate==0.26.0
+  - datasets
+  - fschat
+  - human_eval
+  - pybind11
+  - psutil
+  - ninja
+  notes:
+  - Building the CUDA extension requires nvcc and correct GPU compute capability set
+    in setup.py (default arch="80" for A100; change for other GPUs).
+  - The package bundles a modified FlashAttention v2.4.2 csrc with tree-attention
+    bitmask support; external flash-attn installation is not needed.
+  - Compilation links against cublas.
+  - See setup.py install_requires for the complete dependency set.
+risk_flags:
+- custom_cuda_build
+- large_model_download
+- gated_model_access
+- compilation_complexity
+risk_notes:
+- CUDA compilation requires matching nvcc version and correct compute capability in
+  setup.py; misconfiguration causes silent wrong results or build failures.
+- Gated Llama-3 weights require HuggingFace license acceptance; this may block fully
+  automated runs.
+- EAGLE-2 draft model weights are a separate download not fully documented in the
+  snapshot; recovery may require consulting the EAGLE repository.
+- MT-Bench quality judging (as opposed to throughput measurement) may require OpenAI
+  API keys, but throughput-only evaluation does not.
+- The default arch="80" in setup.py targets A100; users on other GPUs must manually
+  edit this value.
+recommended_when: 'Use this benchmark when you want an LLM-systems optimization task
+  focused on custom C/CUDA kernel development, speculative decoding acceleration,
+  and inference throughput measurement for large-vocabulary models (128k+ tokens).
+  Well-suited for evaluating GPU kernel optimization skills and understanding the
+  interplay between vocabulary size, LM Head computation, and speculative sampling
+  acceptance rates.
+
+  '
+not_recommended_when: 'Do not use this if you lack a modern NVIDIA GPU with ≥24 GB
+  VRAM, cannot compile custom CUDA extensions, or need a task focused on model training
+  rather than inference optimization. Not appropriate if you need a pure-Python benchmark
+  without native code compilation.
+
+  '
+paper:
+  title: 'FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked
+    Speculative Sampling'
+  venue: ACL 2025 Main
+  year: 2025
+  url: https://arxiv.org/abs/2502.14856
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.015_frspec.zip
+  archive_type: zip
+  local_dir_name: paper-15-FRSpec
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.015_frspec.zip
+  sha256: bf4aaa74d5faae067f125c2df48fa0eb7d8e120988bec5fbd748666fc6e1a9e9
+  size_bytes: 974875
+commercial:
+  annual_fee: null
+display:
+  palette_seed: neon-slate-kernel
+  art_style: systems-benchmark
+  accent_priority: high
+image_path: ../image/015_aisb.t3.015_frspec.jpg

package/AISB/catalog/aisb.t3.015_frspec.zh.yaml

@@ -0,0 +1,163 @@
+schema_version: 1
+id: aisb.t3.015_frspec
+name: 'FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked
+  Speculative Sampling'
+version: 0.1.0
+one_line: 通过构建自定义C/CUDA内核并测量Spec-Bench、HumanEval和GSM8K上相较于EAGLE-2基线的tokens/s加速比，来对大词表LLM的频率排序投机解码进行基准测试。
+task_description: '本基准测试评估FR-Spec，这是一种频率排序投机采样框架，通过将draft模型的LM Head限制在高频token子集（例如词汇表前25%），减少约75%的LM Head计算量，同时在验证过程中保持输出分布等价性，从而加速大词表LLM推理。该任务要求：（1）使用nvcc编译面向正确GPU计算能力目标的自定义C/CUDA扩展（llamacu），（2）下载目标LLM权重（如Llama-3-8B-Instruct、Qwen2-7B-Instruct）及相应的EAGLE-2 draft模型权重，（3）通过fr/fr.py从SlimPajama-627B获取或生成token频率统计数据，（4）使用捆绑的shell脚本在三个基准测试套件（spec_bench、human_eval、gsm8k）上运行基线（自回归）、EAGLE-2和FR-Spec推理，（5）测量tokens-per-second吞吐量和可选地在human_eval和gsm8k上检查正确性。主要指标是七个Spec-Bench子任务（MT、对话、RAG、数学、QA、摘要、代码）以及HumanEval和GSM8K速度的吞吐量（tokens/s）。代码包含支持树注意力位掩码和预分配内存管理的改进版FlashAttention。无需外部评估服务；所有评分均在本地完成。
+
+  '
+capability_tags:
+- research_code_optimization
+- large_language_models
+- speculative_decoding
+- cuda_kernels
+- inference_acceleration
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+cost_band: high
+time_band: 6-24h
+difficulty: hard
+data_access: public
+primary_outputs:
+- mt_bench_tokens_per_second
+- translation_tokens_per_second
+- summarization_tokens_per_second
+- qa_tokens_per_second
+- math_reasoning_tokens_per_second
+- rag_tokens_per_second
+- task_throughput_report
+- speed_benchmark
+launch_profiles:
+- id: quick_check
+  label: 快速检查
+  description: '运行单个打包的速度基准测试（例如使用Llama-3-8B-Instruct的spec_bench），以验证FR-Spec相较于EAGLE-2基线的吞吐量提升。需要已下载模型权重并编译好CUDA扩展。
+
+    '
+- id: full_speed_suite
+  label: 完整速度套件
+  description: '为基线、EAGLE-2和FR-Spec配置运行所有三个基准测试套件（spec_bench、human_eval、gsm8k）。包括速度测量和可选的human_eval与gsm8k正确性检查。覆盖所有七个Spec-Bench子任务。
+
+    '
+- id: freq_stats_generation
+  label: 频率统计生成
+  description: '使用fr/fr.py从SlimPajama-627B生成自定义token频率统计数据，然后再运行评估。当预计算统计数据不可用或针对不同模型时，此选项非常有用。
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec
+    access: public
+    note: Llama-3-8B-Instruct的预计算token频率统计数据。
+  - kind: huggingface
+    url: https://huggingface.co/thunlp/LLaMA3.2-Instruct-1B-FR-Spec
+    access: public
+    note: Llama-3.2-1B-Instruct的预计算token频率统计数据。
+  - kind: huggingface
+    url: https://huggingface.co/thunlp/Qwen2-7B-Instruct-FR-Spec
+    access: public
+    note: Qwen2-7B-Instruct的预计算token频率统计数据。
+  - kind: huggingface
+    url: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
+    access: gated
+    note: 目标LLM权重。需要接受HuggingFace上的Meta许可协议。
+  - kind: huggingface
+    url: https://huggingface.co/cerebras/SlimPajama-627B
+    access: public
+    note: fr/fr.py用于生成token频率统计的流式语料库。如果使用预计算统计数据，则不需要。
+  notes:
+  - EAGLE-2 draft模型权重也必须下载；请参阅EAGLE仓库获取URL。
+  - 目标LLM权重根据模型不同为15-30 GB；请预留50-100 GB总空间用于模型 + draft权重 + 统计数据。
+  - Spec-Bench评估数据集由评估脚本捆绑或获取。
+credential_requirements:
+  mode: optional
+  items:
+  - HuggingFace token（仅在访问门控模型权重如Llama-3时需要）
+  notes:
+  - 如果仅使用Qwen2或已下载的权重，则不需要凭据。
+  - setup.py install_requires中列出的OpenAI/Anthropic API密钥（fschat、openai、anthropic）用于MT-Bench评判，在仅吞吐量运行中可能不会被调用。
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 32
+    disk_gb: 100
+    gpu_count: 1
+    gpu_vram_gb: 24
+  recommended:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 200
+    gpu_count: 1
+    gpu_vram_gb: 48
+  notes:
+  - CUDA编译（nvcc）每个并行作业需要约8-9 GB RAM；setup.py会根据可用内存自动限制MAX_JOBS。
+  - 24 GB显存足以支持FP16精度下的Llama-3-8B；建议使用48 GB以获得充足余量和更大的批处理实验空间。
+  - 编译本身可能需要10-30分钟，具体取决于核心数量。
+environment:
+  python: '3.11'
+  cuda: 11.8+
+  pytorch: 2.0+
+  key_packages:
+  - transformers==4.46.2
+  - accelerate==0.26.0
+  - datasets
+  - fschat
+  - human_eval
+  - pybind11
+  - psutil
+  - ninja
+  notes:
+  - 构建CUDA扩展需要nvcc和正确的GPU计算能力设置在setup.py中（默认arch="80"适用于A100；其他GPU请更改此值）。
+  - 该包捆绑了支持树注意力位掩码的改进版FlashAttention v2.4.2 csrc；无需外部flash-attn安装。
+  - 编译链接cublas。
+  - 请参阅setup.py install_requires获取完整的依赖集。
+risk_flags:
+- custom_cuda_build
+- large_model_download
+- gated_model_access
+- compilation_complexity
+risk_notes:
+- CUDA编译需要匹配的nvcc版本和setup.py中正确的计算能力设置；配置错误会导致静默的错误结果或构建失败。
+- 门控的Llama-3权重需要接受HuggingFace许可协议；这可能阻止完全自动化运行。
+- EAGLE-2 draft模型权重是本快照中未完全记录的单独下载；恢复可能需要查阅EAGLE仓库。
+- MT-Bench质量评判（而非吞吐量测量）可能需要OpenAI API密钥，但仅吞吐量评估不需要。
+- setup.py中的默认arch="80"面向A100；其他GPU用户必须手动编辑此值。
+recommended_when: '当您需要一个专注于自定义C/CUDA内核开发、投机解码加速和大词表模型（128k+ tokens）推理吞吐量测量的LLM系统优化任务时，请使用此基准测试。非常适合评估GPU内核优化技能并理解词汇表大小、LM Head计算与投机采样接受率之间的相互作用。
+
+  '
+not_recommended_when: '如果您没有配备≥24 GB显存的现代NVIDIA GPU、无法编译自定义CUDA扩展，或需要一个专注于模型训练而非推理优化的任务，请勿使用此基准测试。如果需要一个没有原生代码编译的纯Python基准测试，也不适用。
+
+  '
+paper:
+  title: 'FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked
+    Speculative Sampling'
+  venue: ACL 2025 Main
+  year: 2025
+  url: https://arxiv.org/abs/2502.14856
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.015_frspec.zip
+  archive_type: zip
+  local_dir_name: paper-15-FRSpec
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.015_frspec.zip
+  sha256: bf4aaa74d5faae067f125c2df48fa0eb7d8e120988bec5fbd748666fc6e1a9e9
+  size_bytes: 974875
+commercial:
+  annual_fee: null
+display:
+  palette_seed: neon-slate-kernel
+  art_style: systems-benchmark
+  accent_priority: high
+image_path: ../image/015_aisb.t3.015_frspec.jpg