@researai/deepscientist 1.5.17 → 1.6.0

package/AISB/catalog/aisb.t3.012_efficientqat.zh.yaml

@@ -0,0 +1,159 @@
+schema_version: 1
+id: aisb.t3.012_efficientqat
+name: 'EfficientQAT：大型语言模型的高效量化感知训练'
+version: 0.1.0
+one_line: '针对 Llama-2/3 和 Mistral 系列（7B–70B）的两阶段低位宽量化感知训练（Block-AP → E2E-QP），通过 WikiText-2 困惑度、C4 困惑度及五个推理基准零样本准确率进行评估。
+
+'
+task_description: '本基准实现了 EfficientQAT，这是一种针对大型语言模型在 2/3/4 位权重仅量化精度下的两阶段量化感知训练流程。阶段一（Block-AP）使用 4096 个 RedPajama 样本（上下文长度 2048）的重构损失，对所有参数——权重、缩放因子和零点——进行逐块训练。阶段二（E2E-QP）冻结量化后的整数权重，仅在目标数据集（RedPajama 或 Alpaca，上下文长度 4096）上端到端微调量化缩放因子。主要可执行入口为 main_block_ap.py 和 main_e2e_qp.py，并配有辅助的缩放因子校准脚本（calibrate_scales.py、calibrate_scales_v2.py）。评估指标包括 WikiText-2 困惑度、C4 困惑度，以及在 WinoGrande、PIQA、HellaSwag、ARC-Easy 和 ARC-Challenge 上使用 lm-eval v0.4.2 测量的平均零样本准确率。多种模型/位宽配置的预量化检查点可在 HuggingFace 上获取，并可通过捆绑的转换脚本转换为 GPTQ 或 BitBLAS 格式。论文报告称，2 位 Llama-2-70B 可在单张 A100-80GB 上于 41 小时内完成，且精度下降小于 3 个点。无需外部评估服务，所有指标均在本地计算。
+
+'
+capability_tags:
+- research_code_optimization
+- large_language_models
+- quantization
+- model_compression
+- efficient_inference
+- quantization_aware_training
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: advanced
+time_band: 1d+
+cost_band: high
+difficulty: hard
+data_access: public
+primary_outputs:
+- wikitext2_ppl
+- c4_ppl
+- quantized_checkpoint
+- avg_accuracy
+launch_profiles:
+- id: calibration_only
+  label: 仅校准
+  description: '使用 WikiText-2 训练数据对预量化检查点运行缩放因子校准（calibrate_scales.py 或 calibrate_scales_v2.py）。生成更新后的缩放因子参数，无需完整的 Block-AP 或 E2E-QP 再训练。最快的路径；需要磁盘上有预量化模型。
+
+'
+- id: qat_eval
+  label: QAT + 评估
+  description: '运行完整的两阶段 EfficientQAT 流程：通过 main_block_ap.py 进行 Block-AP 逐块训练，然后通过 main_e2e_qp.py 进行 E2E-QP 端到端缩放因子训练，最后评估 WikiText-2/C4 困惑度及五个推理任务的零样本准确率。这是论文忠实路线，需要大量 GPU 时间（根据模型大小可能需要数小时到数天）。
+
+'
+- id: eval_pretrained
+  label: 评估预量化模型
+  description: '从 HuggingFace 下载预量化的 EfficientQAT 检查点，无需任何训练即可评估困惑度和零样本准确率。使用带 --resume_quant 参数的 main_block_ap.py。
+
+'
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample
+    access: public
+    note: 'RedPajama 校准数据，用于 Block-AP（4096 样本，序列长度 2048）和 E2E-QP（4096 样本，序列长度 4096）。由 datautils_block.py 自动下载。
+
+'
+  - kind: huggingface
+    url: https://huggingface.co/datasets/wikitext
+    access: public
+    note: 'WikiText-2（wikitext-2-raw-v1），用于困惑度评估和缩放因子校准。在评估期间自动下载。
+
+'
+  - kind: huggingface
+    url: https://huggingface.co/ChenMnZ
+    access: public
+    note: 'Llama-2（7B/13B/70B）、Llama-3（8B/70B）、Llama-3-Instruct 和 Mistral-Large 的预量化模型检查点，支持 EQAT/GPTQ/BitBLAS 格式。各检查点大小从约 2 GB 到约 39 GB 不等。
+
+'
+  - kind: huggingface
+    url: https://huggingface.co/datasets/tatsu-lab/alpaca
+    access: public
+    note: 'Alpaca 指令微调数据集，用作指令微调场景的替代 E2E-QP 训练数据。
+
+'
+  notes:
+  - 全精度基模型（如约 131 GB 的 Llama-2-70B）如需从头运行 Block-AP，必须从其原始 HuggingFace 仓库单独下载。
+  - 预量化检查点要小得多（例如 w2g64 Llama-2-70B 约 20 GB）。
+  - 总磁盘使用量很大程度上取决于目标模型系列和位宽。
+credential_requirements:
+  mode: none
+  items:
+  - HuggingFace 账户可能需要用于门控模型（Llama-2、Llama-3），但访问 EfficientQAT 检查点本身无需账户。
+  notes:
+  - Meta Llama 模型权重需要在 HuggingFace 上接受许可协议后方可下载。
+  - ChenMnZ 命名空间下的预量化 EfficientQAT 检查点可公开访问。
+resources:
+  minimum:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 200
+    gpu_count: 1
+    gpu_vram_gb: 48
+  recommended:
+    cpu_cores: 32
+    ram_gb: 128
+    disk_gb: 500
+    gpu_count: 2
+    gpu_vram_gb: 80
+environment:
+  python: '3.11'
+  cuda: null
+  pytorch: 2.2.2
+  flash_attn: null
+  key_packages:
+  - bitsandbytes==0.41.0
+  - transformers==4.40.1
+  - lm-eval==0.4.2
+  - accelerate
+  - datasets
+  notes:
+  - 完整的依赖项列表请参阅捆绑的 requirements.txt。
+  - CUDA 工具包版本在仓库中未固定；任何支持 PyTorch 2.2.2 的版本均可使用。
+  - GPTQModel（已测试 v0.9.8）仅在模型格式转换为 GPTQ/BitBLAS 时需要；核心 QAT 和评估不需要。
+  - bitsandbytes 用于 E2E-QP 训练中的 AdamW 优化器。
+risk_flags:
+- large_model_download
+- high_gpu_memory
+- long_training_time
+- gated_model_access
+risk_notes:
+- 70B 模型路线需要单张 A100-80GB（论文报告 2 位 Llama-2-70B 的 Block-AP + E2E-QP 需 41 小时）或多人 GPU 配置。
+- 7B 模型路线可在单张 48GB GPU 上完成，但完整的 Block-AP + E2E-QP 仍需数小时。
+- Llama-2/3 的全精度基模型权重在 HuggingFace 上受门控限制，需要接受许可协议。
+- Block-AP 中的 --off_load_to_disk 标志可以训练速度换取降低 CPU 内存使用量。
+- 打包过程中未执行基准测试运行；仍需运行时验证。
+- 论文报告 2 位 Llama-2-70B 的 E2E-QP 内存需求为 34.2 GB。
+recommended_when: '当您需要一个以激进位宽（2-4 位）量化感知训练为中心的 LLM 系统任务，且具备自包含的训练和评估流程，覆盖从 7B 到 70B 的多种模型规模时，可使用此基准。适用于评估低位宽 QAT 的优化策略、与 PTQ 和 Q-PEFT 基线对比，或生成可部署的 GPTQ/BitBLAS 格式量化检查点。
+
+'
+not_recommended_when: '如果您无法访问至少一块 ≥48 GB VRAM 的 GPU、需要亚小时级基准测试周转时间，或关注的是小型模型（<7B 参数），则不应使用此基准。此外，如需权重-激活量化基准也不适用（此基准为权重仅量化）。
+
+'
+paper:
+  title: 'EfficientQAT: Efficient Quantization-Aware Training for Large Language Models'
+  venue: ACL 2025 Main
+  year: 2025
+  url: https://arxiv.org/abs/2407.11062
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.012_efficientqat.zip
+  archive_type: zip
+  local_dir_name: paper-12-EfficientQAT
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.012_efficientqat.zip
+  sha256: 8f53850f12f1bdbc4e3212b21fb51a7479ebb8ab715f3138d96ef1da979b2977
+  size_bytes: 107235
+commercial:
+  annual_fee: null
+display:
+  palette_seed: amber-steel-llm
+  art_style: hardware-editorial
+  accent_priority: high
+image_path: ../image/012_aisb.t3.012_efficientqat.jpg

package/AISB/catalog/aisb.t3.013_appl.yaml

@@ -0,0 +1,152 @@
+schema_version: 1
+id: aisb.t3.013_appl
+name: 'APPL: A Prompt Programming Language for Harmonious Integration of Programs
+  and Large Language Model Prompts'
+version: 0.1.0
+one_line: Measure and optimize the AST compactness of APPL prompt-programming constructs
+  by running a code-backed evaluator on canonical code snippets from the paper.
+task_description: 'This benchmark evaluates the structural compactness of APPL, a
+  Python-native prompt programming language that seamlessly integrates LLM calls into
+  conventional programs. The primary task is to optimize the APPL runtime and language
+  constructs so that canonical prompt-program patterns (e.g., Chain-of-Thought with
+  self-consistency from Figure 7a of the paper) yield smaller Python ASTs, measured
+  by counting AST nodes via the bundled eval_ast_size.py script. The evaluator parses
+  a fixed code snippet with Python''s ast module and reports the total node count.
+  No model training is involved; the benchmark is CPU-only and focuses on compiler-style
+  simplification and language design efficiency. An LLM API key (e.g., OpenAI) is
+  required only if you wish to exercise the APPL runtime end-to-end beyond the AST
+  metric; the core AST evaluation itself needs no external API.
+
+  '
+capability_tags:
+- research_code_optimization
+- prompt_programming
+- software_language_tools
+- llm_tooling
+- python
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: turnkey
+cost_band: low
+time_band: 30-60m
+difficulty: medium
+data_access: public
+primary_outputs:
+- ast_size
+- runtime_output
+launch_profiles:
+- id: quick_check
+  label: Quick Check
+  description: 'Run eval_ast_size.py to compute the AST node count of the canonical
+    CoT-SC snippet. No LLM API key needed. Completes in seconds.
+
+    '
+- id: ast_eval
+  label: AST Eval
+  description: 'Run the full compactness-focused APPL evaluation workflow, including
+    any optimizations applied to the APPL language constructs, and report the resulting
+    ast_size metric.
+
+    '
+- id: runtime_exercise
+  label: Runtime Exercise
+  description: 'Install applang, configure an LLM backend (e.g., OpenAI), and run
+    the bundled examples to verify end-to-end runtime behavior alongside the AST metric.
+
+    '
+dataset_download:
+  primary_method: self_contained
+  sources: []
+  notes:
+  - No external dataset download is required. The evaluation code snippet is embedded
+    in eval_ast_size.py.
+  - The full APPL library source is included in the snapshot under src/appl/.
+credential_requirements:
+  mode: optional
+  items:
+  - OpenAI API key (only for end-to-end runtime exercises, not for the core AST metric)
+  notes:
+  - The primary ast_size metric requires no credentials or network access.
+  - Set OPENAI_API_KEY in .env or as an environment variable if exercising runtime
+    examples.
+  - Other LLM backends supported via litellm may require their own API keys.
+resources:
+  minimum:
+    cpu_cores: 4
+    ram_gb: 8
+    disk_gb: 10
+    gpu_count: 0
+    gpu_vram_gb: 0
+  recommended:
+    cpu_cores: 8
+    ram_gb: 16
+    disk_gb: 20
+    gpu_count: 0
+    gpu_vram_gb: 0
+environment:
+  python: '3.9'
+  cuda: null
+  pytorch: null
+  flash_attn: null
+  key_packages:
+  - applang>=0.2.2
+  - litellm>=1.59.8
+  - openai>=1.13.3
+  - pydantic>=2.6.3
+  - libcst>=1.4.0
+  notes:
+  - CPU-only execution is sufficient for the core AST evaluation metric.
+  - Python 3.9+ required; tested through 3.13.
+  - Install via pip install -U applang or from the bundled pyproject.toml with pdm.
+  - See pyproject.toml for the full dependency set including optional extras (lunary,
+    instructor, langfuse).
+risk_flags:
+- optional_api_dependency
+risk_notes:
+- The core ast_size metric is fully self-contained and reproducible without any external
+  service.
+- End-to-end runtime exercises require a working LLM API (OpenAI or other litellm-supported
+  backend), which incurs API costs and introduces non-determinism in generated text.
+- No benchmark execution was performed during the packaging pass; metric values should
+  be verified by running eval_ast_size.py.
+recommended_when: 'Use this benchmark when you want a software-oriented LLM task that
+  emphasizes runtime behavior, prompt-program structure, and code compactness rather
+  than GPU training. Suitable for evaluating compiler-style optimizations to prompt
+  programming language constructs.
+
+  '
+not_recommended_when: 'Do not use this if you are looking for a heavy model-training
+  benchmark, a purely offline task with no LLM backend integration path, or a benchmark
+  that measures model accuracy on downstream NLP tasks.
+
+  '
+paper:
+  title: 'APPL: A Prompt Programming Language for Harmonious Integration of Programs
+    and Large Language Model Prompts'
+  venue: arXiv preprint
+  year: 2024
+  url: https://arxiv.org/abs/2406.13161
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.013_appl.zip
+  archive_type: zip
+  local_dir_name: paper-13-APPL
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.013_appl.zip
+  sha256: 92f377259bd60724229ee6c61adda761177ee60e47813ebb71dc37f43c0e24f2
+  size_bytes: 2259394
+commercial:
+  annual_fee: null
+display:
+  palette_seed: apple-ink-notebook
+  art_style: language-design
+  accent_priority: medium
+image_path: ../image/013_aisb.t3.013_appl.jpg

package/AISB/catalog/aisb.t3.013_appl.zh.yaml

@@ -0,0 +1,126 @@
+schema_version: 1
+id: aisb.t3.013_appl
+name: 'APPL: A Prompt Programming Language for Harmonious Integration of Programs
+  and Large Language Model Prompts'
+version: 0.1.0
+one_line: 通过对论文中的规范代码片段运行代码支持评估器，测量并优化APPL提示程序结构的AST紧凑性。
+task_description: '该基准测试用于评估APPL（一种原生Python提示编程语言，可将LLM调用无缝集成到传统程序中）的结构紧凑性。主要任务是优化APPL运行时和语言结构，使规范的提示程序模式（例如论文图7a中的带自洽的思维链）生成更小的Python AST，通过捆绑的eval_ast_size.py脚本统计AST节点数量来测量。评估器使用Python的ast模块解析固定代码片段并报告总节点数。不涉及模型训练；基准测试仅需CPU，专注于编译器风格的简化和语言设计效率。如果您希望在AST指标之外对APPL运行时进行端到端练习，则需要LLM API密钥（如OpenAI）；核心AST评估本身无需外部API。
+
+  '
+capability_tags:
+- research_code_optimization
+- prompt_programming
+- software_language_tools
+- llm_tooling
+- python
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: evaluation_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: turnkey
+cost_band: low
+time_band: 30-60m
+difficulty: medium
+data_access: public
+primary_outputs:
+- ast_size
+- runtime_output
+launch_profiles:
+- id: quick_check
+  label: 快速检查
+  description: '运行eval_ast_size.py计算规范CoT-SC代码片段的AST节点数。无需LLM API密钥。几秒内完成。
+
+    '
+- id: ast_eval
+  label: AST评估
+  description: '运行完整的以紧凑性为重点的APPL评估工作流程，包括对APPL语言结构应用的任何优化，并报告最终的ast_size指标。
+
+    '
+- id: runtime_exercise
+  label: 运行时练习
+  description: '安装applang，配置LLM后端（如OpenAI），并运行捆绑的示例以验证端到端运行时行为以及AST指标。
+
+    '
+dataset_download:
+  primary_method: self_contained
+  sources: []
+  notes:
+  - 无需外部数据集下载。评估代码片段已嵌入eval_ast_size.py中。
+  - 完整的APPL库源代码包含在快照的src/appl/目录下。
+credential_requirements:
+  mode: optional
+  items:
+  - OpenAI API密钥（仅用于端到端运行时练习，不用于核心AST指标）
+  notes:
+  - 主要的ast_size指标无需凭据或网络访问。
+  - 如果需要运行运行时示例，请在.env中设置OPENAI_API_KEY或作为环境变量。
+  - 通过litellm支持的其他LLM后端可能需要各自的API密钥。
+resources:
+  minimum:
+    cpu_cores: 4
+    ram_gb: 8
+    disk_gb: 10
+    gpu_count: 0
+    gpu_vram_gb: 0
+  recommended:
+    cpu_cores: 8
+    ram_gb: 16
+    disk_gb: 20
+    gpu_count: 0
+    gpu_vram_gb: 0
+environment:
+  python: '3.9'
+  cuda: null
+  pytorch: null
+  flash_attn: null
+  key_packages:
+  - applang>=0.2.2
+  - litellm>=1.59.8
+  - openai>=1.13.3
+  - pydantic>=2.6.3
+  - libcst>=1.4.0
+  notes:
+  - 纯CPU执行足以完成核心AST评估指标。
+  - 需要Python 3.9+；已测试至3.13版本。
+  - 可通过pip install -U applang安装，或使用pdm从捆绑的pyproject.toml安装。
+  - 完整的依赖项集合（包括可选的lunary、instructor、langfuse）请参见pyproject.toml。
+risk_flags:
+- optional_api_dependency
+risk_notes:
+- 核心ast_size指标完全自包含，无需任何外部服务即可复现。
+- 端到端运行时练习需要可用的LLM API（OpenAI或其他litellm支持的后端），会产生API费用，并在生成的文本中引入非确定性。
+- 打包过程中未执行基准测试；应通过运行eval_ast_size.py来验证指标值。
+recommended_when: '当您需要一个强调运行时行为、提示程序结构和代码紧凑性而非GPU训练的面向软件的LLM任务时使用此基准测试。适用于评估对提示编程语言结构的编译器风格优化。
+
+  '
+not_recommended_when: '如果您正在寻找重型模型训练基准测试、完全离线且无LLM后端集成路径的任务，或测量模型在下游NLP任务上准确性的基准测试，请勿使用此基准测试。
+
+  '
+paper:
+  title: 'APPL: A Prompt Programming Language for Harmonious Integration of Programs
+    and Large Language Model Prompts'
+  venue: arXiv preprint
+  year: 2024
+  url: https://arxiv.org/abs/2406.13161
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.013_appl.zip
+  archive_type: zip
+  local_dir_name: paper-13-APPL
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.013_appl.zip
+  sha256: 92f377259bd60724229ee6c61adda761177ee60e47813ebb71dc37f43c0e24f2
+  size_bytes: 2259394
+commercial:
+  annual_fee: null
+display:
+  palette_seed: apple-ink-notebook
+  art_style: language-design
+  accent_priority: medium
+image_path: ../image/013_aisb.t3.013_appl.jpg

package/AISB/catalog/aisb.t3.014_piguard.yaml

@@ -0,0 +1,207 @@
+schema_version: 1
+id: aisb.t3.014_piguard
+name: 'PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free'
+version: 0.1.0
+one_line: 'Train or fine-tune a DeBERTa-based prompt-injection guard model (PIGuard)
+  and evaluate it on four benchmarks (NotInject, PINT, BIPIA, WildGuard-Benign) measuring
+  malicious detection, benign accuracy, and over-defense accuracy across trigger-word
+  difficulty levels.
+
+  '
+task_description: 'This benchmark packages the full PIGuard pipeline: a lightweight
+  DeBERTa-based binary classifier that detects prompt-injection attacks while minimizing
+  over-defense (false positives on benign inputs containing trigger words like "ignore").
+  The optimization task is to improve injection recall without increasing false positives.
+  Training data from 20 open-source datasets plus LLM-augmented samples is bundled
+  in the snapshot. Evaluation runs across four test sets: NotInject (339 benign samples
+  with 1/2/3 trigger words, split into three difficulty subsets of 113 each), BIPIA
+  (malicious), WildGuard-Benign, and PINT. The PINT benchmark requires a separate
+  access request. Three evaluation scripts are provided: eval.py (loads local checkpoint
+  via PIGuard.py), eval_piguard.py (loads HuggingFace-format weights from /tmp/PIGuard_weights),
+  and run_official_eval.py (same but with a custom benign-score threshold of 0.10).
+  Primary metrics are over-defense accuracy on NotInject (overall and per-trigger-count),
+  malicious accuracy on PINT/BIPIA, and benign accuracy on PINT/WildGuard-Benign.
+  The model weights can be loaded from HuggingFace (leolee99/PIGuard) or from a Google
+  Drive checkpoint.
+
+  '
+capability_tags:
+- research_code_optimization
+- llm_security
+- classification
+- prompt_injection
+- robustness
+aisb_direction: T3
+track_fit:
+- paper_track
+- benchmark_track
+task_mode: experiment_driven
+requires_execution: true
+requires_paper: true
+integrity_level: cas_plus_canary
+snapshot_status: runnable
+support_level: turnkey
+cost_band: medium
+time_band: 6-24h
+difficulty: medium
+data_access: restricted
+primary_outputs:
+- over_defense_accuracy
+- one_trigger_accuracy
+- two_trigger_accuracy
+- three_trigger_accuracy
+- malicious_accuracy
+- benign_accuracy
+- trigger_accuracy_breakdown
+- evaluation_report
+launch_profiles:
+- id: quick_check
+  label: Quick Check (HuggingFace weights)
+  description: 'Run eval_hf.py or eval_piguard.py to evaluate the pretrained PIGuard
+    model from HuggingFace on all bundled test sets (NotInject, BIPIA, WildGuard-Benign).
+    No training needed. PINT evaluation is skipped unless you have obtained access
+    separately.
+
+    '
+- id: full_eval
+  label: Full Eval (all four benchmarks)
+  description: 'Run eval.py --resume <checkpoint> or run_official_eval.py across NotInject,
+    BIPIA, WildGuard-Benign, and PINT. Requires PINT access request and YAML-to-JSON
+    conversion via util.py.
+
+    '
+- id: train_and_eval
+  label: Train + Evaluate
+  description: 'Run train.py to retrain PIGuard with the MOF strategy on the bundled
+    20-source training set, then evaluate the resulting checkpoint on all four benchmarks.
+    Expect 6-24h on a single GPU depending on VRAM and batch size.
+
+    '
+dataset_download:
+  primary_method: mixed
+  sources:
+  - kind: huggingface
+    url: https://huggingface.co/datasets/leolee99/NotInject
+    access: public
+    note: 'NotInject over-defense evaluation dataset (339 samples, three subsets of
+      113 each). Also bundled in the snapshot under datasets/.
+
+      '
+  - kind: huggingface
+    url: https://huggingface.co/leolee99/PIGuard
+    access: public
+    note: Pretrained PIGuard model weights (DeBERTa-based).
+  - kind: google_drive
+    url: https://drive.google.com/file/d/1JpiVb_wtnbBLNEjIx1KS7PHuvmARQKTu/view?usp=sharing
+    access: public
+    note: Alternative checkpoint download.
+  - kind: external
+    url: https://share-eu1.hsforms.com/1TwiBEvLXRrCjJSdnbnHpLwfdfs3
+    access: request_required
+    note: 'PINT benchmark from Lakera AI. Not public; requires filling out an access
+      request form. Must convert from YAML to JSON via util.py after download.
+
+      '
+  - kind: bundled
+    url: null
+    access: public
+    note: 'Training data (20 open-source datasets + LLM augmentations), validation
+      set (144 samples), and test sets (NotInject, BIPIA, WildGuard-Benign) are all
+      bundled under PIGuard/datasets/.
+
+      '
+  notes:
+  - Total bundled data is modest (tens of MB). No large-scale download required for
+    most profiles.
+  - PINT is the only external dataset that requires a separate access request.
+credential_requirements:
+  mode: none
+  items: []
+  notes:
+  - PINT benchmark access requires a form submission to Lakera AI, but no API key.
+  - HuggingFace model download is public; no token needed.
+resources:
+  minimum:
+    cpu_cores: 8
+    ram_gb: 32
+    disk_gb: 50
+    gpu_count: 1
+    gpu_vram_gb: 12
+  recommended:
+    cpu_cores: 16
+    ram_gb: 64
+    disk_gb: 120
+    gpu_count: 1
+    gpu_vram_gb: 24
+environment:
+  python: '3.10'
+  cuda: '11.8'
+  pytorch: 2.4.0
+  flash_attn: 2.6.1
+  key_packages:
+  - flash-attn==2.6.1
+  - vllm==0.5.4
+  - transformers==4.44.0
+  - ptflops
+  notes:
+  - See the bundled requirements.txt for the full dependency set.
+  - The model is DeBERTa-based (not an LLM); vllm is used only in certain evaluation
+    modes.
+  - flash-attn requires compatible CUDA and GPU architecture.
+risk_flags:
+- restricted_dataset_component
+- external_model_weights
+risk_notes:
+- 'The PINT benchmark is not public and not bundled. Full four-benchmark evaluation
+  requires requesting access from Lakera AI via a web form. Without PINT, three of
+  four test sets are still available.
+
+  '
+- 'Model weights must be downloaded from HuggingFace or Google Drive at runtime unless
+  pre-staged at /tmp/PIGuard_weights.
+
+  '
+- 'eval_piguard.py and run_official_eval.py hardcode MODEL_PATH to /tmp/PIGuard_weights
+  and DATASET_ROOT to /repo/datasets. These paths may need adjustment.
+
+  '
+- 'run_official_eval.py uses a custom BENIGN_THRESHOLD of 0.10 (lowered from 0.50),
+  which changes the classification boundary compared to the default argmax approach
+  in eval.py.
+
+  '
+recommended_when: 'Use this benchmark when you want a security-oriented text classification
+  task with dual pressure: improving prompt-injection detection recall while controlling
+  over-defense (false positives on benign inputs containing attack-like trigger words).
+  Good fit for studying shortcut learning, DeBERTa fine-tuning strategies, and threshold
+  calibration on lightweight guard models.
+
+  '
+not_recommended_when: 'Do not use this if you cannot provide a GPU (the model requires
+  CUDA for practical training and batched inference), if you need the full PINT evaluation
+  but cannot obtain access, or if you are looking for a generative LLM benchmark rather
+  than a binary classification task.
+
+  '
+paper:
+  title: 'PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free'
+  venue: ACL 2025
+  year: 2025
+  url: https://aclanthology.org/2025.acl-long.1468/
+download:
+  url: https://github.com/ResearAI/DeepScientist/releases/download/aisb-v0.0.1/aisb.t3.014_piguard.zip
+  archive_type: zip
+  local_dir_name: paper-14-PIGuard
+  provider: github_release
+  repo: ResearAI/DeepScientist
+  tag: aisb-v0.0.1
+  asset_name: aisb.t3.014_piguard.zip
+  sha256: 9a7996fc3b40709caa357b763d7a51e1f1f2ab449480e403ec34a7c248eb4c81
+  size_bytes: 1621668
+commercial:
+  annual_fee: null
+display:
+  palette_seed: rust-sand-guard
+  art_style: safety-dashboard
+  accent_priority: high
+image_path: ../image/014_aisb.t3.014_piguard.jpg