eduevidence 5.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +506 -0
- package/README.zh-CN.md +434 -0
- package/SKILL.md +195 -0
- package/bin/eduevidence.js +127 -0
- package/domains/education/manifest.json +183 -0
- package/domains/education/outcome_taxonomy.json +127 -0
- package/domains/manifest.json +26 -0
- package/domains/policy/frame.schema.json +234 -0
- package/domains/policy/manifest.json +10 -0
- package/domains/policy/methodology_checklist.json +109 -0
- package/domains/policy/outcome_taxonomy.json +53 -0
- package/domains/policy/references/causal-identification.md +45 -0
- package/domains/policy/references/cost-evidence.md +44 -0
- package/domains/policy/references/equity.md +42 -0
- package/domains/policy/references/evidence-hierarchy.md +41 -0
- package/domains/policy/references/implementation-evidence.md +47 -0
- package/eduevidence_cli.py +26 -0
- package/engine/__init__.py +11 -0
- package/engine/__pycache__/__init__.cpython-312.pyc +0 -0
- package/engine/__pycache__/analysis.cpython-312.pyc +0 -0
- package/engine/__pycache__/bias.cpython-312.pyc +0 -0
- package/engine/__pycache__/briefs.cpython-312.pyc +0 -0
- package/engine/__pycache__/capabilities.cpython-312.pyc +0 -0
- package/engine/__pycache__/citation_check.cpython-312.pyc +0 -0
- package/engine/__pycache__/contracts.cpython-312.pyc +0 -0
- package/engine/__pycache__/datasets.cpython-312.pyc +0 -0
- package/engine/__pycache__/events.cpython-312.pyc +0 -0
- package/engine/__pycache__/evidence_graph.cpython-312.pyc +0 -0
- package/engine/__pycache__/evidence_review.cpython-312.pyc +0 -0
- package/engine/__pycache__/evidencecore.cpython-312.pyc +0 -0
- package/engine/__pycache__/gap_lens.cpython-312.pyc +0 -0
- package/engine/__pycache__/gaps.cpython-312.pyc +0 -0
- package/engine/__pycache__/graph_store.cpython-312.pyc +0 -0
- package/engine/__pycache__/graph_validate.cpython-312.pyc +0 -0
- package/engine/__pycache__/ids.cpython-312.pyc +0 -0
- package/engine/__pycache__/library.cpython-312.pyc +0 -0
- package/engine/__pycache__/library_builtin.cpython-312.pyc +0 -0
- package/engine/__pycache__/living.cpython-312.pyc +0 -0
- package/engine/__pycache__/log.cpython-312.pyc +0 -0
- package/engine/__pycache__/meta_analysis.cpython-312.pyc +0 -0
- package/engine/__pycache__/meta_synthesis.cpython-312.pyc +0 -0
- package/engine/__pycache__/migration.cpython-312.pyc +0 -0
- package/engine/__pycache__/mode_router.cpython-312.pyc +0 -0
- package/engine/__pycache__/paths.cpython-312.pyc +0 -0
- package/engine/__pycache__/pilot.cpython-312.pyc +0 -0
- package/engine/__pycache__/planner.cpython-312.pyc +0 -0
- package/engine/__pycache__/project.cpython-312.pyc +0 -0
- package/engine/__pycache__/projections.cpython-312.pyc +0 -0
- package/engine/__pycache__/robustness.cpython-312.pyc +0 -0
- package/engine/__pycache__/run.cpython-312.pyc +0 -0
- package/engine/__pycache__/semantics.cpython-312.pyc +0 -0
- package/engine/__pycache__/study_design.cpython-312.pyc +0 -0
- package/engine/__pycache__/synthesis.cpython-312.pyc +0 -0
- package/engine/__pycache__/tribunal.cpython-312.pyc +0 -0
- package/engine/__pycache__/update.cpython-312.pyc +0 -0
- package/engine/__pycache__/versions.cpython-312.pyc +0 -0
- package/engine/analysis.py +308 -0
- package/engine/bias.py +178 -0
- package/engine/briefs.py +106 -0
- package/engine/capabilities.py +99 -0
- package/engine/citation_check.py +192 -0
- package/engine/contracts.py +117 -0
- package/engine/datasets.py +165 -0
- package/engine/events.py +67 -0
- package/engine/evidence_graph.py +571 -0
- package/engine/evidence_review.py +88 -0
- package/engine/evidencecore.py +182 -0
- package/engine/gap_lens.py +132 -0
- package/engine/gaps.py +169 -0
- package/engine/graph_store.py +335 -0
- package/engine/graph_validate.py +87 -0
- package/engine/ids.py +77 -0
- package/engine/library.py +268 -0
- package/engine/library_builtin.py +301 -0
- package/engine/living.py +671 -0
- package/engine/log.py +39 -0
- package/engine/meta_analysis.py +333 -0
- package/engine/meta_synthesis.py +111 -0
- package/engine/migration.py +397 -0
- package/engine/mode_router.py +72 -0
- package/engine/paths.py +15 -0
- package/engine/pilot.py +368 -0
- package/engine/planner.py +126 -0
- package/engine/project.py +118 -0
- package/engine/projections.py +240 -0
- package/engine/robustness.py +109 -0
- package/engine/run.py +85 -0
- package/engine/semantics.py +135 -0
- package/engine/study_design.py +87 -0
- package/engine/synthesis.py +187 -0
- package/engine/tribunal.py +408 -0
- package/engine/update.py +113 -0
- package/engine/versions.py +12 -0
- package/install.sh +510 -0
- package/integrations/__init__.py +1 -0
- package/integrations/__pycache__/__init__.cpython-312.pyc +0 -0
- package/integrations/__pycache__/agent_mcp.cpython-312.pyc +0 -0
- package/integrations/__pycache__/smart_web_fetch.cpython-312.pyc +0 -0
- package/integrations/agent_mcp.py +856 -0
- package/integrations/smart_web_fetch.py +59 -0
- package/package.json +50 -0
- package/pyproject.toml +55 -0
- package/references/applicability-policy.md +88 -0
- package/references/education-framing.md +132 -0
- package/references/effect_size_formulas.md +35 -0
- package/references/evaluation-design.md +111 -0
- package/references/evidence-quality.md +79 -0
- package/references/grade_framework.md +29 -0
- package/references/intervention-design.md +98 -0
- package/references/methodology-audit.md +103 -0
- package/references/outcome-taxonomy.md +106 -0
- package/references/retrieval-protocol.md +142 -0
- package/references/skeptic-protocol.md +93 -0
- package/references/social_science_pitfalls.md +48 -0
- package/references/source-validity.md +140 -0
- package/references/tribunal-policy.md +112 -0
- package/references/wwc_standards.md +29 -0
- package/retrieval/__init__.py +1 -0
- package/retrieval/__pycache__/__init__.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/corpus_store.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/dedupe.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/failures.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/fetch.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/search.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/source.cpython-312.pyc +0 -0
- package/retrieval/__pycache__/validate.cpython-312.pyc +0 -0
- package/retrieval/corpus_store.py +181 -0
- package/retrieval/dedupe.py +127 -0
- package/retrieval/failures.py +90 -0
- package/retrieval/fetch.py +435 -0
- package/retrieval/search.py +493 -0
- package/retrieval/source.py +160 -0
- package/retrieval/validate.py +257 -0
- package/schemas/agent-mcp-approval.schema.json +57 -0
- package/schemas/chart-spec.schema.json +88 -0
- package/schemas/cross-model-review.schema.json +28 -0
- package/schemas/education-frame.schema.json +106 -0
- package/schemas/evaluation.schema.json +35 -0
- package/schemas/evidence.schema.json +81 -0
- package/schemas/fetch-result.schema.json +119 -0
- package/schemas/intervention.schema.json +46 -0
- package/schemas/methodology.schema.json +102 -0
- package/schemas/report-result.schema.json +381 -0
- package/schemas/report-spec.schema.json +130 -0
- package/schemas/source.schema.json +311 -0
- package/schemas/v2/analysis-plan.schema.json +28 -0
- package/schemas/v2/analysis-run.schema.json +33 -0
- package/schemas/v2/claim.schema.json +26 -0
- package/schemas/v2/dataset-asset.schema.json +40 -0
- package/schemas/v2/decision-snapshot.schema.json +53 -0
- package/schemas/v2/evidence-link.schema.json +38 -0
- package/schemas/v2/finding.schema.json +47 -0
- package/schemas/v2/graph-revision.schema.json +30 -0
- package/schemas/v2/knowledge-gap.schema.json +40 -0
- package/schemas/v2/methodology-audit.schema.json +30 -0
- package/schemas/v2/outcome.schema.json +18 -0
- package/schemas/v2/project.schema.json +31 -0
- package/schemas/v2/research-intent.schema.json +24 -0
- package/schemas/v2/run.schema.json +43 -0
- package/schemas/v2/source.schema.json +24 -0
- package/schemas/v2/study-design.schema.json +67 -0
- package/schemas/v2/study.schema.json +37 -0
- package/schemas/v3/pilot-outcome.schema.json +132 -0
- package/schemas/v3/run-manifest.schema.json +193 -0
- package/schemas/v3/synthesis.schema.json +86 -0
- package/schemas/v4/drift-report.schema.json +66 -0
- package/schemas/v4/evidence-library.schema.json +42 -0
- package/schemas/v4/living-subscription.schema.json +55 -0
- package/schemas/v4/meta-analysis.schema.json +152 -0
- package/schemas/verdict.schema.json +56 -0
- package/scripts/__init__.py +0 -0
- package/scripts/__pycache__/__init__.cpython-312.pyc +0 -0
- package/scripts/__pycache__/benchmark.cpython-312.pyc +0 -0
- package/scripts/__pycache__/benchmark_evaluator.cpython-312.pyc +0 -0
- package/scripts/__pycache__/benchmark_judge.cpython-312.pyc +0 -0
- package/scripts/__pycache__/benchmark_routing.cpython-312.pyc +0 -0
- package/scripts/__pycache__/benchmark_v2.cpython-312.pyc +0 -0
- package/scripts/__pycache__/benchmark_v3.cpython-312.pyc +0 -0
- package/scripts/__pycache__/build_result.cpython-312.pyc +0 -0
- package/scripts/__pycache__/claim_audit.cpython-312.pyc +0 -0
- package/scripts/__pycache__/complexity_gate.cpython-312.pyc +0 -0
- package/scripts/__pycache__/compute_confidence.cpython-312.pyc +0 -0
- package/scripts/__pycache__/dashboard_server.cpython-312.pyc +0 -0
- package/scripts/__pycache__/did_regression.cpython-312.pyc +0 -0
- package/scripts/__pycache__/effect_calculator.cpython-312.pyc +0 -0
- package/scripts/__pycache__/evidence_matrix.cpython-312.pyc +0 -0
- package/scripts/__pycache__/evidence_score.cpython-312.pyc +0 -0
- package/scripts/__pycache__/evidence_semantics.cpython-312.pyc +0 -0
- package/scripts/__pycache__/fetch_benchmark.cpython-312.pyc +0 -0
- package/scripts/__pycache__/lint_report_layout.cpython-312.pyc +0 -0
- package/scripts/__pycache__/orchestrator.cpython-312.pyc +0 -0
- package/scripts/__pycache__/pre_verdict_gate.cpython-312.pyc +0 -0
- package/scripts/__pycache__/recompute_demo_quality.cpython-312.pyc +0 -0
- package/scripts/__pycache__/render_report.cpython-312.pyc +0 -0
- package/scripts/__pycache__/render_report_html.cpython-312.pyc +0 -0
- package/scripts/__pycache__/run_workspace.cpython-312.pyc +0 -0
- package/scripts/__pycache__/skill_lint.cpython-312.pyc +0 -0
- package/scripts/__pycache__/startup_probe.cpython-312.pyc +0 -0
- package/scripts/__pycache__/sync_killer_demo_report.cpython-312.pyc +0 -0
- package/scripts/__pycache__/test_adversarial_empirical.cpython-312-pytest-9.0.2.pyc +0 -0
- package/scripts/__pycache__/test_adversarial_empirical.cpython-312-pytest-9.1.1.pyc +0 -0
- package/scripts/__pycache__/validate_schema.cpython-312.pyc +0 -0
- package/scripts/audit_dois.py +292 -0
- package/scripts/bake_pack.sh +37 -0
- package/scripts/benchmark.py +183 -0
- package/scripts/benchmark_evaluator.py +371 -0
- package/scripts/benchmark_judge.py +535 -0
- package/scripts/benchmark_routing.py +120 -0
- package/scripts/benchmark_v2.py +304 -0
- package/scripts/benchmark_v3.py +552 -0
- package/scripts/build_esl_artifacts.py +1921 -0
- package/scripts/build_evidence_library.py +307 -0
- package/scripts/build_killer_demo.py +295 -0
- package/scripts/build_result.py +311 -0
- package/scripts/check_version_consistency.py +96 -0
- package/scripts/citation_check.py +123 -0
- package/scripts/claim_audit.py +157 -0
- package/scripts/complexity_gate.py +180 -0
- package/scripts/compute_confidence.py +176 -0
- package/scripts/dashboard_server.py +536 -0
- package/scripts/did_regression.py +315 -0
- package/scripts/effect_calculator.py +99 -0
- package/scripts/enrich_projects_human_and_lieflat.py +315 -0
- package/scripts/evidence_matrix.py +129 -0
- package/scripts/evidence_score.py +234 -0
- package/scripts/evidence_semantics.py +87 -0
- package/scripts/fetch_benchmark.py +177 -0
- package/scripts/generate_metrics.py +99 -0
- package/scripts/generate_new_projects.py +686 -0
- package/scripts/generate_promo.py +192 -0
- package/scripts/lint_report_layout.py +182 -0
- package/scripts/orchestrator.py +1456 -0
- package/scripts/pre_verdict_gate.py +513 -0
- package/scripts/quickstart.py +121 -0
- package/scripts/rebake_all_5themes.py +88 -0
- package/scripts/recompute_demo_quality.py +205 -0
- package/scripts/render_report.py +270 -0
- package/scripts/render_report_html.py +356 -0
- package/scripts/retraction_watch.py +110 -0
- package/scripts/run_workspace.py +337 -0
- package/scripts/serve_web.py +54 -0
- package/scripts/skill_lint.py +150 -0
- package/scripts/startup_probe.py +265 -0
- package/scripts/sync_killer_demo_report.py +270 -0
- package/scripts/test_adversarial_empirical.py +541 -0
- package/scripts/validate_schema.py +256 -0
- package/skill/agents/education-planner.md +80 -0
- package/skill/agents/evaluation-designer.md +74 -0
- package/skill/agents/evidence-analyst.md +106 -0
- package/skill/agents/evidence-judge.md +111 -0
- package/skill/agents/evidence-retriever.md +80 -0
- package/skill/agents/intervention-designer.md +82 -0
- package/skill/agents/method-reviewer.md +104 -0
- package/skill/agents/skeptic.md +89 -0
- package/skill/sub-skills/aihot-trend-analysis/SKILL.md +31 -0
- package/skill/sub-skills/contradiction-analysis/SKILL.md +17 -0
- package/skill/sub-skills/data-analysis/SKILL.md +23 -0
- package/skill/sub-skills/ethics-review/SKILL.md +25 -0
- package/skill/sub-skills/evidence-extraction/SKILL.md +19 -0
- package/skill/sub-skills/evidence-review/SKILL.md +18 -0
- package/skill/sub-skills/gap-analysis/SKILL.md +25 -0
- package/skill/sub-skills/literature-review/SKILL.md +21 -0
- package/skill/sub-skills/methodology-audit/SKILL.md +20 -0
- package/skill/sub-skills/report-generation/SKILL.md +51 -0
- package/skill/sub-skills/research-planning/SKILL.md +21 -0
- package/skill/sub-skills/study-design/SKILL.md +16 -0
- package/skill/task-briefs/adjudicate.md +17 -0
- package/skill/task-briefs/audit.md +15 -0
- package/skill/task-briefs/challenge.md +15 -0
- package/skill/task-briefs/evaluate.md +13 -0
- package/skill/task-briefs/extract.md +16 -0
- package/skill/task-briefs/frame.md +17 -0
- package/skill/task-briefs/intervene.md +14 -0
- package/skill/task-briefs/present.md +16 -0
- package/skill/task-briefs/retrieve.md +15 -0
- package/visualization/eduevidence-report/assets/base.css +337 -0
- package/visualization/eduevidence-report/motion/motion.css +157 -0
- package/visualization/eduevidence-report/motion/motion.js +107 -0
- package/visualization/eduevidence-report/references/bilingual-style.md +7 -0
- package/visualization/eduevidence-report/references/component-catalog.md +145 -0
- package/visualization/eduevidence-report/references/evidence-expansion.md +65 -0
- package/visualization/eduevidence-report/references/full-report-outline.md +86 -0
- package/visualization/eduevidence-report/references/layout-constraints.md +63 -0
- package/visualization/eduevidence-report/references/lieflat-composition.md +79 -0
- package/visualization/eduevidence-report/references/motion-system.md +31 -0
- package/visualization/eduevidence-report/schemas/adapter-envelope.schema.json +22 -0
- package/visualization/eduevidence-report/schemas/visual-layout.schema.json +87 -0
- package/visualization/eduevidence-report/scripts/__pycache__/adapter_contract.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/build_artifact_manifest.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/build_charts.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/build_figures.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/build_infographics.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/build_report.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/charts_data.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/lieflat_engine.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/__pycache__/zh_labels.cpython-312.pyc +0 -0
- package/visualization/eduevidence-report/scripts/adapter_contract.py +72 -0
- package/visualization/eduevidence-report/scripts/build_artifact_manifest.py +70 -0
- package/visualization/eduevidence-report/scripts/build_charts.py +283 -0
- package/visualization/eduevidence-report/scripts/build_figures.py +515 -0
- package/visualization/eduevidence-report/scripts/build_infographics.py +268 -0
- package/visualization/eduevidence-report/scripts/build_report.py +3211 -0
- package/visualization/eduevidence-report/scripts/charts_data.py +617 -0
- package/visualization/eduevidence-report/scripts/check_mobile_layout.js +220 -0
- package/visualization/eduevidence-report/scripts/lieflat_engine.py +852 -0
- package/visualization/eduevidence-report/scripts/zh_labels.py +245 -0
- package/visualization/eduevidence-report/themes/academic.css +94 -0
- package/visualization/eduevidence-report/themes/claude.css +96 -0
- package/visualization/eduevidence-report/themes/datalab-dark.css +147 -0
- package/visualization/eduevidence-report/themes/datalab.css +151 -0
- package/visualization/eduevidence-report/themes/presentation.css +140 -0
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# Intervention Design(干预设计)
|
|
2
|
+
|
|
3
|
+
## 1. 最小可验证试点原则
|
|
4
|
+
|
|
5
|
+
**不允许"直接全面部署"**。任何新的 AI 使用政策必须先做小规模、有时间上限的试点:
|
|
6
|
+
|
|
7
|
+
- 范围小:1–2 个班(或 ≤ 120 人),覆盖目标 learner 的代表性子集;
|
|
8
|
+
- 周期短:4–12 周,有明确结束节点;
|
|
9
|
+
- 目标明确:只验证一个 decision(如"是否允许某种 AI 用法");
|
|
10
|
+
- 全程测量:试点期间必须采集 C1/C2/C3/C4 指标(见 outcome-taxonomy.md);
|
|
11
|
+
- 有出口:达到 stop_conditions 立即暂停或回退。
|
|
12
|
+
|
|
13
|
+
> 试点不是"试试看",而是带有预先注册假设与退出条件的受控实验。
|
|
14
|
+
|
|
15
|
+
## 2. TeachingIntervention 字段
|
|
16
|
+
|
|
17
|
+
| 字段 | 说明 |
|
|
18
|
+
| --- | --- |
|
|
19
|
+
| `decision` | 本干预要支撑的决策(必须与 Frame 的 decision_target 一致) |
|
|
20
|
+
| `target_learners` | 试点对象(班级、层次、先验水平) |
|
|
21
|
+
| `learning_goals` | 2–3 条可测量的学习目标(绑定 C1 指标) |
|
|
22
|
+
| `pilot_duration` | 试点总时长与各阶段时长 |
|
|
23
|
+
| `phase_1/2/3` | 阶段化设计:每阶段的 目标 / AI 用法 / 教师角色 / 学生任务 |
|
|
24
|
+
| `ai_usage_policy` | 每阶段允许与禁止的 AI 用法(允许使用逐条列出) |
|
|
25
|
+
| `teacher_role` | 教师在各阶段做什么(讲解、监督、反馈) |
|
|
26
|
+
| `student_role` | 学生各阶段的任务与责任 |
|
|
27
|
+
| `reflection_requirement` | 学生需提交的反思(如每周解释自己的解题思路、标注 AI 使用部分) |
|
|
28
|
+
| `assessment` | 过程评估与结果评估的测量方式与时间点 |
|
|
29
|
+
| `risk_control` | 风险指标清单与监控频率 |
|
|
30
|
+
| `stop_conditions` | 触发暂停/回退的具体阈值条件 |
|
|
31
|
+
|
|
32
|
+
## 3. 示例:C语言课 4 阶段试点(8 周,2 个班各 60 人)
|
|
33
|
+
|
|
34
|
+
### 阶段总览
|
|
35
|
+
|
|
36
|
+
| 阶段 | 时间 | 主题 | AI 用法 |
|
|
37
|
+
| --- | --- | --- | --- |
|
|
38
|
+
| Phase 1 | 第 1–2 周 | Independent Foundation | 禁止使用 AI |
|
|
39
|
+
| Phase 2 | 第 3–4 周 | Explain Don't Solve | 允许解释报错,禁止生成代码 |
|
|
40
|
+
| Phase 3 | 第 5–6 周 | Structured Collaboration | 允许局部生成,必须解释关键逻辑 |
|
|
41
|
+
| Phase 4 | 第 7–8 周 | Transfer Check | 无 AI 新任务(评估迁移) |
|
|
42
|
+
|
|
43
|
+
### Phase 1 Independent Foundation(禁止完整代码生成)
|
|
44
|
+
|
|
45
|
+
- **目标**:学生独立掌握 顺序/分支/循环 三大语法,形成基础能力基线。
|
|
46
|
+
- **AI 用法**:实验课全程禁用 AI;作业同样禁用。
|
|
47
|
+
- **教师角色**:逐个检查上机基础操作;对卡壳学生给提示不给代码。
|
|
48
|
+
- **学生任务**:独立完成 4 个基础编程作业;提交时附上思路草稿。
|
|
49
|
+
- **评估**:第 2 周末基线测验(无 AI),记录 `independent_problem_solving` 基线。
|
|
50
|
+
|
|
51
|
+
### Phase 2 Explain Don't Solve(允许解释报错)
|
|
52
|
+
|
|
53
|
+
- **目标**:学习"读错误信息 + 定位问题"的诊断技能。
|
|
54
|
+
- **AI 用法**:仅允许粘贴报错信息让 AI 解释错误含义;**禁止**让 AI 给出修复代码或完整解决方案。
|
|
55
|
+
- **教师角色**:抽查学生是否越界使用;示范"报错解读 → 自己改"的流程。
|
|
56
|
+
- **学生任务**:每道错题先自行尝试 15 分钟,再使用"解释报错"通道。
|
|
57
|
+
- **评估**:记录 help_seeking 比例与无 AI 诊断小测。
|
|
58
|
+
|
|
59
|
+
### Phase 3 Structured Collaboration(允许局部生成须解释关键逻辑)
|
|
60
|
+
|
|
61
|
+
- **目标**:在受控范围内体验协作式开发,同时保持理解责任在学生。
|
|
62
|
+
- **AI 用法**:允许 AI 生成**局部代码片段**(单个函数体);但提交时必须在每个 AI 生成处
|
|
63
|
+
标注,并用自然语言解释该段的关键逻辑(如"为什么用指针传参")。
|
|
64
|
+
- **教师角色**:批改时重点检查"AI 标注 + 解释"是否真实,随机抽问解释。
|
|
65
|
+
- **学生任务**:完成 2 个综合作业(数组、函数、指针应用),附 AI 使用清单。
|
|
66
|
+
- **评估**:记录 `code_quality`、`reflection_requirement` 完成率。
|
|
67
|
+
|
|
68
|
+
### Phase 4 Transfer Check(无AI新任务)
|
|
69
|
+
|
|
70
|
+
- **目标**:检验迁移能力与依赖程度(对应 C1/transfer 与 C4/ai_dependency)。
|
|
71
|
+
- **AI 用法**:全禁 AI;任务为**新题型/新语言环境**(如改用 C++ 实现同样算法)中的全新题目,
|
|
72
|
+
与训练题不重复。
|
|
73
|
+
- **教师角色**:监考式无 AI 测验;统计 AI 撤除后的表现对比。
|
|
74
|
+
- **学生任务**:完成迁移测验 + 依赖自评问卷。
|
|
75
|
+
- **评估**:`transfer_test` 成绩与 Phase 1 基线对比;C4 指标报告。
|
|
76
|
+
|
|
77
|
+
### 阶段公共字段
|
|
78
|
+
|
|
79
|
+
- `ai_usage_policy`(汇总):Phase1 禁用;Phase2 仅报错解释;Phase3 局部生成+强制解释;
|
|
80
|
+
Phase4 禁用。所有阶段禁止用 AI 完成整份作业或生成答案文本。
|
|
81
|
+
- `reflection_requirement`:每阶段结束提交 200 字反思,回答"我用了 AI 吗?用了哪一步?
|
|
82
|
+
删掉 AI 我还能做吗?"。
|
|
83
|
+
- `risk_control`:每周汇总 C4 指标(ai_dependency / over_reliance / false_confidence)。
|
|
84
|
+
- `stop_conditions`(任一触发即暂停,退回上一阶段或中止试点):
|
|
85
|
+
1. 无 AI 独立题正确率连续 2 周低于 Phase 1 基线 15% 以上;
|
|
86
|
+
2. 代写检测命中率 > 10%;
|
|
87
|
+
3. 超过 40% 学生在 Phase 2 越界使用完整代码生成;
|
|
88
|
+
4. 学生虚假自信自评(高自信 + 低实绩)比例超过 30%。
|
|
89
|
+
|
|
90
|
+
## 4. 试点执行规则
|
|
91
|
+
|
|
92
|
+
| 编号 | 规则 |
|
|
93
|
+
| --- | --- |
|
|
94
|
+
| ID-01 | 试点必须预先写明 stop_conditions 并在每阶段检查,禁止事后补写。 |
|
|
95
|
+
| ID-02 | 每个阶段结束做一次 mini-evaluation(≤ 20 分钟),数据进 Evidence Matrix。 |
|
|
96
|
+
| ID-03 | 试点期间禁止同时调整教材、教师、工具等多变量;一次只变 AI 用法。 |
|
|
97
|
+
| ID-04 | 试点结果无论正负都需走 skeptic-protocol.md 的 9 项检查后再决定是否扩大。 |
|
|
98
|
+
| ID-05 | 全面部署只有在试点满足 success_condition 且适用性分析(applicability-policy.md)通过后才能讨论。 |
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# Methodology Audit(方法学审查清单)
|
|
2
|
+
|
|
3
|
+
## 1. 最高优先级规则
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
任务完成表现不能自动等价为学习效果
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
**Rule:** 任何研究若仅以"AI 辅助下的任务完成度/正确率/速度"作为判断学习效果的依据,
|
|
10
|
+
该证据的方法学权重直接下调一档,且结论只能表述为"任务表现改善",不得表述为"学习效果改善"。
|
|
11
|
+
审查结果中此项单独标注 `HIGH-PRIORITY-VIOLATION` 或 `HIGH-PRIORITY-PASS`。
|
|
12
|
+
|
|
13
|
+
## 2. 审查项清单
|
|
14
|
+
|
|
15
|
+
对每条证据逐项审查,每项给出 `通过 / 未通过 / 不适用` + 证据摘录 + 对结论的影响。
|
|
16
|
+
|
|
17
|
+
### 2.1 实验设计类
|
|
18
|
+
|
|
19
|
+
| 审查项 | 审查问题 | 通过标准 |
|
|
20
|
+
| --- | --- | --- |
|
|
21
|
+
| `control_group` | 有无对照组?对照组是否接受等量教学时间? | 有对照且时间/作业量平衡 |
|
|
22
|
+
| `randomization` | 分组是否随机?是否为自选/报名? | 随机或准随机;自选视为失败 |
|
|
23
|
+
| `pre_test` | 是否测量了干预前基线? | 有前测且报告基线可比性 |
|
|
24
|
+
| `post_test` | 是否有干预后即时测量? | 有标准化后测 |
|
|
25
|
+
| `retention_test` | 是否有延迟保持测量(≥ 2 周)? | 有;缺失则时间强度降档 |
|
|
26
|
+
| `transfer_test` | 是否有无 AI / 新情境迁移测量? | 有;缺失则无法支持迁移结论 |
|
|
27
|
+
|
|
28
|
+
### 2.2 样本类
|
|
29
|
+
|
|
30
|
+
| 审查项 | 审查问题 | 通过标准 |
|
|
31
|
+
| --- | --- | --- |
|
|
32
|
+
| `sample_bias` | 样本是否代表目标 learner? | 教育层次/专业/先验知识与目标一致 |
|
|
33
|
+
| `self_selection` | 参与者是否自行选择组别? | 无自选;自选需报告动机差异并做敏感性分析 |
|
|
34
|
+
|
|
35
|
+
### 2.3 测量类
|
|
36
|
+
|
|
37
|
+
| 审查项 | 审查问题 | 通过标准 |
|
|
38
|
+
| --- | --- | --- |
|
|
39
|
+
| `measurement_validity` | 测量是否真的测到了声称的 Outcome(见 outcome-taxonomy.md)? | 指标与结论对应;未把 C2 当 C1 |
|
|
40
|
+
|
|
41
|
+
### 2.4 混淆因素类
|
|
42
|
+
|
|
43
|
+
| 审查项 | 审查问题 | 通过标准 |
|
|
44
|
+
| --- | --- | --- |
|
|
45
|
+
| `confounders` | 有无变量同时影响干预与结果(如课外补习、作业时间差异)? | 识别并控制/统计调整 |
|
|
46
|
+
| `instructor_effect` | 干预与教师是否绑定?不同班是否不同教师? | 教师交叉或教师效应已分析 |
|
|
47
|
+
| `novelty_effect` | 新工具带来的短期新鲜感是否可能解释效果? | 干预 ≥ 8 周 或 有反实验期 |
|
|
48
|
+
| `tool_version_effect` | 工具版本是否固定并报告? | 指定版本与配置 |
|
|
49
|
+
| `ai_usage_policy` | 学生实际使用方式是否符合规定的 allowed_usage? | 有日志/抽查,确认无未授权使用 |
|
|
50
|
+
|
|
51
|
+
### 2.5 完整性类
|
|
52
|
+
|
|
53
|
+
| 审查项 | 审查问题 | 通过标准 |
|
|
54
|
+
| --- | --- | --- |
|
|
55
|
+
| `dropout` | 流失率与流失原因?流失是否与组别相关? | 流失率报告且两组可比较;否则结果有偏 |
|
|
56
|
+
|
|
57
|
+
## 3. 审查输出格式
|
|
58
|
+
|
|
59
|
+
每项固定输出三行:
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
项: control_group
|
|
63
|
+
判定: 通过 | 未通过 | 不适用
|
|
64
|
+
依据: <原文证据摘录 + 原文页码/段落>
|
|
65
|
+
影响: <对结论置信度的具体影响,如"对照组时间不平衡,效应可能来自练习量而非 AI">
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## 4. 综合判定规则
|
|
69
|
+
|
|
70
|
+
| 编号 | 规则 | 后果 |
|
|
71
|
+
| --- | --- | --- |
|
|
72
|
+
| MA-01 | 缺失 `control_group` 或 `randomization` 中的任意一项 | 因果结论降级为相关/描述性结论 |
|
|
73
|
+
| MA-02 | 缺失 `retention_test` | 禁止声称"长期保持" |
|
|
74
|
+
| MA-03 | 缺失 `transfer_test` | 禁止声称"迁移能力"或"无 AI 时也能胜任" |
|
|
75
|
+
| MA-04 | `sample_bias` 或 `self_selection` 未通过 | 外推时必须在 applicability 中标注"样本自选,存在动机偏差" |
|
|
76
|
+
| MA-05 | `measurement_validity` 未通过且属于"把 C2 当 C1" | 触发 HIGH-PRIORITY-VIOLATION,结论只能谈任务表现 |
|
|
77
|
+
| MA-06 | `novelty_effect` 未排除 | 效果时间衰减假设,推广前需更长试点 |
|
|
78
|
+
| MA-07 | `tool_version_effect` 未报告版本 | 结论限定"该版本"或降置信度 |
|
|
79
|
+
| MA-08 | `ai_usage_policy` 未核查实际使用 | 结论限定"按规范使用的前提下" |
|
|
80
|
+
| MA-09 | `dropout` 未报告或两组不平衡 | 视为存在系统性流失,可能高估效果 |
|
|
81
|
+
|
|
82
|
+
## 5. 审查示例
|
|
83
|
+
|
|
84
|
+
> 某研究声称"允许 Copilot 的班级作业成绩显著更高,故 AI 提高学习效果"。
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
control_group : 通过(有平行班)
|
|
88
|
+
randomization : 未通过(实验班为教师主动报名)
|
|
89
|
+
pre_test : 未通过(未做前测,无法排除基线差异)
|
|
90
|
+
post_test : 通过(期末统一试卷)
|
|
91
|
+
retention_test : 未通过(无延迟测量)
|
|
92
|
+
transfer_test : 未通过(考试即平时作业变体,且考场允许 AI)
|
|
93
|
+
measurement_validity: 未通过(作业成绩=AI 辅助任务表现 C2,被当作学习效果 C1)
|
|
94
|
+
→ HIGH-PRIORITY-VIOLATION
|
|
95
|
+
confounders : 未通过(未报告课外使用 AI 时长)
|
|
96
|
+
novelty_effect : 未通过(干预仅 4 周)
|
|
97
|
+
tool_version_effect: 通过(固定 Copilot 版本)
|
|
98
|
+
ai_usage_policy : 未通过(无使用日志)
|
|
99
|
+
dropout : 未报告
|
|
100
|
+
|
|
101
|
+
综合:仅能得出"在自选报名、无前测、允许 AI 考试的情况下,作业成绩更高"这一
|
|
102
|
+
受限描述;禁止得出"AI 提高学习效果"。建议重做 RCT 并补保持/迁移测量。
|
|
103
|
+
```
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# Outcome Taxonomy(学习结果分类)
|
|
2
|
+
|
|
3
|
+
## 0. 三条核心不等式(必须遵守)
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
任务完成 ≠ 学习 (Task completion ≠ Learning)
|
|
7
|
+
短期 ≠ 长期保持 (Short-term ≠ Long-term retention)
|
|
8
|
+
AI协助 ≠ 无AI迁移 (Performance with AI ≠ Transfer without AI)
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
这三条是解读任何教育研究证据的默认假设。任何声称"AI 有效/无效"的结论,
|
|
12
|
+
必须先说明它测量的是哪一类 Outcome,否则结论不可采信。
|
|
13
|
+
|
|
14
|
+
## 1. 四类 Outcome 与指标清单
|
|
15
|
+
|
|
16
|
+
### C1 学习效果(Learning Effects)——回答"学到了什么"
|
|
17
|
+
|
|
18
|
+
| 指标 | 说明 | 测量示例 |
|
|
19
|
+
| --- | --- | --- |
|
|
20
|
+
| `knowledge_gain` | 知识增量 | 前后测知识题得分差 |
|
|
21
|
+
| `concept_understanding` | 概念理解深度 | 概念解释题、错误概念辨识 |
|
|
22
|
+
| `retention` | 长期保持 | 结课 2–4 周后的延迟测验 |
|
|
23
|
+
| `transfer` | 迁移到新情境 | 换语言/换题型/无 AI 环境的新任务 |
|
|
24
|
+
| `independent_problem_solving` | 独立解题能力 | 闭卷、无 AI 环境中完成新题 |
|
|
25
|
+
|
|
26
|
+
> 这是 primary outcome 的首选来源。判断"是否允许用 AI"主要看 C1。
|
|
27
|
+
|
|
28
|
+
### C2 任务表现(Task Performance)——回答"任务做得怎样"
|
|
29
|
+
|
|
30
|
+
| 指标 | 说明 | 测量示例 |
|
|
31
|
+
| --- | --- | --- |
|
|
32
|
+
| `completion_time` | 完成时间 | 从开始到通过所有测试的时间 |
|
|
33
|
+
| `accuracy` | 正确率 | 测试通过率、题面覆盖率 |
|
|
34
|
+
| `code_quality` | 代码质量 | 可读性、结构、命名、冗余度 |
|
|
35
|
+
| `assignment_score` | 作业/考试成绩 | 常规评分(含 AI 参与环境) |
|
|
36
|
+
|
|
37
|
+
> C2 最容易测量,也最容易误导:用 AI 当然快而准。C2 改善**不能**推断出 C1 改善。
|
|
38
|
+
|
|
39
|
+
### C3 学习过程(Learning Process)——回答"学习如何发生"
|
|
40
|
+
|
|
41
|
+
| 指标 | 说明 | 测量示例 |
|
|
42
|
+
| --- | --- | --- |
|
|
43
|
+
| `engagement` | 投入度 | 上机活跃时长、任务尝试次数 |
|
|
44
|
+
| `motivation` | 动机 | 自评量表、课程留存率 |
|
|
45
|
+
| `cognitive_load` | 认知负荷 | NASA-TLX、学后疲劳自评 |
|
|
46
|
+
| `help_seeking` | 求助行为 | 求助 AI vs 求助教师/同伴的比例 |
|
|
47
|
+
| `metacognition` | 元认知 | 能否解释自己的思路、是否先规划再动手 |
|
|
48
|
+
|
|
49
|
+
### C4 风险指标(Risk Indicators)——回答"代价是什么"
|
|
50
|
+
|
|
51
|
+
| 指标 | 说明 | 测量示例 |
|
|
52
|
+
| --- | --- | --- |
|
|
53
|
+
| `ai_dependency` | AI 依赖 | 无 AI 条件下完成任务的能力降幅 |
|
|
54
|
+
| `over_reliance` | 过度依赖 | 求助 AI 的频率/比例、跳过思考步骤的行为 |
|
|
55
|
+
| `reduced_effort` | 努力下降 | 思考时间缩短、照抄输出 |
|
|
56
|
+
| `reduced_transfer` | 迁移下降 | 无 AI 新题得分低于对照组 |
|
|
57
|
+
| `academic_integrity_risk` | 学术诚信风险 | 代写检测、抄袭代码比例 |
|
|
58
|
+
| `false_confidence` | 虚假自信 | 高估自身能力而低估对 AI 的依赖 |
|
|
59
|
+
|
|
60
|
+
## 2. 证据绑定规则(每条证据必须绑定一个 Outcome)
|
|
61
|
+
|
|
62
|
+
### 规则(必须遵守)
|
|
63
|
+
|
|
64
|
+
| 编号 | 内容 |
|
|
65
|
+
| --- | --- |
|
|
66
|
+
| OT-01 | 每条证据/数据必须标注其 Outcome 类别与具体指标名,格式:`Outcome = C1/knowledge_gain`。 |
|
|
67
|
+
| OT-02 | 未标注 Outcome 的结论一律视为不可验证,不得进入 Evidence Matrix。 |
|
|
68
|
+
| OT-03 | 一项测量只能属于一类 Outcome;例如"用 AI 时的完成时间"属 C2,不能同时当作 C1。 |
|
|
69
|
+
| OT-04 | 结论中使用哪个指标,就必须声明该指标属于哪类,禁止跨类互换表述。 |
|
|
70
|
+
| OT-05 | 若证据只覆盖 C2,则结论只能谈任务表现,禁止外推为"学习效果得到改善"。 |
|
|
71
|
+
| OT-06 | 涉及 AI 工具的结论,必须同时报告至少一项 C1(学习)与至少一项 C4(风险)。 |
|
|
72
|
+
|
|
73
|
+
### 证据绑定示例
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
证据 1(实验组比对照组提交更快):
|
|
77
|
+
Outcome = C2/completion_time → 结论:AI 提高了任务效率。
|
|
78
|
+
✗ 不得改写为"AI 提高了学习效果"。
|
|
79
|
+
|
|
80
|
+
证据 2(四周后无 AI 独立测试两组无差异):
|
|
81
|
+
Outcome = C1/independent_problem_solving → 结论:短期任务效率优势未转化为独立能力。
|
|
82
|
+
|
|
83
|
+
证据 3(对照前测后均无下降,实验组无 AI 时错误明显增多):
|
|
84
|
+
Outcome = C4/ai_dependency → 结论:存在依赖风险,需限制使用方式。
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## 3. 结果解读矩阵(Outcome 组合怎么解读)
|
|
88
|
+
|
|
89
|
+
| C2(任务表现) | C1(学习/迁移) | C4(风险) | 建议解读 |
|
|
90
|
+
| --- | --- | --- | --- |
|
|
91
|
+
| 上升 | 上升 | 无风险 | 正面证据,可考虑推广 |
|
|
92
|
+
| 上升 | 持平 | 无风险 | 工具可作效率提升,但勿声称"学得更快" |
|
|
93
|
+
| 上升 | 持平 | 有风险 | 效率红利伴随依赖,需设计防依赖策略 |
|
|
94
|
+
| 上升 | 下降 | — | **危险信号**:成绩好但没学会,优先查 C4 |
|
|
95
|
+
| 持平/下降 | 上升 | 无风险 | 慢但扎实,视课程目标取舍 |
|
|
96
|
+
| 上升 | 上升 | 有风险 | 需长期跟踪 retention/transfer 再下结论 |
|
|
97
|
+
|
|
98
|
+
## 4. 最小报告要求
|
|
99
|
+
|
|
100
|
+
任何"AI 使用政策"相关结论至少包含:
|
|
101
|
+
|
|
102
|
+
1. 一个 C1 指标(知识/理解/保持/迁移/独立解题);
|
|
103
|
+
2. 一个 C4 指标(依赖或诚信);
|
|
104
|
+
3. 明确区分"有 AI 时表现"与"无 AI 时表现"。
|
|
105
|
+
|
|
106
|
+
缺少以上任一 → 结论降级为"待验证假设"。
|
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
# Retrieval Protocol(检索协议)
|
|
2
|
+
|
|
3
|
+
## 1. 目的与定位
|
|
4
|
+
|
|
5
|
+
本协议是工作流第 2 步 Retrieve 的执行细则,回答三个问题:
|
|
6
|
+
|
|
7
|
+
1. **查什么**:从 EducationResearchFrame 构造覆盖正面与反方证据的查询词族;
|
|
8
|
+
2. **从哪查**:按来源分级(对齐 `source.schema.json` 的 `authority_level`)决定哪些来源可进入候选;
|
|
9
|
+
3. **查多少**:轮次、数量与饱和规则,避免检索不足或检索冗余。
|
|
10
|
+
|
|
11
|
+
检索的产出是 `sources.jsonl` 中状态为 `DISCOVERED` 的来源条目。**检索结果与摘要只是线索,
|
|
12
|
+
不是证据内容**(RULE 2);只有通过 Fetch(步骤 3)与 Validate(步骤 4,见
|
|
13
|
+
source-validity.md)后才能进入 Evidence Extraction。
|
|
14
|
+
|
|
15
|
+
未经本协议规范检索的结论一律视为"证据链不完整",不得进入 Tribunal。
|
|
16
|
+
|
|
17
|
+
## 2. 查询构造(教育 PICO 式转化)
|
|
18
|
+
|
|
19
|
+
### 2.1 从 Frame 提取三要素
|
|
20
|
+
|
|
21
|
+
每个查询必须覆盖 干预(Intervention)× 学习者(Population)× 结果(Outcome)三要素。
|
|
22
|
+
缺失任一要素的查询视为无效查询,不得执行。
|
|
23
|
+
|
|
24
|
+
| 要素 | 来源字段(education-framing.md) | 示例 |
|
|
25
|
+
| --- | --- | --- |
|
|
26
|
+
| 干预 I | `intervention.ai_tool` + `allowed_usage` | `GitHub Copilot`、`AI code completion`、`"代码补全"` |
|
|
27
|
+
| 学习者 P | `learner.education_level` + `major` + `prior_knowledge` | `undergraduate introductory programming`、`大一 编程入门` |
|
|
28
|
+
| 结果 O | `outcomes.primary` + `risk` | `independent problem solving`、`learning`、`transfer` |
|
|
29
|
+
|
|
30
|
+
### 2.2 查询词族规则
|
|
31
|
+
|
|
32
|
+
- 每个要素生成 2–3 个同义/近义表达(如 `AI assistant` / `AI code completion` / `LLM-based tutor`),
|
|
33
|
+
组合成词族;词族间必须实际执行,禁止只跑一个代表性查询后"脑补"其余结果。
|
|
34
|
+
- AI 工具结论必须限定到**工具类型或版本**:`complete code generation` 与
|
|
35
|
+
`code completion` 与 `explain errors only` 是三个不同的干预(冲突来源分类
|
|
36
|
+
`tool`,见 tribunal-policy.md),不得混用一个查询。
|
|
37
|
+
- 语言覆盖:**中文查询与英文查询必须各至少 1 个**。国内高校场景的证据(如
|
|
38
|
+
中文期刊、国内试点报告)只有中文检索才能发现;applicability-policy.md 要求
|
|
39
|
+
地域标注,检索端必须提供两类地域的候选。
|
|
40
|
+
- 每个查询记录到来源条目的 `discovered_by` / `discovery_provider`,并保存查询串,
|
|
41
|
+
保证可复现。
|
|
42
|
+
|
|
43
|
+
### 2.3 反方证据查询独立构造
|
|
44
|
+
|
|
45
|
+
正面证据与反方证据的查询**必须独立构造、独立执行**(RULE 4)。禁止把正面检索的
|
|
46
|
+
同一批结果"翻面"充当反方证据。反方查询模式见第 5 节。
|
|
47
|
+
|
|
48
|
+
## 3. 来源分级与可接受性
|
|
49
|
+
|
|
50
|
+
来源分级直接使用 `source.schema.json` 的 `authority_level` 枚举(Tier 1–5),
|
|
51
|
+
本协议不另设分级。**fetch provider 不能提升此值**——抓取渠道只改变读取路径,
|
|
52
|
+
不改变来源权威级别。
|
|
53
|
+
|
|
54
|
+
| 级别 | 含义 | 示例 | 可作 SUPPORTED 证据? |
|
|
55
|
+
| --- | --- | --- | --- |
|
|
56
|
+
| `tier1_paper_doi` | 有 DOI 的同行评审论文(期刊/会议) | ACM/IEEE/Elsevier 论文 | ✅ 是 |
|
|
57
|
+
| `tier2_academic_database` | 学术数据库/预印本/机构知识库条目(无 DOI 但有稳定标识) | arXiv、SSRN、大学学位论文库 | ✅ 是(按 source-validity.md 预印本规则) |
|
|
58
|
+
| `tier3_professional_institution` | 专业机构报告与页面 | 大学教学中心、教育部、ACM/IEEE 官方综述 | ⚠️ 可作为背景与上下文,不作独立因果证据 |
|
|
59
|
+
| `tier4_news_secondary` | 新闻与二手来源 | 媒体报道、他人转述的研究解读 | ❌ 仅线索 |
|
|
60
|
+
| `tier5_general_web` | 一般网页 | 博客、无署名页面、厂商页面 | ❌ 仅线索;厂商声明禁止作证据 |
|
|
61
|
+
|
|
62
|
+
### 分级判定规则
|
|
63
|
+
|
|
64
|
+
| 编号 | 内容 |
|
|
65
|
+
| --- | --- |
|
|
66
|
+
| RP-01 | 每条候选来源必须标注 `authority_level`,禁止省略;无法判定时标 `tier5_general_web` 并说明。 |
|
|
67
|
+
| RP-02 | `tier4` / `tier5` 来源只能进入"线索池",不得直接成为 Evidence 的绑定来源;如其中确有可核实的研究发现,必须回溯到其引用的原始 `tier1`/`tier2` 来源。 |
|
|
68
|
+
| RP-03 | 厂商/行业声明(如 Copilot 官方博客、AI 产品宣传页)**一律不得**作为独立证据,即使域名是 `.edu` / `.gov`(需核查内容是否厂商资助)。 |
|
|
69
|
+
| RP-04 | 无法回溯到原始来源的二手转述,不得进入 Evidence Matrix。 |
|
|
70
|
+
|
|
71
|
+
## 4. 检索轮次与饱和规则
|
|
72
|
+
|
|
73
|
+
### 4.1 最小轮次
|
|
74
|
+
|
|
75
|
+
| 轮次 | 内容 | 最小查询数 |
|
|
76
|
+
| --- | --- | --- |
|
|
77
|
+
| 第 1 轮 | 主体检索:按 2.2 词族执行正面证据检索 | 3–5 个查询 |
|
|
78
|
+
| 第 2 轮 | 反方检索:按第 5 节执行独立反方证据检索 | 3–5 个查询 |
|
|
79
|
+
| 第 3 轮(如需要) | 补漏检索:针对证据冲突、样本不匹配或 Frame 的 inclusion_criteria 定向补充 | 按缺口定 |
|
|
80
|
+
|
|
81
|
+
### 4.2 饱和与停止
|
|
82
|
+
|
|
83
|
+
- **饱和规则**:连续 3 个不同查询未发现任何新来源(去重后),视为达到检索饱和,可停止。
|
|
84
|
+
- **数量下限**:进入 Tribunal 前,独立研究/独立样本数量必须满足
|
|
85
|
+
Pre-Verdict Gate 的 `Independent study/sample count checked` 项;不足时
|
|
86
|
+
标注 `INSUFFICIENT_SOURCES`(见 SKILL.md 第 15 章),禁止强行裁决。
|
|
87
|
+
- **正反比例**:反方证据检索结果即使为零,也必须如实输出
|
|
88
|
+
`NO CONTRADICTORY EVIDENCE FOUND`(skeptic-protocol.md SK-06),不得以"查过了没有"的模糊表述代替。
|
|
89
|
+
|
|
90
|
+
### 4.3 范围控制
|
|
91
|
+
|
|
92
|
+
- 检索范围由 Frame 的 `scope`(time_range / geography / study_types)与
|
|
93
|
+
`inclusion_criteria` / `exclusion_criteria` 控制(education-framing.md)。
|
|
94
|
+
- 超出 `scope.time_range` 的经典研究可以保留作背景,但结论时效性按
|
|
95
|
+
evidence-quality.md 的 D4/D5 评分,不得因"经典"而豁免。
|
|
96
|
+
- 检索结果命中排除标准(如样本为研究生、干预不足 2 周)的,直接排除并记录排除原因,
|
|
97
|
+
不进入 Fetch。
|
|
98
|
+
|
|
99
|
+
## 5. 反方证据检索战术(RULE 4 执行细则)
|
|
100
|
+
|
|
101
|
+
Skeptic 的 9 项固定任务(skeptic-protocol.md)需要对应的独立查询模式。每项任务
|
|
102
|
+
至少执行一个针对性查询,查询方向与正面检索相反或正交:
|
|
103
|
+
|
|
104
|
+
| 任务 | 检索目标 | 查询模式示例 |
|
|
105
|
+
| --- | --- | --- |
|
|
106
|
+
| S-01 null result | 无显著差异研究 | `"no significant difference" AI coding assistant`;`copilot exam performance equivalent` |
|
|
107
|
+
| S-02 negative result | 效果更差的研究 | `AI code generation harms learning`;`ChatGPT 编程 能力下降` |
|
|
108
|
+
| S-03 相反方向 | 结论相反的研究 | `banning AI improves outcomes`;`限制 AI 使用 更好` |
|
|
109
|
+
| S-04 alternative explanation | 替代解释证据 | `practice time confound AI study`;`novelty effect AI education` |
|
|
110
|
+
| S-05 measurement mismatch | 测量错配案例 | `satisfaction measured as learning AI`(用于识别把 C2/C3 当 C1 的常见做法) |
|
|
111
|
+
| S-06 sampling bias | 自选/高动机样本 | `self-selection AI course study`;`volunteer bias` |
|
|
112
|
+
| S-07 novelty effect | 新奇效应衰减 | `novelty effect longitudinal AI tutor`;`long-term AI education study` |
|
|
113
|
+
| S-08 AI dependency | 撤除 AI 后表现 | `performance drop without AI`;`AI dependency students`;`withdrawal AI assessment` |
|
|
114
|
+
| S-09 超出范围 | 边界外推证据 | 针对目标 Frame 的特定维度组合检索(如 MOOC 场景、无教师支持场景) |
|
|
115
|
+
|
|
116
|
+
反方证据同样必须通过 Fetch + Validate + 方法学审查(methodology-audit.md)
|
|
117
|
+
后才能纳入裁判(skeptic-protocol.md SK-05)。检索到的反方候选若质量低于
|
|
118
|
+
`weak`(evidence-quality.md),按 SK-04 只保留最高质量 2 条。
|
|
119
|
+
|
|
120
|
+
## 6. 检索记录要求
|
|
121
|
+
|
|
122
|
+
| 字段(source.schema.json) | 要求 |
|
|
123
|
+
| --- | --- |
|
|
124
|
+
| `source_id` | 稳定命名,如 `S-2025-<firstauthor>` |
|
|
125
|
+
| `title` / `authors` / `year` / `doi` | 完整填写;DOI 缺失时标注"未找到 DOI" |
|
|
126
|
+
| `canonical_url` | 原始来源的规范 URL;抓取渠道 URL 不得冒充 |
|
|
127
|
+
| `authority_level` | 按第 3 节判定 |
|
|
128
|
+
| `discovered_by` / `discovery_provider` | 检索渠道与提供方 |
|
|
129
|
+
| `query`(扩展字段) | 命中该来源的查询串,保证可复现 |
|
|
130
|
+
| `status` | 初始为 `DISCOVERED`,后续按 source-validity.md 流转 |
|
|
131
|
+
|
|
132
|
+
## 7. 执行规则(必须遵守)
|
|
133
|
+
|
|
134
|
+
| 编号 | 内容 |
|
|
135
|
+
| --- | --- |
|
|
136
|
+
| RP-05 | 检索未执行(跳过 Retrieve 直接回答)即违反 RULE 1,流程无效。 |
|
|
137
|
+
| RP-06 | 三要素(I × P × O)任一缺失的查询不得执行。 |
|
|
138
|
+
| RP-07 | 中英双语各至少 1 个查询;仅单语检索视为检索不完整。 |
|
|
139
|
+
| RP-08 | 反方证据必须独立查询,禁止复用正面结果充当反方。 |
|
|
140
|
+
| RP-09 | snippet 与摘要不得直接作为证据内容(RULE 2);检索阶段产物只能是线索。 |
|
|
141
|
+
| RP-10 | 厂商声明与二手转述不得作为独立证据(RP-03 / RP-04)。 |
|
|
142
|
+
| RP-11 | 达到饱和规则或数量下限后仍不足的,如实输出 `INSUFFICIENT_SOURCES`,禁止降低纳入标准凑数。 |
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Skeptic Protocol(怀疑者协议)
|
|
2
|
+
|
|
3
|
+
## 1. 目的
|
|
4
|
+
|
|
5
|
+
Skeptic 的职责是**找反方**:在正面证据之外主动检索可能推翻或限制结论的证据。
|
|
6
|
+
未经 Skeptic 检查的结论一律视为"未定论"。
|
|
7
|
+
|
|
8
|
+
## 2. 固定 9 项任务(每次分析必须全部执行)
|
|
9
|
+
|
|
10
|
+
Skeptic 对候选结论逐项执行以下 9 个任务,每项输出 `找到 / 未找到` + 证据。
|
|
11
|
+
|
|
12
|
+
| 编号 | 任务 | 具体内容 |
|
|
13
|
+
| --- | --- | --- |
|
|
14
|
+
| S-01 | 找 null result(无效结果) | 检索同类型干预报告"无显著差异"的研究,如"AI 辅助 vs 无 AI 下成绩无差异"。 |
|
|
15
|
+
| S-02 | 找 negative result(负面结果) | 检索"效果更差"的研究,如"使用 AI 后独立解题能力反而下降"。 |
|
|
16
|
+
| S-03 | 找相反方向研究 | 检索结论方向与当前结论相反的研究,如主流结论是"允许",找"禁止/受限更好"的证据。 |
|
|
17
|
+
| S-04 | Alternative explanation(替代解释) | 对正面结果提出至少 2 种替代解释(练习量、教师投入、新鲜感、额外辅导),并检查原研究是否排除。 |
|
|
18
|
+
| S-05 | Measurement mismatch(测量错配) | 检查正面结论的测量是否与宣称 Outcome 错配(把 C2 当 C1、满意度当学习效果)。 |
|
|
19
|
+
| S-06 | Sampling bias(抽样偏差) | 检查样本是否偏向高动机/高先验人群,是否存在自选实验组。 |
|
|
20
|
+
| S-07 | Novelty effect(新奇效应) | 检查正面效果是否可能来自工具新鲜感,而非持续学习收益。 |
|
|
21
|
+
| S-08 | AI dependency(AI 依赖) | 检索工具撤除后表现下降的证据,检查正面研究是否报告了无 AI 条件的迁移测试。 |
|
|
22
|
+
| S-09 | 结论超出研究范围 | 检查原结论是否超出了证据的 学习者×课程×工具×时长×情境 范围。 |
|
|
23
|
+
|
|
24
|
+
## 3. 反方证据来源要求
|
|
25
|
+
|
|
26
|
+
- **反方证据必须来自真实文献、真实研究报告或真实数据。**
|
|
27
|
+
- 禁止虚构、编造、推测性的"可能存在的反方证据"。
|
|
28
|
+
- 若依据的是个人经验或推理,必须标注 `来源=推理`,并单独列为"假设性反驳",
|
|
29
|
+
不得与文献证据混同。
|
|
30
|
+
|
|
31
|
+
## 4. 无反方证据时的输出
|
|
32
|
+
|
|
33
|
+
当全部 9 项任务均未找到有效反方证据时,**必须逐字输出**:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
NO CONTRADICTORY EVIDENCE FOUND
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
该语句是唯一合法的"无反驳"输出;禁止用"似乎没问题""应该没反例"等模糊表述代替。
|
|
40
|
+
|
|
41
|
+
## 5. 输出格式
|
|
42
|
+
|
|
43
|
+
### 每个任务的输出
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
S-01 null result : 找到 / 未找到
|
|
47
|
+
证据: <文献标题/作者/年份 + 关键结果一句话>
|
|
48
|
+
影响: <对候选结论的具体限制,或"无影响">
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### 汇总输出
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
候选结论: <一句话>
|
|
55
|
+
Skeptic 汇总:
|
|
56
|
+
- 找到反方证据的任务: S-xx, S-yy
|
|
57
|
+
- 反方证据强度: 强(可推翻)/ 中(需限范围)/ 弱(提示作用)
|
|
58
|
+
- 未找到的任务: S-zz, ...
|
|
59
|
+
- 结论: NO CONTRADICTORY EVIDENCE FOUND
|
|
60
|
+
| 存在反方证据,候选结论需修改或限定为: ...
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## 6. 执行规则(必须遵守)
|
|
64
|
+
|
|
65
|
+
| 编号 | 内容 |
|
|
66
|
+
| --- | --- |
|
|
67
|
+
| SK-01 | 9 项任务缺一不可;缺项则 Skeptic 报告不完整,裁判庭不得使用。 |
|
|
68
|
+
| SK-02 | 虚构反方证据属于严重违规;发现即废弃该 Skeptic 报告并重新执行。 |
|
|
69
|
+
| SK-03 | 假设性反驳(推理来源)必须单独列出,不得算作"找到反方证据"。 |
|
|
70
|
+
| SK-04 | 某任务找到多条证据时按证据质量(evidence-quality.md)排序,只取最高质量的 2 条。 |
|
|
71
|
+
| SK-05 | 反方证据即使与候选结论方向相反,也需完成方法学审查(methodology-audit.md)后才能纳入裁判。 |
|
|
72
|
+
| SK-06 | 无证据输出必须使用固定字符串 `NO CONTRADICTORY EVIDENCE FOUND`。 |
|
|
73
|
+
|
|
74
|
+
## 7. 示例
|
|
75
|
+
|
|
76
|
+
> 候选结论:允许 Copilot 辅助的大一 C 语言班作业正确率更高,说明 AI 提升学习效果。
|
|
77
|
+
|
|
78
|
+
```
|
|
79
|
+
S-01 null result : 找到。Lee et al. (2024) 报告代码补全组与对照组
|
|
80
|
+
在期末闭卷成绩上无显著差异(p=0.42)。
|
|
81
|
+
S-02 negative result: 未找到。
|
|
82
|
+
S-03 相反方向 : 找到。Yuan (2025) 报告无 AI 提示组在迁移测试显著优于全自动补全组。
|
|
83
|
+
S-04 替代解释 : 找到。原研究未排除"实验班课后多花 2 小时/周"的练习量差异。
|
|
84
|
+
S-05 measurement : 找到。原研究用"作业正确率(C2)"得出"学习效果(C1)"结论,错配。
|
|
85
|
+
S-06 sampling bias : 未找到。
|
|
86
|
+
S-07 novelty : 找到。干预仅 4 周,存在新鲜感解释空间。
|
|
87
|
+
S-08 AI dependency : 找到。原研究未提供撤除 AI 后的独立测试数据。
|
|
88
|
+
S-09 超出范围 : 找到。原研究样本为研究生,被用来支撑本科生结论。
|
|
89
|
+
|
|
90
|
+
Skeptic 汇总: 存在中强反方证据。候选结论需改写为:
|
|
91
|
+
"AI 辅助可提高 AI 环境下的作业完成表现,但无证据表明提升学习效果;
|
|
92
|
+
存在迁移下降与依赖风险,且原证据有测量错配与样本外推问题。"
|
|
93
|
+
```
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Social Science & Educational Research Pitfalls Checklist
|
|
2
|
+
|
|
3
|
+
> **Methodology Tribunal Core Audit Rule**
|
|
4
|
+
> When evaluating empirical claims in education, learning sciences, and educational technology, every study must be audited against these four foundational pitfalls. Failure to isolate these confounds results in immediate methodological confidence downgrading.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## 1. Task Performance != Genuine Learning (任务表现 != 概念习得)
|
|
9
|
+
|
|
10
|
+
- **The Threat**: An intervention (e.g., AI coding assistant, interactive hint system, intelligent tutor) allows students to complete learning tasks faster or with fewer syntax errors, but fails to build enduring conceptual mental models or deep problem-solving schema.
|
|
11
|
+
- **Audit Criteria**:
|
|
12
|
+
- Does the study evaluate conceptual understanding beyond automated task completion?
|
|
13
|
+
- Are assessment tasks distinct from the training tasks (avoiding near-transfer memorization)?
|
|
14
|
+
- Is there evidence of cognitive offloading where the tool performed the task instead of the learner?
|
|
15
|
+
- **Decision Penalty**: If an outcome only measures in-task completion speed/accuracy without conceptual probing, classify outcome as (never ).
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## 2. Short-Term Score != Long-Term Retention (短期得分 != 长期保持)
|
|
20
|
+
|
|
21
|
+
- **The Threat**: Immediate post-tests administered right after tool use capture transient cognitive scaffolding and working memory activation. As scaffolding fades, learned gains frequently decay rapidly.
|
|
22
|
+
- **Audit Criteria**:
|
|
23
|
+
- Was a delayed post-test administered at least 2 to 4+ weeks post-intervention?
|
|
24
|
+
- Does the effect size persist or decay over the retention window?
|
|
25
|
+
- Is there longitudinal follow-up across academic grading cycles?
|
|
26
|
+
- **Decision Penalty**: If only immediate post-test data is available without delayed retention, downgrade Evidence Tier and flag .
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## 3. AI-Assisted Performance != Independent Transfer (辅助表现 != 独立迁移)
|
|
31
|
+
|
|
32
|
+
- **The Threat**: Measuring student proficiency while the AI/intervention tool is actively present conflates human capability with human-AI joint capability. True educational value requires independent transfer to solo, unassisted contexts.
|
|
33
|
+
- **Audit Criteria**:
|
|
34
|
+
- Were post-tests and transfer tests conducted in an isolated, tool-free environment?
|
|
35
|
+
- Can learners solve novel, isomorphic, and far-transfer problems independently?
|
|
36
|
+
- Did the intervention foster tool dependency or autonomous mastery?
|
|
37
|
+
- **Decision Penalty**: Claims asserting student skill improvement based on tool-in-the-loop assessments must be rejected or marked as .
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## 4. Correlation != Causal Pedagogical Efficacy (相关关系 != 因果实证)
|
|
42
|
+
|
|
43
|
+
- **The Threat**: High-performing, intrinsically motivated students voluntarily adopt AI tools or new learning methods at higher rates (Self-Selection Bias). Observing higher test scores among tool users reflects baseline ability, not treatment effect.
|
|
44
|
+
- **Audit Criteria**:
|
|
45
|
+
- Is there true random assignment (RCT) or rigorous quasi-experimental control (DID, Propensity Score Matching, Regression Discontinuity)?
|
|
46
|
+
- Was baseline equivalence established on pre-tests and demographic confounders?
|
|
47
|
+
- Was overall and differential attrition tracked and within WWC standard thresholds?
|
|
48
|
+
- **Decision Penalty**: Observational correlational studies without causal identification strategies cannot receive WWC Tier 1/2 or support definitive recommendations.
|