paperlint 13.0.3 → 13.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +9 -1
- package/dist/adapters/git/index.d.ts.map +1 -1
- package/dist/adapters/git/index.js +13 -0
- package/dist/adapters/git/index.js.map +1 -1
- package/dist/adapters/study-registry/index.d.ts +142 -1
- package/dist/adapters/study-registry/index.d.ts.map +1 -1
- package/dist/adapters/study-registry/index.js +160 -1
- package/dist/adapters/study-registry/index.js.map +1 -1
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +3 -0
- package/dist/cli.js.map +1 -1
- package/dist/domain/decide-first.d.ts +43 -0
- package/dist/domain/decide-first.d.ts.map +1 -1
- package/dist/domain/decide-first.js +45 -2
- package/dist/domain/decide-first.js.map +1 -1
- package/dist/domain/study-gates.d.ts +96 -0
- package/dist/domain/study-gates.d.ts.map +1 -0
- package/dist/domain/study-gates.js +56 -0
- package/dist/domain/study-gates.js.map +1 -0
- package/dist/domain/study-mode.d.ts +6 -0
- package/dist/domain/study-mode.d.ts.map +1 -1
- package/dist/domain/study-mode.js.map +1 -1
- package/dist/domain/study-registry.d.ts +24 -4
- package/dist/domain/study-registry.d.ts.map +1 -1
- package/dist/domain/study-registry.js +39 -2
- package/dist/domain/study-registry.js.map +1 -1
- package/dist/domain/study-result.d.ts +151 -0
- package/dist/domain/study-result.d.ts.map +1 -0
- package/dist/domain/study-result.js +175 -0
- package/dist/domain/study-result.js.map +1 -0
- package/dist/outputs-before-decision-rule.d.ts.map +1 -1
- package/dist/outputs-before-decision-rule.js +37 -3
- package/dist/outputs-before-decision-rule.js.map +1 -1
- package/dist/ports/git-history.d.ts +12 -0
- package/dist/ports/git-history.d.ts.map +1 -1
- package/dist/ports/study-registry-codec.d.ts +3 -0
- package/dist/ports/study-registry-codec.d.ts.map +1 -1
- package/dist/skills/build-label-page/scripts/label-page-cli.d.ts +6 -0
- package/dist/skills/build-label-page/scripts/label-page-commands.d.ts +14 -0
- package/dist/skills/build-label-page/scripts/label-page-core.d.ts +149 -0
- package/dist/skills/build-label-page/scripts/label-page-html.d.ts +5 -0
- package/dist/skills/build-label-page/scripts/label-page.d.ts +2 -0
- package/dist/study-modes.d.ts +3 -0
- package/dist/study-modes.d.ts.map +1 -1
- package/dist/study-modes.js +8 -1
- package/dist/study-modes.js.map +1 -1
- package/dist/study-registry-rule.d.ts.map +1 -1
- package/dist/study-registry-rule.js +1 -0
- package/dist/study-registry-rule.js.map +1 -1
- package/dist/study-result-rule.d.ts +12 -0
- package/dist/study-result-rule.d.ts.map +1 -0
- package/dist/study-result-rule.js +87 -0
- package/dist/study-result-rule.js.map +1 -0
- package/docs/design/study-gates.md +208 -0
- package/docs/rules/repro/outputs-before-decision.md +8 -0
- package/docs/rules/repro/study-registered.md +38 -15
- package/docs/rules/repro/study-result.md +100 -0
- package/docs/rules.md +1 -0
- package/eslint.config.mjs +1 -1
- package/hooks/hooks.harness.ts +51 -13
- package/hooks/paper-skills-nudge.hook.d.mts +22 -0
- package/hooks/paper-skills-nudge.hook.mjs +85 -23
- package/hooks/paper-skills-nudge.test.ts +82 -4
- package/package.json +2 -1
- package/plugin/hooks/hooks.json +7 -2
- package/schemas/plan.v2.schema.json +341 -0
- package/skills/argument-arc/SKILL.md.spec.ts +1 -1
- package/skills/build-label-page/SKILL.md +85 -54
- package/skills/build-label-page/SKILL.md.spec.ts +85 -53
- package/skills/build-label-page/build-label-page.test.ts +10 -941
- package/skills/build-label-page/references/checklist.md +45 -3
- package/skills/build-label-page/references/closed-choice-generator.md +88 -0
- package/skills/build-label-page/scripts/label-page-cli.js +72 -0
- package/skills/build-label-page/scripts/label-page-cli.ts +85 -0
- package/skills/build-label-page/scripts/label-page-commands.js +14 -0
- package/skills/build-label-page/scripts/label-page-commands.ts +14 -0
- package/skills/build-label-page/scripts/label-page-core.js +850 -0
- package/skills/build-label-page/scripts/label-page-core.test.ts +1164 -0
- package/skills/build-label-page/scripts/label-page-core.ts +1210 -0
- package/skills/build-label-page/scripts/label-page-html.js +33 -0
- package/skills/build-label-page/scripts/label-page-html.test.ts +413 -0
- package/skills/build-label-page/scripts/label-page-html.ts +43 -0
- package/skills/build-label-page/scripts/label-page.js +70 -0
- package/skills/build-label-page/scripts/label-page.test.ts +71 -0
- package/skills/build-label-page/scripts/label-page.ts +71 -0
- package/skills/build-label-page/scripts/page-template.html +513 -0
- package/skills/find-venue/SKILL.md.spec.ts +1 -1
- package/skills/paper-status/SKILL.md.spec.ts +1 -1
- package/skills/render-paper/SKILL.md.spec.ts +1 -1
- package/src/adapters/git/index.test.ts +33 -0
- package/src/adapters/git/index.ts +19 -0
- package/src/adapters/study-registry/gates.test.ts +181 -0
- package/src/adapters/study-registry/index.ts +219 -1
- package/src/adapters/study-registry/mode.test.ts +6 -0
- package/src/cli.ts +3 -0
- package/src/domain/decide-first.test.ts +131 -0
- package/src/domain/decide-first.ts +92 -2
- package/src/domain/study-gates.ts +125 -0
- package/src/domain/study-measurements.test.ts +71 -0
- package/src/domain/study-mode.ts +6 -0
- package/src/domain/study-registry.ts +64 -2
- package/src/domain/study-result.test.ts +300 -0
- package/src/domain/study-result.ts +358 -0
- package/src/outputs-before-decision-rule.test.ts +115 -2
- package/src/outputs-before-decision-rule.ts +50 -2
- package/src/ports/git-history.ts +12 -0
- package/src/ports/study-registry-codec.ts +3 -0
- package/src/rule-cases.test.ts +74 -0
- package/src/studies.test.ts +3 -0
- package/src/study-modes.ts +10 -1
- package/src/study-registry-rule.ts +1 -0
- package/src/study-result-rule.test.ts +281 -0
- package/src/study-result-rule.ts +123 -0
- package/skills/build-label-page/references/example-generator.py +0 -803
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"study-registry-rule.js","sourceRoot":"","sources":["../src/study-registry-rule.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AACH,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,MAAM,WAAW,CAAC;AAC3D,OAAO,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,MAAM,QAAQ,CAAC;AAC3C,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAC9C,OAAO,EACL,MAAM,EACN,gBAAgB,EAChB,SAAS,EACT,UAAU,GACX,MAAM,0BAA0B,CAAC;AAClC,OAAO,EACL,gBAAgB,EAChB,kBAAkB,EAClB,YAAY,EACZ,YAAY,EACZ,SAAS,EACT,QAAQ,EACR,SAAS,EACT,YAAY,GAOb,MAAM,4BAA4B,CAAC;AASpC,OAAO,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAExD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAChD,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAE/C,MAAM,WAAW,GAAG,oBAAoB,CAAC;AASzC,uFAAuF;AACvF,MAAM,CAAC,MAAM,MAAM,GAAG,CAAC,KAAY,EAAE,CAAS,EAAiB,EAAE;IAC/D,MAAM,CAAC,GAAG,KAAK,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,SAAS,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IAC9E,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,WAAW,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;AACzD,CAAC,CAAC;AAEF,oFAAoF;AACpF,MAAM,UAAU,UAAU,CACxB,GAAW,EACX,IAAgD;IAEhD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,YAAY,CAAC,CAAC,CAAC;IACpE,IAAI,IAAI,KAAK,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC;IAC7C,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IACtC,OAAO,MAAM,CAAC,EAAE;QACd,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE;QAC7C,CAAC,CAAC,EAAE,IAAI,EAAE,YAAY,EAAE,GAAG,EAAE,MAAM,CAAC,GAAG,EAAE,CAAC;AAC9C,CAAC;AAED,2GAA2G;AAC3G,MAAM,YAAY,GAAG,CAAC,GAAW,EAAE,IAAsB,EAAqB,EAAE,CAC9E,cAAc,CAAC,GAAG,EAAE,IAAI,CAAC;KACtB,MAAM,CACL,CAAC,CAAC,EAAE,EAAE,CACJ,CAAC,CAAC,IAAI,KAAK,KAAK;IAChB,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC;IAC5B,CAAC,gBAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CACrC;KACA,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;AAExB,kGAAkG;AAClG,MAAM,SAAS,GAAG,CAChB,CAAgB,EAChB,MAAmB,EACW,EAAE,CAChC,IAAI,CACF,SAAS,CAAC,CAAC,CAAC,EACZ,MAAM,CAAC,aAAa,CAAC,EACrB,GAAG,CACD,CAAC,CAAC,EAAE,EAAE,CACJ;IACE,SAAS,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,WAAW,CAAC;IAClC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,WAAW,CAAC;CAC9B,CACb,EACD,CAAC,OAAO,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC,OAAO,CAAC,CAC9B,CAAC;AAEJ,sCAAsC;AACtC,MAAM,aAAa,GAAG,CACpB,CAAQ,EACuC,EAAE,CAAC,CAAC,CAAC,WAAW,KAAK,SAAS,CAAC;AAEhF;;;GAGG;AACH,MAAM,CAAC,MAAM,OAAO,GAAG,CACrB,CAAgB,EAChB,MAAmB,EACnB,KAAyB,EACO,EAAE,CAClC,IAAI,GAAG,CACL,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,OAAO,CAAC,CAAC,GAAG,EAAE,EAAE;IAC9D,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;IAC9C,OAAO,KAAK,KAAK,IAAI;QACnB,CAAC,CAAC,EAAE;QACJ,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,IAAI,CAAC,EAAE,KAAK,CAAU,CAAC,CAAC;AAC1D,CAAC,CAAC,CACH,CAAC;AAEJ,2FAA2F;AAC3F,MAAM,WAAW,GAAG,CAClB,EAAE,MAAM,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAW,EACnC,MAAmB,EACnB,KAAyB,EACP,EAAE;IACpB,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;IACnC,QAAQ,IAAI,CAAC,IAAI,EAAE,CAAC;QAClB,KAAK,QAAQ;YACX,OAAO,IAAI,CAAC;QACd,KAAK,QAAQ;YACX,OAAO,EAAE,IAAI,EAAE,QAAQ,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC;QAC7C,KAAK,MAAM;YACT,OAAO,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC,KAAK,gBAAgB;gBAC9C,CAAC,CAAC,KAAK,CAAC,eAAe,CAAC,IAAI,CAAC,IAAI,CAAC;gBAClC,CAAC,CAAC,KAAK,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IACzC,CAAC;AACH,CAAC,CAAC;AAEF,iGAAiG;AACjG,MAAM,UAAU,aAAa,CAC3B,GAAW,EACX,IAAuB;IAEvB,MAAM,QAAQ,GAAG,UAAU,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IACvC,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IACtC,OAAO,YAAY,CAAC;QAClB,OAAO,EAAE,YAAY,CAAC,GAAG,EAAE,IAAI,CAAC;QAChC,QAAQ;QACR,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,MAAM,CAAC,CAAC;QACxD,OAAO,EAAE,SAAS,CAAC,QAAQ,EAAE,MAAM,CAAC;QACpC,KAAK,EAAE,OAAO,CAAC,QAAQ,EAAE,MAAM,EAAE,IAAI,CAAC,KAAK,CAAC;KAC7C,CAAC,CAAC;AACL,CAAC;AAED,0GAA0G;AAC1G,MAAM,QAAQ,GAAuC;IACnD,aAAa;IACb,YAAY;IACZ,cAAc;IACd,WAAW;IACX,UAAU;IACV,QAAQ;IACR,cAAc;IACd,aAAa;IACb,YAAY;IACZ,WAAW;IACX,aAAa;IACb,cAAc;IACd,aAAa;IACb,qBAAqB;IACrB,mBAAmB;IACnB,qBAAqB;IACrB,mBAAmB;IACnB,UAAU;IACV,cAAc;CACf,CAAC;AAEF,kGAAkG;AAClG,MAAM,MAAM,GAAG,CACb,IAAY,EACZ,WAAmB,EACnB,QAA4C,EAClB,EAAE,CAAC,CAAC;IAC9B,IAAI,EAAE,YAAY;IAClB,IAAI,EAAE,EAAE,WAAW,EAAE,GAAG,EAAE,WAAW,CAAC,IAAI,CAAC,EAAE;IAC7C,MAAM,EAAE,EAAE;IACV,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC,CAAC;CACnE,CAAC,CAAC;AAEH,yFAAyF;AACzF,MAAM,MAAM,GAAG,CACb,IAA8B,EAC9B,QAA4C,EAC5C,IAAuB,EACL,EAAE,CAAC,CAAC;IACtB,IAAI;IACJ,MAAM,CAAC,OAA0B;QAC/B,IAAI,QAAQ,CAAC,OAAO,CAAC,QAAQ,CAAC,KAAK,WAAW;YAAE,OAAO,EAAE,CAAC;QAC1D,OAAO;YACL,IAAI;gBACF,MAAM,EAAE,GAAG,OAAO,CAAC,UAAU,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;gBACjD,aAAa,CAAC,OAAO,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,IAAI,CAAC;qBAC3C,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;qBAC3C,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE;oBACb,OAAO,CAAC,MAAM,CAAC;wBACb,GAAG,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE;wBAC3B,SAAS,EAAE,CAAC,CAAC,OAAO;wBACpB,IAAI,EAAE,EAAE,IAAI,EAAE,YAAY,CAAC,CAAC,CAAC,EAAE;qBAChC,CAAC,CAAC;gBACL,CAAC,CAAC,CAAC;YACP,CAAC;SACF,CAAC;IACJ,CAAC;CACF,CAAC,CAAC;AAEH;;;;GAIG;AACH,MAAM,UAAU,kBAAkB,CAChC,IAAuB;IAEvB,OAAO;QACL,kBAAkB,EAAE,MAAM,CACxB,MAAM,CACJ,wBAAwB,EACxB,8OAA8O,EAC9O,QAAQ,CACT,EACD,QAAQ,EACR,IAAI,CACL;QACD,iBAAiB,EAAE,MAAM,CACvB,MAAM,CACJ,uBAAuB,EACvB,oDAAoD,EACpD,kBAAkB,CACnB,EACD,kBAAkB,EAClB,IAAI,CACL;KACF,CAAC;AACJ,CAAC;AAED,mGAAmG;AACnG,MAAM,CAAC,MAAM,0BAA0B,GAGlC,EAAE,wBAAwB,EAAE,OAAO,EAAE,uBAAuB,EAAE,MAAM,EAAE,CAAC"}
|
|
1
|
+
{"version":3,"file":"study-registry-rule.js","sourceRoot":"","sources":["../src/study-registry-rule.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AACH,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,MAAM,WAAW,CAAC;AAC3D,OAAO,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,MAAM,QAAQ,CAAC;AAC3C,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAC;AAC9C,OAAO,EACL,MAAM,EACN,gBAAgB,EAChB,SAAS,EACT,UAAU,GACX,MAAM,0BAA0B,CAAC;AAClC,OAAO,EACL,gBAAgB,EAChB,kBAAkB,EAClB,YAAY,EACZ,YAAY,EACZ,SAAS,EACT,QAAQ,EACR,SAAS,EACT,YAAY,GAOb,MAAM,4BAA4B,CAAC;AASpC,OAAO,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AAExD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAChD,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAE/C,MAAM,WAAW,GAAG,oBAAoB,CAAC;AASzC,uFAAuF;AACvF,MAAM,CAAC,MAAM,MAAM,GAAG,CAAC,KAAY,EAAE,CAAS,EAAiB,EAAE;IAC/D,MAAM,CAAC,GAAG,KAAK,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,SAAS,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IAC9E,OAAO,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,WAAW,EAAE,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;AACzD,CAAC,CAAC;AAEF,oFAAoF;AACpF,MAAM,UAAU,UAAU,CACxB,GAAW,EACX,IAAgD;IAEhD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,YAAY,CAAC,CAAC,CAAC;IACpE,IAAI,IAAI,KAAK,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC;IAC7C,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IACtC,OAAO,MAAM,CAAC,EAAE;QACd,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE;QAC7C,CAAC,CAAC,EAAE,IAAI,EAAE,YAAY,EAAE,GAAG,EAAE,MAAM,CAAC,GAAG,EAAE,CAAC;AAC9C,CAAC;AAED,2GAA2G;AAC3G,MAAM,YAAY,GAAG,CAAC,GAAW,EAAE,IAAsB,EAAqB,EAAE,CAC9E,cAAc,CAAC,GAAG,EAAE,IAAI,CAAC;KACtB,MAAM,CACL,CAAC,CAAC,EAAE,EAAE,CACJ,CAAC,CAAC,IAAI,KAAK,KAAK;IAChB,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC;IAC5B,CAAC,gBAAgB,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CACrC;KACA,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;AAExB,kGAAkG;AAClG,MAAM,SAAS,GAAG,CAChB,CAAgB,EAChB,MAAmB,EACW,EAAE,CAChC,IAAI,CACF,SAAS,CAAC,CAAC,CAAC,EACZ,MAAM,CAAC,aAAa,CAAC,EACrB,GAAG,CACD,CAAC,CAAC,EAAE,EAAE,CACJ;IACE,SAAS,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,WAAW,CAAC;IAClC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,WAAW,CAAC;CAC9B,CACb,EACD,CAAC,OAAO,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC,OAAO,CAAC,CAC9B,CAAC;AAEJ,sCAAsC;AACtC,MAAM,aAAa,GAAG,CACpB,CAAQ,EACuC,EAAE,CAAC,CAAC,CAAC,WAAW,KAAK,SAAS,CAAC;AAEhF;;;GAGG;AACH,MAAM,CAAC,MAAM,OAAO,GAAG,CACrB,CAAgB,EAChB,MAAmB,EACnB,KAAyB,EACO,EAAE,CAClC,IAAI,GAAG,CACL,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,OAAO,CAAC,CAAC,GAAG,EAAE,EAAE;IAC9D,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;IAC9C,OAAO,KAAK,KAAK,IAAI;QACnB,CAAC,CAAC,EAAE;QACJ,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,IAAI,CAAC,EAAE,KAAK,CAAU,CAAC,CAAC;AAC1D,CAAC,CAAC,CACH,CAAC;AAEJ,2FAA2F;AAC3F,MAAM,WAAW,GAAG,CAClB,EAAE,MAAM,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAW,EACnC,MAAmB,EACnB,KAAyB,EACP,EAAE;IACpB,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;IACnC,QAAQ,IAAI,CAAC,IAAI,EAAE,CAAC;QAClB,KAAK,QAAQ;YACX,OAAO,IAAI,CAAC;QACd,KAAK,QAAQ;YACX,OAAO,EAAE,IAAI,EAAE,QAAQ,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,CAAC;QAC7C,KAAK,MAAM;YACT,OAAO,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC,KAAK,gBAAgB;gBAC9C,CAAC,CAAC,KAAK,CAAC,eAAe,CAAC,IAAI,CAAC,IAAI,CAAC;gBAClC,CAAC,CAAC,KAAK,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IACzC,CAAC;AACH,CAAC,CAAC;AAEF,iGAAiG;AACjG,MAAM,UAAU,aAAa,CAC3B,GAAW,EACX,IAAuB;IAEvB,MAAM,QAAQ,GAAG,UAAU,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IACvC,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;IACtC,OAAO,YAAY,CAAC;QAClB,OAAO,EAAE,YAAY,CAAC,GAAG,EAAE,IAAI,CAAC;QAChC,QAAQ;QACR,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,EAAE,SAAS,EAAE,MAAM,CAAC,CAAC;QACxD,OAAO,EAAE,SAAS,CAAC,QAAQ,EAAE,MAAM,CAAC;QACpC,KAAK,EAAE,OAAO,CAAC,QAAQ,EAAE,MAAM,EAAE,IAAI,CAAC,KAAK,CAAC;KAC7C,CAAC,CAAC;AACL,CAAC;AAED,0GAA0G;AAC1G,MAAM,QAAQ,GAAuC;IACnD,aAAa;IACb,YAAY;IACZ,cAAc;IACd,WAAW;IACX,UAAU;IACV,QAAQ;IACR,cAAc;IACd,aAAa;IACb,YAAY;IACZ,WAAW;IACX,aAAa;IACb,cAAc;IACd,aAAa;IACb,qBAAqB;IACrB,mBAAmB;IACnB,qBAAqB;IACrB,mBAAmB;IACnB,UAAU;IACV,cAAc;IACd,cAAc;CACf,CAAC;AAEF,kGAAkG;AAClG,MAAM,MAAM,GAAG,CACb,IAAY,EACZ,WAAmB,EACnB,QAA4C,EAClB,EAAE,CAAC,CAAC;IAC9B,IAAI,EAAE,YAAY;IAClB,IAAI,EAAE,EAAE,WAAW,EAAE,GAAG,EAAE,WAAW,CAAC,IAAI,CAAC,EAAE;IAC7C,MAAM,EAAE,EAAE;IACV,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC,CAAC;CACnE,CAAC,CAAC;AAEH,yFAAyF;AACzF,MAAM,MAAM,GAAG,CACb,IAA8B,EAC9B,QAA4C,EAC5C,IAAuB,EACL,EAAE,CAAC,CAAC;IACtB,IAAI;IACJ,MAAM,CAAC,OAA0B;QAC/B,IAAI,QAAQ,CAAC,OAAO,CAAC,QAAQ,CAAC,KAAK,WAAW;YAAE,OAAO,EAAE,CAAC;QAC1D,OAAO;YACL,IAAI;gBACF,MAAM,EAAE,GAAG,OAAO,CAAC,UAAU,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;gBACjD,aAAa,CAAC,OAAO,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,IAAI,CAAC;qBAC3C,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;qBAC3C,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE;oBACb,OAAO,CAAC,MAAM,CAAC;wBACb,GAAG,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE;wBAC3B,SAAS,EAAE,CAAC,CAAC,OAAO;wBACpB,IAAI,EAAE,EAAE,IAAI,EAAE,YAAY,CAAC,CAAC,CAAC,EAAE;qBAChC,CAAC,CAAC;gBACL,CAAC,CAAC,CAAC;YACP,CAAC;SACF,CAAC;IACJ,CAAC;CACF,CAAC,CAAC;AAEH;;;;GAIG;AACH,MAAM,UAAU,kBAAkB,CAChC,IAAuB;IAEvB,OAAO;QACL,kBAAkB,EAAE,MAAM,CACxB,MAAM,CACJ,wBAAwB,EACxB,8OAA8O,EAC9O,QAAQ,CACT,EACD,QAAQ,EACR,IAAI,CACL;QACD,iBAAiB,EAAE,MAAM,CACvB,MAAM,CACJ,uBAAuB,EACvB,oDAAoD,EACpD,kBAAkB,CACnB,EACD,kBAAkB,EAClB,IAAI,CACL;KACF,CAAC;AACJ,CAAC;AAED,mGAAmG;AACnG,MAAM,CAAC,MAAM,0BAA0B,GAGlC,EAAE,wBAAwB,EAAE,OAAO,EAAE,uBAAuB,EAAE,MAAM,EAAE,CAAC"}
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import { type ResultStudy } from "./domain/study-result.ts";
|
|
2
|
+
import type { FolderRuleModule } from "./folder-venue-rule.ts";
|
|
3
|
+
import type { StudyDeps } from "./ports/study-deps.ts";
|
|
4
|
+
/** Every done study of a ready plan v2 in the paper folder `paper`. */
|
|
5
|
+
export declare const resultStudies: (paper: string, deps: StudyDeps) => readonly ResultStudy[];
|
|
6
|
+
/** The rule, reading the folder through `deps`. */
|
|
7
|
+
export declare const studyResultRules: (deps: StudyDeps) => Readonly<Record<"study-result", FolderRuleModule>>;
|
|
8
|
+
/** Its level in paperlint's own config: an error. */
|
|
9
|
+
export declare const RESULT_RULE_LEVELS: Readonly<{
|
|
10
|
+
"repro/study-result": "error";
|
|
11
|
+
}>;
|
|
12
|
+
//# sourceMappingURL=study-result-rule.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"study-result-rule.d.ts","sourceRoot":"","sources":["../src/study-result-rule.ts"],"names":[],"mappings":"AAcA,OAAO,EAML,KAAK,WAAW,EACjB,MAAM,0BAA0B,CAAC;AAClC,OAAO,KAAK,EAEV,gBAAgB,EACjB,MAAM,wBAAwB,CAAC;AAChC,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AA0CvD,uEAAuE;AACvE,eAAO,MAAM,aAAa,GACxB,OAAO,MAAM,EACb,MAAM,SAAS,KACd,SAAS,WAAW,EAOtB,CAAC;AAqBF,mDAAmD;AACnD,eAAO,MAAM,gBAAgB,GAC3B,MAAM,SAAS,KACd,QAAQ,CAAC,MAAM,CAAC,cAAc,EAAE,gBAAgB,CAAC,CAclD,CAAC;AAEH,qDAAqD;AACrD,eAAO,MAAM,kBAAkB,EAAE,QAAQ,CAAC;IAAE,oBAAoB,EAAE,OAAO,CAAA;CAAE,CAE1E,CAAC"}
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* `repro/study-result` — a `done` study whose plan is v2 shows, in its `result.json`, what its plan
|
|
3
|
+
* fixed before the run: the number its claim stands or falls by, each labelling step's kappa, its
|
|
4
|
+
* self-test, its funnel and its extractor's recall audit. What is judged, and why, is `src/domain/study-result.ts` and
|
|
5
|
+
* docs/rules/repro/study-result.md; this file finds those studies, reads their result through the
|
|
6
|
+
* study folder's reader (no link followed), and reports.
|
|
7
|
+
*
|
|
8
|
+
* A plan that does not parse, is a draft, or is v1 is `repro/study-registered`'s to report; here it
|
|
9
|
+
* asks nothing. Reported on `PIPELINE-STATUS.md`, first line, like the other `repro/` rules.
|
|
10
|
+
*/
|
|
11
|
+
import { basename, dirname, posix } from "node:path";
|
|
12
|
+
import { filter, map, pipe } from "remeda";
|
|
13
|
+
import { RESULT_FILE } from "./domain/study-gates.js";
|
|
14
|
+
import { studiesOf } from "./domain/study-registry.js";
|
|
15
|
+
import { RESULT_PROBLEMS, gatesOf, judgeResult, resultMessage, } from "./domain/study-result.js";
|
|
16
|
+
import { studyFolder } from "./study-folder.js";
|
|
17
|
+
import { registryOf } from "./study-registry-rule.js";
|
|
18
|
+
import { rulePageUrl } from "./venue-rules.js";
|
|
19
|
+
const STATUS_FILE = "PIPELINE-STATUS.md";
|
|
20
|
+
const NAME = "repro/study-result";
|
|
21
|
+
/** What the folder holds of a study's `result.json`, beside its plan (`plan`, relative to the folder). */
|
|
22
|
+
const resultState = (folder, of, plan, codec) => {
|
|
23
|
+
const read = folder.read(of, posix.join(posix.dirname(plan), RESULT_FILE));
|
|
24
|
+
return read.kind === "text" ? codec.parseResult(read.text) : read;
|
|
25
|
+
};
|
|
26
|
+
/** The row `s` as a study to judge: `done`, its plan ready, every gate decided; else null. */
|
|
27
|
+
const resultStudy = (s, folder, codec) => {
|
|
28
|
+
if (s.status !== "done" || s.plan === undefined)
|
|
29
|
+
return null;
|
|
30
|
+
const read = folder.read(s.folder, s.plan);
|
|
31
|
+
if (read.kind !== "text")
|
|
32
|
+
return null;
|
|
33
|
+
const parsed = codec.parsePlan(read.text, s.kind);
|
|
34
|
+
const gates = parsed.kind === "ready" ? gatesOf(parsed.plan) : null;
|
|
35
|
+
return gates === null
|
|
36
|
+
? null
|
|
37
|
+
: {
|
|
38
|
+
id: s.id,
|
|
39
|
+
folder: s.folder,
|
|
40
|
+
...gates,
|
|
41
|
+
state: resultState(folder, s.folder, s.plan, codec),
|
|
42
|
+
};
|
|
43
|
+
};
|
|
44
|
+
/** Every done study of a ready plan v2 in the paper folder `paper`. */
|
|
45
|
+
export const resultStudies = (paper, deps) => {
|
|
46
|
+
const folder = studyFolder(paper, deps);
|
|
47
|
+
return pipe(studiesOf(registryOf(paper, deps)), map((s) => resultStudy(s, folder, deps.codec)), filter((s) => s !== null));
|
|
48
|
+
};
|
|
49
|
+
/** On `PIPELINE-STATUS.md`, every finding of every such study of its paper, at its first line. */
|
|
50
|
+
const reportOn = (context, deps) => {
|
|
51
|
+
if (basename(context.filename) !== STATUS_FILE)
|
|
52
|
+
return {};
|
|
53
|
+
return {
|
|
54
|
+
root() {
|
|
55
|
+
const at = context.sourceCode.getLocFromIndex(0);
|
|
56
|
+
resultStudies(dirname(context.filename), deps)
|
|
57
|
+
.flatMap(judgeResult)
|
|
58
|
+
.forEach((f) => {
|
|
59
|
+
context.report({
|
|
60
|
+
loc: { start: at, end: at },
|
|
61
|
+
messageId: f.problem,
|
|
62
|
+
data: { text: resultMessage(f) },
|
|
63
|
+
});
|
|
64
|
+
});
|
|
65
|
+
},
|
|
66
|
+
};
|
|
67
|
+
};
|
|
68
|
+
/** The rule, reading the folder through `deps`. */
|
|
69
|
+
export const studyResultRules = (deps) => ({
|
|
70
|
+
"study-result": {
|
|
71
|
+
meta: {
|
|
72
|
+
type: "suggestion",
|
|
73
|
+
docs: {
|
|
74
|
+
description: "a done study whose plan is v2 shows in its result.json what its plan fixed before the run: the number its claim stands or falls by, each labelling step's kappa, its self-test, its funnel and its extractor's recall audit",
|
|
75
|
+
url: rulePageUrl(NAME),
|
|
76
|
+
},
|
|
77
|
+
schema: [],
|
|
78
|
+
messages: Object.fromEntries(RESULT_PROBLEMS.map((p) => [p, "{{text}}"])),
|
|
79
|
+
},
|
|
80
|
+
create: (context) => reportOn(context, deps),
|
|
81
|
+
},
|
|
82
|
+
});
|
|
83
|
+
/** Its level in paperlint's own config: an error. */
|
|
84
|
+
export const RESULT_RULE_LEVELS = {
|
|
85
|
+
"repro/study-result": "error",
|
|
86
|
+
};
|
|
87
|
+
//# sourceMappingURL=study-result-rule.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"study-result-rule.js","sourceRoot":"","sources":["../src/study-result-rule.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AACH,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,WAAW,CAAC;AACrD,OAAO,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,MAAM,QAAQ,CAAC;AAC3C,OAAO,EAAE,WAAW,EAAE,MAAM,yBAAyB,CAAC;AACtD,OAAO,EAAE,SAAS,EAAc,MAAM,4BAA4B,CAAC;AACnE,OAAO,EACL,eAAe,EACf,OAAO,EACP,WAAW,EACX,aAAa,GAGd,MAAM,0BAA0B,CAAC;AAQlC,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAChD,OAAO,EAAE,UAAU,EAAE,MAAM,0BAA0B,CAAC;AACtD,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAE/C,MAAM,WAAW,GAAG,oBAAoB,CAAC;AACzC,MAAM,IAAI,GAAG,oBAAoB,CAAC;AAElC,0GAA0G;AAC1G,MAAM,WAAW,GAAG,CAClB,MAAmB,EACnB,EAAU,EACV,IAAY,EACZ,KAAyB,EACZ,EAAE;IACf,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,EAAE,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,WAAW,CAAC,CAAC,CAAC;IAC3E,OAAO,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;AACpE,CAAC,CAAC;AAEF,8FAA8F;AAC9F,MAAM,WAAW,GAAG,CAClB,CAAQ,EACR,MAAmB,EACnB,KAAyB,EACL,EAAE;IACtB,IAAI,CAAC,CAAC,MAAM,KAAK,MAAM,IAAI,CAAC,CAAC,IAAI,KAAK,SAAS;QAAE,OAAO,IAAI,CAAC;IAC7D,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;IAC3C,IAAI,IAAI,CAAC,IAAI,KAAK,MAAM;QAAE,OAAO,IAAI,CAAC;IACtC,MAAM,MAAM,GAAG,KAAK,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;IAClD,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,KAAK,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IACpE,OAAO,KAAK,KAAK,IAAI;QACnB,CAAC,CAAC,IAAI;QACN,CAAC,CAAC;YACE,EAAE,EAAE,CAAC,CAAC,EAAE;YACR,MAAM,EAAE,CAAC,CAAC,MAAM;YAChB,GAAG,KAAK;YACR,KAAK,EAAE,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,EAAE,KAAK,CAAC;SACpD,CAAC;AACR,CAAC,CAAC;AAEF,uEAAuE;AACvE,MAAM,CAAC,MAAM,aAAa,GAAG,CAC3B,KAAa,EACb,IAAe,EACS,EAAE;IAC1B,MAAM,MAAM,GAAG,WAAW,CAAC,KAAK,EAAE,IAAI,CAAC,CAAC;IACxC,OAAO,IAAI,CACT,SAAS,CAAC,UAAU,CAAC,KAAK,EAAE,IAAI,CAAC,CAAC,EAClC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,WAAW,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,EAC9C,MAAM,CAAC,CAAC,CAAC,EAAoB,EAAE,CAAC,CAAC,KAAK,IAAI,CAAC,CAC5C,CAAC;AACJ,CAAC,CAAC;AAEF,kGAAkG;AAClG,MAAM,QAAQ,GAAG,CAAC,OAA0B,EAAE,IAAe,EAAE,EAAE;IAC/D,IAAI,QAAQ,CAAC,OAAO,CAAC,QAAQ,CAAC,KAAK,WAAW;QAAE,OAAO,EAAE,CAAC;IAC1D,OAAO;QACL,IAAI;YACF,MAAM,EAAE,GAAG,OAAO,CAAC,UAAU,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;YACjD,aAAa,CAAC,OAAO,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,IAAI,CAAC;iBAC3C,OAAO,CAAC,WAAW,CAAC;iBACpB,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE;gBACb,OAAO,CAAC,MAAM,CAAC;oBACb,GAAG,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE;oBAC3B,SAAS,EAAE,CAAC,CAAC,OAAO;oBACpB,IAAI,EAAE,EAAE,IAAI,EAAE,aAAa,CAAC,CAAC,CAAC,EAAE;iBACjC,CAAC,CAAC;YACL,CAAC,CAAC,CAAC;QACP,CAAC;KACF,CAAC;AACJ,CAAC,CAAC;AAEF,mDAAmD;AACnD,MAAM,CAAC,MAAM,gBAAgB,GAAG,CAC9B,IAAe,EACqC,EAAE,CAAC,CAAC;IACxD,cAAc,EAAE;QACd,IAAI,EAAE;YACJ,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE;gBACJ,WAAW,EACT,6NAA6N;gBAC/N,GAAG,EAAE,WAAW,CAAC,IAAI,CAAC;aACvB;YACD,MAAM,EAAE,EAAE;YACV,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,eAAe,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,UAAU,CAAC,CAAC,CAAC;SAC1E;QACD,MAAM,EAAE,CAAC,OAAO,EAAE,EAAE,CAAC,QAAQ,CAAC,OAAO,EAAE,IAAI,CAAC;KAC7C;CACF,CAAC,CAAC;AAEH,qDAAqD;AACrD,MAAM,CAAC,MAAM,kBAAkB,GAAgD;IAC7E,oBAAoB,EAAE,OAAO;CAC9B,CAAC"}
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
# The gates of a plan v2
|
|
2
|
+
|
|
3
|
+
Issue [#332](https://github.com/zernie/paperlint/issues/332). A measurement study's `plan.json` says,
|
|
4
|
+
before the run, what makes its number worth believing; after the run its `result.json` shows it.
|
|
5
|
+
Two rules hold it: [`repro/study-registered`](../rules/repro/study-registered.md) reads the plan,
|
|
6
|
+
[`repro/study-result`](../rules/repro/study-result.md) reads the result. The code is
|
|
7
|
+
`src/domain/study-gates.ts`, `src/domain/study-result.ts` and the parser in
|
|
8
|
+
`src/adapters/study-registry/`.
|
|
9
|
+
|
|
10
|
+
## Where it came from
|
|
11
|
+
|
|
12
|
+
A consumer paper's main measurement drew rules out of repositories, ran several executed "readings" of
|
|
13
|
+
each rule, and counted how often the readings disagreed. Refuting its design by hand found two holes:
|
|
14
|
+
|
|
15
|
+
- **It could not lose.** A reading that flagged nothing made every other reading "disagree", so the
|
|
16
|
+
number came out in the design's favour whatever the rules said.
|
|
17
|
+
- **Readings were not tied to the rule's text.** For "max line length 100", readings of 79, 88, 100
|
|
18
|
+
and 120 counted as interpretations, though three of them contradict the text. The number measured
|
|
19
|
+
how varied configurations are, not how vague the rule is.
|
|
20
|
+
|
|
21
|
+
The neighbours of that paper that hold up report four checks this design lacked: two raters with an
|
|
22
|
+
agreement statistic (Cohen's kappa 0.86 and 0.89 in two of them), errors planted to see that the
|
|
23
|
+
checker catches them, every dropped unit counted, and an audit of the extractor — one neighbour read
|
|
24
|
+
50 files in full and reports that its extraction caught 66.3% of the eligible rules. A preregistration template —
|
|
25
|
+
AsPredicted's eight questions — asks "Specify exactly which analyses you will conduct to examine the
|
|
26
|
+
main question/hypothesis"; the plan had no field for what decides the question.
|
|
27
|
+
|
|
28
|
+
## Versions
|
|
29
|
+
|
|
30
|
+
A plan says it is v2 with `"$schema": "https://unpkg.com/paperlint/schemas/plan.v2.schema.json"`.
|
|
31
|
+
|
|
32
|
+
| plan | judged |
|
|
33
|
+
| ----------------- | ------------------------------------------------------------------------- |
|
|
34
|
+
| no `$schema` (v1) | as before; it may carry none of the gates |
|
|
35
|
+
| v2 | every gate present: `null` (not decided: a draft), `"none"`, or its value |
|
|
36
|
+
|
|
37
|
+
A measurement registered on or after **2026-10-31** must have a plan v2 (`plan-version`), so dropping
|
|
38
|
+
the marker is no way around the gates. A measurement registered earlier keeps its plan v1; nothing it
|
|
39
|
+
had turns red. `paperlint studies new` writes a measurement's plan as v2. A later field is a v3, the
|
|
40
|
+
same way: v2 plans keep passing.
|
|
41
|
+
|
|
42
|
+
**There is no optional gate.** A gate left out of a plan v2 makes the file "not a study plan". The one
|
|
43
|
+
way to say "not this study" is the literal `"none"`, written where the reader of the diff sees it.
|
|
44
|
+
|
|
45
|
+
## The five gates
|
|
46
|
+
|
|
47
|
+
### `mode.we_lose_if` — the claim can lose
|
|
48
|
+
|
|
49
|
+
```json
|
|
50
|
+
"mode": {
|
|
51
|
+
"kind": "decide_before_run",
|
|
52
|
+
"outputs": ["result.json"],
|
|
53
|
+
"decision": { "file": "repro/P1-S07-readings/decision.md", "note": "…" },
|
|
54
|
+
"we_lose_if": { "result": "share_upper95", "below": 0.20 }
|
|
55
|
+
}
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`result` names a number the study writes in `result.json`; `below` **or** `above` (exactly one) is
|
|
59
|
+
the threshold. It lives only in a decide-first mode: an `exploratory` study has no claim to lose, and
|
|
60
|
+
its mode cannot hold one. The name is the study's own: `share_upper95` in one study, `fix_rate` in
|
|
61
|
+
another.
|
|
62
|
+
|
|
63
|
+
- **Checked:** present, its shape; after the run, the number is there, and when it crosses the
|
|
64
|
+
threshold `result.json` says `"claim": "lost"` (`claim-lost`), never of a claim that holds
|
|
65
|
+
(`claim-holds`).
|
|
66
|
+
- **Not checked:** that the number named is the right one to decide the question — the author's
|
|
67
|
+
check.
|
|
68
|
+
|
|
69
|
+
**Frozen at the first output.** All five gates are held by git's history
|
|
70
|
+
([`repro/outputs-before-decision`](../rules/repro/outputs-before-decision.md)): the plan as it stood
|
|
71
|
+
at the commit of the study's first output (the first that came after its decision) must declare each
|
|
72
|
+
gate exactly as HEAD's plan does. A gate edited since is `gates-changed`: a threshold moved after the
|
|
73
|
+
result was seen can only follow the result. A plan that was not in history yet at that commit, or was
|
|
74
|
+
a v1 or unreadable there, is `gates-not-fixed`. Edits to other parts of the plan (its question, its
|
|
75
|
+
source) are not reported.
|
|
76
|
+
|
|
77
|
+
### `second_check` — two raters on each labelling step
|
|
78
|
+
|
|
79
|
+
```json
|
|
80
|
+
"second_check": [
|
|
81
|
+
{ "step": "lint-checkable", "min_kappa": 0.60 },
|
|
82
|
+
{ "step": "admissible", "min_kappa": 0.60 }
|
|
83
|
+
]
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
One entry per step on which a model or a person labels units, with the least Cohen's kappa
|
|
87
|
+
(agreement between two raters beyond chance) the two raters must reach. The threshold is fixed
|
|
88
|
+
before labelling, so a kappa of 0.4 cannot be called good enough after it is seen. Landis and Koch's
|
|
89
|
+
bands (0.61–0.80 "substantial") are a common reference, and an opinion, not evidence: the study
|
|
90
|
+
chooses its own and says why in its design. `"none"` when nothing is labelled.
|
|
91
|
+
|
|
92
|
+
In the consumer paper the second hole above is closed here: whether a reading fits the rule's text
|
|
93
|
+
is the `admissible` labelling step.
|
|
94
|
+
|
|
95
|
+
- **Checked:** each step's kappa is in `result.json`'s `kappa` and at least its `min_kappa`.
|
|
96
|
+
- **Not checked:** who the raters were, how the subset was drawn — that stays in the design. A
|
|
97
|
+
`"none"` on a study that does label is caught only by the author's check.
|
|
98
|
+
|
|
99
|
+
### `self_test` — the checker is run on known answers first
|
|
100
|
+
|
|
101
|
+
```json
|
|
102
|
+
"self_test": { "made_by": "scripts/extract.selftest.mjs" }
|
|
103
|
+
"self_test": { "must_catch": "self-test/bad.js", "must_pass": "self-test/good.js" }
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
When the study runs a check (a linter rule, a script, a classifier) its zero means nothing until the
|
|
107
|
+
check has caught an input it must catch and passed one it must pass. Either a script makes and runs
|
|
108
|
+
such pairs (`made_by`), or the plan names one pair; the inputs are of whatever type the check reads.
|
|
109
|
+
Paths are inside the study's folder. `"none"` when the study runs no check.
|
|
110
|
+
|
|
111
|
+
- **Checked:** the paths stay in the folder; after the run, `result.json`'s
|
|
112
|
+
`"self_test": { "passed": n, "failed": n }` is there and `failed` is 0.
|
|
113
|
+
|
|
114
|
+
### `funnel` — every drop counted
|
|
115
|
+
|
|
116
|
+
```json
|
|
117
|
+
"funnel": ["no linter rule", "< 2 admissible readings", "no files in scope", "clone failed"]
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The reasons a drawn unit may be dropped, each once, fixed before the draw — a study cannot invent a
|
|
121
|
+
reason for a unit it would rather not count. Like a trial's flow diagram, which "displays the
|
|
122
|
+
progress of all participants through the trial" (CONSORT). `"none"` when no unit is dropped.
|
|
123
|
+
|
|
124
|
+
- **Checked:** `result.json`'s `funnel` holds a count for every reason (0 when none).
|
|
125
|
+
|
|
126
|
+
### `recall_audit` — how much the extractor misses
|
|
127
|
+
|
|
128
|
+
```json
|
|
129
|
+
"recall_audit": { "n_files": 50, "min_recall": 0.6 }
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
A study that draws units out of artefacts (rules out of instruction files, functions out of code)
|
|
133
|
+
counts only what its extractor found. A person reads `n_files` files of a seeded sample in full and
|
|
134
|
+
marks every unit there; the recall is the share of those units the extractor found too. Without it, a
|
|
135
|
+
reviewer's "how many did your parser miss?" has no number. `"none"` when nothing is extracted (the
|
|
136
|
+
units are given, or a person draws them by hand).
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
STAGE WHO WRITES WHICH RULE RED WHEN
|
|
140
|
+
───────────── ───────────────────────── ────────────────────────── ──────────────────────────────
|
|
141
|
+
1 plan the author (+ agent): repro/study-registered key missing (schema error);
|
|
142
|
+
(before "recall_audit": null while the study runs
|
|
143
|
+
the run) {n_files, min_recall} repro/outputs-before- changed after the first output
|
|
144
|
+
decision (gates-changed)
|
|
145
|
+
2 run the study's script: — —
|
|
146
|
+
extracts units
|
|
147
|
+
3 audit a PERSON reads n_files — —
|
|
148
|
+
sampled files in full,
|
|
149
|
+
marks every unit (a
|
|
150
|
+
build-label-page page)
|
|
151
|
+
4 result the study's script: repro/study-result counts missing (no-recall-audit);
|
|
152
|
+
(after "recall_audit": (when the study is done) files < n_files (recall-sample);
|
|
153
|
+
the run) {files, units_in_files, extracted / units_in_files
|
|
154
|
+
extracted} < min_recall (recall);
|
|
155
|
+
extracted > units_in_files
|
|
156
|
+
(result unreadable)
|
|
157
|
+
FIRES: in the editor on save · npx paperlint lint · the paper's CI
|
|
158
|
+
CANNOT SEE: whether the person really read the files in full, or counted right;
|
|
159
|
+
whether the sample was drawn as the design says
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
`result.json` carries the counts, not a share: the files are checked against the plan, and the
|
|
163
|
+
recall is computed by the rule (`extracted / units_in_files`), so a share rounded in the study's
|
|
164
|
+
favour cannot pass.
|
|
165
|
+
|
|
166
|
+
- **Checked:** the counts are there; at least `n_files` files; the recall at least `min_recall`.
|
|
167
|
+
- **Not checked:** the reading itself — the labels of the audit are data the study exports and
|
|
168
|
+
commits, like any other labels.
|
|
169
|
+
|
|
170
|
+
## `result.json`
|
|
171
|
+
|
|
172
|
+
The study's own script writes it beside the plan; the rule only reads it.
|
|
173
|
+
|
|
174
|
+
```json
|
|
175
|
+
{
|
|
176
|
+
"share_upper95": 0.31,
|
|
177
|
+
"kappa": { "lint-checkable": 0.71, "admissible": 0.64 },
|
|
178
|
+
"self_test": { "passed": 212, "failed": 0 },
|
|
179
|
+
"funnel": {
|
|
180
|
+
"no linter rule": 400,
|
|
181
|
+
"< 2 admissible readings": 37,
|
|
182
|
+
"no files in scope": 5,
|
|
183
|
+
"clone failed": 3
|
|
184
|
+
},
|
|
185
|
+
"recall_audit": { "files": 50, "units_in_files": 95, "extracted": 63 }
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Its own numbers go at the top under any name. `kappa`, `self_test`, `funnel`, `recall_audit` and
|
|
190
|
+
`claim` have the shapes above. It is judged once the study is `done`.
|
|
191
|
+
|
|
192
|
+
## What a machine never sees
|
|
193
|
+
|
|
194
|
+
A `"none"` that is false, a threshold chosen to be easy, a `result` that names the wrong number. These
|
|
195
|
+
are read by the author when they approve the plan (`author_check`) — which is why the gates are few,
|
|
196
|
+
named in plain words, and visible in the diff.
|
|
197
|
+
|
|
198
|
+
## Rejected
|
|
199
|
+
|
|
200
|
+
- **Optional gates, "when they apply".** An author who forgets one would read as an author who did
|
|
201
|
+
not need it. Every gate is present and says `"none"` out loud.
|
|
202
|
+
- **A "kind of measurement" field choosing which gates apply.** One more field that can be wrong;
|
|
203
|
+
`"none"` per gate says the same thing where it is checked.
|
|
204
|
+
- **Free-text gates** (`why`, `raters`, the subset's share and seed). No check reads them; they stay
|
|
205
|
+
in the study's design.
|
|
206
|
+
- **A gate per study design** (admissibility of executed versions, a replication frame, one unit per
|
|
207
|
+
cluster). Specific to one paper so far; a paper keeps such keys in the plan's `extra` until two
|
|
208
|
+
papers have needed them unchanged.
|
|
@@ -42,6 +42,12 @@ A study decided before its run — its plan's `mode` is `decide_before_run`, the
|
|
|
42
42
|
- **`shallow`** — the clone is shallow, so every file's first commit is the clone's: no order can be
|
|
43
43
|
read, and the rule says so instead of judging.
|
|
44
44
|
- **`no-history`** — git could not answer (not installed, not a repository).
|
|
45
|
+
- **`gates-changed`** — the plan is a plan v2 ([`docs/design/study-gates.md`](../../design/study-gates.md)),
|
|
46
|
+
and one of its gates (`mode.we_lose_if`, `second_check`, `self_test`, `funnel`, `recall_audit`) differs from what
|
|
47
|
+
the plan declared at the commit of the study's first output after its decision: a threshold moved
|
|
48
|
+
after a result exists. One finding per changed gate, with its value then and now.
|
|
49
|
+
- **`gates-not-fixed`** — the plan v2 was not in history at that commit, or was a plan v1 or no
|
|
50
|
+
readable plan there: its gates were not fixed before the run.
|
|
45
51
|
|
|
46
52
|
The study's own files — `plan.json`, `README.md` and the decision file — are never outputs. A symbolic
|
|
47
53
|
link in the study's folder is one entry, as git stores it, and is never followed: a link to the folder
|
|
@@ -93,6 +99,8 @@ No rule options. The study declares its mode in its plan.
|
|
|
93
99
|
- What the decision says. The gate is that it is committed, not its content.
|
|
94
100
|
- **A decision amended after the outputs**: the order is between FIRST commits. A later edit of the
|
|
95
101
|
decision file is visible in its history (`git log -- <file>`) but not reported.
|
|
102
|
+
- A plan v1, or a plan v2's parts other than its four gates: the question, the source and the note
|
|
103
|
+
may be reworded after the outputs. Only the gates are frozen.
|
|
96
104
|
- That nobody ran the study before the decision and kept the outputs out of git. History records
|
|
97
105
|
what was committed; a run on a machine that never committed it leaves no trace.
|
|
98
106
|
- A study whose row names a folder that is not one inside `repro/` (`../x`), or a folder that is a
|
|
@@ -132,6 +132,7 @@ author's check of the drawn units. `paperlint studies new` writes the template b
|
|
|
132
132
|
|
|
133
133
|
```json
|
|
134
134
|
{
|
|
135
|
+
"$schema": "https://unpkg.com/paperlint/schemas/plan.v2.schema.json",
|
|
135
136
|
"question": "",
|
|
136
137
|
"population": {
|
|
137
138
|
"source": "",
|
|
@@ -143,23 +144,36 @@ author's check of the drawn units. `paperlint studies new` writes the template b
|
|
|
143
144
|
"mode": {
|
|
144
145
|
"kind": "decide_before_run",
|
|
145
146
|
"outputs": [],
|
|
146
|
-
"decision": { "file": "repro/P1-S02-census/decision.md", "note": "" }
|
|
147
|
-
|
|
147
|
+
"decision": { "file": "repro/P1-S02-census/decision.md", "note": "" },
|
|
148
|
+
"we_lose_if": null
|
|
149
|
+
},
|
|
150
|
+
"second_check": null,
|
|
151
|
+
"self_test": null,
|
|
152
|
+
"funnel": null,
|
|
153
|
+
"recall_audit": null
|
|
148
154
|
}
|
|
149
155
|
```
|
|
150
156
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
|
154
|
-
|
|
|
155
|
-
| `
|
|
156
|
-
| `population.
|
|
157
|
-
| `population.draw.
|
|
158
|
-
| `
|
|
159
|
-
| `
|
|
160
|
-
| `
|
|
161
|
-
| `
|
|
162
|
-
| `
|
|
157
|
+
That is a measurement's template, a plan v2. Another kind's has no `$schema`, `mode` or gates.
|
|
158
|
+
|
|
159
|
+
| field | what it holds |
|
|
160
|
+
| ------------------------ | ------------------------------------------------------------------------------------ |
|
|
161
|
+
| `question` | what the study measures, one sentence |
|
|
162
|
+
| `population.source` | where the units come from: the census, crawl or registry, with its date |
|
|
163
|
+
| `population.draw.kind` | `all` (the whole population, a census: no seed) or `seeded` (a sample) |
|
|
164
|
+
| `population.draw.method` | how the sample is drawn: `census`, `random`, `stratified` … |
|
|
165
|
+
| `population.draw.seed` | `seeded` only, and required there: the draw's seed, a string or a number |
|
|
166
|
+
| `arms` | the conditions compared, each `{ "name": … }`, each name once; `[]` for a census |
|
|
167
|
+
| `primary_outcome` | the one number the study reports first |
|
|
168
|
+
| `author_check` | `null` until the author has seen the drawn units; then `by`, `date`, `quote` |
|
|
169
|
+
| `extra` | optional: the paper's own keys (hypotheses, models, budget), kept unread |
|
|
170
|
+
| `mode` | a measurement's: exactly one of `decide_before_run` or `exploratory` |
|
|
171
|
+
| `$schema` | a plan v2 says so here; without it the plan is v1 |
|
|
172
|
+
| `mode.we_lose_if` | v2, decide-first only: the number its claim loses by, `below` or `above` a threshold |
|
|
173
|
+
| `second_check` | v2: per labelling step, the least kappa between two raters, or `"none"` |
|
|
174
|
+
| `self_test` | v2: how the study's check is run on known answers first, or `"none"` |
|
|
175
|
+
| `funnel` | v2: the reasons a drawn unit may be dropped, or `"none"` |
|
|
176
|
+
| `recall_audit` | v2: files a person reads in full, and the least recall of the extractor, or `"none"` |
|
|
163
177
|
|
|
164
178
|
`mode` is required of a `measurement` (other kinds may omit it): how its outcomes are kept from the
|
|
165
179
|
decisions on its inputs — decided before the run (the default the template writes; its `outputs`,
|
|
@@ -168,6 +182,12 @@ and its `decision` file relative to the paper, committed before the run) or `exp
|
|
|
168
182
|
rule that holds the order in git, [`repro/outputs-before-decision`](outputs-before-decision.md), are
|
|
169
183
|
in [`docs/design/study-modes.md`](../../design/study-modes.md).
|
|
170
184
|
|
|
185
|
+
The four gates of a plan v2 — what each is for, what the machine checks and what it cannot — are in
|
|
186
|
+
[`docs/design/study-gates.md`](../../design/study-gates.md). Each is `null` until decided (the plan is
|
|
187
|
+
a draft), `"none"` when the study has none of it, or its value; a plan v2 that leaves one out is not
|
|
188
|
+
a plan. A plan without `$schema` is v1 and is judged as before, and may carry none of the gates. The
|
|
189
|
+
result a done study's plan v2 asks for is judged by [`repro/study-result`](study-result.md).
|
|
190
|
+
|
|
171
191
|
The template's draw is of all the population; a sampled study changes it to
|
|
172
192
|
`{ "kind": "seeded", "method": "random", "seed": 7 }`. The population's size is not stored: it is
|
|
173
193
|
counted from the source, and a stored copy could only contradict it.
|
|
@@ -212,7 +232,10 @@ The rule reports:
|
|
|
212
232
|
- a row's `plan` that is outside its folder, not a file, reached through a symbolic link, or not of the plan's shape — and, once the
|
|
213
233
|
study has run (`running`, `done`, `stopped`, `superseded`), a plan that leaves `question`,
|
|
214
234
|
`population.source`, `population.draw.method` or `primary_outcome` empty, or — for a measurement —
|
|
215
|
-
declares no `mode` or leaves one of its texts empty
|
|
235
|
+
declares no `mode` or leaves one of its texts empty, or — for a plan v2 — leaves a gate `null`
|
|
236
|
+
(`bad-plan`);
|
|
237
|
+
- a measurement registered on or after 2026-10-31 whose plan is v1, without `$schema`
|
|
238
|
+
(`plan-version`): its gates are not declared. A measurement registered earlier keeps its plan v1;
|
|
216
239
|
- a study of a `headline` measurement that has run with no `author_check` in its plan — or no plan —
|
|
217
240
|
or, when the registry names its `author`, a check by anyone else (`author-check`).
|
|
218
241
|
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# repro/study-result
|
|
2
|
+
|
|
3
|
+
**Level:** error · **Reads:** `repro/studies.json`, each `done` study's `plan.json` (v2) and
|
|
4
|
+
`result.json` · **Reported on:** `PIPELINE-STATUS.md`, line 1
|
|
5
|
+
|
|
6
|
+
A study whose plan is v2 fixed, before its run, what makes its number believable: when its claim
|
|
7
|
+
loses, how well two raters must agree, that its checker is tried on known answers, and why a unit may
|
|
8
|
+
be dropped ([the gates](../../design/study-gates.md)). Once the study is `done`, its `result.json`
|
|
9
|
+
must show each of them.
|
|
10
|
+
|
|
11
|
+
## What it catches
|
|
12
|
+
|
|
13
|
+
For each study that is `done`, whose plan is ready and v2:
|
|
14
|
+
|
|
15
|
+
| problem | when |
|
|
16
|
+
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
17
|
+
| `no-result-file` | the study's folder has no `result.json` |
|
|
18
|
+
| `result-linked` | `result.json` is reached through a symbolic link |
|
|
19
|
+
| `result-unreadable` | `result.json` is not JSON, or `kappa`, `self_test`, `funnel`, `recall_audit` or `claim` is not of its shape (a kappa outside -1..1, such as a percentage written as `75`, is not one; an audit counting more units extracted than the person found, or no unit found, is not one) |
|
|
20
|
+
| `no-number` | the number `mode.we_lose_if.result` names is not a number in `result.json` |
|
|
21
|
+
| `claim-lost` | that number crosses the threshold, and `result.json` does not say `"claim": "lost"` |
|
|
22
|
+
| `claim-holds` | `result.json` says `"claim": "lost"`, and the number does not cross the threshold |
|
|
23
|
+
| `kappa` | a step of `second_check` has no kappa in `result.json`'s `kappa`, or one below its `min_kappa` |
|
|
24
|
+
| `self-test` | `self_test` is declared, and `result.json`'s `self_test` is missing or `failed` is not 0 |
|
|
25
|
+
| `self-test-empty` | `self_test` is declared, and `result.json`'s `self_test` ran no case: `"passed": 0, "failed": 0` |
|
|
26
|
+
| `funnel` | a reason of `funnel` has no count in `result.json`'s `funnel` |
|
|
27
|
+
| `no-recall-audit` | `recall_audit` is declared, and `result.json` has no `recall_audit` counts |
|
|
28
|
+
| `recall-sample` | the audit read fewer `files` than the plan's `n_files` |
|
|
29
|
+
| `recall` | `extracted / units_in_files` is below the plan's `min_recall`: the extractor misses too much |
|
|
30
|
+
|
|
31
|
+
A gate that is `"none"` asks nothing; an exploratory study has no `we_lose_if`.
|
|
32
|
+
|
|
33
|
+
## Why
|
|
34
|
+
|
|
35
|
+
A threshold fixed before the run and checked by hand after it can be moved, forgotten or quietly
|
|
36
|
+
reinterpreted. Here the plan holds the threshold and the result holds the number, and the engine
|
|
37
|
+
compares them on every lint. A claim that lost stays red until the result says so in so many words —
|
|
38
|
+
`"claim": "lost"` — where the paper reads its numbers from, so the loss cannot sit unnoticed in a
|
|
39
|
+
table. The same holds for a kappa below the bar set before labelling, a checker that missed a known
|
|
40
|
+
answer, and a dropped unit nobody counted.
|
|
41
|
+
|
|
42
|
+
## Examples
|
|
43
|
+
|
|
44
|
+
Failing — the plan declares the funnel reason `timed out`, the result counts none:
|
|
45
|
+
|
|
46
|
+
```json
|
|
47
|
+
{
|
|
48
|
+
"pass_rate": 0.7,
|
|
49
|
+
"kappa": { "pass": 0.71 },
|
|
50
|
+
"self_test": { "passed": 4, "failed": 0 },
|
|
51
|
+
"funnel": {}
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
```text
|
|
56
|
+
PIPELINE-STATUS.md
|
|
57
|
+
1:1 error The `result.json` of `P1-S01` counts no drop for `timed out`, declared in its plan's
|
|
58
|
+
`funnel`. What to do: write each in `funnel`, 0 when none was dropped for it.
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Failing — `we_lose_if` is `{ "result": "pass_rate", "below": 0.5 }` and the rate is 0.4:
|
|
62
|
+
|
|
63
|
+
```text
|
|
64
|
+
1:1 error `P1-S01`'s claim LOST: its `pass_rate` is 0.4, below 0.5, the threshold its plan fixed
|
|
65
|
+
before the run. What to do: report the loss in the paper, and say so in `result.json`
|
|
66
|
+
with `"claim": "lost"`; the threshold is not moved after the run.
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Passing — every gate shown:
|
|
70
|
+
|
|
71
|
+
```json
|
|
72
|
+
{
|
|
73
|
+
"pass_rate": 0.7,
|
|
74
|
+
"kappa": { "pass": 0.71 },
|
|
75
|
+
"self_test": { "passed": 4, "failed": 0 },
|
|
76
|
+
"funnel": { "timed out": 2 }
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
## Options / preset fields
|
|
81
|
+
|
|
82
|
+
None. The rule is on at `error` in paperlint's config; a paper turns it off in its `paperlint.json`.
|
|
83
|
+
|
|
84
|
+
## What it does not check
|
|
85
|
+
|
|
86
|
+
- A study that is not `done`, a plan v1, a draft or a plan that does not parse
|
|
87
|
+
([`repro/study-registered`](study-registered.md) reports those).
|
|
88
|
+
- Whether a gate's `"none"` is true, whether the threshold is a sensible one, or whether the number
|
|
89
|
+
is computed right: the author's check of the plan, and the study's own artifact.
|
|
90
|
+
- Whether the gates were fixed before the outputs: [`repro/outputs-before-decision`](outputs-before-decision.md)
|
|
91
|
+
holds them to the plan as it stood at the first output.
|
|
92
|
+
- Whether the person of the recall audit read the files in full, or counted right: the audit's own
|
|
93
|
+
labels, exported and committed like any other.
|
|
94
|
+
|
|
95
|
+
## How to fix
|
|
96
|
+
|
|
97
|
+
Have the study's script write `result.json` beside its plan, with the number `we_lose_if` names at
|
|
98
|
+
the top, and `kappa`, `self_test`, `funnel` and `recall_audit` as the plan declares them
|
|
99
|
+
([the shape](../../design/study-gates.md#resultjson)). A lost claim is reported, not hidden: add
|
|
100
|
+
`"claim": "lost"` and say so in the paper. Never edit the plan's thresholds after the run.
|
package/docs/rules.md
CHANGED
|
@@ -21,6 +21,7 @@ reads and when it fails. Errors fail `paperlint lint`; warnings print and do not
|
|
|
21
21
|
| [`repro/study-registered`](rules/repro/study-registered.md) | error | the listing of the paper's `repro/`, `repro/studies.json`, `repro/README.md`, each study's `plan.json` | a study folder with no row in `repro/studies.json`, a row whose folder is missing or not `<id>-<slug>`, a bad or repeated id, a title over 8 words, `done` without a result and an existing result file, `superseded` without an existing successor, a row naming a measurement the registry does not declare, a measurement with no study or an unknown requirement, a headline measurement not approved by the author, a plan that is missing, malformed or — once the study runs — unfilled, a headline study that has run without the author's check in its plan, or the index in `repro/README.md` out of sync with the registry; `paperlint studies new` and `paperlint studies index` write both — [page](rules/repro/study-registered.md) |
|
|
22
22
|
| [`repro/study-plan-name`](rules/repro/study-plan-name.md) | warn | `repro/studies.json` | a study's plan still named `prereg.json`, its old name: read as the plan, and to be renamed `plan.json` — [page](rules/repro/study-plan-name.md) |
|
|
23
23
|
| [`repro/outputs-before-decision`](rules/repro/outputs-before-decision.md) | error | `repro/studies.json`, each decide-first study's `plan.json` and folder, git's history | an output of a study decided before its run whose first commit does not descend from its decision's first commit, an output while the decision is uncommitted, a decision its outputs match, a shallow or unreadable history; `paperlint studies gate` opens the run only once the decision is committed — [page](rules/repro/outputs-before-decision.md) |
|
|
24
|
+
| [`repro/study-result`](rules/repro/study-result.md) | error | `repro/studies.json`, each done study's `plan.json` (v2) and `result.json` | a done study whose plan is v2 and whose `result.json` is missing, lacks the number its claim is judged by, crosses that threshold without saying `"claim": "lost"`, lacks a labelling step's kappa or has one below the plan's bar, shows a failed self-test, counts no drop for a declared funnel reason, or audits its extractor on too few files or finds its recall below the plan's bar — [page](rules/repro/study-result.md) |
|
|
24
25
|
| [`paper/sources-fresh`](rules/paper/sources-fresh.md) | warn | `paper.tex` → `_build/sources.json` | the paper has not been built (no record of the files TeX read), or a file changed since the last build (its SHA-256 differs) — run `npx paperlint build` |
|
|
25
26
|
| `paper/author-list` | error | `paper.tex` → `_build/references.json`, `_build/sources.json` | an entry's authors are not those of the version it cites — typically the arXiv preprint's list under a published venue. Checked online by `paperlint build` (DBLP), reported on the entry's own line (or at the top of `paper.tex`, naming the entry's file); silent without a current build record |
|
|
26
27
|
| `paper/section-word` | warn | `paper.tex` and its body files | `§` or `\S\ref` instead of the word Section. **Fixable:** `--fix` writes `Section~\ref{…}` / `Section 5` |
|
package/eslint.config.mjs
CHANGED
|
@@ -687,7 +687,7 @@ export default [
|
|
|
687
687
|
* not itself.
|
|
688
688
|
*
|
|
689
689
|
* Measurement on the first run: 15 files with dead imports (`resolve`, `pathToFileURL`) and
|
|
690
|
-
* orphaned constants — remnants of a move from
|
|
690
|
+
* orphaned constants — remnants of a move from the downstream repository this package was extracted from, where those names were needed. All
|
|
691
691
|
* cleaned to zero in the same commit, so the rule opens as a GATE on a clean corpus,
|
|
692
692
|
* not as debt to be silenced.
|
|
693
693
|
*
|