@bompus/house-rules 0.0.0-stage → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +54 -0
- package/LICENSE +21 -0
- package/README.md +347 -2
- package/THIRD_PARTY_NOTICES.md +188 -0
- package/compose.mjs +119 -0
- package/composition.mjs +353 -0
- package/config-view.mjs +194 -0
- package/config.mjs +320 -0
- package/docs/assets/banner-dark.png +0 -0
- package/docs/assets/banner-light.png +0 -0
- package/docs/assets/overview.png +0 -0
- package/docs/configuration.md +193 -0
- package/docs/npm-release.md +45 -0
- package/docs/skill-names.md +50 -0
- package/examples/person/house-rules.json +15 -0
- package/examples/person/rules/10-tooling.md +8 -0
- package/examples/person/rules/20-reporting.md +9 -0
- package/examples/project/AGENTS.md +18 -0
- package/package.json +40 -4
- package/rule-references.mjs +91 -0
- package/rules/core.md +431 -0
- package/rules/modifiers/coded-offers.md +40 -0
- package/rules/modifiers/effort-estimates.md +41 -0
- package/rules/modifiers/land-when-done.md +12 -0
- package/rules/modifiers/low-quota-handoff.md +19 -0
- package/rules/modifiers/multi-agent.md +34 -0
- package/rules/modifiers/no-attribution.md +10 -0
- package/rules/modifiers/plan-files.md +50 -0
- package/rules/modifiers/question-cards.md +48 -0
- package/rules/modifiers/release-batching.md +13 -0
- package/rules/modifiers/scratch-on-disk.md +15 -0
- package/rules/modifiers/shared-host-load.md +48 -0
- package/rules/modifiers/solo-operator.md +13 -0
- package/rules/modifiers/squash-landing.md +22 -0
- package/rules/modifiers/swarmail.md +12 -0
- package/rules/references/github-actions.md +45 -0
- package/rules/references/landing.md +39 -0
- package/rules/references/multi-agent.md +47 -0
- package/rules/references/release-batching.md +43 -0
- package/rules/references/squash-landing.md +44 -0
- package/setup.mjs +375 -0
- package/skills/hr-agent-guidance-audit/LICENSE +21 -0
- package/skills/hr-agent-guidance-audit/SKILL.md +86 -0
- package/skills/hr-agent-guidance-audit/agents/openai.yaml +2 -0
- package/skills/hr-agent-guidance-audit/references/records-and-lessons.md +60 -0
- package/skills/hr-agent-guidance-refresh/LICENSE +21 -0
- package/skills/hr-agent-guidance-refresh/SKILL.md +68 -0
- package/skills/hr-agent-guidance-refresh/agents/openai.yaml +2 -0
- package/skills/hr-agent-guidance-refresh/references/verification.md +47 -0
- package/skills/hr-api-exposure-check/SKILL.md +159 -0
- package/skills/hr-api-exposure-check/agents/openai.yaml +3 -0
- package/skills/hr-audit-choices/SKILL.md +67 -0
- package/skills/hr-audit-choices/agents/openai.yaml +2 -0
- package/skills/hr-benchmarking/NOTICE.md +36 -0
- package/skills/hr-benchmarking/SKILL.md +316 -0
- package/skills/hr-benchmarking/agents/openai.yaml +2 -0
- package/skills/hr-benchmarking/references/javascript.md +17 -0
- package/skills/hr-benchmarking/references/priority-findings-20261006.md +115 -0
- package/skills/hr-benchmarking/references/tools.md +160 -0
- package/skills/hr-benchmarking/scripts/process-attribution.ps1 +20 -0
- package/skills/hr-benchmarking/scripts/process-attribution.py +107 -0
- package/skills/hr-better-accessibility/LICENSE +22 -0
- package/skills/hr-better-accessibility/NOTICE.md +46 -0
- package/skills/hr-better-accessibility/SKILL.md +119 -0
- package/skills/hr-better-accessibility/agents/openai.yaml +3 -0
- package/skills/hr-better-accessibility/focus-and-keyboard.md +130 -0
- package/skills/hr-better-accessibility/forms.md +82 -0
- package/skills/hr-better-accessibility/hit-areas.md +98 -0
- package/skills/hr-better-accessibility/motion-and-zoom.md +81 -0
- package/skills/hr-better-accessibility/screen-readers.md +101 -0
- package/skills/hr-better-accessibility/semantics-and-aria.md +84 -0
- package/skills/hr-better-colors/LICENSE +21 -0
- package/skills/hr-better-colors/NOTICE.md +37 -0
- package/skills/hr-better-colors/SKILL.md +114 -0
- package/skills/hr-better-colors/agents/openai.yaml +3 -0
- package/skills/hr-better-colors/color-formats.md +90 -0
- package/skills/hr-better-colors/color-usage.md +140 -0
- package/skills/hr-better-colors/contrast.md +80 -0
- package/skills/hr-better-colors/palette-generation.md +104 -0
- package/skills/hr-better-colors/palette-structure.md +76 -0
- package/skills/hr-better-colors/token-naming.md +97 -0
- package/skills/hr-better-layout/LICENSE +22 -0
- package/skills/hr-better-layout/NOTICE.md +48 -0
- package/skills/hr-better-layout/SKILL.md +92 -0
- package/skills/hr-better-layout/agents/openai.yaml +3 -0
- package/skills/hr-better-layout/grouping-and-alignment.md +144 -0
- package/skills/hr-better-layout/spacing-and-adaptivity.md +215 -0
- package/skills/hr-better-typography/LICENSE +22 -0
- package/skills/hr-better-typography/NOTICE.md +46 -0
- package/skills/hr-better-typography/SKILL.md +172 -0
- package/skills/hr-better-typography/agents/openai.yaml +3 -0
- package/skills/hr-better-typography/choosing-fonts.md +69 -0
- package/skills/hr-better-typography/css-cheat-sheet.md +65 -0
- package/skills/hr-better-typography/details-and-accessibility.md +119 -0
- package/skills/hr-better-typography/spacing-and-sizing.md +131 -0
- package/skills/hr-better-typography/variable-fonts-and-opentype.md +105 -0
- package/skills/hr-better-typography/wrapping-and-punctuation.md +57 -0
- package/skills/hr-better-writing/LICENSE +22 -0
- package/skills/hr-better-writing/NOTICE.md +46 -0
- package/skills/hr-better-writing/SKILL.md +122 -0
- package/skills/hr-better-writing/agents/openai.yaml +3 -0
- package/skills/hr-change-impact/SKILL.md +113 -0
- package/skills/hr-change-impact/agents/openai.yaml +5 -0
- package/skills/hr-code-review/LICENSE +21 -0
- package/skills/hr-code-review/NOTICE.md +26 -0
- package/skills/hr-code-review/SKILL.md +147 -0
- package/skills/hr-code-review/agents/openai.yaml +3 -0
- package/skills/hr-code-review/references/evidence.md +57 -0
- package/skills/hr-design-exploration/SKILL.md +82 -0
- package/skills/hr-design-exploration/agents/openai.yaml +4 -0
- package/skills/hr-diagnosing-bugs/NOTICE.md +45 -0
- package/skills/hr-diagnosing-bugs/SKILL.md +108 -0
- package/skills/hr-diagnosing-bugs/agents/openai.yaml +3 -0
- package/skills/hr-diagnosing-bugs/references/reporting.md +72 -0
- package/skills/hr-explain-code/NOTICE.md +37 -0
- package/skills/hr-explain-code/SKILL.md +110 -0
- package/skills/hr-explain-code/agents/openai.yaml +5 -0
- package/skills/hr-explain-code/references/helper-brief.md +39 -0
- package/skills/hr-extract-shared-steps/SKILL.md +134 -0
- package/skills/hr-extract-shared-steps/agents/openai.yaml +3 -0
- package/skills/hr-handoff/SKILL.md +91 -0
- package/skills/hr-handoff/agents/openai.yaml +5 -0
- package/skills/hr-handoff/assets/ledger.example.json +10 -0
- package/skills/hr-handoff/assets/receipt.example.json +76 -0
- package/skills/hr-handoff/assets/source.example.json +20 -0
- package/skills/hr-handoff/references/reconciliation.md +105 -0
- package/skills/hr-handoff/scripts/check-reconciliation.mjs +282 -0
- package/skills/hr-house-rules-setup/SKILL.md +100 -0
- package/skills/hr-house-rules-setup/agents/openai.yaml +5 -0
- package/skills/hr-issue-tracker-setup/SKILL.md +49 -0
- package/skills/hr-issue-tracker-setup/agents/openai.yaml +5 -0
- package/skills/hr-lean-plan/SKILL.md +142 -0
- package/skills/hr-lean-plan/agents/openai.yaml +5 -0
- package/skills/hr-maintainability-review/SKILL.md +176 -0
- package/skills/hr-maintainability-review/agents/openai.yaml +5 -0
- package/skills/hr-manual-qa/NOTICE.md +36 -0
- package/skills/hr-manual-qa/SKILL.md +37 -0
- package/skills/hr-manual-qa/agents/openai.yaml +3 -0
- package/skills/hr-navigation-retro/LICENSE +21 -0
- package/skills/hr-navigation-retro/NOTICE.md +5 -0
- package/skills/hr-navigation-retro/SKILL.md +86 -0
- package/skills/hr-navigation-retro/agents/openai.yaml +4 -0
- package/skills/hr-navigation-retro/references/inputs.md +67 -0
- package/skills/hr-navigation-retro/scripts/retro.py +552 -0
- package/skills/hr-navigation-retro/scripts/transcripts.py +205 -0
- package/skills/hr-ordering-tests/SKILL.md +51 -0
- package/skills/hr-ordering-tests/agents/openai.yaml +2 -0
- package/skills/hr-plain-prose/SKILL.md +261 -0
- package/skills/hr-plain-prose/agents/openai.yaml +3 -0
- package/skills/hr-plain-prose/scripts/check.mjs +331 -0
- package/skills/hr-pr-followup/LICENSE +21 -0
- package/skills/hr-pr-followup/NOTICE.md +9 -0
- package/skills/hr-pr-followup/SKILL.md +71 -0
- package/skills/hr-pr-followup/agents/openai.yaml +2 -0
- package/skills/hr-pr-followup/references/review-helpers.md +90 -0
- package/skills/hr-pr-followup/scripts/check-pr-review.ts +510 -0
- package/skills/hr-pr-followup/scripts/resolve-pr-thread.ts +219 -0
- package/skills/hr-progress-report/SKILL.md +49 -0
- package/skills/hr-progress-report/agents/openai.yaml +5 -0
- package/skills/hr-read-reddit/SKILL.md +44 -0
- package/skills/hr-read-reddit/agents/openai.yaml +2 -0
- package/skills/hr-read-reddit/scripts/reddit.ts +252 -0
- package/skills/hr-read-x-links/SKILL.md +42 -0
- package/skills/hr-read-x-links/agents/openai.yaml +2 -0
- package/skills/hr-read-x-links/scripts/x-post.ts +264 -0
- package/skills/hr-split-to-prs/LICENSE +21 -0
- package/skills/hr-split-to-prs/NOTICE.md +9 -0
- package/skills/hr-split-to-prs/SKILL.md +61 -0
- package/skills/hr-split-to-prs/agents/openai.yaml +2 -0
- package/skills/hr-stock-ui-audit/SKILL.md +132 -0
- package/skills/hr-stock-ui-audit/agents/openai.yaml +3 -0
- package/skills/hr-stock-ui-audit/scripts/scan.mjs +619 -0
- package/skills/hr-tdd/LICENSE +21 -0
- package/skills/hr-tdd/NOTICE.md +10 -0
- package/skills/hr-tdd/SKILL.md +40 -0
- package/skills/hr-tdd/agents/openai.yaml +3 -0
- package/skills/hr-tdd/mocking.md +39 -0
- package/skills/hr-tdd/tests.md +77 -0
- package/skills/hr-test-audit/SKILL.md +146 -0
- package/skills/hr-test-audit/agents/openai.yaml +3 -0
- package/skills/hr-test-audit/scripts/floor-guard.ts +137 -0
- package/skills/hr-what-next/SKILL.md +61 -0
- package/skills/hr-what-next/agents/openai.yaml +4 -0
- package/skills/hr-what-next/references/reporting.md +68 -0
- package/skills/hr-writing-for-agents/NOTICE.md +46 -0
- package/skills/hr-writing-for-agents/SKILL-MECHANICS.md +83 -0
- package/skills/hr-writing-for-agents/SKILL.md +108 -0
- package/skills/hr-writing-for-agents/agents/openai.yaml +3 -0
- package/skills/hr-writing-pr/NOTICE.md +48 -0
- package/skills/hr-writing-pr/SKILL.md +104 -0
- package/skills/hr-writing-pr/agents/openai.yaml +2 -0
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Verify installed guidance and observed behavior
|
|
2
|
+
|
|
3
|
+
Use this method when guidance verification is selected. It grants no authority
|
|
4
|
+
for model calls, installation, settings changes, restarts or additional probes.
|
|
5
|
+
The host and repository own the supported commands and safeguards.
|
|
6
|
+
|
|
7
|
+
## Keep the evidence separate
|
|
8
|
+
|
|
9
|
+
| Claim | Evidence needed | Limit |
|
|
10
|
+
| ---------- | ---------------------------------------------------------------------------------- | --------------------------------------------------- |
|
|
11
|
+
| Installed | Compare destination bytes with the selected source revision | Does not establish discovery or loading |
|
|
12
|
+
| Discovered | Inspect the actual host catalog from the relevant repository and account | Does not establish loaded bodies |
|
|
13
|
+
| Loaded | Inspect a fresh context or an approved probe for the selected instruction identity | Does not establish compliance or message visibility |
|
|
14
|
+
| Followed | Inspect the complete trace of a bounded behavior case | Applies to that case, host and tested revision |
|
|
15
|
+
| Prevented | Verify enforcement on the actual host and tool paths | A request or mode label alone is insufficient |
|
|
16
|
+
|
|
17
|
+
Record the source and deployed revisions, host/account, repository, test scope,
|
|
18
|
+
observed result and missing evidence. Keep unavailable, failed and untested
|
|
19
|
+
claims explicit. Linux and native Windows checks are separate; one does not
|
|
20
|
+
qualify the other. Existing sessions may still hold earlier instructions.
|
|
21
|
+
|
|
22
|
+
## Check behavior without supplying the answer
|
|
23
|
+
|
|
24
|
+
For an authorized no-tools context probe, ask for an existing distinctive
|
|
25
|
+
instruction without including its answer. Request that the session stop after
|
|
26
|
+
answering. Capture the full prompt and trace, with private data redacted under
|
|
27
|
+
the owning policy. Missing or incomplete traces leave adherence unverified.
|
|
28
|
+
|
|
29
|
+
Distinguish requested no-tools or skill-read-only behavior, observed adherence
|
|
30
|
+
and verified enforcement. A request alone does not prevent reads, network
|
|
31
|
+
access or tool effects. Read-only mode and never-approval settings do not
|
|
32
|
+
establish those restrictions either. Verify the controls for the actual host
|
|
33
|
+
and tool paths before claiming prevention. A missing check does not prove that
|
|
34
|
+
suitable controls are absent. Do not grant trust or change permissions to make
|
|
35
|
+
a probe pass.
|
|
36
|
+
|
|
37
|
+
## Check adoption without changing personal choices
|
|
38
|
+
|
|
39
|
+
Follow the owning source's supported pin, generator and scoped installer when
|
|
40
|
+
adoption is selected. Preserve account settings, exclusions, invocation policy,
|
|
41
|
+
local patches and repository safeguards. Check destinations and repeat the
|
|
42
|
+
scoped restoration to verify no further writes. Do not infer approval or
|
|
43
|
+
active-context loading from byte equality or an upstream commit.
|
|
44
|
+
|
|
45
|
+
Record each layer's result separately. Unsupported refresh paths remain
|
|
46
|
+
unverified; restarts require their existing authority. A local pass grants no
|
|
47
|
+
production or other-account write authority.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: hr-api-exposure-check
|
|
3
|
+
description: Keep API responses to the fields a real consumer reads and the caller is allowed to read. Use when designing, changing or reviewing what an endpoint, resolver, webhook, serializer or server action returns, or when checking an API for excess data exposure.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# API exposure check
|
|
7
|
+
|
|
8
|
+
A response is both a contract and a disclosure: clients come to depend on
|
|
9
|
+
every field, and anyone holding the response can read it. Build responses
|
|
10
|
+
from what consumers use, and remove or flag the rest.
|
|
11
|
+
|
|
12
|
+
## Two answers per field
|
|
13
|
+
|
|
14
|
+
For every field a response returns, at every depth, write down:
|
|
15
|
+
|
|
16
|
+
1. **Who uses it.** A named consumer: a client screen, an SDK method, a
|
|
17
|
+
partner integration, a background job, or a published contract that
|
|
18
|
+
promises it.
|
|
19
|
+
2. **Why this caller may see it.** The rule that lets this caller, in this
|
|
20
|
+
role, read this value on this object.
|
|
21
|
+
|
|
22
|
+
Client behavior counts as a use, including pagination cursors, concurrency
|
|
23
|
+
ETags or version numbers, and retry timing. Metadata still needs a consumer
|
|
24
|
+
and an access rule; its transport role alone does not justify disclosure.
|
|
25
|
+
Where a consumer needs only to identify a related object, use a small summary
|
|
26
|
+
or an opaque reference instead of the full object.
|
|
27
|
+
|
|
28
|
+
A field with both answers stays. A field missing either one is removed. When
|
|
29
|
+
removal might break a consumer you cannot see (a public API, a third-party
|
|
30
|
+
integration, an old mobile build still in use), do not remove it silently:
|
|
31
|
+
report it as unresolved and say what evidence would settle it.
|
|
32
|
+
|
|
33
|
+
## Finding consumers
|
|
34
|
+
|
|
35
|
+
Search for actual reads, not declarations:
|
|
36
|
+
|
|
37
|
+
- client code, including old app versions still supported
|
|
38
|
+
- generated or hand-written SDKs and their published docs
|
|
39
|
+
- integrations, webhook receivers, scheduled jobs and other services
|
|
40
|
+
- OpenAPI, GraphQL schema or protobuf files that are published to others
|
|
41
|
+
- contract tests and consumer-driven pacts
|
|
42
|
+
|
|
43
|
+
A field that appears only in a type definition, a fixture, a mock or a
|
|
44
|
+
snapshot has no proven consumer. Code that spreads or forwards the whole
|
|
45
|
+
object proves nothing either; follow it to the place that reads a named
|
|
46
|
+
field. If a field is part of a published, versioned contract, it leaves
|
|
47
|
+
through a deprecation: mark it, announce it, keep it for the stated window,
|
|
48
|
+
then remove it in a new version.
|
|
49
|
+
|
|
50
|
+
## Cover the whole response
|
|
51
|
+
|
|
52
|
+
The top-level object is the easy part. Check each of these too:
|
|
53
|
+
|
|
54
|
+
- nested objects and embedded relations, which often serialize a full model
|
|
55
|
+
- every item in a list, and the shape of an empty list
|
|
56
|
+
- pagination and metadata blocks (total counts can reveal what the caller
|
|
57
|
+
cannot otherwise see)
|
|
58
|
+
- response headers, including debug, server and timing headers
|
|
59
|
+
- error bodies, for every status the endpoint can return
|
|
60
|
+
- each role or tenant variant of the same endpoint
|
|
61
|
+
- webhook payloads and server-action return values, which are easy to forget
|
|
62
|
+
|
|
63
|
+
Hiding a field in the UI does not protect it. The raw response is visible in
|
|
64
|
+
browser tools, proxies and logs. If the screen should not show it, the server
|
|
65
|
+
should not send it.
|
|
66
|
+
|
|
67
|
+
## Building responses
|
|
68
|
+
|
|
69
|
+
- Map from an explicit allowlist or a purpose-built output type per use. Never
|
|
70
|
+
serialize a whole model or database row and then delete the fields you
|
|
71
|
+
remember. A column added next month will leak by default under that pattern.
|
|
72
|
+
- Check access to the object and to each sensitive field before mapping, not
|
|
73
|
+
after. Mapping first and filtering later invites a code path that forgets
|
|
74
|
+
the filter.
|
|
75
|
+
- When the client chooses fields (GraphQL selections, `fields=` or `include=`
|
|
76
|
+
parameters, sparse fieldsets), intersect its request with a server-side
|
|
77
|
+
allowlist for that caller. A client request never widens what the caller
|
|
78
|
+
may see.
|
|
79
|
+
- When the stack supports runtime output validation, use a closed schema that
|
|
80
|
+
rejects undeclared properties. Keep the explicit output allowlist either way.
|
|
81
|
+
- Fetch only the columns the response needs where practical. The explicit
|
|
82
|
+
output type stays the boundary either way.
|
|
83
|
+
- In GraphQL, check access on every sensitive field that stays queryable,
|
|
84
|
+
not only on the parent object.
|
|
85
|
+
- Give errors their own minimal shape: a stable code, a short message safe to
|
|
86
|
+
show, and a request or correlation ID. Stack traces, SQL, internal paths,
|
|
87
|
+
upstream responses and validation internals go to the server log.
|
|
88
|
+
- Prefer separate output types for separate audiences (public, owner, admin)
|
|
89
|
+
over one type with conditional blanks. A `null` where a field used to be
|
|
90
|
+
still tells the reader the field exists.
|
|
91
|
+
|
|
92
|
+
## High-risk fields
|
|
93
|
+
|
|
94
|
+
Treat these as exposure by default and require a strong second answer:
|
|
95
|
+
|
|
96
|
+
- credentials, tokens, password hashes, API keys, session or reset data
|
|
97
|
+
- personal data: contact details, addresses, birth dates, government IDs,
|
|
98
|
+
location, health data
|
|
99
|
+
- financial data: card or bank details, balances, billing history
|
|
100
|
+
- permission internals, role lists, feature flags, fraud or risk scores,
|
|
101
|
+
moderation notes
|
|
102
|
+
- internal identifiers that enable enumeration, plus hostnames, IPs, bucket
|
|
103
|
+
names, queue names, versions and other infrastructure details
|
|
104
|
+
- any data belonging to another user or tenant
|
|
105
|
+
|
|
106
|
+
## Tests
|
|
107
|
+
|
|
108
|
+
Test the endpoint's real response, not only its mapping function. For each
|
|
109
|
+
caller type the endpoint supports, assert the exact set of keys or validate
|
|
110
|
+
against a closed schema that rejects unknown properties. Cover anonymous,
|
|
111
|
+
owner, other-user, other-tenant, admin and reduced-access callers where those
|
|
112
|
+
cases apply. Include:
|
|
113
|
+
|
|
114
|
+
- fields that must be absent for this caller, named explicitly
|
|
115
|
+
- nested objects and list items, not only the top level
|
|
116
|
+
- supported detail, list and search paths, including empty results
|
|
117
|
+
- error responses, for each status the endpoint returns
|
|
118
|
+
- a guard that adding a field to the underlying model does not change the
|
|
119
|
+
response; a closed schema or exact key set does this for free
|
|
120
|
+
|
|
121
|
+
A test that only checks that expected fields are present will pass while
|
|
122
|
+
extra fields leak.
|
|
123
|
+
|
|
124
|
+
## Scope
|
|
125
|
+
|
|
126
|
+
Keep the work to the response surface. Do not widen it into general input
|
|
127
|
+
validation or security cleanup unless asked, and leave unrelated
|
|
128
|
+
working-tree changes alone.
|
|
129
|
+
|
|
130
|
+
## Reporting changes
|
|
131
|
+
|
|
132
|
+
After changing responses, report the endpoints changed, the fields removed
|
|
133
|
+
and why, the fields kept with the consumer that needs each, the access rules
|
|
134
|
+
added or kept, and the tests run with any unresolved compatibility risk.
|
|
135
|
+
|
|
136
|
+
## Reporting findings
|
|
137
|
+
|
|
138
|
+
When reviewing, sort each finding into one of three kinds:
|
|
139
|
+
|
|
140
|
+
- **Exposure**: a caller can read data it should not. State who can read
|
|
141
|
+
what, how they would get it, and the impact.
|
|
142
|
+
- **Unneeded surface**: a field with no consumer but no sensitive content.
|
|
143
|
+
Worth removing to shrink the contract; not a vulnerability.
|
|
144
|
+
- **Unresolved**: you could not confirm the consumer or the access rule.
|
|
145
|
+
Name what you checked and what would settle it.
|
|
146
|
+
|
|
147
|
+
Give each finding a confidence level and the evidence behind it. Do not call
|
|
148
|
+
every extra field a security issue; that buries the findings that are. For
|
|
149
|
+
a broader change review, hand the rest to `hr-code-review`.
|
|
150
|
+
|
|
151
|
+
## Sources
|
|
152
|
+
|
|
153
|
+
- OWASP API Security Top 10 (2023), API3: [Broken Object Property Level
|
|
154
|
+
Authorization](https://owasp.org/API-Security/editions/2023/en/0xa3-broken-object-property-level-authorization/),
|
|
155
|
+
which merges excessive data exposure and mass assignment.
|
|
156
|
+
- OWASP Web Security Testing Guide, on testing for excessive data exposure.
|
|
157
|
+
- GDPR ([Regulation 2016/679](https://eur-lex.europa.eu/eli/reg/2016/679/oj))
|
|
158
|
+
Article 5(1)(c) on data minimisation and Article 25 on data protection by
|
|
159
|
+
design and by default.
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: hr-audit-choices
|
|
3
|
+
description: "Audit the decisions an implementing agent made on the user's behalf, not its diff: trace the work into a choices ledger with a verdict per choice, changing no code. Use when the user asks which choices or assumptions went into implemented work, before landing delegated or AI-implemented work the user wants to review by decision, or when a fix works but may be shaped to one failing case."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Audit choices
|
|
7
|
+
|
|
8
|
+
Where a task is underspecified, the implementer decides for the user, and the
|
|
9
|
+
diff does not flag it. Working code can still rest on a data shape, storage
|
|
10
|
+
location, dependency, API contract or tradeoff the user never chose, and later
|
|
11
|
+
work inherits each one. This audit lists those decisions so the user reviews
|
|
12
|
+
choices instead of lines. It changes no code and never blocks a run; acting on
|
|
13
|
+
the verdicts belongs to the caller. For whether the diff meets its spec and the
|
|
14
|
+
repository's standards, use `hr-code-review`.
|
|
15
|
+
|
|
16
|
+
## Steps
|
|
17
|
+
|
|
18
|
+
1. **Fix the baseline.** Collect what the user actually specified: the prompt,
|
|
19
|
+
later user replies, the plan file and any spec. Choices those sources
|
|
20
|
+
explicitly left to the implementer are discretion, not audit items. Done
|
|
21
|
+
when each requirement cites its source.
|
|
22
|
+
2. **Trace every choice.** Walk the session's steps, subagent reports and
|
|
23
|
+
transcripts, diffs and commits. Ask a live implementer which choices it is
|
|
24
|
+
least sure of, but treat its answer as a floor, since implementers
|
|
25
|
+
under-report. Sweep each category: data shapes and formats; storage, paths
|
|
26
|
+
and naming schemes; API contracts and error behavior; added dependencies;
|
|
27
|
+
concurrency, performance and resource tradeoffs; scope interpretation;
|
|
28
|
+
patterns later code will copy. Done when every category has been checked
|
|
29
|
+
against the diff and every choice found is either listed or cited to the
|
|
30
|
+
baseline.
|
|
31
|
+
3. **Judge each choice.** Give one verdict:
|
|
32
|
+
- **sound**: general and defensible.
|
|
33
|
+
- **unsound**: state the corrected decision, the property that must hold in
|
|
34
|
+
general, rather than a patch on top. A fix shaped to the one failing case
|
|
35
|
+
is unsound even when the tests pass.
|
|
36
|
+
- **needs-user**: only for taste, product direction or external cost.
|
|
37
|
+
Record a reversible provisional call so an unattended run can continue.
|
|
38
|
+
4. **Write the ledger** below. When the task has a plan file, promote
|
|
39
|
+
load-bearing sound choices into it as givens. Done when every choice from step 2 has an entry;
|
|
40
|
+
trivial discretion (internal names, cosmetic calls) may collapse to a
|
|
41
|
+
one-line count.
|
|
42
|
+
5. **Report.** Group by verdict in the order needs-user, unsound, sound. Within
|
|
43
|
+
each group, put first the choices the user is least likely to have made the
|
|
44
|
+
same way. When the ledger is long, open with the two or three such choices
|
|
45
|
+
overall. Each needs-user entry becomes a close-out question, shaped as the house
|
|
46
|
+
rules' § Offers describes, with its provisional call as the recommended
|
|
47
|
+
option. List sound entries too: they
|
|
48
|
+
are the architecture the user now owns.
|
|
49
|
+
|
|
50
|
+
## Ledger
|
|
51
|
+
|
|
52
|
+
Write `choices.md` beside the task's plan file, or in the task's notes
|
|
53
|
+
directory (§ Durable notes) when there is no plan file. Each entry carries:
|
|
54
|
+
|
|
55
|
+
- **Choice**: a one-line headline, then a scenario that stands alone without
|
|
56
|
+
the diff: the triggering event, what the work does now, and what the
|
|
57
|
+
alternative would do. Define any term of art where it first appears.
|
|
58
|
+
- **Gap**: what the baseline left unspecified.
|
|
59
|
+
- **Reach**: what later work this constrains or enables.
|
|
60
|
+
- **Verdict**, with a one-line reason, plus the corrected decision or the
|
|
61
|
+
provisional call and how to reverse it.
|
|
62
|
+
- **Where**: the commit or pass that introduced it.
|
|
63
|
+
|
|
64
|
+
A `sound` choice already in the ledger or promoted into the plan is settled;
|
|
65
|
+
later passes do not list or decide it again. `unsound` and `needs-user` entries
|
|
66
|
+
stay open until they are fixed or the user decides them. When entries cluster in one area, the
|
|
67
|
+
plan was vague there, so say so in the report.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Source notices
|
|
2
|
+
|
|
3
|
+
This notice accompanies the hr-benchmarking bundle when copied on its own.
|
|
4
|
+
|
|
5
|
+
## cursor/plugins (pstack)
|
|
6
|
+
|
|
7
|
+
Source: https://github.com/cursor/plugins (compared at revision `e43c7ee`)
|
|
8
|
+
|
|
9
|
+
- `skills/hr-benchmarking/SKILL.md`: the validity checks are adapted from
|
|
10
|
+
`pstack/skills/benchmark-checklist/`.
|
|
11
|
+
|
|
12
|
+
Copyright (c) 2026 Lauren Tan
|
|
13
|
+
|
|
14
|
+
## house-rules and MIT permission notice
|
|
15
|
+
|
|
16
|
+
MIT License
|
|
17
|
+
|
|
18
|
+
Copyright (c) 2026 Aaron Queen
|
|
19
|
+
|
|
20
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
21
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
22
|
+
in the Software without restriction, including without limitation the rights
|
|
23
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
24
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
25
|
+
furnished to do so, subject to the following conditions:
|
|
26
|
+
|
|
27
|
+
The above copyright notice and this permission notice shall be included in all
|
|
28
|
+
copies or substantial portions of the Software.
|
|
29
|
+
|
|
30
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
31
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
32
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
33
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
34
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
35
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
36
|
+
SOFTWARE.
|
|
@@ -0,0 +1,316 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: hr-benchmarking
|
|
3
|
+
description: Benchmark equivalent work with repeated timing, CPU and memory measurements under stated inputs and controls, and check whether saved results support a performance claim. Use when asked to benchmark, to measure which option is faster or uses less memory, or to time agent or model runs. Public model leaderboard research is outside this measurement workflow.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Benchmarking
|
|
7
|
+
|
|
8
|
+
Follow the repository and host's commands, resource limits, authorization and
|
|
9
|
+
landing rules. Reuse an existing workload and runner before building a harness.
|
|
10
|
+
Read [the tool reference](references/tools.md) when selecting a benchmark,
|
|
11
|
+
profiler or correctness check, or interpreting its measurements.
|
|
12
|
+
|
|
13
|
+
## Define the comparison
|
|
14
|
+
|
|
15
|
+
State the question, baseline, candidate, representative input and timed region.
|
|
16
|
+
Exercise ordinary supported inputs and cases that disable an optimization.
|
|
17
|
+
State any restrictions behind a performance claim.
|
|
18
|
+
|
|
19
|
+
For a requested candidate set, record its inventory and distinguish
|
|
20
|
+
configurations or capacity modes. Keep untested, invalid and unsupported
|
|
21
|
+
candidates visible with reasons. A subset comparison establishes results only
|
|
22
|
+
for its tested scope, not a winner across the full inventory.
|
|
23
|
+
|
|
24
|
+
Before running, list the selected engines, workloads, metrics, controls and
|
|
25
|
+
repetitions. Count the planned runs explicitly, including controls and stages;
|
|
26
|
+
state which decision each retained dimension helps make. Choose combinations
|
|
27
|
+
for that decision rather than automatically running every combination. Use a
|
|
28
|
+
broader matrix when its required coverage is stated.
|
|
29
|
+
Name the measurement unit: request, operation, batch or whole command. State
|
|
30
|
+
whether setup, startup, compilation, input loading, retries and cleanup count.
|
|
31
|
+
|
|
32
|
+
Record exact runtime, compiler, dependency and runner versions, build flags,
|
|
33
|
+
input identity and source revisions. Freeze copied quality and control artifacts;
|
|
34
|
+
record their hashes and the executed artifact's identity. Agreement across
|
|
35
|
+
runtimes alone does not verify the intended artifact. Keep these fixed across
|
|
36
|
+
repeated arms; reject mixed versions when resuming a checkpoint.
|
|
37
|
+
|
|
38
|
+
Compare runtime knobs within the same build. Use equivalent outputs, release
|
|
39
|
+
settings and comparable tuning. An untuned arm compares configurations rather
|
|
40
|
+
than implementations.
|
|
41
|
+
|
|
42
|
+
Keep warm and cold conditions explicit. Reset state or use fresh processes
|
|
43
|
+
where the question requires independence. Record warmup and cache conditions.
|
|
44
|
+
A reused service needs equivalent state across arms, not an arbitrary restart.
|
|
45
|
+
|
|
46
|
+
## Confirm the work
|
|
47
|
+
|
|
48
|
+
Validate outputs, consumed results and error counts outside the timed region
|
|
49
|
+
where practical. Confirm requests arrived and asynchronous work was awaited.
|
|
50
|
+
Include failures, timeouts and unresolved attempts in both arms' denominators.
|
|
51
|
+
A faster wrong result or skipped operation is not a performance improvement.
|
|
52
|
+
|
|
53
|
+
Identify the limiting resource from CPU versus wall time, resource counters or
|
|
54
|
+
a separate profile. Check that throughput is possible given the machine and
|
|
55
|
+
input. When the generator limits the result, report that limit.
|
|
56
|
+
|
|
57
|
+
## Measure without changing the experiment
|
|
58
|
+
|
|
59
|
+
Coordinate resource use under the host's rules. During isolated local performance
|
|
60
|
+
measurements, the measuring session only monitors. Other sessions may do light
|
|
61
|
+
work within the run's resource limits. Keep measured files, toolchains and
|
|
62
|
+
services unchanged.
|
|
63
|
+
|
|
64
|
+
Prefer removing competing work to raising priority. Record effective CPU and
|
|
65
|
+
I/O scheduling policies and memory protections; keep them equal across arms.
|
|
66
|
+
Use normal scheduling and all available CPUs for ordinary measurements. Omit
|
|
67
|
+
automatic `nice` and `taskset` wrappers; verify inherited priority and affinity
|
|
68
|
+
rather than assuming their omission resets either. Use restricted affinity or
|
|
69
|
+
changed priority only for representative deployment settings, an explicit resource
|
|
70
|
+
agreement or a stated sensitivity question. Record the reason and effective values.
|
|
71
|
+
Preserve frozen protocols and report constrained results within their tested scope.
|
|
72
|
+
|
|
73
|
+
Treat a priority or affinity change as a separate sensitivity pilot under the host's rules, with preset variation
|
|
74
|
+
criteria and retained raw results. Verify that the controls apply and that
|
|
75
|
+
dependencies and monitoring still progress before adopting them for the
|
|
76
|
+
declared experiment. Priority changes never replace admission checks, resource
|
|
77
|
+
caps or pressure rejection. Read [scheduling controls](references/tools.md#scheduling-controls)
|
|
78
|
+
before choosing them. When brief bursts from other sessions keep failing host-wide
|
|
79
|
+
admission on a shared host, read the [fenced-core arm](references/tools.md#fenced-core-arm)
|
|
80
|
+
before proposing one; it needs the user's authorization. For questions about negative, zero or positive nice values,
|
|
81
|
+
read the [recorded priority findings](references/priority-findings-20261006.md)
|
|
82
|
+
before proposing another experiment.
|
|
83
|
+
|
|
84
|
+
Classify the timed region against the host's budget for concurrent light work.
|
|
85
|
+
A local phase is heavy when measured or expected CPU, memory or I/O exceeds
|
|
86
|
+
that budget. Inspect the command, its hooks, wrappers and children before calling
|
|
87
|
+
demand
|
|
88
|
+
unknown. Inspection can qualify bounded ordinary work within that budget;
|
|
89
|
+
missing resource samples alone does not require a trial. When inspection leaves
|
|
90
|
+
potentially incompatible demand unresolved, contact the active reservation owner
|
|
91
|
+
or host coordinator. If neither is identified, ask the operator. Launch that
|
|
92
|
+
workload only after recording agreement on a bounded trial with CPU, memory and
|
|
93
|
+
I/O limits, or an exclusive run window. Resolve input and authorization conflicts
|
|
94
|
+
separately; resource admission does not override them.
|
|
95
|
+
|
|
96
|
+
Apply worker limits to requested processing concurrency. Report sampled OS-thread
|
|
97
|
+
counts separately; runtime background threads are not processing workers. A
|
|
98
|
+
thread count alone does not establish CPU contention. Use CPU, memory, I/O and
|
|
99
|
+
elapsed-time controls for the admitted resource budget. Keep thread count
|
|
100
|
+
diagnostic unless a host limit or experiment requirement separately justifies
|
|
101
|
+
using it to stop or reject a run.
|
|
102
|
+
|
|
103
|
+
When an existing admission binds thread count, obtain revised admission before
|
|
104
|
+
relaxing that condition. Preserve the other controls, held inputs and original
|
|
105
|
+
run verdicts. Record the revised observer and limits before execution.
|
|
106
|
+
|
|
107
|
+
For an unknown competing job, record a preset time or query limit, then inspect
|
|
108
|
+
its process identity, actual demand and access to measured inputs. Stop at that
|
|
109
|
+
limit. A cap, service name or active state alone does not establish incompatible
|
|
110
|
+
work. Keep unresolved overlap or observer gaps unqualified; input mutation remains
|
|
111
|
+
incompatible even at low demand.
|
|
112
|
+
|
|
113
|
+
A retained benchmark lock does not pause other sessions or reserve every phase
|
|
114
|
+
of their tasks. Qualified non-heavy checks, edits and coordination proceed without
|
|
115
|
+
acquiring that lock while staying within the concurrent-work budget and preserving
|
|
116
|
+
measured inputs. Classify actual commands and children, including hooks; reuse
|
|
117
|
+
qualification only for comparable inputs, runtimes and worker counts. When
|
|
118
|
+
inspection leaves demand unresolved, identify the missing evidence and arrange
|
|
119
|
+
the bounded qualification above. Defer only the incompatible phase, naming its
|
|
120
|
+
resource or input conflict;
|
|
121
|
+
do not use lock ownership alone to block qualified non-heavy work. Keep repository
|
|
122
|
+
integration and installation write locks separate from benchmark admission.
|
|
123
|
+
|
|
124
|
+
Reserve the host only for heavy local phases or deliberately isolated local
|
|
125
|
+
performance measurements. Remote model
|
|
126
|
+
inference, light CLI/API work and remote waits need no exclusive slot; a
|
|
127
|
+
memory cap or elapsed-time record alone does not establish heavy work. For
|
|
128
|
+
remote agent/model evaluations, record the timing regime and concurrent local
|
|
129
|
+
work. Apply the local admission and pressure rules below when measuring local
|
|
130
|
+
performance, rather than treating every model call as a CPU benchmark.
|
|
131
|
+
|
|
132
|
+
Before collecting results, declare required observers, their measurement interval
|
|
133
|
+
and coverage criteria. Require successful execution and recorded coverage that
|
|
134
|
+
meets those criteria. Preserve observer exit status, errors and missing samples;
|
|
135
|
+
reject evidence with unexplained observer failures or insufficient coverage.
|
|
136
|
+
Require empty stderr only when the observer's declared contract requires it;
|
|
137
|
+
documented harmless diagnostics alone do not invalidate observation.
|
|
138
|
+
|
|
139
|
+
Declare observer CPU, resident-memory and I/O budgets separately from foreign-load
|
|
140
|
+
and pressure gates. Include every collector and reader, with process birth
|
|
141
|
+
identities and aligned intervals. Record CPU time, average core-equivalent use,
|
|
142
|
+
workload-relative CPU, resident peaks, logical log bytes and attributable storage
|
|
143
|
+
I/O. Separate setup from collection; disclose missing platform counters and
|
|
144
|
+
shared provider or writeback costs. A log-byte count is not physical disk I/O.
|
|
145
|
+
Retain over-budget runs with an over-budget status and exclude them from accepted
|
|
146
|
+
performance evidence. Declare any stop rule and replacement limit before
|
|
147
|
+
execution. When automation applies a stop rule, compute it with the same code as
|
|
148
|
+
the acceptance analysis. Record budget failures separately from foreign-load and pressure
|
|
149
|
+
failures.
|
|
150
|
+
|
|
151
|
+
Choose budgets for the host, workload duration and decision precision; there is
|
|
152
|
+
no universal workload-relative CPU percentage. A budget permits collection but
|
|
153
|
+
does not prove unchanged latency. Before claiming negligible observer impact or
|
|
154
|
+
a small decisive gain, test sensitivity to observation separately with balanced
|
|
155
|
+
controls and a preset precision criterion. Retain inconclusive controls; do not
|
|
156
|
+
subtract observer CPU from workload wall time. Assign and record each run's
|
|
157
|
+
protocol version before execution. A changed observer budget creates a new
|
|
158
|
+
protocol version. Keep historical rejection verdicts with their original version.
|
|
159
|
+
|
|
160
|
+
Before each arm, inspect CPU outside the measuring scope, memory headroom and
|
|
161
|
+
pressure, swap and competing CPU, memory or I/O jobs. Use the host's admission
|
|
162
|
+
limits; when it has none, declare the metrics and pass/fail thresholds before
|
|
163
|
+
collecting performance results. Start only when every admission condition passes.
|
|
164
|
+
During measurements, attribute CPU, load and resident memory to the recorded
|
|
165
|
+
benchmark process tree or resource scope before calling them competing work.
|
|
166
|
+
Expected benchmark utilization is not contention; total host CPU and load
|
|
167
|
+
average include it. Collect boundary counters adjacent to the measured work,
|
|
168
|
+
with unrelated probes outside that interval. Record sample times and observer
|
|
169
|
+
overhead. A counter interval extending past completion cannot by itself
|
|
170
|
+
establish overlap. Keep post-run observations diagnostic and use a fresh
|
|
171
|
+
settled interval before the next arm.
|
|
172
|
+
|
|
173
|
+
Keep uncalibrated activity counters diagnostic. Before using an activity cutoff
|
|
174
|
+
to reject measurements, establish its relevance through separate sensitivity
|
|
175
|
+
evidence with comparison conditions and an interference criterion set before
|
|
176
|
+
analysis. Retain raw observations and inconclusive outcomes. Preserve declared
|
|
177
|
+
pressure and observer checks; distinguish a conservative limit from evidence
|
|
178
|
+
that interference occurred.
|
|
179
|
+
|
|
180
|
+
Reject a run when inputs change, competing work crosses a declared contention
|
|
181
|
+
limit, or a declared pressure condition fails. Before collecting performance
|
|
182
|
+
results, define pressure rejection using stall duration, swap activity and
|
|
183
|
+
memory-limit/OOM events; name any additional signals and their thresholds.
|
|
184
|
+
Set a stall-duration budget from the experiment's timing precision. Record
|
|
185
|
+
smaller stalls without rejecting them. Stalls exceeding the budget, swapping
|
|
186
|
+
or memory-limit/OOM events invalidate measurements even when the benchmark
|
|
187
|
+
causes them. Resource rejection invalidates the measurement; it alone does not
|
|
188
|
+
establish an implementation defect. Investigate uncertain overlap and repeat
|
|
189
|
+
affected runs within authorized retry budgets before a decisive comparison.
|
|
190
|
+
A light session's presence alone is not rejection evidence.
|
|
191
|
+
|
|
192
|
+
Declare per-case retry budgets before attempts; a budget grants no retry
|
|
193
|
+
authority. Retain rejected attempts outside accepted samples. A later qualifying
|
|
194
|
+
attempt neither erases a rejection nor relaxes its gates. After rejecting a case,
|
|
195
|
+
continue other independently selected cases only when their admission passes
|
|
196
|
+
and the failure cannot invalidate shared controls. Otherwise hold affected cases.
|
|
197
|
+
|
|
198
|
+
Hold reservations during heavy local phases or declared isolated local performance
|
|
199
|
+
measurements. Release them during remote waits, light work and reviews.
|
|
200
|
+
Declare finite phase boundaries or stopping conditions, release at those
|
|
201
|
+
boundaries and seek fresh admission for the next phase. While waiting, keep the
|
|
202
|
+
blocked phase in the ledger and advance already-authorized independent light
|
|
203
|
+
work within the active measurement's allowance. Preserve measured inputs;
|
|
204
|
+
backlog presence does not authorize new implementation.
|
|
205
|
+
|
|
206
|
+
Screen correctness and quality first, then use one run per selected arm to
|
|
207
|
+
explore timing and resource behavior. Preserve screened-out results and their
|
|
208
|
+
reasons. Label screening exploratory; one run does not support a performance
|
|
209
|
+
claim. A correctness-only check can use one candidate run; report its coverage.
|
|
210
|
+
|
|
211
|
+
Before exploratory runs, define the decision-relevant signal for selecting
|
|
212
|
+
comparisons for confirmation. Choose comparisons that meet that signal, plus
|
|
213
|
+
decision-critical comparisons, for repeated confirmation; keep necessary controls.
|
|
214
|
+
Set the confirmation plan before its runs and update the explicit run count
|
|
215
|
+
when the scope changes. Repetition counts depend on the
|
|
216
|
+
decision and observed variation; there is no universal minimum.
|
|
217
|
+
|
|
218
|
+
For a performance claim, add alternating runs to assess run-to-run variation.
|
|
219
|
+
Before the first run used in that claim, set a repetition count or stopping
|
|
220
|
+
rule and a reproducible variation criterion (statistic and threshold).
|
|
221
|
+
|
|
222
|
+
For short or tail metrics, such as the longest stall in a run, give the
|
|
223
|
+
threshold an absolute floor or use a percentile or count metric; a percentage
|
|
224
|
+
of a single small maximum can fail on the workload's own variation.
|
|
225
|
+
Retain every result. Keep pilots that informed the plan separate from the
|
|
226
|
+
claim's sample. Capture raw results, exit status, elapsed time, CPU and memory
|
|
227
|
+
observations. Report admission and monitoring overhead separately from the
|
|
228
|
+
measured workload; batch compatible probes when they dominate elapsed time.
|
|
229
|
+
Report the selected repetition count or stopping rule, variation statistic and
|
|
230
|
+
threshold, sample count, median and range. Apply the preset
|
|
231
|
+
criterion and treat a gap within run-to-run variation as inconclusive. Profile
|
|
232
|
+
separately from the runs used to claim a speed difference.
|
|
233
|
+
|
|
234
|
+
When using saved evidence for a performance claim, bind archived inputs, source,
|
|
235
|
+
cohort and results to their recorded identities and artifact hashes. Validate
|
|
236
|
+
historical reports against that frozen archive; later changes in the live
|
|
237
|
+
checkout alone do not invalidate historical evidence. Verify run identities and
|
|
238
|
+
stages against the declared plan.
|
|
239
|
+
|
|
240
|
+
Recompute qualification from recorded raw counters and preset limits; an accepted
|
|
241
|
+
summary or changed label is insufficient. Reject missing, duplicate or
|
|
242
|
+
misclassified required observations. When an evidence reader exists, put these
|
|
243
|
+
checks and focused corruption regressions there. After a rebase or merge
|
|
244
|
+
touches the reader or a check it imports, run the integrated reader over the
|
|
245
|
+
saved archive before landing, and compare its accepted and excluded records
|
|
246
|
+
with the frozen reader's. A shared check changed on the base branch can reject
|
|
247
|
+
records that the frozen reader accepted.
|
|
248
|
+
|
|
249
|
+
Record memory by what it measures. Whole-process peak RSS includes startup,
|
|
250
|
+
inputs and dependencies. Heap snapshots describe retained objects; allocation
|
|
251
|
+
profiles describe allocation activity. Forced-GC probes are separate from
|
|
252
|
+
ordinary latency measurements. Match retention semantics and payloads when
|
|
253
|
+
comparing caches; equal requested capacities need not mean equal memory use.
|
|
254
|
+
For cache-retention probes, keep the cache owner reachable through the final
|
|
255
|
+
observation; otherwise collection of the owner can be mistaken for release by
|
|
256
|
+
the cache operation.
|
|
257
|
+
|
|
258
|
+
For WeakRef-based retention probes, use bounded allocation batches and calibrated
|
|
259
|
+
collection boundaries after creating or dereferencing targets. Avoid observer
|
|
260
|
+
arrays or locals that keep measured values strongly reachable; preserve the
|
|
261
|
+
intended owner lifetime. Report live payload, cache occupancy and process RSS
|
|
262
|
+
separately. Keep this collection protocol outside ordinary latency runs; yielding
|
|
263
|
+
or forced collection alone does not guarantee that a target is reclaimed.
|
|
264
|
+
|
|
265
|
+
Batch latency percentiles describe batches, not individual requests.
|
|
266
|
+
|
|
267
|
+
## Report and decide
|
|
268
|
+
|
|
269
|
+
Keep the full workload, versions, conditions, repetitions, output identity,
|
|
270
|
+
errors and measurement units in the measurement record. Include excluded costs,
|
|
271
|
+
coverage limits, collector overhead, scheduling, resource caps, admission checks
|
|
272
|
+
and rejected attempts there.
|
|
273
|
+
|
|
274
|
+
Pull requests and other change summaries show the workload, baseline and
|
|
275
|
+
candidate, measured results, relevant checks and limits that affect the claim.
|
|
276
|
+
Omit operational measurement details unless the reader needs them to interpret
|
|
277
|
+
the result or asks for them. A claimed difference with unknown limits, unequal
|
|
278
|
+
work or missing repetitions is unverified or inconclusive.
|
|
279
|
+
|
|
280
|
+
Show numeric before/after results in a table with a benefit column containing
|
|
281
|
+
percent change or a times improvement. Use the metric's direction and label
|
|
282
|
+
worsening, unchanged or inconclusive results. With a zero or missing baseline,
|
|
283
|
+
show an absolute change or state that the benefit cannot be calculated.
|
|
284
|
+
|
|
285
|
+
Use 3% as the default practical no-clear-change band for reported differences.
|
|
286
|
+
This is a reporting convention, not a statistical margin of error. Use the
|
|
287
|
+
variation statistic and threshold declared before collecting results. When that
|
|
288
|
+
threshold, expressed as a percentage of the baseline, exceeds 3%, widen the
|
|
289
|
+
reporting band to at least that percentage and name the statistic and threshold.
|
|
290
|
+
If the variation criterion cannot support a percentage band, keep its result
|
|
291
|
+
separate and label unsupported differences inconclusive. A change outside the
|
|
292
|
+
band still needs qualified evidence and must meet the declared variation criterion. Set a different
|
|
293
|
+
precision target before collecting results when the workload requires it;
|
|
294
|
+
changing presentation does not change frozen acceptance or rejection gates.
|
|
295
|
+
|
|
296
|
+
For agent, model, rule or skill comparisons, distinguish development cases
|
|
297
|
+
from held-out cases and report runs per case. Include a one-line instruction
|
|
298
|
+
control when claiming an improvement from a rule or skill. Keep helper,
|
|
299
|
+
setup and retry costs visible. Before collecting results, define a per-case
|
|
300
|
+
contribution threshold. If a case exceeds it, report results both with and
|
|
301
|
+
without that case.
|
|
302
|
+
|
|
303
|
+
For bug-review evaluations, freeze the supported input contract and label
|
|
304
|
+
procedure before calls. A passing reference suite does not certify a clean
|
|
305
|
+
control. Adjudicate additional findings against that contract. A confirmed extra
|
|
306
|
+
defect makes the affected specificity estimate unavailable; retain earlier
|
|
307
|
+
receipts and introduce a new control version. Leave disputed domains unresolved.
|
|
308
|
+
|
|
309
|
+
Report provider completion, output-format compliance and independently checked
|
|
310
|
+
artifact acceptance separately. Distinguish observed call time from verification
|
|
311
|
+
and rework time. Keep returned usage units separate from attributable account
|
|
312
|
+
charges; unknown attribution remains unknown.
|
|
313
|
+
|
|
314
|
+
Record every experiment, rejected ones included, with the observed result and
|
|
315
|
+
disposition. End with adopt, reject or inconclusive, and the evidence for that
|
|
316
|
+
decision. Any implementation or publication still follows the user's scope.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# JavaScript and TypeScript measurement
|
|
2
|
+
|
|
3
|
+
Use built-in runtime profilers and oha from [the tool reference](tools.md).
|
|
4
|
+
TypeScript applications execute through a runtime; compiler timing and
|
|
5
|
+
application timing answer different questions.
|
|
6
|
+
|
|
7
|
+
## TypeScript measurement
|
|
8
|
+
|
|
9
|
+
Use [compiler diagnostics](https://www.typescriptlang.org/tsconfig/extendedDiagnostics.html)
|
|
10
|
+
for compiler phases and repeated whole-command timing for builds. For application
|
|
11
|
+
performance, record the executed artifact, build target, module format and source
|
|
12
|
+
maps. Use the same compiled JS artifact for an engine-only comparison; report
|
|
13
|
+
startup or transformation separately when it is part of the question.
|
|
14
|
+
|
|
15
|
+
[Node type stripping](https://nodejs.org/api/typescript.html) performs no type
|
|
16
|
+
checking and ignores `tsconfig.json`. A successful direct `.ts` execution does
|
|
17
|
+
not establish TypeScript correctness.
|