@contextrail/assess 0.0.0-stage → 1.0.0-alpha.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (73) hide show
  1. package/LICENSE +202 -0
  2. package/README.md +82 -2
  3. package/assets/brand/icon-dark.png +0 -0
  4. package/assets/brand/icon-light.png +0 -0
  5. package/dist/assess.d.ts +109 -0
  6. package/dist/assess.js +1548 -0
  7. package/dist/assessment.d.ts +566 -0
  8. package/dist/assessment.js +404 -0
  9. package/dist/budget.d.ts +24 -0
  10. package/dist/budget.js +52 -0
  11. package/dist/candidate.d.ts +147 -0
  12. package/dist/candidate.js +321 -0
  13. package/dist/change-summary.d.ts +33 -0
  14. package/dist/change-summary.js +26 -0
  15. package/dist/cli.d.ts +2 -0
  16. package/dist/cli.js +406 -0
  17. package/dist/contracts.d.ts +99 -0
  18. package/dist/contracts.js +62 -0
  19. package/dist/diagnostics.d.ts +44 -0
  20. package/dist/diagnostics.js +412 -0
  21. package/dist/evidence.d.ts +73 -0
  22. package/dist/evidence.js +297 -0
  23. package/dist/feedback.d.ts +21 -0
  24. package/dist/feedback.js +140 -0
  25. package/dist/git.d.ts +40 -0
  26. package/dist/git.js +207 -0
  27. package/dist/github.d.ts +90 -0
  28. package/dist/github.js +1342 -0
  29. package/dist/index.d.ts +14 -0
  30. package/dist/index.js +14 -0
  31. package/dist/mcp.d.ts +28 -0
  32. package/dist/mcp.js +173 -0
  33. package/dist/pack.d.ts +160 -0
  34. package/dist/pack.js +377 -0
  35. package/dist/phases.d.ts +65 -0
  36. package/dist/phases.js +63 -0
  37. package/dist/planning.d.ts +64 -0
  38. package/dist/planning.js +135 -0
  39. package/dist/pr-context.d.ts +316 -0
  40. package/dist/pr-context.js +198 -0
  41. package/dist/prepare.d.ts +71 -0
  42. package/dist/prepare.js +335 -0
  43. package/dist/presentation.d.ts +309 -0
  44. package/dist/presentation.js +1026 -0
  45. package/dist/provider-diagnostics.d.ts +93 -0
  46. package/dist/provider-diagnostics.js +233 -0
  47. package/dist/providers.d.ts +45 -0
  48. package/dist/providers.js +291 -0
  49. package/dist/report-contract.d.ts +132 -0
  50. package/dist/report-contract.js +240 -0
  51. package/dist/results.d.ts +90 -0
  52. package/dist/results.js +128 -0
  53. package/dist/resume.d.ts +25 -0
  54. package/dist/resume.js +116 -0
  55. package/dist/source.d.ts +167 -0
  56. package/dist/source.js +675 -0
  57. package/dist/verification.d.ts +386 -0
  58. package/dist/verification.js +168 -0
  59. package/dist/version.d.ts +2 -0
  60. package/dist/version.js +9 -0
  61. package/docs/README.md +24 -0
  62. package/docs/action-reference.md +85 -0
  63. package/docs/architecture.md +156 -0
  64. package/docs/configuration.md +83 -0
  65. package/docs/contributing.md +46 -0
  66. package/docs/getting-started.md +92 -0
  67. package/docs/github-actions.md +141 -0
  68. package/docs/operations.md +203 -0
  69. package/docs/review-results.md +173 -0
  70. package/docs/security.md +64 -0
  71. package/docs/standards.md +62 -0
  72. package/examples/assess-pull-request.yml +109 -0
  73. package/package.json +56 -5
@@ -0,0 +1,85 @@
1
+ # Action inputs, outputs and delivery
2
+
3
+ [Documentation index](README.md) · [Assess overview](../README.md)
4
+
5
+ Look up a specific setting here. For a first integration, use [GitHub Actions setup](github-actions.md).
6
+
7
+ ## Inputs and credentials
8
+
9
+ | Input | Contract |
10
+ | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
11
+ | `package-version` | Required exact published npm SemVer. No tags, ranges, URLs, local builds or unavailable-version fallback. |
12
+ | `pr-number`, `base-branch` | Required PR number; optional expected target branch. |
13
+ | `reviewer` | Required reusable published `skill://` ID; never an execution or repository ID. |
14
+ | `mcp-url`, `mcp-org-id` | Required HTTPS endpoint and organization scope, using CLI connection vocabulary. |
15
+ | `provider`, `model` | Required `openai` or `openrouter` and explicit provider-native model. No fallback. |
16
+ | `enforcement` | `advisory` (default), `required`, or `strict`. See the [outcome table](review-results.md#interpret-judgments-and-exit-codes); execution failures always return 2. |
17
+ | `effort` | Optional CLI effort; empty delegates to provider default. |
18
+ | `max-time-ms` | Required positive workflow model deadline, at most 2147483647 ms. Does not bound total job time. |
19
+ | `feedback` | `send` (default) or `off`; CLI owns report submission and its status. |
20
+ | `repository-attribution` | `true` or `false` (default); explicitly includes repository identity in CLI retrieval/reporting. |
21
+ | `artifact-retention-days` | 1–90, default 7, within repository/organization retention policy. |
22
+ | `post-comment` | `false` by default; `true` creates/updates the bot PR summary. Requires `pull-requests: write`. |
23
+ | `pr-context` | `false` by default; collect bounded PR discussion/history for a matching CLI release supporting `--pr-context`. Sends that context to the selected model. |
24
+ | `github-token` | Defaults to `github.token`; Git/API reads, plus PR comment writes when enabled. |
25
+ | `mcp-auth-token` | Required bearer credential; map from `secrets.CONTEXTRAIL_MCP_AUTH_TOKEN`. |
26
+ | `openai-api-key`, `openrouter-api-key` | Supply the selected provider's credential from its named secret. |
27
+
28
+ Additional optional inputs: `phase-models` (JSON, default `{}`),
29
+ `log-level` (`quiet|info|debug`, default `info`) and `inline-comments` (default `false`,
30
+ requires `post-comment: 'true'`). The model deadline covers the complete model workflow,
31
+ not a separate allowance for each phase or continuation. PR title/body are prepared as intent; `pr-context`
32
+ controls the additional bounded discussions/history. Supply both existing provider secret
33
+ inputs only when explicitly configured phases use both providers.
34
+
35
+ Credentials are explicit Action inputs, never CLI arguments. Installation gets no
36
+ GitHub, MCP or provider credentials. Git/API read credentials reach only comparison
37
+ preparation, the final freshness read and the opt-in comment step; they are not written into Git configuration
38
+ or passed to model-execution commands. Trusted npm GitHub adapter commands receive only their
39
+ required GitHub credential. MCP and the explicitly configured phase providers' keys reach only the review subprocess;
40
+ ambient CLI policy/provider settings are excluded. Prefer secret inputs over caller
41
+ `env`: environment values on the calling step are inherited by every composite step.
42
+ Consumer steps can still access their job's GitHub token; composite steps are not a
43
+ security boundary against malicious trusted Action code.
44
+
45
+ ## Outputs and artifacts
46
+
47
+ | Output | Meaning |
48
+ | ------------------- | --------------------------------------------------------------------------------------------------------------- |
49
+ | `assessment-exit` | Actual CLI 0/1/2, empty if not invoked or interrupted. Action/setup/upload failure can fail the job separately. |
50
+ | `run-id` | Unique CLI `runId` from confirmed state; no substituted workflow/check/reviewer identity. |
51
+ | `feedback-status` | `received`, `failed`, `pending`, `off` or `unavailable`. Feedback failure preserves CLI exit. |
52
+ | `comparison-status` | `current_at_finish`, `superseded` or `unknown`, when a comparison was prepared. |
53
+ | `artifact-url` | Evidence artifact URL when upload succeeds. |
54
+ | `comment-status` | `created`, `updated`, `skipped_superseded`, `skipped_newer` or `failed`; empty when disabled. |
55
+
56
+ Available state, reports, phase diagnostics, CLI diagnostics, comparison/setup status and npm lock
57
+ with resolved dependency integrity are preserved after failure. Installed dependencies
58
+ and fetched source objects are not uploaded. A separate post-publication artifact retains
59
+ validated write payloads and delivery outcomes, including unknown/partial writes;
60
+ `publication-artifact-url` exposes its URL. It uses the same retention setting. Review evidence includes cited source and generated assessments;
61
+ restrict artifact access and retention accordingly. Ordinary logs and summaries use
62
+ controlled statuses, identities, counts, bounded finding/correction text and unresolved reasons, not conversations, full source citations or raw upstream errors.
63
+ The summary confirms completion only from matching persisted comparison/reviewer
64
+ state. An ingestion receipt does not endorse a review.
65
+
66
+ The final API read labels a changed comparison (identity, selected target or base/head SHAs)
67
+ **SUPERSEDED; historical result only**. Closing a PR alone does not change its comparison. A failed freshness read means unknown. No result certifies future currency;
68
+ verify both SHAs when using it. A provider call may race a subsequent PR push, and
69
+ feedback can already have been sent for the accurately recorded fixed comparison.
70
+ Cancellation or job timeout can prevent later upload/summary steps; use a finite job
71
+ timeout with headroom beyond the model-session limit (for example, use 20 minutes for
72
+ 600000 ms). Evidence preservation is attempted with `always()` after ordinary errors.
73
+
74
+ ## Comment delivery and reruns
75
+
76
+ Comment creation/update is attempted after ordinary review failures with `always()`. A
77
+ permission/API failure is visible in the Action summary and fails that step independently of
78
+ `assessment-exit`; there is no automatic comment-write retry. Comments are scoped to the configured
79
+ PR/reviewer and current comparison, with workflow/run-attempt links for traceability. Cancellation
80
+ or job timeout can still prevent summary/comment execution. The caller must serialize publications per repository/PR/reviewer for rerun deduplication.
81
+ GitHub creation has no atomic compare-and-set; concurrent runs can race after absence checks.
82
+ The npm package owns presentation and GitHub adapter behavior; the Action invokes it.
83
+
84
+ After publishing this Action update, adopt its full public commit SHA as well as the intended
85
+ exact npm version. Re-running an old workflow retains its old Action code and inputs.
@@ -0,0 +1,156 @@
1
+ # Assessment lifecycle and contracts
2
+
3
+ [Documentation index](README.md) · [Assess overview](../README.md)
4
+
5
+ For maintainers and agents changing the harness. Setup and usage are covered in [getting started](getting-started.md).
6
+
7
+ ## Lifecycle
8
+
9
+ 1. **Prepare:** resolve fixed commits, retrieve the published reviewer and governing contexts, and write an evidence pack.
10
+ 2. **Observe:** one coherent assessment reads pinned source and records independently answerable observations, possible standards concerns, assumptions and decisive unknowns for every selected context.
11
+ 3. **Validate concerns:** independently examine each nominated concern against its original governing publication and cited source. Confirm a violation only when both applicability and contradiction are established. Keep missing evidence and ambiguous guidance distinct from violations.
12
+ 4. **Report:** code derives outcomes, consolidates equivalent same-duty corrections, applies policy, persists artifacts and delivers configured feedback. An optional final pass checks the reader change summary separately.
13
+
14
+ The selected reviewer skill and original pinned contexts retain authority. Reviewer instructions may
15
+ narrow the investigation, but this is standards adherence auditing, not unrestricted bug hunting.
16
+ Each additional observation cites the original passage and pinned source; code supplies identities
17
+ and quotations from selected evidence segments. Accepted context groups are immutable and remain
18
+ available when a sibling group needs repair. Checkpoints retain observations and concern decisions.
19
+ An older judgment-based checkpoint cannot be silently interpreted as an observation-based one.
20
+
21
+ A no-concern observation is **observed**, not independently validated compliance. Structural coverage
22
+ means each selected context has observations; it does not prove every obligation was discovered.
23
+ Verification counters describe nominated concerns, separately from context coverage. Only confirmed
24
+ artifact concerns become findings. Conditional applicability remains a question. Actual execution of external operations is outside the audit; source inspection assesses the
25
+ mechanisms that implement applicable operational requirements.
26
+
27
+ The assessment allows 60 provider turns and 50 source-tool calls per response. Each focused validation
28
+ and optional summary session allows 12 provider turns. Excess reads receive
29
+ retry feedback rather than discarding accepted work. Each source tool retains its own path/range/size
30
+ bounds. The configured whole-review deadline is shared, across assessment and validation.
31
+ Global caller turn/token/time limits still apply. Validators receive a focused packet for their
32
+ concern and can read supporting code for specific missing facts; they do not receive every other
33
+ standard's assessment. The default transport, model settings, request retries and diagnostics are
34
+ shared with the production CLI and Action.
35
+
36
+ ## Choose the reviewer and comparison
37
+
38
+ `--reviewer` takes a published `skill://...` ID, not a display name. Its instructions
39
+ provide the review lens; explicit context and domain selections plus required
40
+ context prerequisites determine the expected scope. The CLI retrieves exact
41
+ publication versions and reports missing, inaccessible or unsupported prerequisites.
42
+ Changing publications affects future runs, not the versions already recorded.
43
+
44
+ The reviewer contract consists of skill identity/metadata, review instructions,
45
+ `content.reviewer.selections` (explicit context IDs and domain selections with
46
+ `includeDescendants`), and optional supporting resources. Instruction-only reviewers
47
+ use the same contract with no assessment selections. Empty scope stays instruction-only.
48
+
49
+ `--repo DIR` defaults to the current directory. `--base REF` and `--head REF` are
50
+ required and resolve once to fixed commit and tree IDs. The comparison is base
51
+ commit → head commit; it does not automatically choose a merge base. Use exact
52
+ commit SHAs for automation. Fetch both histories and their blobs before running;
53
+ partial clones can leave supporting source unavailable. Later branch movement,
54
+ working-tree edits and index changes do not alter the review. The source tools read
55
+ both snapshots and unchanged supporting files, with bounded responses and
56
+ continuations. They do not execute repository scripts, tests, hooks or model shell
57
+ commands. Binary files, submodule contents and conventional credential files have
58
+ explicit source-access limits; this is a text/source review, not runtime testing.
59
+
60
+ To check retrieval and scope without calling a model, replace `run` with
61
+ `prepare` and omit provider/model arguments. Inspect `state.json` and its gaps.
62
+ Preparation exit 0 means the scope was prepared; entries remain **not evaluated**.
63
+ Preparation exits 2 for invalid configuration or incomplete retrieval and never 1.
64
+
65
+ ## Evidence and final corrections
66
+
67
+ The assessment boundary is the proposed artifact, not general operational assurance. Each applicable
68
+ check relates a changed behavior to a governing obligation. Missing evidence makes that artifact
69
+ judgment unresolved only when it is decisive for the claim under review. Source-visible defects and
70
+ contradictions remain findings without requiring runtime reproduction. The current PR author's process, CI status and execution of external operations are not assessment subjects.
71
+ Out-of-scope operational nominations remain in the raw audit record but cannot create findings,
72
+ unresolved artifact outcomes or actionable follow-ups. Verification challenges applicability and proportionality
73
+ as well as supporting evidence, including whether a proposed follow-up actually conceals a code defect.
74
+
75
+ For configurable code, judge the source-controlled selector, validation, defaults and authority over
76
+ inputs. An unknown deployed value alone is outside the audit. Missing source for a newly
77
+ selected executable can still leave a specific artifact behavior unresolved. Prior reviewer requests
78
+ are reassessed against this boundary; they do not become additional governing requirements.
79
+
80
+ Assessment produces observations and nominates concerns against the original pinned standards.
81
+ A focused validation session checks each nominated concern: its governing obligation, applicability,
82
+ source evidence and proposed correction. A decisive unknown must nominate a concern; it cannot
83
+ bypass validation by appearing only in an unknowns list. Final check records reflect the validator
84
+ decision while original proposals remain available for audit. Up to two concerns run concurrently. No-concern observations
85
+ remain observed; they are not independently certified as compliant. Missing decisive evidence remains
86
+ unresolved, without suppressing a separately supported violation in the same standard.
87
+
88
+ The assessment receives the evidence pack and can read permitted files in either pinned snapshot.
89
+ Current CI check status and the reviewer runtime identity are omitted from assessment and concern
90
+ validation inputs; the original PR context remains in the run audit. PR intent, discussion and
91
+ history remain available as context, not additional governing authority.
92
+ Each verifier receives the governing publication, reviewer constraints, nominated observation and cited
93
+ source fragments. It can read additional supporting code. Repository text cannot waive governing requirements.
94
+ Code extracts exact citations, generates identities and derives policy outcomes. Decisions and observations
95
+ remain in sealed local artifacts; metadata feedback identifies this method as `candidate-adherence-v2`.
96
+
97
+ If assessment supplies a change summary, a separate bounded summary session checks its evidence.
98
+ An unavailable or unsupported summary is omitted without invalidating completed standards results.
99
+ The summary explains changed behavior and review focus; it creates no additional requirements.
100
+
101
+ Model citations select `{ref, startSegment, endSegment}` from numbered evidence segments,
102
+ using inclusive indices relative to that evidence record. Code extracts the exact quotation,
103
+ character offsets and source location; models do not supply quotation text. An explicit range
104
+ also identifies an occurrence when text repeats. Canonical reports retain quotations and provenance.
105
+ Within each model session, short `context_N` and `evidence_N` handles identify the assigned
106
+ contexts and available evidence. Exact lookup restores canonical identities before validation;
107
+ unknown handles remain invalid. Audit artifacts retain canonical identities. Assessment result
108
+ citations are derived from the checks' governing passages and source references, so the model
109
+ does not have to repeat the same evidence in a separate result field.
110
+ Model inputs carry citation references and source/publication locations rather than repeated report
111
+ quotations; original evidence is supplied separately and remains available through the read tools. Code supplies revision
112
+ metadata from the current context state. The assessor supplies observations and nominated concerns. A focused verifier supplies the concern decision, applicability, duty strength and any accepted correction. Code derives aggregate outcomes without inventing confidence or treating absent concerns as compliance.
113
+ Canonical validators still check evidence, authority and consistency with findings.
114
+ Code also supplies actual package/Node/model identities and fixed comparison SHAs. When GitHub
115
+ context is collected, exact-head check/status receipts distinguish observed results from unavailable
116
+ API access. Neither runtime identity nor passing check metadata proves package safety, deployment
117
+ readiness or compliance with every governed requirement. Repository authority remains the pinned
118
+ repository's declared authority, not mutable PR discussion.
119
+ Selection failures identify the precise field and safe evidence identity in correction feedback
120
+ and diagnostics. Source bodies stay out of routine diagnostics. Selection bounds, exact text,
121
+ governing authority and independent verification remain enforced without extra correction rounds.
122
+
123
+ ## Recovery and validation
124
+
125
+ The assessment can submit multiple ready context observation groups in one tool call. The advertised
126
+ schema describes the full contract; server-side validation handles each group independently so an
127
+ invalid group does not discard valid siblings. Accepted groups are immutable; identical replay is
128
+ acknowledged, while changed replay is rejected. Only remaining groups need correction. Retained
129
+ partial progress is evidence of an incomplete phase, not a verified assessment.
130
+
131
+ Each nominated concern has an independent focused validation record, with two workers sharing the
132
+ overall budget. A failure does not erase completed sibling decisions. Completion requires a decision
133
+ for every nominated concern. Resume verifies immutable source, publication and phase identities and
134
+ reuses accepted concern decisions; it never silently upgrades old judgment-based checkpoints.
135
+
136
+ Invalid submissions share a correction round; three invalid rounds without newly accepted work
137
+ stop the session. Accepted progress resets the allowance. Reads and duplicate submissions do not.
138
+ Every attempt still consumes the configured execution budget. Use one exact committed file for
139
+ scoped literal search, or omit the path for bounded whole-tree navigation.
140
+
141
+ Provider recovery allows at most two additional attempts per failed request, with 250ms then 500ms
142
+ backoff, for transport failures marked retryable by the SDK, HTTP 408/429/5xx and response
143
+ parsing/validation failures. It repeats only the failed model request with the existing conversation;
144
+ completed tools and accepted assessments are not replayed. A successful response resets the retry
145
+ allowance for the next request. Each request consumes the original
146
+ call/time/token budget, and cancellation interrupts the backoff. Authentication, credit-limit and
147
+ configuration failures stop immediately. Recovery is separate from submission correction and
148
+ unfinished-assessment continuation. `provider_retry`, `provider_recovered` and
149
+ `provider_retry_exhausted` events appear at info/debug verbosity. See [diagnostics](operations.md#health-logging-and-cost).
150
+ The observed token cutoff can overshoot by one call and is not a dollar ceiling.
151
+ Execution completion, assessment coverage, verification coverage and policy acceptance
152
+ are separate fields. Schema validity and exact citations do not prove a judgment correct.
153
+
154
+ Each phase persists input/output identities and evidence references. [Artifacts and resume](operations.md#inspect-evidence-and-resume) describes how these records support recovery.
155
+
156
+ The package exports `prepareAssessment`, `runAssessment`, `FixedSnapshotSource`, `McpPublishedReader`, provider constructors and policy helpers. Shipped type declarations describe their arguments; these APIs use the CLI's fixed-publication and source boundaries.
@@ -0,0 +1,83 @@
1
+ # CLI configuration
2
+
3
+ [Documentation index](README.md) · [Assess overview](../README.md)
4
+
5
+ Use this reference for the standalone CLI. CLI flags override their environment aliases.
6
+ For GitHub Actions, start with [the workflow guide](github-actions.md); its inputs are separate from local environment aliases.
7
+
8
+ | Environment variable | Purpose |
9
+ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
10
+ | `CONTEXTRAIL_MCP_URL` | Authorized HTTP(S) MCP endpoint, including its configured path. Use HTTPS for remote credentials. URL credentials, queries and fragments are rejected. |
11
+ | `CONTEXTRAIL_MCP_AUTH_TOKEN` | Bearer token authorized to read the reviewer and selected contexts, and to submit feedback when enabled. |
12
+ | `CONTEXTRAIL_MCP_ORG_ID` | Organization scope, sent as `x-v2-org-id`. |
13
+ | `CONTEXTRAIL_MCP_TIMEOUT_MS` | Per connection/read timeout; default 30000 ms. `--mcp-timeout-ms N` overrides it. |
14
+ | `OPENAI_API_KEY` | Credential for `--provider openai`. |
15
+ | `OPENROUTER_API_KEY` | Credential for `--provider openrouter`. |
16
+
17
+ The MCP endpoint must support protocol **2026-07-28**. The process inherits exported
18
+ variables from its shell or the workflow step's `env`. Provider/model selection is
19
+ explicit, with no fallback. `--provider` and `--model` override
20
+ `CONTEXTRAIL_ASSESS_PROVIDER` and `CONTEXTRAIL_ASSESS_MODEL`.
21
+
22
+ OpenAI uses its Responses API. OpenRouter uses the OpenAI-compatible chat adapter at its own
23
+ endpoint. Submission tools request strict schema output; local validation remains mandatory because
24
+ provider enforcement can vary. Source tools retain their optional navigation arguments. Use a model ID
25
+ available to your provider account and capable of tool calling. For OpenRouter,
26
+ export `OPENROUTER_API_KEY` securely and replace the provider/model portion of the
27
+ quickstart with:
28
+
29
+ ```sh
30
+ --provider openrouter --model YOUR_OPENROUTER_MODEL_ID
31
+ ```
32
+
33
+ `--effort none|minimal|low|medium|high|xhigh` requests provider-specific reasoning
34
+ effort only when that model supports it. A valid CLI setting does not ensure that
35
+ a provider accepts it. Source excerpts and published guidance are sent to your
36
+ selected model provider; use an account and data-handling arrangement appropriate
37
+ for that material.
38
+
39
+ ## Models by phase and logging
40
+
41
+ `--provider`, `--model` and `--effort` supply the explicit shared default. Optional
42
+ `--phase-models` JSON overrides `assess` or `verify`; an override includes provider
43
+ and model together, plus optional effort. Omitted effort uses that provider's default.
44
+ Unknown keys and unsupported configuration fail without model substitution.
45
+
46
+ ```sh
47
+ --phase-models '{"verify":{"provider":"openrouter","model":"YOUR_VERIFICATION_MODEL"}}' \
48
+ --log-level info
49
+ ```
50
+
51
+ The selected reviewer uses one coherent assessment session, followed by independent validation of each nominated concern (up to two concurrently). A separate optional session checks the change summary.
52
+ `--log-level quiet|info|debug` defaults to info; stdout retains machine JSON while safe progress
53
+ goes to stderr. All levels retain required diagnostics. Execution limits cover the invocation,
54
+ including bounded continuation of unfinished submissions. No stronger-model fallback is implicit.
55
+
56
+ ## Limits and environment aliases
57
+
58
+ `--purpose TEXT` records your stated objective or stage for the reviewer.
59
+ Every option in this section, plus provider, model, effort, enforcement, feedback and repository-attribution, has the environment
60
+ form `CONTEXTRAIL_ASSESS_<UPPERCASE_FLAG_WITH_UNDERSCORES>`; CLI values override it.
61
+ For example, `--max-turns` maps to `CONTEXTRAIL_ASSESS_MAX_TURNS`.
62
+ `CONTEXTRAIL_ASSESS_ENFORCEMENT` accepts `advisory` (default), `required`, or `strict`.
63
+ Repository-attribution environment values must be `true` or `false`. Git arguments
64
+ and reviewer ID have no environment aliases. See `--help` for the current option list.
65
+
66
+ | Flag | Default / unit |
67
+ | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
68
+ | `--max-turns N` | Unset; model calls/steps. |
69
+ | `--max-output-tokens N` | Unset; per-call output allowance, including reasoning tokens where applicable. |
70
+ | `--max-total-tokens N` | Unset; observed provider total-token cutoff between calls. Can overshoot by one call; missing usage prevents honest enforcement and stops the review. |
71
+ | `--max-time-ms N` | Unset; model-session deadline in ms, excluding preparation. |
72
+ | `--source-page-entries N` | 100 entries per list/change response. |
73
+ | `--source-search-matches N` / `--source-search-files N` | 100 each, per search page. |
74
+ | `--source-read-lines N` / `--source-read-bytes N` | 200 lines / 65536 bytes per source response; bytes >=4. |
75
+ | `--source-timeout-ms N` | 30000 ms per source operation. |
76
+
77
+ The per-call output allowance includes reasoning and structured tool arguments; a small cap can
78
+ truncate a submission even when source reads fit. Numeric limits are positive safe integers;
79
+ timers use 1..2147483647 ms. Unset execution limits still leave provider constraints in place.
80
+ The observed token limit can overshoot by one call and is not a dollar ceiling.
81
+ Workflow timeouts are separate; leave time for finalization and upload.
82
+
83
+ See [review policy](review-results.md#interpret-judgments-and-exit-codes) for the three enforcement modes, and [recovery](architecture.md#recovery-and-validation) for retry semantics.
@@ -0,0 +1,46 @@
1
+ # Contributing and releases
2
+
3
+ [Documentation index](README.md) · [Assess overview](../README.md)
4
+
5
+ ContextRail maintains Assess. Development and release selection happen upstream; the public repository is a one-way distribution mirror.
6
+
7
+ ## Where to make changes
8
+
9
+ Propose changes for upstream incorporation. Direct edits in the public mirror can block the next
10
+ export. The package, CLI, Action and documentation are maintained together.
11
+
12
+ The package root `README.md` is the canonical overview and is exported as the public README.
13
+ The shared `docs/` directory is the detailed documentation source. Keep task links and
14
+ examples valid in the exported repository and installed package. Public docs must not depend
15
+ on private repository paths, services, working notes or uncommitted plans.
16
+
17
+ ## Validate a public checkout
18
+
19
+ ```sh
20
+ pnpm install --frozen-lockfile
21
+ pnpm run typecheck
22
+ pnpm run lint
23
+ pnpm test
24
+ pnpm run build
25
+ npm pack --dry-run
26
+ ```
27
+
28
+ In the upstream monorepo, follow its `AGENTS.md` and run `pnpm -w check`.
29
+ That gate includes export checks. Package-local builds remain available for normal development.
30
+ A built tarball or passing fixture does not prove that a version was published or that a live
31
+ provider integration works.
32
+
33
+ ## Export and publication
34
+
35
+ Upstream semantic-release selects the public version. The development manifest's version is
36
+ not a registry release. Export uses an explicit file inventory; new public docs must be listed
37
+ there. The npm package's `files` list also includes docs so installed README links remain usable.
38
+ `release.json` binds the selected version and channel to the upstream commit; the public Action
39
+ commit is a separate immutable identity.
40
+
41
+ After successful publication, consumers update both the full Action SHA and exact npm version.
42
+ Verify registry availability before dispatching a review. The README badges track npm's
43
+ `latest` and `alpha` channels automatically; they do not choose either consumer pin.
44
+
45
+ See [Action qualification](github-actions.md#qualify-and-operate-a-consumer) for the live
46
+ integration check and [architecture](architecture.md) before changing the harness.
@@ -0,0 +1,92 @@
1
+ # Run your first assessment
2
+
3
+ [Documentation index](README.md) · [Assess overview](../README.md)
4
+
5
+ This guide takes a Git comparison to a local report. Use [GitHub Actions](github-actions.md) if you
6
+ want PR comments and CI integration.
7
+
8
+ ## Before you start
9
+
10
+ Have these ready:
11
+
12
+ - Node >=22.18.0 and Git >=2.45.
13
+ - A Git repository with both commits and their supporting objects available locally.
14
+ - Your organization's ContextRail MCP endpoint, organization ID and authorized token.
15
+ - A published reviewer skill ID such as `skill://engineering/your-reviewer` that selects published contexts.
16
+ - An OpenAI or OpenRouter API key and a tool-capable model available to that account.
17
+
18
+ If you do not yet have the ContextRail inputs, follow [standards setup](standards.md).
19
+
20
+ ## Install an exact published version
21
+
22
+ Inspect available releases, choose a version, then replace `EXACT_VERSION` below:
23
+
24
+ ```sh
25
+ npm view @contextrail/assess dist-tags
26
+ npm install --ignore-scripts @contextrail/assess@EXACT_VERSION
27
+ ./node_modules/.bin/contextrail-assess --help
28
+ ```
29
+
30
+ Pin the version in your dependency lockfile. The badges show npm channels; they do not select a
31
+ version for your project. A source checkout's manifest can contain a development version rather
32
+ than a registry release. Public exports include `release.json` with the selected release and source commit.
33
+
34
+ ## Check your guidance connection
35
+
36
+ Supply `CONTEXTRAIL_MCP_AUTH_TOKEN` securely through your secret manager or process environment.
37
+ The CLI does not load `.env` files. Set the nonsecret connection values and replace the remaining placeholders:
38
+
39
+ ```sh
40
+ export CONTEXTRAIL_MCP_URL='https://YOUR_CONTEXTRAIL_HOST/mcp'
41
+ export CONTEXTRAIL_MCP_ORG_ID='YOUR_ORGANIZATION_ID'
42
+
43
+ ./node_modules/.bin/contextrail-assess prepare \
44
+ --reviewer skill://engineering/your-reviewer \
45
+ --repo /path/to/repository \
46
+ --base BASE_COMMIT_SHA --head HEAD_COMMIT_SHA
47
+ ```
48
+
49
+ Preparation retrieves guidance and captures the comparison without a model call. Open the returned
50
+ run directory's `preparation.md`: confirm the commits, selected standards and any missing material.
51
+ A prepared scope is ready to assess; it is not a passing assessment.
52
+
53
+ The comparison is exactly base → head. The local CLI does not choose a merge base or include
54
+ uncommitted edits. For a PR comparison, use its merge base and head, or use the GitHub Action,
55
+ which prepares that comparison for you.
56
+
57
+ ## Run the assessment
58
+
59
+ Supply `OPENAI_API_KEY` securely and choose a provider-native model ID:
60
+
61
+ ```sh
62
+ ./node_modules/.bin/contextrail-assess run \
63
+ --reviewer skill://engineering/your-reviewer \
64
+ --repo /path/to/repository \
65
+ --base BASE_COMMIT_SHA --head HEAD_COMMIT_SHA \
66
+ --provider openai --model YOUR_OPENAI_MODEL_ID \
67
+ --enforcement advisory --max-time-ms 600000
68
+ ```
69
+
70
+ For OpenRouter, supply `OPENROUTER_API_KEY` and use
71
+ `--provider openrouter --model YOUR_OPENROUTER_MODEL_ID`. One model serves assessment and
72
+ validation by default. [Configuration](configuration.md) covers optional effort and phase overrides.
73
+
74
+ The deadline bounds model execution, not preparation or total process time. It is not a dollar cap.
75
+ Standard `info` logging shows progress on stderr; stdout remains machine-readable JSON.
76
+
77
+ ## Read the result
78
+
79
+ The final JSON identifies the run directory and decision. Open
80
+ `.contextrail/assess/<run-id>/report.md` in the reviewed repository.
81
+
82
+ - Act on supported findings and their suggested corrections.
83
+ - Inspect unresolved questions for a specific missing implementation fact or ambiguous standard.
84
+ - Check completion and execution status before relying on the result.
85
+ - Keep the bundle if you need the exact work versions, guidance versions and cited evidence later.
86
+
87
+ Advisory exit 0 permits continuation; it does not mean every standard was satisfied. See
88
+ [results and next actions](review-results.md) for the outcome and exit-code table.
89
+
90
+ By default, structured assessment metadata is reported to ContextRail after local finalization.
91
+ Use `--feedback off` to disable that submission. Either way, source and guidance needed for the
92
+ assessment go to your model provider. Read [security and data handling](security.md) before sharing artifacts.
@@ -0,0 +1,141 @@
1
+ # Assess pull requests with GitHub Actions
2
+
3
+ [Documentation index](README.md) · [Assess overview](../README.md)
4
+
5
+ Start with a maintainer-triggered advisory assessment. It produces a PR comment and evidence bundle
6
+ without making standards findings a required merge gate. Execution failures still fail the job.
7
+
8
+ ## Prepare the repository settings
9
+
10
+ Complete [standards setup](standards.md), then add these in **Settings → Secrets and variables → Actions**:
11
+
12
+ | Kind | Name | Value |
13
+ | -------- | ---------------------------- | ------------------------------------------------------------- |
14
+ | Variable | `CONTEXTRAIL_MCP_URL` | Your authorized HTTPS MCP endpoint |
15
+ | Variable | `CONTEXTRAIL_MCP_ORG_ID` | Your organization ID |
16
+ | Variable | `ASSESS_VERSION` | An exact published `@contextrail/assess` version |
17
+ | Variable | `ASSESS_MODEL` | A tool-capable OpenAI model ID available to your account |
18
+ | Secret | `CONTEXTRAIL_MCP_AUTH_TOKEN` | Token authorized for the selected skill/contexts and feedback |
19
+ | Secret | `OPENAI_API_KEY` | Your provider key |
20
+
21
+ Choose a version with `npm view @contextrail/assess dist-tags`. Use the resulting exact version,
22
+ not `alpha`, `latest` or a range. The public Action revision and installed npm version are separate
23
+ identities: review the Action source and copy its full 40-character commit SHA from
24
+ [the public repository](https://github.com/contextrail/assess). Neither identity automatically qualifies the other.
25
+
26
+ ## Add the workflow
27
+
28
+ Save this as `.github/workflows/contextrail-assess.yml`. Replace `ACTION_COMMIT_SHA` and the
29
+ reviewer skill ID before committing it. No contribution checkout is needed.
30
+
31
+ ```yaml
32
+ name: ContextRail Assess
33
+ on:
34
+ workflow_dispatch:
35
+ inputs:
36
+ pr-number:
37
+ description: Pull request number to assess
38
+ required: true
39
+ type: string
40
+ permissions: {}
41
+ concurrency:
42
+ group: contextrail-assess-${{ github.repository }}-${{ inputs.pr-number }}
43
+ cancel-in-progress: false
44
+ jobs:
45
+ assess:
46
+ runs-on: ubuntu-latest
47
+ timeout-minutes: 20
48
+ permissions:
49
+ contents: read
50
+ pull-requests: write
51
+ steps:
52
+ - name: Assess proposed changes
53
+ uses: contextrail/assess/action@ACTION_COMMIT_SHA
54
+ with:
55
+ package-version: ${{ vars.ASSESS_VERSION }}
56
+ pr-number: ${{ inputs.pr-number }}
57
+ reviewer: skill://engineering/your-reviewer
58
+ mcp-url: ${{ vars.CONTEXTRAIL_MCP_URL }}
59
+ mcp-org-id: ${{ vars.CONTEXTRAIL_MCP_ORG_ID }}
60
+ mcp-auth-token: ${{ secrets.CONTEXTRAIL_MCP_AUTH_TOKEN }}
61
+ provider: openai
62
+ model: ${{ vars.ASSESS_MODEL }}
63
+ openai-api-key: ${{ secrets.OPENAI_API_KEY }}
64
+ enforcement: advisory
65
+ max-time-ms: '600000'
66
+ post-comment: 'true'
67
+ ```
68
+
69
+ The workflow must be available on the repository's default branch for the manual Run workflow
70
+ control. Run a trusted workflow revision and enter the PR number. With the GitHub CLI:
71
+
72
+ ```sh
73
+ gh workflow run contextrail-assess.yml --repo OWNER/REPOSITORY -f pr-number=123
74
+ ```
75
+
76
+ For OpenRouter, change `provider` to `openrouter`, use `openrouter-api-key` with
77
+ `secrets.OPENROUTER_API_KEY`, and set `ASSESS_MODEL` to its provider-native model ID.
78
+ Do not include an unused provider secret.
79
+
80
+ After the first run, inspect the PR comment, Action summary and artifact bundle. Confirm the expected
81
+ commits and guidance versions. ContextRail feedback status is separate from execution: a receipt
82
+ confirms ingestion only. [Results and next actions](review-results.md) explains what to do with findings.
83
+
84
+ ## Tune only what you need
85
+
86
+ - **Inline feedback:** add `inline-comments: 'true'` to publish eligible verified findings and code suggestions.
87
+ - **More progress detail:** add `log-level: debug`; routine logs still exclude full conversations and raw provider errors.
88
+ - **PR discussion/history:** add `pr-context: 'true'`. Read the [data implications](security.md) first.
89
+ - **Merge policy:** set `enforcement: required` for supported binding violations, or `strict` to also block unresolved evidence. Configure branch protection to require the job if desired. Do not use `continue-on-error` to create advisory behavior; it would hide execution failures.
90
+ - **Retention:** set `artifact-retention-days` to a permitted value from 1–90. The default is seven days.
91
+ - **Models:** one model serves both roles by default. Add `effort` only if the model supports it; [phase overrides](configuration.md#models-by-phase-and-logging) let you select a different verifier later.
92
+
93
+ Repository variable names belong to your caller workflow. In this example `ASSESS_MODEL` is passed
94
+ explicitly to the Action; setting it alone in a local shell does not configure the CLI. The maintained
95
+ monorepo caller additionally supports `ASSESS_EFFORT`, `ASSESS_VERIFY_MODEL`, `ASSESS_VERIFY_EFFORT`
96
+ and `ASSESS_ENFORCEMENT`. Standalone CLI aliases use `CONTEXTRAIL_ASSESS_*`.
97
+
98
+ ## Triggers, permissions and trust
99
+
100
+ You decide which events and people may spend provider budget or access credentials. Manual dispatch
101
+ is a starting policy, not permission for untrusted PR authors to run credentialed code. If you add
102
+ labels, automatic PR events or schedules, define fork/bot/branch admission in the caller workflow.
103
+ The Action does not authorize a run from the PR author's association or account type.
104
+
105
+ The Action prepares a fixed PR comparison using GitHub and Git objects. PR events bind to event
106
+ SHAs; manual and scheduled invocations resolve current PR SHAs. The comparison uses the unique Git
107
+ merge base, excluding changes made only on the target branch. `comparison.json` records the analysis
108
+ base and observed target tip separately. Missing/ambiguous ancestry or stale event identities stop
109
+ preparation. Optional `base-branch` checks an expected target branch; it is not needed for the example.
110
+
111
+ Fork heads are fetched through the base repository's PR ref. The Action does not check out contribution
112
+ files or execute their scripts/hooks, and rechecks comparison identity before introducing provider
113
+ credentials. A final freshness check labels changed comparisons as superseded before publication.
114
+
115
+ A composite Action shares its job and runner. **Do not execute untrusted contribution code anywhere
116
+ in this credentialed job**, restore untrusted caches or run untrusted services alongside it. Only a
117
+ trusted Action revision and qualified npm artifact may receive credentials. Disabled npm lifecycle
118
+ scripts do not make runtime package code trusted. The Action requires Linux, Git >=2.45, access to
119
+ GitHub.com and the public npm registry; it installs Node 22.18.0. GHES/private registries are outside
120
+ this interface.
121
+
122
+ For read-only results, set `post-comment: 'false'` and reduce `pull-requests` to `read`; keep
123
+ `contents: read`. Optional `checks: read` and `statuses: read` permit capture of GitHub status metadata,
124
+ not logs. Those receipts are not the artifact assessment's subject. Use the default GitHub Actions
125
+ token for comments; it updates only the matching bot-owned comment. No labels or Slack messages are written.
126
+
127
+ ## Qualify and operate a consumer
128
+
129
+ 1. Check the installed package identity and the reviewed Action SHA.
130
+ 2. Confirm the intended published reviewer and context versions are readable.
131
+ 3. Run one small authorized PR and inspect its findings, source links, completion and evidence artifacts.
132
+ 4. Check feedback delivery separately, and correct any endpoint/contract or credential problem.
133
+ 5. Decide whether advisory behavior meets your needs before making the job a required check.
134
+
135
+ Use a fresh dispatch for current commits. Rerunning an old event can reject superseded comparisons;
136
+ removing/readding a trigger label is a new paid request. Duplicate execution requests are not deduplicated.
137
+ Disable the workflow to stop new runs. Rotate expired credentials through their issuing service and
138
+ GitHub secrets, then verify a small run without logging secret values.
139
+
140
+ See [Action reference](action-reference.md) for every input/output and failure-delivery behavior,
141
+ and [operations](operations.md) for diagnostics and recovery.