judgelint 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +33 -24
- data/README.md +118 -128
- data/lib/judgelint/cli.rb +2 -2
- data/lib/judgelint/protocol.rb +42 -8
- data/lib/judgelint/question.rb +62 -61
- data/lib/judgelint/reporter.rb +21 -7
- data/lib/judgelint/runner.rb +76 -20
- data/lib/judgelint/source.rb +5 -0
- data/lib/judgelint/verdict.rb +11 -7
- data/lib/judgelint/version.rb +1 -1
- metadata +6 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 204f26e50fb67239d805530c3616f298539f80eddc738f971065996042a6b4cc
|
|
4
|
+
data.tar.gz: 3c57e02dad382cd5f82ecc6c1936c20ee00b60549e5d9cdfed020f91109c7d05
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c50ecb91cc07c801bc9fa913e2ed9719391b6603fee87fdcb03d2a9e2df6071e3dee5c62e4b74b6e82c7bcf5de9992d9a6174b718bfb9bb3b910e9076fd21a54
|
|
7
|
+
data.tar.gz: 670f8f274d0409b8682be0a6757f74a8b9bfd3d37b2f6fc02af8399866c4e2eae842c7ebd0c4d23bbe7e3e520f61645f58fae07b6a94fc9be92155a5ec583bc0
|
data/CHANGELOG.md
CHANGED
|
@@ -1,32 +1,41 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.2 (2026-10-24)
|
|
4
|
+
|
|
5
|
+
- Each bullet is one choice between breaking the rule, following it, and the
|
|
6
|
+
rule not applying, in place of two yes/no questions. On the demo app, Jev
|
|
7
|
+
finds 13 of 14 known violations, up from 12, with half the questions.
|
|
8
|
+
- Findings name the line that breaks the rule, or the file in a directory,
|
|
9
|
+
when the model is sure of one.
|
|
10
|
+
- Both the System One and OpenAI Decisions protocols send choice questions.
|
|
11
|
+
|
|
3
12
|
## 0.1.1 (2026-10-23)
|
|
4
13
|
|
|
5
|
-
- `judgehint:` comments replace `judgelint:disable`. A hint
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
-
|
|
14
|
+
- `judgehint:` comments replace `judgelint:disable`. A hint explains why code
|
|
15
|
+
breaks a rule. judgelint asks each rule whether the hint gives a reason that
|
|
16
|
+
applies to it, and reports the finding as excused if it does.
|
|
17
|
+
- Each GitHub release has native binaries for macOS and Linux. Install them
|
|
9
18
|
with Homebrew (`brew install jpcamara/tap/judgelint`) or `install.sh`.
|
|
10
19
|
|
|
11
20
|
## 0.1.0 (2026-10-23)
|
|
12
21
|
|
|
13
|
-
- Specs
|
|
14
|
-
|
|
15
|
-
with the file and is asked
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
-
|
|
20
|
-
|
|
21
|
-
- `check`, `test`
|
|
22
|
-
`
|
|
23
|
-
|
|
24
|
-
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
22
|
+
- Specs: a README or `*.judgelint.md` with `paths:` in its frontmatter,
|
|
23
|
+
where each top-level bullet is a rule. The model reads
|
|
24
|
+
the spec with the file, and each bullet is asked both ways and averaged.
|
|
25
|
+
`violation:` and `compliant:` lines under a bullet describe it more
|
|
26
|
+
precisely. Rules that don't fit in a bullet can be written as questions in
|
|
27
|
+
`.judgelint/rules/`.
|
|
28
|
+
- Rules can apply to every 40 lines, a whole file, or a whole directory.
|
|
29
|
+
`context:` adds files the model reads but doesn't judge.
|
|
30
|
+
- Commands: `check`, `test`, `rules`, `providers`, and `init`. `--base` and
|
|
31
|
+
`--pr` check only what changed. Output as text, JSON, GitHub annotations,
|
|
32
|
+
or SARIF.
|
|
33
|
+
- Requests are planned and priced before any are sent, and `max_cost_usd`
|
|
34
|
+
stops an expensive run. Every run prints its cost, and answers are cached
|
|
35
|
+
per model.
|
|
36
|
+
- 16 providers over the System One and OpenAI Decisions APIs. List more than
|
|
37
|
+
one in `models:` to average their answers.
|
|
38
|
+
- `judgelint:disable` comments, with a required reason.
|
|
39
|
+
- No runtime dependencies. The macOS and Linux gems include a native binary
|
|
40
|
+
built with Spinel.
|
|
41
|
+
- `examples/scoop-society`, a TypeScript demo app with known violations.
|
data/README.md
CHANGED
|
@@ -1,58 +1,66 @@
|
|
|
1
1
|
# judgelint
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
judgelint checks code against conventions written in plain English. You write
|
|
4
|
+
them as bullets in a README. judgelint asks a decision model, such as
|
|
5
|
+
TypeSafe's Jev, whether each file breaks each bullet, and reports violations
|
|
6
|
+
with line numbers and an exit code like any other linter. Each answer takes a
|
|
7
|
+
few hundred milliseconds and costs a fraction of a cent.
|
|
4
8
|
|
|
5
|
-
|
|
6
|
-
One decision model, such as TypeSafe's Jev, whether each file breaks each
|
|
7
|
-
one, and reports violations with line ranges and exit codes, like a linter.
|
|
8
|
-
Answers take a few hundred milliseconds and cost a fraction of a cent.
|
|
9
|
-
|
|
10
|
-
Inspired by [praxis](https://github.com/zarpay/praxis-cli), which reads the
|
|
11
|
-
same kind of specs with chat models. judgelint trades praxis's written
|
|
12
|
-
findings for speed and cost, and reads praxis's specs as they are. Written in
|
|
13
|
-
Ruby with no dependencies, and compiled to a native binary with
|
|
14
|
-
[Spinel](https://github.com/matz/spinel), so it doesn't need Ruby to run.
|
|
15
|
-
|
|
16
|
-
## Quick start
|
|
9
|
+
## Install
|
|
17
10
|
|
|
18
11
|
```sh
|
|
19
12
|
brew install jpcamara/tap/judgelint
|
|
20
|
-
# or
|
|
21
|
-
|
|
13
|
+
# or
|
|
14
|
+
curl -fsSL https://raw.githubusercontent.com/jpcamara/judgelint/main/install.sh | sh
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
The script installs to `~/.local/bin`, or to `$JUDGELINT_INSTALL_DIR`. The
|
|
18
|
+
binaries are on the [releases page](https://github.com/jpcamara/judgelint/releases)
|
|
19
|
+
for macOS and Linux, on arm64 and x86_64. Linux needs glibc 2.35 or newer
|
|
20
|
+
(Ubuntu 22.04, Debian 12). `gem install judgelint` works too, and runs as
|
|
21
|
+
plain Ruby on Windows.
|
|
22
22
|
|
|
23
|
-
|
|
23
|
+
## Getting started
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
judgelint init # writes .judgelint.yml and an example spec
|
|
24
27
|
export TYPESAFE_API_KEY=...
|
|
25
|
-
judgelint test
|
|
26
|
-
judgelint
|
|
27
|
-
judgelint
|
|
28
|
+
judgelint test # checks the example spec against its examples
|
|
29
|
+
judgelint # checks every file a spec covers
|
|
30
|
+
judgelint --base main # checks only what changed since main
|
|
28
31
|
```
|
|
29
32
|
|
|
30
|
-
|
|
33
|
+
Here's a run on the [demo app](examples/scoop-society), with the messages shortened:
|
|
31
34
|
|
|
32
35
|
```text
|
|
33
36
|
$ judgelint src/services/apply-refund.ts src/features/flavor-of-day
|
|
34
37
|
src/services/apply-refund.ts
|
|
35
|
-
|
|
36
|
-
x error src/services/
|
|
37
|
-
x error src/services/
|
|
38
|
+
? uncertain src/services/a-service-module-exports-exactly-one 1-21 0.47 A service module exports exactly one entry point.
|
|
39
|
+
x error src/services/domain-failures-are-values-never-exceptions 13 1.00 Domain failures are values, never exceptions.
|
|
40
|
+
x error src/services/error-messages-are-written-for-the 13 1.00 Error messages are written for the API consumer.
|
|
41
|
+
? uncertain src/services/input-is-validated-before-any-work 10 0.51 Input is validated before any work happens.
|
|
42
|
+
x error src/services/every-service-has-a-doc-comment 1-21 1.00 Every service has a doc comment …
|
|
38
43
|
src/features/flavor-of-day
|
|
39
|
-
x error src/features/exactly-one-entry-point-index-ts
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
44
|
+
x error src/features/exactly-one-entry-point-index-ts old-rotation.ts 1.00 Exactly one entry point, `index.ts`, …
|
|
45
|
+
? uncertain src/features/types-live-in-a-dedicated-feature flavor.ts 0.53 Types live in a dedicated `<feature>-types.ts` file.
|
|
46
|
+
x error src/features/logic-files-are-verb-first-kebab flavor.ts 0.99 Logic files are verb-first kebab-case …
|
|
47
|
+
x error src/features/no-orphaned-files old-rotation.ts 0.79 No orphaned files.
|
|
43
48
|
|
|
44
|
-
2 files,
|
|
45
|
-
judgelint: 2 requests,
|
|
49
|
+
2 files, 6 violations, 3 uncertain
|
|
50
|
+
judgelint: 2 requests, 6,437 input tokens, $0.0003
|
|
46
51
|
```
|
|
47
52
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
53
|
+
A finding names the line that breaks the rule, or the file in a directory,
|
|
54
|
+
when the model is sure of one. Otherwise it shows the lines it was asked
|
|
55
|
+
about.
|
|
51
56
|
|
|
52
|
-
|
|
57
|
+
It exits 1 when there are violations, and 2 on a usage, config, or model
|
|
58
|
+
error. Config problems are caught before any request is sent.
|
|
53
59
|
|
|
54
|
-
|
|
55
|
-
|
|
60
|
+
## Writing specs
|
|
61
|
+
|
|
62
|
+
A spec is a README, or any file ending in `.judgelint.md`, with `paths:` in
|
|
63
|
+
its frontmatter. Each top-level bullet is a rule.
|
|
56
64
|
|
|
57
65
|
```markdown
|
|
58
66
|
---
|
|
@@ -66,16 +74,13 @@ excludes: ["src/services/legacy-import.ts"]
|
|
|
66
74
|
- **No I/O except through the injected Store.** No timers, no fetch, no console.
|
|
67
75
|
```
|
|
68
76
|
|
|
69
|
-
The model
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
are
|
|
74
|
-
and a bullet that says "should" or "prefer" is a warning, anything else an
|
|
75
|
-
error.
|
|
77
|
+
The model gets the spec and the numbered file. For each bullet, it chooses
|
|
78
|
+
whether the code breaks it, follows it, or isn't covered by it, and which line
|
|
79
|
+
breaks it most clearly. A rule's id comes from the start of its bullet, like
|
|
80
|
+
`src/services/no-i-o-except-through-the`. Bullets that say "should" or
|
|
81
|
+
"prefer" are warnings, and the rest are errors.
|
|
76
82
|
|
|
77
|
-
|
|
78
|
-
indented lines under it:
|
|
83
|
+
If the model misreads a bullet, describe breaking and following it:
|
|
79
84
|
|
|
80
85
|
```markdown
|
|
81
86
|
- **No I/O except through the injected Store.** No timers, no fetch, no console.
|
|
@@ -83,22 +88,24 @@ indented lines under it:
|
|
|
83
88
|
compliant: reads and writes only through the Store it's given
|
|
84
89
|
```
|
|
85
90
|
|
|
86
|
-
They make the model more decisive
|
|
87
|
-
|
|
91
|
+
They make the model more decisive. A loose description can also flag code
|
|
92
|
+
you meant to allow, so check them against examples.
|
|
88
93
|
|
|
89
|
-
| Frontmatter | |
|
|
94
|
+
| Frontmatter | What it does |
|
|
90
95
|
|---|---|
|
|
91
|
-
| `paths`, `excludes` | Globs for the files the spec
|
|
92
|
-
| `context` | Files the model reads
|
|
93
|
-
| `scope` | `window
|
|
94
|
-
| `severity`, `threshold` |
|
|
96
|
+
| `paths`, `excludes` | Globs for the files the spec covers, relative to the project root. |
|
|
97
|
+
| `context` | Files the model reads along with each file, like shared types. They're never judged. |
|
|
98
|
+
| `scope` | `window`, the default, asks about every 40 lines so findings have line numbers. `file` asks once per file. `directory` asks once per directory, shows the model every file in it, and names the file that breaks the rule. |
|
|
99
|
+
| `severity`, `threshold` | Set the severity or threshold for every rule in the spec. |
|
|
100
|
+
|
|
101
|
+
Each answer is the probability that the code breaks the rule. At 0.7 or above
|
|
102
|
+
it's a violation. Between 0.4 and 0.7 it's reported as uncertain, which
|
|
103
|
+
doesn't fail the run.
|
|
95
104
|
|
|
96
|
-
|
|
97
|
-
threshold (0.7) it's a violation, within 0.3 below it's uncertain (reported,
|
|
98
|
-
never failing).
|
|
105
|
+
### Rules written as questions
|
|
99
106
|
|
|
100
|
-
|
|
101
|
-
`.judgelint/rules/<id>.md
|
|
107
|
+
When a convention doesn't fit in a bullet, write it as a question in
|
|
108
|
+
`.judgelint/rules/<id>.md`:
|
|
102
109
|
|
|
103
110
|
```markdown
|
|
104
111
|
---
|
|
@@ -111,27 +118,29 @@ paths: ["src/features/*"]
|
|
|
111
118
|
Every file in a feature is reachable from index.ts.
|
|
112
119
|
```
|
|
113
120
|
|
|
114
|
-
`fail_when: no`
|
|
115
|
-
input first?"
|
|
121
|
+
Add `fail_when: no` when a "no" is the violation, as in "Does this service
|
|
122
|
+
validate its input first?"
|
|
116
123
|
|
|
117
|
-
|
|
124
|
+
## Explaining exceptions
|
|
125
|
+
|
|
126
|
+
When code breaks a rule on purpose, explain why in a `judgehint:` comment. Any
|
|
127
|
+
comment style works.
|
|
118
128
|
|
|
119
129
|
```ts
|
|
120
130
|
// judgehint: throwing is intentional. Only the nightly job calls this, after
|
|
121
131
|
// checking the review, so a bad id here is a programmer error.
|
|
122
132
|
```
|
|
123
133
|
|
|
124
|
-
For each rule,
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
134
|
+
For each rule, judgelint asks whether a hint in the same 40 lines gives a
|
|
135
|
+
reason the code can break it. If it does, the finding is reported as excused
|
|
136
|
+
and doesn't fail the run. A hint only excuses the rule it explains, and a hint
|
|
137
|
+
with no reason, like "this is intentional", doesn't excuse anything.
|
|
128
138
|
|
|
129
|
-
##
|
|
139
|
+
## Testing rules
|
|
130
140
|
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
(must be), and `judgelint test` reports what the model gets wrong:
|
|
141
|
+
Test each rule against examples. Put files that follow it in
|
|
142
|
+
`.judgelint/examples/<rule id>/pass/` and files that break it in `fail/`.
|
|
143
|
+
`judgelint test` shows which ones the model gets wrong:
|
|
135
144
|
|
|
136
145
|
```text
|
|
137
146
|
lib/no-silent-rescues 9/10
|
|
@@ -140,61 +149,52 @@ lib/no-silent-rescues 9/10
|
|
|
140
149
|
ok fail 0.94 .judgelint/examples/lib/no-silent-rescues/fail/assigns_default.rb
|
|
141
150
|
```
|
|
142
151
|
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
example's path relative to `pass/` or `fail/`, so the directory never gives
|
|
147
|
-
the answer away. For a directory rule, each example is a directory.
|
|
152
|
+
When an example fails, reword the bullet or split it into narrower rules. The
|
|
153
|
+
model doesn't see the `pass/` or `fail/` part of the path. For a directory
|
|
154
|
+
rule, each example is a directory.
|
|
148
155
|
|
|
149
|
-
##
|
|
156
|
+
## CI
|
|
150
157
|
|
|
151
158
|
```sh
|
|
152
|
-
judgelint --base main # since the merge base with main, uncommitted work
|
|
153
|
-
judgelint --pr # the pull request
|
|
154
|
-
judgelint --pr --format github # annotations on the changed lines (
|
|
159
|
+
judgelint --base main # changes since the merge base with main, plus uncommitted work
|
|
160
|
+
judgelint --pr # changes in the current pull request
|
|
161
|
+
judgelint --pr --format github # GitHub annotations on the changed lines (json and sarif work too)
|
|
155
162
|
```
|
|
156
163
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
164
|
+
These only ask about the 40-line windows that changed, so existing code
|
|
165
|
+
doesn't block a new pull request. `--pr` gets the base branch from GitHub
|
|
166
|
+
Actions, GitLab, Buildkite, Bitbucket, Azure Pipelines,
|
|
167
|
+
`JUDGELINT_BASE_BRANCH`, or `gh`. Fetch enough history for git to find the
|
|
168
|
+
merge base.
|
|
162
169
|
|
|
163
170
|
## Configuration
|
|
164
171
|
|
|
165
172
|
`.judgelint.yml` has four settings:
|
|
166
173
|
|
|
167
174
|
```yaml
|
|
168
|
-
models: [typesafe] #
|
|
175
|
+
models: [typesafe] # provider or provider:model; list two to average them
|
|
169
176
|
threshold: 0.7
|
|
170
|
-
ignore: ["vendor/**"] # files no rule
|
|
171
|
-
max_cost_usd: 1.00 # refuse a run estimated
|
|
177
|
+
ignore: ["vendor/**"] # files no rule checks, and READMEs that aren't specs
|
|
178
|
+
max_cost_usd: 1.00 # refuse a run estimated to cost more, unless --yes
|
|
172
179
|
```
|
|
173
180
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
Every run ends with its cost. Answers are cached in `.judgelint/cache.json`
|
|
178
|
-
by file, question, and model, so an unchanged file costs nothing; commit the
|
|
179
|
-
cache to share it.
|
|
181
|
+
Every run prints what it cost. Answers are cached in `.judgelint/cache.json`,
|
|
182
|
+
so a file that hasn't changed costs nothing. Commit the cache to share it.
|
|
183
|
+
`judgelint --help` lists the flags.
|
|
180
184
|
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
the same 12 of 14 violations found and none false, for $0.02 a run
|
|
184
|
-
(`models: [typesafe, cloudflare]`). Each model's answers are cached on their
|
|
185
|
-
own, and all of a file's requests go out at once. The
|
|
186
|
-
[example](examples/scoop-society) compares the models one at a time.
|
|
185
|
+
List two models to average their answers, like `models: [typesafe,
|
|
186
|
+
cloudflare]`. Each model's answers are cached separately.
|
|
187
187
|
|
|
188
188
|
### Providers
|
|
189
189
|
|
|
190
|
-
`judgelint providers` lists them.
|
|
191
|
-
`openrouter`, `celeris`, `cloudflare`, and `openai
|
|
192
|
-
their docs: `perplexity`, `vercel`, `digitalocean`, `upstage`,
|
|
193
|
-
`foundry`, `databricks`,
|
|
194
|
-
`litellm`. Each reads its key from the usual variable
|
|
195
|
-
`
|
|
190
|
+
`judgelint providers` lists them. Tested against the live APIs: `typesafe`
|
|
191
|
+
(Jev, the default), `openrouter`, `celeris`, `cloudflare`, and `openai`.
|
|
192
|
+
Set up from their docs: `perplexity`, `vercel`, `digitalocean`, `upstage`,
|
|
193
|
+
`hanzo`, `foundry`, `databricks`, `ollama`, `llamacpp`, `vllm`, and
|
|
194
|
+
`litellm`. Each one reads its key from the provider's usual variable, like
|
|
195
|
+
`TYPESAFE_API_KEY`.
|
|
196
196
|
|
|
197
|
-
|
|
197
|
+
For any other System One or Decisions endpoint, add a mapping to `models`:
|
|
198
198
|
|
|
199
199
|
```yaml
|
|
200
200
|
models:
|
|
@@ -205,36 +205,26 @@ models:
|
|
|
205
205
|
# headers: {} provider_options: {} price_per_million: 0.04 batch_size: 32
|
|
206
206
|
```
|
|
207
207
|
|
|
208
|
-
##
|
|
208
|
+
## Building from source
|
|
209
209
|
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
SPINEL=../spinel/bin/spinel rake spinel # build/judgelint
|
|
213
|
-
SPINEL=../spinel/bin/spinel rake test:spinel # the tests, against it too
|
|
214
|
-
```
|
|
215
|
-
|
|
216
|
-
Homebrew and `install.sh` (which installs to `~/.local/bin`, or
|
|
217
|
-
`$JUDGELINT_INSTALL_DIR`, and takes `JUDGELINT_VERSION=v0.1.0`) install the
|
|
218
|
-
binary from the [releases](https://github.com/jpcamara/judgelint/releases),
|
|
219
|
-
for macOS and Linux on arm64 and x86_64; the Linux builds need glibc 2.35 or
|
|
220
|
-
newer. On those platforms `gem install` picks a platform gem with the same
|
|
221
|
-
binary inside and links the `judgelint` command straight to it; elsewhere,
|
|
222
|
-
Windows included, the gem runs as plain Ruby. The binary starts in
|
|
223
|
-
4 ms against CRuby's 110, and in a 300-file benchmark it was 1.4x faster than
|
|
224
|
-
CRuby on one thread and 2x on eight. Under Bundler, `bundle exec` spends about
|
|
225
|
-
100 ms loading Bundler first; to skip it, link the binary
|
|
226
|
-
(`ln -s "$(bundle info judgelint --path)/libexec/judgelint" bin/judgelint`). The code stays inside what Spinel
|
|
227
|
-
compiles: plain classes instead of `Data.define`, a hand-written option
|
|
228
|
-
parser, no gems.
|
|
229
|
-
|
|
230
|
-
## Development
|
|
210
|
+
judgelint is written in Ruby and compiled with
|
|
211
|
+
[Spinel](https://github.com/matz/spinel).
|
|
231
212
|
|
|
232
213
|
```sh
|
|
233
214
|
bundle install
|
|
234
215
|
bundle exec rake test
|
|
235
|
-
|
|
216
|
+
git clone https://github.com/matz/spinel ../spinel && (cd ../spinel && make deps && make)
|
|
217
|
+
SPINEL=../spinel/bin/spinel bundle exec rake spinel # builds build/judgelint
|
|
218
|
+
SPINEL=../spinel/bin/spinel bundle exec rake test:spinel # runs the tests against the binary
|
|
219
|
+
bundle exec rake example # asks Jev about the demo app again
|
|
236
220
|
```
|
|
237
221
|
|
|
222
|
+
## Credits
|
|
223
|
+
|
|
224
|
+
judgelint was inspired by [praxis](https://github.com/zarpay/praxis-cli), and
|
|
225
|
+
it reads praxis specs, including `cohort: by_file` and `by_directory`. The
|
|
226
|
+
demo app in [examples/scoop-society](examples/scoop-society) comes from praxis.
|
|
227
|
+
|
|
238
228
|
## License
|
|
239
229
|
|
|
240
|
-
MIT
|
|
230
|
+
MIT. The demo app keeps praxis's MIT license.
|
data/lib/judgelint/cli.rb
CHANGED
|
@@ -10,8 +10,8 @@ module Judgelint
|
|
|
10
10
|
HELP = <<~TEXT
|
|
11
11
|
Usage: judgelint [command] [options] [paths...]
|
|
12
12
|
|
|
13
|
-
Checks
|
|
14
|
-
|
|
13
|
+
Checks code against the conventions in your specs. For each file and each
|
|
14
|
+
rule, it asks a decision model (Jev by default) whether the code breaks it.
|
|
15
15
|
|
|
16
16
|
Commands:
|
|
17
17
|
check [paths...] Check the files the specs govern, or these (default).
|
data/lib/judgelint/protocol.rb
CHANGED
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module Judgelint
|
|
4
|
-
# The two request shapes decision models are served with.
|
|
5
|
-
# yes/no
|
|
6
|
-
#
|
|
4
|
+
# The two request shapes decision models are served with. Each renders a
|
|
5
|
+
# Question as a yes/no question or a choice, and reads back the probability
|
|
6
|
+
# of yes, or a choice's probabilities as {option => probability}. A nil
|
|
7
|
+
# answer is a refusal.
|
|
7
8
|
module Protocol
|
|
8
|
-
# TypeSafe's System One: questions keyed by name, yes/no is "noul"
|
|
9
|
+
# TypeSafe's System One: questions keyed by name, yes/no is "noul", and a
|
|
10
|
+
# choice's options are its criteria.
|
|
9
11
|
module SystemOne
|
|
10
12
|
def self.payload(model, state, questions)
|
|
11
13
|
rendered = {}
|
|
@@ -16,6 +18,8 @@ module Judgelint
|
|
|
16
18
|
end
|
|
17
19
|
|
|
18
20
|
def self.render(question)
|
|
21
|
+
return { "type" => "choice", "instructions" => question.instructions, "criteria" => question.options } if question.choice?
|
|
22
|
+
|
|
19
23
|
rendered = { "type" => "noul", "instructions" => question.instructions }
|
|
20
24
|
criteria = {}
|
|
21
25
|
criteria["true"] = question.yes_means if question.yes_means
|
|
@@ -30,14 +34,18 @@ module Judgelint
|
|
|
30
34
|
|
|
31
35
|
probabilities = {}
|
|
32
36
|
questions.each do |question|
|
|
33
|
-
|
|
37
|
+
answer = answers[question.key]
|
|
38
|
+
probabilities[question.key] =
|
|
39
|
+
question.choice? ? Protocol.distribution(answer, answer.is_a?(Hash) ? answer["probabilities"] : nil, question)
|
|
40
|
+
: Protocol.probability(answer, "noul", question)
|
|
34
41
|
end
|
|
35
42
|
probabilities
|
|
36
43
|
end
|
|
37
44
|
end
|
|
38
45
|
|
|
39
46
|
# OpenAI's Decisions API: questions are a list with names, yes/no is
|
|
40
|
-
# "predicate" with no criteria field,
|
|
47
|
+
# "predicate" with no criteria field, a choice lists its options as
|
|
48
|
+
# {value, description}, and answers come back as a list.
|
|
41
49
|
module Decisions
|
|
42
50
|
# Decisions' input is text, so a structured state goes as its JSON.
|
|
43
51
|
def self.payload(model, state, questions)
|
|
@@ -45,8 +53,16 @@ module Judgelint
|
|
|
45
53
|
{ "model" => model, "input" => input, "questions" => questions.map { |question| render(question) } }
|
|
46
54
|
end
|
|
47
55
|
|
|
48
|
-
#
|
|
56
|
+
# A yes/no question's criteria become lines of its instructions, as
|
|
57
|
+
# RubyLLM and LiteLLM do.
|
|
49
58
|
def self.render(question)
|
|
59
|
+
if question.choice?
|
|
60
|
+
choices = question.options.map do |value, description|
|
|
61
|
+
description ? { "value" => value, "description" => description } : { "value" => value }
|
|
62
|
+
end
|
|
63
|
+
return { "type" => "choice", "name" => question.key, "instructions" => question.instructions, "choices" => choices }
|
|
64
|
+
end
|
|
65
|
+
|
|
50
66
|
lines = [question.instructions]
|
|
51
67
|
lines << "Yes: #{question.yes_means}" if question.yes_means
|
|
52
68
|
lines << "No: #{question.no_means}" if question.no_means
|
|
@@ -68,10 +84,19 @@ module Judgelint
|
|
|
68
84
|
probabilities = {}
|
|
69
85
|
questions.each do |question|
|
|
70
86
|
answer = by_name.key?(question.key) ? by_name[question.key] : unnamed.shift
|
|
71
|
-
probabilities[question.key] =
|
|
87
|
+
probabilities[question.key] =
|
|
88
|
+
question.choice? ? Protocol.distribution(answer, listed(answer), question) : Protocol.probability(answer, "probability", question)
|
|
72
89
|
end
|
|
73
90
|
probabilities
|
|
74
91
|
end
|
|
92
|
+
|
|
93
|
+
# [{value, probability}] as {value => probability}.
|
|
94
|
+
def self.listed(answer)
|
|
95
|
+
list = answer.is_a?(Hash) ? answer["probabilities"] : nil
|
|
96
|
+
return nil unless list.is_a?(Array)
|
|
97
|
+
|
|
98
|
+
list.to_h { |entry| [entry["value"].to_s, entry["probability"]] }
|
|
99
|
+
end
|
|
75
100
|
end
|
|
76
101
|
|
|
77
102
|
ALL = { "system_one" => SystemOne, "decisions" => Decisions }.freeze
|
|
@@ -84,5 +109,14 @@ module Judgelint
|
|
|
84
109
|
|
|
85
110
|
value.to_f
|
|
86
111
|
end
|
|
112
|
+
|
|
113
|
+
def self.distribution(answer, probabilities, question)
|
|
114
|
+
return nil if answer.is_a?(Hash) && answer["type"] == "refusal"
|
|
115
|
+
raise ApiError, "No choice answer for #{question.key}: #{answer.inspect}" unless probabilities.is_a?(Hash)
|
|
116
|
+
|
|
117
|
+
found = {}
|
|
118
|
+
question.options.each_key { |option| found[option] = probabilities[option].to_f }
|
|
119
|
+
found
|
|
120
|
+
end
|
|
87
121
|
end
|
|
88
122
|
end
|
data/lib/judgelint/question.rb
CHANGED
|
@@ -4,52 +4,75 @@ module Judgelint
|
|
|
4
4
|
# One rule asked about one range of a file. The model sees the whole file;
|
|
5
5
|
# the question points at the lines.
|
|
6
6
|
#
|
|
7
|
-
# A spec's bullet is
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
#
|
|
7
|
+
# A spec's bullet is a choice between breaking the rule, following it, and
|
|
8
|
+
# the rule not applying. With the three side by side, a decision model
|
|
9
|
+
# commits to an answer, where a yes/no question gets a hedge. The same
|
|
10
|
+
# request asks which line (or, for a directory, which file) breaks the rule
|
|
11
|
+
# most clearly, so a finding can name it.
|
|
11
12
|
#
|
|
12
|
-
#
|
|
13
|
-
#
|
|
14
|
-
# told about hints inside the rule's question, it only hedges.
|
|
13
|
+
# A rule written as a question is asked yes or no, as written. Code with a
|
|
14
|
+
# judgehint comment is also asked, yes or no, whether the hint excuses it.
|
|
15
15
|
class Question
|
|
16
16
|
BREAKS = "The code clearly breaks the rule."
|
|
17
|
-
FOLLOWS = "The code follows the rule
|
|
17
|
+
FOLLOWS = "The code follows the rule."
|
|
18
|
+
NOT_APPLICABLE = "The rule doesn't apply to this code."
|
|
18
19
|
EXCUSED = "A judgehint comment is about this rule, and gives a reason the code may break it."
|
|
19
20
|
NOT_EXCUSED = "No judgehint comment is about this rule. Any judgehint is about a different rule, or gives no reason."
|
|
20
21
|
|
|
21
|
-
attr_reader :rule, :first, :last
|
|
22
|
+
attr_reader :rule, :first, :last, :places
|
|
22
23
|
|
|
23
|
-
|
|
24
|
+
# +kind+ is :verdict, :where, or :excuse. +places+ are a :where question's
|
|
25
|
+
# options: line ids (L12) or the directory's files, as the model sees them.
|
|
26
|
+
def initialize(rule, first, last, kind: :verdict, places: [])
|
|
24
27
|
@rule = rule
|
|
25
28
|
@first = first
|
|
26
29
|
@last = last
|
|
27
|
-
@
|
|
28
|
-
@
|
|
30
|
+
@kind = kind
|
|
31
|
+
@places = places
|
|
29
32
|
end
|
|
30
33
|
|
|
31
34
|
def excuse?
|
|
32
|
-
@excuse
|
|
35
|
+
@kind == :excuse
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def where?
|
|
39
|
+
@kind == :where
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# A spec's bullet and its location are choices; everything else is yes or no.
|
|
43
|
+
def choice?
|
|
44
|
+
where? || (@kind == :verdict && !rule.convention.nil?)
|
|
33
45
|
end
|
|
34
46
|
|
|
35
47
|
def key
|
|
36
48
|
base = "#{rule.question_key}.#{first}-#{last}"
|
|
37
|
-
return "#{base}.excused" if
|
|
38
|
-
return base
|
|
49
|
+
return "#{base}.excused" if excuse?
|
|
50
|
+
return "#{base}.where" if where?
|
|
51
|
+
|
|
52
|
+
base
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# A choice's options, as {name => description or nil}.
|
|
56
|
+
def options
|
|
57
|
+
return places.to_h { |place| [place, nil] } if where?
|
|
39
58
|
|
|
40
|
-
|
|
59
|
+
{ "breaks" => rule.yes_means || BREAKS, "follows" => rule.no_means || FOLLOWS, "not_applicable" => NOT_APPLICABLE }
|
|
41
60
|
end
|
|
42
61
|
|
|
43
|
-
# The probability the code breaks the rule, from
|
|
44
|
-
|
|
45
|
-
|
|
62
|
+
# The probability the code breaks the rule, from a verdict's answer: a
|
|
63
|
+
# choice's probabilities, or a yes/no question's probability of yes.
|
|
64
|
+
def violation(answer)
|
|
65
|
+
return answer["breaks"].to_f if answer.is_a?(Hash)
|
|
66
|
+
|
|
67
|
+
rule.violation_probability(answer)
|
|
46
68
|
end
|
|
47
69
|
|
|
48
70
|
# "Answer yes only when..." keeps the model from flagging code the rule
|
|
49
71
|
# doesn't cover.
|
|
50
72
|
def instructions
|
|
51
|
-
return excuse_instructions if
|
|
52
|
-
return
|
|
73
|
+
return excuse_instructions if excuse?
|
|
74
|
+
return where_instructions if where?
|
|
75
|
+
return "How #{rule.scope == 'file' ? 'does' : 'do'} #{subject} relate to the following team rule? Rule: #{rule.convention}" if rule.convention
|
|
53
76
|
|
|
54
77
|
text = rule.window? ? "Look at lines #{first}-#{last} of the file; the rest is context. #{rule.question}" : rule.question
|
|
55
78
|
return text unless rule.fail_when == "yes"
|
|
@@ -60,63 +83,41 @@ module Judgelint
|
|
|
60
83
|
end
|
|
61
84
|
|
|
62
85
|
def yes_means
|
|
63
|
-
return EXCUSED if
|
|
64
|
-
return
|
|
86
|
+
return EXCUSED if excuse?
|
|
87
|
+
return nil if choice?
|
|
65
88
|
|
|
66
|
-
|
|
89
|
+
rule.yes_means
|
|
67
90
|
end
|
|
68
91
|
|
|
69
92
|
def no_means
|
|
70
|
-
return NOT_EXCUSED if
|
|
71
|
-
return
|
|
93
|
+
return NOT_EXCUSED if excuse?
|
|
94
|
+
return nil if choice?
|
|
72
95
|
|
|
73
|
-
|
|
96
|
+
rule.no_means
|
|
74
97
|
end
|
|
75
98
|
|
|
76
99
|
def digest
|
|
77
|
-
@digest ||= Digest::SHA256.hexdigest(JSON.generate([instructions, yes_means.to_s, no_means.to_s
|
|
100
|
+
@digest ||= Digest::SHA256.hexdigest(JSON.generate([instructions, yes_means.to_s, no_means.to_s,
|
|
101
|
+
choice? ? options.to_a : []]))
|
|
78
102
|
end
|
|
79
103
|
|
|
80
104
|
private
|
|
81
105
|
|
|
106
|
+
def subject
|
|
107
|
+
{ "window" => "lines #{first}-#{last} of the file", "file" => "the file", "directory" => "the files" }
|
|
108
|
+
.fetch(rule.scope)
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
def where_instructions
|
|
112
|
+
which = rule.directory? ? "Which file" : "Which line, given by its number as L<number>,"
|
|
113
|
+
"#{which} most clearly breaks the following team rule? Rule: #{rule.convention}"
|
|
114
|
+
end
|
|
115
|
+
|
|
82
116
|
def excuse_instructions
|
|
83
117
|
where = { "window" => "in lines #{first}-#{last} of the file", "file" => "in the file", "directory" => "in the files" }
|
|
84
118
|
.fetch(rule.scope)
|
|
85
119
|
"Does a judgehint comment #{where} explain why the code intentionally departs from the following team " \
|
|
86
120
|
"rule, with a reason specific to this rule? Rule: #{rule.convention || rule.message}"
|
|
87
121
|
end
|
|
88
|
-
|
|
89
|
-
# The spec's prose is in the state as team_rules, so the bullet is "the
|
|
90
|
-
# following team rule".
|
|
91
|
-
def convention_instructions
|
|
92
|
-
ask =
|
|
93
|
-
case rule.scope
|
|
94
|
-
when "window"
|
|
95
|
-
if @inverted
|
|
96
|
-
"Do lines #{first}-#{last} of the file follow the following team rule, or does it not apply to them? " \
|
|
97
|
-
"Answer false only when those lines clearly break it."
|
|
98
|
-
else
|
|
99
|
-
"Do lines #{first}-#{last} of the file clearly violate the following team rule? Answer true only " \
|
|
100
|
-
"when the code breaks the rule, not when the rule does not apply to it."
|
|
101
|
-
end
|
|
102
|
-
when "file"
|
|
103
|
-
if @inverted
|
|
104
|
-
"Does the file follow the following team rule, or does it not apply to it? Answer false only when " \
|
|
105
|
-
"the file clearly breaks it."
|
|
106
|
-
else
|
|
107
|
-
"Does the file clearly violate the following team rule? Answer true only when the code breaks the " \
|
|
108
|
-
"rule, not when the rule does not apply to it."
|
|
109
|
-
end
|
|
110
|
-
else
|
|
111
|
-
if @inverted
|
|
112
|
-
"Do the files follow the following team rule, or does it not apply to them? Answer false only when " \
|
|
113
|
-
"they clearly break it."
|
|
114
|
-
else
|
|
115
|
-
"Do the files clearly violate the following team rule? Answer true only when the code breaks the " \
|
|
116
|
-
"rule, not when the rule does not apply to it."
|
|
117
|
-
end
|
|
118
|
-
end
|
|
119
|
-
"#{ask} Rule: #{rule.convention}"
|
|
120
|
-
end
|
|
121
122
|
end
|
|
122
123
|
end
|
data/lib/judgelint/reporter.rb
CHANGED
|
@@ -46,21 +46,27 @@ module Judgelint
|
|
|
46
46
|
def text(verdicts, file_count)
|
|
47
47
|
shown = verdicts.reject { |verdict| verdict.status == :pass }
|
|
48
48
|
width = shown.map { |verdict| verdict.rule.id.size }.max
|
|
49
|
+
places = [7, *shown.map { |verdict| range(verdict).size }].max
|
|
49
50
|
shown.group_by(&:path).each do |path, group|
|
|
50
51
|
@out.puts path
|
|
51
|
-
group.each { |verdict| @out.puts " #{line(verdict, width)}" }
|
|
52
|
+
group.each { |verdict| @out.puts " #{line(verdict, width, places)}" }
|
|
52
53
|
end
|
|
53
54
|
@out.puts "" unless shown.empty?
|
|
54
55
|
@out.puts summary(verdicts, file_count)
|
|
55
56
|
end
|
|
56
57
|
|
|
57
|
-
def line(verdict, width)
|
|
58
|
+
def line(verdict, width, places)
|
|
58
59
|
label = verdict.status == :violation ? verdict.rule.severity : verdict.status.to_s
|
|
59
|
-
format("%s %-10s %-#{width}s
|
|
60
|
+
format("%s %-10s %-#{width}s %-#{places}s %s %s", MARKS.fetch(verdict.status), label, verdict.rule.id,
|
|
60
61
|
range(verdict), probability(verdict), verdict.rule.message)
|
|
61
62
|
end
|
|
62
63
|
|
|
64
|
+
# The line that breaks the rule, or the range asked about; for a
|
|
65
|
+
# directory, the file in it.
|
|
63
66
|
def range(verdict)
|
|
67
|
+
return verdict.file.delete_prefix("#{verdict.path}/") if verdict.file
|
|
68
|
+
return verdict.line.to_s if verdict.line
|
|
69
|
+
|
|
64
70
|
verdict.lines ? verdict.lines.join("-") : ""
|
|
65
71
|
end
|
|
66
72
|
|
|
@@ -107,8 +113,12 @@ module Judgelint
|
|
|
107
113
|
level = { violation: verdict.rule.error? ? "error" : "warning", uncertain: "notice" }[verdict.status]
|
|
108
114
|
return unless level
|
|
109
115
|
|
|
110
|
-
properties = ["file=#{escape(verdict.path, true)}"]
|
|
111
|
-
|
|
116
|
+
properties = ["file=#{escape(verdict.file || verdict.path, true)}"]
|
|
117
|
+
if verdict.line
|
|
118
|
+
properties << "line=#{verdict.line}"
|
|
119
|
+
elsif verdict.lines
|
|
120
|
+
properties += ["line=#{verdict.lines[0]}", "endLine=#{verdict.lines[1]}"]
|
|
121
|
+
end
|
|
112
122
|
properties << "title=#{escape("judgelint #{verdict.rule.id}", true)}"
|
|
113
123
|
@out.puts "::#{level} #{properties.join(',')}::#{escape("#{verdict.rule.message} (#{probability(verdict)})", false)}"
|
|
114
124
|
end
|
|
@@ -135,12 +145,16 @@ module Judgelint
|
|
|
135
145
|
end
|
|
136
146
|
|
|
137
147
|
def sarif_result(verdict)
|
|
138
|
-
region =
|
|
148
|
+
region =
|
|
149
|
+
if verdict.line then { "startLine" => verdict.line }
|
|
150
|
+
elsif verdict.lines then { "startLine" => verdict.lines[0], "endLine" => verdict.lines[1] }
|
|
151
|
+
else {}
|
|
152
|
+
end
|
|
139
153
|
level = verdict.status == :uncertain ? "note" : (verdict.rule.error? ? "error" : "warning")
|
|
140
154
|
{
|
|
141
155
|
"ruleId" => verdict.rule.id, "level" => level,
|
|
142
156
|
"message" => { "text" => "#{verdict.rule.message} (#{probability(verdict)})" },
|
|
143
|
-
"locations" => [{ "physicalLocation" => { "artifactLocation" => { "uri" => verdict.path }, "region" => region } }]
|
|
157
|
+
"locations" => [{ "physicalLocation" => { "artifactLocation" => { "uri" => verdict.file || verdict.path }, "region" => region } }]
|
|
144
158
|
}
|
|
145
159
|
end
|
|
146
160
|
end
|
data/lib/judgelint/runner.rb
CHANGED
|
@@ -43,6 +43,25 @@ module Judgelint
|
|
|
43
43
|
end
|
|
44
44
|
end
|
|
45
45
|
|
|
46
|
+
# One range's averaged answers: the probability it breaks the rule, and
|
|
47
|
+
# where, when the model is sure enough of one line or file.
|
|
48
|
+
class Window
|
|
49
|
+
attr_reader :first, :last, :probability, :place
|
|
50
|
+
|
|
51
|
+
def initialize(first, last, probability, place)
|
|
52
|
+
@first = first
|
|
53
|
+
@last = last
|
|
54
|
+
@probability = probability
|
|
55
|
+
@place = place
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# A finding names a line or file when it gets at least this share of the
|
|
60
|
+
# "where" answer.
|
|
61
|
+
PLACE_AT = 0.5
|
|
62
|
+
# A choice takes at most this many options.
|
|
63
|
+
MAX_OPTIONS = 255
|
|
64
|
+
|
|
46
65
|
# +clients+ has one Client per model in config.models.
|
|
47
66
|
def initialize(config:, rules:, clients:, cache:)
|
|
48
67
|
@config = config
|
|
@@ -150,8 +169,9 @@ module Judgelint
|
|
|
150
169
|
asked = []
|
|
151
170
|
ranges.each do |first, last|
|
|
152
171
|
asked << Question.new(rule, first, last)
|
|
153
|
-
|
|
154
|
-
asked << Question.new(rule, first, last,
|
|
172
|
+
places = places(source, rule, first, last)
|
|
173
|
+
asked << Question.new(rule, first, last, kind: :where, places: places) unless places.empty?
|
|
174
|
+
asked << Question.new(rule, first, last, kind: :excuse) if source.hinted?(first, last)
|
|
155
175
|
end
|
|
156
176
|
questions[rule] = asked
|
|
157
177
|
end
|
|
@@ -160,6 +180,16 @@ module Judgelint
|
|
|
160
180
|
job
|
|
161
181
|
end
|
|
162
182
|
|
|
183
|
+
# The options for asking where a bullet is broken: the range's line ids, or
|
|
184
|
+
# the directory's files. None for a rule written as a question, or past
|
|
185
|
+
# what a choice can hold.
|
|
186
|
+
def places(source, rule, first, last)
|
|
187
|
+
return [] unless rule.convention
|
|
188
|
+
|
|
189
|
+
places = rule.directory? ? source.members.map { |_path, shown| shown } : (first..last).map { |line| "L#{line}" }
|
|
190
|
+
places.size > MAX_OPTIONS ? [] : places
|
|
191
|
+
end
|
|
192
|
+
|
|
163
193
|
# Rules that read different context files or spec prose can't share a request.
|
|
164
194
|
def by_context(rules)
|
|
165
195
|
rules.group_by { |rule| [rule.context, rule.prose] }.map { |(globs, _prose), group| yield(Glob.expand(globs), group) }
|
|
@@ -230,11 +260,11 @@ module Judgelint
|
|
|
230
260
|
job.expected ? found.map { |verdict| verdict.as_example(job.expected) } : found
|
|
231
261
|
end
|
|
232
262
|
|
|
233
|
-
# A window's probability of a violation is the average
|
|
234
|
-
#
|
|
235
|
-
#
|
|
236
|
-
#
|
|
237
|
-
# breaks the rule.
|
|
263
|
+
# A window's probability of a violation is the average of its verdict
|
|
264
|
+
# over the models, and its place is the line or file the models' "where"
|
|
265
|
+
# answers agree on. A rule's verdict is its worst window. A window whose
|
|
266
|
+
# judgehint excuses it, by the same average and threshold, is reported as
|
|
267
|
+
# excused instead, unless another window breaks the rule.
|
|
238
268
|
def verdict_for(rule, path, questions, job)
|
|
239
269
|
return Verdict.new(rule: rule, path: path, probability: nil, status: :skipped) if job.skipped
|
|
240
270
|
|
|
@@ -243,21 +273,24 @@ module Judgelint
|
|
|
243
273
|
questions.group_by { |question| [question.first, question.last] }.each do |(first, last), asked|
|
|
244
274
|
values = []
|
|
245
275
|
excuses = []
|
|
276
|
+
spread = {}
|
|
246
277
|
asked.each do |question|
|
|
247
278
|
job.answers.each_value do |answers|
|
|
248
|
-
|
|
249
|
-
next if
|
|
279
|
+
answer = answers[question.key]
|
|
280
|
+
next if answer.nil?
|
|
250
281
|
|
|
251
282
|
if question.excuse?
|
|
252
|
-
excuses <<
|
|
283
|
+
excuses << answer
|
|
284
|
+
elsif question.where?
|
|
285
|
+
answer.each { |place, probability| spread[place] = (spread[place] || 0.0) + probability }
|
|
253
286
|
else
|
|
254
|
-
values << question.violation(
|
|
287
|
+
values << question.violation(answer)
|
|
255
288
|
end
|
|
256
289
|
end
|
|
257
290
|
end
|
|
258
291
|
next if values.empty?
|
|
259
292
|
|
|
260
|
-
window =
|
|
293
|
+
window = Window.new(first, last, values.sum / values.size, agreed(spread))
|
|
261
294
|
if !excuses.empty? && excuses.sum / excuses.size >= rule.threshold
|
|
262
295
|
excused << window
|
|
263
296
|
else
|
|
@@ -266,22 +299,45 @@ module Judgelint
|
|
|
266
299
|
end
|
|
267
300
|
return Verdict.new(rule: rule, path: path, probability: nil, status: :refused) if open.empty? && excused.empty?
|
|
268
301
|
|
|
269
|
-
found = worst(rule, path, open)
|
|
302
|
+
found = worst(rule, path, open, job.source)
|
|
270
303
|
return found if found && found.status != :pass
|
|
271
304
|
|
|
272
|
-
hidden = worst(rule, path, excused)
|
|
305
|
+
hidden = worst(rule, path, excused, job.source)
|
|
273
306
|
return found || hidden if hidden.nil? || hidden.status == :pass
|
|
274
307
|
|
|
275
|
-
Verdict.new(rule: rule, path: path, probability: hidden.probability, status: :excused, lines: hidden.lines
|
|
308
|
+
Verdict.new(rule: rule, path: path, probability: hidden.probability, status: :excused, lines: hidden.lines,
|
|
309
|
+
line: hidden.line, file: hidden.file)
|
|
310
|
+
end
|
|
311
|
+
|
|
312
|
+
# The place with at least PLACE_AT of the summed "where" probabilities.
|
|
313
|
+
# A loop rather than max_by, which Spinel's Hash doesn't have yet.
|
|
314
|
+
def agreed(spread)
|
|
315
|
+
total = 0.0
|
|
316
|
+
best = nil
|
|
317
|
+
spread.each do |place, value|
|
|
318
|
+
total += value
|
|
319
|
+
best = place if best.nil? || value > spread[best]
|
|
320
|
+
end
|
|
321
|
+
return nil if total <= 0
|
|
322
|
+
|
|
323
|
+
spread[best] / total >= PLACE_AT ? best : nil
|
|
276
324
|
end
|
|
277
325
|
|
|
278
|
-
# The worst of +windows
|
|
279
|
-
def worst(rule, path, windows)
|
|
326
|
+
# The worst of +windows+, or nil if there are none.
|
|
327
|
+
def worst(rule, path, windows, source)
|
|
280
328
|
return nil if windows.empty?
|
|
281
329
|
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
330
|
+
window = windows.max_by(&:probability)
|
|
331
|
+
line = nil
|
|
332
|
+
file = nil
|
|
333
|
+
if rule.directory?
|
|
334
|
+
member = source.members.find { |_member, shown| shown == window.place }
|
|
335
|
+
file = member[0] if member
|
|
336
|
+
elsif window.place
|
|
337
|
+
line = window.place.delete_prefix("L").to_i
|
|
338
|
+
end
|
|
339
|
+
lines = rule.directory? ? nil : [window.first, window.last]
|
|
340
|
+
Verdict.classify(rule, path, window.probability, warn_at: @config.warn_at, lines: lines, line: line, file: file)
|
|
285
341
|
end
|
|
286
342
|
|
|
287
343
|
# Each model's answers are cached on their own, so adding a model asks
|
data/lib/judgelint/source.rb
CHANGED
|
@@ -54,6 +54,11 @@ module Judgelint
|
|
|
54
54
|
(1..line_count).step(size).map { |first| [first, [first + size - 1, line_count].min] }
|
|
55
55
|
end
|
|
56
56
|
|
|
57
|
+
# The directory's files as [path, the path the model sees].
|
|
58
|
+
def members
|
|
59
|
+
@files.map { |file, _lines| [file, shown(file)] }
|
|
60
|
+
end
|
|
61
|
+
|
|
57
62
|
def hinted?(first, last)
|
|
58
63
|
directory? ? !@hints.empty? : @hints.any? { |line| line.between?(first, last) }
|
|
59
64
|
end
|
data/lib/judgelint/verdict.rb
CHANGED
|
@@ -8,35 +8,39 @@ module Judgelint
|
|
|
8
8
|
# large to judge), :refused (the model declined the question), or :excused
|
|
9
9
|
# (a violation or uncertain, but a judgehint excuses it). Skipped and
|
|
10
10
|
# refused files are reported, never silently passed. +lines+ is the [first,
|
|
11
|
-
# last] range the verdict came from.
|
|
11
|
+
# last] range the verdict came from. +line+ is the line in it that breaks
|
|
12
|
+
# the rule, and +file+ the file in a directory, when the model could say.
|
|
12
13
|
#
|
|
13
14
|
# An example's verdict also knows the answer it should have had: +expected+
|
|
14
15
|
# is :pass or :fail for files under rules/<id>/pass and rules/<id>/fail.
|
|
15
16
|
# Not Data.define: Spinel can't compile a Data or Struct in a program that also uses net/https and json.
|
|
16
17
|
class Verdict
|
|
17
|
-
attr_reader :rule, :path, :probability, :status, :lines, :expected
|
|
18
|
+
attr_reader :rule, :path, :probability, :status, :lines, :line, :file, :expected
|
|
18
19
|
|
|
19
|
-
def initialize(rule:, path:, probability:, status:, lines: nil, expected: nil)
|
|
20
|
+
def initialize(rule:, path:, probability:, status:, lines: nil, line: nil, file: nil, expected: nil)
|
|
20
21
|
@rule = rule
|
|
21
22
|
@path = path
|
|
22
23
|
@probability = probability
|
|
23
24
|
@status = status
|
|
24
25
|
@lines = lines
|
|
26
|
+
@line = line
|
|
27
|
+
@file = file
|
|
25
28
|
@expected = expected
|
|
26
29
|
end
|
|
27
30
|
|
|
28
31
|
def as_example(expected)
|
|
29
|
-
Verdict.new(rule: rule, path: path, probability: probability, status: status, lines: lines,
|
|
32
|
+
Verdict.new(rule: rule, path: path, probability: probability, status: status, lines: lines, line: line, file: file,
|
|
33
|
+
expected: expected)
|
|
30
34
|
end
|
|
31
35
|
|
|
32
36
|
# +probability+ is the probability the code breaks the rule.
|
|
33
|
-
def self.classify(rule, path, probability, warn_at:, lines: nil)
|
|
37
|
+
def self.classify(rule, path, probability, warn_at:, lines: nil, line: nil, file: nil)
|
|
34
38
|
status =
|
|
35
39
|
if probability >= rule.threshold then :violation
|
|
36
40
|
elsif probability >= warn_at then :uncertain
|
|
37
41
|
else :pass
|
|
38
42
|
end
|
|
39
|
-
new(rule: rule, path: path, probability: probability, status: status, lines: lines)
|
|
43
|
+
new(rule: rule, path: path, probability: probability, status: status, lines: lines, line: line, file: file)
|
|
40
44
|
end
|
|
41
45
|
|
|
42
46
|
def failing?
|
|
@@ -53,7 +57,7 @@ module Judgelint
|
|
|
53
57
|
|
|
54
58
|
def to_h
|
|
55
59
|
{
|
|
56
|
-
"path" => path, "lines" => lines, "rule" => rule.id, "severity" => rule.severity, "status" => status.to_s,
|
|
60
|
+
"path" => path, "lines" => lines, "line" => line, "file" => file, "rule" => rule.id, "severity" => rule.severity, "status" => status.to_s,
|
|
57
61
|
"probability" => probability&.round(3), "message" => rule.message
|
|
58
62
|
}.merge(expected ? { "expected" => expected.to_s, "correct" => correct? } : {})
|
|
59
63
|
end
|
data/lib/judgelint/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: judgelint
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- JP Camara
|
|
@@ -10,10 +10,10 @@ cert_chain: []
|
|
|
10
10
|
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
11
|
dependencies: []
|
|
12
12
|
description: |
|
|
13
|
-
judgelint checks
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
13
|
+
judgelint checks code against conventions you write as bullets in a README.
|
|
14
|
+
It asks a decision model, such as TypeSafe's Jev, whether each file breaks
|
|
15
|
+
each bullet, and reports violations with line numbers. On macOS and Linux
|
|
16
|
+
the gem includes a native binary.
|
|
17
17
|
email:
|
|
18
18
|
- johnpcamara@gmail.com
|
|
19
19
|
executables:
|
|
@@ -69,6 +69,6 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
69
69
|
requirements: []
|
|
70
70
|
rubygems_version: 3.6.9
|
|
71
71
|
specification_version: 4
|
|
72
|
-
summary:
|
|
72
|
+
summary: Checks code against conventions written in plain English, using decision
|
|
73
73
|
models.
|
|
74
74
|
test_files: []
|