leanscreen 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- leanscreen-0.1.0/.gitignore +10 -0
- leanscreen-0.1.0/LICENSE +105 -0
- leanscreen-0.1.0/PKG-INFO +255 -0
- leanscreen-0.1.0/README.md +130 -0
- leanscreen-0.1.0/docs/PUBLISH_BRIEF.md +112 -0
- leanscreen-0.1.0/pyproject.toml +96 -0
- leanscreen-0.1.0/src/leanscreen/__init__.py +16 -0
- leanscreen-0.1.0/src/leanscreen/backtranslate.py +205 -0
- leanscreen-0.1.0/src/leanscreen/config.py +69 -0
- leanscreen-0.1.0/src/leanscreen/errors.py +47 -0
- leanscreen-0.1.0/src/leanscreen/falsify.py +116 -0
- leanscreen-0.1.0/src/leanscreen/lean_server.py +252 -0
- leanscreen-0.1.0/src/leanscreen/lints.py +445 -0
- leanscreen-0.1.0/src/leanscreen/llm.py +457 -0
- leanscreen-0.1.0/src/leanscreen/logging.py +78 -0
- leanscreen-0.1.0/src/leanscreen/py.typed +0 -0
- leanscreen-0.1.0/src/leanscreen/quality.py +524 -0
- leanscreen-0.1.0/src/leanscreen/score.py +282 -0
- leanscreen-0.1.0/src/leanscreen/server.py +421 -0
- leanscreen-0.1.0/src/leanscreen/verify.py +338 -0
- leanscreen-0.1.0/tests/test_backtranslate.py +92 -0
- leanscreen-0.1.0/tests/test_lean_server.py +226 -0
- leanscreen-0.1.0/tests/test_lints.py +261 -0
- leanscreen-0.1.0/tests/test_llm.py +680 -0
- leanscreen-0.1.0/tests/test_mcp_server.py +219 -0
- leanscreen-0.1.0/tests/test_quality.py +251 -0
- leanscreen-0.1.0/tests/test_score.py +195 -0
- leanscreen-0.1.0/tests/test_verify.py +117 -0
- leanscreen-0.1.0/tests/test_verify_batch.py +99 -0
leanscreen-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# Functional Source License, Version 1.1, Apache 2.0 Future License
|
|
2
|
+
|
|
3
|
+
## Abbreviation
|
|
4
|
+
|
|
5
|
+
FSL-1.1-Apache-2.0
|
|
6
|
+
|
|
7
|
+
## Notice
|
|
8
|
+
|
|
9
|
+
Copyright 2026 Millennium Research
|
|
10
|
+
|
|
11
|
+
## Terms and Conditions
|
|
12
|
+
|
|
13
|
+
### Licensor ("We")
|
|
14
|
+
|
|
15
|
+
The party offering the Software under these Terms and Conditions.
|
|
16
|
+
|
|
17
|
+
### The Software
|
|
18
|
+
|
|
19
|
+
The "Software" is each version of the software that we make available under
|
|
20
|
+
these Terms and Conditions, as indicated by our inclusion of these Terms and
|
|
21
|
+
Conditions with the Software.
|
|
22
|
+
|
|
23
|
+
### License Grant
|
|
24
|
+
|
|
25
|
+
Subject to your compliance with this License Grant and the Patents,
|
|
26
|
+
Redistribution and Trademark clauses below, we hereby grant you the right to
|
|
27
|
+
use, copy, modify, create derivative works, publicly perform, publicly display
|
|
28
|
+
and redistribute the Software for any Permitted Purpose identified below.
|
|
29
|
+
|
|
30
|
+
### Permitted Purpose
|
|
31
|
+
|
|
32
|
+
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
|
|
33
|
+
means making the Software available to others in a commercial product or
|
|
34
|
+
service that:
|
|
35
|
+
|
|
36
|
+
1. substitutes for the Software;
|
|
37
|
+
|
|
38
|
+
2. substitutes for any other product or service we offer using the Software
|
|
39
|
+
that exists as of the date we make the Software available; or
|
|
40
|
+
|
|
41
|
+
3. offers the same or substantially similar functionality as the Software.
|
|
42
|
+
|
|
43
|
+
Permitted Purposes specifically include using the Software:
|
|
44
|
+
|
|
45
|
+
1. for your internal use and access;
|
|
46
|
+
|
|
47
|
+
2. for non-commercial education;
|
|
48
|
+
|
|
49
|
+
3. for non-commercial research; and
|
|
50
|
+
|
|
51
|
+
4. in connection with professional services that you provide to a licensee
|
|
52
|
+
using the Software in accordance with these Terms and Conditions.
|
|
53
|
+
|
|
54
|
+
### Patents
|
|
55
|
+
|
|
56
|
+
To the extent your use for a Permitted Purpose would necessarily infringe our
|
|
57
|
+
patents, the license grant above includes a license under our patents. If you
|
|
58
|
+
make a claim against any party that the Software infringes or contributes to
|
|
59
|
+
the infringement of any patent, then your patent license to the Software ends
|
|
60
|
+
immediately.
|
|
61
|
+
|
|
62
|
+
### Redistribution
|
|
63
|
+
|
|
64
|
+
The Terms and Conditions apply to all copies, modifications and derivatives of
|
|
65
|
+
the Software.
|
|
66
|
+
|
|
67
|
+
If you redistribute any copies, modifications or derivatives of the Software,
|
|
68
|
+
you must include a copy of or a link to these Terms and Conditions and not
|
|
69
|
+
remove any copyright notices provided in or with the Software.
|
|
70
|
+
|
|
71
|
+
### Disclaimer
|
|
72
|
+
|
|
73
|
+
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
|
|
74
|
+
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
|
|
75
|
+
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
|
|
76
|
+
|
|
77
|
+
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
|
|
78
|
+
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
|
|
79
|
+
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
|
|
80
|
+
|
|
81
|
+
### Trademarks
|
|
82
|
+
|
|
83
|
+
Except for displaying the License Details and identifying us as the origin of
|
|
84
|
+
the Software, you have no right under these Terms and Conditions to use our
|
|
85
|
+
trademarks, trade names, service marks or product names.
|
|
86
|
+
|
|
87
|
+
## Grant of Future License
|
|
88
|
+
|
|
89
|
+
We hereby irrevocably grant you an additional license to use the Software under
|
|
90
|
+
the Apache License, Version 2.0 that is effective on the second anniversary of
|
|
91
|
+
the date we make the Software available. On or after that date, you may use the
|
|
92
|
+
Software under the Apache License, Version 2.0, in which case the following
|
|
93
|
+
will apply:
|
|
94
|
+
|
|
95
|
+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
|
96
|
+
this file except in compliance with the License.
|
|
97
|
+
|
|
98
|
+
You may obtain a copy of the License at
|
|
99
|
+
|
|
100
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
101
|
+
|
|
102
|
+
Unless required by applicable law or agreed to in writing, software distributed
|
|
103
|
+
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
|
104
|
+
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
105
|
+
specific language governing permissions and limitations under the License.
|
|
@@ -0,0 +1,255 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: leanscreen
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: A calibrated faithfulness screen for informal↔Lean 4 statement pairs, served over MCP. Screens only — never certifies.
|
|
5
|
+
Project-URL: Homepage, https://millenniumresearch.ai/leanscreen
|
|
6
|
+
Project-URL: Repository, https://github.com/ibrahimmian36/leanscreen
|
|
7
|
+
Author-email: Millennium Research <ibrahimnmian@gmail.com>
|
|
8
|
+
License: # Functional Source License, Version 1.1, Apache 2.0 Future License
|
|
9
|
+
|
|
10
|
+
## Abbreviation
|
|
11
|
+
|
|
12
|
+
FSL-1.1-Apache-2.0
|
|
13
|
+
|
|
14
|
+
## Notice
|
|
15
|
+
|
|
16
|
+
Copyright 2026 Millennium Research
|
|
17
|
+
|
|
18
|
+
## Terms and Conditions
|
|
19
|
+
|
|
20
|
+
### Licensor ("We")
|
|
21
|
+
|
|
22
|
+
The party offering the Software under these Terms and Conditions.
|
|
23
|
+
|
|
24
|
+
### The Software
|
|
25
|
+
|
|
26
|
+
The "Software" is each version of the software that we make available under
|
|
27
|
+
these Terms and Conditions, as indicated by our inclusion of these Terms and
|
|
28
|
+
Conditions with the Software.
|
|
29
|
+
|
|
30
|
+
### License Grant
|
|
31
|
+
|
|
32
|
+
Subject to your compliance with this License Grant and the Patents,
|
|
33
|
+
Redistribution and Trademark clauses below, we hereby grant you the right to
|
|
34
|
+
use, copy, modify, create derivative works, publicly perform, publicly display
|
|
35
|
+
and redistribute the Software for any Permitted Purpose identified below.
|
|
36
|
+
|
|
37
|
+
### Permitted Purpose
|
|
38
|
+
|
|
39
|
+
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
|
|
40
|
+
means making the Software available to others in a commercial product or
|
|
41
|
+
service that:
|
|
42
|
+
|
|
43
|
+
1. substitutes for the Software;
|
|
44
|
+
|
|
45
|
+
2. substitutes for any other product or service we offer using the Software
|
|
46
|
+
that exists as of the date we make the Software available; or
|
|
47
|
+
|
|
48
|
+
3. offers the same or substantially similar functionality as the Software.
|
|
49
|
+
|
|
50
|
+
Permitted Purposes specifically include using the Software:
|
|
51
|
+
|
|
52
|
+
1. for your internal use and access;
|
|
53
|
+
|
|
54
|
+
2. for non-commercial education;
|
|
55
|
+
|
|
56
|
+
3. for non-commercial research; and
|
|
57
|
+
|
|
58
|
+
4. in connection with professional services that you provide to a licensee
|
|
59
|
+
using the Software in accordance with these Terms and Conditions.
|
|
60
|
+
|
|
61
|
+
### Patents
|
|
62
|
+
|
|
63
|
+
To the extent your use for a Permitted Purpose would necessarily infringe our
|
|
64
|
+
patents, the license grant above includes a license under our patents. If you
|
|
65
|
+
make a claim against any party that the Software infringes or contributes to
|
|
66
|
+
the infringement of any patent, then your patent license to the Software ends
|
|
67
|
+
immediately.
|
|
68
|
+
|
|
69
|
+
### Redistribution
|
|
70
|
+
|
|
71
|
+
The Terms and Conditions apply to all copies, modifications and derivatives of
|
|
72
|
+
the Software.
|
|
73
|
+
|
|
74
|
+
If you redistribute any copies, modifications or derivatives of the Software,
|
|
75
|
+
you must include a copy of or a link to these Terms and Conditions and not
|
|
76
|
+
remove any copyright notices provided in or with the Software.
|
|
77
|
+
|
|
78
|
+
### Disclaimer
|
|
79
|
+
|
|
80
|
+
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
|
|
81
|
+
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
|
|
82
|
+
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
|
|
83
|
+
|
|
84
|
+
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
|
|
85
|
+
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
|
|
86
|
+
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
|
|
87
|
+
|
|
88
|
+
### Trademarks
|
|
89
|
+
|
|
90
|
+
Except for displaying the License Details and identifying us as the origin of
|
|
91
|
+
the Software, you have no right under these Terms and Conditions to use our
|
|
92
|
+
trademarks, trade names, service marks or product names.
|
|
93
|
+
|
|
94
|
+
## Grant of Future License
|
|
95
|
+
|
|
96
|
+
We hereby irrevocably grant you an additional license to use the Software under
|
|
97
|
+
the Apache License, Version 2.0 that is effective on the second anniversary of
|
|
98
|
+
the date we make the Software available. On or after that date, you may use the
|
|
99
|
+
Software under the Apache License, Version 2.0, in which case the following
|
|
100
|
+
will apply:
|
|
101
|
+
|
|
102
|
+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
|
103
|
+
this file except in compliance with the License.
|
|
104
|
+
|
|
105
|
+
You may obtain a copy of the License at
|
|
106
|
+
|
|
107
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
108
|
+
|
|
109
|
+
Unless required by applicable law or agreed to in writing, software distributed
|
|
110
|
+
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
|
111
|
+
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
112
|
+
specific language governing permissions and limitations under the License.
|
|
113
|
+
License-File: LICENSE
|
|
114
|
+
Keywords: autoformalization,faithfulness,lean4,mathlib,mcp
|
|
115
|
+
Requires-Python: <3.14,>=3.12
|
|
116
|
+
Requires-Dist: httpx>=0.27.0
|
|
117
|
+
Requires-Dist: mcp<2,>=1.28.0
|
|
118
|
+
Requires-Dist: pydantic-settings>=2.3.0
|
|
119
|
+
Requires-Dist: pydantic>=2.7.0
|
|
120
|
+
Provides-Extra: dev
|
|
121
|
+
Requires-Dist: mypy>=1.10.0; extra == 'dev'
|
|
122
|
+
Requires-Dist: pytest>=8.2.0; extra == 'dev'
|
|
123
|
+
Requires-Dist: ruff>=0.5.0; extra == 'dev'
|
|
124
|
+
Description-Content-Type: text/markdown
|
|
125
|
+
|
|
126
|
+
# leanscreen
|
|
127
|
+
|
|
128
|
+
A calibrated faithfulness screen for informal↔Lean 4 statement pairs, served
|
|
129
|
+
over [MCP](https://modelcontextprotocol.io) so Claude (Code, Desktop, or any
|
|
130
|
+
MCP client) can check statements **while you draft them**.
|
|
131
|
+
|
|
132
|
+
**The one thing to understand before using it:** this screen may only
|
|
133
|
+
*reject*. `passed_screening` means "no defect found by this harness" — it is
|
|
134
|
+
**not** a certification of faithfulness. Measured against 886 frozen human
|
|
135
|
+
verdicts, statements a human reviewer had rejected still passed the full
|
|
136
|
+
screen 17.0% of the time for theorems and 35.6% for definitions; statements
|
|
137
|
+
a human had certified faithful were flagged 15–18% of the time. Every
|
|
138
|
+
response carries this calibration verbatim.
|
|
139
|
+
|
|
140
|
+
## Two tools
|
|
141
|
+
|
|
142
|
+
**`check_fast`** — deterministic only: lints (unused binders, trivially
|
|
143
|
+
satisfiable existentials, pinned `∃!` witnesses, suspicious ℕ-arithmetic,
|
|
144
|
+
…), vacuity checks (reflexive goals, `True` goals, withheld declarations),
|
|
145
|
+
and Lean 4 elaboration against your own mathlib environment. **Zero API
|
|
146
|
+
calls, no key needed, ~0.1s per statement once the REPL is warm.** Call it
|
|
147
|
+
constantly while drafting.
|
|
148
|
+
|
|
149
|
+
**`check_deep`** — everything in `check_fast`, plus two independent LLM
|
|
150
|
+
judges under strict consensus (a back-translation judge and a
|
|
151
|
+
clause-by-clause checklist judge on separate models) and an adversarial
|
|
152
|
+
counterexample probe. Runs on **your** `ANTHROPIC_API_KEY`; measured cost is
|
|
153
|
+
roughly $0.17–0.27 per statement (the response reports actual spend as
|
|
154
|
+
`actual_cost_usd`), 30–60 seconds. Call it deliberately, before something
|
|
155
|
+
ships.
|
|
156
|
+
|
|
157
|
+
Both take `informal` (the natural-language statement), `lean` (the Lean 4
|
|
158
|
+
statement), and an optional `kind` (`theorem` | `definition`, inferred from
|
|
159
|
+
the declaration head when omitted). Responses rank their evidence —
|
|
160
|
+
`counterexample` > `deterministic` > `two-judge-consensus` > `single-judge`
|
|
161
|
+
— and a single-judge flag is explicitly labeled as below the reporting bar.
|
|
162
|
+
|
|
163
|
+
## Install
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
pip install git+https://github.com/ibrahimmian36/leanscreen
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Requires Python ≥3.12. Runtime dependencies: `httpx`, `pydantic`,
|
|
170
|
+
`pydantic-settings`, `mcp` — nothing else.
|
|
171
|
+
|
|
172
|
+
## Lean setup (optional but recommended)
|
|
173
|
+
|
|
174
|
+
Without a Lean project, the server still runs — `check_fast` does lints +
|
|
175
|
+
vacuity and says plainly that elaboration was skipped. With one, statements
|
|
176
|
+
are elaborated for real:
|
|
177
|
+
|
|
178
|
+
1. A Lean 4 project with mathlib, built: `lake build` inside it.
|
|
179
|
+
2. The [community REPL](https://github.com/leanprover-community/repl),
|
|
180
|
+
built against the **same toolchain**: `lake build` inside the repl repo
|
|
181
|
+
gives you `.lake/build/bin/repl`.
|
|
182
|
+
3. `lake` on the server's PATH.
|
|
183
|
+
|
|
184
|
+
mathlib imports once at server startup (~100 seconds, in the background —
|
|
185
|
+
calls arriving mid-warm-up answer immediately with a "still warming" note),
|
|
186
|
+
then each check takes ~0.1s.
|
|
187
|
+
|
|
188
|
+
## Configuration
|
|
189
|
+
|
|
190
|
+
Environment variables (or a `.env` in the working directory), all
|
|
191
|
+
`LEANSCREEN_`-prefixed:
|
|
192
|
+
|
|
193
|
+
| Variable | Default | Meaning |
|
|
194
|
+
|---|---|---|
|
|
195
|
+
| `LEANSCREEN_LEAN_PROJECT_PATH` | unset | Lean 4 + mathlib project (elaboration off when unset) |
|
|
196
|
+
| `LEANSCREEN_LEAN_REPL_PATH` | unset | community REPL binary; without it every check pays a full `lake env lean` |
|
|
197
|
+
| `LEANSCREEN_LEAN_TIMEOUT_SECONDS` | `180` | per-statement Lean budget |
|
|
198
|
+
| `LEANSCREEN_ANTHROPIC_MODEL` | `claude-opus-4-8` | judge A + probe (the calibrated default) |
|
|
199
|
+
| `LEANSCREEN_JUDGE_B_MODEL` | `claude-fable-5` | checklist judge (calibrated default; locked-surface models get a 32k token budget automatically) |
|
|
200
|
+
| `LEANSCREEN_MAX_TOKENS` | `4096` | judge A response budget |
|
|
201
|
+
| `ANTHROPIC_API_KEY` | unset | needed for `check_deep` only |
|
|
202
|
+
|
|
203
|
+
Claude Code (`.mcp.json` in your project) or Claude Desktop
|
|
204
|
+
(`claude_desktop_config.json`):
|
|
205
|
+
|
|
206
|
+
```json
|
|
207
|
+
{
|
|
208
|
+
"mcpServers": {
|
|
209
|
+
"lean-faithfulness-screen": {
|
|
210
|
+
"command": "leanscreen",
|
|
211
|
+
"env": {
|
|
212
|
+
"LEANSCREEN_LEAN_PROJECT_PATH": "/path/to/your/lean-mathlib-project",
|
|
213
|
+
"LEANSCREEN_LEAN_REPL_PATH": "/path/to/repl/.lake/build/bin/repl",
|
|
214
|
+
"ANTHROPIC_API_KEY": "sk-ant-…"
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
}
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
## What this does not guarantee
|
|
222
|
+
|
|
223
|
+
The judge configuration was calibrated 2026-07-15 against 886 frozen human
|
|
224
|
+
verdicts (595 faithful / 291 unfaithful) from a production research-math
|
|
225
|
+
corpus. Under strict two-judge consensus, human-rejected pairs still passed
|
|
226
|
+
17.0% (theorems) / 35.6% (definitions) of the time, and human-certified
|
|
227
|
+
pairs were flagged 15–18% of the time. Both judges are Anthropic-family
|
|
228
|
+
models, so correlated blind spots cannot be ruled out. The counterexample
|
|
229
|
+
probe confabulates: on one PutnamBench sample its counterexamples were wrong
|
|
230
|
+
4 times out of 5. **Treat every flag as a candidate for human confirmation
|
|
231
|
+
and every pass as "nothing found", never "faithful."**
|
|
232
|
+
|
|
233
|
+
Human certification — an expert reviewer confirming that the Lean means the
|
|
234
|
+
informal statement — is what this screen deliberately does not automate. We
|
|
235
|
+
offer it as a service: **contact ibrahimnmian@gmail.com**.
|
|
236
|
+
|
|
237
|
+
## License
|
|
238
|
+
|
|
239
|
+
[FSL-1.1-Apache-2.0](LICENSE) (the Functional Source License): free to
|
|
240
|
+
use, copy, modify, and redistribute — including internal commercial use,
|
|
241
|
+
non-commercial education and research, and professional services — but not
|
|
242
|
+
to offer as a competing commercial product or service. **Each version
|
|
243
|
+
automatically becomes Apache 2.0 two years after its release** (the same
|
|
244
|
+
license as mathlib). Not OSI-approved until the conversion — read it
|
|
245
|
+
before building on it commercially.
|
|
246
|
+
|
|
247
|
+
## Provenance
|
|
248
|
+
|
|
249
|
+
Extracted from Millennium Research's private formalization platform
|
|
250
|
+
(2026-07-28); the detector stack, judge prompts, and calibration figures are
|
|
251
|
+
the ones behind our benchmark audits — the miniF2F and ProofNet# filings are
|
|
252
|
+
public, and the PutnamBench, ProofNetVerif, and CLEVER audits have been
|
|
253
|
+
shared with their maintainers. The calibration *data* is not included.
|
|
254
|
+
|
|
255
|
+
Project page: [millenniumresearch.ai/leanscreen](https://millenniumresearch.ai/leanscreen)
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# leanscreen
|
|
2
|
+
|
|
3
|
+
A calibrated faithfulness screen for informal↔Lean 4 statement pairs, served
|
|
4
|
+
over [MCP](https://modelcontextprotocol.io) so Claude (Code, Desktop, or any
|
|
5
|
+
MCP client) can check statements **while you draft them**.
|
|
6
|
+
|
|
7
|
+
**The one thing to understand before using it:** this screen may only
|
|
8
|
+
*reject*. `passed_screening` means "no defect found by this harness" — it is
|
|
9
|
+
**not** a certification of faithfulness. Measured against 886 frozen human
|
|
10
|
+
verdicts, statements a human reviewer had rejected still passed the full
|
|
11
|
+
screen 17.0% of the time for theorems and 35.6% for definitions; statements
|
|
12
|
+
a human had certified faithful were flagged 15–18% of the time. Every
|
|
13
|
+
response carries this calibration verbatim.
|
|
14
|
+
|
|
15
|
+
## Two tools
|
|
16
|
+
|
|
17
|
+
**`check_fast`** — deterministic only: lints (unused binders, trivially
|
|
18
|
+
satisfiable existentials, pinned `∃!` witnesses, suspicious ℕ-arithmetic,
|
|
19
|
+
…), vacuity checks (reflexive goals, `True` goals, withheld declarations),
|
|
20
|
+
and Lean 4 elaboration against your own mathlib environment. **Zero API
|
|
21
|
+
calls, no key needed, ~0.1s per statement once the REPL is warm.** Call it
|
|
22
|
+
constantly while drafting.
|
|
23
|
+
|
|
24
|
+
**`check_deep`** — everything in `check_fast`, plus two independent LLM
|
|
25
|
+
judges under strict consensus (a back-translation judge and a
|
|
26
|
+
clause-by-clause checklist judge on separate models) and an adversarial
|
|
27
|
+
counterexample probe. Runs on **your** `ANTHROPIC_API_KEY`; measured cost is
|
|
28
|
+
roughly $0.17–0.27 per statement (the response reports actual spend as
|
|
29
|
+
`actual_cost_usd`), 30–60 seconds. Call it deliberately, before something
|
|
30
|
+
ships.
|
|
31
|
+
|
|
32
|
+
Both take `informal` (the natural-language statement), `lean` (the Lean 4
|
|
33
|
+
statement), and an optional `kind` (`theorem` | `definition`, inferred from
|
|
34
|
+
the declaration head when omitted). Responses rank their evidence —
|
|
35
|
+
`counterexample` > `deterministic` > `two-judge-consensus` > `single-judge`
|
|
36
|
+
— and a single-judge flag is explicitly labeled as below the reporting bar.
|
|
37
|
+
|
|
38
|
+
## Install
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
pip install git+https://github.com/ibrahimmian36/leanscreen
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Requires Python ≥3.12. Runtime dependencies: `httpx`, `pydantic`,
|
|
45
|
+
`pydantic-settings`, `mcp` — nothing else.
|
|
46
|
+
|
|
47
|
+
## Lean setup (optional but recommended)
|
|
48
|
+
|
|
49
|
+
Without a Lean project, the server still runs — `check_fast` does lints +
|
|
50
|
+
vacuity and says plainly that elaboration was skipped. With one, statements
|
|
51
|
+
are elaborated for real:
|
|
52
|
+
|
|
53
|
+
1. A Lean 4 project with mathlib, built: `lake build` inside it.
|
|
54
|
+
2. The [community REPL](https://github.com/leanprover-community/repl),
|
|
55
|
+
built against the **same toolchain**: `lake build` inside the repl repo
|
|
56
|
+
gives you `.lake/build/bin/repl`.
|
|
57
|
+
3. `lake` on the server's PATH.
|
|
58
|
+
|
|
59
|
+
mathlib imports once at server startup (~100 seconds, in the background —
|
|
60
|
+
calls arriving mid-warm-up answer immediately with a "still warming" note),
|
|
61
|
+
then each check takes ~0.1s.
|
|
62
|
+
|
|
63
|
+
## Configuration
|
|
64
|
+
|
|
65
|
+
Environment variables (or a `.env` in the working directory), all
|
|
66
|
+
`LEANSCREEN_`-prefixed:
|
|
67
|
+
|
|
68
|
+
| Variable | Default | Meaning |
|
|
69
|
+
|---|---|---|
|
|
70
|
+
| `LEANSCREEN_LEAN_PROJECT_PATH` | unset | Lean 4 + mathlib project (elaboration off when unset) |
|
|
71
|
+
| `LEANSCREEN_LEAN_REPL_PATH` | unset | community REPL binary; without it every check pays a full `lake env lean` |
|
|
72
|
+
| `LEANSCREEN_LEAN_TIMEOUT_SECONDS` | `180` | per-statement Lean budget |
|
|
73
|
+
| `LEANSCREEN_ANTHROPIC_MODEL` | `claude-opus-4-8` | judge A + probe (the calibrated default) |
|
|
74
|
+
| `LEANSCREEN_JUDGE_B_MODEL` | `claude-fable-5` | checklist judge (calibrated default; locked-surface models get a 32k token budget automatically) |
|
|
75
|
+
| `LEANSCREEN_MAX_TOKENS` | `4096` | judge A response budget |
|
|
76
|
+
| `ANTHROPIC_API_KEY` | unset | needed for `check_deep` only |
|
|
77
|
+
|
|
78
|
+
Claude Code (`.mcp.json` in your project) or Claude Desktop
|
|
79
|
+
(`claude_desktop_config.json`):
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
{
|
|
83
|
+
"mcpServers": {
|
|
84
|
+
"lean-faithfulness-screen": {
|
|
85
|
+
"command": "leanscreen",
|
|
86
|
+
"env": {
|
|
87
|
+
"LEANSCREEN_LEAN_PROJECT_PATH": "/path/to/your/lean-mathlib-project",
|
|
88
|
+
"LEANSCREEN_LEAN_REPL_PATH": "/path/to/repl/.lake/build/bin/repl",
|
|
89
|
+
"ANTHROPIC_API_KEY": "sk-ant-…"
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
## What this does not guarantee
|
|
97
|
+
|
|
98
|
+
The judge configuration was calibrated 2026-07-15 against 886 frozen human
|
|
99
|
+
verdicts (595 faithful / 291 unfaithful) from a production research-math
|
|
100
|
+
corpus. Under strict two-judge consensus, human-rejected pairs still passed
|
|
101
|
+
17.0% (theorems) / 35.6% (definitions) of the time, and human-certified
|
|
102
|
+
pairs were flagged 15–18% of the time. Both judges are Anthropic-family
|
|
103
|
+
models, so correlated blind spots cannot be ruled out. The counterexample
|
|
104
|
+
probe confabulates: on one PutnamBench sample its counterexamples were wrong
|
|
105
|
+
4 times out of 5. **Treat every flag as a candidate for human confirmation
|
|
106
|
+
and every pass as "nothing found", never "faithful."**
|
|
107
|
+
|
|
108
|
+
Human certification — an expert reviewer confirming that the Lean means the
|
|
109
|
+
informal statement — is what this screen deliberately does not automate. We
|
|
110
|
+
offer it as a service: **contact ibrahimnmian@gmail.com**.
|
|
111
|
+
|
|
112
|
+
## License
|
|
113
|
+
|
|
114
|
+
[FSL-1.1-Apache-2.0](LICENSE) (the Functional Source License): free to
|
|
115
|
+
use, copy, modify, and redistribute — including internal commercial use,
|
|
116
|
+
non-commercial education and research, and professional services — but not
|
|
117
|
+
to offer as a competing commercial product or service. **Each version
|
|
118
|
+
automatically becomes Apache 2.0 two years after its release** (the same
|
|
119
|
+
license as mathlib). Not OSI-approved until the conversion — read it
|
|
120
|
+
before building on it commercially.
|
|
121
|
+
|
|
122
|
+
## Provenance
|
|
123
|
+
|
|
124
|
+
Extracted from Millennium Research's private formalization platform
|
|
125
|
+
(2026-07-28); the detector stack, judge prompts, and calibration figures are
|
|
126
|
+
the ones behind our benchmark audits — the miniF2F and ProofNet# filings are
|
|
127
|
+
public, and the PutnamBench, ProofNetVerif, and CLEVER audits have been
|
|
128
|
+
shared with their maintainers. The calibration *data* is not included.
|
|
129
|
+
|
|
130
|
+
Project page: [millenniumresearch.ai/leanscreen](https://millenniumresearch.ai/leanscreen)
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# Build prompt — extract the screen into `leanscreen`, ready for public release
|
|
2
|
+
|
|
3
|
+
> This is the brief the extraction was executed against (2026-07-28). It is
|
|
4
|
+
> written so a fresh Claude Code session could run it end to end; it also
|
|
5
|
+
> records every decision made, so future maintainers know what was deliberate.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
I want the MCP faithfulness screener extracted from the private
|
|
10
|
+
`arxiv-math-platform` repo into a standalone, publicly publishable package.
|
|
11
|
+
The private repo stays the product; the public package is the funnel.
|
|
12
|
+
|
|
13
|
+
## What the product is
|
|
14
|
+
|
|
15
|
+
`leanscreen`: one pip-installable package, one console script (`leanscreen`),
|
|
16
|
+
serving the two MCP tools that already exist — `check_fast` (free,
|
|
17
|
+
deterministic: lints + vacuity + Lean elaboration) and `check_deep` (paid:
|
|
18
|
+
dual-judge strict consensus + counterexample probe, on the user's own
|
|
19
|
+
`ANTHROPIC_API_KEY`). Local-first, single-user, no hosting, no auth, no
|
|
20
|
+
telemetry.
|
|
21
|
+
|
|
22
|
+
## What moves, what stays private
|
|
23
|
+
|
|
24
|
+
**Moves** (the screen subtree — it depends only on `httpx` at runtime):
|
|
25
|
+
`llm.py`, `backtranslate.py` (judge prompts), `lints.py`, `quality.py`,
|
|
26
|
+
`falsify.py`, `verify.py`, `lean_server.py`, `logging.py`, the single-pair
|
|
27
|
+
`score_pair` core of `score_external.py`, and the MCP server. Unit tests for
|
|
28
|
+
each module come along.
|
|
29
|
+
|
|
30
|
+
**Stays private** (the commercial harness and everything upstream of it):
|
|
31
|
+
the batch layer (`load_pairs` / `run` / `render_report` / budget rails /
|
|
32
|
+
crash-safe resume), the numeric cross-evaluation probe and its sandbox, the
|
|
33
|
+
whole data platform (ingestion, extraction, DB, review UI, export,
|
|
34
|
+
benchmark, grader), the 886-verdict calibration set, and every census
|
|
35
|
+
artifact. The public package quotes the calibration *numbers*; the data that
|
|
36
|
+
produced them is the moat and never ships.
|
|
37
|
+
|
|
38
|
+
## Decisions (made, revisit deliberately)
|
|
39
|
+
|
|
40
|
+
- **Name**: `leanscreen`; package `leanscreen`, script `leanscreen`, MCP
|
|
41
|
+
server name stays `lean-faithfulness-screen`.
|
|
42
|
+
- **License**: FSL-1.1-Apache-2.0 (revised from ELv2 on 2026-07-28 after
|
|
43
|
+
license research, on Ibby's request). Why FSL won: (1) its Competing Use
|
|
44
|
+
clause covers "any other product or service we offer using the Software" —
|
|
45
|
+
protecting the certification/screening *service*, where ELv2 only blocks
|
|
46
|
+
hosted versions of the software itself; (2) Permitted Purposes explicitly
|
|
47
|
+
allow internal use, non-commercial education, and research — the mathlib
|
|
48
|
+
audience; (3) each version converts to Apache 2.0 (mathlib's license)
|
|
49
|
+
after 2 years, the best community optics available, and a 2-year-old
|
|
50
|
+
screener without current calibration is not a competitive threat; (4) it
|
|
51
|
+
is simpler than BUSL, whose Additional Use Grant ambiguity is exactly what
|
|
52
|
+
Sentry created FSL to fix. (Not legal advice; have a lawyer confirm before
|
|
53
|
+
anything heavy rides on it.)
|
|
54
|
+
- **Env prefix**: `LEANSCREEN_` (`LEAN_PROJECT_PATH`, `LEAN_REPL_PATH`,
|
|
55
|
+
`ANTHROPIC_MODEL`, `JUDGE_B_MODEL`, `MAX_TOKENS`, …). A fresh, slim
|
|
56
|
+
`Settings` — none of the platform's 50 knobs.
|
|
57
|
+
- **Judge B is now a setting** (`LEANSCREEN_JUDGE_B_MODEL`, default
|
|
58
|
+
`claude-fable-5` = the calibrated configuration) instead of a hard-coded
|
|
59
|
+
constant; the locked-surface 32k max_tokens rule keys off the value.
|
|
60
|
+
- **Numeric probe dropped** from the public `score_pair` (the MCP tools never
|
|
61
|
+
used it; smaller surface, smaller audit).
|
|
62
|
+
- **`infer_kind` made public** (was `_infer_kind`).
|
|
63
|
+
- **Fork policy**: this is a fork-with-provenance, not a shared library. The
|
|
64
|
+
private repo does NOT depend on the public package. When a detector
|
|
65
|
+
improves in one repo, port it to the other deliberately; the commit
|
|
66
|
+
message must name the source commit.
|
|
67
|
+
|
|
68
|
+
## Non-negotiable constraints (carried over verbatim)
|
|
69
|
+
|
|
70
|
+
- The screen may only reject — never certify. `CALIBRATION_DISCLOSURE` ships
|
|
71
|
+
verbatim in every payload; both tool descriptions state passing is not
|
|
72
|
+
certification (test-pinned).
|
|
73
|
+
- Evidence tiers stay ranked, never flattened: counterexample >
|
|
74
|
+
deterministic > two-judge-consensus > single-judge (below the reporting
|
|
75
|
+
bar).
|
|
76
|
+
- Every flag is a candidate, not a verdict.
|
|
77
|
+
- No secrets, no internal paths, no customer or reviewer names anywhere in
|
|
78
|
+
the public tree — audit before every push.
|
|
79
|
+
|
|
80
|
+
## Quality gate
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
.venv/bin/ruff check . && .venv/bin/ruff format --check . && .venv/bin/mypy && .venv/bin/pytest
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Strict mypy, zero errors, same bar as the private repo.
|
|
87
|
+
|
|
88
|
+
## Acceptance criteria
|
|
89
|
+
|
|
90
|
+
1. `pip install -e .` in a fresh venv pulls only httpx, pydantic(-settings),
|
|
91
|
+
and mcp (plus their transitive deps) — none of the platform's heavy deps.
|
|
92
|
+
2. The full test suite passes; the MCP-server tests are the same ones that
|
|
93
|
+
pass in the private repo, re-pointed.
|
|
94
|
+
3. `leanscreen` starts over stdio with no Lean project and no API key and
|
|
95
|
+
answers `check_fast`, exactly as the private server does.
|
|
96
|
+
4. README works for a stranger: REPL build walkthrough, generic paths,
|
|
97
|
+
config snippets for Claude Code / Claude Desktop, the calibration table,
|
|
98
|
+
the ELv2 summary, and a contact for certification engagements.
|
|
99
|
+
5. A pre-publish audit pass over the whole tree: secret scan, internal-path
|
|
100
|
+
scan, license header presence, README claims match measured numbers.
|
|
101
|
+
|
|
102
|
+
## Steps reserved for Ibby (do not automate)
|
|
103
|
+
|
|
104
|
+
1. Confirm the license choice and the public contact address.
|
|
105
|
+
2. `gh repo create` (public) + first push — publishing is a send.
|
|
106
|
+
3. Any PyPI upload, registry listing, or announcement post (10am rule).
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
*Executed 2026-07-28. Origin: arxiv-math-platform @ 0f57167 (commits e39d2f9
|
|
111
|
+
+ 0f57167 are the MCP server and the proof-stub fix this package forked
|
|
112
|
+
from).*
|