demiurge-pyrrho 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +70 -0
- package/LICENSE +21 -0
- package/README.md +137 -0
- package/SKILL.md +76 -0
- package/bin/pyrrho.js +142 -0
- package/docs/HONEST-NUMBERS.md +33 -0
- package/lib/bias.js +122 -0
- package/lib/claims.js +45 -0
- package/lib/epoche.js +64 -0
- package/lib/sample.js +38 -0
- package/lib/siblings.js +56 -0
- package/lib/sqlrisk.js +56 -0
- package/package.json +12 -0
package/CLAUDE.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# PYRRHO companion (guided setup and ongoing help)
|
|
2
|
+
|
|
3
|
+
You are Claude, reading this inside the PYRRHO repo or skill folder. You are the setup companion and, after setup, the ongoing guide. One step at a time, plain language, never pressure an optional step.
|
|
4
|
+
|
|
5
|
+
## What PYRRHO is (tell them in one line)
|
|
6
|
+
|
|
7
|
+
Every AI finds you an insight. PYRRHO tells you which ones the data cannot support, and it refuses with arithmetic rather than opinion.
|
|
8
|
+
|
|
9
|
+
## When to reach for it (say this once)
|
|
10
|
+
|
|
11
|
+
- **PYRRHO** when a number is about to become a decision, or you need SQL that will not silently lie.
|
|
12
|
+
- **ZOILUS** when a finished analysis needs reviewing as an artifact against a quality bar. PYRRHO produces, ZOILUS judges.
|
|
13
|
+
- **ATHENA** when the question is what to DO about a finding. PYRRHO only reports what the numbers do and do not say.
|
|
14
|
+
|
|
15
|
+
If they ask for a data scientist, tell them plainly: PYRRHO is not one. No modelling, no pipelines. Its only product is refusal. That space is crowded and PYRRHO deliberately does not enter it.
|
|
16
|
+
|
|
17
|
+
## Setup, conversationally
|
|
18
|
+
|
|
19
|
+
1. **Install is one copy.** `SKILL.md`, `lib/`, `bin/` and `references/` into `~/.claude/skills/pyrrho/`. Show the command for their OS before running it.
|
|
20
|
+
|
|
21
|
+
PowerShell:
|
|
22
|
+
```powershell
|
|
23
|
+
Copy-Item -Recurse . "$env:USERPROFILE\.claude\skills\pyrrho"
|
|
24
|
+
```
|
|
25
|
+
bash:
|
|
26
|
+
```bash
|
|
27
|
+
cp -r . ~/.claude/skills/pyrrho
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
2. **Check it landed.** `node bin/pyrrho.js setup` prints a state-aware readout. It should list ten bias catalog entries and find `SKILL.md`.
|
|
31
|
+
|
|
32
|
+
3. **Try one thing.** Have them run the line that sells the product:
|
|
33
|
+
```powershell
|
|
34
|
+
node bin/pyrrho.js diff --p1 0.04 --n1 200 --p2 0.05 --n2 200
|
|
35
|
+
```
|
|
36
|
+
It will tell them the difference they were about to celebrate is not distinguishable from chance.
|
|
37
|
+
|
|
38
|
+
4. **Zero config.** No API key, no account, no service. The arithmetic and the linters run locally with no model call.
|
|
39
|
+
|
|
40
|
+
## The rule that matters most
|
|
41
|
+
|
|
42
|
+
**Get the context before the numbers.** Ask what a row represents, how it was collected, the sample size, and the time range. If the context is missing, ask for it. Do not analyse. PYRRHO screens the context you describe, not the data you hold, so a wrong description produces a wrong screen. This is its largest limit and it is structural, not a bug.
|
|
43
|
+
|
|
44
|
+
## The CLI helpers (all zero-LLM)
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
pyrrho claim "<claim>" [--n N] [--self-selected] [--experiment] [--p1 .04 --n1 200 --p2 .05 --n2 200]
|
|
48
|
+
pyrrho screen [--n N] [--survivors-only] [--groups-differ] [--comparisons N] [--outliers] [--days N]
|
|
49
|
+
pyrrho ci --p 0.04 --n 200 the confidence interval they forgot
|
|
50
|
+
pyrrho diff --p1 .. --n1 .. --p2 .. --n2 .. is that difference noise?
|
|
51
|
+
pyrrho n-for --moe 0.01 how many observations they actually need
|
|
52
|
+
pyrrho sql <file> lint a query for silent lies and scans
|
|
53
|
+
pyrrho catalog | pyrrho setup
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Ongoing companion duties (never retire)
|
|
57
|
+
|
|
58
|
+
- When they paste a claim, run it through the gate before agreeing with it. That is the whole point of having installed this.
|
|
59
|
+
- When PYRRHO says SUSPEND, always read them the rule-out path. A refusal without a way forward is sulking, and the tool is built not to do that.
|
|
60
|
+
- If the same refusal keeps firing (they keep making causal claims on observational data), offer CHIRON: a refusal you keep hitting should become a permanent rule.
|
|
61
|
+
- Recommend a sibling only when it is missing and relevant: ZOILUS (review the finished analysis), CHIRON (refusal becomes rule), HORKOS (receipts that the analysis was actually run).
|
|
62
|
+
|
|
63
|
+
## Laws you must not break
|
|
64
|
+
|
|
65
|
+
1. The default verdict is SUSPEND. A claim earns SUPPORTED and never assumes it.
|
|
66
|
+
2. Every refusal is arithmetic or a named bias. Never a vibe.
|
|
67
|
+
3. Name what would rule it out, every time.
|
|
68
|
+
4. Data is observational unless they say they ran an experiment.
|
|
69
|
+
5. Fewer than five in a cell means report the counts, not a percentage.
|
|
70
|
+
6. No em dashes in anything you write for this product. Plain language. Never fake completeness.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Lee Jun Ying
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+
|
|
5
|
+
# PYRRHO
|
|
6
|
+
|
|
7
|
+
*Every AI finds you an insight. PYRRHO tells you which ones the data cannot support.*
|
|
8
|
+
|
|
9
|
+
*Suspends judgment where the evidence stops.*
|
|
10
|
+
|
|
11
|
+
[](LICENSE)
|
|
12
|
+
[](package.json)
|
|
13
|
+
[](benchmarks/run.js)
|
|
14
|
+
|
|
15
|
+
</div>
|
|
16
|
+
|
|
17
|
+
**Every AI finds you an insight. PYRRHO tells you which ones the data cannot support.**
|
|
18
|
+
|
|
19
|
+
> I am Pyrrho of Elis. I taught *epoche*: suspend judgment where the evidence stops. I am not remembered for what I concluded. That is the point.
|
|
20
|
+
|
|
21
|
+
Point any model at a spreadsheet and it hands you confident bullet points. It finds a trend in twelve rows. It calls a correlation a cause. I do the other job. I say exactly what the numbers carry, then I name, precisely, everything they do not.
|
|
22
|
+
|
|
23
|
+
## Before and after
|
|
24
|
+
|
|
25
|
+
**Your AI, handed a dashboard:**
|
|
26
|
+
|
|
27
|
+
> Conversion is up 25 percent since the redesign. The redesign worked. Roll it out everywhere.
|
|
28
|
+
|
|
29
|
+
**PYRRHO, handed the same numbers:**
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
$ pyrrho claim "The redesign caused conversion to rise" --n 12 --self-selected \
|
|
33
|
+
--p1 0.04 --n1 200 --p2 0.05 --n2 200
|
|
34
|
+
|
|
35
|
+
verdict: SUSPEND (the data does not carry this claim)
|
|
36
|
+
claim: "The redesign caused conversion to rise"
|
|
37
|
+
|
|
38
|
+
DO NOT CONCLUDE:
|
|
39
|
+
[blocking] The claim asserts cause, but the data is observational. Correlation is not causation.
|
|
40
|
+
rule it out: Run a controlled experiment, or name and measure the confounder.
|
|
41
|
+
[blocking] n = 12. Below roughly 30 observations the estimate is dominated by noise.
|
|
42
|
+
rule it out: Collect more observations, or report the confidence interval.
|
|
43
|
+
[blocking] The sample selected itself, so it does not represent the population you want to speak about.
|
|
44
|
+
rule it out: Sample at random from the target population, or restate the claim as being about volunteers.
|
|
45
|
+
[blocking] The observed difference of 1.00 percentage points sits inside the noise band of
|
|
46
|
+
plus or minus 3.86 points. It is not distinguishable from chance.
|
|
47
|
+
rule it out: Collect enough observations to shrink the interval, or report no difference.
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
That "25 percent" was one point of movement inside a four point noise band, on twelve self-selected users, described with a verb that claims causation. Four independent reasons the claim is worthless. Nobody else was going to tell you.
|
|
51
|
+
|
|
52
|
+
## The five laws
|
|
53
|
+
|
|
54
|
+
1. **Suspend where the evidence stops.** The default verdict is SUSPEND. A claim earns SUPPORTED; it is never assumed.
|
|
55
|
+
2. **Every refusal is arithmetic or a named bias.** No vibes. "That difference sits inside a plus or minus 2.8 point band" beats "I'm not sure about that."
|
|
56
|
+
3. **Name what would rule it out.** Every refusal carries the specific test that would settle it. Refusing without a path forward is sulking.
|
|
57
|
+
4. **Data is observational until proven otherwise.** You did not run an experiment unless you say you did.
|
|
58
|
+
5. **Report the counts when the rate is theatre.** Fewer than five in either cell, and a percentage is a lie with a decimal point.
|
|
59
|
+
|
|
60
|
+
And the oldest smell in the trade: **a result that comes back too clean is itself a finding.** A perfect trend, an all-green benchmark, a distribution with no outliers: each is suspect until you can explain why it is real. Clean earns scrutiny before it earns belief.
|
|
61
|
+
|
|
62
|
+
## The EPOCHE engine
|
|
63
|
+
|
|
64
|
+
Three deterministic checks, composed:
|
|
65
|
+
|
|
66
|
+
- **The sentence.** Causal language on observational data. Absolute language (always, never, proves). Extrapolation with no stated model.
|
|
67
|
+
- **The context.** A ten entry bias catalog that screens what you described: small sample, rate not estimable, correlation as causation, confounding, survivorship, selection, multiple comparisons, outlier as signal, data quality, window shorter than the cycle. Each entry detects itself and knows what would rule it out.
|
|
68
|
+
- **The arithmetic.** Confidence intervals, whether a claimed difference is distinguishable from noise, and how many observations you would actually need.
|
|
69
|
+
|
|
70
|
+
Any blocking finding, and the verdict is SUSPEND. That is a return value, not a suggestion.
|
|
71
|
+
|
|
72
|
+
## It also writes SQL
|
|
73
|
+
|
|
74
|
+
PYRRHO writes production SQL, states its ASSUMPTIONS first (NULLs, duplicates, ties, empty groups, time zones, and the grain of one output row, which is where most silently wrong queries go wrong), then walks you through it. The linter then catches: `UPDATE` or `DELETE` with no `WHERE`, implicit cross joins, the `NOT IN` null trap, fan-out double counting from an aggregate over a one-to-many join, unbounded windows with no `PARTITION BY`, non-sargable predicates, leading wildcards, unbounded reads. A blocking finding means the query does not ship.
|
|
75
|
+
|
|
76
|
+
## Install for your agent
|
|
77
|
+
|
|
78
|
+
> **From npm:** `npm install -g demiurge-pyrrho`, then `pyrrho setup` — or `npx demiurge-pyrrho claim "<claim>"` with no install. (Skill copy uses the source clone below.)
|
|
79
|
+
|
|
80
|
+
PowerShell:
|
|
81
|
+
```powershell
|
|
82
|
+
git clone https://github.com/eragonlonelyboy-lab/pyrrho; cd pyrrho; node bin/pyrrho.js setup
|
|
83
|
+
```
|
|
84
|
+
bash:
|
|
85
|
+
```bash
|
|
86
|
+
git clone https://github.com/eragonlonelyboy-lab/pyrrho && cd pyrrho && node bin/pyrrho.js setup
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Copy the folder into `~/.claude/skills/pyrrho/` and invoke with `/pyrrho`. Zero config. The CLI is the deterministic half:
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
pyrrho claim "<claim>" [--n N] [--self-selected] [--experiment]
|
|
93
|
+
pyrrho screen | pyrrho ci --p 0.04 --n 200 | pyrrho diff --p1 .04 --n1 200 --p2 .05 --n2 200
|
|
94
|
+
pyrrho n-for --moe 0.01 | pyrrho sql <file> | pyrrho catalog | pyrrho setup
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Not for you if
|
|
98
|
+
|
|
99
|
+
- You want a data scientist. PYRRHO does no modelling, no feature engineering, no pipelines. That space is commoditised and crowded. Its only product is refusal.
|
|
100
|
+
- You want an insight machine. There are hundreds. This is the thing that stands behind them and says no.
|
|
101
|
+
- You want to be told your number is good.
|
|
102
|
+
|
|
103
|
+
## FAQ
|
|
104
|
+
|
|
105
|
+
**"Isn't this just a statistics library?"**
|
|
106
|
+
A statistics library computes what you ask. I refuse what you were about to say. The arithmetic is the smaller half.
|
|
107
|
+
|
|
108
|
+
**"Ten biases is not all the biases."**
|
|
109
|
+
It is not. It is ten that detect themselves from a context you can actually describe. See [HONEST-NUMBERS](docs/HONEST-NUMBERS.md), where I say exactly what I cannot see.
|
|
110
|
+
|
|
111
|
+
**"What if I really did run an experiment?"**
|
|
112
|
+
Then pass `--experiment` and I will stop calling your causal claim an overreach. I default to observational because you did not run one.
|
|
113
|
+
|
|
114
|
+
**"Will it just refuse everything?"**
|
|
115
|
+
`4% vs 8% on n=5000` returns SUPPORTED, blocking count zero. The bar is arithmetic, not mood.
|
|
116
|
+
|
|
117
|
+
## The family
|
|
118
|
+
|
|
119
|
+
**PYRRHO** produces the analysis. **ZOILUS** reviews it as an artifact. **ATHENA** decides what to do about it. I only report what the numbers do and do not say.
|
|
120
|
+
|
|
121
|
+
**HORKOS** proves you ran it. **MONETA** budgets it. **CHIRON** turns a refusal you keep hitting into a permanent rule.
|
|
122
|
+
|
|
123
|
+
The working standard the whole house runs on is public too: [ARETE](https://github.com/eragonlonelyboy-lab/arete), five discipline gates any model can run. PYRRHO is its treat-good-news-as-suspect rule, with arithmetic behind it.
|
|
124
|
+
|
|
125
|
+
## Verify me
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
node benchmarks/run.js # 34/34
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
The tests prove the arithmetic and the linters. They cannot prove that any given reading of data is wise. I say so in [HONEST-NUMBERS](docs/HONEST-NUMBERS.md).
|
|
132
|
+
|
|
133
|
+
If PYRRHO stops you from shipping one wrong number, star it. If it does not, do not.
|
|
134
|
+
|
|
135
|
+
MIT. Copyright (c) 2026 Lee Jun Ying. Built by Eragon Lee.
|
|
136
|
+
|
|
137
|
+
*Named for Pyrrho of Elis, who suspended judgment where the evidence stopped.*
|
package/SKILL.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: pyrrho
|
|
3
|
+
description: The skeptic. Use PYRRHO when a claim is being drawn from data, or when you need SQL you can trust. It states exactly what the numbers support, names every conclusion they do NOT support and the bias that would produce each one, and suspends judgment where the evidence stops. Trigger on "what does this data say", "is this significant", "write me a query", "/pyrrho", or any moment a number is about to become a decision. Not for deciding what to DO about a finding (that is ATHENA), and not for judging a finished analysis as an artifact (that is ZOILUS).
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# PYRRHO: the skeptic
|
|
7
|
+
|
|
8
|
+
> Pyrrho of Elis taught *epoche*: suspend judgment where the evidence is insufficient. He is not remembered for what he concluded.
|
|
9
|
+
|
|
10
|
+
**Every AI finds you an insight. PYRRHO tells you which ones the data cannot support.**
|
|
11
|
+
|
|
12
|
+
Point any model at a spreadsheet and it will hand you confident bullet points. It will find a trend in twelve rows. It will call a correlation a cause. PYRRHO does the opposite job: it says exactly what the numbers carry, then names, precisely, everything they do not.
|
|
13
|
+
|
|
14
|
+
## When to use / when not
|
|
15
|
+
|
|
16
|
+
- **Use PYRRHO** when a number is about to become a decision, when a claim rests on data, or when you need a query that will not silently lie.
|
|
17
|
+
- **Not for deciding what to do** about a finding. That is **ATHENA** (wisdom of a decision).
|
|
18
|
+
- **Not for judging a finished analysis** as an artifact against a quality bar. That is **ZOILUS** (quality of execution). PYRRHO *produces* the analysis; ZOILUS *reviews* it.
|
|
19
|
+
- **Not a data scientist.** No modelling, no feature engineering, no pipelines. That space is commoditised. PYRRHO's only product is refusal.
|
|
20
|
+
|
|
21
|
+
## The five laws (non-negotiable)
|
|
22
|
+
|
|
23
|
+
**Cross-model agreement is not evidence.** Two models can share training priors, prompts and blind spots. For any claim supported mainly by reviewer consensus, state the independence class and demand an observation, deterministic test or external source. A suspiciously clean unanimous review triggers an extra check rather than higher confidence.
|
|
24
|
+
|
|
25
|
+
1. **Suspend where the evidence stops.** The default is SUSPEND. A claim earns SUPPORTED; it is never assumed.
|
|
26
|
+
2. **Every refusal is arithmetic or a named bias.** No vibes. "That difference sits inside a plus-or-minus 2.8 point noise band" beats "I'm not sure about that."
|
|
27
|
+
3. **Name what would rule it out.** Every DO NOT CONCLUDE carries the specific test that would settle it. Refusing without a path forward is sulking.
|
|
28
|
+
4. **Data is observational until proven otherwise.** You did not run an experiment unless you say you did. Causal language on observational data is a blocking finding.
|
|
29
|
+
5. **Report the counts when the rate is theatre.** Fewer than five in either cell, and a percentage is a lie with a decimal point.
|
|
30
|
+
|
|
31
|
+
## The EPOCHE engine
|
|
32
|
+
|
|
33
|
+
EPOCHE (Greek *epokhe*, the suspension of judgment) is the gate. Given a claim and the context the data came from, it composes three deterministic checks and returns SUPPORTED or SUSPEND with the full DO NOT CONCLUDE list:
|
|
34
|
+
|
|
35
|
+
1. **The sentence** (`lib/claims.js`), causal language on observational data, absolute language (always, never, proves), extrapolation with no stated forecasting model.
|
|
36
|
+
2. **The context** (`lib/bias.js`), the catalog screens what you described: small sample, rate not estimable, correlation-as-causation, confounding, survivorship, selection, multiple comparisons, outlier-as-signal, data quality, window shorter than the cycle. Each entry detects itself and knows what would rule it out.
|
|
37
|
+
3. **The arithmetic** (`lib/sample.js`), confidence intervals, whether a claimed difference is distinguishable from noise, and how many observations you would actually need.
|
|
38
|
+
|
|
39
|
+
Any blocking finding, and the verdict is SUSPEND. This is not a suggestion; it is a return value.
|
|
40
|
+
|
|
41
|
+
## The workflow
|
|
42
|
+
|
|
43
|
+
1. **Get the context before the numbers.** Ask what a row represents, the **units**, how it was collected, the sample size, and the time range. Also ask **the decision this analysis serves**: an analysis with no decision behind it is trivia, and the decision determines which finding matters. If the context is missing, ask; do not analyse.
|
|
44
|
+
2. **Screen the context** (`pyrrho screen`) before you look at a single finding. Data-quality problems (nulls, duplicates, impossible values) are screened here, before anything is treated as signal. A quality problem can create the whole effect.
|
|
45
|
+
3. **State the top 3 findings** as specific claims, each with the numbers that support it, ordered by importance, not by where they appear.
|
|
46
|
+
4. **For each finding, give the likely explanation** (what plausibly drives it) **and the alternative explanation** you would rule out or test.
|
|
47
|
+
5. **Run the gate** on each claim (`pyrrho claim "..."`). Report the DO NOT CONCLUDE list in full, each entry with what would rule it out.
|
|
48
|
+
6. **The one chart.** Name the single visualisation that best carries the key finding, what goes on each axis, and why that chart over the obvious alternative.
|
|
49
|
+
7. **The next question.** The one follow-up worth investigating and the specific data that would answer it.
|
|
50
|
+
8. **If the data cannot support a real conclusion, say so plainly.** Do not force one. This is the whole product.
|
|
51
|
+
|
|
52
|
+
## SQL
|
|
53
|
+
|
|
54
|
+
PYRRHO writes production SQL that has to be correct the first time. Correctness first, then performance, then elegance.
|
|
55
|
+
|
|
56
|
+
**Before writing, ask** for the dialect (PostgreSQL, BigQuery, MySQL, Snowflake), the schema with types and keys, the approximate data size, and anything known about indexes or partitions. If the request is ambiguous in a way that changes the result, ask rather than guess: the grain of the output, which date column, how an event is defined, the time zone.
|
|
57
|
+
|
|
58
|
+
**Deliver in this order:**
|
|
59
|
+
|
|
60
|
+
1. **ASSUMPTIONS.** How you read the schema, and how you handle NULLs, duplicates, ties, empty groups, and time zones. **State the grain of one output row.** This section is not optional: most silently wrong queries are wrong here, not in the syntax.
|
|
61
|
+
2. **THE QUERY**, in the requested dialect. Meaningful aliases, explicit `JOIN` syntax, qualified columns, CTEs instead of nested subqueries where it aids readability, short comments on non-obvious logic. Never `SELECT *`; name the columns you need.
|
|
62
|
+
3. **HOW IT WORKS.** Two to four sentences walking through the query, so the reader understands it rather than pasting it.
|
|
63
|
+
4. **PERFORMANCE AND CORRECTNESS RISKS.** Name what hurts at scale and give a concrete fix or an index to add: a full table scan, a missing index, an exploding join, an unbounded window, a non-sargable filter. Then name any way the query could silently return a wrong answer (fan-out from a one-to-many join, double counting) and how you prevented it.
|
|
64
|
+
5. **A SET-BASED REWRITE.** If a set-based rewrite is meaningfully faster than the obvious approach, show it. Correctness first, then performance, then elegance.
|
|
65
|
+
|
|
66
|
+
Then `lib/sqlrisk.js` lints it deterministically. Blocking: unbounded writes (`UPDATE`/`DELETE` with no `WHERE`), implicit cross joins. Flagged: the `NOT IN` null trap (a silent wrong answer), **fan-out double counting from an aggregate over a one-to-many join**, **unbounded windows** (no `PARTITION BY`), non-sargable predicates, leading wildcards, unbounded reads. A blocking finding means the query does not ship.
|
|
67
|
+
|
|
68
|
+
## Boundaries (do not blur the pantheon)
|
|
69
|
+
|
|
70
|
+
- **ZOILUS** reviews a finished analysis against a quality bar. PYRRHO produces it. Same producer/judge split as PEITHO and ZOILUS.
|
|
71
|
+
- **ATHENA** decides what to do about a finding. PYRRHO only reports what the numbers do and do not say.
|
|
72
|
+
- **HORKOS** proves the analysis was actually run. **MONETA** budgets. **CHIRON** turns a repeated PYRRHO refusal into a permanent rule.
|
|
73
|
+
- **ARGUS** (parked) would audit automated scorers. That is a different question: PYRRHO audits *analysis integrity*, not *scorer integrity*.
|
|
74
|
+
|
|
75
|
+
## Sibling recommendation
|
|
76
|
+
On setup, detect installed siblings and recommend only the missing natural pairs: ZOILUS (to review the finished analysis), CHIRON (a refusal you keep hitting becomes a rule), HORKOS (receipts).
|
package/bin/pyrrho.js
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
// pyrrho, the skeptic. Deterministic CLI half (zero-LLM): the EPOCHE gate,
|
|
4
|
+
// sample arithmetic, the bias catalog, the SQL risk linter. The reading of the
|
|
5
|
+
// data is skill-driven (SKILL.md); this is the machinery that refuses.
|
|
6
|
+
const fs = require('fs');
|
|
7
|
+
const { suspend } = require('../lib/epoche');
|
|
8
|
+
const { ciHalfWidth, proportionsDiffer, minimumSampleFor } = require('../lib/sample');
|
|
9
|
+
const { screen, ids } = require('../lib/bias');
|
|
10
|
+
const { lintSql } = require('../lib/sqlrisk');
|
|
11
|
+
const siblings = require('../lib/siblings');
|
|
12
|
+
const pkg = require('../package.json');
|
|
13
|
+
|
|
14
|
+
const args = process.argv.slice(2);
|
|
15
|
+
const cmd = args[0];
|
|
16
|
+
const out = (s) => process.stdout.write(s + '\n');
|
|
17
|
+
const num = (name) => { const i = args.indexOf(name); return i >= 0 ? Number(args[i + 1]) : undefined; };
|
|
18
|
+
const str = (name) => { const i = args.indexOf(name); return i >= 0 ? args[i + 1] : undefined; };
|
|
19
|
+
|
|
20
|
+
function printSuspension(r) {
|
|
21
|
+
out(`verdict: ${r.verdict}${r.verdict === 'SUSPEND' ? ' (the data does not carry this claim)' : ''}`);
|
|
22
|
+
if (r.claim) out(`claim: "${r.claim}"`);
|
|
23
|
+
if (!r.doNotConclude.length) { out('\n(nothing to suspend: the evidence carries it)'); return; }
|
|
24
|
+
out('\nDO NOT CONCLUDE:');
|
|
25
|
+
for (const d of r.doNotConclude) {
|
|
26
|
+
out(` [${d.severity}] ${d.summary}`);
|
|
27
|
+
out(` rule it out: ${d.ruleOut}`);
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
switch (cmd) {
|
|
32
|
+
case '--version': case '-v': out(`pyrrho ${pkg.version}`); break;
|
|
33
|
+
|
|
34
|
+
case 'ci': {
|
|
35
|
+
const p = num('--p'), n = num('--n');
|
|
36
|
+
if (p == null || n == null) { out('usage: pyrrho ci --p 0.04 --n 200'); process.exit(1); }
|
|
37
|
+
const h = ciHalfWidth(p, n);
|
|
38
|
+
out(`${(p * 100).toFixed(2)}% on n=${n} -> 95% CI: ${((p - h) * 100).toFixed(2)}% to ${((p + h) * 100).toFixed(2)}% (plus or minus ${(h * 100).toFixed(2)} points)`);
|
|
39
|
+
break;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
case 'diff': {
|
|
43
|
+
const p1 = num('--p1'), n1 = num('--n1'), p2 = num('--p2'), n2 = num('--n2');
|
|
44
|
+
if ([p1, n1, p2, n2].some((x) => x == null)) { out('usage: pyrrho diff --p1 0.04 --n1 200 --p2 0.05 --n2 200'); process.exit(1); }
|
|
45
|
+
const r = proportionsDiffer(p1, n1, p2, n2);
|
|
46
|
+
out(`difference: ${(r.diff * 100).toFixed(2)} points | noise band: plus or minus ${(r.ciHalf * 100).toFixed(2)} points | z = ${r.z.toFixed(2)}`);
|
|
47
|
+
out(r.significant ? 'DISTINGUISHABLE from chance.' : 'NOT distinguishable from chance. Report no difference.');
|
|
48
|
+
break;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
case 'n-for': {
|
|
52
|
+
const p = num('--p') ?? 0.5, moe = num('--moe');
|
|
53
|
+
if (moe == null) { out('usage: pyrrho n-for --moe 0.01 [--p 0.5]'); process.exit(1); }
|
|
54
|
+
out(`need n = ${minimumSampleFor(p, moe)} to pin ${(p * 100).toFixed(0)}% to plus or minus ${(moe * 100).toFixed(2)} points`);
|
|
55
|
+
break;
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
case 'screen': {
|
|
59
|
+
// pyrrho screen --n 12 --self-selected --comparisons 8
|
|
60
|
+
const ctx = {
|
|
61
|
+
n: num('--n'), successes: num('--successes'), comparisons: num('--comparisons'), days: num('--days'),
|
|
62
|
+
selfSelected: args.includes('--self-selected'),
|
|
63
|
+
survivorsOnly: args.includes('--survivors-only'),
|
|
64
|
+
groupsDifferBaseline: args.includes('--groups-differ'),
|
|
65
|
+
outliersPresent: args.includes('--outliers'),
|
|
66
|
+
outliersHandled: args.includes('--outliers-handled'),
|
|
67
|
+
observational: !args.includes('--experiment'),
|
|
68
|
+
modelAgreement: args.includes('--model-agreement'),
|
|
69
|
+
independentProof: args.includes('--independent-proof'),
|
|
70
|
+
reviewers: num('--reviewers'),
|
|
71
|
+
identicalReasoning: args.includes('--identical-reasoning'),
|
|
72
|
+
};
|
|
73
|
+
const hits = screen(ctx);
|
|
74
|
+
if (!hits.length) { out('no bias triggered by the context you described.'); break; }
|
|
75
|
+
for (const h of hits) { out(`[${h.severity}] ${h.name}: ${h.summary}`); out(` rule it out: ${h.ruleOut}`); }
|
|
76
|
+
break;
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
case 'catalog': out('bias catalog: ' + ids().join(', ')); break;
|
|
80
|
+
|
|
81
|
+
case 'claim': {
|
|
82
|
+
// pyrrho claim "X causes Y" --n 12 --self-selected
|
|
83
|
+
const claim = args[1];
|
|
84
|
+
if (!claim || claim.startsWith('--')) { out('usage: pyrrho claim "<the claim>" [--n N] [--experiment] [--self-selected] ...'); process.exit(1); }
|
|
85
|
+
const ctx = {
|
|
86
|
+
n: num('--n'), successes: num('--successes'), comparisons: num('--comparisons'), days: num('--days'),
|
|
87
|
+
selfSelected: args.includes('--self-selected'), survivorsOnly: args.includes('--survivors-only'),
|
|
88
|
+
groupsDifferBaseline: args.includes('--groups-differ'), outliersPresent: args.includes('--outliers'),
|
|
89
|
+
observational: !args.includes('--experiment'),
|
|
90
|
+
modelAgreement: args.includes('--model-agreement'), independentProof: args.includes('--independent-proof'),
|
|
91
|
+
reviewers: num('--reviewers'), identicalReasoning: args.includes('--identical-reasoning'),
|
|
92
|
+
};
|
|
93
|
+
const comparison = num('--p1') != null ? { p1: num('--p1'), n1: num('--n1'), p2: num('--p2'), n2: num('--n2') } : undefined;
|
|
94
|
+
printSuspension(suspend({ claim, ctx, comparison }));
|
|
95
|
+
process.exit(0);
|
|
96
|
+
break;
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
case 'sql': {
|
|
100
|
+
const file = args[1];
|
|
101
|
+
const sql = file && fs.existsSync(file) ? fs.readFileSync(file, 'utf8') : str('--q');
|
|
102
|
+
if (!sql) { out('usage: pyrrho sql <file.sql> | pyrrho sql --q "SELECT * FROM t"'); process.exit(1); }
|
|
103
|
+
const r = lintSql(sql);
|
|
104
|
+
if (!r.findings.length) { out('OK: no risk patterns found.'); break; }
|
|
105
|
+
for (const f of r.findings) { out(`[${f.severity}] ${f.summary}`); out(` fix: ${f.fix}`); }
|
|
106
|
+
process.exit(r.ok ? 0 : 1);
|
|
107
|
+
break;
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
case 'siblings': {
|
|
111
|
+
const s = siblings.detect();
|
|
112
|
+
out('installed: ' + (s.installed.join(', ') || 'none'));
|
|
113
|
+
const rec = siblings.recommend(s.installed);
|
|
114
|
+
out('recommended (missing): ' + (rec.map((r) => r.name).join(', ') || 'none, all set'));
|
|
115
|
+
for (const r of rec) out(` - ${r.name}: ${r.why}`);
|
|
116
|
+
break;
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
case 'setup': {
|
|
120
|
+
out('PYRRHO setup check');
|
|
121
|
+
out('------------------');
|
|
122
|
+
out(` bias catalog entries : ${ids().length}`);
|
|
123
|
+
out(` SKILL.md : ${fs.existsSync(__dirname + '/../SKILL.md') ? 'yes' : 'MISSING'}`);
|
|
124
|
+
out('');
|
|
125
|
+
out('PYRRHO is skill-first: invoke it in Claude Code with /pyrrho or "have PYRRHO check this claim".');
|
|
126
|
+
out('This CLI is the deterministic half: EPOCHE gate, sample arithmetic, bias catalog, SQL risk linter.');
|
|
127
|
+
out('It produces the analysis. ZOILUS judges it. ATHENA decides what to do about it.');
|
|
128
|
+
break;
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
default:
|
|
132
|
+
out(`pyrrho ${pkg.version}, suspends judgment where the evidence stops`);
|
|
133
|
+
out('');
|
|
134
|
+
out(' pyrrho claim "<claim>" [ctx flags] run the EPOCHE gate on a claim');
|
|
135
|
+
out(' pyrrho screen [ctx flags] screen a context against the bias catalog');
|
|
136
|
+
out(' pyrrho ci --p 0.04 --n 200 the confidence interval you forgot');
|
|
137
|
+
out(' pyrrho diff --p1 .04 --n1 200 --p2 .05 --n2 200 is that difference noise?');
|
|
138
|
+
out(' pyrrho n-for --moe 0.01 how many observations you actually need');
|
|
139
|
+
out(' pyrrho sql <file> lint a query for silent lies and scans');
|
|
140
|
+
out(' pyrrho siblings detect installed Demiurge gods');
|
|
141
|
+
out(' pyrrho catalog | setup');
|
|
142
|
+
}
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# HONEST-NUMBERS
|
|
2
|
+
|
|
3
|
+
Every claim PYRRHO makes about itself, and every one it cannot make. A product built to refuse unsupported conclusions does not get to make them about itself.
|
|
4
|
+
|
|
5
|
+
## What is checkable
|
|
6
|
+
|
|
7
|
+
`node benchmarks/run.js` runs 34 deterministic tests, no model involved. They prove:
|
|
8
|
+
|
|
9
|
+
- **The arithmetic is right.** 4 percent against 5 percent on n=200 each is not distinguishable from noise. 4 percent against 8 percent on n=5000 each is. `minimumSampleFor(0.5, 0.05)` returns 385, the familiar number.
|
|
10
|
+
- **The sentence linter fires correctly.** Causal language on observational data is blocking. The same claim marked `--experiment` is not flagged. A careful descriptive claim passes with zero findings.
|
|
11
|
+
- **The bias catalog detects and does not manufacture.** A clean context (n=5000, experimental) triggers nothing. Every triggered entry carries a rule-out path.
|
|
12
|
+
- **EPOCHE composes correctly.** It suspends a causal claim on a tiny self-selected sample, suspends a difference inside the noise band, supports a careful claim from a clean experiment, and does not double-report causation as both a lint and a bias.
|
|
13
|
+
- **The SQL linter catches real traps.** `DELETE` with no `WHERE`, implicit cross joins, the `NOT IN` null trap, fan-out double counting, unbounded windows, non-sargable predicates. A careful query passes clean, and a trap pattern inside a string literal does not false-positive.
|
|
14
|
+
|
|
15
|
+
That is the whole receipt. It is a receipt about machinery.
|
|
16
|
+
|
|
17
|
+
## What is NOT checkable
|
|
18
|
+
|
|
19
|
+
**No test can prove a reading of data is wise.** The tests prove the gate fires when it should. They say nothing about whether the analysis you built on the other side was any good. Judging that is ZOILUS's job, not a benchmark's.
|
|
20
|
+
|
|
21
|
+
**PYRRHO screens the context you describe, not the data you have.** If you say the sample was random and it was not, the selection bias never fires. Garbage context in, wrong screen out. It cannot see a confounder you did not mention. This is the largest limit, and it is structural: PYRRHO reads your description of the world, not the world.
|
|
22
|
+
|
|
23
|
+
**Ten biases is not all the biases.** The catalog holds ten that can detect themselves from a context a human can actually state. Simpson's paradox, regression to the mean, base rate neglect, and a long tail of others are not in it. Absence from the catalog is not absolution.
|
|
24
|
+
|
|
25
|
+
**The arithmetic uses the normal approximation.** Which is why PYRRHO refuses when a cell holds fewer than five. Outside that guard it is a Wald interval, adequate for the decisions this tool exists to interrupt, not adequate for a paper.
|
|
26
|
+
|
|
27
|
+
**Reject-on-doubt means friction by design.** PYRRHO will sometimes suspend a claim that was, in fact, true. That is the trade. A tool that never falsely suspends is a tool that never catches anything, and the cost of a wrong number that shipped is higher than the cost of collecting more data.
|
|
28
|
+
|
|
29
|
+
**It is not a data scientist.** No modelling, no feature engineering, no pipelines, no charts it drew itself. If you want an AI data science team, several exist and they are better at that than PYRRHO will ever be. PYRRHO's only product is refusal.
|
|
30
|
+
|
|
31
|
+
## What it costs
|
|
32
|
+
|
|
33
|
+
Nothing. It is a skill and a zero-LLM CLI. The arithmetic and the linters run locally with no model call. The reading of the data is done by whatever agent invoked the skill, and costs whatever that agent costs.
|
package/lib/bias.js
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
// bias.js, the catalog. Each entry knows how to detect itself from the
|
|
3
|
+
// context you describe, and what would rule it out. No entry is a vibe.
|
|
4
|
+
|
|
5
|
+
const CATALOG = [
|
|
6
|
+
{
|
|
7
|
+
id: 'small-sample',
|
|
8
|
+
name: 'Sample too small',
|
|
9
|
+
detect: (c) => c.n != null && c.n < 30,
|
|
10
|
+
summary: (c) => `n = ${c.n}. Below roughly 30 observations the estimate is dominated by noise.`,
|
|
11
|
+
ruleOut: 'Collect more observations, or report the confidence interval instead of a point estimate.',
|
|
12
|
+
severity: 'blocking',
|
|
13
|
+
},
|
|
14
|
+
{
|
|
15
|
+
id: 'rate-not-estimable',
|
|
16
|
+
name: 'Rate not estimable',
|
|
17
|
+
detect: (c) => c.successes != null && c.n != null && (c.successes < 5 || (c.n - c.successes) < 5),
|
|
18
|
+
summary: (c) => `${c.successes} of ${c.n}. With fewer than 5 in either cell, a percentage is theatre.`,
|
|
19
|
+
ruleOut: 'Report the raw counts, not a rate.',
|
|
20
|
+
severity: 'blocking',
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
id: 'correlation-causation',
|
|
24
|
+
name: 'Correlation treated as causation',
|
|
25
|
+
detect: (c) => c.observational !== false && c.causalClaim === true,
|
|
26
|
+
summary: () => 'A causal claim drawn from observational data.',
|
|
27
|
+
ruleOut: 'Run an experiment, or name the confounder and measure it.',
|
|
28
|
+
severity: 'blocking',
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
id: 'confounding',
|
|
32
|
+
name: 'Confounding variable',
|
|
33
|
+
detect: (c) => c.observational !== false && c.groupsDifferBaseline === true,
|
|
34
|
+
summary: () => 'The compared groups differ at baseline, so the outcome difference may be the baseline difference.',
|
|
35
|
+
ruleOut: 'Stratify or match on the baseline variable, then re-compare.',
|
|
36
|
+
severity: 'blocking',
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
id: 'survivorship',
|
|
40
|
+
name: 'Survivorship bias',
|
|
41
|
+
detect: (c) => c.survivorsOnly === true,
|
|
42
|
+
summary: () => 'Only the survivors are in the data. The ones that failed are invisible and would change the answer.',
|
|
43
|
+
ruleOut: 'Find the dropped population and include it, or restate the claim as being about survivors only.',
|
|
44
|
+
severity: 'blocking',
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
id: 'selection',
|
|
48
|
+
name: 'Selection bias',
|
|
49
|
+
detect: (c) => c.selfSelected === true,
|
|
50
|
+
summary: () => 'The sample selected itself, so it does not represent the population you want to speak about.',
|
|
51
|
+
ruleOut: 'Sample at random from the target population, or restate the claim as being about volunteers.',
|
|
52
|
+
severity: 'blocking',
|
|
53
|
+
},
|
|
54
|
+
{
|
|
55
|
+
id: 'multiple-comparisons',
|
|
56
|
+
name: 'Multiple comparisons',
|
|
57
|
+
detect: (c) => c.comparisons != null && c.comparisons > 1,
|
|
58
|
+
summary: (c) => `${c.comparisons} comparisons were run. At the usual 5% threshold you expect about ${(0.05 * c.comparisons).toFixed(2)} false positives by chance alone.`,
|
|
59
|
+
ruleOut: 'Correct the threshold (Bonferroni: divide by the number of comparisons), or pre-register the one comparison that mattered.',
|
|
60
|
+
severity: 'major',
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
id: 'outlier-as-signal',
|
|
64
|
+
name: 'Outlier treated as signal',
|
|
65
|
+
detect: (c) => c.outliersPresent === true && c.outliersHandled !== true,
|
|
66
|
+
summary: () => 'Outliers are present and unhandled. A single extreme value can create the whole effect.',
|
|
67
|
+
ruleOut: 'Re-run the analysis with the outliers excluded and report whether the finding survives.',
|
|
68
|
+
severity: 'major',
|
|
69
|
+
},
|
|
70
|
+
{
|
|
71
|
+
id: 'data-quality',
|
|
72
|
+
name: 'Data quality not screened',
|
|
73
|
+
// Distinct from outlier-as-signal: an outlier is a real value at the tail, a
|
|
74
|
+
// quality problem is a value that should not exist. Nulls silently drop rows
|
|
75
|
+
// from a join, duplicates double a count, an impossible value skews a mean.
|
|
76
|
+
detect: (c) => c.nullsPresent === true || c.duplicatesPossible === true || c.impossibleValues === true || c.dataQualityChecked === false,
|
|
77
|
+
summary: () => 'Nulls, duplicate rows, or impossible values are present or unchecked. A quality problem can create the entire effect before any analysis begins.',
|
|
78
|
+
ruleOut: 'Count the nulls per column, dedupe on the key, and check the value ranges. Re-run and report whether the finding survives.',
|
|
79
|
+
severity: 'blocking',
|
|
80
|
+
},
|
|
81
|
+
{
|
|
82
|
+
id: 'consensus-as-evidence',
|
|
83
|
+
name: 'Agreement substituted for evidence',
|
|
84
|
+
detect: (c) => c.modelAgreement === true && c.independentProof !== true,
|
|
85
|
+
summary: () => 'Multiple models agreed, but no independent observation or runnable proof supports the claim.',
|
|
86
|
+
ruleOut: 'Run an independent proof against the artifact or data. Agreement changes confidence only after evidence exists.',
|
|
87
|
+
severity: 'blocking',
|
|
88
|
+
},
|
|
89
|
+
{
|
|
90
|
+
id: 'too-clean-convergence',
|
|
91
|
+
name: 'Suspiciously clean convergence',
|
|
92
|
+
detect: (c) => Number(c.reviewers || 0) >= 2 && c.identicalReasoning === true,
|
|
93
|
+
summary: (c) => `${c.reviewers} reviewers returned materially identical reasoning, so their independence is unproven.`,
|
|
94
|
+
ruleOut: 'Blind a fresh reviewer to prior verdicts and require a different evidence path or discriminating test.',
|
|
95
|
+
severity: 'major',
|
|
96
|
+
},
|
|
97
|
+
{
|
|
98
|
+
id: 'short-window',
|
|
99
|
+
name: 'Window too short for the cycle',
|
|
100
|
+
detect: (c) => c.days != null && c.days < 14,
|
|
101
|
+
summary: (c) => `The observation window is ${c.days} days, shorter than a fortnight, so weekly seasonality is not separable from trend.`,
|
|
102
|
+
ruleOut: 'Extend the window past at least two full cycles, or compare like-for-like weekdays.',
|
|
103
|
+
severity: 'major',
|
|
104
|
+
},
|
|
105
|
+
];
|
|
106
|
+
|
|
107
|
+
// Screen a described context against the whole catalog.
|
|
108
|
+
function screen(ctx = {}) {
|
|
109
|
+
return CATALOG.filter((b) => {
|
|
110
|
+
try { return !!b.detect(ctx); } catch { return false; }
|
|
111
|
+
}).map((b) => ({
|
|
112
|
+
id: b.id,
|
|
113
|
+
name: b.name,
|
|
114
|
+
severity: b.severity,
|
|
115
|
+
summary: typeof b.summary === 'function' ? b.summary(ctx) : b.summary,
|
|
116
|
+
ruleOut: b.ruleOut,
|
|
117
|
+
}));
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
function ids() { return CATALOG.map((b) => b.id); }
|
|
121
|
+
|
|
122
|
+
module.exports = { screen, ids, CATALOG };
|
package/lib/claims.js
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
// claims.js, the language linter.
|
|
3
|
+
// Data is observational until proven otherwise. A claim written in causal
|
|
4
|
+
// language on top of observational data is an overreach, no matter how
|
|
5
|
+
// confident the prose sounds. This catches the overreach in the sentence.
|
|
6
|
+
|
|
7
|
+
const CAUSAL = /\b(causes?|caused|causing|drives?|driving|drove|leads? to|led to|results? in|resulted in|because of|due to|thanks to|makes? (?:us|them|it|people)|responsible for)\b/i;
|
|
8
|
+
const ABSOLUTE = /\b(always|never|all|none|every|proves?|proven|guarantees?|guaranteed|definitely|certainly|clearly shows)\b/i;
|
|
9
|
+
const EXTRAPOLATION = /\b(will|going to|expect(?:ed)? to|predicts?|forecasts?|projects? to|by next (?:year|quarter|month)|on track to)\b/i;
|
|
10
|
+
|
|
11
|
+
// A claim + the context it was drawn from. observational defaults to true:
|
|
12
|
+
// you did not run an experiment unless you say you did.
|
|
13
|
+
function lintClaim(claim, ctx = {}) {
|
|
14
|
+
const text = String(claim || '');
|
|
15
|
+
const observational = ctx.observational !== false; // default true
|
|
16
|
+
const findings = [];
|
|
17
|
+
|
|
18
|
+
if (observational && CAUSAL.test(text)) {
|
|
19
|
+
findings.push({
|
|
20
|
+
severity: 'blocking',
|
|
21
|
+
id: 'causal-language-on-observational-data',
|
|
22
|
+
summary: 'The claim asserts cause, but the data is observational. Correlation is not causation.',
|
|
23
|
+
ruleOut: 'Run a controlled experiment, or name and measure the confounder that would produce this pattern without the causal link.',
|
|
24
|
+
});
|
|
25
|
+
}
|
|
26
|
+
if (ABSOLUTE.test(text)) {
|
|
27
|
+
findings.push({
|
|
28
|
+
severity: 'blocking',
|
|
29
|
+
id: 'absolute-language',
|
|
30
|
+
summary: 'The claim uses absolute language (always, never, proves, guarantees) that no sample can support.',
|
|
31
|
+
ruleOut: 'Restate as a bounded estimate with its confidence interval.',
|
|
32
|
+
});
|
|
33
|
+
}
|
|
34
|
+
if (EXTRAPOLATION.test(text) && !ctx.forecastModel) {
|
|
35
|
+
findings.push({
|
|
36
|
+
severity: 'major',
|
|
37
|
+
id: 'unbacked-extrapolation',
|
|
38
|
+
summary: 'The claim projects forward without a stated forecasting model.',
|
|
39
|
+
ruleOut: 'Name the model and its assumptions, or restate the claim in the past tense about what was observed.',
|
|
40
|
+
});
|
|
41
|
+
}
|
|
42
|
+
return findings;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
module.exports = { lintClaim, CAUSAL, ABSOLUTE, EXTRAPOLATION };
|
package/lib/epoche.js
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
// epoche.js, the EPOCHE engine (Greek epokhe, the suspension of judgment).
|
|
3
|
+
// Pyrrho's whole discipline in one function: given a claim and the context the
|
|
4
|
+
// data came from, decide whether the evidence carries the claim. When it does
|
|
5
|
+
// not, we do not argue and we do not soften. We suspend, and we say exactly why.
|
|
6
|
+
const { lintClaim, CAUSAL } = require('./claims');
|
|
7
|
+
const { screen } = require('./bias');
|
|
8
|
+
const { proportionsDiffer, rateIsEstimable } = require('./sample');
|
|
9
|
+
|
|
10
|
+
// ctx fields (all optional, all honest): n, successes, observational, groupsDifferBaseline,
|
|
11
|
+
// survivorsOnly, selfSelected, comparisons, outliersPresent, outliersHandled, days, forecastModel
|
|
12
|
+
// comparison (optional): { p1, n1, p2, n2 } to test a claimed difference against noise
|
|
13
|
+
function suspend({ claim, ctx = {}, comparison } = {}) {
|
|
14
|
+
const doNotConclude = [];
|
|
15
|
+
|
|
16
|
+
// 1. the sentence itself
|
|
17
|
+
const causalClaim = CAUSAL.test(String(claim || ''));
|
|
18
|
+
for (const f of lintClaim(claim, ctx)) {
|
|
19
|
+
doNotConclude.push({ severity: f.severity, id: f.id, summary: f.summary, ruleOut: f.ruleOut });
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
// 2. the context it came from (feed the causal reading back into the catalog)
|
|
23
|
+
for (const b of screen({ ...ctx, causalClaim })) {
|
|
24
|
+
if (doNotConclude.some((d) => d.id === b.id || (d.id === 'causal-language-on-observational-data' && b.id === 'correlation-causation'))) continue;
|
|
25
|
+
doNotConclude.push({ severity: b.severity, id: b.id, summary: b.summary, ruleOut: b.ruleOut });
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
// 3. the arithmetic, if a difference is being claimed
|
|
29
|
+
let noise = null;
|
|
30
|
+
if (comparison && comparison.n1 != null && comparison.n2 != null) {
|
|
31
|
+
const r = proportionsDiffer(comparison.p1, comparison.n1, comparison.p2, comparison.n2);
|
|
32
|
+
noise = r;
|
|
33
|
+
if (!r.significant) {
|
|
34
|
+
doNotConclude.push({
|
|
35
|
+
severity: 'blocking',
|
|
36
|
+
id: 'difference-inside-noise',
|
|
37
|
+
summary: `The observed difference of ${(r.diff * 100).toFixed(2)} percentage points sits inside the noise band of plus or minus ${(r.ciHalf * 100).toFixed(2)} points. It is not distinguishable from chance.`,
|
|
38
|
+
ruleOut: `Collect enough observations to shrink the interval, or report no difference.`,
|
|
39
|
+
});
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
if (ctx.successes != null && ctx.n != null && !rateIsEstimable(ctx.successes, ctx.n)) {
|
|
43
|
+
if (!doNotConclude.some((d) => d.id === 'rate-not-estimable')) {
|
|
44
|
+
doNotConclude.push({
|
|
45
|
+
severity: 'blocking',
|
|
46
|
+
id: 'rate-not-estimable',
|
|
47
|
+
summary: `${ctx.successes} of ${ctx.n}. With fewer than 5 in either cell, a percentage is theatre.`,
|
|
48
|
+
ruleOut: 'Report the raw counts, not a rate.',
|
|
49
|
+
});
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
const blocking = doNotConclude.filter((d) => d.severity === 'blocking');
|
|
54
|
+
const verdict = blocking.length ? 'SUSPEND' : 'SUPPORTED';
|
|
55
|
+
return {
|
|
56
|
+
verdict, // SUSPEND = the data does not carry this claim
|
|
57
|
+
claim: String(claim || ''),
|
|
58
|
+
blocking: blocking.length,
|
|
59
|
+
doNotConclude, // every reason, each with what would rule it out
|
|
60
|
+
noise, // the arithmetic, when a difference was tested
|
|
61
|
+
};
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
module.exports = { suspend };
|
package/lib/sample.js
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
// sample.js, the arithmetic that refuses.
|
|
3
|
+
// Most bad data claims die here, not in argument. If the difference you are
|
|
4
|
+
// excited about sits inside the noise band, no amount of narrative saves it.
|
|
5
|
+
|
|
6
|
+
const Z95 = 1.96;
|
|
7
|
+
|
|
8
|
+
// 95% confidence half-width for a proportion (Wald). Returns Infinity for n<=0.
|
|
9
|
+
function ciHalfWidth(p, n, z = Z95) {
|
|
10
|
+
if (!(n > 0)) return Infinity;
|
|
11
|
+
if (p < 0 || p > 1) throw new Error('p must be a proportion between 0 and 1');
|
|
12
|
+
return z * Math.sqrt((p * (1 - p)) / n);
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
// Can two observed proportions be distinguished from noise?
|
|
16
|
+
// Returns the difference, its standard error, the z score, and the verdict.
|
|
17
|
+
function proportionsDiffer(p1, n1, p2, n2, z = Z95) {
|
|
18
|
+
if (!(n1 > 0) || !(n2 > 0)) return { diff: Math.abs(p1 - p2), se: Infinity, z: 0, significant: false, ciHalf: Infinity };
|
|
19
|
+
const se = Math.sqrt((p1 * (1 - p1)) / n1 + (p2 * (1 - p2)) / n2);
|
|
20
|
+
const diff = Math.abs(p1 - p2);
|
|
21
|
+
const zscore = se > 0 ? diff / se : 0;
|
|
22
|
+
return { diff, se, z: zscore, significant: se > 0 && zscore >= z, ciHalf: z * se };
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
// How many observations to pin a proportion to +/- marginOfError?
|
|
26
|
+
function minimumSampleFor(p, marginOfError, z = Z95) {
|
|
27
|
+
if (!(marginOfError > 0)) throw new Error('marginOfError must be > 0');
|
|
28
|
+
return Math.ceil((z * z * p * (1 - p)) / (marginOfError * marginOfError));
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
// A rate observed as k successes in n trials: is the rate even estimable?
|
|
32
|
+
// Rule of thumb: fewer than 5 successes AND fewer than 5 failures makes the
|
|
33
|
+
// normal approximation meaningless. Say so instead of quoting a percentage.
|
|
34
|
+
function rateIsEstimable(k, n) {
|
|
35
|
+
return n > 0 && k >= 5 && (n - k) >= 5;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
module.exports = { ciHalfWidth, proportionsDiffer, minimumSampleFor, rateIsEstimable, Z95 };
|
package/lib/siblings.js
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
// siblings.js: Demiurge sibling detection (PYR-11).
|
|
3
|
+
// PYRRHO owns what the numbers say. It defers everything else, and it recommends
|
|
4
|
+
// only the missing natural pairs rather than nagging about the whole pantheon.
|
|
5
|
+
const fs = require('fs');
|
|
6
|
+
const path = require('path');
|
|
7
|
+
const os = require('os');
|
|
8
|
+
|
|
9
|
+
const SIBLINGS = {
|
|
10
|
+
zoilus: 'reviews the finished analysis as an artifact (PYRRHO produces, ZOILUS judges)',
|
|
11
|
+
chiron: 'turns a refusal you keep hitting into a permanent rule',
|
|
12
|
+
horkos: 'proves the analysis was actually run (receipts)',
|
|
13
|
+
athena: 'decides what to DO about a finding (PYRRHO only reports what the numbers say)',
|
|
14
|
+
peitho: 'sells the result, and must pass PYRRHO before any numeric claim ships',
|
|
15
|
+
veritas: 'prose hygiene on the write-up',
|
|
16
|
+
moneta: 'budgets the loop',
|
|
17
|
+
calliope: 'designs the one chart',
|
|
18
|
+
hypnos: 'memory consolidation',
|
|
19
|
+
maat: 'the dashboard',
|
|
20
|
+
};
|
|
21
|
+
|
|
22
|
+
// The pairs PYRRHO actually needs. Everything else is noise at setup time.
|
|
23
|
+
const NATURAL_PAIRS = ['zoilus', 'chiron', 'horkos'];
|
|
24
|
+
|
|
25
|
+
function defaultSearchPaths() {
|
|
26
|
+
return [
|
|
27
|
+
path.join(os.homedir(), '.claude', 'skills'),
|
|
28
|
+
path.join(__dirname, '..', '..'), // sibling product dirs (…/products/*)
|
|
29
|
+
];
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
function detect(searchPaths) {
|
|
33
|
+
const paths = searchPaths || defaultSearchPaths();
|
|
34
|
+
const found = new Set();
|
|
35
|
+
for (const base of paths) {
|
|
36
|
+
if (!base || !fs.existsSync(base)) continue;
|
|
37
|
+
let entries = [];
|
|
38
|
+
try { entries = fs.readdirSync(base); } catch { continue; }
|
|
39
|
+
for (const name of entries) {
|
|
40
|
+
const key = name.toLowerCase();
|
|
41
|
+
if (SIBLINGS[key] && key !== 'pyrrho') {
|
|
42
|
+
try { if (fs.statSync(path.join(base, name)).isDirectory()) found.add(key); } catch { /* skip */ }
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
const installed = Object.keys(SIBLINGS).filter((k) => found.has(k));
|
|
47
|
+
const missing = Object.keys(SIBLINGS).filter((k) => !found.has(k));
|
|
48
|
+
return { installed, missing };
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
function recommend(installed) {
|
|
52
|
+
const have = new Set(installed || detect().installed);
|
|
53
|
+
return NATURAL_PAIRS.filter((k) => !have.has(k)).map((k) => ({ name: k, why: SIBLINGS[k] }));
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
module.exports = { detect, recommend, SIBLINGS, NATURAL_PAIRS };
|
package/lib/sqlrisk.js
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
// sqlrisk.js, the query linter.
|
|
3
|
+
// PYRRHO writes SQL from plain English (that part is the model's job). What is
|
|
4
|
+
// deterministic is refusing to hand you a query that will quietly lie, wreck a
|
|
5
|
+
// table, or crawl. Every finding names the risk and the fix.
|
|
6
|
+
|
|
7
|
+
function stripped(sql) {
|
|
8
|
+
// remove string literals and comments so patterns do not match inside them
|
|
9
|
+
return String(sql || '')
|
|
10
|
+
.replace(/--[^\n]*/g, ' ')
|
|
11
|
+
.replace(/\/\*[\s\S]*?\*\//g, ' ')
|
|
12
|
+
.replace(/'(?:[^']|'')*'/g, "''")
|
|
13
|
+
.replace(/\s+/g, ' ');
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
function lintSql(sql) {
|
|
17
|
+
const s = stripped(sql);
|
|
18
|
+
const findings = [];
|
|
19
|
+
const has = (re) => re.test(s);
|
|
20
|
+
|
|
21
|
+
if (/\b(UPDATE|DELETE)\b/i.test(s) && !/\bWHERE\b/i.test(s)) {
|
|
22
|
+
findings.push({ severity: 'blocking', id: 'unbounded-write', summary: 'UPDATE or DELETE with no WHERE clause. This rewrites the whole table.', fix: 'Add a WHERE clause, and run it as a SELECT first to see what it would touch.' });
|
|
23
|
+
}
|
|
24
|
+
if (/\bSELECT\s+\*/i.test(s)) {
|
|
25
|
+
findings.push({ severity: 'major', id: 'select-star', summary: 'SELECT * pulls every column, including ones you do not need and ones that may be added later.', fix: 'Name the columns you actually use.' });
|
|
26
|
+
}
|
|
27
|
+
if (/\bFROM\s+\w+\s*,\s*\w+/i.test(s) && !/\bWHERE\b/i.test(s)) {
|
|
28
|
+
findings.push({ severity: 'blocking', id: 'implicit-cross-join', summary: 'Two tables in FROM with no join predicate. This is a cross join and will multiply the rows.', fix: 'Use an explicit JOIN with an ON clause.' });
|
|
29
|
+
}
|
|
30
|
+
if (/\bWHERE\b[^;]*\b(LOWER|UPPER|DATE|CAST|COALESCE|SUBSTR|SUBSTRING)\s*\(\s*\w+\s*\)/i.test(s)) {
|
|
31
|
+
findings.push({ severity: 'major', id: 'non-sargable-predicate', summary: 'A function is applied to a column inside WHERE, so an index on that column cannot be used and the query degrades to a full table scan.', fix: 'Move the function to the compared value, or add a matching expression index. A missing index here is the difference between a seek and a scan.' });
|
|
32
|
+
}
|
|
33
|
+
if (/\bLIKE\s*'%/i.test(String(sql || ''))) {
|
|
34
|
+
findings.push({ severity: 'minor', id: 'leading-wildcard', summary: "A LIKE pattern beginning with % cannot use an index and forces a scan.", fix: 'Anchor the pattern, or use a full-text index.' });
|
|
35
|
+
}
|
|
36
|
+
if (has(/\bSELECT\b/i) && !has(/\bLIMIT\b|\bTOP\b|\bFETCH FIRST\b/i) && !has(/\b(COUNT|SUM|AVG|MIN|MAX)\s*\(/i) && !has(/\bGROUP BY\b/i)) {
|
|
37
|
+
findings.push({ severity: 'minor', id: 'unbounded-read', summary: 'An exploratory SELECT with no LIMIT can return the whole table (a full table scan when no index covers the predicate).', fix: 'Add a LIMIT while exploring.' });
|
|
38
|
+
}
|
|
39
|
+
if (/\bNOT\s+IN\s*\(\s*SELECT\b/i.test(s)) {
|
|
40
|
+
findings.push({ severity: 'major', id: 'not-in-null-trap', summary: 'NOT IN with a subquery returns no rows if the subquery yields a single NULL. This is a silent wrong answer.', fix: 'Use NOT EXISTS, or filter NULLs inside the subquery.' });
|
|
41
|
+
}
|
|
42
|
+
// Fan-out: aggregating across a one-to-many join double counts the "one" side.
|
|
43
|
+
// A heuristic, so it flags rather than blocks: we cannot see the cardinality.
|
|
44
|
+
if (/\bJOIN\b/i.test(s) && /\b(SUM|COUNT|AVG)\s*\(/i.test(s) && !/\bCOUNT\s*\(\s*DISTINCT\b/i.test(s)) {
|
|
45
|
+
findings.push({ severity: 'major', id: 'fan-out-double-count', summary: 'An aggregate over a JOIN (an exploding join). If the join is one-to-many, every row on the "one" side is counted once per match, silently inflating the total.', fix: 'Pre-aggregate the many side in a CTE before joining, or aggregate over a DISTINCT key. Check the join cardinality before trusting the number.' });
|
|
46
|
+
}
|
|
47
|
+
// A window with no PARTITION BY scans and ranks the entire result set.
|
|
48
|
+
if (/\bOVER\s*\(/i.test(s) && !/\bOVER\s*\([^)]*\bPARTITION\s+BY\b/i.test(s)) {
|
|
49
|
+
findings.push({ severity: 'major', id: 'unbounded-window', summary: 'A window function with no PARTITION BY runs over the whole result set, which does not scale and is often not the grain you meant.', fix: 'Add PARTITION BY the grain you actually want, or state plainly that the whole set is intended.' });
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const blocking = findings.filter((f) => f.severity === 'blocking').length;
|
|
53
|
+
return { ok: blocking === 0, blocking, findings };
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
module.exports = { lintSql, stripped };
|
package/package.json
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "demiurge-pyrrho",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Every AI finds you an insight. PYRRHO tells you which ones the data cannot support.",
|
|
5
|
+
"bin": { "pyrrho": "bin/pyrrho.js" },
|
|
6
|
+
"scripts": { "test": "node benchmarks/run.js" },
|
|
7
|
+
"keywords": ["data", "statistics", "skepticism", "bias", "sql", "analysis", "claude", "agent-skill", "demiurge"],
|
|
8
|
+
"author": "Eragon Lee",
|
|
9
|
+
"license": "MIT",
|
|
10
|
+
"repository": { "type": "git", "url": "https://github.com/eragonlonelyboy-lab/pyrrho" },
|
|
11
|
+
"files": ["bin", "lib", "references", "docs", "SKILL.md", "CLAUDE.md", "LICENSE", "README.md"]
|
|
12
|
+
}
|