undetermined 0.1.0__tar.gz

undetermined-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Seth Wheeler
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

undetermined-0.1.0/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: undetermined
+Version: 0.1.0
+Summary: Point it at a program, get back what can and cannot be determined: a constant with an error bar, and an explicit undetermined list with reasons.
+License: MIT
+Keywords: empirical,constant,measurement,plateau,algorithm-analysis,benchmark,estimation,reproducibility,uncertainty,minimum-detectable-effect,zero-dependency
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# undetermined
+
+**Point it at a program; get back what can and cannot be determined about it.**
+
+Plenty of libraries fit a curve to measurements and hand you back a number. This one
+hands back a number *with the error bar it was decided by*, plus an explicit
+`undetermined` list with a reason on each entry — and it will put an observable on that
+list rather than fit a plateau to a drift.
+
+```
+heads: 1.9978 +/- 0.0032   4 rungs from truth=8 agree within 2.0 sigma
+flat:  UNDETERMINED        no run of 3 rungs agrees; the constant is still moving
+                           at the top of the ladder
+```
+
+The second line is the point. A tool that always produced the first line would be
+useless and would still pass every test that checks it produces one.
+
+Ships as `undetermined` on **PyPI** and on **npm**, from one tree, at one version. No
+dependencies in either half.
+
+---
+
+## Install
+
+```bash
+pip install undetermined
+```
+
+```bash
+npm install undetermined
+```
+
+## Use
+
+You supply an **adapter**: an object that knows how to run the thing you are measuring
+at a controllable input size, and nothing else. This library never knows what program it
+is looking at.
+
+```python
+import random
+from undetermined import characterize
+
+class Coin:
+    def truths(self):
+        # The controllable input values, known by construction. A ladder, not a point:
+        # a constant that is only measured at one size cannot be shown to have settled.
+        return [8, 32, 128, 512]
+
+    @property
+    def observables(self):
+        def heads(truth, seed):
+            r = random.Random(seed)
+            return sum(1 for _ in range(truth) if r.random() < 0.5)
+
+        def flat(truth, seed):
+            r = random.Random(seed)
+            return sum(1 for _ in range(12) if r.random() < 0.5) + 0.5
+
+        return {"heads": heads, "flat": flat}
+
+report = characterize(Coin(), trials=2500)
+
+report["per_observable"]["heads"]["constant"]      # 1.9978...
+report["per_observable"]["heads"]["constant_se"]   # 0.0032...
+report["undetermined"]                             # ['flat']
+report["notes"]                                    # why each one is on the list
+```
+
+```js
+import { characterize } from "undetermined";
+
+const report = characterize(adapter, { trials: 2500 });
+report.per_observable.heads.constant;   // 1.9978...
+report.undetermined;                    // ['flat']
+```
+
+### The adapter protocol
+
+| member | required | meaning |
+| --- | --- | --- |
+| `truths()` | yes | the controllable input values, known by construction |
+| `observables` | yes | `{name: (truth, seed) -> number}`, at least two |
+| `instances()` | no | sibling instances of the same family |
+| `knobs` | no | `{name: [values]}` — parameters of the program itself |
+| `perturbed(knob, v)` | with `knobs` | the observables with that knob set to that value |
+
+At least two observables, always. With one there is no choice to make, so the library
+cannot be shown to make one — it raises rather than reporting a confident single answer.
+
+### Deriving the trial count instead of guessing it
+
+`characterize` takes a `trials` count. If you would rather state the precision you need
+and have the budget derived:
+
+```python
+from undetermined import to_tolerance
+report = to_tolerance(MyAdapter(), tolerance=0.01)   # 1% on the constant
+```
+
+It doubles the trial count until every determined constant is inside the tolerance or the
+cap is reached, and reports which ones never got there. `tolerance=0` raises: a tolerance
+is a decision about your problem, and this library will not choose it for you.
+
+---
+
+## What it refuses, and why
+
+Three refusals, and each is the answer to a way of being confidently wrong.
+
+**An observable that ignores its seed** raises immediately. `fit` averages an observable
+over `trials` *different* seeds, so an observable that reads the clock or an unseeded RNG
+produces a mean over noise. A mean over noise still has a standard error, still forms a
+ladder, and can still plateau — every guard downstream compares against that error, so a
+broken adapter does not produce a wrong-looking answer, it produces a **confident** one.
+Each observable is called twice with the same `(truth, seed)` at the first and last rung;
+a disagreement is a wiring error, not a finding, so it throws.
+
+**A constant that never settles** is reported as `UNDETERMINED` with the reason. A run of
+three consecutive rungs must agree within two combined standard errors before any
+plateau is reported; the value is then the inverse-variance-weighted mean over that run,
+and the report says which rung it started from.
+
+**A choice between observables that is not earned** is reported as `UNDETERMINED`. With
+`instances()`, an observable is only called informative if its constant varies at least
+3x its own measurement error across instances, and it is only *chosen* over the runner-up
+if it beats it by 3x. Two observables that both vary a lot are not a choice.
+
+### The rule underneath all of it
+
+> Compare against the **noise**, never against the **size**.
+
+A spread only means something in units of the error on the thing that spread. Dividing by
+the magnitude instead is how a large number gets mistaken for a real one, and it is the
+single mistake this library is shaped around not making.
+
+### The thresholds
+
+They are the contract, and `python/tests/test_parity.py` asserts both halves hold the
+same ones and produce the same output on the same numbers — including the same
+explanatory strings.
+
+| constant | value | what it gates |
+| --- | --- | --- |
+| `PLATEAU_K` | 2.0 | sigma within which rungs must agree |
+| `PLATEAU_RUN` | 3 | consecutive agreeing rungs required |
+| `MIN_RATIO` | 3.0 | error-multiples an observable must vary by to be informative |
+| `MIN_MARGIN` | 3.0 | factor by which the winner must beat the runner-up |
+| `SIGMAS` | 3.0 | standard errors in a minimum detectable effect |
+| `FLOOR_TRIALS` / `CAP_TRIALS` | 400 / 40000 | the budget search bounds |
+
+---
+
+## This is a library, not a CLI
+
+An adapter is a code object with closures in it. There is nothing to pass on a command
+line that would not amount to naming a Python or JavaScript symbol and importing it, so
+the package ships an import and no console script.
+
+## Where it sits
+
+**Layer 0**: no dependencies inside or outside this network of packages, by design.
+
+The `nondet` edge was considered and **rejected**. `nondet` addresses a function as
+`FILE::NAME` so it can re-run it in fresh processes; this library's observables are
+closures inside an adapter object and have no such address, so `nondet` cannot probe
+them. Wiring it in would have meant either a fake file path or a check that never ran —
+a dependency that looks like a guarantee and is not. The reproducibility precondition is
+implemented natively in both halves instead, and it is checked in the same call that
+would have needed the guarantee. `nondet` remains the right tool for the *functions your
+adapter calls*, which do have addresses; running it on those is a good idea and is not
+something this package can do on your behalf.
+
+## Development
+
+```bash
+python3 -m unittest discover -s python/tests -v   # PYTHONPATH=python
+node --test js/test/*.test.js
+```
+
+The parity suite skips when `node` is not on PATH, so a Python-only contributor can still
+run everything else. CI asserts it was **not** skipped — a skipped test and a passing one
+look identical in a tally.
+
+## License
+
+MIT

undetermined-0.1.0/README.md

@@ -0,0 +1,187 @@
+# undetermined
+
+**Point it at a program; get back what can and cannot be determined about it.**
+
+Plenty of libraries fit a curve to measurements and hand you back a number. This one
+hands back a number *with the error bar it was decided by*, plus an explicit
+`undetermined` list with a reason on each entry — and it will put an observable on that
+list rather than fit a plateau to a drift.
+
+```
+heads: 1.9978 +/- 0.0032   4 rungs from truth=8 agree within 2.0 sigma
+flat:  UNDETERMINED        no run of 3 rungs agrees; the constant is still moving
+                           at the top of the ladder
+```
+
+The second line is the point. A tool that always produced the first line would be
+useless and would still pass every test that checks it produces one.
+
+Ships as `undetermined` on **PyPI** and on **npm**, from one tree, at one version. No
+dependencies in either half.
+
+---
+
+## Install
+
+```bash
+pip install undetermined
+```
+
+```bash
+npm install undetermined
+```
+
+## Use
+
+You supply an **adapter**: an object that knows how to run the thing you are measuring
+at a controllable input size, and nothing else. This library never knows what program it
+is looking at.
+
+```python
+import random
+from undetermined import characterize
+
+class Coin:
+    def truths(self):
+        # The controllable input values, known by construction. A ladder, not a point:
+        # a constant that is only measured at one size cannot be shown to have settled.
+        return [8, 32, 128, 512]
+
+    @property
+    def observables(self):
+        def heads(truth, seed):
+            r = random.Random(seed)
+            return sum(1 for _ in range(truth) if r.random() < 0.5)
+
+        def flat(truth, seed):
+            r = random.Random(seed)
+            return sum(1 for _ in range(12) if r.random() < 0.5) + 0.5
+
+        return {"heads": heads, "flat": flat}
+
+report = characterize(Coin(), trials=2500)
+
+report["per_observable"]["heads"]["constant"]      # 1.9978...
+report["per_observable"]["heads"]["constant_se"]   # 0.0032...
+report["undetermined"]                             # ['flat']
+report["notes"]                                    # why each one is on the list
+```
+
+```js
+import { characterize } from "undetermined";
+
+const report = characterize(adapter, { trials: 2500 });
+report.per_observable.heads.constant;   // 1.9978...
+report.undetermined;                    // ['flat']
+```
+
+### The adapter protocol
+
+| member | required | meaning |
+| --- | --- | --- |
+| `truths()` | yes | the controllable input values, known by construction |
+| `observables` | yes | `{name: (truth, seed) -> number}`, at least two |
+| `instances()` | no | sibling instances of the same family |
+| `knobs` | no | `{name: [values]}` — parameters of the program itself |
+| `perturbed(knob, v)` | with `knobs` | the observables with that knob set to that value |
+
+At least two observables, always. With one there is no choice to make, so the library
+cannot be shown to make one — it raises rather than reporting a confident single answer.
+
+### Deriving the trial count instead of guessing it
+
+`characterize` takes a `trials` count. If you would rather state the precision you need
+and have the budget derived:
+
+```python
+from undetermined import to_tolerance
+report = to_tolerance(MyAdapter(), tolerance=0.01)   # 1% on the constant
+```
+
+It doubles the trial count until every determined constant is inside the tolerance or the
+cap is reached, and reports which ones never got there. `tolerance=0` raises: a tolerance
+is a decision about your problem, and this library will not choose it for you.
+
+---
+
+## What it refuses, and why
+
+Three refusals, and each is the answer to a way of being confidently wrong.
+
+**An observable that ignores its seed** raises immediately. `fit` averages an observable
+over `trials` *different* seeds, so an observable that reads the clock or an unseeded RNG
+produces a mean over noise. A mean over noise still has a standard error, still forms a
+ladder, and can still plateau — every guard downstream compares against that error, so a
+broken adapter does not produce a wrong-looking answer, it produces a **confident** one.
+Each observable is called twice with the same `(truth, seed)` at the first and last rung;
+a disagreement is a wiring error, not a finding, so it throws.
+
+**A constant that never settles** is reported as `UNDETERMINED` with the reason. A run of
+three consecutive rungs must agree within two combined standard errors before any
+plateau is reported; the value is then the inverse-variance-weighted mean over that run,
+and the report says which rung it started from.
+
+**A choice between observables that is not earned** is reported as `UNDETERMINED`. With
+`instances()`, an observable is only called informative if its constant varies at least
+3x its own measurement error across instances, and it is only *chosen* over the runner-up
+if it beats it by 3x. Two observables that both vary a lot are not a choice.
+
+### The rule underneath all of it
+
+> Compare against the **noise**, never against the **size**.
+
+A spread only means something in units of the error on the thing that spread. Dividing by
+the magnitude instead is how a large number gets mistaken for a real one, and it is the
+single mistake this library is shaped around not making.
+
+### The thresholds
+
+They are the contract, and `python/tests/test_parity.py` asserts both halves hold the
+same ones and produce the same output on the same numbers — including the same
+explanatory strings.
+
+| constant | value | what it gates |
+| --- | --- | --- |
+| `PLATEAU_K` | 2.0 | sigma within which rungs must agree |
+| `PLATEAU_RUN` | 3 | consecutive agreeing rungs required |
+| `MIN_RATIO` | 3.0 | error-multiples an observable must vary by to be informative |
+| `MIN_MARGIN` | 3.0 | factor by which the winner must beat the runner-up |
+| `SIGMAS` | 3.0 | standard errors in a minimum detectable effect |
+| `FLOOR_TRIALS` / `CAP_TRIALS` | 400 / 40000 | the budget search bounds |
+
+---
+
+## This is a library, not a CLI
+
+An adapter is a code object with closures in it. There is nothing to pass on a command
+line that would not amount to naming a Python or JavaScript symbol and importing it, so
+the package ships an import and no console script.
+
+## Where it sits
+
+**Layer 0**: no dependencies inside or outside this network of packages, by design.
+
+The `nondet` edge was considered and **rejected**. `nondet` addresses a function as
+`FILE::NAME` so it can re-run it in fresh processes; this library's observables are
+closures inside an adapter object and have no such address, so `nondet` cannot probe
+them. Wiring it in would have meant either a fake file path or a check that never ran —
+a dependency that looks like a guarantee and is not. The reproducibility precondition is
+implemented natively in both halves instead, and it is checked in the same call that
+would have needed the guarantee. `nondet` remains the right tool for the *functions your
+adapter calls*, which do have addresses; running it on those is a good idea and is not
+something this package can do on your behalf.
+
+## Development
+
+```bash
+python3 -m unittest discover -s python/tests -v   # PYTHONPATH=python
+node --test js/test/*.test.js
+```
+
+The parity suite skips when `node` is not on PATH, so a Python-only contributor can still
+run everything else. CI asserts it was **not** skipped — a skipped test and a passing one
+look identical in a tally.
+
+## License
+
+MIT

undetermined-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "undetermined"
+version = "0.1.0"
+description = "Point it at a program, get back what can and cannot be determined: a constant with an error bar, and an explicit undetermined list with reasons."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+keywords = [
+  "empirical", "constant", "measurement", "plateau", "algorithm-analysis",
+  "benchmark", "estimation", "reproducibility", "uncertainty",
+  "minimum-detectable-effect", "zero-dependency",
+]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+]
+# LAYER 0. No in-network dependencies. The `nondet` edge was considered and rejected:
+# nondet addresses a function as FILE::NAME, and this package's observables are closures
+# inside an adapter object, so it cannot probe them. The reproducibility precondition is
+# implemented natively in both halves instead. See the README.
+dependencies = []
+
+[tool.setuptools]
+package-dir = { "" = "python" }
+packages = ["undetermined"]

undetermined-0.1.0/python/undetermined/__init__.py

@@ -0,0 +1,26 @@
+"""undetermined — point it at a program, get back what can and cannot be determined.
+
+    from undetermined import characterize, to_tolerance
+
+    report = characterize(MyAdapter(), trials=2500)
+    report = to_tolerance(MyAdapter(), tolerance=0.01)   # derive the trial count instead
+
+The name is the differentiator. Plenty of things fit a curve to measurements and hand
+back a number; this one has an `undetermined` list with reasons, and will put an
+observable on it rather than fit a plateau to a drift.
+"""
+
+from .budget import mde, to_tolerance, trials_for
+from .core import (
+    UNDETERMINED,
+    characterize,
+    fit,
+    heterogeneity,
+    ladder_for,
+    plateau,
+    reproducible,
+)
+
+__all__ = ["characterize", "to_tolerance", "reproducible", "fit", "plateau",
+           "heterogeneity", "ladder_for", "mde", "trials_for", "UNDETERMINED"]
+__version__ = "0.1.0"

undetermined-0.1.0/python/undetermined/budget.py

@@ -0,0 +1,114 @@
+"""Spend trials until the answer is supportable, and refuse to report one that is not.
+
+`characterize(adapter, trials=2500)` and `discover(adapter, trials=1200)` both take a trial
+count the caller has to guess. Exp 310 measured what guessing costs, on twelve sealed
+capabilities with the same rule and only the precision differing:
+
+    a coarse guess     2 of 12 correct
+    a good guess      11 of 12
+    a fine guess      12 of 12
+    DERIVED           11 of 12   -- and you cannot guess wrong
+
+So the derived budget is not more accurate than a good guess. What it buys is that there is
+no guess: the trial count is computed from the size of effect you said you cared about and
+the variance actually observed, and it varied 100x across those twelve capabilities (400 to
+40,000) with nothing in the rule knowing which capability was which.
+
+THE SECOND HALF MATTERS MORE. Exp 309 published twelve verdicts and five of them were
+unresolved -- right, and reached by measurements that could not have detected the effect that
+decides the question. It reported a minimum detectable effect only on the branch where it
+abstained. So here EVERY constant carries its MDE, and one whose MDE exceeds the tolerance is
+reported `supported: False` with what it would cost, instead of being reported as a number.
+
+    from budget import to_tolerance
+    report = to_tolerance(MyAdapter(), tolerance=0.01)   # "1% in the constant matters to me"
+
+`tolerance` is a statement of what difference matters -- like a significance level, it is part
+of the question and not a nuisance parameter. It is the one number this module will not choose
+for you, and exp 310's TARGETS said so before that round ran.
+"""
+from . import core as CH
+
+SIGMAS = 3.0            # exp 309's RESOLVABLE_SIGMAS; an MDE is this many standard errors
+FLOOR_TRIALS = 400      # exp 310's floor
+CAP_TRIALS = 40000      # exp 310's cap
+GROWTH = 2.0
+
+
+def mde(se, value):
+    """The smallest RELATIVE effect a measurement of this precision could have detected."""
+    if se is None or value in (None, 0):
+        return None
+    return SIGMAS * abs(se / value)
+
+
+def trials_for(current_mde, tolerance, spent):
+    """How many trials would bring the MDE down to the tolerance.
+
+    se falls as 1/sqrt(N), so being k times too coarse costs k^2 the trials. Returns None
+    when the measurement already supports the claim.
+    """
+    if current_mde is None or spent <= 0 or tolerance <= 0:
+        return None
+    if current_mde <= tolerance:
+        return None
+    return int(spent * (current_mde / tolerance) ** 2) + 1
+
+
+def _annotate(report, tolerance):
+    """Attach an MDE and a support verdict to every constant the report carries."""
+    worst = None
+    for name, row in report["per_observable"].items():
+        m = mde(row.get("constant_se"), row.get("constant"))
+        row["mde"] = m
+        row["tolerance"] = tolerance
+        row["supported"] = m is not None and m <= tolerance
+        if not row["supported"]:
+            row["why_unsupported"] = (
+                "no constant was determined" if m is None else
+                "the constant is %.6g but this measurement could not have detected a "
+                "%.3g relative effect (MDE %.3g), so it does not support a claim at that "
+                "tolerance" % (row["constant"], tolerance, m))
+            if m is not None and (worst is None or m > worst):
+                worst = m
+    return worst
+
+
+def to_tolerance(adapter, tolerance, seed0=17, floor=FLOOR_TRIALS, cap=CAP_TRIALS,
+                 on_step=None):
+    """Run `characterize`, raising the trial count until every constant it determined is
+    supported at `tolerance`, or the budget is spent.
+
+    Returns characterize's report with three things added per observable -- `mde`,
+    `tolerance`, `supported` -- plus a top-level `budget` block recording what was spent,
+    whether every determined constant is supported, and what the shortfall would cost.
+
+    An observable that comes back UNDETERMINED is left alone: no amount of precision turns a
+    ladder that never flattens into a constant, and pretending otherwise is what exp 310's
+    leg 0 found exp 309 doing on five of twelve.
+    """
+    if tolerance <= 0:
+        raise ValueError("tolerance must be positive: it is the size of effect you care "
+                         "about, and this module will not choose it for you")
+    trials = floor
+    history = []
+    while True:
+        report = CH.characterize(adapter, trials=trials, seed0=seed0)
+        worst = _annotate(report, tolerance)
+        determined = [n for n, r in report["per_observable"].items()
+                      if r["constant"] is not CH.UNDETERMINED]
+        unsupported = [n for n in determined if not report["per_observable"][n]["supported"]]
+        history.append({"trials": trials, "worst_mde": worst,
+                        "unsupported": sorted(unsupported)})
+        if on_step:
+            on_step(history[-1])
+        want = trials_for(worst, tolerance, trials) if unsupported else None
+        if not unsupported or want is None or trials >= cap:
+            report["budget"] = {
+                "tolerance": tolerance, "trials": trials, "floor": floor, "cap": cap,
+                "all_supported": not unsupported, "unsupported": sorted(unsupported),
+                "worst_mde": worst, "history": history,
+                "would_need": None if want is None else min(want, 10 ** 12),
+                "shortfall": None if want is None else round(want / float(cap), 2)}
+            return report
+        trials = min(cap, max(int(trials * GROWTH), want))

undetermined-0.1.0/python/undetermined/core.py

@@ -0,0 +1,209 @@
+"""Point it at a program; get back what can and cannot be determined about it.
+
+One interface, no capability-specific logic.
+
+An adapter exposes only:
+    truths()              -> the controllable input values, known by construction
+    observables           -> {name: f(truth, seed) -> float}
+    knobs                 -> {name: [values]}  (optional)
+    perturbed(knob, v)    -> {name: f(truth, seed)}  (optional, required if knobs given)
+
+Everything below is assembled from instruments earlier rounds paid for:
+  * fit c = truth / E[raw]                    (exp 297)
+  * plateau across a ladder                   (exp 298)
+  * heterogeneity = spread / own error        (exp 302)
+  * report what is UNDETERMINED, never guess  (exps 298, 300, 303)
+
+The rule that survived every round: compare against the NOISE, never against the SIZE.
+"""
+import math
+
+UNDETERMINED = None
+MIN_RATIO = 3.0
+MIN_MARGIN = 3.0
+PLATEAU_K = 2.0
+PLATEAU_RUN = 3
+
+
+# ---------------------------------------------------------------- primitives
+
+def fit(sample, truth, trials, seed0=0):
+    """c such that c * E[raw] == truth. Returns (c, standard_error)."""
+    raws = [sample(truth, seed0 + i) for i in range(trials)]
+    mean = sum(raws) / len(raws)
+    if mean == 0.0:
+        return None, None
+    c = truth / mean
+    var = sum((r - mean) ** 2 for r in raws) / (len(raws) - 1) if len(raws) > 1 else 0.0
+    cv = math.sqrt(var) / abs(mean)
+    return c, abs(c) * cv / math.sqrt(len(raws))
+
+
+def _sd(xs):
+    if len(xs) < 2:
+        return None
+    m = sum(xs) / len(xs)
+    return math.sqrt(sum((x - m) ** 2 for x in xs) / (len(xs) - 1))
+
+
+def heterogeneity(values, errors):
+    """Spread across cases in units of the measurement error. Never divided by magnitude."""
+    sd = _sd(values)
+    if sd is None:
+        return None
+    mean_e = sum(errors) / len(errors)
+    return None if mean_e <= 0 else sd / mean_e
+
+
+def plateau(ladder, k=PLATEAU_K, need=PLATEAU_RUN):
+    """Earliest rung from which every later rung agrees within k combined errors."""
+    rungs = sorted(ladder, key=lambda r: r["truth"])
+    if len(rungs) < need:
+        return {"value": UNDETERMINED, "se": None, "from_truth": None,
+                "why": "ladder shorter than the required run of %d" % need}
+    for i in range(len(rungs) - need + 1):
+        tail = rungs[i:]
+        if all(abs(tail[0]["c"] - r["c"]) <=
+               k * math.sqrt(tail[0]["se"] ** 2 + r["se"] ** 2) for r in tail[1:]):
+            w = sum(1.0 / r["se"] ** 2 for r in tail)
+            return {"value": sum(r["c"] / r["se"] ** 2 for r in tail) / w,
+                    "se": (1.0 / w) ** 0.5, "from_truth": tail[0]["truth"],
+                    "why": "%d rungs from truth=%g agree within %.1f sigma"
+                           % (len(tail), tail[0]["truth"], k)}
+    return {"value": UNDETERMINED, "se": None, "from_truth": None,
+            "why": "no run of %d rungs agrees; the constant is still moving at the top of "
+                   "the ladder" % need}
+
+
+# ---------------------------------------------------------------- the report
+
+def ladder_for(sample, truths, trials, seed0=0):
+    out = []
+    for t in truths:
+        c, se = fit(sample, t, trials, seed0)
+        if c is not None and se:
+            out.append({"truth": t, "c": c, "se": se})
+    return out
+
+
+def reproducible(obs, truths, seed0=17):
+    """Every observable must answer the same for the same (truth, seed), or refuse.
+
+    THE PRECONDITION NOTHING ELSE HERE CAN SUBSTITUTE FOR. `fit` averages an observable
+    over `trials` DIFFERENT seeds, so an observable that ignores its seed and reads the
+    clock, the environment or an unseeded RNG produces a mean over noise -- and a mean
+    over noise still has a standard error, still forms a ladder, and can still plateau.
+    Every guard downstream compares against that error, so a broken adapter does not
+    produce a wrong-looking answer: it produces a confident one.
+
+    Checked at the cheapest place it can fail -- the first and last rung, twice each --
+    because an observable that is stable at one size and not another is stable at
+    neither, and four calls is a price nobody notices.
+
+    A raise, not a `look`. The other refusals in this module describe what the program
+    would not reveal; this one says the instrument was wired up wrong, and continuing
+    would report a number about nothing.
+    """
+    for name, f in sorted(obs.items()):
+        for truth in ({truths[0], truths[-1]} if truths else ()):
+            first, second = f(truth, seed0), f(truth, seed0)
+            if first != second:
+                raise ValueError(
+                    "observable %r is not reproducible: at truth=%r and seed=%r it "
+                    "returned %r and then %r. Every constant below is fitted from a mean "
+                    "over seeds, so an observable that ignores its seed yields a mean "
+                    "over noise -- which still plateaus, and would be reported as an "
+                    "answer." % (name, truth, seed0, first, second))
+
+
+def characterize(adapter, trials=2500, seed0=17):
+    """The whole report. Nothing here knows what program it is looking at."""
+    truths = list(adapter.truths())
+    obs = adapter.observables
+    if len(obs) < 2:
+        raise ValueError("an adapter must expose at least two observables, or there is no "
+                         "choice to make and the tool cannot be shown to make one")
+    reproducible(obs, truths, seed0)
+    per = {}
+    for name, f in obs.items():
+        lad = ladder_for(f, truths, trials, seed0)
+        p = plateau(lad)
+        per[name] = {"ladder": lad, "plateau": p,
+                     "constant": p["value"], "constant_se": p["se"],
+                     "regime_from": p["from_truth"],
+                     "top_rung": lad[-1]["c"] if lad else None,
+                     "top_rung_se": lad[-1]["se"] if lad else None}
+
+    # which observable carries information about THIS instance?  Only answerable with more
+    # than one instance; with one, say so rather than guessing (exp 303's family C).
+    inst = getattr(adapter, "instances", None)
+    informative = {"choice": UNDETERMINED,
+                   "why": "only one instance was supplied, so no constant can be shown to "
+                          "vary across instances; supply >=2 to decide"}
+    if inst:
+        vals, errs = {}, {}
+        for name in obs:
+            vs, es = [], []
+            for sub in inst():
+                c, se = fit(sub.observables[name], truths[-1], trials, seed0)
+                if c is not None and se:
+                    vs.append(c); es.append(se)
+            vals[name], errs[name] = vs, es
+        ratios = {n: heterogeneity(vals[n], errs[n]) for n in obs}
+        informative = _pick(ratios)
+        informative["ratios"] = ratios
+
+    # does anything respond to perturbing the program itself?
+    knobs = getattr(adapter, "knobs", None)
+    response = {}
+    if knobs and hasattr(adapter, "perturbed"):
+        for name in obs:
+            response[name] = {}
+            for knob, values in knobs.items():
+                pts = []
+                for v in values:
+                    c, _ = fit(adapter.perturbed(knob, v)[name], truths[-1], max(400, trials // 4), seed0)
+                    if c and c > 0:
+                        pts.append((math.log(v), math.log(c)))
+                response[name][knob] = _slope(pts)
+
+    undetermined = [n for n in obs if per[n]["constant"] is UNDETERMINED]
+    return {"observables": sorted(obs), "per_observable": per,
+            "informative": informative, "perturbation_response": response,
+            "undetermined": undetermined,
+            "notes": _notes(per, informative, undetermined)}
+
+
+def _slope(pts):
+    if len(pts) < 2:
+        return None
+    mx = sum(p[0] for p in pts) / len(pts)
+    my = sum(p[1] for p in pts) / len(pts)
+    den = sum((p[0] - mx) ** 2 for p in pts)
+    return None if den <= 0 else sum((p[0]-mx)*(p[1]-my) for p in pts) / den
+
+
+def _pick(ratios):
+    usable = {k: v for k, v in ratios.items() if v is not None}
+    live = {k: v for k, v in usable.items() if v >= MIN_RATIO}
+    if not live:
+        return {"choice": UNDETERMINED,
+                "why": "no constant varies beyond %.0fx its own error across instances"
+                       % MIN_RATIO}
+    rank = sorted(live.items(), key=lambda kv: -kv[1])
+    if len(rank) > 1 and rank[0][1] < MIN_MARGIN * rank[1][1]:
+        return {"choice": UNDETERMINED,
+                "why": "%s (%.1f) does not beat %s (%.1f) by %.0fx"
+                       % (rank[0][0], rank[0][1], rank[1][0], rank[1][1], MIN_MARGIN)}
+    return {"choice": rank[0][0],
+            "why": "%s varies %.1fx its own error" % (rank[0][0], rank[0][1])}
+
+
+def _notes(per, informative, undetermined):
+    out = []
+    for n in sorted(undetermined):
+        out.append("`%s`: no plateau -- %s" % (n, per[n]["plateau"]["why"]))
+    if informative["choice"] is UNDETERMINED:
+        out.append("which observable characterises the program: UNDETERMINED -- %s"
+                   % informative["why"])
+    return out

undetermined-0.1.0/python/undetermined.egg-info/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: undetermined
+Version: 0.1.0
+Summary: Point it at a program, get back what can and cannot be determined: a constant with an error bar, and an explicit undetermined list with reasons.
+License: MIT
+Keywords: empirical,constant,measurement,plateau,algorithm-analysis,benchmark,estimation,reproducibility,uncertainty,minimum-detectable-effect,zero-dependency
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# undetermined
+
+**Point it at a program; get back what can and cannot be determined about it.**
+
+Plenty of libraries fit a curve to measurements and hand you back a number. This one
+hands back a number *with the error bar it was decided by*, plus an explicit
+`undetermined` list with a reason on each entry — and it will put an observable on that
+list rather than fit a plateau to a drift.
+
+```
+heads: 1.9978 +/- 0.0032   4 rungs from truth=8 agree within 2.0 sigma
+flat:  UNDETERMINED        no run of 3 rungs agrees; the constant is still moving
+                           at the top of the ladder
+```
+
+The second line is the point. A tool that always produced the first line would be
+useless and would still pass every test that checks it produces one.
+
+Ships as `undetermined` on **PyPI** and on **npm**, from one tree, at one version. No
+dependencies in either half.
+
+---
+
+## Install
+
+```bash
+pip install undetermined
+```
+
+```bash
+npm install undetermined
+```
+
+## Use
+
+You supply an **adapter**: an object that knows how to run the thing you are measuring
+at a controllable input size, and nothing else. This library never knows what program it
+is looking at.
+
+```python
+import random
+from undetermined import characterize
+
+class Coin:
+    def truths(self):
+        # The controllable input values, known by construction. A ladder, not a point:
+        # a constant that is only measured at one size cannot be shown to have settled.
+        return [8, 32, 128, 512]
+
+    @property
+    def observables(self):
+        def heads(truth, seed):
+            r = random.Random(seed)
+            return sum(1 for _ in range(truth) if r.random() < 0.5)
+
+        def flat(truth, seed):
+            r = random.Random(seed)
+            return sum(1 for _ in range(12) if r.random() < 0.5) + 0.5
+
+        return {"heads": heads, "flat": flat}
+
+report = characterize(Coin(), trials=2500)
+
+report["per_observable"]["heads"]["constant"]      # 1.9978...
+report["per_observable"]["heads"]["constant_se"]   # 0.0032...
+report["undetermined"]                             # ['flat']
+report["notes"]                                    # why each one is on the list
+```
+
+```js
+import { characterize } from "undetermined";
+
+const report = characterize(adapter, { trials: 2500 });
+report.per_observable.heads.constant;   // 1.9978...
+report.undetermined;                    // ['flat']
+```
+
+### The adapter protocol
+
+| member | required | meaning |
+| --- | --- | --- |
+| `truths()` | yes | the controllable input values, known by construction |
+| `observables` | yes | `{name: (truth, seed) -> number}`, at least two |
+| `instances()` | no | sibling instances of the same family |
+| `knobs` | no | `{name: [values]}` — parameters of the program itself |
+| `perturbed(knob, v)` | with `knobs` | the observables with that knob set to that value |
+
+At least two observables, always. With one there is no choice to make, so the library
+cannot be shown to make one — it raises rather than reporting a confident single answer.
+
+### Deriving the trial count instead of guessing it
+
+`characterize` takes a `trials` count. If you would rather state the precision you need
+and have the budget derived:
+
+```python
+from undetermined import to_tolerance
+report = to_tolerance(MyAdapter(), tolerance=0.01)   # 1% on the constant
+```
+
+It doubles the trial count until every determined constant is inside the tolerance or the
+cap is reached, and reports which ones never got there. `tolerance=0` raises: a tolerance
+is a decision about your problem, and this library will not choose it for you.
+
+---
+
+## What it refuses, and why
+
+Three refusals, and each is the answer to a way of being confidently wrong.
+
+**An observable that ignores its seed** raises immediately. `fit` averages an observable
+over `trials` *different* seeds, so an observable that reads the clock or an unseeded RNG
+produces a mean over noise. A mean over noise still has a standard error, still forms a
+ladder, and can still plateau — every guard downstream compares against that error, so a
+broken adapter does not produce a wrong-looking answer, it produces a **confident** one.
+Each observable is called twice with the same `(truth, seed)` at the first and last rung;
+a disagreement is a wiring error, not a finding, so it throws.
+
+**A constant that never settles** is reported as `UNDETERMINED` with the reason. A run of
+three consecutive rungs must agree within two combined standard errors before any
+plateau is reported; the value is then the inverse-variance-weighted mean over that run,
+and the report says which rung it started from.
+
+**A choice between observables that is not earned** is reported as `UNDETERMINED`. With
+`instances()`, an observable is only called informative if its constant varies at least
+3x its own measurement error across instances, and it is only *chosen* over the runner-up
+if it beats it by 3x. Two observables that both vary a lot are not a choice.
+
+### The rule underneath all of it
+
+> Compare against the **noise**, never against the **size**.
+
+A spread only means something in units of the error on the thing that spread. Dividing by
+the magnitude instead is how a large number gets mistaken for a real one, and it is the
+single mistake this library is shaped around not making.
+
+### The thresholds
+
+They are the contract, and `python/tests/test_parity.py` asserts both halves hold the
+same ones and produce the same output on the same numbers — including the same
+explanatory strings.
+
+| constant | value | what it gates |
+| --- | --- | --- |
+| `PLATEAU_K` | 2.0 | sigma within which rungs must agree |
+| `PLATEAU_RUN` | 3 | consecutive agreeing rungs required |
+| `MIN_RATIO` | 3.0 | error-multiples an observable must vary by to be informative |
+| `MIN_MARGIN` | 3.0 | factor by which the winner must beat the runner-up |
+| `SIGMAS` | 3.0 | standard errors in a minimum detectable effect |
+| `FLOOR_TRIALS` / `CAP_TRIALS` | 400 / 40000 | the budget search bounds |
+
+---
+
+## This is a library, not a CLI
+
+An adapter is a code object with closures in it. There is nothing to pass on a command
+line that would not amount to naming a Python or JavaScript symbol and importing it, so
+the package ships an import and no console script.
+
+## Where it sits
+
+**Layer 0**: no dependencies inside or outside this network of packages, by design.
+
+The `nondet` edge was considered and **rejected**. `nondet` addresses a function as
+`FILE::NAME` so it can re-run it in fresh processes; this library's observables are
+closures inside an adapter object and have no such address, so `nondet` cannot probe
+them. Wiring it in would have meant either a fake file path or a check that never ran —
+a dependency that looks like a guarantee and is not. The reproducibility precondition is
+implemented natively in both halves instead, and it is checked in the same call that
+would have needed the guarantee. `nondet` remains the right tool for the *functions your
+adapter calls*, which do have addresses; running it on those is a good idea and is not
+something this package can do on your behalf.
+
+## Development
+
+```bash
+python3 -m unittest discover -s python/tests -v   # PYTHONPATH=python
+node --test js/test/*.test.js
+```
+
+The parity suite skips when `node` is not on PATH, so a Python-only contributor can still
+run everything else. CI asserts it was **not** skipped — a skipped test and a passing one
+look identical in a tally.
+
+## License
+
+MIT

undetermined-0.1.0/python/undetermined.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+LICENSE
+README.md
+pyproject.toml
+python/undetermined/__init__.py
+python/undetermined/budget.py
+python/undetermined/core.py
+python/undetermined.egg-info/PKG-INFO
+python/undetermined.egg-info/SOURCES.txt
+python/undetermined.egg-info/dependency_links.txt
+python/undetermined.egg-info/top_level.txt

undetermined-0.1.0/python/undetermined.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

undetermined-0.1.0/python/undetermined.egg-info/top_level.txt

@@ -0,0 +1 @@
+undetermined

undetermined-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+