copela 0.1.0__tar.gz

copela-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Felipe Santibanez-Leal
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

copela-0.1.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: copela
+Version: 0.1.0
+Summary: Run narrative-to-formal translation across many language models and score it with oracles that are not language models: executable, structural and property layers, reported separately.
+Author-email: Felipe Santibanez-Leal <fsantibanez@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/fsantibanezleal/CAOS_Copela
+Project-URL: Source, https://github.com/fsantibanezleal/CAOS_Copela
+Project-URL: Issues, https://github.com/fsantibanezleal/CAOS_Copela/issues
+Project-URL: Changelog, https://github.com/fsantibanezleal/CAOS_Copela/blob/main/CHANGELOG.md
+Keywords: evaluation,benchmark,optimization,formalization,autoformalization,metamorphic-testing,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: planteo>=0.1.0
+Provides-Extra: solvers
+Requires-Dist: pyomo>=6.7; extra == "solvers"
+Requires-Dist: highspy>=1.7; extra == "solvers"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai>=1.50; extra == "openai"
+Provides-Extra: all
+Requires-Dist: pyomo>=6.7; extra == "all"
+Requires-Dist: highspy>=1.7; extra == "all"
+Requires-Dist: anthropic>=0.40; extra == "all"
+Requires-Dist: openai>=1.50; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Dynamic: license-file
+
+# copela
+
+[![PyPI](https://img.shields.io/pypi/v/copela.svg)](https://pypi.org/project/copela/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Run narrative-to-formal translation across many language models, and score the result with oracles
+that are **not** language models.
+
+The name is the cupel used in fire assay: the vessel that separates the metal from the lead. That is
+the job here, separating a formalization that is faithful from one that merely runs.
+
+## The number this produces
+
+```
+gap report
+============================================================
+
+  anthropic/claude-sonnet-5 [optimization]  ran 0.940 [0.832, 0.981] n=50  faithful 0.720 [0.584, 0.826] n=50  gap +0.220
+```
+
+**ran** is what the field reports. **faithful** is what was asked for. The gap between them is the
+output, and no source found reports it across target families.
+
+That gap is not hypothetical. Where it has been measured carefully, in natural-language to Lean
+formalization, it runs [3.0 to 29.0 percentage points](https://arxiv.org/abs/2606.31002), and the
+strongest system measured had the largest gap: 89.5% compiling, 60.5% faithful.
+
+## Four layers, never merged
+
+| Layer | Asks | Strength |
+|---|---|---|
+| **executable** | did it run, solve, compile | necessary, weak, the layer the field over-reports |
+| **structural** | is it the same model as the reference | strong where it applies |
+| **property** | do the invariants of this class hold | strong, catches what structure misses |
+| **judge** | what would a model say | a labelled screening aggregate, never truth |
+
+There is deliberately **no combined score**, and a test fails if anyone adds one. A single number
+lets a high "it ran" rate conceal a low "it was right" rate, which is the distance this exists to
+show.
+
+The judge layer is reported because the literature reports it and comparability matters. It carries
+a label on every record saying it is not an oracle, because the study that calibrated a two-judge
+consensus against human majority states exactly that.
+
+## The property layer
+
+Metamorphic relations: instead of checking an exact output, check how the output **must** change
+when the input changes in a controlled way.
+
+| Relation | Guarantee |
+|---|---|
+| scale the objective by `k > 0` | the argmin cannot move |
+| add a redundant constraint | the feasible set cannot change |
+| tighten a constraint | the optimum cannot improve |
+
+The standard objection to metamorphic testing is that the relations must be authored per problem
+class and so do not generalise to arbitrary programs. That is the design here: the target families
+are narrow typed classes, so the relations are written once per class.
+
+A candidate that solves and then fails one of these is wrong in a way no solver would have reported.
+
+## Install
+
+```bash
+pip install copela                    # the harness
+pip install "copela[solvers]"         # plus Pyomo and HiGHS
+pip install "copela[all]"             # plus the provider SDKs
+```
+
+## Use
+
+```python
+from copela import Budget, Case, Ledger, Sweep, Target, build
+from copela.providers import get
+from copela.solvers.highs import make_solver
+
+sweep = Sweep(
+    ledger=Ledger("runs.jsonl"),
+    budget=Budget(limit_usd=5.00, max_consecutive_failures=10),
+    providers={"anthropic": get("anthropic"), "ollama": get("ollama")},
+    build_prompt=my_prompt,        # the prompt strategy is what a study varies
+    parse_response=my_parser,
+    solve=make_solver(),
+    repeats=5,
+)
+
+sweep.run(cases, [Target("anthropic", "claude-sonnet-5"),
+                  Target("ollama", "qwen3:8b")])
+
+print(build(Ledger("runs.jsonl")).to_text())
+```
+
+From the shell:
+
+```bash
+copela models                      # what each provider serves, and what it costs
+copela solve problem.json          # solve one formalization, no model involved
+copela report runs.jsonl           # the gap
+```
+
+## Why many models, not one
+
+Because a benchmark of AI-assisted modelling and simulation reports that
+[no single model dominates across engine types](https://arxiv.org/abs/2605.28994), with
+task-specific tradeoffs between speed and accuracy. A ranking claimed from one model contradicts a
+published result, so the provider seam is a requirement rather than tidiness: Anthropic, Groq and
+local models through Ollama, behind one interface, with no vendor name anywhere outside
+`copela/providers/`. A test enforces that.
+
+## What reproducibility means here
+
+Temperature zero does not make hosted inference deterministic. The dominant cause is the batch-size
+dependence of reduction kernels rather than floating-point non-associativity, and bitwise
+determinism costs a third to two thirds of throughput and cannot be bought over a hosted API.
+
+So the harness pins what it can (model, version, temperature, seed, provider fingerprint), records
+`n` repeats, and reports a **rate with a Wilson interval**. It never presents a single run as the
+result, and 5 of 5 is not reported as certainly 1.0.
+
+## The ledger
+
+One append-only JSONL record per call, carrying its full provenance. A record is never edited,
+because a ledger that can be rewritten is not evidence. It is also the resume mechanism: a sweep
+reads it and skips the calls already done.
+
+## Cost
+
+Every sweep declares a budget and a kill criterion before it runs, and the guard refuses the call
+that **would** exceed the ceiling rather than noticing afterwards.
+
+## Documentation
+
+The wiki is in [`docs/`](docs/). The design document, written before the code, is
+[`docs/design/SDD.md`](docs/design/SDD.md); each of its twelve requirements names the test that
+verifies it.
+
+## Related
+
+[`planteo`](https://github.com/fsantibanezleal/CAOS_Planteo) is the representation this consumes: a
+typed problem with dimensions on every quantity and provenance on every element.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

copela-0.1.0/README.md

@@ -0,0 +1,144 @@
+# copela
+
+[![PyPI](https://img.shields.io/pypi/v/copela.svg)](https://pypi.org/project/copela/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Run narrative-to-formal translation across many language models, and score the result with oracles
+that are **not** language models.
+
+The name is the cupel used in fire assay: the vessel that separates the metal from the lead. That is
+the job here, separating a formalization that is faithful from one that merely runs.
+
+## The number this produces
+
+```
+gap report
+============================================================
+
+  anthropic/claude-sonnet-5 [optimization]  ran 0.940 [0.832, 0.981] n=50  faithful 0.720 [0.584, 0.826] n=50  gap +0.220
+```
+
+**ran** is what the field reports. **faithful** is what was asked for. The gap between them is the
+output, and no source found reports it across target families.
+
+That gap is not hypothetical. Where it has been measured carefully, in natural-language to Lean
+formalization, it runs [3.0 to 29.0 percentage points](https://arxiv.org/abs/2606.31002), and the
+strongest system measured had the largest gap: 89.5% compiling, 60.5% faithful.
+
+## Four layers, never merged
+
+| Layer | Asks | Strength |
+|---|---|---|
+| **executable** | did it run, solve, compile | necessary, weak, the layer the field over-reports |
+| **structural** | is it the same model as the reference | strong where it applies |
+| **property** | do the invariants of this class hold | strong, catches what structure misses |
+| **judge** | what would a model say | a labelled screening aggregate, never truth |
+
+There is deliberately **no combined score**, and a test fails if anyone adds one. A single number
+lets a high "it ran" rate conceal a low "it was right" rate, which is the distance this exists to
+show.
+
+The judge layer is reported because the literature reports it and comparability matters. It carries
+a label on every record saying it is not an oracle, because the study that calibrated a two-judge
+consensus against human majority states exactly that.
+
+## The property layer
+
+Metamorphic relations: instead of checking an exact output, check how the output **must** change
+when the input changes in a controlled way.
+
+| Relation | Guarantee |
+|---|---|
+| scale the objective by `k > 0` | the argmin cannot move |
+| add a redundant constraint | the feasible set cannot change |
+| tighten a constraint | the optimum cannot improve |
+
+The standard objection to metamorphic testing is that the relations must be authored per problem
+class and so do not generalise to arbitrary programs. That is the design here: the target families
+are narrow typed classes, so the relations are written once per class.
+
+A candidate that solves and then fails one of these is wrong in a way no solver would have reported.
+
+## Install
+
+```bash
+pip install copela                    # the harness
+pip install "copela[solvers]"         # plus Pyomo and HiGHS
+pip install "copela[all]"             # plus the provider SDKs
+```
+
+## Use
+
+```python
+from copela import Budget, Case, Ledger, Sweep, Target, build
+from copela.providers import get
+from copela.solvers.highs import make_solver
+
+sweep = Sweep(
+    ledger=Ledger("runs.jsonl"),
+    budget=Budget(limit_usd=5.00, max_consecutive_failures=10),
+    providers={"anthropic": get("anthropic"), "ollama": get("ollama")},
+    build_prompt=my_prompt,        # the prompt strategy is what a study varies
+    parse_response=my_parser,
+    solve=make_solver(),
+    repeats=5,
+)
+
+sweep.run(cases, [Target("anthropic", "claude-sonnet-5"),
+                  Target("ollama", "qwen3:8b")])
+
+print(build(Ledger("runs.jsonl")).to_text())
+```
+
+From the shell:
+
+```bash
+copela models                      # what each provider serves, and what it costs
+copela solve problem.json          # solve one formalization, no model involved
+copela report runs.jsonl           # the gap
+```
+
+## Why many models, not one
+
+Because a benchmark of AI-assisted modelling and simulation reports that
+[no single model dominates across engine types](https://arxiv.org/abs/2605.28994), with
+task-specific tradeoffs between speed and accuracy. A ranking claimed from one model contradicts a
+published result, so the provider seam is a requirement rather than tidiness: Anthropic, Groq and
+local models through Ollama, behind one interface, with no vendor name anywhere outside
+`copela/providers/`. A test enforces that.
+
+## What reproducibility means here
+
+Temperature zero does not make hosted inference deterministic. The dominant cause is the batch-size
+dependence of reduction kernels rather than floating-point non-associativity, and bitwise
+determinism costs a third to two thirds of throughput and cannot be bought over a hosted API.
+
+So the harness pins what it can (model, version, temperature, seed, provider fingerprint), records
+`n` repeats, and reports a **rate with a Wilson interval**. It never presents a single run as the
+result, and 5 of 5 is not reported as certainly 1.0.
+
+## The ledger
+
+One append-only JSONL record per call, carrying its full provenance. A record is never edited,
+because a ledger that can be rewritten is not evidence. It is also the resume mechanism: a sweep
+reads it and skips the calls already done.
+
+## Cost
+
+Every sweep declares a budget and a kill criterion before it runs, and the guard refuses the call
+that **would** exceed the ceiling rather than noticing afterwards.
+
+## Documentation
+
+The wiki is in [`docs/`](docs/). The design document, written before the code, is
+[`docs/design/SDD.md`](docs/design/SDD.md); each of its twelve requirements names the test that
+verifies it.
+
+## Related
+
+[`planteo`](https://github.com/fsantibanezleal/CAOS_Planteo) is the representation this consumes: a
+typed problem with dimensions on every quantity and provenance on every element.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

copela-0.1.0/pyproject.toml

@@ -0,0 +1,73 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "copela"
+version = "0.1.0"
+description = "Run narrative-to-formal translation across many language models and score it with oracles that are not language models: executable, structural and property layers, reported separately."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+authors = [{ name = "Felipe Santibanez-Leal", email = "fsantibanez@gmail.com" }]
+keywords = [
+    "evaluation",
+    "benchmark",
+    "optimization",
+    "formalization",
+    "autoformalization",
+    "metamorphic-testing",
+    "llm",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Typing :: Typed",
+]
+dependencies = ["planteo>=0.1.0"]
+
+[project.optional-dependencies]
+solvers = ["pyomo>=6.7", "highspy>=1.7"]
+anthropic = ["anthropic>=0.40"]
+openai = ["openai>=1.50"]
+all = ["pyomo>=6.7", "highspy>=1.7", "anthropic>=0.40", "openai>=1.50"]
+dev = ["pytest>=8.0", "ruff>=0.6"]
+
+[project.scripts]
+copela = "copela.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/fsantibanezleal/CAOS_Copela"
+Source = "https://github.com/fsantibanezleal/CAOS_Copela"
+Issues = "https://github.com/fsantibanezleal/CAOS_Copela/issues"
+Changelog = "https://github.com/fsantibanezleal/CAOS_Copela/blob/main/CHANGELOG.md"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+copela = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+pythonpath = ["."]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+ignore = [
+    "E501",
+    # UP042 wants StrEnum. These enums inherit str deliberately: their .value is what lands in the
+    # JSONL ledger, and StrEnum changes str() behaviour in a way that would alter what is written.
+    "UP042",
+]

copela-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

copela-0.1.0/src/copela/__init__.py

@@ -0,0 +1,61 @@
+"""copela: run narrative-to-formal translation across many models, and score it with oracles
+that are not language models.
+
+The name is the cupel used in fire assay, the vessel that separates the metal from the lead. That is
+the job here: separating a formalization that is faithful from one that merely runs.
+
+Four layers, reported separately and never merged into one score:
+
+1. executable, did it run, solve, compile
+2. structural, is it the same model as the reference
+3. property, do the invariants of this class hold
+4. judge, what a model says, recorded as a labelled screening aggregate and never as truth
+
+The headline is the subtraction: how often the artifact ran, minus how often it was right.
+"""
+
+from __future__ import annotations
+
+from .budget import Budget, BudgetExceeded, estimate
+from .ledger import CallKey, Ledger, LedgerError, Record
+from .providers import Provider, ProviderError, StubProvider
+from .report import Cell, Report, build
+from .sweep import Case, Sweep, Target
+from .verdicts import (
+    JUDGE_LABEL,
+    CandidateVerdict,
+    Layer,
+    LayerResult,
+    Outcome,
+    Rate,
+)
+
+__version__ = "0.1.0"
+__display_version__ = "0.01.000"
+
+__all__ = [
+    "JUDGE_LABEL",
+    "Budget",
+    "BudgetExceeded",
+    "CallKey",
+    "CandidateVerdict",
+    "Case",
+    "Cell",
+    "Layer",
+    "LayerResult",
+    "Ledger",
+    "LedgerError",
+    "Outcome",
+    "Provider",
+    "ProviderError",
+    "Rate",
+    "Record",
+    "Report",
+    "StubProvider",
+    "Sweep",
+    "Target",
+    "__display_version__",
+    "__version__",
+    "build",
+    "estimate",
+]

copela-0.1.0/src/copela/budget.py

@@ -0,0 +1,84 @@
+"""The budget guard: stop before the limit, not after it.
+
+A sweep is cases times models times repeats, and each cell costs money. The guard exists because the
+failure it prevents has happened on this account: an unattended job consumed a week of quota in
+about a day.
+
+The rule is that the guard refuses the call that *would* exceed the budget, rather than noticing
+afterwards. A guard that reports an overrun is an accountant, not a guard.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+class BudgetExceeded(RuntimeError):
+    """Raised when a call would take the sweep past its declared budget."""
+
+
+@dataclass
+class Budget:
+    """A spend ceiling and a kill criterion, both declared before the sweep runs.
+
+    ``limit_usd`` is the hard ceiling. ``max_consecutive_failures`` is the kill criterion: a sweep
+    whose calls are all failing is buying nothing, and continuing to the ceiling is waste.
+    """
+
+    limit_usd: float
+    max_consecutive_failures: int = 10
+    spent_usd: float = 0.0
+    consecutive_failures: int = 0
+    calls: int = 0
+
+    def __post_init__(self) -> None:
+        if self.limit_usd < 0:
+            raise ValueError("a budget cannot be negative")
+
+    @property
+    def remaining_usd(self) -> float:
+        return max(0.0, self.limit_usd - self.spent_usd)
+
+    def check(self, estimated_usd: float) -> None:
+        """Raise if this call would exceed the ceiling. Call BEFORE spending."""
+        if self.spent_usd + estimated_usd > self.limit_usd:
+            raise BudgetExceeded(
+                f"this call is estimated at {estimated_usd:.4f} USD and "
+                f"{self.spent_usd:.4f} of {self.limit_usd:.4f} is already spent; "
+                "stopping before the budget rather than after it"
+            )
+        if self.consecutive_failures >= self.max_consecutive_failures:
+            raise BudgetExceeded(
+                f"{self.consecutive_failures} consecutive failures reached the kill criterion; "
+                "a sweep that is failing every call is buying nothing"
+            )
+
+    def charge(self, actual_usd: float, *, failed: bool = False) -> None:
+        """Record what a completed call actually cost."""
+        self.spent_usd += actual_usd
+        self.calls += 1
+        self.consecutive_failures = self.consecutive_failures + 1 if failed else 0
+
+    def describe(self) -> str:
+        return (
+            f"{self.spent_usd:.4f} of {self.limit_usd:.4f} USD over {self.calls} call(s), "
+            f"{self.remaining_usd:.4f} remaining"
+        )
+
+
+def estimate(
+    prompt: str,
+    expected_output_tokens: int,
+    input_per_mtok: float,
+    output_per_mtok: float,
+) -> float:
+    """A cost estimate before the call, from a crude token count.
+
+    Four characters per token is a rough English average and it is deliberately not refined: the
+    estimate exists to keep the guard conservative, and a guard that under-estimates is worse than
+    one that stops slightly early.
+    """
+    input_tokens = max(1, len(prompt) // 4)
+    return (
+        input_tokens * input_per_mtok + expected_output_tokens * output_per_mtok
+    ) / 1_000_000