mache 0.1.0__tar.gz

mache-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022-2026 Andrew Wright
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mache-0.1.0/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.4
+Name: mache
+Version: 0.1.0
+Summary: Read a sharded engine match: the pooled estimate and the ccrl fit
+Author: Andrew Wright
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/aywrite/mache
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest~=9.1; extra == "test"
+Dynamic: license-file
+
+# mache
+
+Measure A CHess Engine.
+
+A collection of tools for measuring and benchmarking chess engines. From mache
+(μάχη), Greek for battle.
+
+## Why this exists
+
+[OpenBench](https://github.com/AndyGrant/OpenBench) is how engine testing is
+normally done: an instance hands out tests and client machines attach to it and
+play the games. In almost every case using OpenBench is a far better choice.
+
+This tool exists for two reasons.
+
+The first is that it was not planned. It grew as I wrote
+[arche](https://github.com/aywrite/arche), my first chess engine, starting as a
+basic CI job that got out of hand.
+
+The second is that OpenBench needs machines. There is a shared instance a good
+many engines develop against, and a dozen or more projects run their own. mache
+is for the case where you have neither. It has no instance and no clients, and
+it runs on the hosted CI runners a repository already gets, as part of the pull
+requests and releases it already runs, so testing an engine costs no machine you
+have to own, administer or ask anyone to lend you. Both play their games with
+fastchess underneath.
+
+Hosted runners are the point of mache and also what make it awkward. They are
+slow, they are noisy, they are shared, and a job is killed at a time limit often
+long before a match worth reading has finished. So a match is split across jobs
+that run at once and pooled afterwards, which is most of what the tools below
+are for, and why they take the care described further down.
+
+## What is here
+
+`mache`, a Python package with four tools:
+
+| Tool | What it answers |
+| --- | --- |
+| `match-estimate` | How much stronger, over the pooled games of every shard. Or, with bounds, whether |
+| `rating-estimate` | Where an engine sits on a published rating scale, from a gauntlet |
+| `match-terminations` | How the games actually ended |
+| `book-slice` | Which openings a shard plays, so that no two shards share one |
+
+`actions/setup`, a composite action that builds
+[fastchess](https://github.com/Disservin/fastchess) at a pinned commit,
+fetches an opening book and checks it against a recorded hash, and puts the
+package on `PYTHONPATH`. Nothing is installed at match time.
+
+## The part that is not obvious
+
+A sharded match is not a long match cut up. Three things have to hold or the
+number it produces is wrong.
+
+**The estimate is over the pool.** fastchess prints one, but only for the
+games its own process played. With five shards that is a fifth of the
+evidence, and averaging five such figures is a different calculation.
+`match-estimate` reads the games themselves.
+
+**The error bar is over pairs, not games.** Under `-repeat` the two games of a
+round are one opening with the colours reversed, so they are one observation.
+Counting them as two understates the spread.
+
+**A sequential test looks only at batch boundaries.** A per-shard SPRT that
+stopped when its own games settled the question would be one look per shard at
+a bound priced for one, on a sample chosen by what it said. Here the shards
+play their slices out with nothing watching and the test is judged once over
+all of them. A run is one batch, and `--prior-pairs` carries its pairs into
+the next, so repeated runs accumulate into one test rather than several.
+
+Openings follow from a seed rather than a shuffle, so a schedule can be played
+again from what the run recorded.
+
+## Reading a rating estimate
+
+`rating-estimate` holds every opponent at its published figure and fits the one
+free parameter, so the figure is the rating at which the expected score equals
+the score actually made.
+
+The `±` is a 95% interval and it describes the games and nothing else. Whether
+one rating can describe the results at all is asked separately: when the
+opponents disagree with each other by more than chance allows, a note says so,
+and the interval is an understatement rather than an estimate.
+
+**A placement against a published list carries a systematic error no number of
+games reduces.** The opponents earned their ratings on other hardware at
+slower time controls. Treat the figure as a placement worth about a hundred
+points either way, not as a rating.
+
+## Install
+
+```
+pip install mache
+```
+
+The engine side needs no install. The action puts the package on the path.
+
+## Status
+
+Alpha. mache is used by [arche](https://github.com/aywrite/arche), which is
+where it was written, and it has not yet been used by an engine that is not
+arche. Until it has, expect the rough edges of a tool with one user.
+
+mache is written with heavy AI assistance.
+
+## Licence
+
+MIT. See [LICENSE](LICENSE).
+
+The generalized log likelihood ratio follows Van den Bergh's note on the
+pentanomial model, written from the note and checked against fastchess: for
+the same pairs the number here is the number it prints. Two test cases are
+fastchess's own, attributed where they are used. fastchess is MIT.

mache-0.1.0/README.md

@@ -0,0 +1,113 @@
+# mache
+
+Measure A CHess Engine.
+
+A collection of tools for measuring and benchmarking chess engines. From mache
+(μάχη), Greek for battle.
+
+## Why this exists
+
+[OpenBench](https://github.com/AndyGrant/OpenBench) is how engine testing is
+normally done: an instance hands out tests and client machines attach to it and
+play the games. In almost every case using OpenBench is a far better choice.
+
+This tool exists for two reasons.
+
+The first is that it was not planned. It grew as I wrote
+[arche](https://github.com/aywrite/arche), my first chess engine, starting as a
+basic CI job that got out of hand.
+
+The second is that OpenBench needs machines. There is a shared instance a good
+many engines develop against, and a dozen or more projects run their own. mache
+is for the case where you have neither. It has no instance and no clients, and
+it runs on the hosted CI runners a repository already gets, as part of the pull
+requests and releases it already runs, so testing an engine costs no machine you
+have to own, administer or ask anyone to lend you. Both play their games with
+fastchess underneath.
+
+Hosted runners are the point of mache and also what make it awkward. They are
+slow, they are noisy, they are shared, and a job is killed at a time limit often
+long before a match worth reading has finished. So a match is split across jobs
+that run at once and pooled afterwards, which is most of what the tools below
+are for, and why they take the care described further down.
+
+## What is here
+
+`mache`, a Python package with four tools:
+
+| Tool | What it answers |
+| --- | --- |
+| `match-estimate` | How much stronger, over the pooled games of every shard. Or, with bounds, whether |
+| `rating-estimate` | Where an engine sits on a published rating scale, from a gauntlet |
+| `match-terminations` | How the games actually ended |
+| `book-slice` | Which openings a shard plays, so that no two shards share one |
+
+`actions/setup`, a composite action that builds
+[fastchess](https://github.com/Disservin/fastchess) at a pinned commit,
+fetches an opening book and checks it against a recorded hash, and puts the
+package on `PYTHONPATH`. Nothing is installed at match time.
+
+## The part that is not obvious
+
+A sharded match is not a long match cut up. Three things have to hold or the
+number it produces is wrong.
+
+**The estimate is over the pool.** fastchess prints one, but only for the
+games its own process played. With five shards that is a fifth of the
+evidence, and averaging five such figures is a different calculation.
+`match-estimate` reads the games themselves.
+
+**The error bar is over pairs, not games.** Under `-repeat` the two games of a
+round are one opening with the colours reversed, so they are one observation.
+Counting them as two understates the spread.
+
+**A sequential test looks only at batch boundaries.** A per-shard SPRT that
+stopped when its own games settled the question would be one look per shard at
+a bound priced for one, on a sample chosen by what it said. Here the shards
+play their slices out with nothing watching and the test is judged once over
+all of them. A run is one batch, and `--prior-pairs` carries its pairs into
+the next, so repeated runs accumulate into one test rather than several.
+
+Openings follow from a seed rather than a shuffle, so a schedule can be played
+again from what the run recorded.
+
+## Reading a rating estimate
+
+`rating-estimate` holds every opponent at its published figure and fits the one
+free parameter, so the figure is the rating at which the expected score equals
+the score actually made.
+
+The `±` is a 95% interval and it describes the games and nothing else. Whether
+one rating can describe the results at all is asked separately: when the
+opponents disagree with each other by more than chance allows, a note says so,
+and the interval is an understatement rather than an estimate.
+
+**A placement against a published list carries a systematic error no number of
+games reduces.** The opponents earned their ratings on other hardware at
+slower time controls. Treat the figure as a placement worth about a hundred
+points either way, not as a rating.
+
+## Install
+
+```
+pip install mache
+```
+
+The engine side needs no install. The action puts the package on the path.
+
+## Status
+
+Alpha. mache is used by [arche](https://github.com/aywrite/arche), which is
+where it was written, and it has not yet been used by an engine that is not
+arche. Until it has, expect the rough edges of a tool with one user.
+
+mache is written with heavy AI assistance.
+
+## Licence
+
+MIT. See [LICENSE](LICENSE).
+
+The generalized log likelihood ratio follows Van den Bergh's note on the
+pentanomial model, written from the note and checked against fastchess: for
+the same pairs the number here is the number it prints. Two test cases are
+fastchess's own, attributed where they are used. fastchess is MIT.

mache-0.1.0/mache/__init__.py

@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2022-2026 Andrew Wright
+
+"""The tools a match is read with: the pooled estimate, the ccrl fit, the
+terminations count and the book slice.
+
+They are one package because they read one thing. The fit owns the two regular
+expressions a fastchess pgn is split into games with, and the estimate reads
+the terminations count as well as the fit.
+"""
+
+__version__ = "0.1.0"
+
+# The version of the --json shape, and the one thing every --json object says
+# about itself. It is 1 because the shape is released: a tagged version is what
+# another repository can pin, and a shape nobody can pin is not a contract.
+# Fields are added from here on. None is removed, and none is given a new
+# meaning under the name it already has. A change that cannot be made that way
+# raises this number, and a reader that finds a number it does not know should
+# say so rather than read the object anyway.
+JSON_FORMAT = 1
+
+
+def tool(command: str) -> dict[str, str]:
+    """What produced a --json object, so a figure can be read back against the
+    version of the tooling that produced it."""
+    return {"name": "mache", "version": __version__, "command": command}

mache-0.1.0/mache/book_slice.py

@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2022-2026 Andrew Wright
+
+"""Print the opening a shard of a match starts at.
+
+The shards of a match play at the same time and are pooled afterwards, so no
+two of them may play the same opening: a position played twice would be counted
+twice and the games would not be the independent sample the estimate reads them
+as. The book is taken in order rather than drawn, and each shard is given a
+slice of its own, which is where its games begin.
+
+The run's first shard starts at a remainder of the seed, so two runs of the same
+size play different regions of a book far larger than either of them uses, and
+each shard after it starts a slice further along. The starts are worked out from
+the seed and the counts alone, so the manifest is enough to play the schedule
+again, without depending on how fastchess draws its own openings.
+
+The index printed is one based, which is what fastchess's `start=` takes.
+"""
+
+import argparse
+import sys
+
+
+def first_opening(openings: int, wanted: int, seed: int) -> int:
+    """Where the first shard begins. The offset leaves room for every shard, so
+    the last one still ends inside the book."""
+    room = openings - wanted
+    if room < 1:
+        return 1
+    return 1 + seed % room
+
+
+def start(openings: int, pairs: int, shards: int, shard: int, seed: int) -> int:
+    """The opening this shard begins at, one based."""
+    return first_opening(openings, pairs * shards, seed) + shard * pairs
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--openings", type=int, required=True, help="what the book has")
+    parser.add_argument("--pairs", type=int, required=True, help="openings per shard")
+    parser.add_argument("--shards", type=int, required=True, help="shards in the run")
+    parser.add_argument("--shard", type=int, required=True, help="which one, from zero")
+    parser.add_argument("--seed", type=int, required=True, help="the run's seed")
+    args = parser.parse_args()
+
+    if min(args.openings, args.pairs, args.shards) < 1 or args.seed < 0:
+        sys.exit("the counts are all at least one and the seed is not negative")
+    if not 0 <= args.shard < args.shards:
+        sys.exit(f"shard {args.shard} is not one of {args.shards}")
+
+    wanted = args.pairs * args.shards
+    if args.openings - wanted < 1:
+        # fastchess reads on around the end of the book, so the match still
+        # plays. It is the shards no longer being disjoint that is worth saying
+        print(
+            f"the book holds {args.openings} openings and the run wants {wanted},"
+            " so the shards repeat each other",
+            file=sys.stderr,
+        )
+    print(start(args.openings, args.pairs, args.shards, args.shard, args.seed))
+
+
+if __name__ == "__main__":
+    main()