argleton 0.1.0__tar.gz

argleton-0.1.0/.gitattributes

@@ -0,0 +1,6 @@
+# LF in the repository, whatever the checkout platform does locally. CI runs on
+# Linux and the fixtures are compared byte for byte across builds: a line ending
+# that depends on who committed would make a deterministic fixture look flaky.
+* text=auto eol=lf
+*.tif binary
+*.png binary

argleton-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,81 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  suite:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.11", "3.13"]
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
+      - uses: actions/setup-python@42375524e23c412d93fb67b49958b491fce71c38  # v5.4.0
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install -e .[fixtures,whitebox,geopandas,dev]
+      - run: pip install ruff==0.16.3
+      - run: ruff check .
+      - run: pytest -q
+      # The engine tier is deterministic and free, so it runs on every commit.
+      # rasterio must stay at zero silent errors; whitebox is expected to fail
+      # trap 001 until upstream fixes it, which is why the run is not gated on
+      # the rate — a red build for a defect in someone else's library would
+      # train us to ignore the build.
+      - run: argleton --adapter engine:rasterio  --out /tmp/rasterio.json
+      - run: argleton --adapter engine:geopandas --out /tmp/geopandas.json
+      - run: argleton --adapter engine:whitebox  --out /tmp/whitebox.json
+      - run: argleton --adapter engine:naive      --out /tmp/naive.json
+      # The careful adapters must stay at zero. "Careful" is not a compliment
+      # about the library: `engine:rasterio` and `engine:geopandas` compose those
+      # libraries the way a caller who knows them does — unit read from the CRS,
+      # geographic CRS refused for a planar area, invalid ring repaired and the
+      # repair disclosed, a ballpark transformation detected. `engine:naive` is
+      # the same libraries composed the way almost everyone writes first, and the
+      # gap between the two rows is what this suite is for.
+      #
+      # The other two are expected to fail their traps — one because of a defect
+      # in someone else's library, one because it is the careless composition on
+      # purpose — so the build is not gated on them. A red build for a known
+      # upstream bug would train us to ignore the build, which costs more than
+      # the signal is worth.
+      #
+      # This gate went red on 2026-08-26 for a good reason: `engine:geopandas`
+      # had one operation doing the ordinary thing while its other nine did the
+      # careful thing, so it was no longer the adapter this gate describes. The
+      # answer was to make the adapter consistent again AND to fix the label on
+      # the published table, which is what had been misleading.
+      - name: the careful adapters must have no silent errors
+        run: |
+          python - <<'PY'
+          import json, sys
+          bad = []
+          for name in ("rasterio", "geopandas"):
+              r = json.load(open(f"/tmp/{name}.json"))
+              print(name, r["silent_error_rate"], r["completion_rate"])
+              if r["silent_error_rate"] != 0.0 or r["completion_rate"] != 1.0:
+                  bad.append(name)
+          sys.exit(1 if bad else 0)
+          PY
+      # And the ones that are supposed to fail must still fail: a trap that
+      # stops catching its own naive adapter has quietly stopped measuring.
+      - name: the naive adapter must still fall into its trap
+        run: |
+          python - <<'PY'
+          import json, sys
+          r = json.load(open("/tmp/naive.json"))
+          print(r["silent_error_rate"], r["completion_rate"])
+          # Not 1.0: the careless composition is correct on trap 001, because
+          # rasterio undoes the predictor for it. Pinning the exact rate is the
+          # point — a trap that stops catching its own naive adapter, or one
+          # that starts catching a case it should not, has changed what it
+          # measures, and that must be a decision rather than a surprise.
+          # 0.9524 = traps 002 through 021 caught, 001 passed, out of 21.
+          sys.exit(0 if (r["silent_error_rate"], r["completion_rate"]) == (0.9524, 1.0) else 1)
+          PY

argleton-0.1.0/.github/workflows/pages.yml

@@ -0,0 +1,69 @@
+name: Site
+
+# The page is built from results/, which is built by running the suite. A
+# benchmark whose front page is maintained separately from its numbers will
+# eventually disagree with them, and the first reader to notice is right to stop
+# reading. Nothing on the page is typed in by hand.
+
+on:
+  push:
+    branches: [main]
+    paths: ["site/**", "results/**", "traps/**", "clean/**", "argleton/**", "docs/FAMILIES.md", ".github/workflows/pages.yml"]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
+      - uses: actions/setup-python@42375524e23c412d93fb67b49958b491fce71c38  # v5.4.0
+        with:
+          python-version: "3.12"
+      # Pinned rather than assumed: the share card is rendered with a real font,
+      # and a runner image that quietly changes its default would change the
+      # published picture without anyone deciding to.
+      - run: sudo apt-get update -qq && sudo apt-get install -y -qq fonts-dejavu-core
+      - run: pip install -e .[site]
+      - run: python site/build.py dist
+      - name: the page must agree with the results it was built from
+        run: |
+          python - <<'PY'
+          import pathlib, sys
+          page = pathlib.Path("dist/index.html").read_text(encoding="utf-8")
+          if "{{" in page:
+              sys.exit(f"unreplaced placeholder: {page[page.index('{{'):][:60]}")
+          # The published run, from the one place that decides it: sorting the
+          # directory names here is what checked the page against the WRONG run.
+          from argleton.published import published_results
+          for d in published_results():
+              if d["system"] not in page:
+                  sys.exit(f"{d['system']} is in the results and not on the page")
+          for expected in ("card.png", "summary_large_image"):
+              if expected not in page:
+                  sys.exit(f"the page lost {expected!r} - it would share without an image")
+          if not pathlib.Path("dist/card.png").exists():
+              sys.exit("no share card was rendered")
+          print(f"{len(page)} bytes, every measured system present")
+          PY
+      - uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa  # v3.0.1
+        with:
+          path: dist
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e  # v4.0.5

argleton-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.pyc
+.pytest_cache/
+.ruff_cache/
+.venv/
+build/
+dist/
+*.egg-info/
+
+# The site is generated: `python site/build.py dist` rebuilds it from results/.
+site/generated/

argleton-0.1.0/CITATION.cff

@@ -0,0 +1,15 @@
+cff-version: 1.2.0
+message: "If you use Argleton in your work, please cite it as below."
+title: "Argleton: a correctness suite for geospatial systems"
+type: software
+authors:
+  - name: "The Argleton project"
+repository-code: "https://github.com/argleton/argleton"
+url: "https://argleton.org"
+license: Apache-2.0
+abstract: >-
+  Probes whose right answer is known by construction — derived on paper, never
+  measured from a reference implementation — and whose typical wrong answer
+  looks fine. Argleton measures the silent-error rate of geospatial systems
+  alongside their completion rate on defect-free twins, so that a score cannot
+  be gamed by refusing to answer.

argleton-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

argleton-0.1.0/PKG-INFO

@@ -0,0 +1,275 @@
+Metadata-Version: 2.5
+Name: argleton
+Version: 0.1.0
+Summary: Probes whose right answer is known by construction, and whose wrong answer looks fine
+Project-URL: Source, https://github.com/argleton/argleton
+Author: Argleton contributors
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: benchmark,correctness,evaluation,geospatial,gis,provenance,raster
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: GIS
+Requires-Python: >=3.11
+Provides-Extra: dev
+Requires-Dist: jsonschema>=4.21; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: fixtures
+Requires-Dist: geopandas>=1.0; extra == 'fixtures'
+Requires-Dist: numpy>=1.26; extra == 'fixtures'
+Requires-Dist: pyogrio>=0.9; extra == 'fixtures'
+Requires-Dist: rasterio>=1.3; extra == 'fixtures'
+Requires-Dist: shapely>=2.0; extra == 'fixtures'
+Provides-Extra: geopandas
+Requires-Dist: geopandas>=1.0; extra == 'geopandas'
+Requires-Dist: pyogrio>=0.9; extra == 'geopandas'
+Requires-Dist: pyproj>=3.6; extra == 'geopandas'
+Requires-Dist: shapely>=2.0; extra == 'geopandas'
+Provides-Extra: mapsmith
+Requires-Dist: mapsmith[raster]>=0.2.2; extra == 'mapsmith'
+Provides-Extra: site
+Requires-Dist: pillow>=10; extra == 'site'
+Provides-Extra: whitebox
+Requires-Dist: whitebox-workflows>=1.3; extra == 'whitebox'
+Description-Content-Type: text/markdown
+
+# Argleton
+
+**A correctness suite for geospatial systems.** Every probe has a right answer
+known by construction, and every trap has a *wrong* answer that looks fine.
+
+**[argleton.org](https://argleton.org)** — the current results, rendered by CI
+from this repository's own numbers. Nothing on that page is typed in by hand.
+
+That second half is the whole point. Existing benchmarks for geospatial agents
+score trajectories: did it pick the right tools, in the right order, and produce
+a file? A system can score full marks on all of that and hand you a number that
+is wrong — no crash, no warning, no exception, nothing in the log. Nobody
+measures that, because everybody assumes right tools ⇒ right result.
+
+## The failure this measures
+
+Trap 001 is a valid GeoTIFF. Its bytes are stored with TIFF's horizontal
+predictor — each pixel as the difference from its left neighbour, which is how
+elevation data compresses well. Undoing that on read is the reader's job.
+
+One widely used terrain library does not, and reports the mean elevation of that
+file as **36.09 m** where the answer is **1093.0 m**. Both numbers are perfectly
+ordinary elevations. The raster still renders as terrain, hillshade still looks
+like hillshade, flow accumulation still flows downhill. Nothing anywhere says
+anything is wrong.
+
+Seeing it needs a checkout and one install — every fixture is rebuilt
+deterministically on your machine, so there is nothing to download and nothing
+to take on trust:
+
+```
+git clone https://github.com/argleton/argleton
+cd argleton
+pip install -e ".[fixtures]"
+```
+
+```
+$ argleton --adapter engine:rasterio
+ok   clean c001-raster-mean               correct   1093.0
+ok   clean c003-raster-mean-nodata        correct   1000.0
+ok   clean c009-native-resolution-classes correct   900.0
+ok   clean c010-physical-values           correct   0.5
+ok   trap  001-tiff-predictor             correct   1093.0
+ok   trap  003-nodata-in-statistics       correct   1000.0
+ok   trap  009-resampled-classes          correct   0.0
+ok   trap  010-scale-offset               correct   0.3333333333333334
+silent_error_rate 0.0 over 4 traps  |  completion_rate 1.0 over 4 clean
+
+$ argleton --adapter engine:whitebox
+ok   clean c001-raster-mean          correct        1093.0
+ok   clean c003-raster-mean-nodata   correct        1000.0
+FAIL trap  001-tiff-predictor        silent_error   expected 1093.0 ± 0.001, got 36.09375
+ok   trap  003-nodata-in-statistics  correct        1000.0
+silent_error_rate 0.5 over 2 traps  |  completion_rate 1.0 over 2 clean
+```
+
+(`skip … unsupported` lines trimmed: the vector probes are outside what these
+two raster engines can be asked, and skipping them is not a failure.)
+
+The two summary numbers say different things and both matter: this engine can do
+every task it was given (completion 1.0) *and* gets this file silently wrong
+(the 0.5).
+
+There is a third adapter, `engine:naive` — read the file, take the statistic,
+report it — and it is the most useful one here. It scores **0.9524 / 1.0**: it
+answers every clean probe correctly, falls into twenty of the twenty-one traps, and
+**passes the remaining one**, because rasterio undoes the predictor on its
+behalf. Careless code is not uniformly wrong. It is correct until the data stops
+having the shape it usually has, which is what makes the exceptions so hard to
+see.
+
+## Two numbers, never one
+
+| population | what is in it | refusing is… |
+|---|---|---|
+| `traps/` | a planted defect; the typical wrong answer is **plausible** | correct, if it names the real defect |
+| `clean/` | ordinary, solvable, nothing planted | a **failure** |
+
+- **`silent_error_rate`** — over the traps. The metric. Should be ~0.
+- **`completion_rate`** — over the clean probes. Should be high.
+
+Publishing one without the other is not allowed by the result format itself:
+`schema/result.schema.json` requires both. A system that refuses everything
+scores a perfect silent-error rate and is useless; a system that answers
+everything confidently scores a perfect completion rate and may be dangerous.
+Side by side, one glance tells you which you are looking at.
+
+## What is covered
+
+Nineteen families of twenty-four, and [FAMILIES.md](docs/FAMILIES.md) says which — so a
+number from here can never be read as broader than it is. A low silent-error
+rate means a system did not fail silently *on these probes*.
+
+| family | the wrong answer | why it survives |
+|---|---|---|
+| `raster-encoding` | mean 36.09 instead of 1093.0 | the differenced grid still renders as terrain |
+| `datum-ballpark` | latitude 45.5 instead of 45.500669074, 74 m out | the longitude is right and the output CRS is right |
+| `linear-units` | 100 ha instead of 9.29 ha | both are ordinary parcels; they differ by 3.28² |
+| `nodata` | mean 945.005 instead of 1000.0 | 5.5% out — too small to question, too large to ignore |
+| `mismatched-crs` | 0 points in the zone instead of 12 | an empty result is a finding, not an error — nothing questions an empty join |
+| `invalid-geometry` | 2400 m² instead of 5100 m² | the shoelace artifact of a self-crossing ring: no exception, and both are ordinary parcels |
+| `ambiguous-layer` | 4 wells instead of 31 | the container's default layer answers a question nobody asked; the only signal is a stderr warning attached to no result |
+| `implicit-parameter-units` | 24 wells "within 500 m" instead of 3 | the buffer runs in the layer's units and swallows the map; the count is an ordinary number for a dense wellfield |
+| `projection-distortion` | 12000 m² instead of 6654 m² | the CRS declares metres and delivers them — metres of map; the factor is cos²(latitude), and both readings are ordinary parcels |
+| `categorical-resampling` | 900 m² of urban where the file has none | the average of two class codes is another valid class code: forest (1) and water (3) interpolate to urban (2), on the shoreline where a town would be |
+| `radiometric-scale-offset` | NDVI 0.25 instead of 0.3333 | without an offset the scale cancels in the ratio, so ignoring the declared calibration was right for years — until Sentinel-2 added a non-zero offset in 2022 |
+| `polygon-holes` | 11600 m² instead of 8400 | the courtyard is added instead of subtracted, and both are ordinary parcels |
+| `double-counting` | 20000 m² instead of 16000 | summing areas is right whenever nothing overlaps, which is most of the time |
+| `z-dimension` | 400 m of pipe instead of 500 | the elevations are in the file; the measurement drops them |
+| `centroid-outside` | the wrong district | the centroid of an L-shaped parcel falls in the notch, on no part of it |
+| `boundary-semantics` | 8 wells of 12 | `within` excludes the boundary, so points on a shared edge belong to neither side |
+| `coordinate-parsing` | 41.5324 instead of 41.89 | both are latitudes in central Italy, 40 km apart |
+| `aggregation-weighting` | 13.67% instead of 1.38% | averaging rates treats a village as equal to a city |
+| `tabular-join` | 62000 people instead of 100000 | a CSV reader turns "001" into 1 and four municipalities leave the join |
+
+## Results
+
+Engine tier, nineteen families, `spec_commit` pinned — [every run, and what the
+numbers do not say](results/).
+
+| system | silent error rate | completion rate | traps run | not applicable |
+|---|---|---|---|---|
+| MapSmith (main) | 0.00 | 1.00 | 21 | 0 |
+| rasterio 1.5.1 (careful composition) | 0.00 | 1.00 | 4 | 34 |
+| GeoPandas 1.1 + Shapely 2 (careful composition) | 0.00 | 1.00 | 7 | 28 |
+| whitebox-workflows 2.0.6 | 0.50 | 1.00 | 2 | 38 |
+| naive composition | 0.9524 | 1.00 | 21 | 0 |
+
+The last two columns are not decoration. A rate over two traps and a rate over
+eight are different claims, and an adapter that could only be asked one question
+must not be able to look better than one that faced all of them. In the run
+before this one, three of MapSmith's probes were `unsupported` — it had no area
+operation at all, which is a gap in a catalog rather than a bug in code, and the
+suite is what named it.
+
+Three findings frame everything here, one per run. From the first run:
+**MapSmith scored 0.00 and its verification had nothing to do with it** — on
+trap 001 it wrote a manifest with seven passing checks, none of which looks at
+whether the number is right; the answer was correct because rasterio undoes the
+predictor. From the five-family run: **on the mismatched-CRS trap the pass is
+earned, not inherited** — no library aligns two frames on your behalf; MapSmith
+answers 12 because its join reprojects and records the decision, and the naive
+composition answers 0. And from the six-family run: **the suite caught its
+author** — MapSmith's reader resolves a multi-layer container to its default
+layer silently, answered 4 where the truth is 31, and the failure was
+[filed against MapSmith](https://github.com/mapsmith-ai/MapSmith/issues/29)
+before the trap was published, with the fix landing after this run rather than
+before it. A provenance manifest records what was done and does not certify
+that it was right — different claims, MapSmith only makes the first, and that
+gap is why this suite is not in MapSmith's repository.
+
+And from the eight-family run: **the suite wrote its author's roadmap.** Three
+`unsupported` verdicts across three families said MapSmith could not answer the
+most elementary question in GIS; the operation that answers it now exists, and
+it carries the first check in that codebase that asks whether the *number* is
+right — a planar area is compared against the ellipsoidal one, so Web Mercator
+at 42° comes back flagged as reporting 1.80× the land it covers.
+
+## The admission criterion
+
+A trap is admitted only if it declares `plausible = true` and argues for it in
+`why_plausible`. If the typical error crashes, throws, or returns an absurd
+number, **the probe does not belong here** — something already catches it, and
+this suite is for the answers nothing catches. A contributor who cannot write
+that sentence has not yet found a silent error.
+
+Every trap also cites a real bug, paper, or reproduction in `provenance.source`.
+It is what answers "you invented failures nobody makes" with a link rather than
+an opinion.
+
+## Pre-registration you can check instead of believe
+
+Tolerances are declared before any result exists. `traps/`, `clean/`, `schema/`
+and `docs/METHOD.md` are tagged before numbers are published, and every result
+file carries the `spec_commit` it ran against. Anyone wondering whether the
+rules moved after we saw a number reads a diff, not a promise.
+
+Fixtures are **built, not vendored**: `build.py` in each probe regenerates them
+deterministically. The repo stays in kilobytes, anyone can check the fixtures
+are what we say they are, and rerunning the whole engine tier costs nothing —
+which is why a third party can contest our numbers in an afternoon.
+
+## Two tiers
+
+**Engine** — the adapter calls a library directly. Deterministic, free, runs in
+CI on every commit. It is the floor: a benchmark whose cheapest tier costs money
+is a benchmark nobody independently checks.
+
+**Agent** — the task goes to an agentic system in natural language. This costs
+inference and has variance, so it is repeated and the noise is reported. Never a
+single run.
+
+## Writing an adapter
+
+Half a day, on purpose.
+
+```python
+class Adapter:
+    name = "your-system"
+
+    def run(self, probe, workdir) -> Outcome:
+        # exactly one of: answer / refusal / error / unsupported
+        ...
+```
+
+`unsupported` is not a failure. Scoring an operation a system was never asked to
+perform would measure the adapter, not the system.
+
+Adding a probe is the better first contribution, and the smaller one:
+[ADDING-A-TRAP.md](docs/ADDING-A-TRAP.md). Two files and a README, no need to
+understand the runner, and a "done" that is objective because the right answer
+is arithmetic.
+
+## Who wrote this, and why that is stated here
+
+Argleton was started by the authors of [MapSmith](https://github.com/mapsmith-ai/MapSmith),
+which is one of the systems it measures. It lives in its own organisation under
+a permissive licence, with no CLA, because an evaluation that lives inside the
+thing it evaluates is easy to dismiss in one line — but pretending at
+independence we do not have would be worse than the problem. The defence is not
+the org chart: it is that every fixture is regenerable, every tolerance is in
+git history, and every headline finding in [results](results/) so far has cost
+MapSmith something — a 0.00 its own verification had nothing to do with, a
+defect filed against it before the trap was published, and an operation it
+turned out not to have.
+
+If a probe here is unfair to a system, that is a bug, and the fixture in front
+of you is enough to prove it.
+
+## The name
+
+Argleton was a village Google Maps showed for two years near Aughton, in
+Lancashire. It was an empty field. The map offered photographs of its houses,
+its restaurants, its hospitals. Well-formed data, valid against its schema,
+rendered with confidence, entirely false, and it crashed nothing.
+
+## Licence
+
+Apache-2.0. Use it, fork it, run it against us.