mapsmith 0.7.1__tar.gz → 0.9.0__tar.gz

{mapsmith-0.7.1 → mapsmith-0.9.0}/.gitignore

@@ -33,6 +33,7 @@ data/
 tests/data/*
 !tests/data/manifest_spec_validator.py
 !tests/data/manifest-v1.schema.json
+!tests/data/manifest_spec_multi_file_digest.py
 # The discovery benchmark: 95 queries written by a second model from work
 # scenarios, never from this catalog. It is tracked BECAUSE it is a dataset — an
 # evaluation set nobody can inspect is a claim, not a measurement.

{mapsmith-0.7.1 → mapsmith-0.9.0}/.zenodo.json

@@ -1,7 +1,7 @@
 {
   "upload_type": "software",
   "title": "MapSmith: geospatial processing over MCP, with verifiable provenance",
-  "version": "0.7.1",
+  "version": "0.9.0",
   "creators": [
     {"name": "The MapSmith project"}
   ],

{mapsmith-0.7.1 → mapsmith-0.9.0}/CHANGELOG.md

@@ -6,7 +6,272 @@ All notable changes to MapSmith are documented here, in the format of
 
 ## [Unreleased]
 
-Nothing yet.
+## [0.9.0] - 2026-10-08
+
+Records still declare `spec_version` `1.0.0-draft.10`, and no key is renamed.
+Three keys of MapSmith's own are new, each declared with the reason it is not
+a synonym of anything the specification has: `x-mapsmith:areas_measured_on` in
+`crs_decisions`, from `apportion_by_area`, and `x-mapsmith:ballpark_share` and
+`x-mapsmith:outside_area_of_use` inside `crs_decisions.transformation`. Three
+check names are new, all from `apportion_by_area`:
+`x-mapsmith:extensive_total_preserved`, `x-mapsmith:every_source_value_placed`
+and `x-mapsmith:no_zone_gives_more_than_it_holds`. **What
+`reproject_layer` writes changes in two cases**, both described under *Fixed*:
+coordinates outside every published operation's area are no longer moved by
+an operation from elsewhere, and a geographic CRS counted from another
+meridian whose operations are published only for its Greenwich twin
+(EPSG:4806, Monte Mario from Rome) goes through that twin. Shapely 2.1 and
+pyarrow 23.0.1 or later are required.
+
+### Added
+
+- **`apportion_by_area`: move counts and rates from the zones they are reported
+  in to zones that cross them.** Cases by municipality to health districts,
+  census-tract population to school catchments. Each column is declared
+  extensive (a count, split by share of area) or intensive (a rate, averaged by
+  area); a call declaring neither is refused, because splitting a rate or
+  averaging a count gives a plausible wrong number. Each count comes with
+  `_min` and `_max` that hold however it is distributed inside the source
+  zones; rate bounds rest on the same uniformity the estimate assumes, and the
+  record says which is which. Zones are cut in the source zones' own CRS,
+  where a boundary both layers share is the same line, and a zone counts as
+  wholly inside a target by a topological test, not by a ratio of areas.
+  Shares are ratios of areas measured on an equal-area surface -- the source
+  CRS when it is one, otherwise a plane centred on the zones on their own
+  datum, named in the record -- so the pieces of each zone add up to it.
+  Whether targets overlap is decided zone by zone, so a few square metres of
+  overlapping boundaries are said in the record where they put a zone's value
+  in two targets, and a check fails if a zone with no overlap gives out more
+  than it holds. Invalid zones are repaired and the repair recorded; a zone
+  the repair leaves with no area or with a coordinate that is not a number,
+  zones spread over more than a hemisphere, layers writing longitude in two
+  conventions, and a target that the source CRS would tear at one of its
+  seams (the 180th meridian, a pole: caught by area and by position, whatever
+  the units) are refused; targets keep their own CRS; and the counts written
+  are checked against the pieces they came from.
+
+### Fixed
+
+- **`crs_decisions.transformation` now describes what PROJ did to the data's
+  own coordinates.** Which operation PROJ selects depends on where a
+  coordinate is, and the record asked at one point chosen from the source CRS
+  alone -- for EPSG:4326, the middle of the world, in the Gulf of Guinea. It
+  was wrong in both directions: WGS 84 data in Europe moved to EPSG:3035, or in
+  England to EPSG:27700, was recorded as a ballpark (`is_ballpark: true`) where
+  PROJ applied a published 1 m or 2 m shift; and NAD27 coordinates in Italy were
+  recorded as a 7 m shift where PROJ applied none -- the error a ballpark record
+  exists to report, hidden by it. Every operation that moves an input now hands
+  over the moved layer, PROJ is asked what it did at a spread of the data's own
+  points, inside and outside the areas of use of the operations it has -- outside
+  them it applies no shift for some pairs (NAD27) and stretches a published
+  operation for others (DHDN, the Swiss grid), so the answer is PROJ's and not
+  inferred. A layer straddling the edge of a datum grid is recorded as a
+  ballpark with `x-mapsmith:ballpark_share` saying how many of its vertices got
+  no shift; `x-mapsmith:outside_area_of_use` says how many lie outside the area
+  of every operation published for the pair, whatever PROJ then did with them;
+  a move of no coordinates at all records that nothing was transformed; and a
+  check with no data to hand asks where the two CRSs' areas of use overlap.
+- **`reproject_layer` no longer applies an operation outside its area of use,
+  and its output changes where it did.** When PROJ's own choice at its probe
+  point was a ballpark it substituted the first published operation for the
+  pair, wherever the data was: WGS 84 data in Madrid moved into EPSG:3003 by
+  the Italian operation came out 84.5 m from PROJ's own result, and WGS 84
+  data in Italy moved into EPSG:27700 by the British one, 248 m. Now it
+  substitutes only an operation whose area of use contains every vertex, and
+  otherwise leaves each point where PROJ does. A geographic CRS counted from
+  another meridian whose operations are published only for its Greenwich twin
+  goes through that twin -- EPSG:4806 (Monte Mario, Rome) and EPSG:4819 (Nord
+  Sahara 1959, Paris) among the 23 tried; NTF Paris, EPSG:4807, has an
+  operation of its own and does not change. A station in Piedmont on EPSG:4806
+  now lands on the published 4 m operation's answer, where it was 6.3 m off.
+- **`better_available_m` names only an operation that covers the data and is
+  not installed.** It reported any published operation for the pair: EPSG:4806
+  data in Piedmont was told a 44 m operation needed its grid installed, when
+  that operation needs none and is valid in Sardinia only.
+- **A record written on a failing path no longer claims an earlier run's
+  output.** With a dataset already at the output path, a run refused by its
+  preconditions, or one that crashed before writing, recorded the digest of
+  the earlier bytes as its own output -- and a lineage walk by digest then
+  attributed them to the inputs of the run that failed -- and its record
+  replaced theirs, leaving them with none. Such a record now omits `output`
+  and says why, and where a record on disk still describes the bytes at the
+  path it is left in place and the failing run's record is written beside it
+  as `<output>.failed.provenance.json`, which the error names. Bytes the
+  failing run did write keep their digest, including when a check of the
+  output crashes after the write.
+
+### Changed
+
+- **Shapely 2.1 or later is required** (was 2.0): `apportion_by_area` uses its
+  maximum inscribed circle to check that a reprojected target still holds its
+  own interior.
+- **pyarrow 23.0.1 or later is required** (was 21), the release fixing
+  CVE-2026-25087, a use-after-free reading Arrow IPC files; MapSmith reads none,
+  and the floor keeps an install at the minimum from carrying it. pytest 9.0.3
+  or later for the test extra.
+
+## [0.8.0] - 2026-10-05
+
+Records declare `spec_version` `1.0.0-draft.10`. `draft.9` defines the digest
+of a dataset made of several files, which is what the first fix below
+implements; `draft.10` gives each input a role and a configuration of its own.
+**One key moves**: a second input's own georeferencing facts,
+`inputs[].x-mapsmith:environment` on 0.7.x, are now `inputs[].environment`.
+
+### Added
+
+- **`cluster_points_by_distance`: group points chained together within a
+  distance.** "Group everything within half a kilometre" is single linkage, not
+  thinning: two points share a cluster when a chain of points joins them with
+  every link at most the distance long, so chains merge, and every point is kept
+  with its `cluster_id` and `cluster_size`. `thin_points` keeps one point per
+  crowd, and its count is close to the number of groups but not equal to it. A
+  check re-reads the written file and re-derives the chains by a second
+  algorithm. A geographic CRS is refused, and so are lines, polygons and
+  multipart points, which a point cluster has no single position for.
+- **`list_operations` takes a `geometry`: what the caller's layer holds**, as
+  `point`, `line` or `polygon`, or a list of them for a mixed layer. The
+  catalogue now has 31 operations taking one vector layer to another, one more
+  than the size it hands over whole, so a caller declaring everything it knew
+  got a shortlist back for the most common request there is; with the geometry
+  declared, the right operation is in the answer for 116 of the 118 benchmark
+  requests again. An entry declares the families it accepts only where its code
+  refuses the others (fifteen do), an entry without a declaration is kept for
+  every family, and a test calls each declared entry with every family it
+  leaves out and requires the refusal.
+- **`benchmarks/choice_at_scale.py`**, with every choice it scores in
+  `benchmarks/choice_at_scale.json`: each of the 118 requests handed to Claude
+  Opus 5.5 in sets of 30, 40 and 50 candidates and compared with itself. From
+  40 it chose right 6.9 points less often than from 30, from 50 5.1 points
+  less, so the size `list_operations` hands over whole stays 30, and a test
+  holds it at or below the largest size measured without loss.
+- **Each input of an operation that reads more than one dataset says which
+  argument read it** (`inputs[].argument`, spec `draft.10`). The order of
+  `inputs` is not significant, so until now a weighted zonal statistic with
+  values and weights swapped produced a record that differed only in file
+  names. Eighteen operations name their inputs (twenty code paths:
+  `spatial_join` has three engines), with the names the input checks
+  already use in `verification[].argument`, so the two can be joined; a
+  single-input record is unchanged.
+
+- **`get_lineage` takes a `layer`.** The history of one layer of a file
+  geodatabase now starts from the geodatabase itself: its record sits beside
+  the container (spec section 3.1), and the walk matches the layer at every hop
+  into a container. A layer given for a path that is not a container is
+  refused rather than ignored.
+
+### Changed
+
+- **`get_lineage` steps carry `inputs[].argument`, `inputs[].environment` and
+  `output.layer`** when the records hold them.
+- **A shapefile output whose member files are ambiguous** (`out.dbf` beside
+  `out.DBF`, possible only on a case-sensitive filesystem) is written with a
+  manifest that has no `output` digest and a note saying why, instead of a
+  digest that would choose one of the two.
+- **`inputs[].x-mapsmith:environment` is `inputs[].environment`**, the field
+  spec `draft.10` defines for the configuration that changed how one input was
+  read. Same content; only the name moves, and a reader of 0.7.x records reads
+  the old one.
+- **With more than one input, no file's facts sit on the record's
+  `environment`, the first input's included.** A sidecar beside the first
+  raster was recorded there, which the format reads as true of the whole run,
+  since the order of `inputs` means nothing: in a `clip_raster` it read as true
+  of the mask. It is now on that input's own `environment`, and only process
+  settings stay on the record. Seven operations can meet it (clip_raster,
+  sample_raster_at_points, elevation_profile, least_cost_path, viewshed,
+  watershed, zonal_statistics), and only when a sidecar is present. A
+  single-input record is unchanged.
+
+### Fixed
+
+- **`cluster_points_by_distance` is bounded.** The pairs of points within the
+  distance grow with the square of the points when the distance is large, and
+  6,000 points at a distance of 1e9 took two minutes and 2.3 GB. The pairs are
+  now found in blocks and the call is refused once they pass
+  `MAPSMITH_MAX_SAMPLES`, the operator's limit, before memory is spent.
+  `hot_spots` with `weights="distance_band"` uses the same query: its weights
+  were a double loop over every pair of features in Python, quadratic at any
+  band, and a band that is not a finite positive number is now refused.
+- **`get_lineage` no longer hashes every shapefile in the workspace.** Since
+  the draft.9 digest rule it computed, while indexing, the other rule's digest
+  of every shapefile record's output, whatever the chain asked about. A
+  shapefile is now hashed only when a hop misses on it.
+- **A Windows junction inside a `.gdb` is not followed.** The digest of a
+  directory container walked into junctions, which `is_symlink()` does not
+  report, so files outside the container could enter its digest. Section 3.3 of
+  the specification says links are neither members nor followed. The size used
+  for the re-hash cap now counts the same files as the digest.
+- **A single-layer `.gdb` read by any operation records its layer.** Every
+  operation now reads a single-layer file geodatabase, not only
+  `extract_layer`, and the input records the layer it read: without it, a
+  lineage walk meeting the record of that layer stopped with a false "records
+  claim this container's digest, but for other layers".
+- **`get_lineage` refuses a `layer` holding `:` or a control character.** On
+  Windows `:` in a file name opens an alternate data stream of another file.
+- **`thin_points` refuses a distance that is not a finite positive number**,
+  like `cluster_points_by_distance`; and `least_cost_path` accepts a start or
+  end given as a single-part MultiPoint, which it used to count as no point.
+- **`zonal_statistics` refuses zones that are not polygons.** Line zones were
+  accepted and returned the line's length in pixels as the `count`, with a
+  mean over the cells it crossed: a plausible table, and not a zonal
+  statistic. `count_in_polygons` and `summarize_points_in_polygons` refuse a
+  polygon layer holding anything else the same way. Found by the test that
+  holds the new geometry declarations to the code.
+- **`least_cost_path` is found by a caller holding points.** It reads a cost
+  raster and two point layers but declared its input as raster only, so a
+  caller who said, correctly, that it held a vector layer never saw it. A start
+  or end layer holding a line or a polygon beside its point is now refused; it
+  used to pass with the extra feature ignored.
+- **`snap_layer` snaps to a polygon or multipart reference.** Both crashed with
+  a raw `NotImplementedError` from shapely; every vertex of every part, holes
+  included, is now a target.
+- **A feature with no geometry is refused, with a count, by `thin_points`,
+  `sample_raster_at_points`, `points_along_lines` and `line_intersections`.**
+  They crashed on it with an `AttributeError`, "cannot convert float NaN to
+  integer" and a failed sort.
+- **A shapefile's digest covers its attributes and its CRS.** An input or
+  output shapefile was digested as its `.shp` alone, so a record kept matching
+  a dataset whose `.dbf` (attributes) or `.prj` (CRS) had been rewritten, while
+  it said `sha256` is of the bytes. The digest is now the listing of the
+  `.shp`, `.shx`, `.dbf`, `.prj` and `.cpg` files, as section 3.3 of the
+  specification defines it since draft.9; a file geodatabase input is digested
+  the same way, lock files excluded. `get_lineage` recomputes digests with the
+  same function, so a chain written by this release still resolves. A record
+  written by an earlier release for a shapefile carries the old digest:
+  `get_lineage` compares it under the rule it was written with, so an
+  unchanged shapefile is not reported as edited, and a chain that crosses
+  from a record written by 0.7.1 to one written now still resolves. A record
+  written before this release can only vouch for the `.shp`: an edit to the
+  attributes or the CRS made after it was written stays invisible to it, so
+  such a hop is reported as `reverified_shp_only`, not `reverified`, and a
+  history that contains one is not `verified`.
+- **`extract_layer` works on a file geodatabase.** The catalog listed `.gdb`
+  among its containers, and the input digest opened the path as a file: a
+  geodatabase is a directory, so the operation died before writing anything.
+- **`get_lineage` matches the layer when a digest belongs to a container.**
+  Two records for two layers of one geodatabase share the container's digest
+  (spec section 6, draft.9); a hop into a container now resolves to the record
+  for the layer it names, and stops with `other_layer` rather than attribute
+  one layer to the operation that wrote another. It re-checks a layer's record
+  against the container it sits beside (`<container>.<layer>.provenance.json`,
+  spec section 3.1), and two layers of one container reached on two branches
+  are two histories.
+- **A URI in a manifest path keeps its `//`, so its credentials are masked.**
+  The path normaliser collapsed `https://user:pw@host/x` to `https:/user:pw@…`,
+  and the redaction of URI credentials looks for `://`: the password would have
+  stayed in the record. Not reachable while remote paths are refused, which is
+  the default; found in the 0.7.0 audit. The weights raster of
+  `zonal_statistics` is recorded through the same normaliser now.
+- **A redaction in the environment of a secondary input is declared.** Masked
+  values there did not raise `parameters_redacted`, so the record changed a
+  value without saying so.
+- **The package metadata names the Copernicus NOTICE** beside the AGPL licence,
+  since the source distribution ships the two Copernicus clips the notebook
+  reads.
+- **The links on the PyPI page resolve.** PyPI renders the README with nothing
+  to resolve a relative link against, so every link to a document of the repo
+  and the map-panel screenshot answered 404 there. The build now rewrites them
+  to GitHub URLs pinned to the release tag; on GitHub the README is unchanged.
 
 ## [0.7.1] - 2026-09-26
 

{mapsmith-0.7.1 → mapsmith-0.9.0}/CITATION.cff

@@ -12,8 +12,8 @@ identifiers:
     value: "10.5281/zenodo.22213091"
     description: "Concept DOI — always resolves to the latest release"
 license: AGPL-3.0-or-later
-version: "0.7.1"
-date-released: "2026-09-26"
+version: "0.9.0"
+date-released: "2026-10-08"
 keywords:
   - GIS
   - geospatial

{mapsmith-0.7.1 → mapsmith-0.9.0}/CLAUDE.md

@@ -5,7 +5,7 @@ MapSmith is professional-grade geoprocessing with **verifiable provenance**, rea
 ## Non-negotiable invariants
 
 1. **Deterministic outputs only**: geometry and numbers always come from engine executions, never from a model.
-2. **Provenance on every writer**: any function that writes a dataset emits `<output>.provenance.json` via `ProvenanceRecord` (inputs with sha256, exact parameters, motivated CRS decisions, engine+version).
+2. **Provenance on every writer**: any function that writes a dataset emits `<output>.provenance.json` via `ProvenanceRecord` (inputs with sha256, exact parameters, motivated CRS decisions, engine+version). The digest is `provenance.dataset_sha256` -- for a shapefile or a `.gdb`, the listing digest of spec section 3.3 -- never `sha256_of` on a dataset path, which hashes one file and missed a shapefile's attributes and CRS until 0.7.1.
 3. **Deterministic verification**: the order is the invariant — verify, repair what is mechanically repairable, write the manifest, and *only then* enforce, so the audit trail survives the error instead of being lost with it. `verify.audited(...)` does all four and is how a new writer is written. A minority still open-code the same sequence inline; that is a migration in progress rather than a second sanctioned shape, and two tests hold the line so this sentence does not have to carry a number that rots: one fails if any hand-written writer skips `verify.enforce`, the other ratchets their count downwards only. (It used to say writers go through `audited` "so the invariant cannot be bypassed", which was false for seventeen of fifty-seven — and one of them enforced nothing at all.) Preconditions: `verify.verify_loaded_inputs(...)` (per input: CRS presence is CRITICAL, emptiness is a warning) **before** any CRS alignment, and `verify.verify_input_pairs(...)` (extent overlap) **after** — get that order wrong and a raw pyproj error beats our own message. Never re-read data just to check it. `verify.audit_on_failure(...)` keeps the diagnosis on disk when the engine crashes. Mechanical repair (`make_valid`) is bounded (2 rounds), writes to a temp file and swaps atomically, uses `gdf.geometry.name` (never assumes "geometry"), refuses multi-layer containers and multi-file formats, and lands in `repairs` — in the manifest *and* in the tool result. Anything needing judgement (empty results, CRS choices) is a hinted check in `warnings`, never silently fixed. `on_empty` is `fail` only where emptiness is provably a bug (buffer, reproject), `warn` where it is legitimate but suspicious (clip, join).
 4. **Explicit CRS discipline**: reject inputs without a CRS (helpful message, naming the declaration when the file has one MapSmith will not resolve); never run metric ops on geographic CRS without a recorded reprojection decision. Resolving a CRS and labelling one are different jobs and live apart: `readers.native_crs` returns coordinate systems, `verify.crs_label` returns strings for manifests, and comparisons go through `verify.same_crs` — a label fed back in as a CRS is how a declared CRS got dropped (#28). The keys of `crs_decisions` are the specification's (`analysis_crs`, `source_crs`, `target_crs`, `transformation`, `reason`) or they carry the `x-mapsmith:` prefix that says they are ours — stricter than section 3.5, which only recommends a prefix, and a house rule because a SHOULD with nothing reading it let two operations invent synonyms of `analysis_crs` for a week. `tests/test_verify.py` derives the recommended set from the vendored schema and sweeps every writing site in the source; `tests/test_showcase.py` does the same for every manifest shown on the README or in the gallery, which the source sweep cannot see.
 5. **GPL boundary**: QGIS/GRASS only via subprocess (CLI/files/JSON), never in-process imports.