spatialcv 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,30 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ ## [Unreleased]
6
+
7
+ ## [0.1.0] - 2026-11-08
8
+ ### Added
9
+ - scikit-learn compatible splitters (`split(X, y, groups)`, `get_n_splits`, usable in
10
+ `cross_val_score`, `cross_validate`, `cross_val_predict` and `GridSearchCV`):
11
+ - `SpatialBlockKFold`: square or equal-area hexagonal blocks, random or systematic
12
+ block-to-fold assignment, optional greedy size balancing.
13
+ - `SpatialClusterKFold`: k-means on coordinates (3D unit vectors for lon/lat).
14
+ - `EnvironmentalBlockKFold`: k-means in standardised feature space.
15
+ - `SpatialKFoldWithBuffer`: dead-zone buffer around the test fold of any splitter
16
+ (random K-fold by default).
17
+ - `BufferedLeaveOneOut`: leave-one-out with a dead-zone buffer.
18
+ - `NNDMLeaveOneOut` and `nndm()`: nearest neighbour distance matching LOO
19
+ (Milà et al. 2022).
20
+ - `KNNDMKFold` and `knndm()`: k-fold nearest neighbour distance matching
21
+ (Linnenbrink et al. 2024), Ward hierarchical or k-means clustering.
22
+ - `estimate_autocorrelation_range`, `empirical_variogram`, `fit_variogram_model`:
23
+ Matheron semivariogram and weighted least-squares spherical / exponential fits in pure
24
+ numpy.
25
+ - `fold_diagnostics`: prediction-to-train, CV test-to-train and LOO nearest-neighbour
26
+ distance distributions and the Wasserstein W statistic for any splitter.
27
+ - `plot_folds` (optional matplotlib) and `FoldDiagnostics.plot`, `Variogram.plot`.
28
+ - Planar (`metric="euclidean"`) and lon/lat (`metric="haversine"`, metres) coordinates;
29
+ coordinates from arrays, DataFrame columns or GeoDataFrames.
30
+ - `make_spatial_regression`, `make_prediction_grid`, `gaussian_random_field` synthetic data.
@@ -0,0 +1,21 @@
1
+ cff-version: 1.2.0
2
+ message: "If you use this software, please cite it as below, together with the papers of the methods you use (see README)."
3
+ authors:
4
+ - family-names: "Eminoğlu"
5
+ given-names: "Yusuf"
6
+ email: "yusufeminoglu@gmail.com"
7
+ orcid: "https://orcid.org/0009-0005-6000-2934"
8
+ title: "spatialcv"
9
+ version: "0.1.0"
10
+ date-released: "2026-11-08"
11
+ license: "MIT"
12
+ repository-code: "https://github.com/YusufEminoglu/spatialcv_sdk"
13
+ url: "https://pypi.org/project/spatialcv/"
14
+ abstract: "Honest cross-validation for spatial machine learning: scikit-learn compatible spatial block, cluster, environmental, buffered, NNDM and kNNDM splitters with variogram-based range estimation and nearest-neighbour distance diagnostics."
15
+ keywords:
16
+ - spatial cross-validation
17
+ - spatial autocorrelation
18
+ - GeoAI
19
+ - scikit-learn
20
+ - NNDM
21
+ - kNNDM
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Yusuf Eminoğlu
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,9 @@
1
+ include LICENSE
2
+ include README.md
3
+ include CHANGELOG.md
4
+ include CITATION.cff
5
+ include pyproject.toml
6
+ include src/spatialcv/py.typed
7
+ recursive-include src/spatialcv *.py
8
+ recursive-include tests *.py
9
+ recursive-include examples *.py
@@ -0,0 +1,311 @@
1
+ Metadata-Version: 2.4
2
+ Name: spatialcv
3
+ Version: 0.1.0
4
+ Summary: Honest cross-validation for spatial machine learning: scikit-learn compatible spatial block, cluster, buffered, NNDM and kNNDM splitters.
5
+ Author-email: Yusuf Eminoğlu <yusufeminoglu@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/YusufEminoglu/spatialcv_sdk
8
+ Project-URL: Repository, https://github.com/YusufEminoglu/spatialcv_sdk
9
+ Project-URL: Issues, https://github.com/YusufEminoglu/spatialcv_sdk/issues
10
+ Project-URL: Documentation, https://github.com/YusufEminoglu/spatialcv_sdk#readme
11
+ Project-URL: Changelog, https://github.com/YusufEminoglu/spatialcv_sdk/blob/main/CHANGELOG.md
12
+ Keywords: spatial-cross-validation,cross-validation,spatial-autocorrelation,geoai,machine-learning,scikit-learn,block-cross-validation,nndm,knndm,variogram,gis,remote-sensing,geostatistics,model-evaluation
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3.9
16
+ Classifier: Programming Language :: Python :: 3.10
17
+ Classifier: Programming Language :: Python :: 3.11
18
+ Classifier: Programming Language :: Python :: 3.12
19
+ Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Operating System :: OS Independent
21
+ Classifier: Topic :: Scientific/Engineering :: GIS
22
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
23
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
24
+ Classifier: Intended Audience :: Science/Research
25
+ Classifier: Intended Audience :: Developers
26
+ Classifier: Typing :: Typed
27
+ Requires-Python: >=3.9
28
+ Description-Content-Type: text/markdown
29
+ License-File: LICENSE
30
+ Requires-Dist: numpy>=1.21.0
31
+ Requires-Dist: scikit-learn>=1.0.0
32
+ Provides-Extra: plot
33
+ Requires-Dist: matplotlib>=3.5.0; extra == "plot"
34
+ Provides-Extra: geo
35
+ Requires-Dist: geopandas>=0.12.0; extra == "geo"
36
+ Requires-Dist: matplotlib>=3.5.0; extra == "geo"
37
+ Provides-Extra: dev
38
+ Requires-Dist: pytest>=7.0.0; extra == "dev"
39
+ Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
40
+ Requires-Dist: ruff>=0.1.0; extra == "dev"
41
+ Requires-Dist: mypy>=1.0.0; extra == "dev"
42
+ Requires-Dist: pandas>=1.3.0; extra == "dev"
43
+ Dynamic: license-file
44
+
45
+ # spatialcv
46
+
47
+ **Honest cross-validation for machine learning on spatial data.**
48
+
49
+ `spatialcv` provides scikit-learn compatible splitters that keep spatially autocorrelated
50
+ neighbours out of each test fold's training data, plus the tools to choose a block size or
51
+ buffer and to check whether your cross-validation is as hard as the real prediction task.
52
+
53
+ ```bash
54
+ pip install spatialcv # numpy + scikit-learn
55
+ pip install "spatialcv[plot]" # + matplotlib for fold maps
56
+ pip install "spatialcv[geo]" # + geopandas (GeoDataFrame input) and matplotlib
57
+ ```
58
+
59
+ Python 3.9+. Every splitter has `split(X, y, groups)` and `get_n_splits()`, so it plugs into
60
+ `cross_val_score`, `cross_validate`, `cross_val_predict`, `GridSearchCV`,
61
+ `RandomizedSearchCV` and pipelines unchanged.
62
+
63
+ ## Why spatial cross-validation?
64
+
65
+ Nearby observations are similar (Tobler's first law). Random K-fold puts a sample's
66
+ neighbours in the training set, so a flexible model (random forest, gradient boosting,
67
+ a neural network, anything that sees coordinates or spatially smooth covariates) can
68
+ "look up" the answer instead of learning a transferable relationship. The CV score then
69
+ reflects interpolation between close neighbours, while the map is used far from any
70
+ sample. Large-scale mapping studies have reported random-CV accuracies that collapse
71
+ under spatial validation (Ploton et al. 2020).
72
+
73
+ Spatial CV makes the test data *as far from the training data as the prediction
74
+ locations are*. The splitters here do this in different ways:
75
+
76
+ | Splitter | Idea | Test sets | Key parameters | Reference |
77
+ |---|---|---|---|---|
78
+ | `SpatialBlockKFold` | square or hexagonal blocks are held out whole; random, balanced or systematic block-to-fold assignment | partition | `block_size`, `shape`, `assignment`, `balance` | Roberts et al. 2017; Valavi et al. 2019 |
79
+ | `SpatialClusterKFold` | k-means on coordinates (on the sphere for lon/lat); clusters are held out | partition | `n_clusters`, `balance` | Brenning 2012 |
80
+ | `EnvironmentalBlockKFold` | k-means in standardised *feature* space; tests extrapolation to new conditions | partition | `features`, `standardize` | Roberts et al. 2017; Valavi et al. 2019 |
81
+ | `SpatialKFoldWithBuffer` | any K-fold (random by default) plus a dead zone: training points within `buffer` of a test point are dropped | partition | `buffer`, `cv` | Pohjankukka et al. 2017 |
82
+ | `BufferedLeaveOneOut` | leave-one-out with a circular dead zone around each test point | single samples | `buffer` | Le Rest et al. 2014 |
83
+ | `NNDMLeaveOneOut` | LOO whose training sets are thinned until the CV nearest-neighbour distance distribution matches prediction-to-sample distances | single samples | `prediction_points`, `phi`, `min_train` | Milà et al. 2022 |
84
+ | `KNNDMKFold` | k-fold version of NNDM: clusters are merged into folds to minimise the Wasserstein distance W between those distributions | partition | `prediction_points`, `n_splits`, `maxp`, `clustering` | Linnenbrink et al. 2024 |
85
+
86
+ Helpers:
87
+
88
+ | Function | Purpose |
89
+ |---|---|
90
+ | `estimate_autocorrelation_range` | empirical semivariogram + fitted spherical / exponential model (pure numpy); gives a block size or buffer |
91
+ | `fold_diagnostics` | nearest-neighbour distance distributions: prediction→train (`Gij`), CV test→train (`Gjstar`), LOO (`Gj`) and the W statistic, for *any* splitter |
92
+ | `plot_folds` | map of folds, or of train / test / excluded points of one split (matplotlib) |
93
+ | `nndm`, `knndm` | the NNDM / kNNDM algorithms as functions returning folds and distances |
94
+ | `make_spatial_regression`, `make_prediction_grid` | synthetic autocorrelated data for experiments |
95
+
96
+ ## Quick start
97
+
98
+ ```python
99
+ import numpy as np
100
+ from sklearn.ensemble import RandomForestRegressor
101
+ from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
102
+
103
+ import spatialcv as scv
104
+
105
+ coords, X, y = scv.make_spatial_regression(400, random_state=42)
106
+ features = np.column_stack([X, coords])
107
+ model = RandomForestRegressor(n_estimators=100, random_state=0, n_jobs=-1)
108
+
109
+ # 1. How far does spatial autocorrelation reach?
110
+ vg = scv.estimate_autocorrelation_range(coords, y, model="auto")
111
+ print(f"range = {vg.effective_range:.0f} m, model = {vg.model}, nugget = {vg.nugget:.2f}")
112
+
113
+ # 2. Random vs spatial K-fold
114
+ random_cv = KFold(5, shuffle=True, random_state=0)
115
+ block_cv = scv.SpatialBlockKFold(5, block_size=vg.effective_range, coords=coords, random_state=0)
116
+ for name, cv in [("random KFold", random_cv), ("SpatialBlockKFold", block_cv)]:
117
+ r2 = cross_val_score(model, features, y, cv=cv, scoring="r2")
118
+ print(f"{name:<18} R2 = {r2.mean():.3f} +/- {r2.std():.3f}")
119
+
120
+ # 3. Hyper-parameter search with spatial folds
121
+ search = GridSearchCV(model, {"max_features": [0.3, 1.0]}, cv=block_cv, scoring="r2")
122
+ search.fit(features, y)
123
+ print("best:", search.best_params_, f"R2 = {search.best_score_:.3f}")
124
+
125
+ # 4. Are the CV test points as far from training data as the prediction grid?
126
+ grid = scv.make_prediction_grid(extent=10_000, n_per_side=40)
127
+ for name, cv in [("random KFold", random_cv), ("SpatialBlockKFold", block_cv)]:
128
+ d = scv.fold_diagnostics(cv, features, prediction_points=grid, coords=coords)
129
+ print(
130
+ f"{name:<18} median test->train {np.median(d.Gjstar):6.0f} m | "
131
+ f"prediction->train {np.median(d.Gij):6.0f} m | W = {d.W:.0f}"
132
+ )
133
+ ```
134
+
135
+ Output (scikit-learn 1.9, numpy 2.5):
136
+
137
+ ```text
138
+ range = 2760 m, model = spherical, nugget = 0.00
139
+ random KFold R2 = 0.835 +/- 0.041
140
+ SpatialBlockKFold R2 = 0.143 +/- 0.299
141
+ best: {'max_features': 0.3} R2 = 0.225
142
+ random KFold median test->train 122 m | prediction->train 694 m | W = 711
143
+ SpatialBlockKFold median test->train 827 m | prediction->train 694 m | W = 201
144
+ ```
145
+
146
+ Random K-fold tests points that are typically 122 m from a training sample, while the map
147
+ is predicted 694 m (median) from the nearest sample: its R² of 0.84 measures something the
148
+ model will not be asked to do.
149
+
150
+ ## Which CV matches the true map error?
151
+
152
+ [`examples/compare_cv.py`](examples/compare_cv.py) generates 300 clustered samples of a
153
+ spatial field together with the *true* values on a 40 × 40 prediction grid over the whole
154
+ 10 km × 10 km area, so the real map RMSE is known. Pooled CV predictions
155
+ (`cross_val_predict`) give:
156
+
157
+ ```text
158
+ autocorrelation range of y: 2177 m (spherical)
159
+ true map RMSE: 1.035
160
+
161
+ splitter CV RMSE error W
162
+ KFold (random) 0.499 -0.537 974
163
+ SpatialBlockKFold 1.097 +0.062 482
164
+ SpatialClusterKFold 1.173 +0.138 403
165
+ SpatialKFoldWithBuffer 1.216 +0.180 552
166
+ BufferedLeaveOneOut 1.093 +0.058 760
167
+ NNDMLeaveOneOut 0.946 -0.089 71
168
+ KNNDMKFold 1.210 +0.175 341
169
+ ```
170
+
171
+ Random CV halves the true error. The spatial methods land within ±0.2, and NNDM has by far
172
+ the best-matched distance distribution (lowest W). Spatial CV is not automatically
173
+ "correct" either: strategies that hold out more than the prediction task requires are
174
+ pessimistic (Wadoux et al. 2021), which is why matching the distance distributions
175
+ (NNDM / kNNDM) and checking them with `fold_diagnostics` is worthwhile.
176
+
177
+ ## Coordinates
178
+
179
+ Splitters look up sample coordinates in this order:
180
+
181
+ 1. `coords=`: an `(n_samples, 2)` array, or a GeoSeries / GeoDataFrame (point
182
+ coordinates, or centroids for other geometries);
183
+ 2. `coord_columns=`: two column names (DataFrame) or positions inside `X`;
184
+ 3. `X` itself when it is a GeoDataFrame;
185
+ 4. `X` itself when it has exactly two columns.
186
+
187
+ Units: with `metric="euclidean"` (default) coordinates are planar, e.g. metres in a
188
+ projected CRS, and `block_size` / `buffer` use the same unit. With `metric="haversine"`
189
+ coordinates are `(lon, lat)` in degrees and all distances (`buffer`, `block_size`, `phi`,
190
+ variogram range, diagnostics) are great-circle metres. K-means clustering then runs on 3D
191
+ unit vectors, so clusters are correct across the antimeridian; blocks are laid on a local
192
+ equirectangular projection.
193
+
194
+ ```python
195
+ cv = scv.SpatialKFoldWithBuffer(5, buffer=2_000, coords=lonlat, metric="haversine")
196
+ cv = scv.SpatialClusterKFold(5, coord_columns=["x", "y"]) # columns of a DataFrame X
197
+ cv = scv.SpatialBlockKFold(5, block_size=5_000) # X is a GeoDataFrame
198
+ ```
199
+
200
+ ## Splitter notes
201
+
202
+ **Blocks.** `block_size` should be at least the autocorrelation range. Hexagons have the
203
+ same area as squares of side `block_size`. `assignment="systematic"` numbers grid cells row
204
+ by row and sends cell *i* to fold *i* mod `n_splits` (deterministic); `balance=True`
205
+ (random assignment) gives each block, largest first, to the fold with the fewest samples.
206
+ `block_index(X)` returns the block id of each sample, and every partitioning splitter has
207
+ `assign_folds(X)`.
208
+
209
+ **Buffers.** A training point is dropped when its distance to *any* test point is
210
+ `<= buffer`. Test sets still partition the data, so `cross_val_predict` works; training
211
+ sets shrink, so very large buffers raise an error rather than yield empty training sets.
212
+ Wrap any splitter: `SpatialKFoldWithBuffer(buffer=1000, cv=SpatialBlockKFold(...))`.
213
+
214
+ **NNDM** (Milà et al. 2022) needs the locations where the model will be applied
215
+ (`prediction_points`, e.g. a regular sample of a few thousand points of the area of
216
+ interest). Starting from LOO, it walks through nearest-neighbour distances `r` in
217
+ increasing order and removes test point *j*'s nearest neighbour from its training set
218
+ whenever the LOO distance CDF `Gj*(r)` would still be at or above the prediction CDF
219
+ `Gij(r)`, keeping more than `min_train` of the data and stopping at `phi` (default: the
220
+ largest distance). It follows CAST's reference `nndm()`. Cost is O(n²) memory.
221
+
222
+ **kNNDM** (Linnenbrink et al. 2024) clusters the samples (Ward hierarchical clustering, or
223
+ k-means) into *q* clusters for a log-spaced grid of *q* between `n_splits` and *n* − 2.
224
+ Clusters with at least *n*/*k* samples become folds on their own; the rest, ordered along
225
+ the first principal component of the coordinates, are dealt round-robin into the remaining
226
+ folds. Configurations with a fold larger than `maxp` are rejected and the one with the
227
+ smallest Wasserstein distance W between `Gj*` and `Gij` wins. If random CV already matches
228
+ (W < 1e-4) random folds are returned. It follows CAST's reference `knndm()`
229
+ (geographical space).
230
+
231
+ ```python
232
+ res = scv.knndm(coords, grid, n_splits=5, random_state=0)
233
+ res.labels, res.n_clusters, res.W, res.method # folds, chosen q, W, "hierarchical"
234
+ res = scv.nndm(coords, grid, phi=3_000)
235
+ res.train_indices[0], res.Gjstar, res.W
236
+ ```
237
+
238
+ ## Diagnostics and plots
239
+
240
+ ```python
241
+ d = scv.fold_diagnostics(cv, X, prediction_points=grid, coords=coords)
242
+ print(d) # medians of Gij / Gjstar / Gj, W and W_random
243
+ d.plot() # ECDFs of the three distance distributions
244
+ scv.plot_folds(cv, X) # colour = test fold
245
+ scv.plot_folds(cv, X, fold=0) # train / test / excluded (buffer) points of split 0
246
+ vg.plot() # empirical semivariogram and fitted model
247
+ ```
248
+
249
+ `fold_diagnostics` works with any object that has `split`, including scikit-learn's own
250
+ splitters, so you can check a `GroupKFold` built from administrative units too.
251
+
252
+ ## Variogram details
253
+
254
+ `estimate_autocorrelation_range(coords, values, model="spherical" | "exponential" | "auto")`
255
+ computes Matheron's estimator in `n_lags` equal-width bins up to `max_lag` (default: a
256
+ third of the bounding-box diagonal) and fits nugget + partial sill + range by weighted
257
+ least squares with weights N(h)/h² (Cressie 1985). The returned `Variogram` has
258
+ `effective_range` (the range for the spherical model, 3 × the range parameter for the
259
+ exponential one, i.e. 95 % of the sill), `nugget`, `partial_sill`, `sill`, the empirical
260
+ `lags`/`gamma`/`counts`, and is callable: `vg(h)`. Use model residuals as `values` when
261
+ possible; above 2000 samples a random subset is used (`max_points`).
262
+
263
+ ## References
264
+
265
+ - Brenning, A. (2012). Spatial cross-validation and bootstrap for the assessment of
266
+ prediction rules in remote sensing: the R package sperrorest. *IEEE IGARSS 2012*,
267
+ 5372–5375. https://doi.org/10.1109/IGARSS.2012.6352393
268
+ - Cressie, N. (1985). Fitting variogram models by weighted least squares. *Mathematical
269
+ Geology* 17, 563–586. https://doi.org/10.1007/BF01032109
270
+ - Le Rest, K., Pinaud, D., Monestiez, P., Chadoeuf, J., Bretagnolle, V. (2014). Spatial
271
+ leave-one-out cross-validation for variable selection in the presence of spatial
272
+ autocorrelation. *Global Ecology and Biogeography* 23, 811–820.
273
+ https://doi.org/10.1111/geb.12161
274
+ - Linnenbrink, J., Milà, C., Ludwig, M., Meyer, H. (2024). kNNDM CV: k-fold
275
+ nearest-neighbour distance matching cross-validation for map accuracy estimation.
276
+ *Geoscientific Model Development* 17, 5897–5912.
277
+ https://doi.org/10.5194/gmd-17-5897-2024
278
+ - Milà, C., Mateu, J., Pebesma, E., Meyer, H. (2022). Nearest neighbour distance matching
279
+ Leave-One-Out Cross-Validation for map validation. *Methods in Ecology and Evolution*
280
+ 13, 1304–1316. https://doi.org/10.1111/2041-210X.13851
281
+ - Ploton, P., et al. (2020). Spatial validation reveals poor predictive performance of
282
+ large-scale ecological mapping models. *Nature Communications* 11, 4540.
283
+ https://doi.org/10.1038/s41467-020-18321-y
284
+ - Pohjankukka, J., Pahikkala, T., Nevalainen, P., Heikkonen, J. (2017). Estimating the
285
+ prediction performance of spatial models via spatial k-fold cross validation.
286
+ *International Journal of Geographical Information Science* 31, 2001–2019.
287
+ https://doi.org/10.1080/13658816.2017.1346255
288
+ - Rahimi, A., Recht, B. (2007). Random features for large-scale kernel machines.
289
+ *Advances in Neural Information Processing Systems* 20 (used by the synthetic data
290
+ generator).
291
+ - Roberts, D. R., et al. (2017). Cross-validation strategies for data with temporal,
292
+ spatial, hierarchical, or phylogenetic structure. *Ecography* 40, 913–929.
293
+ https://doi.org/10.1111/ecog.02881
294
+ - Valavi, R., Elith, J., Lahoz-Monfort, J. J., Guillera-Arroita, G. (2019). blockCV: an R
295
+ package for generating spatially or environmentally separated folds for k-fold
296
+ cross-validation of species distribution models. *Methods in Ecology and Evolution* 10,
297
+ 225–232. https://doi.org/10.1111/2041-210X.13107
298
+ - Wadoux, A. M. J.-C., Heuvelink, G. B. M., de Bruin, S., Brus, D. J. (2021). Spatial
299
+ cross-validation is not the right way to evaluate map accuracy. *Ecological Modelling*
300
+ 457, 109692. https://doi.org/10.1016/j.ecolmodel.2021.109692
301
+
302
+ The NNDM and kNNDM implementations follow the R package CAST (Meyer, Milà, Ludwig,
303
+ Linnenbrink et al.); block and buffer designs follow blockCV and spatialsample.
304
+
305
+ ## Citation
306
+
307
+ See [`CITATION.cff`](CITATION.cff). Please also cite the papers of the methods you use.
308
+
309
+ ## License
310
+
311
+ MIT © Yusuf Eminoğlu
@@ -0,0 +1,267 @@
1
+ # spatialcv
2
+
3
+ **Honest cross-validation for machine learning on spatial data.**
4
+
5
+ `spatialcv` provides scikit-learn compatible splitters that keep spatially autocorrelated
6
+ neighbours out of each test fold's training data, plus the tools to choose a block size or
7
+ buffer and to check whether your cross-validation is as hard as the real prediction task.
8
+
9
+ ```bash
10
+ pip install spatialcv # numpy + scikit-learn
11
+ pip install "spatialcv[plot]" # + matplotlib for fold maps
12
+ pip install "spatialcv[geo]" # + geopandas (GeoDataFrame input) and matplotlib
13
+ ```
14
+
15
+ Python 3.9+. Every splitter has `split(X, y, groups)` and `get_n_splits()`, so it plugs into
16
+ `cross_val_score`, `cross_validate`, `cross_val_predict`, `GridSearchCV`,
17
+ `RandomizedSearchCV` and pipelines unchanged.
18
+
19
+ ## Why spatial cross-validation?
20
+
21
+ Nearby observations are similar (Tobler's first law). Random K-fold puts a sample's
22
+ neighbours in the training set, so a flexible model (random forest, gradient boosting,
23
+ a neural network, anything that sees coordinates or spatially smooth covariates) can
24
+ "look up" the answer instead of learning a transferable relationship. The CV score then
25
+ reflects interpolation between close neighbours, while the map is used far from any
26
+ sample. Large-scale mapping studies have reported random-CV accuracies that collapse
27
+ under spatial validation (Ploton et al. 2020).
28
+
29
+ Spatial CV makes the test data *as far from the training data as the prediction
30
+ locations are*. The splitters here do this in different ways:
31
+
32
+ | Splitter | Idea | Test sets | Key parameters | Reference |
33
+ |---|---|---|---|---|
34
+ | `SpatialBlockKFold` | square or hexagonal blocks are held out whole; random, balanced or systematic block-to-fold assignment | partition | `block_size`, `shape`, `assignment`, `balance` | Roberts et al. 2017; Valavi et al. 2019 |
35
+ | `SpatialClusterKFold` | k-means on coordinates (on the sphere for lon/lat); clusters are held out | partition | `n_clusters`, `balance` | Brenning 2012 |
36
+ | `EnvironmentalBlockKFold` | k-means in standardised *feature* space; tests extrapolation to new conditions | partition | `features`, `standardize` | Roberts et al. 2017; Valavi et al. 2019 |
37
+ | `SpatialKFoldWithBuffer` | any K-fold (random by default) plus a dead zone: training points within `buffer` of a test point are dropped | partition | `buffer`, `cv` | Pohjankukka et al. 2017 |
38
+ | `BufferedLeaveOneOut` | leave-one-out with a circular dead zone around each test point | single samples | `buffer` | Le Rest et al. 2014 |
39
+ | `NNDMLeaveOneOut` | LOO whose training sets are thinned until the CV nearest-neighbour distance distribution matches prediction-to-sample distances | single samples | `prediction_points`, `phi`, `min_train` | Milà et al. 2022 |
40
+ | `KNNDMKFold` | k-fold version of NNDM: clusters are merged into folds to minimise the Wasserstein distance W between those distributions | partition | `prediction_points`, `n_splits`, `maxp`, `clustering` | Linnenbrink et al. 2024 |
41
+
42
+ Helpers:
43
+
44
+ | Function | Purpose |
45
+ |---|---|
46
+ | `estimate_autocorrelation_range` | empirical semivariogram + fitted spherical / exponential model (pure numpy); gives a block size or buffer |
47
+ | `fold_diagnostics` | nearest-neighbour distance distributions: prediction→train (`Gij`), CV test→train (`Gjstar`), LOO (`Gj`) and the W statistic, for *any* splitter |
48
+ | `plot_folds` | map of folds, or of train / test / excluded points of one split (matplotlib) |
49
+ | `nndm`, `knndm` | the NNDM / kNNDM algorithms as functions returning folds and distances |
50
+ | `make_spatial_regression`, `make_prediction_grid` | synthetic autocorrelated data for experiments |
51
+
52
+ ## Quick start
53
+
54
+ ```python
55
+ import numpy as np
56
+ from sklearn.ensemble import RandomForestRegressor
57
+ from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
58
+
59
+ import spatialcv as scv
60
+
61
+ coords, X, y = scv.make_spatial_regression(400, random_state=42)
62
+ features = np.column_stack([X, coords])
63
+ model = RandomForestRegressor(n_estimators=100, random_state=0, n_jobs=-1)
64
+
65
+ # 1. How far does spatial autocorrelation reach?
66
+ vg = scv.estimate_autocorrelation_range(coords, y, model="auto")
67
+ print(f"range = {vg.effective_range:.0f} m, model = {vg.model}, nugget = {vg.nugget:.2f}")
68
+
69
+ # 2. Random vs spatial K-fold
70
+ random_cv = KFold(5, shuffle=True, random_state=0)
71
+ block_cv = scv.SpatialBlockKFold(5, block_size=vg.effective_range, coords=coords, random_state=0)
72
+ for name, cv in [("random KFold", random_cv), ("SpatialBlockKFold", block_cv)]:
73
+ r2 = cross_val_score(model, features, y, cv=cv, scoring="r2")
74
+ print(f"{name:<18} R2 = {r2.mean():.3f} +/- {r2.std():.3f}")
75
+
76
+ # 3. Hyper-parameter search with spatial folds
77
+ search = GridSearchCV(model, {"max_features": [0.3, 1.0]}, cv=block_cv, scoring="r2")
78
+ search.fit(features, y)
79
+ print("best:", search.best_params_, f"R2 = {search.best_score_:.3f}")
80
+
81
+ # 4. Are the CV test points as far from training data as the prediction grid?
82
+ grid = scv.make_prediction_grid(extent=10_000, n_per_side=40)
83
+ for name, cv in [("random KFold", random_cv), ("SpatialBlockKFold", block_cv)]:
84
+ d = scv.fold_diagnostics(cv, features, prediction_points=grid, coords=coords)
85
+ print(
86
+ f"{name:<18} median test->train {np.median(d.Gjstar):6.0f} m | "
87
+ f"prediction->train {np.median(d.Gij):6.0f} m | W = {d.W:.0f}"
88
+ )
89
+ ```
90
+
91
+ Output (scikit-learn 1.9, numpy 2.5):
92
+
93
+ ```text
94
+ range = 2760 m, model = spherical, nugget = 0.00
95
+ random KFold R2 = 0.835 +/- 0.041
96
+ SpatialBlockKFold R2 = 0.143 +/- 0.299
97
+ best: {'max_features': 0.3} R2 = 0.225
98
+ random KFold median test->train 122 m | prediction->train 694 m | W = 711
99
+ SpatialBlockKFold median test->train 827 m | prediction->train 694 m | W = 201
100
+ ```
101
+
102
+ Random K-fold tests points that are typically 122 m from a training sample, while the map
103
+ is predicted 694 m (median) from the nearest sample: its R² of 0.84 measures something the
104
+ model will not be asked to do.
105
+
106
+ ## Which CV matches the true map error?
107
+
108
+ [`examples/compare_cv.py`](examples/compare_cv.py) generates 300 clustered samples of a
109
+ spatial field together with the *true* values on a 40 × 40 prediction grid over the whole
110
+ 10 km × 10 km area, so the real map RMSE is known. Pooled CV predictions
111
+ (`cross_val_predict`) give:
112
+
113
+ ```text
114
+ autocorrelation range of y: 2177 m (spherical)
115
+ true map RMSE: 1.035
116
+
117
+ splitter CV RMSE error W
118
+ KFold (random) 0.499 -0.537 974
119
+ SpatialBlockKFold 1.097 +0.062 482
120
+ SpatialClusterKFold 1.173 +0.138 403
121
+ SpatialKFoldWithBuffer 1.216 +0.180 552
122
+ BufferedLeaveOneOut 1.093 +0.058 760
123
+ NNDMLeaveOneOut 0.946 -0.089 71
124
+ KNNDMKFold 1.210 +0.175 341
125
+ ```
126
+
127
+ Random CV halves the true error. The spatial methods land within ±0.2, and NNDM has by far
128
+ the best-matched distance distribution (lowest W). Spatial CV is not automatically
129
+ "correct" either: strategies that hold out more than the prediction task requires are
130
+ pessimistic (Wadoux et al. 2021), which is why matching the distance distributions
131
+ (NNDM / kNNDM) and checking them with `fold_diagnostics` is worthwhile.
132
+
133
+ ## Coordinates
134
+
135
+ Splitters look up sample coordinates in this order:
136
+
137
+ 1. `coords=`: an `(n_samples, 2)` array, or a GeoSeries / GeoDataFrame (point
138
+ coordinates, or centroids for other geometries);
139
+ 2. `coord_columns=`: two column names (DataFrame) or positions inside `X`;
140
+ 3. `X` itself when it is a GeoDataFrame;
141
+ 4. `X` itself when it has exactly two columns.
142
+
143
+ Units: with `metric="euclidean"` (default) coordinates are planar, e.g. metres in a
144
+ projected CRS, and `block_size` / `buffer` use the same unit. With `metric="haversine"`
145
+ coordinates are `(lon, lat)` in degrees and all distances (`buffer`, `block_size`, `phi`,
146
+ variogram range, diagnostics) are great-circle metres. K-means clustering then runs on 3D
147
+ unit vectors, so clusters are correct across the antimeridian; blocks are laid on a local
148
+ equirectangular projection.
149
+
150
+ ```python
151
+ cv = scv.SpatialKFoldWithBuffer(5, buffer=2_000, coords=lonlat, metric="haversine")
152
+ cv = scv.SpatialClusterKFold(5, coord_columns=["x", "y"]) # columns of a DataFrame X
153
+ cv = scv.SpatialBlockKFold(5, block_size=5_000) # X is a GeoDataFrame
154
+ ```
155
+
156
+ ## Splitter notes
157
+
158
+ **Blocks.** `block_size` should be at least the autocorrelation range. Hexagons have the
159
+ same area as squares of side `block_size`. `assignment="systematic"` numbers grid cells row
160
+ by row and sends cell *i* to fold *i* mod `n_splits` (deterministic); `balance=True`
161
+ (random assignment) gives each block, largest first, to the fold with the fewest samples.
162
+ `block_index(X)` returns the block id of each sample, and every partitioning splitter has
163
+ `assign_folds(X)`.
164
+
165
+ **Buffers.** A training point is dropped when its distance to *any* test point is
166
+ `<= buffer`. Test sets still partition the data, so `cross_val_predict` works; training
167
+ sets shrink, so very large buffers raise an error rather than yield empty training sets.
168
+ Wrap any splitter: `SpatialKFoldWithBuffer(buffer=1000, cv=SpatialBlockKFold(...))`.
169
+
170
+ **NNDM** (Milà et al. 2022) needs the locations where the model will be applied
171
+ (`prediction_points`, e.g. a regular sample of a few thousand points of the area of
172
+ interest). Starting from LOO, it walks through nearest-neighbour distances `r` in
173
+ increasing order and removes test point *j*'s nearest neighbour from its training set
174
+ whenever the LOO distance CDF `Gj*(r)` would still be at or above the prediction CDF
175
+ `Gij(r)`, keeping more than `min_train` of the data and stopping at `phi` (default: the
176
+ largest distance). It follows CAST's reference `nndm()`. Cost is O(n²) memory.
177
+
178
+ **kNNDM** (Linnenbrink et al. 2024) clusters the samples (Ward hierarchical clustering, or
179
+ k-means) into *q* clusters for a log-spaced grid of *q* between `n_splits` and *n* − 2.
180
+ Clusters with at least *n*/*k* samples become folds on their own; the rest, ordered along
181
+ the first principal component of the coordinates, are dealt round-robin into the remaining
182
+ folds. Configurations with a fold larger than `maxp` are rejected and the one with the
183
+ smallest Wasserstein distance W between `Gj*` and `Gij` wins. If random CV already matches
184
+ (W < 1e-4) random folds are returned. It follows CAST's reference `knndm()`
185
+ (geographical space).
186
+
187
+ ```python
188
+ res = scv.knndm(coords, grid, n_splits=5, random_state=0)
189
+ res.labels, res.n_clusters, res.W, res.method # folds, chosen q, W, "hierarchical"
190
+ res = scv.nndm(coords, grid, phi=3_000)
191
+ res.train_indices[0], res.Gjstar, res.W
192
+ ```
193
+
194
+ ## Diagnostics and plots
195
+
196
+ ```python
197
+ d = scv.fold_diagnostics(cv, X, prediction_points=grid, coords=coords)
198
+ print(d) # medians of Gij / Gjstar / Gj, W and W_random
199
+ d.plot() # ECDFs of the three distance distributions
200
+ scv.plot_folds(cv, X) # colour = test fold
201
+ scv.plot_folds(cv, X, fold=0) # train / test / excluded (buffer) points of split 0
202
+ vg.plot() # empirical semivariogram and fitted model
203
+ ```
204
+
205
+ `fold_diagnostics` works with any object that has `split`, including scikit-learn's own
206
+ splitters, so you can check a `GroupKFold` built from administrative units too.
207
+
208
+ ## Variogram details
209
+
210
+ `estimate_autocorrelation_range(coords, values, model="spherical" | "exponential" | "auto")`
211
+ computes Matheron's estimator in `n_lags` equal-width bins up to `max_lag` (default: a
212
+ third of the bounding-box diagonal) and fits nugget + partial sill + range by weighted
213
+ least squares with weights N(h)/h² (Cressie 1985). The returned `Variogram` has
214
+ `effective_range` (the range for the spherical model, 3 × the range parameter for the
215
+ exponential one, i.e. 95 % of the sill), `nugget`, `partial_sill`, `sill`, the empirical
216
+ `lags`/`gamma`/`counts`, and is callable: `vg(h)`. Use model residuals as `values` when
217
+ possible; above 2000 samples a random subset is used (`max_points`).
218
+
219
+ ## References
220
+
221
+ - Brenning, A. (2012). Spatial cross-validation and bootstrap for the assessment of
222
+ prediction rules in remote sensing: the R package sperrorest. *IEEE IGARSS 2012*,
223
+ 5372–5375. https://doi.org/10.1109/IGARSS.2012.6352393
224
+ - Cressie, N. (1985). Fitting variogram models by weighted least squares. *Mathematical
225
+ Geology* 17, 563–586. https://doi.org/10.1007/BF01032109
226
+ - Le Rest, K., Pinaud, D., Monestiez, P., Chadoeuf, J., Bretagnolle, V. (2014). Spatial
227
+ leave-one-out cross-validation for variable selection in the presence of spatial
228
+ autocorrelation. *Global Ecology and Biogeography* 23, 811–820.
229
+ https://doi.org/10.1111/geb.12161
230
+ - Linnenbrink, J., Milà, C., Ludwig, M., Meyer, H. (2024). kNNDM CV: k-fold
231
+ nearest-neighbour distance matching cross-validation for map accuracy estimation.
232
+ *Geoscientific Model Development* 17, 5897–5912.
233
+ https://doi.org/10.5194/gmd-17-5897-2024
234
+ - Milà, C., Mateu, J., Pebesma, E., Meyer, H. (2022). Nearest neighbour distance matching
235
+ Leave-One-Out Cross-Validation for map validation. *Methods in Ecology and Evolution*
236
+ 13, 1304–1316. https://doi.org/10.1111/2041-210X.13851
237
+ - Ploton, P., et al. (2020). Spatial validation reveals poor predictive performance of
238
+ large-scale ecological mapping models. *Nature Communications* 11, 4540.
239
+ https://doi.org/10.1038/s41467-020-18321-y
240
+ - Pohjankukka, J., Pahikkala, T., Nevalainen, P., Heikkonen, J. (2017). Estimating the
241
+ prediction performance of spatial models via spatial k-fold cross validation.
242
+ *International Journal of Geographical Information Science* 31, 2001–2019.
243
+ https://doi.org/10.1080/13658816.2017.1346255
244
+ - Rahimi, A., Recht, B. (2007). Random features for large-scale kernel machines.
245
+ *Advances in Neural Information Processing Systems* 20 (used by the synthetic data
246
+ generator).
247
+ - Roberts, D. R., et al. (2017). Cross-validation strategies for data with temporal,
248
+ spatial, hierarchical, or phylogenetic structure. *Ecography* 40, 913–929.
249
+ https://doi.org/10.1111/ecog.02881
250
+ - Valavi, R., Elith, J., Lahoz-Monfort, J. J., Guillera-Arroita, G. (2019). blockCV: an R
251
+ package for generating spatially or environmentally separated folds for k-fold
252
+ cross-validation of species distribution models. *Methods in Ecology and Evolution* 10,
253
+ 225–232. https://doi.org/10.1111/2041-210X.13107
254
+ - Wadoux, A. M. J.-C., Heuvelink, G. B. M., de Bruin, S., Brus, D. J. (2021). Spatial
255
+ cross-validation is not the right way to evaluate map accuracy. *Ecological Modelling*
256
+ 457, 109692. https://doi.org/10.1016/j.ecolmodel.2021.109692
257
+
258
+ The NNDM and kNNDM implementations follow the R package CAST (Meyer, Milà, Ludwig,
259
+ Linnenbrink et al.); block and buffer designs follow blockCV and spatialsample.
260
+
261
+ ## Citation
262
+
263
+ See [`CITATION.cff`](CITATION.cff). Please also cite the papers of the methods you use.
264
+
265
+ ## License
266
+
267
+ MIT © Yusuf Eminoğlu