regscan 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- regscan-0.1.0/.github/workflows/ci.yml +16 -0
- regscan-0.1.0/.github/workflows/publish.yml +41 -0
- regscan-0.1.0/.gitignore +12 -0
- regscan-0.1.0/CONTRIBUTING.md +77 -0
- regscan-0.1.0/LICENSE +29 -0
- regscan-0.1.0/PKG-INFO +244 -0
- regscan-0.1.0/README.md +218 -0
- regscan-0.1.0/publish.yml +43 -0
- regscan-0.1.0/pyproject.toml +43 -0
- regscan-0.1.0/src/regscan/__init__.py +43 -0
- regscan-0.1.0/src/regscan/api.py +53 -0
- regscan-0.1.0/src/regscan/config.py +92 -0
- regscan-0.1.0/src/regscan/families.py +235 -0
- regscan-0.1.0/src/regscan/kernel.py +362 -0
- regscan-0.1.0/src/regscan/kernel_state.py +407 -0
- regscan-0.1.0/src/regscan/py.typed +0 -0
- regscan-0.1.0/src/regscan/registry.py +106 -0
- regscan-0.1.0/src/regscan/result.py +42 -0
- regscan-0.1.0/src/regscan/window.py +85 -0
- regscan-0.1.0/tests/_oracle.py +65 -0
- regscan-0.1.0/tests/conftest.py +4 -0
- regscan-0.1.0/tests/test_kernel.py +239 -0
- regscan-0.1.0/tests/test_registry.py +33 -0
- regscan-0.1.0/tests/test_scan.py +78 -0
- regscan-0.1.0/tests/test_window.py +43 -0
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
on: [push, pull_request]
|
|
3
|
+
jobs:
|
|
4
|
+
test:
|
|
5
|
+
runs-on: ubuntu-latest
|
|
6
|
+
strategy:
|
|
7
|
+
matrix:
|
|
8
|
+
python-version: ["3.9", "3.11", "3.12"]
|
|
9
|
+
steps:
|
|
10
|
+
- uses: actions/checkout@v4
|
|
11
|
+
- uses: actions/setup-python@v5
|
|
12
|
+
with:
|
|
13
|
+
python-version: ${{ matrix.python-version }}
|
|
14
|
+
- run: pip install -e ".[dev]"
|
|
15
|
+
- run: ruff check src tests
|
|
16
|
+
- run: pytest -q --cov=regscan
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
name: publish
|
|
2
|
+
|
|
3
|
+
# Publishes to PyPI when a GitHub Release is created. Uses Trusted Publishing
|
|
4
|
+
# (OpenID Connect), so there is no API token to store or rotate: PyPI verifies
|
|
5
|
+
# the workflow's identity directly. Configure the publisher once on PyPI under
|
|
6
|
+
# the project's Publishing settings before the first release.
|
|
7
|
+
|
|
8
|
+
on:
|
|
9
|
+
release:
|
|
10
|
+
types: [published]
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
build:
|
|
14
|
+
runs-on: ubuntu-latest
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v4
|
|
17
|
+
- uses: actions/setup-python@v5
|
|
18
|
+
with:
|
|
19
|
+
python-version: "3.11"
|
|
20
|
+
- run: pip install build twine
|
|
21
|
+
- run: python -m build
|
|
22
|
+
# Fails on malformed metadata or an unrenderable README, which is worth
|
|
23
|
+
# catching here rather than after the upload is irreversible.
|
|
24
|
+
- run: twine check dist/*
|
|
25
|
+
- uses: actions/upload-artifact@v4
|
|
26
|
+
with:
|
|
27
|
+
name: dist
|
|
28
|
+
path: dist/
|
|
29
|
+
|
|
30
|
+
publish:
|
|
31
|
+
needs: build
|
|
32
|
+
runs-on: ubuntu-latest
|
|
33
|
+
environment: pypi
|
|
34
|
+
permissions:
|
|
35
|
+
id-token: write # required for Trusted Publishing
|
|
36
|
+
steps:
|
|
37
|
+
- uses: actions/download-artifact@v4
|
|
38
|
+
with:
|
|
39
|
+
name: dist
|
|
40
|
+
path: dist/
|
|
41
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
regscan-0.1.0/.gitignore
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Development
|
|
2
|
+
|
|
3
|
+
```bash
|
|
4
|
+
pip install -e ".[dev]"
|
|
5
|
+
pytest -q
|
|
6
|
+
ruff check src tests
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
## Adding a family
|
|
10
|
+
|
|
11
|
+
Register it in `regscan/registry.py` with a *factory*, not an import:
|
|
12
|
+
|
|
13
|
+
```python
|
|
14
|
+
register("my_method", factory=lambda: _my_scan, family="F_X", doc="...")
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Nothing is imported until the method is first requested, which keeps
|
|
18
|
+
`import regscan` cheap. `tests/test_registry.py::test_import_is_light` asserts
|
|
19
|
+
that importing the package pulls in no heavy modules; keep it passing.
|
|
20
|
+
|
|
21
|
+
Every scan function has the signature:
|
|
22
|
+
|
|
23
|
+
```python
|
|
24
|
+
fn(x: np.ndarray, w: int, cfg: ScanConfig | None = None) -> tuple[float, tuple[int, int]]
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
returning `(score, (a, b))` with `b` inclusive.
|
|
28
|
+
|
|
29
|
+
## Testing a fast implementation
|
|
30
|
+
|
|
31
|
+
`scan_nwkr` only recomputes indices within `T = truncation * w` of an interval
|
|
32
|
+
boundary, on the argument that every other index has the same fit restricted
|
|
33
|
+
or not. `tests/test_kernel.py::test_matches_naive_implementation` checks that
|
|
34
|
+
against a deliberately naive scan that refits everything, for both kernels and
|
|
35
|
+
several bandwidths, to machine precision. Any faster implementation — such as
|
|
36
|
+
the paper's incremental O(r) window updates — must pass the same test.
|
|
37
|
+
|
|
38
|
+
## Optimisations still open
|
|
39
|
+
|
|
40
|
+
`mean`, `poly_deg*` and `nwkr_*` are all at their intended complexity:
|
|
41
|
+
|
|
42
|
+
- constant family — prefix sums of `y` and `y²`, O(1) per interval;
|
|
43
|
+
- polynomial family — prefix sums of the moments `t^p` and `t^p y`, O(d³) per
|
|
44
|
+
interval regardless of its length;
|
|
45
|
+
- kernel family — incremental `nin`/`din`, inside buffer and `sse_out`, O(r)
|
|
46
|
+
to extend an interval by one sample.
|
|
47
|
+
|
|
48
|
+
Super-resolution sits on top: `decimate` block-means the signal, and the
|
|
49
|
+
coarse result is refined by scoring the original samples around the winning
|
|
50
|
+
blocks. The refinement window deliberately spans one block either side of each
|
|
51
|
+
coarse endpoint — narrowing it to the winning block halves the rate of exact
|
|
52
|
+
agreement, and does so silently, returning a plausible interval a few samples
|
|
53
|
+
out rather than an error. `test_refinement_searches_beyond_the_winning_block`
|
|
54
|
+
pins that directly.
|
|
55
|
+
|
|
56
|
+
## The kernel scan's correctness contract
|
|
57
|
+
|
|
58
|
+
`scan_nwkr` carries four pieces of mutable state across thousands of O(r)
|
|
59
|
+
updates. A mistake there does not crash — it drifts. So it is pinned against
|
|
60
|
+
`tests/_oracle.py::direct_nwkr`, which refits every index for every interval
|
|
61
|
+
and carries no state at all:
|
|
62
|
+
|
|
63
|
+
- `test_incremental_matches_reference` covers both kernels, three bandwidths
|
|
64
|
+
and two buffer settings, to 1e-11 relative;
|
|
65
|
+
- `test_no_drift_over_a_long_run` uses large `w`, which puts many incremental
|
|
66
|
+
updates between exact refreshes — where accumulated error would appear.
|
|
67
|
+
|
|
68
|
+
Any change to `kernel_state.py` or to the search loop must keep both passing.
|
|
69
|
+
Note that `sse_out` is refreshed exactly every `REFRESH = range_cap` steps; if
|
|
70
|
+
you change that cadence, the drift test is what tells you whether you can.
|
|
71
|
+
|
|
72
|
+
## Scope
|
|
73
|
+
|
|
74
|
+
This package holds the regression scan statistic families only: `F_0`, `F_d`,
|
|
75
|
+
`F_KR`, `F_KRR`. Baseline detectors, benchmarking harnesses and
|
|
76
|
+
application-specific preprocessing belong elsewhere, so that a scan statistic
|
|
77
|
+
never drags ruptures, stumpy or torch into someone's environment.
|
regscan-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, Gazi Abdur Rakib.
|
|
4
|
+
All rights reserved.
|
|
5
|
+
|
|
6
|
+
Redistribution and use in source and binary forms, with or without
|
|
7
|
+
modification, are permitted provided that the following conditions are met:
|
|
8
|
+
|
|
9
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
10
|
+
list of conditions and the following disclaimer.
|
|
11
|
+
|
|
12
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
13
|
+
this list of conditions and the following disclaimer in the documentation
|
|
14
|
+
and/or other materials provided with the distribution.
|
|
15
|
+
|
|
16
|
+
3. Neither the name of the copyright holder nor the names of its contributors
|
|
17
|
+
may be used to endorse or promote products derived from this software
|
|
18
|
+
without specific prior written permission.
|
|
19
|
+
|
|
20
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
21
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
22
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
23
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
24
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
25
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
26
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
27
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
28
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
29
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
regscan-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,244 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: regscan
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Regression-based scan statistics for interval anomaly detection in smoothly varying 1D signals
|
|
5
|
+
Project-URL: Homepage, https://github.com/BeardyMan37/regscan
|
|
6
|
+
Project-URL: Repository, https://github.com/BeardyMan37/regscan
|
|
7
|
+
Project-URL: Paper, https://arxiv.org/abs/2608.22201
|
|
8
|
+
Author: Gazi Abdur Rakib
|
|
9
|
+
License-Expression: BSD-3-Clause
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: anomaly detection,change point,kernel regression,radio astronomy,scan statistics
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering :: Astronomy
|
|
16
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
17
|
+
Requires-Python: >=3.9
|
|
18
|
+
Requires-Dist: numba>=0.57
|
|
19
|
+
Requires-Dist: numpy>=1.22
|
|
20
|
+
Provides-Extra: dev
|
|
21
|
+
Requires-Dist: mypy; extra == 'dev'
|
|
22
|
+
Requires-Dist: pytest-cov; extra == 'dev'
|
|
23
|
+
Requires-Dist: pytest>=7; extra == 'dev'
|
|
24
|
+
Requires-Dist: ruff; extra == 'dev'
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
|
|
27
|
+
# regscan
|
|
28
|
+
|
|
29
|
+
Regression-based scan statistics for detecting interval anomalies in smoothly
|
|
30
|
+
varying 1-D signals.
|
|
31
|
+
|
|
32
|
+
Given a signal and a function family `F`, the score of an interval `I = [a, b]` is
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
S(I) = 1 - (SR_I + SR_O) / SR_A
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
where `SR_A`, `SR_I` and `SR_O` are sums of squared residuals from fitting `F`
|
|
39
|
+
to the whole signal, to the inside of `I`, and to the outside. `S` is near 0
|
|
40
|
+
when splitting explains nothing and approaches 1 when it explains everything.
|
|
41
|
+
The scan returns the highest-scoring interval.
|
|
42
|
+
|
|
43
|
+
## Notation
|
|
44
|
+
|
|
45
|
+
| symbol | meaning |
|
|
46
|
+
|---|---|
|
|
47
|
+
| `n` | length of the signal, in samples |
|
|
48
|
+
| `w` | window width — the kernel bandwidth for `F_KR`, and the scale of local structure the fit can follow |
|
|
49
|
+
| `r` | range cap: the longest candidate interval considered, `r = 3w` |
|
|
50
|
+
| `d` | polynomial degree for the `F_d` family (`poly_deg1` is `d = 1`) |
|
|
51
|
+
| `a`, `b` | inclusive start and end indices of a candidate interval |
|
|
52
|
+
| `I` | the candidate interval `[a, b]` |
|
|
53
|
+
|
|
54
|
+
`n` is fixed by the data. `w` is the one parameter worth thinking about, and
|
|
55
|
+
`r` follows from it. Complexities below are for a full scan over all candidate
|
|
56
|
+
intervals.
|
|
57
|
+
|
|
58
|
+
| method | family | model | cost |
|
|
59
|
+
|---|---|---|---|
|
|
60
|
+
| `mean` | `F_0` | constant | O(nr) |
|
|
61
|
+
| `poly_deg1` … `poly_deg3` | `F_d` | degree-`d` polynomial | O(nr d³) |
|
|
62
|
+
| `nwkr_gaussian`, `nwkr_laplace` | `F_KR` | Nadaraya-Watson kernel regression | O(nrw) |
|
|
63
|
+
| `krr_gaussian`, `krr_laplace` | `F_KRR` | kernel ridge regression | O(n⁴w) |
|
|
64
|
+
|
|
65
|
+
Complexities are for the algorithm; see **Performance** below for what this
|
|
66
|
+
implementation actually achieves and where it falls short.
|
|
67
|
+
|
|
68
|
+
`F_KR` is the method the package exists for. A weak family such as `F_0` or
|
|
69
|
+
`F_1` cannot represent a curved background, so it reduces residual by
|
|
70
|
+
splitting the interval wherever the curvature is worst — flagging smooth
|
|
71
|
+
structure as an anomaly. The kernel fit tracks that structure, so it enters
|
|
72
|
+
`SR_A`, `SR_I` and `SR_O` alike and cancels out of the score.
|
|
73
|
+
|
|
74
|
+
## Install
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
pip install regscan
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
numpy and numba. The kernel scan is a tight scalar loop — the shape numba
|
|
81
|
+
compiles well and numpy vectorises badly — so it is JIT-compiled. The first
|
|
82
|
+
call in a process pays a one-off compile; results are cached on disk after
|
|
83
|
+
that.
|
|
84
|
+
|
|
85
|
+
## Use
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
import numpy as np, regscan
|
|
89
|
+
|
|
90
|
+
t = np.linspace(0, 1, 300)
|
|
91
|
+
x = 4.7 + 0.10 * t + 0.05 * np.random.default_rng(0).normal(0, 1, 300)
|
|
92
|
+
x[140:170] -= 0.25 # the anomaly
|
|
93
|
+
|
|
94
|
+
res = regscan.scan(x, method="nwkr_gaussian", w=12)
|
|
95
|
+
res.score, res.a, res.b # 0.248, 140, 169
|
|
96
|
+
res.width_frac, res.at_edge() # 0.10, False
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
On the same signal with no anomaly planted, `nwkr_gaussian` scores 0.036 while
|
|
100
|
+
`mean` scores 0.185 — the weak family is reacting to the ramp.
|
|
101
|
+
|
|
102
|
+
## Choosing `w` and `r`
|
|
103
|
+
|
|
104
|
+
`w` sets the scale of structure the fit can follow. Too small and the kernel
|
|
105
|
+
reproduces the anomaly itself, so it cancels out of the score; too large and
|
|
106
|
+
the fit cannot follow the background, which is the failure mode of the weak
|
|
107
|
+
families.
|
|
108
|
+
|
|
109
|
+
`r` is the longest interval the scan will consider. An anomaly wider than `r`
|
|
110
|
+
cannot be returned at all, and cost falls linearly as `r` does, so it is the
|
|
111
|
+
knob to reach for when you know roughly how wide the feature is.
|
|
112
|
+
|
|
113
|
+
```python
|
|
114
|
+
regscan.scan(x, method="nwkr_gaussian", w=12, r=90)
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
Omitted, `w` defaults to `max(3, n // 16)` and `r` to `3 * w`. Those are
|
|
118
|
+
fallbacks so the scan runs unattended, not recommendations.
|
|
119
|
+
|
|
120
|
+
Sweeping `w` and checking whether `(a, b)` stays put is a cheap way to tell a
|
|
121
|
+
resolved feature from an artefact of the bandwidth: a real interval holds
|
|
122
|
+
steady, while one that tracks `w` is measuring the kernel.
|
|
123
|
+
|
|
124
|
+
## Super-resolution
|
|
125
|
+
|
|
126
|
+
`F_KR` can block-mean the signal, scan the shorter version, then search the
|
|
127
|
+
original samples around the winning blocks to recover exact endpoints:
|
|
128
|
+
|
|
129
|
+
```python
|
|
130
|
+
from regscan import ScanConfig
|
|
131
|
+
|
|
132
|
+
regscan.scan(x, method="nwkr_gaussian", w=100,
|
|
133
|
+
config=ScanConfig(super_resolution=4)) # or "auto"
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
At n = 1600 with w = 100:
|
|
137
|
+
|
|
138
|
+
| factor | time | interval |
|
|
139
|
+
|---|---|---|
|
|
140
|
+
| 1 (exact) | 3.11 s | (700, 819) |
|
|
141
|
+
| 2 | 0.44 s | (700, 819) |
|
|
142
|
+
| 4 | 0.08 s | (700, 819) |
|
|
143
|
+
| 8 | 0.02 s | (704, 815) |
|
|
144
|
+
|
|
145
|
+
A factor of 4 is **39× faster** and returns the same answer. A factor of 8
|
|
146
|
+
does not, and that is the trade: the coarse pass locates each endpoint only to
|
|
147
|
+
within a block of `factor` samples, and a wrong block is a wrong answer.
|
|
148
|
+
|
|
149
|
+
Refinement searches one block either side of each coarse endpoint, which
|
|
150
|
+
matters more than it sounds — block means smooth an anomaly's edges, so the
|
|
151
|
+
coarse pass picks a neighbouring block often enough that confining the search
|
|
152
|
+
to the winning block alone reproduced the exact interval only 20/40 times at
|
|
153
|
+
factor 4. Including the neighbours makes it 40/40.
|
|
154
|
+
|
|
155
|
+
It remains an approximation. Verify against `super_resolution=1` on a sample
|
|
156
|
+
of your data before trusting it wholesale.
|
|
157
|
+
|
|
158
|
+
`"auto"` picks the factor from the length: 1 below 450 samples, then doubling
|
|
159
|
+
at 900, 1800 and so on. `sr_cap` bounds it, which matters when the feature is
|
|
160
|
+
narrow — an interval has to survive decimation to be found.
|
|
161
|
+
|
|
162
|
+
The default is `1`, exact.
|
|
163
|
+
|
|
164
|
+
## Configuration
|
|
165
|
+
|
|
166
|
+
```python
|
|
167
|
+
from regscan import ScanConfig
|
|
168
|
+
|
|
169
|
+
cfg = ScanConfig(
|
|
170
|
+
kernel="laplace", # gaussian | laplace
|
|
171
|
+
buffer=24, # exclude this many samples at each end
|
|
172
|
+
min_width=0.01, # bounds on interval length, as a fraction of n
|
|
173
|
+
max_width=0.25,
|
|
174
|
+
)
|
|
175
|
+
regscan.scan(x, method="nwkr_gaussian", w=16, config=cfg)
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
`ScanConfig` is immutable and passed explicitly; nothing lives in module
|
|
179
|
+
globals, so scanning several methods in one process cannot leak state between
|
|
180
|
+
them.
|
|
181
|
+
|
|
182
|
+
`buffer` matters when comparing families. An interval can only be placed in
|
|
183
|
+
`[buffer, n-buffer)`, so a non-zero buffer suppresses detections at the ends of
|
|
184
|
+
the signal — give every family the same value or the comparison is not like for
|
|
185
|
+
like.
|
|
186
|
+
|
|
187
|
+
`max_width` is worth capping. As an interval approaches `n/2`, inside and
|
|
188
|
+
outside become comparable and the statistic stops discriminating; the
|
|
189
|
+
maximiser then drifts to whatever split best absorbs slow curvature.
|
|
190
|
+
|
|
191
|
+
## Performance
|
|
192
|
+
|
|
193
|
+
Measured on this implementation, one full scan, `w = n // 16`:
|
|
194
|
+
|
|
195
|
+
One full scan, `w = n // 16`, after JIT warm-up:
|
|
196
|
+
|
|
197
|
+
| n | `mean` | `poly_deg1` | `nwkr_gaussian` |
|
|
198
|
+
|---|---|---|---|
|
|
199
|
+
| 100 | 3 ms | 44 ms | 4 ms |
|
|
200
|
+
| 200 | 12 ms | 181 ms | 18 ms |
|
|
201
|
+
| 400 | 51 ms | 754 ms | 83 ms |
|
|
202
|
+
| 800 | 187 ms | 2.7 s | 0.45 s |
|
|
203
|
+
| 1600 | 738 ms | 11.0 s | 3.2 s |
|
|
204
|
+
|
|
205
|
+
`F_KR` is **6× faster than `F_1`** at n = 800 despite fitting a far richer
|
|
206
|
+
model, which is the practical case for it: the polynomial family pays O(d³) per
|
|
207
|
+
candidate interval, while the kernel family pays O(r) to extend one.
|
|
208
|
+
|
|
209
|
+
Each family reaches its complexity by carrying state rather than refitting.
|
|
210
|
+
`mean` scores an interval from prefix sums of `y` and `y²` in O(1).
|
|
211
|
+
`poly_deg1` uses prefix sums of the moments `t^p` and `t^p y`, so a fit costs
|
|
212
|
+
O(d³) to solve regardless of interval length. `F_KR` grows an interval one
|
|
213
|
+
sample at a time, updating in O(r): the inside buffer and `sse_in`, the
|
|
214
|
+
`nin`/`din` arrays holding the inside points' kernel contribution to every
|
|
215
|
+
index, and `sse_out` obtained from them by subtracting from the all-points
|
|
216
|
+
totals. `sse_out` is adjusted rather than recomputed, so it is refreshed
|
|
217
|
+
exactly on a fixed cadence to keep floating-point error from accumulating.
|
|
218
|
+
|
|
219
|
+
For longer signals, super-resolution (above) cuts this substantially again.
|
|
220
|
+
|
|
221
|
+
## Scores are comparable within a family, not across families
|
|
222
|
+
|
|
223
|
+
Each family divides by its own `SR_A`, and a kernel fit has a smaller `SR_A`
|
|
224
|
+
than a constant fit before any interval is chosen. The same interval therefore
|
|
225
|
+
scores differently under `F_0` and `F_KR`. Compare families by rank, by whether
|
|
226
|
+
they agree on the interval, or by the contrast between anomalous and clean
|
|
227
|
+
signals — not by absolute score.
|
|
228
|
+
|
|
229
|
+
## Citing
|
|
230
|
+
|
|
231
|
+
Rakib et al., *Efficient Regression Models for Scan Statistics*,
|
|
232
|
+
[arXiv:2608.22201](https://arxiv.org/abs/2608.22201) (2026).
|
|
233
|
+
|
|
234
|
+
```bibtex
|
|
235
|
+
@misc{rakib2026efficientregressionmodelsscan,
|
|
236
|
+
title={Efficient Regression Models for Scan Statistics},
|
|
237
|
+
author={Gazi Abdur Rakib and Tristan Ashton and Ryan A. Loomis and Brian S. Mason and Eric J. Murphy and Ci Xue and Jeff M. Phillips},
|
|
238
|
+
year={2026},
|
|
239
|
+
eprint={2608.22201},
|
|
240
|
+
archivePrefix={arXiv},
|
|
241
|
+
primaryClass={stat.ME},
|
|
242
|
+
url={https://arxiv.org/abs/2608.22201},
|
|
243
|
+
}
|
|
244
|
+
```
|
regscan-0.1.0/README.md
ADDED
|
@@ -0,0 +1,218 @@
|
|
|
1
|
+
# regscan
|
|
2
|
+
|
|
3
|
+
Regression-based scan statistics for detecting interval anomalies in smoothly
|
|
4
|
+
varying 1-D signals.
|
|
5
|
+
|
|
6
|
+
Given a signal and a function family `F`, the score of an interval `I = [a, b]` is
|
|
7
|
+
|
|
8
|
+
```
|
|
9
|
+
S(I) = 1 - (SR_I + SR_O) / SR_A
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
where `SR_A`, `SR_I` and `SR_O` are sums of squared residuals from fitting `F`
|
|
13
|
+
to the whole signal, to the inside of `I`, and to the outside. `S` is near 0
|
|
14
|
+
when splitting explains nothing and approaches 1 when it explains everything.
|
|
15
|
+
The scan returns the highest-scoring interval.
|
|
16
|
+
|
|
17
|
+
## Notation
|
|
18
|
+
|
|
19
|
+
| symbol | meaning |
|
|
20
|
+
|---|---|
|
|
21
|
+
| `n` | length of the signal, in samples |
|
|
22
|
+
| `w` | window width — the kernel bandwidth for `F_KR`, and the scale of local structure the fit can follow |
|
|
23
|
+
| `r` | range cap: the longest candidate interval considered, `r = 3w` |
|
|
24
|
+
| `d` | polynomial degree for the `F_d` family (`poly_deg1` is `d = 1`) |
|
|
25
|
+
| `a`, `b` | inclusive start and end indices of a candidate interval |
|
|
26
|
+
| `I` | the candidate interval `[a, b]` |
|
|
27
|
+
|
|
28
|
+
`n` is fixed by the data. `w` is the one parameter worth thinking about, and
|
|
29
|
+
`r` follows from it. Complexities below are for a full scan over all candidate
|
|
30
|
+
intervals.
|
|
31
|
+
|
|
32
|
+
| method | family | model | cost |
|
|
33
|
+
|---|---|---|---|
|
|
34
|
+
| `mean` | `F_0` | constant | O(nr) |
|
|
35
|
+
| `poly_deg1` … `poly_deg3` | `F_d` | degree-`d` polynomial | O(nr d³) |
|
|
36
|
+
| `nwkr_gaussian`, `nwkr_laplace` | `F_KR` | Nadaraya-Watson kernel regression | O(nrw) |
|
|
37
|
+
| `krr_gaussian`, `krr_laplace` | `F_KRR` | kernel ridge regression | O(n⁴w) |
|
|
38
|
+
|
|
39
|
+
Complexities are for the algorithm; see **Performance** below for what this
|
|
40
|
+
implementation actually achieves and where it falls short.
|
|
41
|
+
|
|
42
|
+
`F_KR` is the method the package exists for. A weak family such as `F_0` or
|
|
43
|
+
`F_1` cannot represent a curved background, so it reduces residual by
|
|
44
|
+
splitting the interval wherever the curvature is worst — flagging smooth
|
|
45
|
+
structure as an anomaly. The kernel fit tracks that structure, so it enters
|
|
46
|
+
`SR_A`, `SR_I` and `SR_O` alike and cancels out of the score.
|
|
47
|
+
|
|
48
|
+
## Install
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
pip install regscan
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
numpy and numba. The kernel scan is a tight scalar loop — the shape numba
|
|
55
|
+
compiles well and numpy vectorises badly — so it is JIT-compiled. The first
|
|
56
|
+
call in a process pays a one-off compile; results are cached on disk after
|
|
57
|
+
that.
|
|
58
|
+
|
|
59
|
+
## Use
|
|
60
|
+
|
|
61
|
+
```python
|
|
62
|
+
import numpy as np, regscan
|
|
63
|
+
|
|
64
|
+
t = np.linspace(0, 1, 300)
|
|
65
|
+
x = 4.7 + 0.10 * t + 0.05 * np.random.default_rng(0).normal(0, 1, 300)
|
|
66
|
+
x[140:170] -= 0.25 # the anomaly
|
|
67
|
+
|
|
68
|
+
res = regscan.scan(x, method="nwkr_gaussian", w=12)
|
|
69
|
+
res.score, res.a, res.b # 0.248, 140, 169
|
|
70
|
+
res.width_frac, res.at_edge() # 0.10, False
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
On the same signal with no anomaly planted, `nwkr_gaussian` scores 0.036 while
|
|
74
|
+
`mean` scores 0.185 — the weak family is reacting to the ramp.
|
|
75
|
+
|
|
76
|
+
## Choosing `w` and `r`
|
|
77
|
+
|
|
78
|
+
`w` sets the scale of structure the fit can follow. Too small and the kernel
|
|
79
|
+
reproduces the anomaly itself, so it cancels out of the score; too large and
|
|
80
|
+
the fit cannot follow the background, which is the failure mode of the weak
|
|
81
|
+
families.
|
|
82
|
+
|
|
83
|
+
`r` is the longest interval the scan will consider. An anomaly wider than `r`
|
|
84
|
+
cannot be returned at all, and cost falls linearly as `r` does, so it is the
|
|
85
|
+
knob to reach for when you know roughly how wide the feature is.
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
regscan.scan(x, method="nwkr_gaussian", w=12, r=90)
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Omitted, `w` defaults to `max(3, n // 16)` and `r` to `3 * w`. Those are
|
|
92
|
+
fallbacks so the scan runs unattended, not recommendations.
|
|
93
|
+
|
|
94
|
+
Sweeping `w` and checking whether `(a, b)` stays put is a cheap way to tell a
|
|
95
|
+
resolved feature from an artefact of the bandwidth: a real interval holds
|
|
96
|
+
steady, while one that tracks `w` is measuring the kernel.
|
|
97
|
+
|
|
98
|
+
## Super-resolution
|
|
99
|
+
|
|
100
|
+
`F_KR` can block-mean the signal, scan the shorter version, then search the
|
|
101
|
+
original samples around the winning blocks to recover exact endpoints:
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
from regscan import ScanConfig
|
|
105
|
+
|
|
106
|
+
regscan.scan(x, method="nwkr_gaussian", w=100,
|
|
107
|
+
config=ScanConfig(super_resolution=4)) # or "auto"
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
At n = 1600 with w = 100:
|
|
111
|
+
|
|
112
|
+
| factor | time | interval |
|
|
113
|
+
|---|---|---|
|
|
114
|
+
| 1 (exact) | 3.11 s | (700, 819) |
|
|
115
|
+
| 2 | 0.44 s | (700, 819) |
|
|
116
|
+
| 4 | 0.08 s | (700, 819) |
|
|
117
|
+
| 8 | 0.02 s | (704, 815) |
|
|
118
|
+
|
|
119
|
+
A factor of 4 is **39× faster** and returns the same answer. A factor of 8
|
|
120
|
+
does not, and that is the trade: the coarse pass locates each endpoint only to
|
|
121
|
+
within a block of `factor` samples, and a wrong block is a wrong answer.
|
|
122
|
+
|
|
123
|
+
Refinement searches one block either side of each coarse endpoint, which
|
|
124
|
+
matters more than it sounds — block means smooth an anomaly's edges, so the
|
|
125
|
+
coarse pass picks a neighbouring block often enough that confining the search
|
|
126
|
+
to the winning block alone reproduced the exact interval only 20/40 times at
|
|
127
|
+
factor 4. Including the neighbours makes it 40/40.
|
|
128
|
+
|
|
129
|
+
It remains an approximation. Verify against `super_resolution=1` on a sample
|
|
130
|
+
of your data before trusting it wholesale.
|
|
131
|
+
|
|
132
|
+
`"auto"` picks the factor from the length: 1 below 450 samples, then doubling
|
|
133
|
+
at 900, 1800 and so on. `sr_cap` bounds it, which matters when the feature is
|
|
134
|
+
narrow — an interval has to survive decimation to be found.
|
|
135
|
+
|
|
136
|
+
The default is `1`, exact.
|
|
137
|
+
|
|
138
|
+
## Configuration
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from regscan import ScanConfig
|
|
142
|
+
|
|
143
|
+
cfg = ScanConfig(
|
|
144
|
+
kernel="laplace", # gaussian | laplace
|
|
145
|
+
buffer=24, # exclude this many samples at each end
|
|
146
|
+
min_width=0.01, # bounds on interval length, as a fraction of n
|
|
147
|
+
max_width=0.25,
|
|
148
|
+
)
|
|
149
|
+
regscan.scan(x, method="nwkr_gaussian", w=16, config=cfg)
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
`ScanConfig` is immutable and passed explicitly; nothing lives in module
|
|
153
|
+
globals, so scanning several methods in one process cannot leak state between
|
|
154
|
+
them.
|
|
155
|
+
|
|
156
|
+
`buffer` matters when comparing families. An interval can only be placed in
|
|
157
|
+
`[buffer, n-buffer)`, so a non-zero buffer suppresses detections at the ends of
|
|
158
|
+
the signal — give every family the same value or the comparison is not like for
|
|
159
|
+
like.
|
|
160
|
+
|
|
161
|
+
`max_width` is worth capping. As an interval approaches `n/2`, inside and
|
|
162
|
+
outside become comparable and the statistic stops discriminating; the
|
|
163
|
+
maximiser then drifts to whatever split best absorbs slow curvature.
|
|
164
|
+
|
|
165
|
+
## Performance
|
|
166
|
+
|
|
167
|
+
Measured on this implementation, one full scan, `w = n // 16`:
|
|
168
|
+
|
|
169
|
+
One full scan, `w = n // 16`, after JIT warm-up:
|
|
170
|
+
|
|
171
|
+
| n | `mean` | `poly_deg1` | `nwkr_gaussian` |
|
|
172
|
+
|---|---|---|---|
|
|
173
|
+
| 100 | 3 ms | 44 ms | 4 ms |
|
|
174
|
+
| 200 | 12 ms | 181 ms | 18 ms |
|
|
175
|
+
| 400 | 51 ms | 754 ms | 83 ms |
|
|
176
|
+
| 800 | 187 ms | 2.7 s | 0.45 s |
|
|
177
|
+
| 1600 | 738 ms | 11.0 s | 3.2 s |
|
|
178
|
+
|
|
179
|
+
`F_KR` is **6× faster than `F_1`** at n = 800 despite fitting a far richer
|
|
180
|
+
model, which is the practical case for it: the polynomial family pays O(d³) per
|
|
181
|
+
candidate interval, while the kernel family pays O(r) to extend one.
|
|
182
|
+
|
|
183
|
+
Each family reaches its complexity by carrying state rather than refitting.
|
|
184
|
+
`mean` scores an interval from prefix sums of `y` and `y²` in O(1).
|
|
185
|
+
`poly_deg1` uses prefix sums of the moments `t^p` and `t^p y`, so a fit costs
|
|
186
|
+
O(d³) to solve regardless of interval length. `F_KR` grows an interval one
|
|
187
|
+
sample at a time, updating in O(r): the inside buffer and `sse_in`, the
|
|
188
|
+
`nin`/`din` arrays holding the inside points' kernel contribution to every
|
|
189
|
+
index, and `sse_out` obtained from them by subtracting from the all-points
|
|
190
|
+
totals. `sse_out` is adjusted rather than recomputed, so it is refreshed
|
|
191
|
+
exactly on a fixed cadence to keep floating-point error from accumulating.
|
|
192
|
+
|
|
193
|
+
For longer signals, super-resolution (above) cuts this substantially again.
|
|
194
|
+
|
|
195
|
+
## Scores are comparable within a family, not across families
|
|
196
|
+
|
|
197
|
+
Each family divides by its own `SR_A`, and a kernel fit has a smaller `SR_A`
|
|
198
|
+
than a constant fit before any interval is chosen. The same interval therefore
|
|
199
|
+
scores differently under `F_0` and `F_KR`. Compare families by rank, by whether
|
|
200
|
+
they agree on the interval, or by the contrast between anomalous and clean
|
|
201
|
+
signals — not by absolute score.
|
|
202
|
+
|
|
203
|
+
## Citing
|
|
204
|
+
|
|
205
|
+
Rakib et al., *Efficient Regression Models for Scan Statistics*,
|
|
206
|
+
[arXiv:2608.22201](https://arxiv.org/abs/2608.22201) (2026).
|
|
207
|
+
|
|
208
|
+
```bibtex
|
|
209
|
+
@misc{rakib2026efficientregressionmodelsscan,
|
|
210
|
+
title={Efficient Regression Models for Scan Statistics},
|
|
211
|
+
author={Gazi Abdur Rakib and Tristan Ashton and Ryan A. Loomis and Brian S. Mason and Eric J. Murphy and Ci Xue and Jeff M. Phillips},
|
|
212
|
+
year={2026},
|
|
213
|
+
eprint={2608.22201},
|
|
214
|
+
archivePrefix={arXiv},
|
|
215
|
+
primaryClass={stat.ME},
|
|
216
|
+
url={https://arxiv.org/abs/2608.22201},
|
|
217
|
+
}
|
|
218
|
+
```
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
name: publish
|
|
2
|
+
|
|
3
|
+
# Publishes to PyPI when a GitHub Release is created. Uses Trusted Publishing
|
|
4
|
+
# (OpenID Connect), so there is no API token to store or rotate: PyPI verifies
|
|
5
|
+
# the workflow's identity directly. Configure the publisher once on PyPI under
|
|
6
|
+
# the project's Publishing settings before the first release.
|
|
7
|
+
|
|
8
|
+
on:
|
|
9
|
+
release:
|
|
10
|
+
types: [published]
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
build:
|
|
14
|
+
runs-on: ubuntu-latest
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v4
|
|
17
|
+
- uses: actions/setup-python@v5
|
|
18
|
+
with:
|
|
19
|
+
python-version: "3.11"
|
|
20
|
+
- run: pip install build twine
|
|
21
|
+
- run: python -m build
|
|
22
|
+
# Fails on malformed metadata or an unrenderable README, which is worth
|
|
23
|
+
# catching here rather than after the upload is irreversible.
|
|
24
|
+
- run: twine check dist/*
|
|
25
|
+
- uses: actions/upload-artifact@v4
|
|
26
|
+
with:
|
|
27
|
+
name: dist
|
|
28
|
+
path: dist/
|
|
29
|
+
|
|
30
|
+
publish:
|
|
31
|
+
needs: build
|
|
32
|
+
runs-on: ubuntu-latest
|
|
33
|
+
environment: pypi
|
|
34
|
+
permissions:
|
|
35
|
+
id-token: write # required for Trusted Publishing
|
|
36
|
+
steps:
|
|
37
|
+
- uses: actions/download-artifact@v4
|
|
38
|
+
with:
|
|
39
|
+
name: dist
|
|
40
|
+
path: dist/
|
|
41
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
42
|
+
with:
|
|
43
|
+
repository-url: https://test.pypi.org/legacy/
|