matchedfilter 0.1.0a1__tar.gz

matchedfilter-0.1.0a1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alex Nitz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

matchedfilter-0.1.0a1/MANIFEST.in

@@ -0,0 +1,11 @@
+# setuptools only guesses at Python files, so every C source the build needs
+# has to be named here.  Without this the sdist builds fine and then fails to
+# install: the extension sources are simply absent.
+include README.md LICENSE
+recursive-include src *.c *.h *.py
+recursive-include python *.c *.h
+recursive-include tests *.c *.h *.py
+recursive-include tools *.py
+recursive-include docs *.md
+prune build
+global-exclude *.pyc *.so *.o __pycache__

matchedfilter-0.1.0a1/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.4
+Name: matchedfilter
+Version: 0.1.0a1
+Summary: Fast single-threaded batched matched filter with peak-only output (x86-64)
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ahnitz/matchedfilter
+Keywords: matched-filter,correlation,fft,avx512,signal-processing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: C
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.20
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Provides-Extra: bench
+Requires-Dist: numpy>=1.20; extra == "bench"
+Dynamic: license-file
+
+# matchedfilter
+
+A fast single-threaded matched filter for x86. You give it a batch of data
+segments and a batch of templates; it correlates every pair and hands back only
+the peaks.
+
+Not returning the full correlation is the point. Most searches threshold the
+output and throw the rest away, and once you say so up front the filter can
+skip work that could not have produced a peak anyway.
+
+> **Status: work in progress.** The API still moves, and there is a known
+> calibration weakness in the hierarchical filter. See
+> [Caveats](#caveats).
+
+```python
+import matchedfilter as mf
+
+filt = mf.MatchedFilter(16384, ndata=16, ntemplates=64)
+filt.set_data(data_spectra)          # (16, 16384) complex64, already FFT'd
+filt.set_templates(template_spectra) # (64, 16384) complex64
+
+peaks = filt.run(binsize=1024, threshold=5.5)
+peaks["index"], peaks["value"], peaks["magnitude"]
+```
+
+`import matchedfilter as mf` is the convention used throughout these docs.
+
+## Install
+
+```bash
+pip install --pre matchedfilter
+```
+
+`--pre` because this is an alpha release. Wheels are built for CPython 3.9 to
+3.13 on manylinux x86-64; anywhere else pip falls back to the source
+distribution, which needs numpy and a C compiler.
+
+**x86-64 only for now.** The kernels are AVX2 and AVX-512 intrinsics with no
+portable fallback, so a build on any other architecture stops with an error
+rather than producing a slow one. AVX-512 is used when the CPU has it and AVX2
+otherwise, decided at runtime.
+
+## How it works
+
+Inputs are **frequency domain**: the unnormalised forward transform of each
+segment, in natural order. Produce them with whatever you already use (numpy,
+MKL, FFTW); matchedfilter does not need to own that step.
+
+The filter is built once and reused. Ingest conjugates the templates and
+stores both sides in the layout the correlation loop walks, which costs a few
+percent of a run and less as the batch grows.
+
+`run` returns a structured array of shape `(ndata, ntemplates, nbins)` with
+fields `index`, `value` and `magnitude`. Bins whose peak fell below the
+threshold carry `index == -1`.
+
+Supported lengths are 1024 and the powers of two from 4096 to 1048576.
+
+### Performance
+
+Per (data, template) pair, 8x32 batch, one core of a Zen 5 desktop:
+
+| n | matchedfilter | numpy | |
+|---:|---:|---:|---:|
+| 1024 | 0.61 µs | 18.98 µs | 31x |
+| 4096 | 2.08 µs | 37.81 µs | 18x |
+| 16384 | 9.98 µs | 127.04 µs | 13x |
+| 65536 | 48.87 µs | 568.56 µs | 12x |
+
+numpy is a floor, not a rival. It is there so the comparison runs anywhere.
+Against MKL or FFTW the margin is much smaller, and part of what is left comes
+from computing peaks instead of a full correlation. Measure on your own box:
+
+```bash
+python -m matchedfilter.benchmark
+```
+
+## Hierarchical filtering
+
+`HierarchicalFilter` adds a cheap pre-pass: correlate against a low-frequency
+slice of the template, and only run the full-length filter where that slice
+leaves a peak plausible.
+
+**This helps only under an assumption about your templates**: that enough of
+the matched-filter output power sits in the low band that a narrow slice gives
+a usable bound on the full result. For chirp-like templates whose power is
+concentrated at low frequency that tends to hold. For templates whose power is
+spread flat across the band, or concentrated high, the slice bounds nothing
+useful, and the pre-pass is pure added cost. It is worth checking against your
+own templates before relying on it.
+
+```python
+hf = mf.HierarchicalFilter(16384, ndata=16, ntemplates=64,
+                           snr=6.0,   # threshold you intend to use
+                           fd=1e-3,   # false-dismissal budget
+                           band=2048) # width of the cheap slice
+
+hf.set_reference(expected_output_power)      # real frequency series, the OUTPUT
+hf.set_templates(template_spectra)
+hf.set_data(data_spectra)
+peaks = hf.run(binsize=16384, threshold=6.0)
+```
+
+`set_reference` takes a real frequency series of length `n` holding the expected
+power of the filter **output** in each bin. Only its shape is used; the overall
+normalisation is divided out.
+
+This is the output, not the template. The two differ whenever the data is
+coloured, and passing the template's own power will mis-set the gate: a
+broadband template reconstructing a narrowband signal is the case where it goes
+wrong by the largest factor.
+
+The gate is one-sided by construction: peaks it reports are bit-identical to
+the flat filter's. It can only omit, never invent. `fd` is the budget for how
+often it is allowed to omit one.
+
+On pure noise at n=4096 the gate runs about **7x faster** than the flat filter.
+The saving scales with how little survives, so it grows with your threshold and
+falls toward 1x on data where most pairs trigger.
+
+## Caveats
+
+- **The false-dismissal budget is not currently met at low thresholds.**
+  `fd` is honoured well at snr 6 and above. At snr 5.0 to 5.5 with a coarse
+  band the gate omits more than it should: 1.4% against a 0.1% budget in a
+  418-template search. The cause is that the gate's recovery factors are
+  measured from a mean frequency series, which is not a bound on any
+  individual realisation. Tracked by an `xfail` test in `tests/test_api.py` and written up
+  in [docs/hierarchical.md](docs/hierarchical.md).
+- Single-threaded by design. Parallelism is the caller's to arrange.
+- x86-64 Linux only, as above. Other architectures are not implemented rather
+  than merely untested.
+
+## Development
+
+```bash
+pip install -e .[test]
+pytest              # includes a C test that builds itself from source
+python -m matchedfilter.benchmark
+```
+
+[docs/](docs/) holds the design notes: how the hierarchical gate is
+calibrated, and measurements of the approaches that were tried and rejected.
+
+## License
+
+MIT

matchedfilter-0.1.0a1/README.md

@@ -0,0 +1,147 @@
+# matchedfilter
+
+A fast single-threaded matched filter for x86. You give it a batch of data
+segments and a batch of templates; it correlates every pair and hands back only
+the peaks.
+
+Not returning the full correlation is the point. Most searches threshold the
+output and throw the rest away, and once you say so up front the filter can
+skip work that could not have produced a peak anyway.
+
+> **Status: work in progress.** The API still moves, and there is a known
+> calibration weakness in the hierarchical filter. See
+> [Caveats](#caveats).
+
+```python
+import matchedfilter as mf
+
+filt = mf.MatchedFilter(16384, ndata=16, ntemplates=64)
+filt.set_data(data_spectra)          # (16, 16384) complex64, already FFT'd
+filt.set_templates(template_spectra) # (64, 16384) complex64
+
+peaks = filt.run(binsize=1024, threshold=5.5)
+peaks["index"], peaks["value"], peaks["magnitude"]
+```
+
+`import matchedfilter as mf` is the convention used throughout these docs.
+
+## Install
+
+```bash
+pip install --pre matchedfilter
+```
+
+`--pre` because this is an alpha release. Wheels are built for CPython 3.9 to
+3.13 on manylinux x86-64; anywhere else pip falls back to the source
+distribution, which needs numpy and a C compiler.
+
+**x86-64 only for now.** The kernels are AVX2 and AVX-512 intrinsics with no
+portable fallback, so a build on any other architecture stops with an error
+rather than producing a slow one. AVX-512 is used when the CPU has it and AVX2
+otherwise, decided at runtime.
+
+## How it works
+
+Inputs are **frequency domain**: the unnormalised forward transform of each
+segment, in natural order. Produce them with whatever you already use (numpy,
+MKL, FFTW); matchedfilter does not need to own that step.
+
+The filter is built once and reused. Ingest conjugates the templates and
+stores both sides in the layout the correlation loop walks, which costs a few
+percent of a run and less as the batch grows.
+
+`run` returns a structured array of shape `(ndata, ntemplates, nbins)` with
+fields `index`, `value` and `magnitude`. Bins whose peak fell below the
+threshold carry `index == -1`.
+
+Supported lengths are 1024 and the powers of two from 4096 to 1048576.
+
+### Performance
+
+Per (data, template) pair, 8x32 batch, one core of a Zen 5 desktop:
+
+| n | matchedfilter | numpy | |
+|---:|---:|---:|---:|
+| 1024 | 0.61 µs | 18.98 µs | 31x |
+| 4096 | 2.08 µs | 37.81 µs | 18x |
+| 16384 | 9.98 µs | 127.04 µs | 13x |
+| 65536 | 48.87 µs | 568.56 µs | 12x |
+
+numpy is a floor, not a rival. It is there so the comparison runs anywhere.
+Against MKL or FFTW the margin is much smaller, and part of what is left comes
+from computing peaks instead of a full correlation. Measure on your own box:
+
+```bash
+python -m matchedfilter.benchmark
+```
+
+## Hierarchical filtering
+
+`HierarchicalFilter` adds a cheap pre-pass: correlate against a low-frequency
+slice of the template, and only run the full-length filter where that slice
+leaves a peak plausible.
+
+**This helps only under an assumption about your templates**: that enough of
+the matched-filter output power sits in the low band that a narrow slice gives
+a usable bound on the full result. For chirp-like templates whose power is
+concentrated at low frequency that tends to hold. For templates whose power is
+spread flat across the band, or concentrated high, the slice bounds nothing
+useful, and the pre-pass is pure added cost. It is worth checking against your
+own templates before relying on it.
+
+```python
+hf = mf.HierarchicalFilter(16384, ndata=16, ntemplates=64,
+                           snr=6.0,   # threshold you intend to use
+                           fd=1e-3,   # false-dismissal budget
+                           band=2048) # width of the cheap slice
+
+hf.set_reference(expected_output_power)      # real frequency series, the OUTPUT
+hf.set_templates(template_spectra)
+hf.set_data(data_spectra)
+peaks = hf.run(binsize=16384, threshold=6.0)
+```
+
+`set_reference` takes a real frequency series of length `n` holding the expected
+power of the filter **output** in each bin. Only its shape is used; the overall
+normalisation is divided out.
+
+This is the output, not the template. The two differ whenever the data is
+coloured, and passing the template's own power will mis-set the gate: a
+broadband template reconstructing a narrowband signal is the case where it goes
+wrong by the largest factor.
+
+The gate is one-sided by construction: peaks it reports are bit-identical to
+the flat filter's. It can only omit, never invent. `fd` is the budget for how
+often it is allowed to omit one.
+
+On pure noise at n=4096 the gate runs about **7x faster** than the flat filter.
+The saving scales with how little survives, so it grows with your threshold and
+falls toward 1x on data where most pairs trigger.
+
+## Caveats
+
+- **The false-dismissal budget is not currently met at low thresholds.**
+  `fd` is honoured well at snr 6 and above. At snr 5.0 to 5.5 with a coarse
+  band the gate omits more than it should: 1.4% against a 0.1% budget in a
+  418-template search. The cause is that the gate's recovery factors are
+  measured from a mean frequency series, which is not a bound on any
+  individual realisation. Tracked by an `xfail` test in `tests/test_api.py` and written up
+  in [docs/hierarchical.md](docs/hierarchical.md).
+- Single-threaded by design. Parallelism is the caller's to arrange.
+- x86-64 Linux only, as above. Other architectures are not implemented rather
+  than merely untested.
+
+## Development
+
+```bash
+pip install -e .[test]
+pytest              # includes a C test that builds itself from source
+python -m matchedfilter.benchmark
+```
+
+[docs/](docs/) holds the design notes: how the hierarchical gate is
+calibrated, and measurements of the approaches that were tried and rejected.
+
+## License
+
+MIT

matchedfilter-0.1.0a1/docs/design.md

@@ -0,0 +1,72 @@
+# Batched matched filter: design
+
+D data segments and T template segments arrive as spectra: complex vectors of
+length N, the unnormalised forward transform of each segment. For every pair the
+output is
+
+    z_dt[k] = IFFT( D_d[f] * conj(H_t[f]) )[k]
+
+reported as a binned maximum over a search window, with a detection floor.
+D and T are arbitrary.
+
+## Where the time goes
+
+Forward transforms number D+T. Pair work is D*T. At D=T=16 the pair loop is
+~89% of the work before any optimisation, so every design decision is made about
+the pair loop.
+
+Per pair, done naively:
+
+    read D_d           N complex
+    read H_t           N complex
+    write product P    N complex
+    read P             N complex     <- IFFT stage A
+    write intermediate N complex
+    read intermediate  N complex
+
+At 2^20 that is 48 MiB per pair, and 256 pairs is 12 GiB.
+
+## What reuse buys
+
+1. **Fuse the product into the IFFT's stage-A load.** The product never has to
+   exist in memory: stage A reads `D_d` and `H_t` and multiplies on the way in.
+   Removes 2 of the 6 passes, 16 MiB of 48 per pair.
+
+2. **Pre-permute the stored spectra into stage-A order.** Stage A reads
+   `x[n2*N1 + n1]` walking n2, i.e. with stride N1, measured at 7.6 GB/s against
+   43.9 sequential on this core. Storing the spectra as `X'[n1*N2 + n2]` makes
+   that read contiguous. The permutation costs one transpose per segment (D+T of
+   them) instead of a strided read per pair (D*T of them): at D=T=16, 32
+   transposes to avoid 256 strided passes.
+
+3. **Cache-block the (d,t) loop.** The matmul argument: a tile of nd×nt pairs
+   loads nd+nt spectra and does nd*nt work, so traffic falls by roughly the
+   harmonic mean where the spectra fit.
+
+4. **Conjugate templates once at ingest**, not per pair.
+
+## What does not work, and why
+
+- **Pushing butterflies across the product.** The product is elementwise in
+  frequency and butterflies mix frequencies, so no part of the IFFT can be
+  pre-applied to `D` or `H` separately. Only the *permutation* commutes with an
+  elementwise product, which is why (2) works and a pre-butterflied store does
+  not.
+- **Sharing IFFT work between pairs.** Different pairs have different inputs;
+  there is no common subexpression beyond what (1) to (3) already capture.
+
+## Why the input is frequency domain
+
+Taking time-domain segments would let the ingest rearrangement fuse into a
+forward transform the library performed itself, making it free. Measured, that
+rearrangement is only 1.9 to 4.5% of total, and shrinks as T grows:
+
+| N | D × T | ingest | pair loop | share |
+|---|---|---:|---:|---:|
+| 2^12 | 16×16 | 26.7 µs | 652 µs | 3.9% |
+| 2^12 | 16×256 | 228 µs | 11566 µs | 1.9% |
+| 2^14 | 16×16 | 104 µs | 2940 µs | 3.4% |
+| 2^16 | 16×64 | 2470 µs | 55566 µs | 4.3% |
+
+Owning the forward transform is not worth that, and the caller's pipeline
+generally has the spectra already.