phasenet 0.2.6.dev1__py3-none-any.whl → 0.2.6.dev3__py3-none-any.whl

phasenet/data/README.md

@@ -0,0 +1,313 @@
+# Data pipeline and augmentations
+
+The goal is generalization: a model that picks well on recordings it has never seen. Two rules serve it. Train on
+what inference sees: the waveform is normalized by the same layer inference uses, with the same neighbours around it.
+Widen what training sees: the waveform is changed the way real recordings differ (more events, more noise, other
+filters, missing components, gaps, a missing start), so new stations, cables and conditions look familiar.
+
+Two datasets share the pipeline: `ceed.py`, the stations of one or more events with three components, and
+`das.py`, the channels of one cable with one component. A *trace* below means a station or a channel. The
+transforms all live in `transforms.py`.
+
+## The pipeline
+
+**Input.** Seismic: 16 consecutive station records packed into one sample (several small events together, a large
+event split), each event's stations sorted by distance; 3 components, 12288 samples at 100 Hz (123 s), each record's
+first P at sample 4096; labels are P and S picks per station, first-motion polarity, and magnitude. DAS: one cable
+record, all channels, 1 component, the file's length at its own rate; labels are P and S picks per channel, and magnitude from the events table. One window in
+ten is a noise window instead (Noise).
+
+| # | Stage | Sensor | What it does | Recipe | Why, and what it bought |
+|---|---|---|---|---|---|
+| 1 | `RandomCommonMode` | DAS | removes the median over channels at every sample | p 0.8 | the caller's usual choice at inference |
+| 2 | `RandomHighpass` | both | highpasses the record at a corner log-uniform in 0.5–2 Hz | p 0.5 | the caller's filter, or none; the eval uses 1 Hz |
+| 3 | `ResampleTime` / `ResampleSpace` (loader) | DAS | stretches the record 0.3–3× in time, or in space, after 1 and 2 | ¼ of the visits each | other distances and channel spacings; filtering an upsampled record cost 2.4× |
+| 4 | `RandomCrop` | both | cuts the 4096-sample window plus its context, 1024 samples before it (a cable: 2048 channels), holding a pick; some windows start just before a P | p_start 0.15 | the normalizer's context (Context); a window that ends just after a P |
+| 5 | `StackCopies` | both | adds copies: other windows of the same record, each holding an event, placed where no events overlap and scaled down so every event in the window stays visible. An event is visible when its S/N reaches a floor at its deciding trace: the median of its best traces by S/N | in half the windows (`--stack-prob 0.5`); 1 + Poisson(1) copies, one on average (`--stack-rate 2 2`: the mean copies when it fires); floor S/N 1 on stations, 0.5 on channels (`--stack-snr`); deciding trace: the median of the 5 best stations, or 300 best channels (`--stack-rank`) | several events per window; without it P recall falls 0.08 and the faint cells collapse |
+| 6 | `StackNoise` | both | adds noise (Noise) to lower the faintest P's S/N to a target drawn log-uniform between the floor and its present S/N, capped at 10 (a target never above S/N 10); one amplitude per station, or one per cable | off | a fainter recording |
+| 6b | `FlipPolarity` | seismic | flips the sign of the waveform and of every first-motion label | p 0.5 | the same earthquake with its moment tensor negated: the polarity head reads the sign from the onset, not from a prior |
+| 7 | `RandomDeadComponents` | seismic | zeroes E and N on a share of stations | 0.3 | vertical-only stations at inference |
+| 8 | `RandomArtifact` | seismic | per station, a DC step of 5–50 × rms, or clipping at 10–50 % of the peak | step 0.2, clip 0.1 | two failure modes of continuous station data, at no score cost |
+| 9 | `FlipLR` | DAS | reverses the channel order | p 0.5 | the cable read from its other end |
+| 10 | `RandomGap` | DAS | zeroes 32–256 samples on every channel, 5 s or more from any pick | p 0.2 | a gap in the data |
+| 11 | `RandomDropContext` | both | removes the context before the window (and after it, for `centred`) | p 0.6 | at inference the first window of a file has no samples before it, so the normalizer falls back on its edge rule; without this stage every training window would arrive with context, and the model would never see that case (Context) |
+| 12 | `InputNormalize` | both | the model's input normalization (the modes: Context) | `trailing`: each sample scaled by the 1024-sample span before it; at a trace's start, `mirror`: the first 256 samples mirrored to fill the span | the same layer, with the same neighbours, as at inference |
+| 13 | `DropContext` | both | keeps the last 4096 samples | | the model's window |
+| 14 | labels | both | the label arrays (Labels) | `--weight-sp`: each trace's phase loss weighted by how rare its S − P is in the training set, mean weight 1 (Labels); `--ps-pairs-only`: a window with no P–S pair is skipped, so every window trains the phase head; seismic `--ignore-unassociated`: picks no earthquake took are kept, with no phase loss within 1 s of them, instead of being dropped and their waves trained as noise (DAS drops them: on Arcata 45 % are traffic) | what the model learns |
+| 15 | batch | both | seismic 16 windows × 16 stations; DAS 4 records × 48 windows × 2048 channels | | |
+
+Stages 5 to 11 add what a recording can contain: other events, more noise, missing components, instrument failures,
+gaps, a missing start. The noise comes after the copies and is judged on the faintest of them, so every label stays
+visible and the window still gets fainter. Stages 12 and 13 make it the model's input; nothing else touches the
+waveform. Each `Random…` transform applies with a probability.
+
+## Context: where training and inference differ
+
+The input normalization (`phasenet/normalize.py`, `NormalizeSpec`, saved in the checkpoint) is a fixed layer with no
+parameters: each sample is demeaned and divided by a scale measured on other samples. Its settings:
+
+| Setting | Options | The scale of a sample comes from |
+|---|---|---|
+| `kind` | `trailing` (the recipes) | the span of `window` samples before it (1024): causal, so a small event before a larger one keeps its amplitude; it streams |
+| | `centred` | the span of `window` samples around it (512 on each side); it can stream only with a delay of half the window |
+| | `window` | the whole window the model sees: the crop comes first, and each inference patch is normalized alone |
+| | `tf_interp` | TF PhaseNet's centred window, measured every `window / 2` and interpolated: only for the PhaseNet-2018 port |
+| `scope` | `station` (the recipes) | pooled over a station's components, which keep their relative amplitudes (a DAS channel is a one-component station) |
+| | `component`, `global` | each component alone; one scale for the whole window |
+| `stat` | `std` (the recipes), `l1` | the standard deviation; or the mean absolute deviation, rescaled to the std of Gaussian noise |
+| `edge` | `mirror` 256 (the recipes) | at a trace's start, where the span has no samples before: the first 256 samples mirrored back and forth fill it |
+| | `shared` | at a trace's start: the first `window` samples all take the scale of the first span |
+
+Three lengths, kept apart below:
+
+- the **window**: the 4096 samples the model sees;
+- the **normalization span** (`NormalizeSpec.window`): the 1024 samples whose statistics scale one sample;
+- the **context** (`sample.context`): the samples cut with the window, 1024 before it (and 1024 after, for
+  `centred`), so that the window's first samples have a full normalization span of real data.
+
+Inference normalizes the whole trace, so every sample's span is real data, except at the trace's start, where the
+edge rule fills it. Training sees 4096-sample windows cut from 12288-sample records, so the pipeline has to reproduce
+both cases.
+
+| At inference | In training |
+|---|---|
+| **Mid-stream.** The window's first samples are scaled by the real data before it. | `RandomCrop` keeps 1024 context samples in front of the window, the normalizer runs on both, `DropContext` removes the context. Without it the window's first 10 s would be scaled by statistics that include the P, about 5× smaller than inference produces. |
+| **The first window of a file.** No data before it; the normalizer applies its edge rule. | `RandomDropContext` removes the context from 60 % of the windows before normalizing; the same edge rule applies. One file start per file at inference, but it costs the file's first 10 s. |
+| **The convolutions.** Zero padding at each window's edges. | The same. Nothing to do. |
+
+The edge rule lives in the normalizer only. `centred` truncates its window at both ends, so for it
+`RandomDropContext` also removes the context after the window, the last window of a file.
+
+**A window may start at a record's first sample.** Every CEED record has its P at sample 4096, so a window that ends
+just after that P must start at sample 0, where there is no context before it. `RandomCrop` therefore lets the cut
+begin up to 1024 samples before the record: the window arrives with partial context or none, and the normalizer's
+edge rule fills the span, as at the start of a file. Without this, every window would start at sample 1024 or later,
+a P could never sit in the last 1024 samples of a window, and the model would never learn to pick a P whose S has
+not arrived yet.
+
+## Labels
+
+Labels come from two tables, the interface of iterative training: each round predicts, associates and rewrites
+them, and the next round trains on them (`tables.py`). Both read from csv or parquet.
+
+- **picks**: record key, `origin_id`, `station_id` (a cable: the channel index), `phase_index`, `phase_type` (P or
+  S), `phase_polarity` (U/D, optional).
+- **events**: record key, `origin_id`, `time`, `latitude`, `longitude`, `depth_km`, `magnitude`.
+
+Every row says which record it belongs to, and which earthquake within that record:
+
+| Column | Names | Values |
+|---|---|---|
+| record key: `event_id` (CEED) | the record. A CEED record is cut around one catalog earthquake, so it is named by that earthquake's catalog id | e.g. `nc73891226` |
+| record key: `file_name` (DAS) | the record. A DAS record is a file that may hold no earthquake or several, so it is named by the file (without `.h5`); a DAS picks table is one file per record, its key column repeated | e.g. `20230111T171615Z` |
+| `origin_id` | an earthquake **within** its record, as association numbered them: 0, 1, 2, …; −1 or empty for a pick no earthquake took. The same number in two records means two different earthquakes | 0 for the catalog earthquake of a CEED record in round 0 |
+
+So the pair (record key, `origin_id`) names one earthquake. `origin_index` (written by association, Association
+below) is not a name but a time: that earthquake's origin, as a sample index in the record. A DAS events table may
+also keep, per origin, the catalog `event_id` it matched.
+
+A window can hold several events, on both sensors, so the loaders make one `Target` per associated origin: its
+picks, its P–S pairs (each P with the first S after it on its station) and its magnitude from the events row.
+Round 0 is the catalog: without `--label-path` the seismic loader turns CEED's own columns into the two tables
+(origin 0); the Arcata events table, from the records' catalog attributes joined to the published picks' origins,
+is `quakeflow_das/arcata/events.csv` (`scripts/das_catalog_events.py`).
+
+**How the arrays are made.** `sample.generate_labels(sample, LabelConfig)` turns a window's Targets into the arrays
+the losses read, after every augmentation, so copies and stretched or flipped picks are labelled like any other.
+Positions and widths are in samples (the numbers below are the defaults at 100 Hz): the model knows samples, not a
+rate, so the same weights serve a 100 Hz station and a 250 Hz cable; physical parameters follow
+`sample.sampling_rate` (filter corners, the S/N measure `transforms.levels`, the gap's 5 s margin). A width in
+`LabelConfig` is a full width: sigma is width / 6, drawn over ± 3 sigma, values under 0.05 set to 0.
+
+**What each head learns**, per trace (station or channel) and sample:
+
+| Output | Shape | Value |
+|---|---|---|
+| `phase_pick` | (3, nx, nt) | Channels noise, P, S. A Gaussian at each P pick and each S pick (`phase_width` 60, sigma 10 samples, 0.1 s). Overlapping picks add; noise = max(0, 1 − P − S). |
+| `polarity` | (1, nx, nt) | 0.5 + 0.5 × sign × a Gaussian at each first motion with a sign (`polarity_width` 20, sigma 3.3 samples): 1 up, 0 down, 0.5 elsewhere. |
+| `event_center` | (1, nx, nt) | A Gaussian at the P-S midpoint (P + S) / 2 (`event_width` 150, sigma 25 samples), on traces whose P and S are both in the window. |
+| `event_time` | (1, nx, nt) | Samples since the origin: t − center + the travel time to the center, that travel time from S − P with vp 6 km/s and vp/vs 1.73. Within ± 225 samples of the center. |
+| `magnitude` | (1, nx, nt) | The event's catalog magnitude, over the same span as `event_time`. |
+
+**Where each loss runs** (the masks):
+
+| Mask | Where it is 1 (or a weight) |
+|---|---|
+| `phase_mask` | Where the phase loss runs. Only on traces with an **associated P–S pair**: a P and the first S after it on the trace, both of one origin. A lone P or S, and unassociated picks, get none. After the cut, a pair keeps its mask while the window holds either phase: a P at the window's end whose S was cut, an S at its start whose P was cut. **One event** on every trace of the window: the whole trace. **Several events** on a trace, real or added by stacking: only [P − 5 s, S + 2 (S − P) + 5 s] around each event (`loss_pad_s`), so no event's waves are trained as another's noise (WZ 11-01). The value is 1, or the S − P weight (below). 0 within ± 1 s (`ignore_half_s`) of every unlabelled arrival (a sign-0 entry: an unassociated pick kept by `--ignore-unassociated`). |
+| `polarity_mask` | ± 90 samples (`polarity_mask_width` 180) around each first motion with a sign: as wide as a P pick may be off, wider than the polarity Gaussian, which is 0.5 (no sign) outside its ± 10 samples. |
+| `event_center_mask` | The event's traces: the whole trace for a one-event window, otherwise [P − 5 s, S + 2 (S − P) + 5 s] around each event. |
+| `event_time_mask` | ± 225 samples (`event_time_mask_width` 450) around each event center. |
+| `magnitude_mask` | Same as `event_time_mask`, where the event has a catalog magnitude and the trace was not noised by StackNoise (`sample.nomag`). Copies carry no magnitude. |
+
+**The S − P weight** (`--weight-sp`, `LabelConfig.sp_weight`): w = N / (B · count) over √2-wide log bins of S − P,
+0.7–23 s, counted at start on up to 200 training files (`tables.sp_samples`); capped at 8× the equal share, then scaled
+to mean 1. The phase loss divides by the mask's sum, so it is a weighted mean over the batch, Σ w·loss / Σ w: rare
+distances count more than common ones in the same batch, the overall scale of the weights cancels, and mean 1 keeps
+the weighted traces on par with the noise windows' weight-1 "no pick" rows. On DAS a batch's crops come from 4
+records, whose crops share their S − P, so the weight acts mostly between records.
+
+> **Not yet tested (WZ 11-01).** Whether this weight is optimal is open. To test: train each sensor with no weight,
+> this weight, and its square root (a milder w^½; `--weight-sp 0`, `1`, `0.5`), two seeds each, and compare recall by S − P (distance) cell and the
+> recall in the common near cells (rule: the far cells must gain without the near ones falling, as How the recipe is
+> chosen requires).
+
+## Noise
+
+One rule for both sensors and both uses, the noise windows (one output in ten, `noise_window_prob`, held by
+`NoiseShare`: a failed one stays due) and `StackNoise` (`transforms.noise_sample`, WZ 11-01). Noise is always a
+training crop's length with its context (window + context).
+
+| | From the noise list (`--noise-list`) | From a training record (`SampleBuffer.noise`) |
+|---|---|---|
+| Earthquake check | none: the files are trusted | `earthquake_mask` on the traces with a P–S pair (the screen, below) |
+| Span | a random span of the file | a span drawn uniformly among all the record's earthquake-free starts (`noise_span`), anywhere outside the earthquakes' spans. DAS: between the events of a long record. CEED: the head before 2 s ahead of the P, or more often the tail after the event's span ends (69 % of seismic spans, 11-01) |
+| Shorter than needed | reflected at its end to the length (`np.pad` "symmetric", no jump at the seam); skipped under half of it | the same |
+| "No pick" trained on (`Sample.noise_traces`) | every trace | the traces with a P–S pair: the others may hide an unlabelled event (G17) |
+| Unlabelled arrivals | none | the record's that fall in the span, ±1 s without phase loss (G16) |
+
+`StackNoise` adds noise only on the trusted traces. In a noise window, "no event" is trained on every trace, and there
+is no event-time loss. A seismic head is 2 s shorter than a window and is reflected by 1224 samples when it is drawn; up to half of a
+span can be a mirrored copy of the rest, harmless for quiet noise.
+
+**The earthquake screen** (`earthquake_mask`) marks where a training record holds an earthquake, so noise is cut
+elsewhere. It reads only the traces with a P–S pair: only they estimate an origin time, so only they can be cleared
+(WZ 11-01). One rule picks the case:
+
+| | With origin times (the picks carry `origin_index`) | Without (the catalog round) |
+|---|---|---|
+| Reads | the origin time of every P–S pair, associated or not (Association) | the waves of the P–S traces, 2–15 Hz |
+| An earthquake is | a 2 s stretch where at least `min_origins` traces have an origin and the count rises above the record's background (median + 20 MAD of its 2 s counts) | a 2 s stretch where more than `trigger_fraction` of the traces trigger: a trace's 1 s energy stays above 8× its median (over the seconds with data, not gaps or padding) for 1 s |
+| Thresholds: stations / a cable | 3 stations / 5 channels | any one station / 30 % of the channels (an earthquake crosses a cable in seconds; traffic and noisy channels do not) |
+| It spans | from 2 s before its first P to its last pick + (S − P) + 1 s | the triggered time |
+| Misses | an earthquake no pick found (no origin) | small events under the trigger; a spike trips it |
+| Measured | median 0–27 origins of unassociated pairs per 2 s on Arcata (RESULTS 11-01) | 97 % of catalogued seismic events and 21 of 21 Arcata events found, 11 % of CEED heads and 2 % of DAS noise cuts rejected (RESULTS 10-29) |
+
+## Association: picks to earthquakes by origin time
+
+Each round of iterative training picks every record, then groups the picks into earthquakes by their origin time
+(`phasenet/utils/postprocess.py`, called by `scripts/predict_das.py` and `scripts/predict_ceed.py`). The idea: a
+trace's arrivals move with distance, but the origin time of one earthquake is the same on every trace. So each trace
+estimates an origin time, and traces that agree belong to one earthquake.
+
+**1. One origin time per trace.** Each P is paired with the first S after it on the same trace, with S − P in
+[`min_ps_dt`, `max_ps_dt`) (`estimate_origins_from_ps_pairs`). A trace at distance d records tP = t0 + d / vp and
+tS = t0 + d / vs, so the pair's center and interval are
+
+    tS + tP = 2 t0 + d (1/vs + 1/vp)
+    tS − tP = d (1/vs − 1/vp)
+    =>  t0 = (tS + tP) / 2 − (tS − tP) / 2 · (vp + vs) / (vp − vs)
+
+The origin is the P–S center minus (vp + vs) / (vp − vs) half-intervals (2.74 with vp / vs = 1.73). The center and
+the interval are what the labels hold (`ps_centers`, `ps_intervals`) and the event head learns, so the model's own
+estimate follows the same equation: the predicted event center minus its predicted center-to-origin offset
+(`estimate_origins_from_event_heads`). An error δ in either pick moves the center by δ / 2: neither phase is the
+anchor.
+
+**2. A histogram of origin times.** All traces' t0 go into bins of `bin_size`. An earthquake is a peak of at least
+`mph` traces, at least `mpd` bins from a higher peak (`associate_picks`).
+
+**3. Linking.** For each peak, the median t0 of the pairs within one bin is the event's origin. Every pair within
+`origin_window` of it joins the event. On a cable, a RANSAC fit of the P–S midpoint against the channel (a quadratic
+moveout) then drops pairs off the curve (`moveout_window`), when at least 10 pairs remain.
+
+**4. The picks table.** A linked pick gets its event's `origin_id` and `origin_index` (the event origin, in samples).
+A pick in a pair no event took keeps `origin_id` −1 and its own pair's t0 as `origin_index`; an unpaired pick has
+none. `ps_center` and `ps_interval` record the pair.
+
+| Parameter | Seismic (CEED) | DAS (Arcata) | Why it differs |
+|---|---|---|---|
+| `bin_size` | 1 s | 0.1 s | a cable's channels are close, so their origins agree far more tightly |
+| `mph` (traces at a peak) | 5 | 100 | 16 stations at most, against thousands of channels |
+| `mpd` (bins between peaks) | 5 (5 s) | 50 (5 s) | the same 5 s between two earthquakes |
+| `origin_window` | 1 s | 0.2 s | as `bin_size` |
+| `moveout_window` (RANSAC) | none | 0.2 s | too few stations for a fit |
+| S − P range | 0.5–20 s | 1–30 s | |
+
+The same origin times screen the noise sources (Noise, above).
+
+## The loaders
+
+`CEEDIterableDataset` (`ceed.py`) and `DASIterableDataset` (`das.py`) turn records into training outputs in the same
+steps; only reading a record is sensor-specific. A *record* is what a file holds (a CEED station record, a DAS h5
+file); a *window* is the 4096 samples the model sees.
+
+| Step | Seismic | DAS |
+|---|---|---|
+| Files | QuakeFlow day files (`quakeflow_paths`, GCS or a local copy) or the CEED Hub dataset | one h5 per record, local or GCS (`list_files`) |
+| Workers | the DDP ranks split the files (as many on every rank), then `worker_share`: every n-th file per DataLoader worker; in training, one file each in turn when the workers outnumber the files | the same |
+| Read | `read_records`: the parquet waveforms as one raw buffer; each record carries its day's picks | `read_h5`: the raw data, NaN → 0 (as a seismic record: every stage is scale-free, and the normalizer demeans and scales each trace); kept in memory under 256 MB |
+| A sample | `_groups` → `records_to_sample`: 16 consecutive stations (small events packed together, large ones split), each event's stations by distance, `station_event_index` per station | the whole record, visited again and again, `num_patch` crops per visit |
+| Record-level changes | in the `Compose`: a record gives one window per visit, so once per window is once per record | Each time the loader visits a record it applies, to the whole record: `RandomCommonMode`, then `RandomHighpass`, then `ResampleTime` or `ResampleSpace` (or neither). All `num_patch` crops of that visit share these draws. They run outside the `Compose` for speed: on a 7550 × 52500 record each costs seconds, too slow to repeat for every crop |
+| Output | `label_output`: data, the label arrays (`LABEL_KEYS`, the same keys on both sensors), the event labels every 16th sample, the logged rates | the same, the event labels also max-pooled over 16 channels (the max keeps a group labelled when only some of its channels carry picks; a label is pooled over the channels its mask covers, so an unlabelled 0 never beats a negative event time or magnitude); `polarity` and `polarity_mask` a (1, 1, 1) zero: the dataset knows a cable has no first motions, and two full-size zero arrays made 1.4 GB batches (11-01) |
+
+**A training crop that is redrawn** (DAS): up to 10 tries (`CROP_TRIES`) while its event-center mask covers under 2 %
+(`MIN_EVENT_SHARE`) or, with `--ps-pairs-only`, it has no phase loss.
+
+## Flags
+
+The same flags serve both datasets. `--recipe seismic|das` sets every one of them to the released recipe
+(`RECIPES` in `train.py`); a flag given beside it overrides one value.
+
+| Flag | What it sets |
+|---|---|
+| `--norm-kind`, `--norm-window`, `--norm-edge`, `--norm-edge-window` | `NormalizeSpec`, saved in the checkpoint; inference follows it |
+| `--label-path`, `--events-path` | the round's picks tables (a folder) and events table (a file) |
+| `--ignore-unassociated` | unassociated picks kept as sign-0 entries (moved by every crop and stack): no phase loss within 1 s; else dropped |
+| `--ps-pairs-only` | train only on windows with a P–S pair |
+| `--weight-sp P` | the phase loss weighted by S − P (Labels, `phase_mask`): the inverse-frequency weight to the power P (recipes 1; 0.5 its square root; 0 off) |
+| `--p-start-prob` | `RandomCrop`: the share of windows that start just before a P |
+| `--stack-prob`, `--stack-rate`, `--stack-snr`, `--stack-rank` | `StackCopies`; the rank also sets the cable's `StackNoise` channel |
+| `--noise-stack-prob`, `--noise-list` | `StackNoise`; the trusted noise files the noise windows and `StackNoise` draw from (Noise) |
+| `--noise-window-prob` | the share of noise windows |
+| `--flip-polarity-prob` | `FlipPolarity` (seismic) |
+| `--dead-components-prob` | `RandomDeadComponents` |
+| `--artifact-step-prob`, `--artifact-clip-prob` | `RandomArtifact` (seismic) |
+| `--drop-context-prob` | `RandomDropContext` |
+| `--common-mode-prob`, `--flip-lr-prob`, `--gap-prob` | `RandomCommonMode`, `FlipLR`, `RandomGap` (DAS) |
+
+Training logs the realized rates beside the losses (wandb `rates/`): the share of windows with a copy, copies per
+window, the share noised and the share of noise windows. A recipe sets probabilities; a copy can fail to fit.
+
+## Inference
+
+`predict_2d` normalizes the whole trace first, with the spec stored in the checkpoint. The model does not normalize
+in its `forward`: the normalizer is a fixed layer applied outside it, on the whole record in training (`InputNormalize`,
+with the context) and on the whole trace at inference, so both give every sample the same neighbours. The checkpoint
+carries the spec, so an old model keeps its own normalization. Then one of two modes:
+
+- **Windows** of the training length, overlapping by half. Each sample takes its score from the window whose
+  centre is nearest.
+- **Whole trace** (`--whole-trace`, `input.window_nt` None). The trace is padded to a multiple of the network's
+  downsampling factor with its own tail mirrored, never with zeros: a wall of zeros inside the receptive field
+  suppresses picks near the end of a file.
+
+The `trailing` kind also streams chunk by chunk with state (`Normalizer.stream`). `centred` cannot stream without
+a delay of half its window. The `window` kind normalizes each patch alone and refuses the whole trace.
+
+## Not yet consistent
+
+- No DAS augmentation has been measured against an evaluation set; the 11-01 label and noise changes (`--weight-sp`
+  mean 1, copies counted as events, one noise rule) wait for a training run on each sensor.
+
+## How the recipe is chosen
+
+A recipe is better only if it holds every check against the reference (PhaseNet-2018 for stations; for a cable the
+pretrained PhaseNet-DAS and the main-branch pipeline trained on the same data and steps), in two seeds, one change at
+a time; a gain smaller than the spread between seeds is not believed, and a mean that improves while one cell falls is
+not an improvement. The checks, on the held-out, fixed test set:
+
+1. **Recall by magnitude × distance and by S/N × distance**, cell by cell, on raw and 1 Hz highpassed input; a cell
+   with few events (on a cable: events, not picks) is reported, not decided on.
+2. **Precision-recall over the pick threshold**, at equal recall: unlabelled earthquakes count as false picks, so
+   precision ranks models but its level is biased against the better detector.
+3. **Stability over the window**: the recall at the worst pick position, the first window of a stream, false picks
+   in the coda and in noise.
+4. **The errors themselves**: galleries of the clearest misses and false picks, at the onset (±2 s) and whole; many
+   false picks are real small earthquakes, many misses are picks in the wrong place.
+5. **Timing**: recall at 0.1, 0.2, 0.3 and 0.5 s and the signed residual by distance and S/N.
+6. **Robustness to artifacts**: gaps, spikes, DC steps, clipping on the eval records.
+7. **The other heads hold**: magnitude error and polarity agreement do not get worse.
+
+Measure the data before training (copies per window, label S/N, share noised, picks per window, labels without a
+loss); keep the simplest recipe that holds every check; stop when one does, or when the remaining gap is shown to be a
+label problem, not an augmentation problem.

phasenet/data/__init__.py

@@ -1,7 +1,6 @@
-from .transforms import (
-    Transform, Compose, Identity, Normalize, RandomHighpass, RandomCommonMode,
-    RandomGain, ColoredNoise, RandomBandpass, ResampleTime,
-    RandomCrop, SampleBuffer,
-)
-from .das import DASDataset, DASIterableDataset
-from .ceed import CEEDDataset, CEEDIterableDataset
+"""The training data: one dataset per sensor. Everything else is imported from its module: transforms.py (the
+augmentations), sample.py (Sample, Target, labels), tables.py (picks and events tables, file access)."""
+from .ceed import CEEDIterableDataset
+from .das import DASIterableDataset
+
+__all__ = ["CEEDIterableDataset", "DASIterableDataset"]