@motionstudies/data 0.1.0-alpha.1 → 0.1.0-alpha.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +171 -2
- package/adsb-heatmap.d.mts +56 -0
- package/adsb-heatmap.mjs +223 -0
- package/air-endpoints.d.mts +5 -0
- package/air-endpoints.mjs +130 -0
- package/network-chunk-cli.d.mts +1 -0
- package/network-chunk-cli.mjs +65 -0
- package/package.json +15 -3
package/README.md
CHANGED
|
@@ -4,8 +4,8 @@ Shared packages for the Motion Studies transport instrument. The source workspac
|
|
|
4
4
|
|
|
5
5
|
- `@motionstudies/core`: transport contracts, indexing, interpolation and visual theme contracts; no browser or Node dependencies.
|
|
6
6
|
- `@motionstudies/three`: `NationalNetworkScene`, `HubPulseScene`, `StationFlowScene`, camera framing and label-mode contracts. React, React Three Fiber and Three.js are peers; rendering internals are not public subpaths.
|
|
7
|
-
- `@motionstudies/web`: picker, theme application, mounting, progressive loaders, observed operations and recording. Import `tokens.css` and `mobile-picker.css` for isolated widgets. `shell.css` is an optional full-page study shell scoped to `.motion-study`; `mountMotionStudy` applies that class. Fonts and edition layouts belong to consumers.
|
|
8
|
-
- `@motionstudies/data`: Node-only GTFS readers, network chunking, merging and station ranking. ZIP reading requires `unzip` on the host. Source selection, provenance overrides and compilation commands belong to each edition.
|
|
7
|
+
- `@motionstudies/web`: picker, button tooltips, theme application, mounting, progressive loaders, observed operations and recording. Import `tokens.css` and `mobile-picker.css` for isolated widgets. `shell.css` is an optional full-page study shell scoped to `.motion-study`; `mountMotionStudy` applies that class. Fonts and edition layouts belong to consumers.
|
|
8
|
+
- `@motionstudies/data`: Node-only GTFS readers, ADS-B heatmap compilation, air endpoint enrichment, network chunking, merging and station ranking. ZIP reading requires `unzip` on the host. Source selection, provenance overrides and compilation commands belong to each edition.
|
|
9
9
|
|
|
10
10
|
```tsx
|
|
11
11
|
import { MobilePicker } from '@motionstudies/web/components/MobilePicker'
|
|
@@ -24,3 +24,172 @@ Keep the resolver stable across renders. Manifest paths and their chunk paths ar
|
|
|
24
24
|
Build release candidates with `npm run build:packages`. Distribution manifests and compiled ESM/declarations are written to `.package-dist/`; workspace manifests continue to point at source for fast local iteration. `npm run check:packed` packs and installs those distributions into a separate consumer, builds the lab and validates the public exports. No source aliases or workspace links are used in that consumer.
|
|
25
25
|
|
|
26
26
|
Source workspace manifests always stay private. `npm run check:release` builds public candidates, tests their packed consumer and records the tested tarball hashes; `npm run release:dry-run` inspects the publication without writing to npm. The manual main-branch `release.yml` workflow publishes those same tarballs with public access and provenance. See [release instructions](https://github.com/emmettl/motionstudies/blob/main/docs/RELEASING.md) for bootstrap-token and trusted-publisher setup. All four shared packages are MIT-licensed; each distribution includes `LICENSE`.
|
|
27
|
+
|
|
28
|
+
## Now and browser location
|
|
29
|
+
|
|
30
|
+
`useNowClock` from `@motionstudies/web/use-now-clock` follows the wall clock on each animation frame, so returning from a suspended tab catches up immediately. Call `start()` from a Now button, set the consumer's playback rate to 1, and render its `time` while `active`. With `NationalNetworkScene`, pass `isPlaying={false}` while this external clock owns time. Call `stop()` before pausing, seeking or changing speed, and retain the last clock time for ordinary playback. Clear moving vehicle selections on entry to keep the camera still; panning and zooming remain available.
|
|
31
|
+
|
|
32
|
+
Supply a resolver `(instant: Date) => number | null` that maps the instant into the edition's service-time coordinates. The edition owns timezone, service date, daylight-saving rules and source coverage. Now is appropriate when the data gives a meaningful sense of this place at this time; exact live vehicle positions are not required. An edition may explicitly map the current local clock onto a suitable representative weekday or seasonal timetable, with a quiet label such as “Typical weekday · realtime pace”. Historical or modelled sources need that same meaningful relationship to the present; merely having timestamps is insufficient. Return null outside the chosen data's coverage: the hook stops, retains the last valid time and exposes `unavailable`. Do not silently relabel a recording as live or wrap a partial study window. Timetable and observed-data labels remain the consumer's responsibility; 1× playback does not imply a live vehicle feed.
|
|
33
|
+
|
|
34
|
+
`useBrowserLocation` from `@motionstudies/web/use-browser-location` requests a single position only when `locate()` is called. It exposes `location`, `status` and `clear()`; it neither stores nor sends coordinates. Render localized messages for denied, timeout and unavailable states. Clearing or unmounting discards late responses. A secure browser context and user permission are required.
|
|
35
|
+
|
|
36
|
+
Pass an in-coverage position as `NationalNetworkScene`'s `userLocation` for a steady glowing dot. Use the existing `focus-location` camera command once after locating, with `[longitude, latitude]` and a suitable `distanceScale`. Validate coverage in the consumer and preserve the view when the user is outside it. Show the reported `accuracy` in accessible text; the dot is an approximate position. The marker hides in diagram layouts. The lab's **Now** specimen uses a clearly labelled synthetic UTC timetable and covers clock, permission and out-of-coverage behavior. Public editions still require independent adoption and releases.
|
|
37
|
+
|
|
38
|
+
## Selection labels
|
|
39
|
+
|
|
40
|
+
`NationalNetworkScene` gives the selected station first label priority, followed by the selected route's terminal stops (including branches), then intermediate stops. Selected services use their own endpoints. Priority precedes retained labels and ordinary rank/tier admission; clearing selection restores edition ranking. This is built in for every consumer, including geographic and diagram layouts. Supply complete enabled infrastructure as `referenceSnapshot` to preserve endpoints through timetable gaps; its stop indexes need not match the active snapshot. See the [edition behaviour contract](https://github.com/emmettl/motionstudies/blob/main/docs/EDITIONS.md#selection-and-station-labels).
|
|
41
|
+
|
|
42
|
+
## Button help
|
|
43
|
+
|
|
44
|
+
`mountMotionStudy` installs one shared tooltip surface. Independent consumers such as the lab can render `ButtonTooltips` from `@motionstudies/web/components/ButtonTooltips` once instead. Put concise, action-oriented help in each button’s `data-tooltip`; icon buttons fall back to their `aria-label`. An empty `data-tooltip` opts out. Avoid native `title` attributes on these buttons, which can also appear during touch interaction.
|
|
45
|
+
|
|
46
|
+
Help appears after a short mouse hover or on keyboard focus when the primary pointer is fine and supports hover. Touch input suppresses it, including on hybrid devices. Escape, activation, scrolling and blur dismiss it. The tooltip stays inside the viewport, can itself be hovered, and temporarily extends `aria-describedby` without replacing existing descriptions. Copy and translations stay in the edition; rendering and input handling stay in this package. The Controls specimen and packed-consumer tests exercise this contract.
|
|
47
|
+
|
|
48
|
+
## Airport heroes and split-flap boards
|
|
49
|
+
|
|
50
|
+
`AirportHeroCard` provides an airport identity header and switchable departure/arrival boards. `SplitFlapBoard` is the underlying transport-neutral widget, also suitable for rail stations. Both use scoped package styles, semantic tables, full accessible cell values, keyboard-operable selection, contained horizontal scrolling and reduced-motion support. Only changed characters remount for the flap animation.
|
|
51
|
+
|
|
52
|
+
```tsx
|
|
53
|
+
import { AirportHeroCard } from '@motionstudies/web/components/AirportHeroCard'
|
|
54
|
+
import '@motionstudies/web/airport-hero-card.css'
|
|
55
|
+
|
|
56
|
+
<AirportHeroCard
|
|
57
|
+
key={airport.id}
|
|
58
|
+
airport={airport}
|
|
59
|
+
departures={departures}
|
|
60
|
+
arrivals={arrivals}
|
|
61
|
+
study={{ time, windowStart: metadata.windowStart, windowEnd: metadata.windowEnd }}
|
|
62
|
+
dateLabel={metadata.serviceDate}
|
|
63
|
+
note="Observed study · inferred directions; times are observations."
|
|
64
|
+
onSelectFlight={selectAirTrack}
|
|
65
|
+
selectedFlightId={selectedAirTrackId}
|
|
66
|
+
/>
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Entries have a stable `id`, a `service` label, and optional numeric `time`, `place`, `stand`, `status` and `tone` (`neutral`, `accent` or `warning`). Movement times and `study.time`, `windowStart`, and `windowEnd` must use the same study-relative seconds and service date. Do not parse display strings or normalize numeric times at midnight: an event after 24:00 retains its value above 86,400. `formatTime` optionally controls display formatting; the default uses the study's `formatServiceTime` helper. The header clock is derived directly from `study.time`, with no independent wall clock.
|
|
70
|
+
|
|
71
|
+
The card sorts movements chronologically and shows up to eight rows per direction inside the intersection of the study bounds and a rolling window: ten minutes behind the playback clock and sixty minutes ahead, with inclusive endpoints. Override this with `horizon={{ lookBehindSeconds: 600, lookAheadSeconds: 3600 }}` and `maxRows`. Playback, backward seeking, changed study bounds and updated movement times all recalculate the rows. An out-of-study or invalid clock shows no movements; rows with missing or non-finite times are excluded because they cannot be placed in the window. Other unknown fields render as a dash. Labels include `studyTime`, `boardWindow` and `outsideWindow` for localization. Filtering does not infer operational statuses or clear the consumer's map selection when a row leaves the window.
|
|
72
|
+
|
|
73
|
+
Consumers still own time coordinates, source interpretation and data loading. Supply movements for the displayed horizon, not just aircraft active at the current second, and use the selected study's bounds rather than an individual progressive chunk's bounds. Do not turn an approach-envelope association into a confirmed departure/arrival: unclassified tracks should remain outside these direction lists. Current `AirTrack` data does not supply scheduled times, routes or gates; leave those fields absent, use observation times only when clearly labelled, and explain any inference in the required `note`.
|
|
74
|
+
|
|
75
|
+
Pass `labels` for edition translations, `loading`, or a localized `error` and `onRetry` for data states. The selected direction is local to each card; key the card by airport ID to reset it on selection changes. The Airports lab specimen exercises synthetic timetables, playback, scrubbing, study-window changes, incomplete observations, French labels, long destinations, updates and recovery. Edition adoption happens through their independently pinned package releases; adding this export does not update deployed studies.
|
|
76
|
+
|
|
77
|
+
For a custom rail or transport board, import `SplitFlapBoard` from `@motionstudies/web/components/SplitFlapBoard` and `@motionstudies/web/split-flap-board.css`. Supply `columns` (`key`, `label`, `characters`) and `rows` (`id`, `cells`, optional `tone`). Cell text longer than its flap count is visually ellipsized, with the full value retained for assistive technology and hover. `onSelectRow`, `selectedRowId` and `selectionColumn` optionally make one cell per row selectable.
|
|
78
|
+
|
|
79
|
+
For bus stops and local transport, `DotMatrixBoard` accepts the same rows, selection callbacks, loading state and empty/loading messages. It uses an amber 5 × 7 LED alphabet and its own scoped stylesheet; the flip-board stylesheet is not required. Consumers can switch components without remapping their data.
|
|
80
|
+
|
|
81
|
+
```tsx
|
|
82
|
+
import { DotMatrixBoard } from '@motionstudies/web/components/DotMatrixBoard'
|
|
83
|
+
import '@motionstudies/web/dot-matrix-board.css'
|
|
84
|
+
|
|
85
|
+
<DotMatrixBoard
|
|
86
|
+
label="Bus departures"
|
|
87
|
+
columns={[
|
|
88
|
+
{ key: 'route', label: 'Route', characters: 4, minCharacters: 4 },
|
|
89
|
+
{ key: 'destination', label: 'Destination', characters: 24 },
|
|
90
|
+
{ key: 'time', label: 'Due', characters: 6, minCharacters: 6, align: 'right' },
|
|
91
|
+
]}
|
|
92
|
+
rows={[{ id: 'bus-71', cells: { route: '71', destination: 'City Centre', time: '2 min' } }]}
|
|
93
|
+
lineCount="auto"
|
|
94
|
+
style={{ height: 360 }}
|
|
95
|
+
/>
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
`lineCount` defaults to six display slots and accepts 1–30 (values outside that range are clamped; non-finite values use six). Fixed counts keep all slots and scale their contents to the available height, so dense boards need taller containers to remain readable. `"auto"` observes the actual container and fits 1–30 rows at a target `minRowHeight` of 34px. The default board height is 320px; use `style`, `className`, or `height: '100%'` inside a parent with a defined height. Unused slots stay blank; rows beyond the visible slots are omitted, without pagination or changes to consumer selection.
|
|
99
|
+
|
|
100
|
+
For dot-matrix columns, `characters` is a width weight, while `minCharacters` reserves space before the remaining width is distributed. At very narrow widths all columns scale down. Long cell values are visually ellipsized, with their full original text available to assistive technology and on hover. Characters outside the bitmap alphabet (including accented names and non-Latin scripts) use SVG text as a visual fallback. `--matrix-ink` and `--matrix-unlit` customize the LEDs. Updates are immediate, with no flashing or scrolling animation. The **Bus boards** lab specimen exercises both presentations, height/width resizing, row counts, selection, long names, loading and empty states using synthetic timetable data.
|
|
101
|
+
|
|
102
|
+
`variant="uk-rail"` gives the matrix a square black enclosure, mixed-case amber lettering, matrix column headings and horizontal display bands. Optional `heading` and `headingColumnSpan` replace the visual labels across the leading columns while preserving the individual accessible column headers. `footerLabel` and `clockLabel` add a matrix footer; the consumer owns clock formatting and updates. No live clock or pagination is inferred. `DotMatrixRow.note` adds a detail line and an accessible description on the selection button. Details consume one display slot and stay with their departure: if only one slot remains, the next departure with a detail waits until there is room for both. A one-line board still shows its first departure, retaining its full note for assistive technology.
|
|
103
|
+
|
|
104
|
+
## Rail, bus and airport hero cards
|
|
105
|
+
|
|
106
|
+
Hero cards have separate transport-specific APIs and visual identities, with shared display components underneath:
|
|
107
|
+
|
|
108
|
+
| Card | Identity | Departure fields | Default display |
|
|
109
|
+
| --- | --- | --- | --- |
|
|
110
|
+
| `RailStationHeroCard` | Station name, optional code and locality | Scheduled time, destination, platform, expected time/status, via/service note | UK rail matrix |
|
|
111
|
+
| `BusStopHeroCard` | Stop name, optional stop code and locality | Route, destination, due estimate, optional via | Bus dot matrix |
|
|
112
|
+
| `AirportHeroCard` | IATA code, airport name and city | Flight, time, destination/origin, gate, remarks, direction tabs | Split flap |
|
|
113
|
+
|
|
114
|
+
Import the new cards from `@motionstudies/web/components/RailStationHeroCard` or `@motionstudies/web/components/BusStopHeroCard`, plus `@motionstudies/web/transport-hero-cards.css` (which includes both board styles). Airport imports and study-window behavior remain as documented above.
|
|
115
|
+
|
|
116
|
+
```tsx
|
|
117
|
+
<RailStationHeroCard
|
|
118
|
+
station={{ name: 'Bristol Temple Meads', code: 'BRI', locality: 'Bristol' }}
|
|
119
|
+
departures={[{
|
|
120
|
+
id: 'train-1', time: '17:15', destination: 'Portsmouth Harbour',
|
|
121
|
+
platform: '9', expected: '17:22', via: 'Eastleigh',
|
|
122
|
+
}]}
|
|
123
|
+
lineCount="auto"
|
|
124
|
+
boardHeight={400}
|
|
125
|
+
clockLabel="16:49:26"
|
|
126
|
+
footerLabel="Study timetable"
|
|
127
|
+
note="Synthetic timetable. Example times, not a live service."
|
|
128
|
+
/>
|
|
129
|
+
|
|
130
|
+
<BusStopHeroCard
|
|
131
|
+
stop={{ name: 'Anchor Road', code: 'A1', locality: 'Bristol' }}
|
|
132
|
+
departures={[{ id: 'bus-1', route: '71', destination: 'City Centre', due: '2 min' }]}
|
|
133
|
+
lineCount={6}
|
|
134
|
+
note="Synthetic timetable. Example estimates, not a live service."
|
|
135
|
+
/>
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Rail and bus consumers supply already ordered and formatted departures, including their own filtering, timezones and freshness. Missing times, platforms and statuses stay unknown rather than becoming “On time”. Both cards accept `presentation` (`'uk-rail'`, `'dot-matrix'`, `'split-flap'`), `lineCount`, `boardHeight` in pixels, `minRowHeight`, `loading`, `error`, `onRetry`, localized `labels`, and controlled `onSelectDeparture`/`selectedDepartureId`. A required `note` explains the source. The matrix fits its container; the split-flap alternative contains scrolling and places any service details in an Information column. Fixed line counts represent physical matrix lines; in split-flap mode they limit departure rows. The **Transport heroes** lab compares all three cards, switches the new cards' presentations, and exercises details, row fitting, updates, long names and failure states.
|
|
139
|
+
|
|
140
|
+
The rail card additionally accepts `presentation="sbb"`: a blue-and-white typographic departure board with service badges, scheduled time, destination/via information and prominent track numbers. This layout follows the information hierarchy in [SBB's general display guide](https://www.sbb.ch/en/travel-information/stations/services-station/station-customer-information/general-display-board.html). It uses ordinary text, including accented and non-Latin names. `RailDeparture.service` supplies a train label such as `IC 1`; optional `serviceCategory` (`'intercity'`, `'international'`, `'regional'`, `'suburban'`) selects the badge treatment. `platformSector` supplies a separate sector label when known. Expected times or disruption messages appear below the scheduled time; absence of a message does not manufacture an “On time” assertion.
|
|
141
|
+
|
|
142
|
+
For this layout, `lineCount` counts departures with their inline detail, and `"auto"` fits rows using a default target height of 64px. Dense fixed counts reduce type size; long values remain in the accessible text and hover titles. Labels stay consumer-owned, including the added `service` column label. The lab's **SBB departure board** option selects a synthetic Zürich HB example with German, French, Italian and English labels. Both new board styles are included in `transport-hero-cards.css`.
|
|
143
|
+
|
|
144
|
+
At compact widths the SBB layout stacks the service badge under the time, preserving destination space and the separate track column. Rail and bus hero padding follows the card width rather than the viewport. The lab includes a 240–980px width slider and Compact/Mobile/Panel/Wide presets, plus a 180–640px board-height control. The size regression suite covers seven card widths, four board heights, fixed and automatic line counts, long destinations, selection during updates, mobile viewports and loading/error states in Chromium and WebKit. Prefer `lineCount="auto"` for small panels; high fixed line counts deliberately trade text size for density.
|
|
145
|
+
|
|
146
|
+
```tsx
|
|
147
|
+
<RailStationHeroCard
|
|
148
|
+
presentation="sbb"
|
|
149
|
+
station={{ name: 'Zürich HB' }}
|
|
150
|
+
labels={{ station: 'Bahnhof', departures: 'Abfahrt', service: 'Zug', time: 'Zeit', destination: 'Nach', platform: 'Gleis' }}
|
|
151
|
+
departures={[{ id: 'example-1', service: 'IC 1', serviceCategory: 'intercity',
|
|
152
|
+
time: '09:02', destination: 'Genève-Aéroport', via: 'Bern · Lausanne', platform: '32', platformSector: 'ABCD' }]}
|
|
153
|
+
lineCount="auto"
|
|
154
|
+
boardHeight={430}
|
|
155
|
+
note="Synthetic timetable · Example data."
|
|
156
|
+
/>
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
The shared board also accepts `loading`, a localized `loadingMessage`, and `loadingRows` (default five). While loading, its decorative rows cycle through staggered letters and digits; they are hidden from assistive technology and cannot be selected. A single status message announces loading. When data arrives, characters flip through a short sequence and settle into their actual values; later changes animate only the changed characters. These CSS animations have no JavaScript timers and stop looping when loading ends or the board is removed. Reduced-motion users get static blank loading flaps and immediate final text. `AirportHeroCard` uses this shared loading treatment automatically. Use **Reload board** in the Airports lab to preview the complete loading-to-ready transition.
|
|
160
|
+
|
|
161
|
+
Empty messages also appear on the flaps, in the widest column (the destination/origin column in airport cards), with the other columns blank. Longer localized messages wrap across display rows instead of being truncated. They settle with the same animation as flight details, and one hidden status announces the complete message to assistive technology. This also applies when the study clock moves into a window with no movements.
|
|
162
|
+
|
|
163
|
+
Rail and other transport consumers can share the same time filtering through `movementBoardWindow(study, horizon)` and `movementsForBoard(entries, window, maxRows)` from `@motionstudies/core/domain/movement-board`. Format the returned numeric times when mapping them into `SplitFlapBoard` cells. The lab's rail board follows the same study clock and horizon as its airport card.
|
|
164
|
+
|
|
165
|
+
`@motionstudies/data/air-endpoints` provides offline `enrichAirEndpoints` for existing air manifests, chunks and opening snapshots. Supply cached same-date global ADSB.lol heatmaps, an OurAirports CSV and the service date's local UTC offset. It associates only unambiguous low-altitude endpoints near a reference airport; cruise-only traces and uncertain routes stay unknown. Optional `AirEndpoint` origin/destination fields carry airport identity, observed boundary time and `observed-endpoint` evidence. `airportBoardMovements` maps full manifest entries to board rows without confusing playback chunk boundaries with flight endpoints. Input hashes and source/licence attribution are recorded in fixture metadata. These fields describe inferred observations, never flight schedules, gates or live status.
|
|
166
|
+
|
|
167
|
+
## Optional live airport feed
|
|
168
|
+
|
|
169
|
+
`AirportBoard` from `@motionstudies/web/components/AirportBoard` adds Study/Now controls around an existing `AirportHeroCard` configuration. Pass `studyCard` with the usual card props and `live={{ baseUrl, edition, airport }}` for the shared service. Import `airport-hero-card.css`. `labels` localizes the wrapper's control and availability messages. The lower-level `useAirportFeed` hook and core `domain/live-airport` contract are also public exports.
|
|
170
|
+
|
|
171
|
+
Live timestamps are Unix seconds and use the airport's timezone for display, independently of recorded service time. The wrapper does not pass live flight IDs to the recorded scene's selection callback. The recorded card remains mounted while hidden; the edition still owns playback and can pause its study when appropriate. Only Now mode fetches flight boards; checking capabilities does not query the paid provider. Stale results carry their retrieval time and disappear when expired. Source data never falls back to synthetic or recorded flights under a live label.
|
|
172
|
+
|
|
173
|
+
See [service architecture and operations](../docs/LIVE-AIRPORTS.md). The Worker is deployed separately; npm publication and edition adoption remain explicit release steps.
|
|
174
|
+
|
|
175
|
+
## Shared recorded air compilation
|
|
176
|
+
|
|
177
|
+
`@motionstudies/data/adsb-heatmap` consolidates the offline heatmap pipeline previously copied between editions. `ingestAdsbHeatmaps` reads cached gzip slices, decodes observations, filters transport-scale tracks, splits flights, and writes either an opening snapshot or an indexed day with overlapping chunks. Source hashes, chunk hashes and ODbL attribution accompany the output. It makes no network requests.
|
|
178
|
+
|
|
179
|
+
The same `decodeAdsbHeatmap` now powers `enrichAirEndpoints`; endpoint inference retains full coordinate precision, while playback compilation retains the existing five-decimal coordinates. `transportAirTracks` and `chunkAirSnapshot` are also available for consumers that assemble their own pipeline. All functions have public TypeScript declarations and work in the packed Node package.
|
|
180
|
+
|
|
181
|
+
Editions supply geographic bounds, service date, explicit UTC offset, optional timezone, input files and output paths. Flight IDs and chunk overlap retain the existing contracts. The default splits known callsign changes and gaps over 30 minutes. Set `splitTracks: false` only when reproducing a legacy opening snapshot with one ID per aircraft. See [adoption and compatibility](../docs/AIR-DATA.md).
|
|
182
|
+
|
|
183
|
+
## Shared edition controllers and performance
|
|
184
|
+
|
|
185
|
+
`positionForTrain` now indexes chronological stop times with binary search and retains sequential behavior for unordered observations. Stop arrays are immutable: replace the array when a timetable changes. Arrival/departure boundaries, dwell, cancellation and backward seeking retain the existing contract.
|
|
186
|
+
|
|
187
|
+
`countableVehicleTrains(network, stations, selection)` and `createActiveTimetableVehicleCounter(trains, options)` from `@motionstudies/core/domain/vehicle-counts` separate station/route/category membership from clock updates. Build the selector and counter with `useMemo` when data or selection changes, then call the counter at the displayed time. It includes both interval endpoints and excludes cancellations and inverted intervals. Missing stations yield no matches. By default it counts timetable intervals even if a journey lacks enough stops to position; `{ requirePositionable: true }` excludes journeys with fewer than two stops. This distinction is explicit so an edition can retain its established metric.
|
|
188
|
+
|
|
189
|
+
The renderer now shares active GPU upload ranges, paused frame reuse, label and trail frame budgets, cached text comparators and batched hub lines. Custom layers can import the low-level helpers from `@motionstudies/three/render-performance`. Recreate frame trackers when their geometry/data/selection inputs change; `batchHubLines` takes ownership of two-vertex source line resources. Edition-specific worker transfer, picking and cartographic adapters remain consumer-owned.
|
|
190
|
+
|
|
191
|
+
`useJsonAsset<T>(url, enabled, parse?, optional?)` from `@motionstudies/web/use-json-asset` loads a single asset lazily and exposes `data`, `loading`, `error`, `unavailable`, and `retry`. Keep the parser stable and perform edition-specific schema/source compatibility checks there. Successful data remains cached while disabled; consumers decide whether to display it. Changing the URL or parser immediately discards prior-source state, and disabling/unmounting cancels requests. `retry()` discards cached state and requests again when enabled. Optional HTTP 404 responses are unavailable; other failures are errors. No source fallback or freshness policy is inferred.
|
|
192
|
+
|
|
193
|
+
`useTransitionValue(target, { durationMs, easing, steps })` from `@motionstudies/web/use-transition-value` animates a numeric value, returning `value` and `transitioning`. It reverses from the current frame, cancels on teardown, and settles immediately when reduced motion becomes active. `smoothTransition` is the default easing; `cosineTransition` and stepped progress support existing edition rhythms. Keep custom easing functions stable. Camera actions and lazy layout loading stay in the edition.
|
|
194
|
+
|
|
195
|
+
Edition chunk scripts can call `runNetworkChunkCli()` from `@motionstudies/data/network-chunk-cli`. It accepts the existing `--input`, `--manifest`, `--opening`, `--chunk-hours`, `--opening-start`, `--opening-end`, and `--focus` arguments. Source acquisition, provenance, output paths and command invocation remain edition-owned.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
import type { AirSnapshot, AirTrack } from '@motionstudies/core/domain/air'
|
|
2
|
+
import type { AirDayManifest, AirDayChunk, AirDayChunkDescriptor } from '@motionstudies/core/domain/air-day'
|
|
3
|
+
|
|
4
|
+
export interface HeatmapWindow {
|
|
5
|
+
serviceDate: string
|
|
6
|
+
/** Explicit offset for this recording; the caller owns DST and source date selection. */
|
|
7
|
+
utcOffsetHours: number
|
|
8
|
+
windowStart: number
|
|
9
|
+
windowEnd: number
|
|
10
|
+
bounds?: readonly [west: number, south: number, east: number, north: number]
|
|
11
|
+
/** Numeric 24-bit ICAO addresses; omit to decode all aircraft. */
|
|
12
|
+
addresses?: ReadonlySet<number>
|
|
13
|
+
}
|
|
14
|
+
export type HeatmapSample = [time: number, longitude: number, latitude: number, altitudeFeet: number, speedKnots: number, callsign: string | undefined]
|
|
15
|
+
export interface HeatmapRecord {
|
|
16
|
+
id: string
|
|
17
|
+
callsign: string
|
|
18
|
+
samples: readonly HeatmapSample[]
|
|
19
|
+
}
|
|
20
|
+
export interface HeatmapSource {
|
|
21
|
+
name: string
|
|
22
|
+
sha256: string
|
|
23
|
+
sourceUrl: string
|
|
24
|
+
}
|
|
25
|
+
export type HeatmapMetadata = AirSnapshot['metadata'] & {
|
|
26
|
+
timezone?: string
|
|
27
|
+
utcOffsetHours: number
|
|
28
|
+
sourceFiles: readonly HeatmapSource[]
|
|
29
|
+
}
|
|
30
|
+
export type HeatmapSnapshot = Omit<AirSnapshot, 'metadata'> & { readonly metadata: HeatmapMetadata }
|
|
31
|
+
export type AirChunkIntegrity = AirDayChunkDescriptor & { readonly bytes: number; readonly sha256: string }
|
|
32
|
+
export type ChunkedAirManifest = Omit<AirDayManifest, 'chunks'> & { readonly chunks: readonly AirChunkIntegrity[] }
|
|
33
|
+
export type HeatmapManifest = Omit<ChunkedAirManifest, 'metadata'> & { readonly metadata: HeatmapMetadata }
|
|
34
|
+
|
|
35
|
+
/** Uncompressed binary input; returns the number of accepted observations. Does not round coordinates. An absent callsign is undefined; an explicitly cleared one is empty. */
|
|
36
|
+
export function decodeAdsbHeatmap(bytes: Uint8Array, options: HeatmapWindow, onSample: (address: number, sample: HeatmapSample) => void): number
|
|
37
|
+
/** Does not mutate the input records. Legacy unsplit snapshots can opt out of flight segmentation. */
|
|
38
|
+
export function transportAirTracks(records: Iterable<HeatmapRecord>, options?: { splitTracks?: boolean }): AirTrack[]
|
|
39
|
+
export function chunkAirSnapshot(snapshot: AirSnapshot, options: {
|
|
40
|
+
chunkSeconds: number
|
|
41
|
+
stem: string
|
|
42
|
+
overlapBefore?: number
|
|
43
|
+
overlapAfter?: number
|
|
44
|
+
}): { manifest: ChunkedAirManifest; chunks: { descriptor: AirChunkIntegrity; payload: AirDayChunk }[] }
|
|
45
|
+
export interface HeatmapIngestOptions extends HeatmapWindow {
|
|
46
|
+
inputs: readonly string[]
|
|
47
|
+
output: string
|
|
48
|
+
bounds: readonly [west: number, south: number, east: number, north: number]
|
|
49
|
+
timezone?: string
|
|
50
|
+
chunkHours?: number
|
|
51
|
+
/** Defaults true; use false only to preserve a legacy single-track-per-aircraft snapshot. */
|
|
52
|
+
splitTracks?: boolean
|
|
53
|
+
}
|
|
54
|
+
export function ingestAdsbHeatmaps(options: HeatmapIngestOptions & { chunkHours: number }): Promise<HeatmapManifest>
|
|
55
|
+
export function ingestAdsbHeatmaps(options: HeatmapIngestOptions & { chunkHours?: undefined }): Promise<HeatmapSnapshot>
|
|
56
|
+
export function ingestAdsbHeatmaps(options: HeatmapIngestOptions): Promise<HeatmapSnapshot | HeatmapManifest>
|
package/adsb-heatmap.mjs
ADDED
|
@@ -0,0 +1,223 @@
|
|
|
1
|
+
import { createHash } from 'node:crypto'
|
|
2
|
+
import { readFile, writeFile } from 'node:fs/promises'
|
|
3
|
+
import { basename, dirname, join } from 'node:path'
|
|
4
|
+
import { gunzipSync } from 'node:zlib'
|
|
5
|
+
|
|
6
|
+
const MAGIC = 0x0e7f7c9d
|
|
7
|
+
const CALLSIGN_MARKER = 0x40000000
|
|
8
|
+
const NON_ICAO_MARKER = 0x01000000
|
|
9
|
+
|
|
10
|
+
function serviceDayStart(options) {
|
|
11
|
+
const { serviceDate, utcOffsetHours, windowStart, windowEnd } = options
|
|
12
|
+
const midnight = Date.parse(`${serviceDate}T00:00:00Z`)
|
|
13
|
+
if (!/^\d{4}-\d{2}-\d{2}$/.test(serviceDate) || !Number.isFinite(midnight) || new Date(midnight).toISOString().slice(0, 10) !== serviceDate) {
|
|
14
|
+
throw new Error('A valid serviceDate (YYYY-MM-DD) is required')
|
|
15
|
+
}
|
|
16
|
+
if (!Number.isFinite(utcOffsetHours) || Math.abs(utcOffsetHours) > 14) throw new Error('A valid local UTC offset is required')
|
|
17
|
+
if (!Number.isFinite(windowStart) || !Number.isFinite(windowEnd) || windowStart >= windowEnd) throw new Error('A finite, increasing study window is required')
|
|
18
|
+
if (options.bounds) {
|
|
19
|
+
const [west, south, east, north] = options.bounds
|
|
20
|
+
if (options.bounds.length !== 4 || !options.bounds.every(Number.isFinite) || west < -180 || east > 180 || south < -90 || north > 90 || west >= east || south >= north) {
|
|
21
|
+
throw new Error('bounds must be [west, south, east, north] in geographic degrees')
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
return midnight - utcOffsetHours * 3600000
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
/** Decode an uncompressed readsb heatmap slice. Times remain service-local seconds, including after midnight. */
|
|
28
|
+
export function decodeAdsbHeatmap(bytes, options, onSample) {
|
|
29
|
+
const dayStart = serviceDayStart(options)
|
|
30
|
+
const buffer = Buffer.isBuffer(bytes) ? bytes : Buffer.from(bytes)
|
|
31
|
+
if (buffer.length % 16 !== 0) throw new Error('Truncated heatmap record')
|
|
32
|
+
const [west, south, east, north] = options.bounds ?? [-180, -90, 180, 90]
|
|
33
|
+
const callsigns = new Map()
|
|
34
|
+
let time
|
|
35
|
+
let count = 0
|
|
36
|
+
for (let offset = 0; offset < buffer.length; offset += 16) {
|
|
37
|
+
const address = buffer.readUInt32LE(offset), encodedLatitude = buffer.readInt32LE(offset + 4)
|
|
38
|
+
if (address === MAGIC) {
|
|
39
|
+
const milliseconds = Number((BigInt(encodedLatitude) << 32n) | BigInt(buffer.readUInt32LE(offset + 8)))
|
|
40
|
+
time = Math.round((milliseconds - dayStart) / 1000)
|
|
41
|
+
continue
|
|
42
|
+
}
|
|
43
|
+
const id = address & 0xffffff
|
|
44
|
+
if ((address & NON_ICAO_MARKER) !== 0 || (options.addresses && !options.addresses.has(id))) continue
|
|
45
|
+
// Negative latitude also sets this bit in two's complement; it is still a position.
|
|
46
|
+
if (encodedLatitude >= 0 && (encodedLatitude & CALLSIGN_MARKER) !== 0) {
|
|
47
|
+
callsigns.set(id, buffer.subarray(offset + 8, offset + 16).toString('ascii').replaceAll('\0', '').trim().toUpperCase())
|
|
48
|
+
continue
|
|
49
|
+
}
|
|
50
|
+
if (time === undefined || time < options.windowStart || time > options.windowEnd) continue
|
|
51
|
+
const latitude = encodedLatitude / 1e6, longitude = buffer.readInt32LE(offset + 8) / 1e6
|
|
52
|
+
if (longitude < west || longitude > east || latitude < south || latitude > north) continue
|
|
53
|
+
const altitude = buffer.readInt16LE(offset + 12), speed = buffer.readInt16LE(offset + 14)
|
|
54
|
+
if (altitude < 0 || speed < 0) continue
|
|
55
|
+
onSample(id, [time, longitude, latitude, altitude * 25, speed / 10, callsigns.get(id)])
|
|
56
|
+
count++
|
|
57
|
+
}
|
|
58
|
+
return count
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
function round(value, digits) {
|
|
62
|
+
const scale = 10 ** digits
|
|
63
|
+
return Math.round(value * scale) / scale
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
function transportScaleTrack(record) {
|
|
67
|
+
const samples = [...record.samples]
|
|
68
|
+
.sort((first, second) => first[0] - second[0])
|
|
69
|
+
.filter((sample, index, all) => index === 0 || sample[0] !== all[index - 1][0])
|
|
70
|
+
.map((sample) => sample.slice(0, 5))
|
|
71
|
+
if (samples.length < 4) return undefined
|
|
72
|
+
const maximumSpeed = samples.reduce((maximum, sample) => Math.max(maximum, sample[4]), 0)
|
|
73
|
+
const maximumAltitude = samples.reduce((maximum, sample) => Math.max(maximum, sample[3]), 0)
|
|
74
|
+
const airlineCallsign = /^[A-Z]{2,4}\d[A-Z0-9]*$/.test(record.callsign)
|
|
75
|
+
if (maximumSpeed < 120 || maximumAltitude < 1_500) return undefined
|
|
76
|
+
if (!airlineCallsign && (maximumSpeed < 250 || maximumAltitude < 10_000)) return undefined
|
|
77
|
+
return {
|
|
78
|
+
id: record.id,
|
|
79
|
+
callsign: record.callsign || record.id.toUpperCase(),
|
|
80
|
+
start: samples[0][0],
|
|
81
|
+
end: samples.at(-1)[0],
|
|
82
|
+
samples,
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
function transportScaleTracks(record) {
|
|
87
|
+
const ordered = [...record.samples]
|
|
88
|
+
.sort((first, second) => first[0] - second[0])
|
|
89
|
+
.filter((sample, index, all) => index === 0 || sample[0] !== all[index - 1][0])
|
|
90
|
+
const segments = []
|
|
91
|
+
let current = []
|
|
92
|
+
let currentCallsign = ''
|
|
93
|
+
|
|
94
|
+
for (const sample of ordered) {
|
|
95
|
+
const callsign = sample[5]
|
|
96
|
+
const previous = current.at(-1)
|
|
97
|
+
const callsignChanged = callsign && currentCallsign && callsign !== currentCallsign
|
|
98
|
+
const longGap = previous && sample[0] - previous[0] > 30 * 60
|
|
99
|
+
if (current.length && (callsignChanged || longGap)) {
|
|
100
|
+
segments.push({ callsign: currentCallsign, samples: current })
|
|
101
|
+
current = []
|
|
102
|
+
currentCallsign = ''
|
|
103
|
+
}
|
|
104
|
+
if (callsign) currentCallsign = callsign
|
|
105
|
+
current.push(sample)
|
|
106
|
+
}
|
|
107
|
+
if (current.length) segments.push({ callsign: currentCallsign, samples: current })
|
|
108
|
+
|
|
109
|
+
return segments
|
|
110
|
+
.map((segment) => {
|
|
111
|
+
const base = transportScaleTrack({
|
|
112
|
+
id: record.id,
|
|
113
|
+
callsign: segment.callsign || record.callsign,
|
|
114
|
+
samples: segment.samples,
|
|
115
|
+
})
|
|
116
|
+
if (!base) return undefined
|
|
117
|
+
return {
|
|
118
|
+
...base,
|
|
119
|
+
id: `${record.id}-${base.start}`,
|
|
120
|
+
icaoAddress: record.id,
|
|
121
|
+
}
|
|
122
|
+
})
|
|
123
|
+
.filter(Boolean)
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
|
|
127
|
+
/** Stable ordering and IDs; split on a known callsign change or an observation gap over 30 minutes. */
|
|
128
|
+
export function transportAirTracks(records, { splitTracks = true } = {}) {
|
|
129
|
+
return [...records].flatMap(record => splitTracks
|
|
130
|
+
? transportScaleTracks(record)
|
|
131
|
+
: [transportScaleTrack(record)].filter(Boolean))
|
|
132
|
+
.sort((first, second) => first.id.localeCompare(second.id))
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/** Produce playback chunks with the same 180s trail and 45s interpolation overlap used by the editions. */
|
|
136
|
+
export function chunkAirSnapshot(snapshot, { chunkSeconds, stem, overlapBefore = 180, overlapAfter = 45 }) {
|
|
137
|
+
if (!Number.isInteger(chunkSeconds) || chunkSeconds <= 0) throw new Error('chunkSeconds must be a positive integer')
|
|
138
|
+
if (![overlapBefore, overlapAfter].every(value => Number.isFinite(value) && value >= 0)) throw new Error('Chunk overlap must be non-negative')
|
|
139
|
+
if (!/^[a-zA-Z0-9_-]+$/.test(stem)) throw new Error('A simple chunk filename stem is required')
|
|
140
|
+
const { windowStart: start, windowEnd: end } = snapshot.metadata
|
|
141
|
+
if (!Number.isFinite(start) || !Number.isFinite(end) || start >= end) throw new Error('A finite, increasing study window is required')
|
|
142
|
+
const chunks = []
|
|
143
|
+
for (let windowStart = start; windowStart < end; windowStart += chunkSeconds) {
|
|
144
|
+
const windowEnd = Math.min(end, windowStart + chunkSeconds)
|
|
145
|
+
const id = String(Math.floor(windowStart / chunkSeconds)).padStart(2, '0')
|
|
146
|
+
const tracks = snapshot.tracks.flatMap(track => {
|
|
147
|
+
const samples = track.samples.filter(sample => sample[0] >= windowStart - overlapBefore && sample[0] <= windowEnd + overlapAfter)
|
|
148
|
+
return samples.length < 2 ? [] : [{ ...track, start: samples[0][0], end: samples.at(-1)[0], samples }]
|
|
149
|
+
})
|
|
150
|
+
const payload = { windowStart, windowEnd, tracks }
|
|
151
|
+
const serialized = `${JSON.stringify(payload)}\n`
|
|
152
|
+
chunks.push({ descriptor: {
|
|
153
|
+
id, windowStart, windowEnd, path: `${stem}-${id}.json`, trackCount: tracks.length,
|
|
154
|
+
sampleCount: tracks.reduce((sum, track) => sum + track.samples.length, 0),
|
|
155
|
+
bytes: Buffer.byteLength(serialized), sha256: createHash('sha256').update(serialized).digest('hex'),
|
|
156
|
+
}, payload })
|
|
157
|
+
}
|
|
158
|
+
const aircraft = snapshot.tracks.map(({ samples: _samples, ...track }) => ({
|
|
159
|
+
...track, icaoAddress: track.icaoAddress ?? track.id,
|
|
160
|
+
chunkIds: chunks.filter(({ descriptor }) => track.end >= descriptor.windowStart && track.start <= descriptor.windowEnd)
|
|
161
|
+
.map(({ descriptor }) => descriptor.id),
|
|
162
|
+
}))
|
|
163
|
+
return { manifest: {
|
|
164
|
+
metadata: snapshot.metadata, bounds: snapshot.bounds, trackCount: snapshot.tracks.length,
|
|
165
|
+
sampleCount: snapshot.tracks.reduce((sum, track) => sum + track.samples.length, 0),
|
|
166
|
+
aircraft, chunks: chunks.map(chunk => chunk.descriptor),
|
|
167
|
+
}, chunks }
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
function sourceUrl(date) {
|
|
171
|
+
return `https://github.com/adsblol/globe_history_${date.slice(0, 4)}/releases/tag/v${date.replaceAll('-', '.')}-planes-readsb-prod-0`
|
|
172
|
+
}
|
|
173
|
+
|
|
174
|
+
/** Compile cached gzip heatmaps. The caller owns source selection, timezone and output location; no network calls. */
|
|
175
|
+
export async function ingestAdsbHeatmaps(options) {
|
|
176
|
+
serviceDayStart(options)
|
|
177
|
+
if (!options.bounds) throw new Error('Study bounds are required')
|
|
178
|
+
if (!options.inputs?.length) throw new Error('At least one cached heatmap input is required')
|
|
179
|
+
if (options.chunkHours !== undefined && (!Number.isInteger(options.chunkHours) || options.chunkHours < 1)) throw new Error('chunkHours must be a positive integer')
|
|
180
|
+
if (!options.output?.endsWith('.json')) throw new Error('An output .json path is required')
|
|
181
|
+
if (options.timezone) new Intl.DateTimeFormat('en', { timeZone: options.timezone })
|
|
182
|
+
const records = new Map(), sourceFiles = []
|
|
183
|
+
// Preserve caller ordering for deterministic first-observation wins when input slices overlap.
|
|
184
|
+
for (const input of options.inputs) {
|
|
185
|
+
const raw = await readFile(input), name = basename(input)
|
|
186
|
+
const date = /^\d{4}-\d{2}-\d{2}/.exec(name)?.[0] ?? options.serviceDate
|
|
187
|
+
sourceFiles.push({ name, sha256: createHash('sha256').update(raw).digest('hex'), sourceUrl: sourceUrl(date) })
|
|
188
|
+
decodeAdsbHeatmap(gunzipSync(raw), options, (address, sample) => {
|
|
189
|
+
const id = address.toString(16).padStart(6, '0')
|
|
190
|
+
const record = records.get(id) ?? { id, callsign: '', samples: [] }
|
|
191
|
+
record.callsign = sample[5] || record.callsign
|
|
192
|
+
record.samples.push([sample[0], round(sample[1], 5), round(sample[2], 5), sample[3], round(sample[4], 1), record.callsign])
|
|
193
|
+
records.set(id, record)
|
|
194
|
+
})
|
|
195
|
+
}
|
|
196
|
+
const tracks = transportAirTracks(records.values(), { splitTracks: options.splitTracks ?? true })
|
|
197
|
+
const [minLongitude, minLatitude, maxLongitude, maxLatitude] = options.bounds
|
|
198
|
+
const snapshot = {
|
|
199
|
+
metadata: {
|
|
200
|
+
publisher: 'ADSB.lol', ...(options.timezone ? { timezone: options.timezone } : {}),
|
|
201
|
+
utcOffsetHours: options.utcOffsetHours, sourceFiles,
|
|
202
|
+
serviceDate: options.serviceDate, windowStart: options.windowStart, windowEnd: options.windowEnd,
|
|
203
|
+
sourceUrl: sourceUrl(options.serviceDate), license: 'ODbL 1.0',
|
|
204
|
+
licenseUrl: 'https://opendatacommons.org/licenses/odbl/1-0/',
|
|
205
|
+
model: 'Observed ADS-B / MLAT heatmap positions replayed at 10-second resolution',
|
|
206
|
+
note: 'Ground, non-ICAO, low-speed and low-altitude signals are excluded; altitude is rendered with explicit vertical compression.',
|
|
207
|
+
sampleIntervalSeconds: 10,
|
|
208
|
+
},
|
|
209
|
+
bounds: { minLongitude, minLatitude, maxLongitude, maxLatitude }, tracks,
|
|
210
|
+
}
|
|
211
|
+
if (options.chunkHours !== undefined) {
|
|
212
|
+
const { manifest, chunks } = chunkAirSnapshot(snapshot, {
|
|
213
|
+
chunkSeconds: options.chunkHours * 3600, stem: basename(options.output, '.json').replace(/-manifest$/, ''),
|
|
214
|
+
})
|
|
215
|
+
for (const { descriptor, payload } of chunks) {
|
|
216
|
+
await writeFile(join(dirname(options.output), descriptor.path), `${JSON.stringify(payload)}\n`)
|
|
217
|
+
}
|
|
218
|
+
await writeFile(options.output, `${JSON.stringify(manifest)}\n`)
|
|
219
|
+
return manifest
|
|
220
|
+
}
|
|
221
|
+
await writeFile(options.output, `${JSON.stringify(snapshot)}\n`)
|
|
222
|
+
return snapshot
|
|
223
|
+
}
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
import type { AirEndpoint, AirSample } from '@motionstudies/core/domain/air'
|
|
2
|
+
export interface EndpointAirport { icao: string; iata: string; name: string; city: string; longitude: number; latitude: number; elevation: number }
|
|
3
|
+
export function parseAirports(csv: string): EndpointAirport[]
|
|
4
|
+
export function inferAirEndpoints(samples: readonly AirSample[], airports: readonly EndpointAirport[]): { origin?: AirEndpoint; destination?: AirEndpoint }
|
|
5
|
+
export function enrichAirEndpoints(options: { manifestPath: string; snapshotPaths?: string[]; heatmapDirectory: string; airportCsvPath: string; utcOffsetHours: number }): Promise<{ tracks: number; origins: number; destinations: number }>
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
import { readFile, readdir, writeFile } from 'node:fs/promises'
|
|
2
|
+
import { dirname, join } from 'node:path'
|
|
3
|
+
import { gunzipSync } from 'node:zlib'
|
|
4
|
+
import { createHash } from 'node:crypto'
|
|
5
|
+
import { parseCsvLine } from './gtfs.mjs'
|
|
6
|
+
import { decodeAdsbHeatmap } from './adsb-heatmap.mjs'
|
|
7
|
+
|
|
8
|
+
export function parseAirports(csv) {
|
|
9
|
+
const [header, ...lines] = csv.trim().split(/\r?\n/)
|
|
10
|
+
const keys = parseCsvLine(header)
|
|
11
|
+
return lines.map(line => Object.fromEntries(parseCsvLine(line).map((value, index) => [keys[index], value])))
|
|
12
|
+
.filter(row => ['large_airport', 'medium_airport'].includes(row.type) && row.ident && row.elevation_ft)
|
|
13
|
+
.map(row => ({ icao: row.ident, iata: row.iata_code, name: row.name, city: row.municipality,
|
|
14
|
+
longitude: Number(row.longitude_deg), latitude: Number(row.latitude_deg), elevation: Number(row.elevation_ft) }))
|
|
15
|
+
.filter(row => [row.longitude, row.latitude, row.elevation].every(Number.isFinite))
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
function distance(sample, airport) {
|
|
19
|
+
const longitude = ((sample[1] - airport.longitude + 540) % 360) - 180
|
|
20
|
+
return Math.hypot(longitude * 111.32 * Math.cos((sample[2] + airport.latitude) * Math.PI / 360), (sample[2] - airport.latitude) * 111.32)
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
function atAirport(sample, airport) {
|
|
24
|
+
return sample[3] >= airport.elevation - 500 && sample[3] <= airport.elevation + 1500 && sample[4] <= 250 && distance(sample, airport) <= 6
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
/** Conservative endpoint association; cruise-only traces and ambiguous neighbouring airports stay unknown. */
|
|
28
|
+
export function inferAirEndpoints(samples, airports) {
|
|
29
|
+
if (samples.length < 4) return {}
|
|
30
|
+
const maximumAltitude = Math.max(...samples.map(sample => sample[3]))
|
|
31
|
+
const endpoint = (arrival) => {
|
|
32
|
+
const ordered = arrival ? [...samples].reverse() : samples
|
|
33
|
+
const candidates = airports.filter(airport => atAirport(ordered[0], airport) && maximumAltitude >= airport.elevation + 3000)
|
|
34
|
+
if (candidates.length !== 1) return undefined
|
|
35
|
+
const airport = candidates[0]
|
|
36
|
+
let boundary = ordered[0]
|
|
37
|
+
for (const sample of ordered.slice(1)) {
|
|
38
|
+
if (!atAirport(sample, airport)) break
|
|
39
|
+
boundary = sample
|
|
40
|
+
}
|
|
41
|
+
return { icao: airport.icao, iata: airport.iata, name: airport.name, city: airport.city, time: boundary[0], evidence: 'observed-endpoint' }
|
|
42
|
+
}
|
|
43
|
+
const origin = endpoint(false), destination = endpoint(true)
|
|
44
|
+
if (origin && destination && (origin.icao === destination.icao || origin.time >= destination.time)) return {}
|
|
45
|
+
return { ...(origin ? { origin } : {}), ...(destination ? { destination } : {}) }
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
function segmentsFor(samples) {
|
|
49
|
+
const segments = []
|
|
50
|
+
let current = []
|
|
51
|
+
for (const sample of samples.sort((a, b) => a[0] - b[0])) {
|
|
52
|
+
const previous = current.at(-1)
|
|
53
|
+
if (previous && previous[0] === sample[0]) continue
|
|
54
|
+
if (previous && (sample[0] - previous[0] > 1800 || sample[5] !== previous[5])) {
|
|
55
|
+
segments.push(current)
|
|
56
|
+
current = []
|
|
57
|
+
}
|
|
58
|
+
current.push(sample)
|
|
59
|
+
}
|
|
60
|
+
if (current.length) segments.push(current)
|
|
61
|
+
return segments
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/** Enrich an existing study and its chunks using cached, same-date global heatmaps. No network requests. */
|
|
65
|
+
export async function enrichAirEndpoints({ manifestPath, snapshotPaths = [], heatmapDirectory, airportCsvPath, utcOffsetHours }) {
|
|
66
|
+
const readJson = async path => JSON.parse(await readFile(path, 'utf8'))
|
|
67
|
+
const manifest = await readJson(manifestPath)
|
|
68
|
+
const snapshots = await Promise.all(snapshotPaths.map(readJson))
|
|
69
|
+
const date = manifest.metadata.serviceDate
|
|
70
|
+
if (!/^\d{4}-\d{2}-\d{2}$/.test(date) || !Number.isFinite(utcOffsetHours)) throw new Error('A service date and UTC offset are required')
|
|
71
|
+
if (snapshots.some(snapshot => snapshot.metadata.serviceDate !== date)) throw new Error('Air snapshots must share the manifest service date')
|
|
72
|
+
const tracks = [...manifest.aircraft, ...snapshots.flatMap(snapshot => snapshot.tracks)]
|
|
73
|
+
const wanted = new Set(tracks.map(track => parseInt(track.icaoAddress ?? track.id.split('-')[0], 16)))
|
|
74
|
+
const records = new Map()
|
|
75
|
+
const files = (await readdir(heatmapDirectory)).filter(file => file.endsWith('.bin.ttf')).sort()
|
|
76
|
+
if (!files.length) throw new Error('No cached heatmap slices found')
|
|
77
|
+
const sources = []
|
|
78
|
+
for (const file of files) {
|
|
79
|
+
const raw = await readFile(join(heatmapDirectory, file))
|
|
80
|
+
const used = decodeAdsbHeatmap(gunzipSync(raw), {
|
|
81
|
+
serviceDate: date, utcOffsetHours, windowStart: 0, windowEnd: 86400, addresses: wanted,
|
|
82
|
+
}, (id, sample) => {
|
|
83
|
+
const samples = records.get(id) ?? []
|
|
84
|
+
sample[5] ??= samples.at(-1)?.[5] ?? ''
|
|
85
|
+
samples.push(sample)
|
|
86
|
+
records.set(id, samples)
|
|
87
|
+
})
|
|
88
|
+
if (used) sources.push({ file, sha256: createHash('sha256').update(raw).digest('hex') })
|
|
89
|
+
}
|
|
90
|
+
const csv = await readFile(airportCsvPath, 'utf8')
|
|
91
|
+
if (!sources.length) throw new Error('No same-date observations matched the study aircraft')
|
|
92
|
+
const airports = parseAirports(csv)
|
|
93
|
+
if (!airports.length) throw new Error('No usable airports in the reference CSV')
|
|
94
|
+
const segments = new Map([...records].map(([id, samples]) => [id, segmentsFor(samples).map(samples => ({
|
|
95
|
+
start: samples[0][0], end: samples.at(-1)[0], callsign: samples[0][5], ...inferAirEndpoints(samples, airports),
|
|
96
|
+
}))]))
|
|
97
|
+
const enrich = (track) => {
|
|
98
|
+
const candidates = (segments.get(parseInt(track.icaoAddress ?? track.id.split('-')[0], 16)) ?? [])
|
|
99
|
+
.filter(segment => segment.callsign && segment.callsign === track.callsign && segment.start <= track.start + 30 && segment.end >= track.end - 30)
|
|
100
|
+
const { origin: _origin, destination: _destination, ...rest } = track
|
|
101
|
+
if (candidates.length !== 1) return rest
|
|
102
|
+
const { origin, destination } = candidates[0]
|
|
103
|
+
return { ...rest, ...(origin ? { origin } : {}), ...(destination ? { destination } : {}) }
|
|
104
|
+
}
|
|
105
|
+
const metadata = {
|
|
106
|
+
serviceDate: date, method: 'observed-endpoint', note: 'Origin/destination inferred from full-trace endpoints within 6 km and 1500 ft above an airport; observed times, not schedules or confirmed flight plans.',
|
|
107
|
+
airportSource: 'https://ourairports.com/data/', airportLicense: 'Public Domain', airportSha256: createHash('sha256').update(csv).digest('hex'),
|
|
108
|
+
traceSource: manifest.metadata.sourceUrl, traceLicense: 'ODbL 1.0', utcOffsetHours, sourceFiles: sources,
|
|
109
|
+
}
|
|
110
|
+
const aircraft = manifest.aircraft.map(enrich)
|
|
111
|
+
const byId = new Map(aircraft.map(track => [track.id, track]))
|
|
112
|
+
const writes = []
|
|
113
|
+
for (const descriptor of manifest.chunks) {
|
|
114
|
+
const path = join(dirname(manifestPath), descriptor.path)
|
|
115
|
+
const chunk = await readJson(path)
|
|
116
|
+
chunk.tracks = chunk.tracks.map(track => {
|
|
117
|
+
const { origin: _origin, destination: _destination, ...rest } = track
|
|
118
|
+
const full = byId.get(track.id)
|
|
119
|
+
return { ...rest, ...(full?.origin ? { origin: full.origin } : {}), ...(full?.destination ? { destination: full.destination } : {}) }
|
|
120
|
+
})
|
|
121
|
+
const bytes = Buffer.from(`${JSON.stringify(chunk)}\n`)
|
|
122
|
+
if ('bytes' in descriptor) descriptor.bytes = bytes.length
|
|
123
|
+
if ('sha256' in descriptor) descriptor.sha256 = createHash('sha256').update(bytes).digest('hex')
|
|
124
|
+
writes.push([path, chunk])
|
|
125
|
+
}
|
|
126
|
+
writes.push([manifestPath, { ...manifest, metadata: { ...manifest.metadata, routeEnrichment: metadata }, aircraft }])
|
|
127
|
+
snapshots.forEach((snapshot, index) => writes.push([snapshotPaths[index], { ...snapshot, metadata: { ...snapshot.metadata, routeEnrichment: metadata }, tracks: snapshot.tracks.map(enrich) }]))
|
|
128
|
+
for (const [path, value] of writes) await writeFile(path, `${JSON.stringify(value)}\n`)
|
|
129
|
+
return { tracks: aircraft.length, origins: aircraft.filter(track => track.origin).length, destinations: aircraft.filter(track => track.destination).length }
|
|
130
|
+
}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
export function runNetworkChunkCli(argv?: readonly string[]): Promise<void>
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
import { chunkNetworkSnapshot, extractNetworkWindow } from './network-chunks.mjs'
|
|
2
|
+
|
|
3
|
+
import { createWriteStream } from 'node:fs'
|
|
4
|
+
import { mkdir, readFile } from 'node:fs/promises'
|
|
5
|
+
import { basename, dirname, extname, join, resolve } from 'node:path'
|
|
6
|
+
|
|
7
|
+
function argument(argv, name, fallback) {
|
|
8
|
+
const index = argv.indexOf(`--${name}`)
|
|
9
|
+
return index === -1 ? fallback : argv[index + 1]
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
function parseClock(value) {
|
|
13
|
+
const [hours, minutes = 0] = value.split(':').map(Number)
|
|
14
|
+
if (!Number.isFinite(hours) || !Number.isFinite(minutes)) {
|
|
15
|
+
throw new Error(`Invalid clock value ${value}`)
|
|
16
|
+
}
|
|
17
|
+
return hours * 3600 + minutes * 60
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
async function writeJson(filePath, value) {
|
|
21
|
+
await mkdir(dirname(filePath), { recursive: true })
|
|
22
|
+
await new Promise((resolvePromise, rejectPromise) => {
|
|
23
|
+
const output = createWriteStream(filePath)
|
|
24
|
+
output.once('error', rejectPromise)
|
|
25
|
+
output.once('finish', resolvePromise)
|
|
26
|
+
output.end(JSON.stringify(value))
|
|
27
|
+
})
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
export async function runNetworkChunkCli(argv = process.argv.slice(2)) {
|
|
31
|
+
const input = argument(argv, 'input')
|
|
32
|
+
const manifestOutput = argument(argv, 'manifest')
|
|
33
|
+
const openingOutput = argument(argv, 'opening')
|
|
34
|
+
if (!input || !manifestOutput || !openingOutput) {
|
|
35
|
+
throw new Error(
|
|
36
|
+
'Usage: node scripts/chunk-network-snapshot.mjs --input day.json --manifest day-manifest.json --opening morning.json [--chunk-hours 2] [--opening-start 06:45] [--opening-end 08:45] [--focus 07:45]',
|
|
37
|
+
)
|
|
38
|
+
}
|
|
39
|
+
const snapshot = JSON.parse(await readFile(resolve(input), 'utf8'))
|
|
40
|
+
const manifestPath = resolve(manifestOutput)
|
|
41
|
+
const outputDirectory = dirname(manifestPath)
|
|
42
|
+
const outputStem = basename(manifestPath, extname(manifestPath)).replace(/-manifest$/, '')
|
|
43
|
+
const chunkDirectoryName = `${outputStem}-chunks`
|
|
44
|
+
const { manifest, chunks } = chunkNetworkSnapshot(
|
|
45
|
+
snapshot,
|
|
46
|
+
Number(argument(argv, 'chunk-hours', '2')) * 3600,
|
|
47
|
+
chunkDirectoryName,
|
|
48
|
+
)
|
|
49
|
+
const opening = extractNetworkWindow(
|
|
50
|
+
snapshot,
|
|
51
|
+
parseClock(argument(argv, 'opening-start', '06:45')),
|
|
52
|
+
parseClock(argument(argv, 'opening-end', '08:45')),
|
|
53
|
+
parseClock(argument(argv, 'focus', '07:45')),
|
|
54
|
+
)
|
|
55
|
+
await Promise.all([
|
|
56
|
+
writeJson(manifestPath, manifest),
|
|
57
|
+
writeJson(resolve(openingOutput), opening),
|
|
58
|
+
...chunks.map(({ descriptor, payload }) =>
|
|
59
|
+
writeJson(join(outputDirectory, descriptor.path), payload),
|
|
60
|
+
),
|
|
61
|
+
])
|
|
62
|
+
console.log(
|
|
63
|
+
`Wrote ${manifestOutput}, ${chunks.length} progressive chunks and ${opening.trains.length} opening journeys.`,
|
|
64
|
+
)
|
|
65
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@motionstudies/data",
|
|
3
|
-
"version": "0.1.0-alpha.
|
|
3
|
+
"version": "0.1.0-alpha.10",
|
|
4
4
|
"private": false,
|
|
5
5
|
"type": "module",
|
|
6
6
|
"description": "Offline data preparation primitives for Motion Studies.",
|
|
@@ -20,9 +20,17 @@
|
|
|
20
20
|
"tag": "next"
|
|
21
21
|
},
|
|
22
22
|
"engines": {
|
|
23
|
-
"node": ">=
|
|
23
|
+
"node": ">=24.0.0"
|
|
24
24
|
},
|
|
25
25
|
"exports": {
|
|
26
|
+
"./adsb-heatmap": {
|
|
27
|
+
"types": "./adsb-heatmap.d.mts",
|
|
28
|
+
"import": "./adsb-heatmap.mjs"
|
|
29
|
+
},
|
|
30
|
+
"./air-endpoints": {
|
|
31
|
+
"types": "./air-endpoints.d.mts",
|
|
32
|
+
"import": "./air-endpoints.mjs"
|
|
33
|
+
},
|
|
26
34
|
"./gtfs": {
|
|
27
35
|
"types": "./gtfs.d.mts",
|
|
28
36
|
"import": "./gtfs.mjs"
|
|
@@ -38,6 +46,10 @@
|
|
|
38
46
|
"./merge-network": {
|
|
39
47
|
"types": "./merge-network.d.mts",
|
|
40
48
|
"import": "./merge-network.mjs"
|
|
49
|
+
},
|
|
50
|
+
"./network-chunk-cli": {
|
|
51
|
+
"types": "./network-chunk-cli.d.mts",
|
|
52
|
+
"import": "./network-chunk-cli.mjs"
|
|
41
53
|
}
|
|
42
54
|
},
|
|
43
55
|
"files": [
|
|
@@ -51,6 +63,6 @@
|
|
|
51
63
|
],
|
|
52
64
|
"sideEffects": false,
|
|
53
65
|
"dependencies": {
|
|
54
|
-
"@motionstudies/core": "0.1.0-alpha.
|
|
66
|
+
"@motionstudies/core": "0.1.0-alpha.10"
|
|
55
67
|
}
|
|
56
68
|
}
|