@motionstudies/data 0.1.0-alpha.14 → 0.1.0-alpha.16

package/README.md

@@ -229,3 +229,27 @@ For timetable-backed vehicle selections, `NetworkVehicleHeroCard` from `@motions
 ### Railway preparation
 
 `@motionstudies/data/rail-wtt`, `rail-journeys`, `rail-public-calls`, `rail-routing` and `rail-geometry` expose typed offline railway preparation. XLSX reading, dated source-column audits, UID/date joins, public-call anchors and connected OSM paths preserve source evidence. Editions supply regional identities, calendar exceptions and clipping policy. Node 24+ and `unzip` are required; no Python runtime is shipped. See `docs/SHARED-RAIL-READERS.md` in the source repository for calendar limits and parity evidence.
+
+`mountMotionStudy({ id, theme }, application)` requires presentation identity only; a full `MotionStudyEdition` remains accepted. Applications own their data catalogue and loading lifecycle.
+
+For independent airport infrastructure, `mapStyle.airports.visible` hides the mounted group without rebuilding GPU resources, and `showLabels` controls its labels separately. Both default to `true`. Set these from the edition's layout and label controls; airport abbreviations equal to their IATA code are rendered once.
+
+## Compression at rest in the source store
+
+`capture` and `importCapture` accept `compression: 'gzip'`. The object is stored as `objects/<sha256>.gz` when gzip saves at least ten percent, otherwise raw; the record keeps the raw content hash and byte count as identity and adds `storedBytes` and `encoding`. `readCapture` and the new `readObject(store, hash, expectedBytes)` follow either form, verify the raw hash after expansion, and refuse output beyond the recorded size. `compactObjects(store, { minimumSaving })` compresses existing raw objects in place, verifying each before removing the raw file; records are not rewritten, since reads do not depend on them. Already-compressed sources such as ZIP archives stay raw. Use `storedBytes` for storage accounting and `bytes` for provenance.
+
+## Estimated fields, gridded series, field paths and daylight
+
+These transport-neutral modules came from Zugunruhe's radar-field studies and serve any work that renders a continuous surface between scattered instruments, a gridded weather product, illustrative flow, or the sun's position: modelled electricity demand between substations, rain radar, model wind, or a terminator crossing a national field. They are pure functions with no dependencies. See [the shared-field record](../docs/SHARED-FIELDS.md) for provenance and adoption.
+
+`@motionstudies/core/domain/supported-field` estimates values between stations. `fieldNeighbours` weights stations by a Gaussian kernel with a smooth taper to a finite reach; `estimateField` blends station samples and reports `support`, a fade from the nearest station that supplied a value, and `vectorSupport` for the nearest complete vector. All kilometre scales are `FieldSupportOptions`; `DEFAULT_FIELD_SUPPORT` retains Zugunruhe's 90 km scale, 240 km reach and 120–240 km support fade. Null station values are excluded rather than treated as zero; vector components blend as east/north components, never as bearings. `createFieldGrid` precomputes neighbours for a regular grid whose rows run south to north, `sampleFieldGrid` packs value, east, north and support per cell and level with NaN for unavailable channels, and `sampleFieldGridAt` reads it bilinearly, returning null beside any missing corner. The `movement` flag withholds cells without a complete vector and limits their support to the nearest vector, for path integration.
+
+Render support as opacity or reach, not as confidence in the underlying measurement: it says how far the nearest evidence is, nothing about that evidence's accuracy. The kernel scale is a claim about resolution and belongs in the study's provenance.
+
+`@motionstudies/core/domain/gridded-series` reads a regular latitude/longitude grid of variables at listed instants. `bracketTime` finds the adjacent frames; `sampleGriddedSeries` interpolates bilinearly in space and linearly in time; `meanGriddedSeries` returns an area-weighted mean, with an optional valid range. A missing sample with any weight makes the reading unavailable instead of bridging the gap. Values keep the source's units; the consumer converts percentages or applies its own scale.
+
+`@motionstudies/core/domain/field-paths` integrates illustrative paths through an estimated velocity field. `advanceFieldPath` takes an explicit midpoint step in metres per second, with `secondsPerStep` converting one unit of study time, and refuses to start from, pass through or land on unsupported field. `mixFieldMovement` blends two readings; `visibleTrailSegments` returns the last `trail` steps of a track at a study time with a fade at either end, identical when scrubbing back. Paths are neither observed trajectories nor individual movers and must be labelled as illustration.
+
+`@motionstudies/core/domain/daylight` computes `solarPosition` (altitude and azimuth in degrees, azimuth clockwise from north) from the Astronomy Answers low-precision algorithm, matching SunCalc to a thousandth of a degree without a dependency. `daylightAt` adds a unit `direction` towards the sun in a scene frame with +x east, +y up and +z south, and smooth `twilight`, `warmth` and `daylight` envelopes from `DaylightEnvelopes` (defaults: twilight −18° to −1°, warmth rising −12° to −1° and falling 3° to 15°, daylight −1° to 12°). The envelopes are artistic responses to sun altitude and encode no weather or sky observation. Sample several longitudes to place a terminator across a national field.
+
+The lab's **Fields** specimen renders a synthetic archipelago of seven instruments with a mid-study gap and one vectorless instrument, an invented cover series as haze, integrated trails, and a terminator crossing with the study clock. Brightness is the estimate and opacity is support; clicking reads the estimate, cover and sun altitude at a place.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@motionstudies/data",
-  "version": "0.1.0-alpha.14",
+  "version": "0.1.0-alpha.16",
   "private": false,
   "type": "module",
   "description": "Offline data preparation primitives for Motion Studies.",
@@ -99,7 +99,7 @@
   ],
   "sideEffects": false,
   "dependencies": {
-    "@motionstudies/core": "0.1.0-alpha.14",
+    "@motionstudies/core": "0.1.0-alpha.16",
     "fast-xml-parser": "5.11.1"
   }
 }

package/source-store.d.mts

@@ -1,13 +1,15 @@
 import type {Buffer} from 'node:buffer'
 export interface CaptureSource {id:string;publisher:string;kind:string;documentation:string;license:string;attribution?:string;origins:readonly string[]}
-export interface CaptureRecord {schemaVersion:1;source:Omit<CaptureSource,'origins'>;url:string;sha256:string;bytes:number;retrievedAt:string|null;importedAt?:string;acquisition:'http'|'local-import';validation:string;httpStatus?:number;contentType?:string|null;lastModified?:string|null;etag?:string|null}
+export interface CaptureRecord {schemaVersion:1;source:Omit<CaptureSource,'origins'>;url:string;sha256:string;bytes:number;storedBytes?:number;encoding?:'gzip'|null;retrievedAt:string|null;importedAt?:string;acquisition:'http'|'local-import';validation:string;httpStatus?:number;contentType?:string|null;lastModified?:string|null;etag?:string|null}
 export interface CaptureResult {bytes:Buffer;record:CaptureRecord;cached:boolean}
 export type CaptureValidator=(bytes:Buffer)=>unknown|Promise<unknown>
-export interface CaptureOptions {store:string;source:CaptureSource;url:string;offline?:boolean;refresh?:boolean;maxBytes?:number;timeoutMs?:number;retries?:number;validate?:CaptureValidator;fetchImpl?:typeof fetch;userAgent?:string}
+export interface CaptureOptions {store:string;source:CaptureSource;url:string;offline?:boolean;refresh?:boolean;maxBytes?:number;timeoutMs?:number;retries?:number;validate?:CaptureValidator;fetchImpl?:typeof fetch;userAgent?:string;compression?:'gzip'}
 export function sha256(bytes:string|Uint8Array):string
 export function publicSourceUrl(source:CaptureSource,value:string):string
 export function atomicJson(path:string,value:unknown):Promise<void>
 export function readCapture(store:string,source:CaptureSource,url:string,validate?:CaptureValidator):Promise<CaptureResult>
-export function importCapture(options:Pick<CaptureOptions,'store'|'source'|'url'|'maxBytes'|'validate'>&{input:string}):Promise<CaptureResult>
+export function importCapture(options:Pick<CaptureOptions,'store'|'source'|'url'|'maxBytes'|'validate'|'compression'>&{input:string}):Promise<CaptureResult>
+export function readObject(store:string,hash:string,expectedBytes?:number):Promise<Buffer>
+export function compactObjects(store:string,options?:{minimumSaving?:number}):Promise<{examined:number;compressed:number;kept:number;rawBytes:number;storedBytes:number}>
 export function capture(options:CaptureOptions):Promise<CaptureResult>
 export function jsonDocument(bytes:Buffer):Record<string,unknown>|unknown[]

package/source-store.mjs

@@ -1,7 +1,8 @@
 import { createHash, randomUUID } from 'node:crypto'
-import { mkdir, readFile, writeFile, rename, rm, stat } from 'node:fs/promises'
+import { mkdir, readFile, writeFile, rename, rm, stat, readdir, unlink } from 'node:fs/promises'
 import { join, dirname } from 'node:path'
 import { setTimeout as delay } from 'node:timers/promises'
+import { gzipSync, gunzipSync } from 'node:zlib'
 
 export const sha256 = bytes => createHash('sha256').update(bytes).digest('hex')
 
@@ -35,37 +36,69 @@ function requestKey(source, url) { return sha256(`${source.id}\n${url}`) }
 export async function readCapture(store, source, url, validate = () => {}) {
   const record = JSON.parse(await readFile(join(store, 'requests', `${requestKey(source, url)}.json`), 'utf8'))
   if (record.source.id !== source.id || record.url !== url || !/^[a-f0-9]{64}$/.test(record.sha256)) throw new Error('Capture identity mismatch')
-  const bytes = await readFile(join(store, 'objects', record.sha256))
+  const bytes = await readObject(store, record.sha256, record.bytes)
   if (bytes.length !== record.bytes || sha256(bytes) !== record.sha256) throw new Error('Capture integrity check failed')
   await validate(bytes)
   return { bytes, record, cached: true }
 }
+/** Read an object by its raw-content hash, whether it is stored raw or gzip-compressed at rest. */
+export async function readObject(store, hash, expectedBytes) {
+  try { return await readFile(join(store, 'objects', hash)) } catch (error) { if (error.code !== 'ENOENT') throw error }
+  const compressed = await readFile(join(store, 'objects', `${hash}.gz`))
+  try { return gunzipSync(compressed, { maxOutputLength: (expectedBytes ?? 512 * 1024 * 1024) + 1 }) } catch { throw new Error('Compressed capture could not be expanded within its recorded size') }
+}
+/** Store object bytes, compressing at rest when asked and worthwhile; the hash of the raw bytes remains the identity. */
+async function writeObject(store, hash, bytes, compression) {
+  if (compression === 'gzip') {
+    const compressed = gzipSync(bytes, { level: 6 })
+    if (compressed.length <= bytes.length * 0.9) { await atomicWrite(join(store, 'objects', `${hash}.gz`), compressed); return { storedBytes: compressed.length, encoding: 'gzip' } }
+  } else if (compression !== undefined) throw new Error('Unsupported compression; use gzip or leave unset')
+  await atomicWrite(join(store, 'objects', hash), bytes)
+  return { storedBytes: bytes.length, encoding: null }
+}
+/** Compress existing raw objects in a store where gzip saves at least the given fraction. Records keep their raw hash and byte count; reads follow either form. */
+export async function compactObjects(store, { minimumSaving = 0.1 } = {}) {
+  const result = { examined: 0, compressed: 0, kept: 0, rawBytes: 0, storedBytes: 0 }
+  let names
+  try { names = await readdir(join(store, 'objects')) } catch (error) { if (error.code === 'ENOENT') return result; throw error }
+  for (const name of names.filter(n => /^[a-f0-9]{64}$/.test(n))) {
+    const bytes = await readFile(join(store, 'objects', name))
+    result.examined++; result.rawBytes += bytes.length
+    const compressed = gzipSync(bytes, { level: 6 })
+    if (compressed.length <= bytes.length * (1 - minimumSaving)) {
+      await atomicWrite(join(store, 'objects', `${name}.gz`), compressed)
+      if (sha256(gunzipSync(compressed)) !== name) throw new Error('Compressed object failed verification; raw object retained')
+      await unlink(join(store, 'objects', name)); result.compressed++; result.storedBytes += compressed.length
+    } else { result.kept++; result.storedBytes += bytes.length }
+  }
+  return result
+}
 
-async function persist(store, source, url, bytes, acquisition, response = {}) {
+async function persist(store, source, url, bytes, acquisition, response = {}, compression) {
   const hash = sha256(bytes)
+  const stored = await writeObject(store, hash, bytes, compression)
   const record = {
     schemaVersion: 1, source: { id: source.id, publisher: source.publisher, kind: source.kind, documentation: source.documentation, license: source.license, ...(source.attribution ? { attribution: source.attribution } : {}) },
-    url, sha256: hash, bytes: bytes.length,
+    url, sha256: hash, bytes: bytes.length, storedBytes: stored.storedBytes, encoding: stored.encoding,
     ...(acquisition === 'http' ? { retrievedAt: new Date().toISOString() } : { retrievedAt: null, importedAt: new Date().toISOString() }),
     acquisition, validation: 'raw-capture; see adapter audit for fitness', ...response,
   }
-  await atomicWrite(join(store, 'objects', hash), bytes)
   await atomicJson(join(store, 'records', `${randomUUID()}.json`), record)
   await atomicJson(join(store, 'requests', `${requestKey(source, url)}.json`), record)
   return { bytes, record, cached: false }
 }
 
-export async function importCapture({ store, source, url, input, maxBytes = 512 * 1024 * 1024, validate = () => {} }) {
+export async function importCapture({ store, source, url, input, maxBytes = 512 * 1024 * 1024, validate = () => {}, compression }) {
   url = publicSourceUrl(source, url)
   if (!Number.isSafeInteger(maxBytes) || maxBytes < 1) throw new Error('Expected a positive byte limit')
   if ((await stat(input)).size > maxBytes) throw new Error('Source file exceeds byte limit')
   const bytes = await readFile(input)
   if (!bytes.length || bytes.length > maxBytes) throw new Error('Source file is empty or exceeds byte limit')
   await validate(bytes)
-  return persist(store, source, url, bytes, 'local-import')
+  return persist(store, source, url, bytes, 'local-import', {}, compression)
 }
 
-export async function capture({ store, source, url, offline = false, refresh = false, maxBytes = 32 * 1024 * 1024, timeoutMs = 20000, retries = 2, validate = () => {}, fetchImpl = fetch, userAgent = 'Motion Studies source capture' }) {
+export async function capture({ store, source, url, offline = false, refresh = false, maxBytes = 32 * 1024 * 1024, timeoutMs = 20000, retries = 2, validate = () => {}, fetchImpl = fetch, userAgent = 'Motion Studies source capture', compression }) {
   url = publicSourceUrl(source, url)
   if (!Number.isSafeInteger(maxBytes) || maxBytes < 1 || !Number.isSafeInteger(timeoutMs) || timeoutMs < 1 || !Number.isSafeInteger(retries) || retries < 0 || retries > 10) throw new Error('Invalid capture limits')
   if (offline && refresh) throw new Error('Offline and refresh cannot be combined')
@@ -112,7 +145,7 @@ export async function capture({ store, source, url, offline = false, refresh = f
       return await persist(store, source, url, bytes, 'http', {
         httpStatus: response.status, contentType: response.headers.get('content-type'),
         lastModified: response.headers.get('last-modified'), etag: response.headers.get('etag'),
-      })
+      }, compression)
     } catch (error) {
       if (!retryable || attempt === retries) throw error
       await delay(250 * 2 ** attempt)