@golemio/pid 5.7.2 → 5.7.3-dev.2378110705

package/docs/processing/transferboards_filtering.md

@@ -0,0 +1,312 @@
+## v4/pid/transferboards filtering pipeline (newcomer guide)
+
+This section documents every filtering and transformation step executed by the `GET /v4/pid/transferboards` endpoint, in the order they actually run. The goal is to help a developer who is new to the codebase understand **what** each stage does, **why** it exists, and **where** the code lives.
+
+---
+
+### Overview
+
+The endpoint answers the question: _"Given that I am on vehicle X arriving at stop Y, which onward connections should I show on the transfer board?"_
+
+The answer is not simply "all departures from stop Y in the next hour" — many departures must be filtered out because they go back the way the passenger just came, or depart too soon to catch, or are irrelevant in-network transfers. The pipeline also specially handles **guaranteed transfers** (connections the operator has contractually promised to wait for).
+
+**Primary source files:**
+- `src/output-gateway/pid/controllers/v4/V4TransferBoardsController.ts` — orchestrates the full pipeline
+- `src/output-gateway/pid/service/facade/TransferFacade.ts` — fetches the cache, runs line-direction filters
+- `src/output-gateway/pid/helpers/TransferBoardFilter.ts` — static filter methods (time, name, deduplication)
+- `src/output-gateway/pid/helpers/TransferBoardSorter.ts` — sort pipeline
+- `src/output-gateway/pid/service/transformations/TransferDepartureCacheTransformation.ts` — output grouping/capping
+
+---
+
+### Stage 1 — Request validation and parameter parsing
+
+**Where:** `V4PIDRouter.ts` (validation middleware) + `V4TransferBoardsController.ts` (top of `getTransferBoards`)
+
+The endpoint accepts two mutually exclusive identification modes:
+
+- **ASW mode** — `aswId` + `vehicleRegistrationNumber` + `routeType`
+- **CIS mode** — `cisId` + `tripNumber` (returns HTTP 501; not yet implemented)
+
+If neither set of params is provided, or both are, validation rejects the request early. CIS mode always returns `501 Not Implemented`.
+
+---
+
+### Stage 2 — Stop resolution (ASW → GTFS stop IDs)
+
+**Where:** `TransferFacade.getTransferCache()` → `gtfsStopsCache:*`
+
+A single ASW node can map to multiple GTFS `stop_id` values (e.g. platforms A and B at the same tram stop). The facade:
+
+1. Looks up all GTFS `stop_id` values for the given `aswId` from `gtfsStopsCache:{aswId}`.
+2. Fetches the stop names for those `stop_id` values.
+3. Groups them into **same-name clusters** — stops that share a `stop_name` (e.g. both platforms of "Náměstí Míru") are treated as one logical transfer point.
+
+This grouping matters later: only departures from stops in the same cluster pass the same-stop-name filters.
+
+---
+
+### Stage 3 — Current vehicle lookup
+
+**Where:** `V4TransferBoardsController.ts` + `vpPublicStopTimeCache:*`
+
+The controller looks up the arriving vehicle's trip data using `service-{routeType}-{registrationNumber}` as the key into `vpPublicStopTimeCache`. For buses, a fallback to the trolleybus key is tried if the bus key is missing (registration numbers are shared between the two types).
+
+This gives the controller the vehicle's **current `trip_id`** (called `currentTripId` throughout the codebase), which is central to guaranteed transfer detection and direction filtering.
+
+If the vehicle is not found in the stop-time cache, the response is `404 Not Found`.
+
+---
+
+### Stage 4 — Arrival time detection
+
+**Where:** `V4TransferBoardsController.ts` → `gtfsDelayComputation:{tripId}`
+
+The controller reads the vehicle's current trip from `gtfsDelayComputation`, which contains a per-stop array with both planned and predicted (delayed) times. It finds the entry for the target stop and extracts:
+
+- `plannedTimeFrom` — the GTFS scheduled arrival time at this stop
+- `delayedTimeFrom` — the real-time predicted arrival time (may differ due to delays)
+
+Both values are used to set the departure cache query window in the next stage.
+
+---
+
+### Stage 5 — Departure cache fetch
+
+**Where:** `TransferFacade.getTransferCache()` → `gtfsPublicDepartureCache:*`
+
+The facade fetches candidates from `gtfsPublicDepartureCache:{stopId}` (a Redis sorted set, scored by planned departure timestamp) for each GTFS `stop_id` in scope.
+
+**Time window:**
+```
+from  = plannedTimeFrom − 61 minutes
+to    = plannedTimeFrom + 60 minutes + delayMinutes
+```
+
+The 61-minute lookback exists to catch transfer lines whose scheduled departure is in the past but which may still arrive at the stop after the current vehicle due to their own delay of up to 60 minutes.
+
+**Fetch limit:** `max(80, min(120, limit × 10))` items per stop. The wide over-fetch ensures that enough candidates survive later filtering to fill the requested `limit`.
+
+---
+
+### Stage 6 — Self-removal
+
+**Where:** `V4TransferBoardsController.ts`
+
+The current vehicle's own trip (`currentTripId`) is removed from the candidate list. It should never appear as a "transfer option" for itself.
+
+---
+
+### Stage 7 — Pre-filter: bus/tram/trolleybus at wrong-name stops
+
+**Where:** `TransferFacade.getTransferCache()` (inline, before returning)
+
+Before the expensive line-direction filter runs, a cheap pre-filter drops departures that are:
+- Route type **bus, tram, or trolleybus** (metro/train/ferry/funicular pass through unconditionally)
+- From a stop whose `stop_name` differs from the current stop's name
+
+**Exception:** if the departure has a non-empty `connections[]` array (meaning it is a possible guaranteed transfer), it is kept regardless of stop name. This is because a guaranteed transfer bus might legitimately be at a different-named stop.
+
+This pre-filter is a performance optimization — it eliminates obviously irrelevant candidates before the costlier stages.
+
+---
+
+### Stage 8 — Sort + real-time vehicle position join
+
+**Where:** `V4TransferBoardsController.ts`
+
+Remaining candidates are sorted by `departure_datetime` ascending, then each is joined with live vehicle positions from `vpPublicCache` (if the trip has an active vehicle). This enriches each candidate with:
+
+- `predictedDepartureDate` / `predictedDepartureTimestamp` — the RT-adjusted departure time
+- `is_wheelchair_accessible` — resolved from the RT vehicle descriptor if available (see Stage 9)
+
+---
+
+### Stage 9 — Wheelchair accessibility resolution
+
+**Where:** `DepartureBoardMapper.determineWheelchairAccessibility()`
+
+For each candidate departure, accessibility is determined differently by mode:
+
+| Mode | Source |
+|---|---|
+| **Metro** | `wheelchair_boarding` from the GTFS stop (stop-level, static) |
+| **All other** | `real_wheelchair_accessible` from the RT vehicle descriptor (boolean), or if absent: GTFS static `wheelchair_accessible` (trip-level, `GtfsTripWheelchairAccessEnum`) |
+
+The fallback to GTFS static data is the fix introduced in v5.7.1 (item 3 above).
+
+---
+
+### Stage 10 — Guaranteed transfer detection
+
+**Where:** `V4TransferBoardsController.ts` (`havingConnectionFromTripId`)
+
+A departure is a **guaranteed transfer** (Czech: _zaručený přestup_) when:
+
+1. Its `connections[]` array contains an entry where `from_trip_id === currentTripId`, AND
+2. The current vehicle's predicted arrival time is within the wait window:
+   ```
+   vehicleArrivalMs < departurePlannedMs + max_wait_sec × 1000
+   ```
+
+Guaranteed transfers receive special treatment in all subsequent stages — they can bypass nearly every filter and are always shown if time constraints allow.
+
+The result is two sets: `guaranteedTripIds` and `guaranteedConnections`.
+
+---
+
+### Stage 11 — Minimal transfer time filter
+
+**Where:** `TransferBoardFilter.minimalTransferTime()`
+
+Departures whose scheduled time is too close to the vehicle's arrival are dropped — there is not enough time for passengers to board. Thresholds differ by mode:
+
+| Mode | Minimum gap |
+|---|---|
+| Bus, tram, trolleybus | 60 seconds |
+| Metro, train, ferry, funicular | 120 seconds |
+
+**Guaranteed transfers are exempt** from this filter — the operator has guaranteed the connection, so time is irrelevant.
+
+---
+
+### Stage 12 — Same-stop-name filter (Stage 2)
+
+**Where:** `TransferBoardFilter.sameStopNameTransfer()`
+
+A second pass of the stop-name check, applied after guaranteed transfers are identified. Rules:
+
+- Bus/tram/trolleybus at a stop with a different name than the arrival stop → **dropped**
+- Metro/train/ferry/funicular → **kept** (they serve different physical locations by design)
+- Any trip in `guaranteedTripIds` → **kept** (guaranteed transfer exemption)
+- Any trip in `keepAlwaysTripIds` (see Stage 14) → **kept**
+
+This stage is separate from the Stage 7 pre-filter because at Stage 7 we did not yet know which trips were guaranteed.
+
+---
+
+### Stage 13 — Categorise trips for line-direction filter
+
+**Where:** `V4TransferBoardsController.ts`
+
+Before running the expensive direction filter, the controller categorises remaining candidates:
+
+- **Always keep:** guaranteed transfers + "keep always" types (metro, train, ferry, funicular)
+- **Needs filtering:** everything else (bus, tram, trolleybus that are not guaranteed)
+
+Only the "needs filtering" set is passed to `findRelevantTripIdsFromLines`. This avoids running direction analysis on trips that will be kept unconditionally.
+
+---
+
+### Stage 14 — Guaranteed transfer time adjustment
+
+**Where:** `V4TransferBoardsController.ts`
+
+For guaranteed transfers, the displayed departure time is **replaced** with the arriving vehicle's own predicted arrival time. This is the time the bus/tram will actually wait until, so showing the vehicle's arrival time as the "departure" is more useful to passengers than the originally scheduled departure time.
+
+---
+
+### Stage 15 — Sort (pre-output)
+
+**Where:** `TransferBoardSorter.sort()`
+
+A four-criterion stable sort applied before the direction filter:
+
+1. **Metro first** — metro departures before all others
+2. **Metro name ascending** — within metro, sort by `route_short_name` (A before B before C)
+3. **Departure time ascending** — earlier departures first
+4. **`direction_id` ascending** — tiebreaker for trips with identical departure times
+
+---
+
+### Stage 16 — Line-direction filter (`findRelevantTripIdsFromLines`)
+
+**Where:** `TransferFacade.findRelevantTripIdsFromLines()`
+
+This is the most complex stage. It determines whether each candidate trip is going somewhere _new_ for the passenger, or whether it is effectively doubling back along the route the passenger just arrived on.
+
+For each candidate trip, the facade loads its stop sequence from `gtfsDelayComputation` or `vpPublicStopTimeCache`. It then applies two sequential sub-filters:
+
+#### Sub-filter A: Forward-subset check (`filterOutForwardSubgroupOfLine`)
+
+**Question:** Does this candidate trip's future stops form a strict subset of the current vehicle's future stops?
+
+If every stop the candidate will visit is also a future stop on the current vehicle's route, then there is no point transferring — the passenger will get there anyway without changing. Such trips are **dropped**.
+
+**Example:** Current vehicle is bus 207 going A→B→C→D. Candidate bus 150 goes B→C. Since B and C are both future stops of 207, transferring to 150 is redundant → dropped.
+
+#### Sub-filter B: Backtracking conflict check (`filterOutLinesWithBacktrackingConflict`)
+
+**Question:** Do this candidate trip's next N stops overlap with the current vehicle's previous N stops in a way that suggests it is heading back the same way?
+
+Configured by `countPreviousStopsToAllow` (default `0`). The value defines how many of the current vehicle's previous stops the candidate trip is allowed to share in its upcoming stops. If the candidate shares more than that number, it is **dropped** (it's going backwards). `0` means no overlap is permitted at all.
+
+The strictness of the stop matching is controlled by `isOppositeDirectionFilterStopNameStrict` (default `true`): when true, both `stop_id` AND `stop_name` must match; when false, only `stop_id`.
+
+---
+
+### Stage 17 — Combine approved trip IDs
+
+**Where:** `TransferBoardFilter.keepAlwaysLinesAndTripIds()`
+
+The final set of approved trip IDs is the union of:
+- `guaranteedTripIds` (from Stage 10)
+- `keepAlwaysTripIds` (metro/train/ferry/funicular, from Stage 13)
+- Trips that passed the direction filter (from Stage 16)
+
+All other trips are discarded from the candidate list here.
+
+---
+
+### Stage 18 — Output grouping and per-route cap
+
+**Where:** `TransferDepartureCacheTransformation.transform()`
+
+Before the final limit is applied, departures are grouped by route and capped to prevent one line from dominating the board:
+
+| Mode | Grouping key | Cap |
+|---|---|---|
+| **Metro** | `route_short_name + stop_id` | 2 departure times merged into one `departure_timestamp.minutes[]` entry |
+| **Non-metro** | `route_short_name + direction_id` | 2 separate departure entries per group |
+
+For non-metro, `direction_id` is taken from the live vehicle positions cache (keyed by `route_short_name + trip_headsign`), defaulting to `0` when not available.
+
+---
+
+### Stage 19 — Final filter: limit, time window, deduplication
+
+**Where:** `TransferBoardFilter.filterDepartures()`
+
+The last stage applies three rules:
+
+1. **Time window cut:** departures after `arrivalTime + 1 hour` are dropped.
+2. **Trip deduplication:** if two entries share the same `trip_id`, only the first (earliest) is kept.
+3. **Limit:** the result is truncated to the requested `limit` (query parameter, default typically 10).
+
+**Guaranteed transfers are exempt from the time window and limit** — they always appear regardless of how many other results there are, as long as they are still within the `max_wait_sec` window.
+
+---
+
+### Quick reference: which filter each vehicle type is subject to
+
+| Filter | Bus/Tram/Trolleybus | Metro/Train/Ferry/Funicular | Guaranteed transfer |
+|---|---|---|---|
+| Stage 7 pre-filter (wrong stop name) | Yes | No | No (exempt) |
+| Stage 11 min transfer time | Yes (60 s) | Yes (120 s) | **No** |
+| Stage 12 same-stop-name | Yes | No | **No** |
+| Stage 16 direction filter | Yes | **No** | **No** |
+| Stage 17 combine | All types pass via their category | — | — |
+| Stage 18 per-route cap (2 entries) | Yes | Yes (metro grouped differently) | Yes |
+| Stage 19 limit | Yes | Yes | **No** |
+
+---
+
+### Key Redis keys used
+
+| Key pattern | Purpose |
+|---|---|
+| `gtfsStopsCache:{aswId}` | ASW node → GTFS stop_id mapping |
+| `gtfsPublicDepartureCache:{stopId}` | Sorted set of precomputed departures (score = departure Unix timestamp) |
+| `gtfsDelayComputation:{tripId}` | Per-trip RT stop times, delay, shapes |
+| `vpPublicStopTimeCache:{vehicleKey}` | Per-vehicle stop time lookup (used to identify current trip) |
+| `vpPublicCache:*` | Geo-indexed real-time vehicle positions |

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@golemio/pid",
-    "version": "5.7.2",
+    "version": "5.7.3-dev.2378110705",
     "description": "Golemio PID Module",
     "main": "dist/index.js",
     "types": "dist/index.d.ts",
@@ -9,13 +9,16 @@
         "build-minimal": "run-s 'build -- --sourceMap false --declaration false'",
         "build-watch": "run-s 'build -- --watch --preserveWatchOutput'",
         "refresh-precomputed-tables": "cross-env NODE_ENV='test' TZ='UTC' ts-node -r tsconfig-paths/register -r dotenv/config test/scripts/refresh-precomputed-tables.ts",
-        "pretest": "golemio import-db-data && run-s \"refresh-precomputed-tables\"",
-        "test": "cross-env NODE_ENV='test' TZ='UTC' mocha --exit --check-leaks --timeout 120000 --reporter-option maxDiffSize=0 -r ts-node/register -r tsconfig-paths/register --file 'test/setup.ts' -r dotenv/config 'test/**/*.test.ts'",
-        "test-debug": "run-s 'test -- --inspect-brk=9230'",
+        "prepare-db": "golemio import-db-data && run-s \"refresh-precomputed-tables\"",
+        "test:debug": "run-s 'test -- --inspect-brk=9230'",
+        "test:only": "cross-env NODE_ENV='test' TZ='UTC' mocha --exit --check-leaks --timeout 120000 --reporter-option maxDiffSize=0 -r ts-node/register -r tsconfig-paths/register --file 'test/setup.ts' -r dotenv/config 'test/**/*.test.ts'",
+        "test": "run-s prepare-db test:only",
         "apidocs-test": "npm run apidocs-test-output",
         "apidocs-test-input": "cross-env NODE_ENV=test golemio swagger api-test --oas docs/openapi-input.yaml --script test/api-docs/input-gateway/server.js --filter test/api-docs/input-gateway/portman-filter.json",
         "apidocs-test-output": "cross-env NODE_ENV=test golemio swagger api-test --oas docs/openapi-output.yaml --script test/api-docs/output-gateway/server.js --config test/api-docs/output-gateway/portman-config.json",
-        "code-coverage": "nyc run-s 'test -- -r source-map-support/register'",
+        "coverage:only": "nyc --reporter=text --reporter=lcov run-s 'test:only -- -r source-map-support/register'",
+        "code-coverage": "run-s prepare-db coverage:only",
+        "format:check-staged": "pretty-quick --staged --pattern '**/*.ts'",
         "generate-docs": "typedoc --out docs/typedoc src",
         "lint": "eslint --cache \"{src,test}/**/*.ts\"",
         "validate-dependencies": "dependency-cruiser --config .dependency-cruiser.js src"