swiss-netex 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 47374355daf2b7133d4cdd64c44bc6a2574b3a1f77fc7ede6ed3dec582c85e67
+  data.tar.gz: 6e8dc725d90ae6c5e19a8e1cb7dad5113c20603994f648712d9d169b73a0760e
+SHA512:
+  metadata.gz: 4a5c0a1aca546bac12a3196cc0b52d4822caeb5665bde3586b65678eb5a635cdbb2896cb6c4c7eead48e082ca8ec14a37b5f0762e43f47f039b05be171cf3f87
+  data.tar.gz: 6c51f8a53e717ef9f0be266b7859d9665afae4d1c099bf79af9a17015bb1b4370d78bb59cb046bb6e1b3ca5e34c6b6117695d3425898691103bfae645bee1f3b

data/CHANGELOG.md

@@ -0,0 +1,77 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [1.0.0] — unreleased
+
+First public release. Download, inspect, and filter Swiss national NeTEx
+timetable packages from [opentransportdata.swiss](https://opentransportdata.swiss).
+
+Marked **1.0.0** (not 0.x): the CLI surface is complete for the intended
+scope and treated as a stable contract. Tag/publish waits on team review.
+
+### Added
+
+- CLI binary `swiss-netex` with `help`, `version`, `download`, `operators`,
+  `lines`, `filter`, and `extract`
+- `download` — resolve the latest (or year-pinned) national NeTEx zip via the
+  public catalog + dataset page HTML, stream to disk, cache under
+  `~/.cache/swiss-netex` (override with `--cache-dir` / `--force`)
+- `operators` / `lines` — discovery tables from RESOURCE / SERVICE only
+  (TSV when piped; aligned columns on a TTY). Optional `--operator` /
+  `--line` filters on `lines`. Omit `--from` to download/cache first
+  (`-y` / `-d` / `-c` / `-f`, same rules as `extract`)
+- `filter` — operator/line filter for a local zip or directory package:
+  streaming SERVICE + TIMETABLE passes, transitive RESOURCE / SERVICE /
+  SITE / SERVICECALENDAR / COMMON rewrite, drop empty timetable shards,
+  `README.txt` in the output zip
+- `extract` — download (when `--from` omitted) + filter in one command
+- Operator aliases: GO number, `ch:1:Operator:*`, SBOID, ShortName
+  (repeatable `--operator` = union). Line aliases: PublicCode, id, SLNID,
+  ShortName, Name. With explicit `--line`, journeys must carry a matching
+  `LineRef` (operator-only fallback only when no line filter is set)
+- COMMON connection entities (`JourneyMeeting`, `InterchangeRule`) kept only
+  when every tracked journey/stop/line ref still resolves in the package
+- Bare `-o` stem without extension becomes `stem.zip` for filter/extract;
+  existing directories / trailing `/` still get the default `*_op-….zip`
+  name inside
+- National-scale streaming: short-lived fork workers for TIMETABLE shards
+  and support frames, parent byte pre-scan to skip empty shards, streamed
+  kept-journey bodies, non-DOM COMMON scanner
+- Env knobs: `SWISS_NETEX_TT_WORKERS` (default CPU count; `1` = serial),
+  `SWISS_NETEX_TT_BATCH` (default 1 shard/child), `SWISS_NETEX_NO_FORK=1`
+- `SwissNetex::Package`, `ResourceFrame`, `Operators`, `Lines`, `Filter`,
+  `Download`, `Extract` library entry points
+- Minitest suite (fixture packages only; no national zip in CI), GitHub
+  Actions CI + tag publish gated on test/lint, minimal RuboCop, RubyGems
+  trusted-publish on version tags
+
+### Fixed
+
+- Filtered XML no longer keeps blank “holes” where dropped entities used to sit
+  (streaming whitespace collapse in SERVICE/SITE/RESOURCE/SERVICECALENDAR,
+  TIMETABLE, and COMMON writers — no full DOM pretty-print)
+
+### Notes
+
+- Tool is MIT; timetable data remains under OpenTransportData terms
+- Cache key is the remote filename (export timestamp embedded); a positive-size
+  file must also open as a zip before it counts as a hit. Use `--force` to
+  re-download
+- Downloads request `Accept-Encoding: identity` so size checks compare
+  wire `Content-Length` to the bytes written on disk
+- Filter/extract write the output zip via a same-directory temp file + rename
+- TIMETABLE `TrainNumber` defs are kept whether they appear before or after
+  `vehicleJourneys` (defs-first inputs may emit `trainNumbers` after journeys)
+- Filtered-package `README.txt` and public README document Swiss-profile
+  expectations: `ValidDayBits` calendars, closed COMMON connections, sparse
+  TIMETABLE shard indices
+- Peak system RSS scales roughly with worker count × largest kept shard;
+  prefer a machine with several GB free RAM for full national filters
+- `extract` / `operators` / `lines`: `--from` cannot be combined with
+  `--dataset` / `--year` / `--force` / `--cache-dir` (download-only flags)

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 GateMedia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,237 @@
+# swiss-netex
+
+CLI for Swiss national NeTEx timetable packages
+([opentransportdata.swiss](https://opentransportdata.swiss)): download, inspect,
+and filter by operator/line.
+
+This is **not** a full NeTEx model library. It prepares smaller operator-scoped
+packages from the national SKI NeTEx feed.
+
+## Requirements
+
+- Ruby ≥ 4.0
+
+## Installation
+
+```bash
+gem install swiss-netex
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/gatemedia/swiss-netex.git
+cd swiss-netex
+mise run bootstrap   # or: bundle install
+bundle exec exe/swiss-netex version
+```
+
+## Quick start
+
+Download (or reuse cache) and filter one operator in a single step:
+
+```bash
+swiss-netex extract -y 2026 -O 151 -o tl.zip
+```
+
+Stdout prints the output path. Progress goes to stderr.
+
+```bash
+swiss-netex help
+swiss-netex help extract   # full flags for any command
+swiss-netex version
+```
+
+## Commands
+
+### `download`
+
+Resolve and stream the latest national NeTEx zip. Resource downloads are public
+(no API token on the happy path). Prints the file path on stdout.
+
+```bash
+swiss-netex download
+swiss-netex download --year 2026
+swiss-netex download -y 2026 -o ./cache
+swiss-netex download --dataset timetablenetex_2026 --force
+swiss-netex download --cache-dir ./cache
+```
+
+- Default dataset: highest published `timetablenetex_YYYY` on the public catalog
+- Override with `--year YYYY` or `--dataset` (slug or full page URL)
+- Default cache: `~/.cache/swiss-netex`
+- Cache key is the remote filename (export timestamp embedded). A positive-size
+  local file with that name that also opens as a zip is a cache hit — OTD signed
+  URLs reject HEAD, so remote size is not re-checked. Use `--force` to refresh.
+
+### `operators`
+
+List operators from the RESOURCE frame. Scans RESOURCE only — not TIMETABLE
+shards — so listing stays cheap even on a full national package. Omit `--from`
+to download (or reuse cache) first — same flags as `download` / `extract`.
+
+```bash
+swiss-netex operators -y 2026
+swiss-netex operators --from ./cache/PROD_NETEX_….zip
+swiss-netex operators -y 2026 | cut -f1,4
+```
+
+Columns: `go`, `operator_id`, `sboid`, `short_name`, `name`.
+
+### `lines`
+
+List lines from the SERVICE frame (SERVICE only — not TIMETABLE). Optional
+`--operator` / `--line` narrow the list (same alias rules as `filter`). Omit
+`--from` to download (or reuse cache) first.
+
+```bash
+swiss-netex lines -y 2026
+swiss-netex lines -y 2026 -O 151
+swiss-netex lines --from ./cache/PROD_NETEX_….zip -O 151 -l m1
+```
+
+Columns: `line_id`, `public_code`, `slnid`, `operator_ref`, `name`.
+
+### `filter`
+
+Filter a local package (zip or extracted directory) by operator and optional
+line. Writes a smaller zip with transitive frame contents and a `README.txt`.
+
+```bash
+swiss-netex filter --from ./cache/PROD_NETEX_….zip -O 151 -o tl.zip
+swiss-netex filter --from ./cache/PROD_NETEX_….zip \
+  -O 151 -O 55 -l 1 -o filtered.zip --verbose
+swiss-netex filter --from ./package-dir/ -O 151 -o ./out/
+```
+
+- `--operator` / `-O` is **repeatable** (union). Aliases: GO number,
+  `ch:1:Operator:*`, SBOID, ShortName
+- Optional `--line` / `-l` (repeatable): PublicCode, id, SLNID, ShortName, Name
+- Keeps matching SERVICE lines and TIMETABLE `ServiceJourney`s, then the
+  transitive closure in RESOURCE / SERVICE / SITE / SERVICECALENDAR / COMMON
+- Empty timetable shards are dropped (original national shard indices kept)
+- Without `--allow-empty`, exits non-zero when no lines or journeys match
+- `--verbose` prints per-shard and per-frame keep/drop counters on stderr
+- Output zip includes `README.txt` (filters, counts, Swiss-profile notes)
+
+### `extract`
+
+Download (when `--from` is omitted) then filter. The common one-shot workflow.
+
+```bash
+swiss-netex extract -O 151 -o tl.zip
+swiss-netex extract -y 2026 -O 151 -o tl.zip
+swiss-netex extract --from ./cache/PROD_NETEX_….zip -O 151 -o tl.zip
+```
+
+Same filter flags as `filter`, plus download flags when no `--from` is given.
+`operators` and `lines` share the same package-source rules: omit `--from` to
+download/cache; with `--from`, `--dataset` / `--year` / `--force` /
+`--cache-dir` are rejected.
+
+## Output and conventions
+
+| Stream | Content |
+|--------|---------|
+| stdout | Machine-usable result (file path, or TSV/table for discovery) |
+| stderr | Progress, status, verbose counters, errors |
+
+Discovery commands (`operators`, `lines`): raw TSV when piped / non-TTY;
+aligned columns on an interactive terminal (same cells, two-space padding).
+
+**`-o` path rules** (`filter` / `extract`):
+
+| `-o` value | Result |
+|------------|--------|
+| `tl.zip` | as-is |
+| `tl` (no extension) | `tl.zip` |
+| `tl.tar` | as-is (explicit non-zip extension left alone) |
+| existing directory or trailing `/` | `dir/<source>_op-….zip` |
+
+Exit codes: `0` ok, `1` runtime/domain error, `64` usage/parse error.
+
+## Memory and speed
+
+National packages are large (~hundreds of MB compressed, tens of GB
+uncompressed). `filter` / `extract` stream frames and never load the full
+national set into a DOM:
+
+- short-lived worker processes on platforms with `fork`
+- parent byte pre-scan skips empty TIMETABLE shards
+- process pool for matching shards and support frames
+- non-DOM COMMON scanner
+
+| Env | Effect |
+|-----|--------|
+| `SWISS_NETEX_TT_WORKERS=N` | Pool size (default: CPU count; `1` = serial) |
+| `SWISS_NETEX_TT_BATCH` | Shards per child (default: `1`) |
+| `SWISS_NETEX_NO_FORK=1` | Force in-process mode |
+
+Prefer a machine with **≥4 GB free RAM**. Peak system RSS scales roughly with
+worker count × largest kept shard.
+
+## Swiss profile notes (filtered packages)
+
+These are normal for SKI national exports and operator-scoped subsets — not
+filter bugs. The same points are written into each output `README.txt`.
+
+**Calendar / `DayType`**
+
+Journeys typically reference `AvailabilityCondition`s that carry
+`FromDate` / `ToDate` / `ValidDayBits`. Many selections collect **zero**
+`DayType` / `DayTypeRef` entities. Importers should read validity from the
+kept ACs in SERVICECALENDAR, not assume a DayType model.
+
+**COMMON connections**
+
+`JourneyMeeting` and `InterchangeRule` are kept only when **every** tracked
+journey, stop, and line ref still resolves inside the package (no dangling
+partner `LineRef`). Cross-operator interchanges drop unless you include both
+operators (e.g. `-O 151 -O 55`). An empty connection section is valid.
+
+**TIMETABLE shard names**
+
+Only shards with kept journeys are written. Filenames keep the national
+`TIMETABLE_{i}_{n}_…` indices, so numbering can have gaps (e.g. 45–69 plus
+113). Consumers should glob timetable members, not assume `1…N` contiguous.
+
+## Data license
+
+The **tool** is MIT. **Timetable data** remains under
+[OpenTransportData terms](https://opentransportdata.swiss). Do not commit
+national dumps into this repository.
+
+## Development
+
+This project uses [mise](https://mise.jdx.dev/) for the local toolchain.
+
+```bash
+mise trust          # once, if prompted
+mise install        # Ruby from mise.toml / .ruby-version
+mise run bootstrap  # bundle install
+mise run test       # bundle exec rake test
+mise run lint       # bundle exec rake rubocop
+mise run ci         # tests + rubocop
+mise run swiss-netex help
+mise run swiss-netex version
+mise run build      # gem package into pkg/
+```
+
+Without mise:
+
+```bash
+bundle install
+bundle exec rake        # test + rubocop
+bundle exec rake test
+bundle exec rake rubocop
+```
+
+## Release
+
+1. Bump `SwissNetex::VERSION` and update `CHANGELOG.md`
+2. Tag `vX.Y.Z` and push the tag
+3. GitHub Actions builds and publishes to [RubyGems](https://rubygems.org/gems/swiss-netex)
+
+## License
+
+MIT © GateMedia — see [LICENSE](LICENSE).

data/exe/swiss-netex

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "swiss_netex"
+
+exit SwissNetex::CLI.start(ARGV)

data/lib/swiss_netex/catalog.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module SwissNetex
+  # Parses an opentransportdata.swiss catalog search HTML page and picks the
+  # highest Fahrplanjahr among national NeTEx dataset slugs
+  # (`timetablenetex_YYYY`).
+  #
+  # CKAN JSON APIs require a token on this host; the public HTML catalog does not.
+  class Catalog
+    DATASET_SLUG_RE = %r{
+      /(?:[a-z]{2}/)?dataset/(timetablenetex_(\d{4}))
+      (?:["'\s?#]|$)
+    }ix
+
+    def self.latest_slug(html)
+      new(html).latest_slug
+    end
+
+    def initialize(html)
+      @html = html.to_s
+    end
+
+    def slugs
+      seen = {}
+      @html.scan(DATASET_SLUG_RE).filter_map do |slug, year|
+        key = slug.downcase
+        next if seen[key]
+
+        seen[key] = true
+        { slug: key, year: Integer(year) }
+      end
+    end
+
+    def latest_slug
+      list = slugs
+      raise Error, "No timetablenetex_YYYY dataset found on catalog search page" if list.empty?
+
+      list.max_by { |entry| entry[:year] }[:slug]
+    end
+  end
+end