gtfs 3.2.5 → 3.4.0

package/README.md

@@ -21,13 +21,13 @@
 
 <hr>
 
-`node-GTFS` loads transit data in [GTFS format](https://developers.google.com/transit/) into a SQLite database and provides some methods to query for agencies, routes, stops, times, fares, calendars and other GTFS data. It also offers spatial queries to find nearby stops, routes and agencies and can convert stops and shapes to geoJSON format.
+`node-GTFS` loads transit data in [GTFS format](https://developers.google.com/transit/) into a SQLite database and provides some methods to query for agencies, routes, stops, times, fares, calendars and other GTFS data. It also offers spatial queries to find nearby stops, routes and agencies and can convert stops and shapes to geoJSON format. Additionally, this library can export data from the SQLite database back into GTFS (csv) format.
 
-Additionally, this library can export data from the SQLite database back into GTFS (csv) format.
+The library also supports importing GTFS-Realtime data into the same database. In order to keep the realtime database fresh, it uses `SQLITE REPLACE` which makes it very effective.
 
 You can use it as a [command-line tool](#command-line-examples) or as a [node.js module](#code-example).
 
-This library has three parts: the [GTFS import script](#gtfs-import-script), the [query methods](#query-methods) and the [GTFS export script](#gtfs-export-script)
+This library has four parts: the [GTFS import script](#gtfs-import-script), [GTFS export script](#gtfs-export-script) and [GTFS-Realtime update script](#gtfsrealtime-update-script) and the [query methods](#query-methods)
 
 ## Installation
 
@@ -96,13 +96,13 @@ importGtfs(config)
   </tr>
 </table>
 
-## Command Line Usage
+## Command-Line Usage
 
 The `gtfs-import` command-line utility will import GTFS into SQLite3.
 
 The `gtfs-export` command-line utility will create GTFS from data previously imported into SQLite3.
 
-### gtfs-import Command-line options
+### gtfs-import Command-Line options
 
 `configPath`
 
@@ -219,6 +219,25 @@ API along with your API token.
 }
 ```
 
+- Specify urls for GTFS-Realtime updates. `realtimeUrls` allows an array of GTFS-Realtime URLs. For example, a URL for trip updates, a URL for vehicle updates and a URL for service alerts. In addition, a `realtimeHeaders` parameter allows adding additional HTTP headers to the request.
+
+```json
+{
+  "agencies": [
+    {
+      "url": "http://countyconnection.com/GTFS/google_transit.zip",
+      "realtimeUrls": [
+        "https://opendata.somewhere.com/gtfs-rt/VehicleUpdates.pb",
+        "https://opendata.somewhere.com/gtfs-rt/TripUpdates.pb"
+      ],
+      "realtimeHeaders": {
+        "Authorization": "bearer 1234567890"
+      }
+    }
+  ]
+}
+```
+
 - Specify multiple agencies to be imported into the same database
 
 ```json
@@ -353,6 +372,39 @@ importGtfs(config)
   });
 ```
 
+## `gtfsrealtime-update` Script
+
+The `gtfsrealtime-update` script requests GTFS-Realtime data and importings into a SQLite database. [GTFS-Realtime data](https://gtfs.org/realtime/reference/) can compliment GTFS Static data. [Read more about GTFS-Realtime configuration](#configuration).
+
+### Run the `gtfsrealtime-update` script from command-line
+
+    gtfsrealtime-update
+
+By default, it will look for a `config.json` file in the project root. To specify a different path for the configuration file:
+
+    gtfsrealtime-update --configPath /path/to/your/custom-config.json
+
+### Use `updateGtfsRealtime` script in code
+
+Use `updateGtfsRealtime()` in your code to run an update of a GTFS-Realtime data specified in a config.json file.
+
+```js
+import { updateGtfsRealtime } from 'gtfs';
+import { readFile } from 'fs/promises';
+
+const config = JSON.parse(
+  await readFile(new URL('./config.json', import.meta.url))
+);
+
+updateGtfsRealtime(config)
+  .then(() => {
+    console.log('Update Successful');
+  })
+  .catch((err) => {
+    console.error(err);
+  });
+```
+
 ## `gtfs-export` Script
 
 The `gtfs-export` script reads from a JSON configuration file and exports data in GTFS format from a SQLite database. [Read more on setting up your configuration file](#configuration).
@@ -371,7 +423,7 @@ By default, it will look for a `config.json` file in the project root. To specif
 
     gtfs-export --configPath /path/to/your/custom-config.json
 
-### Command Line options
+### Command-Line options
 
 #### Specify path to config JSON file
 
@@ -381,7 +433,7 @@ You can specify the path to a config file to be used by the export script.
 
 #### Show help
 
-Show all command line options
+Show all command-line options
 
     gtfs-export --help
 
@@ -417,6 +469,8 @@ This library includes many methods you can use in your project to query GTFS dat
 
 Most methods accept three optional arguments: `query`, `fields` and `sortBy`.
 
+More advanced methods include `advancedQuery`, `runRawQuery` and `execRawQuery`.
+
 #### Query
 
 For example, to get a list of all routes with just `route_id`, `route_short_name` and `route_color` sorted by `route_short_name`:
@@ -569,7 +623,7 @@ getStops({
 `getStops` allows passing a `route_id` in the query and it will query trips and stoptimes to find all stops served by that `route_id`.
 
 ```js
-import { getRoutes } from 'gtfs';
+import { getStops } from 'gtfs';
 
 // Get all stops for a specific route
 getStops({
@@ -580,7 +634,7 @@ getStops({
 `getStops` allows passing a `trip_id` in the query and it will query stoptimes to find all stops on that `trip_id`.
 
 ```js
-import { getRoutes } from 'gtfs';
+import { getStops } from 'gtfs';
 
 // Get all stops for a specific trip
 getStops({
@@ -588,6 +642,17 @@ getStops({
 });
 ```
 
+`getStops` allows passing a `shape_id` in the query and it will query trips and stoptimes to find all stops that use that `shape_id`.
+
+```js
+import { getStops } from 'gtfs';
+
+// Get all stops for a specific trip
+getStops({
+  shape_id: 'cal_sf_tam',
+});
+```
+
 ### getStopsAsGeoJSON(query)
 
 Queries stops and returns a promise. The result of the promise is an geoJSON object of stops. All valid queries for `getStops()` work for `getStopsAsGeoJSON()`.
@@ -718,23 +783,28 @@ Returns geoJSON of shapes.
 ```js
 import { getShapesAsGeoJSON } from 'gtfs';
 
-// Get geoJSON of all stops in an agency
+// Get geoJSON of all routes in an agency
 getShapesAsGeoJSON();
 
-// Get geoJSON of stops along a specific route
+// Get geoJSON of shapes for a specific route
 getShapesAsGeoJSON({
   route_id: 'Lo-16APR',
 });
 
-// Get geoJSON of stops for a specific trip
+// Get geoJSON of shapes for a specific trip
 getShapesAsGeoJSON({
   trip_id: '37a',
 });
 
-// Get geoJSON of stops for a specific `service_id`
+// Get geoJSON of shapes for a specific `service_id`
 getShapesAsGeoJSON({
   service_id: 'CT-16APR-Caltrain-Sunday-02',
 });
+
+// Get geoJSON of shapes for a specific `shape_id`
+getShapesAsGeoJSON({
+  shape_id: 'cal_sf_tam',
+});
 ```
 
 ### getCalendars(query, fields, sortBy)
@@ -995,6 +1065,114 @@ getTimetableNotesReferences({
 });
 ```
 
+### getTripsDatedVehicleJourneys(query, fields, sortBy)
+
+Queries trips_dated_vehicle_journey and returns a promise. The result of the promise is an array of trips_dated_vehicle_journey. These are from the non-standard `trips-dated-vehicle-journey.txt` file. See [documentation and examples of this file](https://www.trafiklab.se/api/trafiklab-apis/gtfs-regional/extra-files/).
+
+```js
+import { getTripsDatedVehicleJourneys } from 'gtfs';
+
+// Get all timetable_stop_orders
+getTripsDatedVehicleJourneys();
+```
+
+### getServiceAlerts(query, fields, sortBy)
+
+Queries service alerts and returns a promise. The result of the promise is an array of service alerts.
+These are only valid if you use GTFS-Realtime and have imported Service Alert data.
+
+```js
+import { getServiceAlerts } from 'gtfs';
+
+// Get service alerts
+getServiceAlerts();
+```
+
+### getTripUpdates(query, fields, sortBy)
+
+Queries trip alerts and returns a promise. The result of the promise is an array of trip updates.
+These are only valid if you use GTFS-Realtime and have imported Trip Update data.
+
+```js
+import { getTripUpdates } from 'gtfs';
+
+// Get all trip updates
+getTripUpdates();
+```
+
+### getStopTimesUpdates(query, fields, sortBy)
+
+Queries stop times updates and returns a promise. The result of the promise is an array of stop times updates.
+These are only valid if you use GTFS-Realtime and have imported Trip Update data.
+
+```js
+import { getStopTimesUpdates } from 'gtfs';
+
+// Get all stop times updates
+getStopTimesUpdates();
+```
+
+### getVehiclePositions(query, fields, sortBy)
+
+Queries vehicle positions and returns a promise. The result of the promise is an array of vehicle location data.
+These are only valid if you use GTFS-Realtime and have imported Vehicle Position data.
+
+```js
+import { getVehiclePositions } from 'gtfs';
+
+// Get all vehicle position data
+getVehiclePositions();
+```
+
+### advancedQuery(table, advancedQueryOptions)
+
+Queries the database in a simple manner with support for table joins and custom tables. Returns a promise.
+The result of the promise is an array the selected data. Example shows joining stop_times with trips.
+Used for advanced scenarios.
+
+```js
+import { advancedQuery } from 'gtfs';
+
+const advancedQueryOptions = {
+  query: {
+    'stop_times.trip_id': tripId,
+  },
+  fields: ['stop_times.trip_id', 'arrival_time'],
+  join: [
+    {
+      type: 'INNER',
+      table: 'trips',
+      on: 'stop_times.trip_id=trips.trip_id',
+    },
+  ],
+};
+// Perform a custom query
+advancedQuery('stop_times', advancedQueryOptions);
+```
+
+### runRawQuery(query)
+
+Queries the database using a raw sql statement. Returns a promise.
+The result of the promise is an array the selected data.
+
+```js
+import { runRawQuery } from 'gtfs';
+
+// Perform a raw query
+runRawQuery('SELECT * FROM stop_times WHERE stop_sequence="1"');
+```
+
+### execRawQuery(query)
+
+Executes a statement. Returns a promise containing the result of the execute.
+
+```js
+import { execRawQuery } from 'gtfs';
+
+// Purge trips table
+execRawQuery('DELETE FROM trips');
+```
+
 ## Contributing
 
 Pull requests are welcome, as is feedback and [reporting issues](https://github.com/blinktaginc/node-gtfs/issues).

package/bin/gtfsrealtime-update.js

@@ -0,0 +1,32 @@
+#!/usr/bin/env node
+
+import yargs from 'yargs';
+import { hideBin } from 'yargs/helpers';
+
+import { getConfig } from '../lib/file-utils.js';
+import { formatError } from '../lib/log-utils.js';
+import { updateGtfsRealtime } from '../lib/gtfs.js';
+
+const { argv } = yargs(hideBin(process.argv))
+  .usage('Usage: $0 --configPath ./config.json')
+  .help()
+  .option('c', {
+    alias: 'configPath',
+    describe: 'Path to config file',
+    type: 'string',
+  });
+
+const handleError = (error) => {
+  const text = error || 'Unknown Error';
+  process.stdout.write(`\n${formatError(text)}\n`);
+  console.error(error);
+  process.exit(1);
+};
+
+const setupImport = async () => {
+  const config = await getConfig(argv);
+  await updateGtfsRealtime(config);
+  process.exit();
+};
+
+setupImport().catch(handleError);

package/config-sample-rtupdates.json

@@ -0,0 +1,28 @@
+{
+  "agencies": [
+    {
+      "url": "http://www.bart.gov/dev/schedules/google_transit.zip",
+      "exclude": ["shapes"],
+      "realtimeUrls": [
+        "https://opendata.samtrafiken.se/gtfs-rt/sl/TripUpdates.pb?key=yourkey",
+        "https://opendata.samtrafiken.se/gtfs-rt/sl/VehiclePositions.pb?key=yourkey",
+        "https://opendata.samtrafiken.se/gtfs-rt/sl/ServiceAlerts.pb?key=yourkey"
+      ],
+      "realtimeHeaders": {
+        "X-CustomHeader": "MyCustomHeader"
+      },
+      "headers": {
+        "X-CustomHeader": "MyCustomHeader"
+      }
+    },
+    {
+      "path": "/path/to/gtfs.zip",
+      "exclude": ["stop_times", "shapes"]
+    }
+  ],
+  "csvOptions": {
+    "skip_lines_with_error": true
+  },
+  "sqlitePath": "/tmp/gtfs",
+  "exportPath": "~/path/to/export/gtfs"
+}

package/lib/advancedQuery.js

@@ -0,0 +1,45 @@
+import sqlString from 'sqlstring-sqlite';
+
+import { getDb } from './db.js';
+
+import {
+  formatOrderByClause,
+  formatSelectClause,
+  formatWhereClauses,
+  formatJoinClause,
+} from './utils.js';
+
+/*
+ * Returns an array of all agencies that match the query parameters.
+ */
+export async function advancedQuery(table, advancedQueryOptions) {
+  const defaultOptions = {
+    query: {},
+    fields: [],
+    orderBy: [],
+    join: [],
+    options: {},
+  };
+  const queryOptions = { ...defaultOptions, ...advancedQueryOptions };
+
+  const db = queryOptions.options.db ?? (await getDb());
+  const tableName = sqlString.escapeId(table);
+  const selectClause = formatSelectClause(queryOptions.fields);
+  const whereClause = formatWhereClauses(queryOptions.query);
+  const joinClause = formatJoinClause(queryOptions.join);
+  const orderByClause = formatOrderByClause(queryOptions.orderBy);
+  const sqlStatement = `${selectClause} FROM ${tableName} ${joinClause} ${whereClause} ${orderByClause};`;
+  return db.all(sqlStatement).catch((error) => {
+    console.log(error);
+  });
+}
+
+export async function runRawQuery(sql, options = {}) {
+  const db = options.db ?? (await getDb());
+  return db.all(sql);
+}
+
+export async function execRawQuery(sql, options = {}) {
+  const db = options.db ?? (await getDb());
+  return db.exec(sql);
+}

package/lib/db.js

@@ -1,4 +1,4 @@
-import sqlite3 from '@vscode/sqlite3';
+import sqlite3 from 'sqlite3';
 import { open } from 'sqlite';
 import { setDefaultConfig } from './utils.js';
 const dbs = {};

package/lib/gtfs/shapes.js

@@ -101,11 +101,12 @@ export async function getShapesAsGeoJSON(query = {}, options = {}) {
         (agency) => agency.agency_id === route.agency_id
       );
 
-      const routeProperties = {
+      const geojsonProperties = {
         agency_name: agency ? agency.agency_name : undefined,
+        shape_id: query.shape_id,
         ...route,
       };
-      features.push(...shapesToGeoJSONFeatures(shapes, routeProperties));
+      features.push(...shapesToGeoJSONFeatures(shapes, geojsonProperties));
     })
   );
 

package/lib/gtfs/stops.js

@@ -47,12 +47,14 @@ export async function getStops(
     'trip_id',
     'service_id',
     'direction_id',
+    'shape_id',
   ]);
   const tripQuery = pick(query, [
     'route_id',
     'trip_id',
     'service_id',
     'direction_id',
+    'shape_id',
   ]);
 
   const whereClauses = Object.entries(stopQuery).map(([key, value]) =>

package/lib/gtfs-realtime/service-alerts.js

@@ -0,0 +1,32 @@
+import sqlString from 'sqlstring-sqlite';
+
+import { getDb } from '../db.js';
+
+import {
+  formatOrderByClause,
+  formatSelectClause,
+  formatWhereClauses,
+} from '../utils.js';
+import serviceAlerts from '../../models/gtfs-realtime/service-alerts.js';
+import serviceAlertTargets from '../../models/gtfs-realtime/service-alert-targets.js';
+
+/*
+ * Returns an array of all agencies that match the query parameters.
+ */
+export async function getServiceAlerts(
+  query = {},
+  fields = [],
+  orderBy = [],
+  options = {}
+) {
+  const db = options.db ?? (await getDb());
+  const tableName = sqlString.escapeId(serviceAlerts.filenameBase);
+  const joinTable = sqlString.escapeId(serviceAlertTargets.filenameBase);
+  const selectClause = formatSelectClause(fields);
+  const whereClause = formatWhereClauses(query);
+  const orderByClause = formatOrderByClause(orderBy);
+
+  return db.all(
+    `${selectClause} FROM ${tableName} INNER JOIN ${joinTable} ON ${tableName}.id=${joinTable}.alert_id ${whereClause} ${orderByClause};`
+  );
+}

package/lib/gtfs-realtime/stop-times-updates.js

@@ -0,0 +1,30 @@
+import sqlString from 'sqlstring-sqlite';
+
+import { getDb } from '../db.js';
+
+import {
+  formatOrderByClause,
+  formatSelectClause,
+  formatWhereClauses,
+} from '../utils.js';
+import stopTimeUpdates from '../../models/gtfs-realtime/stop-times-updates.js';
+
+/*
+ * Returns an array of all agencies that match the query parameters.
+ */
+export async function getStopTimesUpdates(
+  query = {},
+  fields = [],
+  orderBy = [],
+  options = {}
+) {
+  const db = options.db ?? (await getDb());
+  const tableName = sqlString.escapeId(stopTimeUpdates.filenameBase);
+  const selectClause = formatSelectClause(fields);
+  const whereClause = formatWhereClauses(query);
+  const orderByClause = formatOrderByClause(orderBy);
+
+  return db.all(
+    `${selectClause} FROM ${tableName} ${whereClause} ${orderByClause};`
+  );
+}

package/lib/gtfs-realtime/trip-updates.js

@@ -0,0 +1,30 @@
+import sqlString from 'sqlstring-sqlite';
+
+import { getDb } from '../db.js';
+
+import {
+  formatOrderByClause,
+  formatSelectClause,
+  formatWhereClauses,
+} from '../utils.js';
+import tripUpdates from '../../models/gtfs-realtime/trip-updates.js';
+
+/*
+ * Returns an array of all agencies that match the query parameters.
+ */
+export async function getTripUpdates(
+  query = {},
+  fields = [],
+  orderBy = [],
+  options = {}
+) {
+  const db = options.db ?? (await getDb());
+  const tableName = sqlString.escapeId(tripUpdates.filenameBase);
+  const selectClause = formatSelectClause(fields);
+  const whereClause = formatWhereClauses(query);
+  const orderByClause = formatOrderByClause(orderBy);
+
+  return db.all(
+    `${selectClause} FROM ${tableName} ${whereClause} ${orderByClause};`
+  );
+}

package/lib/gtfs-realtime/vehicle-positions.js

@@ -0,0 +1,30 @@
+import sqlString from 'sqlstring-sqlite';
+
+import { getDb } from '../db.js';
+
+import {
+  formatOrderByClause,
+  formatSelectClause,
+  formatWhereClauses,
+} from '../utils.js';
+import vehiclePositions from '../../models/gtfs-realtime/vehicle-positions.js';
+
+/*
+ * Returns an array of all agencies that match the query parameters.
+ */
+export async function getVehiclePositions(
+  query = {},
+  fields = [],
+  orderBy = [],
+  options = {}
+) {
+  const db = options.db ?? (await getDb());
+  const tableName = sqlString.escapeId(vehiclePositions.filenameBase);
+  const selectClause = formatSelectClause(fields);
+  const whereClause = formatWhereClauses(query);
+  const orderByClause = formatOrderByClause(orderBy);
+
+  return db.all(
+    `${selectClause} FROM ${tableName} ${whereClause} ${orderByClause};`
+  );
+}

package/lib/gtfs.js

@@ -1,5 +1,5 @@
 // GTFS import script
-import importGtfs from './import.js';
+import { importGtfs, updateGtfsRealtime } from './import.js';
 
 // GTFS export script
 import exportGtfs from './export.js';
@@ -31,6 +31,7 @@ import { getTimetableStopOrders } from './non-standard/timetable-stop-order.js';
 import { getTimetablePages } from './non-standard/timetable-pages.js';
 import { getTimetableNotes } from './non-standard/timetable-notes.js';
 import { getTimetableNotesReferences } from './non-standard/timetable-notes-references.js';
+import { getTripsDatedVehicleJourneys } from './non-standard/trips-dated-vehicle-journey.js';
 
 // GTFS-ride Filenames
 import { getBoardAlights } from './gtfs-ride/board-alights.js';
@@ -39,9 +40,18 @@ import { getRiderTrips } from './gtfs-ride/rider-trips.js';
 import { getRiderships } from './gtfs-ride/riderships.js';
 import { getTripCapacities } from './gtfs-ride/trip-capacities.js';
 
+// GTFS-Realtime Filenames
+import { getStopTimesUpdates } from './gtfs-realtime/stop-times-updates.js';
+import { getTripUpdates } from './gtfs-realtime/trip-updates.js';
+import { getVehiclePositions } from './gtfs-realtime/vehicle-positions.js';
+import { getServiceAlerts } from './gtfs-realtime/service-alerts.js';
+
 // Expose database connection
 import { openDb, closeDb, getDb } from './db.js';
 
+// Advanced Query Support or Raw Query Support
+import { runRawQuery, execRawQuery, advancedQuery } from './advancedQuery.js';
+
 const _importGtfs = importGtfs;
 export { _importGtfs as importGtfs };
 const _exportGtfs = exportGtfs;
@@ -123,6 +133,9 @@ export { _getTimetableNotes as getTimetableNotes };
 const _getTimetableNotesReferences = getTimetableNotesReferences;
 export { _getTimetableNotesReferences as getTimetableNotesReferences };
 
+const _getTripsDatedVehicleJourneys = getTripsDatedVehicleJourneys;
+export { _getTripsDatedVehicleJourneys as getTripsDatedVehicleJourneys };
+
 const _getBoardAlights = getBoardAlights;
 export { _getBoardAlights as getBoardAlights };
 
@@ -138,9 +151,31 @@ export { _getRiderships as getRiderships };
 const _getTripCapacities = getTripCapacities;
 export { _getTripCapacities as getTripCapacities };
 
+const _updateGtfsRealtime = updateGtfsRealtime;
+export { _updateGtfsRealtime as updateGtfsRealtime };
+
+const _getStopTimesUpdates = getStopTimesUpdates;
+export { _getStopTimesUpdates as getStopTimesUpdates };
+
+const _getTripUpdates = getTripUpdates;
+export { _getTripUpdates as getTripUpdates };
+
+const _getVehiclePositions = getVehiclePositions;
+export { _getVehiclePositions as getVehiclePositions };
+
+const _getServiceAlerts = getServiceAlerts;
+export { _getServiceAlerts as getServiceAlerts };
+
 const _openDb = openDb;
 export { _openDb as openDb };
 const _closeDb = closeDb;
 export { _closeDb as closeDb };
 const _getDb = getDb;
 export { _getDb as getDb };
+
+const _advancedQuery = advancedQuery;
+export { _advancedQuery as advancedQuery };
+const _runRawQuery = runRawQuery;
+export { _runRawQuery as runRawQuery };
+const _execRawQuery = execRawQuery;
+export { _execRawQuery as execRawQuery };