@golemio/bicycle-counters 1.0.8 → 1.0.9

package/docs/implementation_documentation.md

@@ -0,0 +1,248 @@
+# Implementační dokumentace modulu *bicycle-counters*
+
+## Záměr
+
+Modul slouží k ukládání informací o sčítačích kol v Praze.
+
+Dostupní poskytovatelé:
+- Camea
+- EcoCounter
+
+
+## Vstupní data
+
+### Data aktivně stahujeme
+
+Měření Camea senzorů jsou validovány a přepočítány každý den o 05:00 (UTC) a můžou být zpětně změněna.
+Data z Camea senzorů obsahují informaci o teplotě.
+
+#### *Camea*
+
+##### Lokalita
+
+- zdroj dat
+  - url: `config.datasources.BicycleCountersCamea`
+  - parametry dotazu: žádné
+- formát dat
+  - protokol: http
+  - datový typ: json
+  - [validační schéma: datasourceCameaJsonSchema](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/schema-definitions/index.ts)
+  - [příklad vstupních dat](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/test/integration-engine/data/bicyclecounters-camea-datasource.json)
+- frekvence stahování
+  - cron definice:
+    - `cron.dataplatform.bicyclecounters.refreshCameaDataLastXHoursInDB` (`0 */5 * * * *`)
+    - `cron.dataplatform.bicyclecounters.refreshCameaDataPreviousDayInDB` (`0 0 5 * * *`)
+- název rabbitmq fronty
+  - `dataplatform.bicyclecounters.refreshCameaDataLastXHoursInDB`
+  - `dataplatform.bicyclecounters.refreshCameaDataPreviousDayInDB`
+  - `dataplatform.bicyclecounters.refreshCameaDataSpecificDayInDB`
+
+##### Měření
+
+- zdroj dat
+    - url: `config.datasources.BicycleCountersCameaMeasurements`
+    - parametry dotazu: žádné
+- formát dat
+    - protokol: http
+    - datový typ: json
+    - [validační schéma: datasourceCameaMeasurementsJsonSchema](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/schema-definitions/index.ts)
+    - [příklad vstupních dat](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/test/integration-engine/data/bicyclecounters-camea-measurements-datasource.json)
+- frekvence stahování
+    - cron definice:
+        - stahuje se spolu s lokalitou
+- název rabbitmq fronty
+    - `dataplatform.bicyclecounters.updateCamea`
+
+#### *EcoCounter*
+
+##### Lokalita
+
+- zdroj dat
+    - url: `config.datasources.BicycleCountersEcoCounter`
+    - parametry dotazu: Bearer `config.datasources.CountersEcoCounterTokens.PRAHA`
+- formát dat
+    - protokol: http
+    - datový typ: json
+    - [validační schéma: datasourceEcoCounterJsonSchema](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/schema-definitions/index.ts)
+    - [příklad vstupních dat](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/test/integration-engine/data/bicyclecounters-ecocounter-datasource.json)
+- frekvence stahování
+    - cron definice:
+        - `cron.dataplatform.bicyclecounters.refreshEcoCounterDataInDB` (`0 */5 * * * *`)
+- název rabbitmq fronty
+    - `dataplatform.bicyclecounters.refreshEcoCounterDataInDB`
+
+##### Měření
+
+- zdroj dat
+    - url: `config.datasources.BicycleCountersEcoCounterMeasurements`
+    - parametry dotazu: Bearer `config.datasources.CountersEcoCounterTokens.PRAHA`
+- formát dat
+    - protokol: http
+    - datový typ: json
+    - [validační schéma: datasourceEcoCounterMeasurementsJsonSchema](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/schema-definitions/index.ts)
+    - [příklad vstupních dat](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/test/integration-engine/data/bicyclecounters-ecocounter-measurements-datasource.json)
+- frekvence stahování
+    - cron definice:
+        - stahuje se spolu s lokalitou
+- název rabbitmq fronty
+    - `dataplatform.bicyclecounters.updateEcoCounter`
+
+## Zpracování dat / transformace
+
+Popis transformace a případného obohacení dat. Zaměřeno hlavně na workery a jejich metody.
+
+### Worker BicycleCountersWorker
+
+#### refreshCameaDataLastXHoursInDB
+
+- vstupní rabbitmq fronta
+  - `dataplatform.bicyclecounters.refreshCameaDataLastXHoursInDB`
+  - bez parametrů
+- závislé fronty (do kterých jsou odesílány zprávy z metody workeru)
+  - `dataplatform.bicyclecounters.updateCamea`
+  - parametry:
+  ```
+  { id: <vendor_id>, duration: CameaRefreshDurations.last3Hours }
+  ```
+- datové zdroje
+  - dataSourceCamea
+- transformace
+  - [CameaTransformation](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/integration-engine/CameaTransformation.ts) - mapování pro `locationsModel` a `directionsModel`
+- data modely
+  - locationsModel -> (schéma public) `bicyclecounters_locations`
+  - directionsModel -> (schéma public) `bicyclecounters_directions`
+
+#### refreshCameaDataPreviousDayInDB
+
+- vstupní rabbitmq fronta
+    - `dataplatform.bicyclecounters.refreshCameaDataPreviousDayInDB`
+    - bez parametrů
+- závislé fronty (do kterých jsou odesílány zprávy z metody workeru)
+    - `dataplatform.bicyclecounters.updateCamea`
+    - parametry:
+  ```
+  { id: <vendor_id>, duration: CameaRefreshDurations.previousDay }
+  ```
+- datové zdroje/transformace/data modely stejně jako `refreshCameaDataLastXHoursInDB`
+
+#### refreshCameaDataSpecificDayInDB
+
+bez cronu, k zpětné opravě dat
+
+- vstupní rabbitmq fronta
+    - `dataplatform.bicyclecounters.refreshCameaDataSpecificDayInDB`
+    - bez parametrů
+- závislé fronty (do kterých jsou odesílány zprávy z metody workeru)
+    - `dataplatform.bicyclecounters.updateCamea`
+    - parametry:
+  ```
+  { id: <vendor_id>, duration: CameaRefreshDurations.specificDay, date: <date, f.e. 2020-12-14> }
+  ```
+- datové zdroje/transformace/data modely stejně jako `refreshCameaDataLastXHoursInDB`
+
+#### updateCamea
+
+- vstupní rabbitmq fronta
+    - `dataplatform.bicyclecounters.updateCamea`
+    - parametry:
+  ```
+  { id: <vendor_id>, duration: CameaRefreshDurations.specificDay, date: <date, f.e. 2020-12-14> }
+  ```
+- datové zdroje
+    - dataSourceCameaMeasurements
+- transformace
+    - [CameaMeasurementsTransformation](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/integration-engine/https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/integration-engine/CameaMeasurementsTransformation.ts) mapování pro `detectionsModel` a `temperaturesModel`
+- data modely
+    - detectionsModel -> (schéma public) `bicyclecounters_detections`
+    - temperaturesModel -> (schéma public) `bicyclecounters_temperatures`
+
+#### refreshEcoCounterDataInDB
+
+- vstupní rabbitmq fronta
+    - `dataplatform.bicyclecounters.refreshEcoCounterDataInDB`
+    - bez parametrů
+- závislé fronty (do kterých jsou odesílány zprávy z metody workeru)
+    - `dataplatform.bicyclecounters.updateEcoCounter`
+    - parametry:
+  ```
+  { category: "bicycle", directions_id: <id>, id: <vendor_id>, locations_id: <locations_id> }
+  ```
+- datové zdroje
+    - dataSourceEcoCounter
+- transformace
+    - [EcoCounterTransformation](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/integration-engine/EcoCounterTransformation.ts) - mapování pro `locationsModel` a `directionsModel`
+- data modely
+    - locationsModel -> (schéma public) `bicyclecounters_locations`
+    - directionsModel -> (schéma public) `bicyclecounters_directions`
+
+#### updateEcoCounter
+
+- vstupní rabbitmq fronta
+    - `dataplatform.bicyclecounters.updateEcoCounter`
+  - parametry:
+  ```
+  { category: "bicycle", directions_id: <id>, id: <vendor_id>, locations_id: <locations_id> }
+  ```
+- datové zdroje
+    - dataSourceEcoCounterMeasurements
+- transformace
+    - [EcoCounterMeasurementsTransformation](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/integration-engine/https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/src/integration-engine/EcoCounterMeasurementsTransformation.ts) mapování pro `detectionsModel` a `temperaturesModel`
+- data modely
+    - detectionsModel -> (schéma public) `bicyclecounters_detections`
+    - temperaturesModel -> (schéma public) `bicyclecounters_temperatures`
+
+
+## Uložení dat
+
+Popis ukládání dat.
+
+### Obecné
+
+- typ databáze
+  - PSQL
+- databázové schéma
+  - ![bicycle-counters er diagram](assets/bicycle-counters_erd.png)
+- retence dat
+  - `locationsModel` a `directionsModel` update
+  - `detectionsModel` a `temperaturesModel` historizace bez promazávání
+
+### *Název datového modelu*
+
+- schéma a název tabulky / kolekce
+- struktura tabulky / kolekce
+  - odkaz na schéma, případně na migrační skripty
+- ukázka dat
+
+
+## Output API
+
+### Obecné
+
+- OpenAPI v3 dokumentace
+  - [odkaz na dokumentaci](https://gitlab.com/operator-ict/golemio/code/modules/bicycle-counters/-/blob/development/docs/openapi.yaml)
+- veřejné / neveřejné endpointy
+  - api je veřejné
+- postman kolekce
+  - TBD
+
+#### */bicyclecounters*
+
+- zdrojové tabulky
+  - `bicyclecounters_locations`
+  - `bicyclecounters_directions`
+
+#### */bicyclecounters/detections*
+
+- zdrojové tabulky
+  - `bicyclecounters_detections`
+
+- (volitelně) nestandardní url a query parametry
+  - `aggregate: boolean` umožňuje agregaci dle `directions_id`
+
+#### */bicyclecounters/temperatures*
+
+- zdrojové tabulky
+  - `bicyclecounters_temperatures`
+
+- (volitelně) nestandardní url a query parametry
+  - `aggregate: boolean` umožňuje agregaci dle `locations_id`

package/docs/openapi.yaml

@@ -0,0 +1,400 @@
+openapi: 3.0.1
+info:
+  title: Bicycle Counter
+  description: |
+    | Vendor | Vendor id prefix | Measurement time aggregation | Temperature availability |
+    | - | - | - | - |
+    | Camea | `camea-` | 5 minutes | available |
+    | EcoCounter | `ecoCounter-` | 15 minutes | not available |
+    ⚠️ Measurements of vendor Camea are every day at 5:00am (UTC) validated and recalculated and can change then.
+  version: 1.0.0
+tags:
+- name: Bicycle Counter
+paths:
+  /bicyclecounters/:
+    get:
+      tags:
+      - Bicycle Counter
+      summary: GET All Bicycle Counters
+      parameters:
+      - name: latlng
+        in: query
+        description: Sorting by location (Latitude and Longitude separated by comma,
+          latitude first).
+        schema:
+          type: string
+          example: 50.124935,14.457204
+      - name: range
+        in: query
+        description: Filter by distance from latlng in meters (range query). Depends
+          on the latlng parameter.
+        schema:
+          type: number
+          example: 5000
+      - name: limit
+        in: query
+        description: Limits number of retrieved items. The maximum is 10000 (default
+          value).
+        schema:
+          type: number
+          example: 10
+      - name: offset
+        in: query
+        description: Number of the first items that are skipped.
+        schema:
+          type: number
+          example: 0
+      - name: x-access-token
+        in: header
+        description: e.g. YOUR_ACCESS_TOKEN
+        schema:
+          type: string
+          example: YOUR_ACCESS_TOKEN
+      responses:
+        200:
+          description: OK
+          content:
+            application/json; charset=utf-8:
+              schema:
+                required:
+                - type
+                type: object
+                properties:
+                  type:
+                    type: string
+                  features:
+                    type: array
+                    items:
+                      $ref: '#/components/schemas/FeaturePoint'
+                example:
+                  type: FeatureCollection
+                  features:
+                  - geometry:
+                      type: Point
+                      coordinates:
+                      - 14.4633
+                      - 50.07827
+                    properties:
+                      id: camea-BC_AT-STLA
+                      name: Altetická
+                      route: A140
+                      updated_at: "2019-05-18T07:38:37.000Z"
+                      directions:
+                      - id: BC_AT_ST
+                        name: Strahov
+                    type: Feature
+        404:
+          description: Not found
+          content:
+            application/json; charset=utf-8:
+              schema:
+                $ref: '#/components/schemas/Error'
+              example:
+                error_message: Not Found
+                error_status: 404
+        403:
+          description: Forbidden
+          content:
+            application/json; charset=utf-8:
+              schema:
+                $ref: '#/components/schemas/Error'
+              example:
+                error_message: Forbidden
+                error_status: 403
+  /bicyclecounters/detections:
+    get:
+      tags:
+      - Bicycle Counter
+      summary: GET Bicycle Counters Detections
+      parameters:
+      - name: limit
+        in: query
+        description: Limits number of retrieved items. The maximum is 10000 (default
+          value).
+        schema:
+          type: number
+          example: 10
+      - name: offset
+        in: query
+        description: Number of the first items that are skipped.
+        schema:
+          type: number
+          example: 0
+      - name: from
+        in: query
+        description: Date in ISO8601, limits data measured from this datetime
+        schema:
+          type: string
+          example: "2019-05-16T04:27:58.000Z"
+      - name: to
+        in: query
+        description: Date in ISO8601, limits data measured up until this datetime
+        schema:
+          type: string
+          example: "2019-05-18T04:27:58.000Z"
+      - name: aggregate
+        in: query
+        description: sum aggregated by directions_id is returned
+        schema:
+          type: boolean
+          example: true
+      - name: id
+        in: query
+        description: direction ids
+        schema:
+          type: string
+          example: camea-BC_ZA-BO
+      - name: x-access-token
+        in: header
+        description: e.g. YOUR_ACCESS_TOKEN
+        schema:
+          type: string
+          example: YOUR_ACCESS_TOKEN
+      responses:
+        200:
+          description: OK
+          content:
+            application/json; charset=utf-8:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/BicycleCounterDetections'
+                example:
+                - id: camea-BC_ZA-BO
+                  value: 3
+                  value_pedestrians: 7
+                  locations_id: camea-BC_PN-VYBR
+                  measured_from: "2019-05-18T07:38:37.000Z"
+                  measured_to: "2019-05-18T07:38:37.000Z"
+                  measurement_count: 1
+        404:
+          description: Not found
+          content:
+            application/json; charset=utf-8:
+              schema:
+                $ref: '#/components/schemas/Error'
+              example:
+                error_message: Not Found
+                error_status: 404
+        403:
+          description: Forbidden
+          content:
+            application/json; charset=utf-8:
+              schema:
+                $ref: '#/components/schemas/Error'
+              example:
+                error_message: Forbidden
+                error_status: 403
+  /bicyclecounters/temperatures:
+    get:
+      tags:
+      - Bicycle Counter
+      summary: GET Bicycle Counters Temperatures
+      parameters:
+      - name: limit
+        in: query
+        description: Limits number of retrieved items. The maximum is 10000 (default
+          value).
+        schema:
+          type: number
+          example: 10
+      - name: offset
+        in: query
+        description: Number of the first items that are skipped.
+        schema:
+          type: number
+          example: 0
+      - name: from
+        in: query
+        description: Date in ISO8601, limits data measured from this datetime
+        schema:
+          type: string
+          example: "2019-05-16T04:27:58.000Z"
+      - name: to
+        in: query
+        description: Date in ISO8601, limits data measured up until this datetime
+        schema:
+          type: string
+          example: "2019-05-18T04:27:58.000Z"
+      - name: aggregate
+        in: query
+        description: average aggregated by locations_id is returned
+        schema:
+          type: boolean
+          example: true
+      - name: id
+        in: query
+        description: location ids
+        schema:
+          type: array
+          items:
+            type: string
+            example: camea-BC_ZA-KLBO
+      - name: x-access-token
+        in: header
+        description: e.g. YOUR_ACCESS_TOKEN
+        schema:
+          type: string
+          example: YOUR_ACCESS_TOKEN
+      responses:
+        200:
+          description: OK
+          content:
+            application/json; charset=utf-8:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/BicycleCounterTemperatures'
+                example:
+                - id: camea-BC_AT-STLA
+                  value: 3
+                  measured_from: "2019-05-18T07:38:37.000Z"
+                  measured_to: "2019-05-18T07:38:37.000Z"
+                  measurement_count: 1
+        404:
+          description: Not found
+          content:
+            application/json; charset=utf-8:
+              schema:
+                $ref: '#/components/schemas/Error'
+              example:
+                error_message: Not Found
+                error_status: 404
+        403:
+          description: Forbidden
+          content:
+            application/json; charset=utf-8:
+              schema:
+                $ref: '#/components/schemas/Error'
+              example:
+                error_message: Forbidden
+                error_status: 403
+components:
+  schemas:
+    Point:
+      type: array
+      items:
+        type: number
+      minItems: 2
+      maxItems: 2
+      example:
+        - 14.4633
+        - 50.07827
+    Error:
+      title: Error
+      required:
+      - error_message
+      - error_status
+      type: object
+      properties:
+        error_message:
+          type: string
+          example: Forbidden
+        error_status:
+          type: number
+          example: 403
+    FeaturePoint:
+      title: Feature Point
+      required:
+      - geometry
+      - properties
+      - type
+      type: object
+      properties:
+        geometry:
+          required:
+          - coordinates
+          - type
+          type: object
+          properties:
+            type:
+              type: string
+            coordinates:
+              $ref: '#/components/schemas/Point'
+        properties:
+          $ref: '#/components/schemas/BicycleCounter'
+        type:
+          type: string
+    BicycleCounter:
+      title: Bicycle Counter
+      required:
+      - id
+      - name
+      type: object
+      properties:
+        id:
+          type: string
+          example: camea-BC_AT-STLA
+        name:
+          type: string
+          example: Altetická
+        route:
+          type: string
+          example: A140
+        updated_at:
+          type: string
+          example: "2019-05-18T07:38:37.000Z"
+        directions:
+          type: array
+          items:
+            $ref: '#/components/schemas/BicycleCounterDirection'
+    BicycleCounterDetections:
+      title: Bicycle Counter Detections
+      type: object
+      properties:
+        id:
+          type: string
+          description: direction_id
+          example: camea-BC_ZA-BO
+        value:
+          type: number
+          example: 3
+        value_pedestrians:
+          type: number
+          example: 7
+        locations_id:
+          type: string
+          example: camea-BC_PN-VYBR
+        measured_from:
+          type: string
+          example: 2019-05-18T07:38:37.000Z
+        measured_to:
+          type: string
+          example: 2019-05-18T07:38:37.000Z
+        measurement_count:
+          type: number
+          example: 1
+    BicycleCounterDirection:
+      title: Bicycle Counter Direction
+      required:
+      - id
+      type: object
+      properties:
+        id:
+          type: string
+          description: direction ID
+          example: BC_AT_ST
+        name:
+          type: string
+          description: direction name
+          example: Strahov
+    BicycleCounterTemperatures:
+      title: Bicycle Counter Temperatures
+      type: object
+      properties:
+        id:
+          type: string
+          description: location_id
+          example: camea-BC_AT-STLA
+        value:
+          type: number
+          example: 3
+        measured_from:
+          type: string
+          example: "2019-05-18T07:38:37.000Z"
+        measured_to:
+          type: string
+          example: "2019-05-18T07:38:37.000Z"
+        measurement_count:
+          type: number
+          example: 1

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@golemio/bicycle-counters",
-    "version": "1.0.8",
+    "version": "1.0.9",
     "description": "Golemio Bicycle Counters Module",
     "main": "dist/index.js",
     "types": "dist/index.d.ts",
@@ -27,9 +27,9 @@
     "devDependencies": {
         "@commitlint/cli": "^11.0.0",
         "@commitlint/config-conventional": "^11.0.0",
-        "@golemio/core": "1.2.13",
-        "@golemio/eslint-config": "^1.0.2",
-        "@golemio/schema-definitions": "2.0.37",
+        "@golemio/core": "1.2.14",
+        "@golemio/eslint-config": "^1.1.0",
+        "@golemio/schema-definitions": "2.0.39",
         "@ovos-media/ts-transform-paths": "^1.7.18-1",
         "@types/chai": "4.2.3",
         "@types/chai-as-promised": "7.1.2",
@@ -42,12 +42,12 @@
         "chai-as-promised": "7.1.1",
         "cross-env": "^7.0.3",
         "dotenv": "^8.2.0",
-        "eslint": "^7.17.0",
+        "eslint": "^8.1.1",
         "husky": "^4.3.7",
         "mocha": "^8.2.1",
         "npm-run-all": "^4.1.5",
         "nyc": "^15.1.0",
-        "prettier": "^2.2.1",
+        "prettier": "^2.5.1",
         "pretty-quick": "^3.1.0",
         "rimraf": "^3.0.2",
         "sinon": "^9.2.3",