atlas_engine 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b097368a07bb9496c6c42c1ef98c785dc25ef5b3aa9d4a6bd97980ac372a8769
-  data.tar.gz: b1383521f226975e0e48b2bd9ea03e3dc45b6903e66e245b7efff37e889b51cd
+  metadata.gz: 4f78501b09c1f8bfeae5d8a5be2ff3b83d7f68f689e66294f704f3c3bcf5830b
+  data.tar.gz: d1100e782d95c89565be662da85ba69788fa457bbefcd11ccd8082c356086df2
 SHA512:
-  metadata.gz: eb1337d027e93333a0f96ced70070a7c540b6d559d1e23490a2580965adc6cc87fd87bb0cb558962a32d306571bf154a70ba9edd36dfdbecbfd9f0f1372e1bb7
-  data.tar.gz: cef1341af8b655e7eb701944308ad7c00ad1047d33c0f0a8866efe580fb0995736e4d8b5aa62d9dcd58daec468d3da7ef9ef0778bad4999e18b6a00da3f328dd
+  metadata.gz: 7d2b8abdf8ae247fcb95b10a0e7f301deea99ee2505f8817c85a8bf10459e3cdedfd7a631738071cf018db7545ca0b2506ec518f31c1e5ba1f507a49915d2f25
+  data.tar.gz: d7b2177df1ca73d45a37993dfdea005d92fc3705ef422c0a1b7e90285825880f4d82f7ae3fff47d8f425abf3a2865636ec5bd8e546a2e60c113de19aeeba8df0

data/README.md

@@ -1,12 +1,164 @@
 # Atlas Engine
 
-This is a rails engine that is meant to provide end-to-end address validation for rails apps
+Atlas Engine is a rails engine that provides a global end-to-end address validation API for rails apps.
+
+The validation API is powered by GraphQL, an example request and explanation of the parameters and response follows:
+
+```graphql
+query validation {
+  validation(
+    address: {
+      address1: "151 O'Connor St"
+      address2: ""
+      city: "Ottawa"
+      provinceCode: "ON"
+      countryCode: CA
+      zip: "K2P 2L8"
+    }
+    locale: "en"
+    matchingStrategy: LOCAL
+  ) {
+    validationScope
+    concerns {
+      code
+      fieldNames
+      suggestionIds
+      message
+    }
+    suggestions {
+      address1
+      address2
+      city
+      provinceCode
+      zip
+    }
+  }
+}
+```
+
+Response:
+
+```json
+{
+  "data": {
+    "validation": {
+      "validationScope": [
+        "country_code",
+        "province_code",
+        "zip",
+        "city",
+        "address1"
+      ],
+      "concerns": [],
+      "suggestions": []
+    }
+  }
+}
+```
+
+**Address:** The raw input for each address line that is to be validated. Requirements for each field's format and even
+presence or absence differs per country.
+
+**Locale:** The language in which to render any messages in the validation API response.
+
+**MatchingStrategy:** The strategy used to evaluate the validity of the address input. Out of the box, Atlas Engine
+supports three different matching strategies: `local`, `es`, and `es_street`.
+  - `local` matching uses the [worldwide](https://github.com/Shopify/worldwide) gem to provide the most basic level of
+  address validation. This may include simple errors (required fields not populated) or more advanced errors (province
+  not belonging to the country, zip code not belonging to the province). This level of matching does not require
+  [ingestion](#ingestion) of country data to work, but the level of support and suggestions it can provide in its
+  responses is minimal.
+  - `es` matching uses data indexed in elasticsearch via our [ingestion](#ingestion) process to validate the city,
+  province, country, and zip code fields of the input address, in addition to all of the basic functionality provided
+  in the `local` strategy. A more detailed explanation for how this strategy works can be found [here](#elasticsearch-matching-strategy).
+  - `es_street` is our most advanced matching strategy and requires the highest quality data indexed in elasticsearch
+  via our [ingestion](#ingestion) process. This matching strategy provides everything that `es` and `local` does along
+  with validation of the address1 and address2 components of the address input. A more detailed explanation of how
+  this strategy works can be found [here](#elasticsearch-matching-strategy).
+
+**Validation Scope:** This response object is populated with the field names from the input that have been successfully
+validated.
+
+**Concerns:** This response object is populated with a code if there is a validation error with the input address.
+A concern may also include a suggestion to fix the issue.
+
+**Suggestions:** This response object provides the corrected value for a field that has a concern if available.
+
+### Example request with a concern:
+
+Navigate to http://localhost:3000/graphiql and initiate the following request. Note the invalid zip field.
+
+```graphql
+query validation {
+  validation(
+    address: {
+      address1: "151 O'Connor St"
+      address2: ""
+      city: "Ottawa"
+      provinceCode: "ON"
+      countryCode: CA
+      zip: "90210"
+    }
+    locale: "en"
+    matchingStrategy: LOCAL
+  ) {
+    validationScope
+    concerns {
+      code
+      fieldNames
+      suggestionIds
+      message
+    }
+    suggestions {
+      address1
+      address2
+      city
+      provinceCode
+      zip
+    }
+  }
+}
+```
+
+Response:
+
+```json
+{
+  "data": {
+    "validation": {
+      "validationScope": [
+        "country_code",
+        "province_code",
+        "city",
+        "address1"
+      ],
+      "concerns": [
+        {
+          "code": "zip_invalid_for_province",
+          "fieldNames": [
+            "zip",
+            "country",
+            "province"
+          ],
+          "suggestionIds": [],
+          "message": "Enter a valid postal code for Ontario"
+        }
+      ],
+      "suggestions": []
+    }
+  }
+}
+```
+
+The concerns object contains a concern code `zip_invalid_for_province` to highlight the validation error of `90210`
+being an invalid zip code for the province `ON`. It also returns the human readable message
+`"Enter a valid postal code for Ontario"` in the provided language `en`.
 
-## Local Setup
+The validation scope excludes zip because the zip was not successfully validated.
 
-### In your rails app
+## Installation of Atlas Engine in your rails app
 
-#### Initial setup
+### Initial setup
 Add the engine to your gemfile
 ```
 gem "atlas_engine"
@@ -19,7 +171,7 @@ bundle lock
 bin/rails generate atlas_engine:install
 ```
 
-#### Updating to a newer version of the engine
+### Updating to a newer version of the engine
 
 Working with migrations
 ```
@@ -30,9 +182,11 @@ rails atlas_engine:install:migrations
 rails db:migrate
 ```
 
-### Developing in the engine
+## Setup Atlas Engine for contribution / local development
 
-#### Setup Docker
+This setup guide is based on a mac os development environment. Your tooling may vary.
+
+### Install + Setup Docker
 
 ```
 brew install docker
@@ -46,62 +200,61 @@ colima start --cpu 4 --memory 8
 colima ssh
   sudo sysctl -w vm.max_map_count=262144
   exit
+```
+
+Verify docker is running with: `docker info`
+
+### Clone the atlas_engine git repository
 
 ```
+git clone https://github.com/Shopify/atlas-engine.git
+```
+
+### Setup Ruby and Rails
 
-Verify if docker is running: `docker info`
+Install ruby >= 3.2.1
 
-#### Setup Rails
+In the newly cloned repository directory run:
 
 ```
 bundle install
 
-# If you get an ssl error with puma installation run
+# *Note* If you get an ssl error during the puma installation run the following command:
 bundle config build.puma --with-pkg-config=$(brew --prefix openssl@3)/lib/pkgconfig
 ```
 
-#### Setting up Elasticsearch, Mysql
+### Setup up Dockerized Elasticsearch and MySQL
 
+In a separate terminal, from the cloned atlas_engine directory run:
 ```
-bash setup
 docker-compose up
 
-# If you encounter an error getting docker credentials, remove or update the `credsStore`
+# *Note* If you encounter an error getting docker credentials, remove or update the `credsStore`
 key in your Docker configuration file:
 
 # ~/.docker/config.json
 "credsStore": "desktop", # remove this line
 ```
 
-Connecting to Docker services
-  * for Mysql : `mysql --host=127.0.0.1 --user=user --password=changeme`
-  * for Elasticsearch : `http://localhost:9200`
-
-  _note: if you have updated any of the ports in your .env file then use those ports instead_
+Verify your connection to the newly created Docker services with the following commands:
+  - MySQL : `mysql --host=127.0.0.1 --user=root`
+  - Elasticsearch : `curl http://localhost:9200`
 
-
-#### Setting up db
+### Setup the local db
 ```
 rails db:setup
 ```
 
-#### Setting up maintenance tasks
-After locally setting up Atlas Engine:
-```
-rails app:maintenance_tasks:install:migrations
-rails db:migrate
-```
-
-## Using the App
-
 ### Infrastructure Requirements
-The elasticsearch implementation depends on the ICU analysis plugin. Refer to the [Dockfile](./Dockfile) leveraged in local setup for plugin installation.
+The elasticsearch implementation depends on the ICU analysis plugin. Refer to the [Dockerfile](./docker/elasticsearch/Dockerfile) leveraged in local setup for plugin installation.
 
-### Starting the App and Testing
+### Starting the App / Running Tests
   * `bin/rails server` to start the server
   * `bin/rails test` to run tests
+  * `bundle exec rubocop` to run ruby style checks
+  * `src tc` to run sorbet typechecks
 
-### Running Sorbet
+### Sorbet
 
 Generate rbi files for custom code
 ```
@@ -117,7 +270,318 @@ bin/tapioca gems
 bin/tapioca gems --all
 ```
 
-Running a sorbet check
+Run sorbet check
 ```
 srb tc
 ```
+
+## Ingestion
+
+In order to power the more advanced validation matching strategies that provide city / state / zip and even street
+level address validation, your app must have a populated elasticsearch index per country available for `atlas_engine`
+to query.
+
+The data we use to power atlas engine validation is free open source data from the [open addresses](https://openaddresses.io/)
+project. The following guide demostrates how to ingest data with the dummy app, but the process is the same with
+the engine mounted into your own rails app.
+
+1. Go to the [open addresses](https://openaddresses.io/) download center, create an account, support the project, and
+download a GeoJSON+LD file for the country or region you wish to validate. For this example, we will be using the
+countrywide addresses data for Australia.
+
+2. Once the file is downloaded, start your app with `rails s` and navigate to `http://localhost:3000/maintenance_tasks`
+(see [the github repo](https://github.com/Shopify/maintenance_tasks) for more information about maintenance_tasks).
+There are two tasks available: `Maintenance::AtlasEngine::GeoJsonImportTask` and `Maintenance::AtlasEngine::ElasticsearchIndexCreateTask`. We will be using both in the ingestion process.
+
+3. Navigate to the `Maintenance::AtlasEngine::GeoJsonImportTask`. This task will transform the raw geo json file into
+records in our mysql database and has the following parameters:
+
+- **clear_records:** If checked, removes any existing records for the country in the database.
+
+- **country_code: (required)** The ISO country code of the data we are ingesting.
+In this example, the country code of Australia is `AU`.
+
+- **geojson_file_path: (required)** The fully qualified path of the previously downloaded geojson data from open addresses.
+
+- **locale: (optional)** The language of the data in the open addresses file.
+
+4. Once properly parameterized, click run. The process will initialize a `country_import` and should succeed immediately.
+
+5. Navigate to `http://localhost:3000/country_imports` to track the progress of the country import. Click the import id
+link for a more detailed view. Once the import status has updated from `in_progress` to `complete` we will have all of
+the raw open address data imported into our mysql database's `atlas_engine_post_addresses` table.
+
+6. Navigate back to `http://localhost:3000/maintenance_tasks` and click on the `Maintenance::AtlasEngine::ElasticsearchIndexCreateTask`. This task will ingest the data we have staged in mysql
+and use it to create documents in a new elasticsearch index which Atlas Engine will ultimately use for validation.
+
+7. The `ElasticsearchIndexCreateTask` includes the following parameters:
+
+- **country_code: (required)** the ISO country code of the data we are ingesting and the name of the elasticsearch index we
+will be creating. In this example, the country code of Australia is `AU`.
+
+- **locale: (optional)** the language of the documents we will be creating. This is required for multi-locale countries
+as our indexes are separated by language.
+
+- **province_codes: (optional)** an allow list of province codes to create documents for. If left blank the task will create
+documents for the entire dataset.
+
+- **shard_override: (optional)** the number of shards to create this index with. If left blank the default will be used.
+
+- **replica_override: (optional)** the number of replicas to create this index with. If left blank the default will be used.
+
+- **activate_index: (optional)** if checked, immediately promote this index to be the index queried by atlas engine.
+If unchecked, the created index will need to be activated manually.
+
+8. Once properly parameterized, click run. The maintenance task UI will track the progress of the index creation.
+
+9. When completed, the index documents may be verified manually with an elasticsearch client.
+We may now use the `es` and `es_street` matching strategies with `AU` addresses. See [below](#elasticsearch-matching-strategy)
+for an example of its usage.
+
+## Elasticsearch Matching Strategy
+
+Once we have successfully created and activated an elasticsearch index using open address data, we may now use
+the more advanced elasticsearch matching strategies `es` and `es_street`.
+
+Consider the following example of an invalid `AU` address:
+
+```graphql
+query validation {
+  validation(
+    address: {
+      address1: "100 miller st"
+      address2: ""
+      city: "sydney"
+      provinceCode: "NSW"
+      countryCode: AU
+      zip: "2060"
+    }
+    locale: "en"
+    matchingStrategy: ES
+  ) {
+    validationScope
+    concerns {
+      code
+      fieldNames
+      suggestionIds
+      message
+    }
+    suggestions {
+      address1
+      address2
+      city
+      provinceCode
+      zip
+    }
+  }
+}
+```
+
+When input into `http://localhost:3000/graphiql`, this query should produce the following response:
+
+```json
+{
+  "data": {
+    "validation": {
+      "candidate": ",NSW,,,,2060,[North Sydney],,Miller Street",
+      "validationScope": [
+        "country_code",
+        "province_code",
+        "zip",
+        "city",
+        "address1"
+      ],
+      "concerns": [
+        {
+          "code": "city_inconsistent",
+          "typeLevel": 3,
+          "fieldNames": [
+            "city"
+          ],
+          "suggestionIds": [
+            "665ffd09-75b8-584d-8e4a-a0f471bfea01"
+          ],
+          "message": "Enter a valid city for New South Wales, 2060"
+        }
+      ],
+      "suggestions": [
+        {
+          "id": "665ffd09-75b8-584d-8e4a-a0f471bfea01",
+          "address1": null,
+          "address2": null,
+          "city": "North Sydney",
+          "province": null,
+          "provinceCode": null,
+          "zip": null
+        }
+      ]
+    }
+  }
+}
+```
+
+The concerns object contains a concern code `city_inconsistent` to highlight the validation error of `sydney`
+being an incorrect city for the rest of the provided address. The concern message field is the human readable
+error nudge `"Enter a valid city for New South Wales, 2060"`, pointing to the supporting pieces of evidence (province
+and zip) that were used to determine city as the inconsistent value in this address input.
+
+The suggestion object contains a corrected city field `North Sydney` which will result in no concerns or suggestions
+for the validation endpoint if applied.
+
+The candidate field contains a representation of the matching document in the elasticsearch index that was found and
+used to determine the suggestions and concerns in the api response.
+
+The `es_street` level of validation can also be used to correct errors in the `address1` or `address2` fields of the
+input. In the following request we have modified our query to make a second error in our input - searching for
+`miller ave` instead of `miller st`.
+
+```graphql
+query validation {
+  validation(
+    address: {
+      address1: "100 miller ave"
+      address2: ""
+      city: "sydney"
+      provinceCode: "NSW"
+      countryCode: AU
+      zip: "2060"
+    }
+    locale: "en"
+    matchingStrategy: ES_STREET
+  ) {
+    validationScope
+    concerns {
+      code
+      fieldNames
+      suggestionIds
+      message
+    }
+    suggestions {
+      address1
+      address2
+      city
+      provinceCode
+      zip
+    }
+  }
+}
+```
+
+This query produces the following response:
+
+```json
+{
+  "data": {
+    "validation": {
+      "candidate": ",NSW,,,,2060,[North Sydney],,Miller Street",
+      "validationScope": [
+        "country_code",
+        "province_code",
+        "zip",
+        "city",
+        "address1"
+      ],
+      "concerns": [
+        {
+          "code": "city_inconsistent",
+          "typeLevel": 3,
+          "fieldNames": [
+            "city"
+          ],
+          "suggestionIds": [
+            "88779db6-2c5d-5dbb-9f77-f7b07c07206a"
+          ],
+          "message": "Enter a valid city for New South Wales, 2060"
+        },
+        {
+          "code": "street_inconsistent",
+          "typeLevel": 3,
+          "fieldNames": [
+            "address1"
+          ],
+          "suggestionIds": [
+            "88779db6-2c5d-5dbb-9f77-f7b07c07206a"
+          ],
+          "message": "Enter a valid street name for New South Wales, 2060"
+        }
+      ],
+      "suggestions": [
+        {
+          "id": "88779db6-2c5d-5dbb-9f77-f7b07c07206a",
+          "address1": "100 Miller Street",
+          "address2": null,
+          "city": "North Sydney",
+          "province": null,
+          "provinceCode": null,
+          "zip": null
+        }
+      ]
+    }
+  }
+}
+```
+
+The concerns object now contains an additional concern code `street_inconsistent` to highlight the validation error of
+`miller ave` being an incorrect street for the rest of the address input. The concern message field is the human
+readable error nudge `"Enter a valid street name for New South Wales, 2060"`, pointing to the supporting pieces of
+evidence (province and zip) that were used to determine street as an inconsistent value in this address input.
+
+The suggestion object contains a corrected street field `100 Miller Street` and a corrected city field `North Sydney`
+If both of these suggestions are applied to the input address the subsequent request will be valid.
+
+The corrected input of
+
+```graphql
+query validation {
+  validation(
+    address: {
+      address1: "100 miller st"
+      address2: ""
+      city: "north sydney"
+      provinceCode: "NSW"
+      countryCode: AU
+      zip: "2060"
+    }
+    locale: "en"
+    matchingStrategy: ES_STREET
+  ) {
+    validationScope
+    concerns {
+      code
+      fieldNames
+      suggestionIds
+      message
+    }
+    suggestions {
+      address1
+      address2
+      city
+      provinceCode
+      zip
+    }
+  }
+}
+```
+
+will produce the response:
+
+```json
+{
+  "data": {
+    "validation": {
+      "candidate": ",NSW,,,,2060,[North Sydney],,Miller Street",
+      "validationScope": [
+        "country_code",
+        "province_code",
+        "zip",
+        "city",
+        "address1"
+      ],
+      "concerns": [],
+      "suggestions": []
+    }
+  }
+}
+```
+
+This response has no concerns or suggestions, and the input address is therefore considered to be valid.

data/app/countries/atlas_engine/be/country_profile.yml

@@ -6,6 +6,8 @@ ingestion:
 validation:
   enabled: true
   default_matching_strategy: local
+  address_parser: AtlasEngine::Be::ValidationTranscriber::AddressParser
+  has_provinces: false 
   index_locales:
   - fr
   - nl

data/app/countries/atlas_engine/be/validation_transcriber/address_parser.rb

@@ -0,0 +1,84 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Be
+    module ValidationTranscriber
+      class AddressParser < AtlasEngine::ValidationTranscriber::AddressParserBase
+        private
+
+        STREET = "(?<street>.+)"
+        NUMBERED_STREET = "(?<street>.+\s+[0-9]+)"
+        BUILDING_NUM = "(?<building_num>[0-9]+[[:alpha:]]*)"
+        UNIT_NUM = "(?<unit_num>[[:alnum:]]+)"
+        PO_BOX = /\b(?<box_type>pb|box|bte|bus|boîte|boite|postbus|antwoordnummer)\s+(?<number>\d+)\b/i
+        STREET_SUFFIXES = %r{
+          \A(
+            dwarsstraat|dwstr|dwarsweg|dwwg|dijk|dk|gracht|gr|kade|kd|kanaal|kan
+            |laan|leane|loane|ln|park|pk|plantsoen|plnts|plein|pln|singel|sngl
+            |straat|strjitte|str|straatweg|strwg|weg|wg
+          )\z
+        }ix
+
+        sig { returns(T::Array[Regexp]) }
+        def country_regex_formats
+          @country_regex_formats ||= [
+            /^#{STREET},?\s+#{BUILDING_NUM}/,
+            /^#{BUILDING_NUM}\s?(,\s?)?#{STREET}/,
+            /^#{NUMBERED_STREET},?\s+#{BUILDING_NUM}$/,
+            %r{^#{STREET},?\s+#{BUILDING_NUM}[\s,-/]+#{UNIT_NUM}$},
+            /^#{NUMBERED_STREET},?\s+#{BUILDING_NUM}[\s,-]+#{UNIT_NUM}$/,
+          ]
+        end
+
+        sig { override.params(address_line: String).returns(T::Array[T.nilable(String)]) }
+        def extract_po_box(address_line)
+          po_box_match = address_line.match(PO_BOX)
+
+          if po_box_match
+            po_box = po_box_match["number"]
+            address_line = address_line.gsub(PO_BOX, "").strip.delete_suffix(",")
+          else
+            po_box = nil
+          end
+
+          [address_line, po_box]
+        end
+
+        # Return true if something's obviously wrong with this regex match
+        sig do
+          override.params(
+            captures: T::Hash[Symbol, T.nilable(String)],
+            address: AddressValidation::TAddress,
+          ).returns(T::Boolean)
+        end
+        def ridiculous?(captures, address)
+          building_num = captures[:building_num]&.downcase
+          street = captures[:street]&.downcase
+          unit_num = captures[:unit_num]&.downcase
+
+          if street.present?
+            return true unless address.address1&.upcase&.include?(street.upcase) ||
+              address.address2&.upcase&.include?(street.upcase)
+          end
+
+          [building_num, unit_num].any? do |token|
+            po_box?(token) || street_suffix?(token)
+          end
+        end
+
+        sig { override.params(token: T.nilable(String)).returns(T::Boolean) }
+        def po_box?(token)
+          return false if token.blank?
+
+          token.match?(PO_BOX)
+        end
+
+        sig { override.params(token: T.nilable(String)).returns(T::Boolean) }
+        def street_suffix?(token)
+          token.present? && token.match?(STREET_SUFFIXES)
+        end
+      end
+    end
+  end
+end