atlas_engine 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 56b5b41a624c3481954aadfa0101c9883dfe3479a3cdec373f8ecccab8bae879
-  data.tar.gz: 73ee021ecf5d8357ae8f000df7e963e7504153fdc3b7bb98f188445aa8cd7989
+  metadata.gz: 1bdadba3857324dae37bf4c22aa4078b4f565876c0f865a4d3b99db69d0eaa41
+  data.tar.gz: 4004ade11bd387145ed6b82cb8b5df9a99989b3d1869cc1d40fd7fb310e3fc7c
 SHA512:
-  metadata.gz: b93dee8561c73619222ca4a4e803fb2f911b3149955c5b1af7803990318a7ebd36e781c41643cf3076197f579833e9c2d361e8e4550a9d2d3de628d8e7819ef7
-  data.tar.gz: f2f593c6e2a7849dae6dce6a52d9402ce7d907a90511664d1218133b5d4a501b5ae98a26e54a06412c3fb37e3dc74c75ae17c575ae24fa4805c02ad888467d21
+  metadata.gz: 846730be55864274ddd3d61e90ffd970102ea54d25221e411cee78c91daaa974a1159d0c976cde708ac6ea355815deb9c1d91230ef9c4eb49dcf38fe59e4be27
+  data.tar.gz: 791c4a0050adab0b39092b660c56968d7f6862a1387914f50341f940153963487f9fc8ec8533ee6519fd8569bd69022c8cb7ce181c3017a14313fd32a49879a7

data/README.md

@@ -290,12 +290,46 @@ level address validation, your app must have a populated elasticsearch index per
 to query.
 
 The data we use to power atlas engine validation is free open source data from the [open addresses](https://openaddresses.io/)
-project. The following guide demostrates how to ingest data with the dummy app, but the process is the same with
+project.
+
+### Supported countries
+At the moment, `atlas_engine` supports advanced address validation for the following countries.
+
+| Country/territory | Two-letter code | Locales  | Street | City | Postal Code | Province/State |
+|-------------------|-----------------|----------|--------|------|-------------|----------------|
+| Australia         | AU              |          |        | x    | x           | x              |
+| Austria           | AT              |          |        | x    | x           | x              |
+| Belgium           | BE              | fr,nl,de |        | x    | x           |                |
+| Bermuda           | BM              |          |        | x    | x           | x              |
+| Czechia           | CZ              |          |        | x    | x           |                |
+| Denmark           | DK              |          |        | x    | x           |                |
+| Faroe Islands     | FO              |          |        | x    | x           |                |
+| France            | FR              |          |        | x    | x           |                |
+| Italy             | IT              |          |        |      | x           |                |
+| Liechtenstein     | LI              |          |        | x    | x           | x              |
+| Luxembourg        | LU              | fr,lb    |        | x    | x           |                |
+| Netherlands       | NL              | nl       |        | x    | x           | x              |
+| Poland            | PL              |          |        | x    | x           | x              |
+| Portugal          | PT              |          |        | x    | x           | x              |
+| Slovenia          | SI              |          |        | x    | x           | x              |
+| South Korea       | KR              |          |        | x    | x           | x              |
+| Switzerland       | CH              | de,fr,it |        | x    | x           |                |
+
+### Downloading and indexing instructions
+
+
+The following guide demonstrates how to ingest data with the dummy app, but the process is the same with
 the engine mounted into your own rails app.
 
 1. Go to the [open addresses](https://openaddresses.io/) download center, create an account, support the project, and
-download a GeoJSON+LD file for the country or region you wish to validate. For this example, we will be using the
-countrywide addresses data for Australia.
+download a GeoJSON+LD file for the country or region you wish to validate.
+
+Restrictions on the file:
+- Must be an `addresses` file, as opposed to a `buildings` or `parcels` file.
+- Must be gzipped (.gz format)
+- Datasets listed under the _Individual Sources_ section work fine. Those under _Data Collections_ must first be unzipped. The `addresses` geojson files within may then be gzipped and imported.
+
+For this example, we will be using the `au/countrywide` --> `addresses - country` data for Australia, in the GeoJSON+LD format.
 
 2. Once the file is downloaded, start your app with `rails s` and navigate to `http://localhost:3000/maintenance_tasks`
 (see [the github repo](https://github.com/Shopify/maintenance_tasks) for more information about maintenance_tasks).
@@ -309,7 +343,7 @@ records in our mysql database and has the following parameters:
 - **country_code: (required)** The ISO country code of the data we are ingesting.
 In this example, the country code of Australia is `AU`.
 
-- **geojson_file_path: (required)** The fully qualified path of the previously downloaded geojson data from open addresses.
+- **geojson_file_path: (required)** The fully qualified path of the previously downloaded geojson data from OpenAddresses. A comma-delimited list of fully-qualified paths is also accepted.
 
 - **locale: (optional)** The language of the data in the open addresses file.
 

data/app/countries/atlas_engine/gg/address_importer/corrections/open_address/city_corrector.rb

@@ -0,0 +1,38 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Gg
+    module AddressImporter
+      module Corrections
+        module OpenAddress
+          class CityCorrector
+            class << self
+              extend T::Sig
+
+              CITY_ALIASES_MAPPING = {
+                "St. Sampson" => ["Saint Samsaon"],
+                "St. Saviour" => ["Saint-Sauveur", "Saint Sauveux"],
+                "St. Peter Port" => ["Saint-Pierre Port"],
+                "St. Andrew" => ["Saint Andri", "Saint-André-de-la-Pommeraye"],
+                "St. Pierre Du Bois" => ["St. Peter's", "St. Pierre"],
+                "Castel" => ["Lé Casté", "Sainte-Marie-du-Câtel"],
+                "Forest" => ["Le Fôret", "La Fouarêt"],
+                "Torteval" => ["Tortévas"],
+                "Vale" => ["Lé Vale", "Le Valle"],
+              }
+
+              sig { params(address: Hash).void }
+              def apply(address)
+                city = address[:city].first
+                if CITY_ALIASES_MAPPING.include?(city)
+                  address[:city] += CITY_ALIASES_MAPPING[city]
+                end
+              end
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/app/countries/atlas_engine/gg/address_validation/validators/full_address/exclusions/city.rb

@@ -0,0 +1,32 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Gg
+    module AddressValidation
+      module Validators
+        module FullAddress
+          module Exclusions
+            class City <
+              AtlasEngine::AddressValidation::Validators::FullAddress::Exclusions::ExclusionBase
+              extend T::Sig
+              class << self
+                sig do
+                  override.params(
+                    candidate: AtlasEngine::AddressValidation::Candidate,
+                    address_comparison: AtlasEngine::AddressValidation::Validators::FullAddress::AddressComparison,
+                  )
+                    .returns(T::Boolean)
+                end
+                def apply?(candidate, address_comparison)
+                  # If the candidate city is already present in the parsings
+                  address_comparison.parsings.parsings.pluck(:city)&.include?(candidate.component(:city)&.value&.first)
+                end
+              end
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/app/countries/atlas_engine/gg/country_profile.yml

@@ -3,5 +3,13 @@ validation:
   enabled: true
   default_matching_strategy: es
   has_provinces: false
+  address_parser: AtlasEngine::Gg::ValidationTranscriber::AddressParser
   restrictions:
     - class: AtlasEngine::Gg::AddressValidation::Validators::FullAddress::Restrictions::UnsupportedCity
+ingestion:
+  correctors:
+      open_address:
+        - AtlasEngine::Gg::AddressImporter::Corrections::OpenAddress::CityCorrector
+  exclusions:
+    city:
+      - AtlasEngine::Gg::AddressValidation::Validators::FullAddress::Exclusions::City

data/app/countries/atlas_engine/gg/synonyms.yml

@@ -0,0 +1,3 @@
+city_synonyms:
+- st, saint
+- ste, sainte

data/app/countries/atlas_engine/gg/validation_transcriber/address_parser.rb

@@ -0,0 +1,57 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Gg
+    module ValidationTranscriber
+      class AddressParser < AtlasEngine::ValidationTranscriber::AddressParserBase
+        private
+
+        CITY = %r{
+          (?<city>
+            st\.?\s?saviour[']?[s]?|
+            saint[-|\s]sauveur|
+            saint\s?sauveux|
+            st\.?\s?sampson[']?[s]?|
+            saint\s?samsaon|
+            st\.?\s?andrew[']?[s]?|
+            saint\s?andri|
+            saint[-|\s]andr[é|e][-|\s]de[-|\s]la[-|\s]pommeraye|
+            st\.?\s?martin[']?[s]?|
+            st\.?\s?peter[']?[s]?\s?port|
+            saint[-|\s]pierre\s?port|
+            st\.?\s?peter[']?[s]?|
+            st\.?\s?pierre\s?du\s?bois|
+            st\.?\s?pierre|
+            vale|
+            l[é|e]\s?vale|
+            le\s?valle|
+            torteval|
+            tort[é|e]vas|
+            castel|
+            l[é|e]\s?cast[é|e]|
+            sainte[-|\s]marie[-|\s]du[-|\s]c[â|a]tel|
+            forest|
+            le\s?f[ô|o]ret|
+            la\s?fouar[ê|e]t
+          )
+        }ix
+
+        sig { returns(T::Array[Regexp]) }
+        def country_regex_formats
+          @country_regex_formats ||= [
+            /^#{BUILDING_NAME},\s+#{CITY}$/,
+            /^#{BUILDING_NAME},\s+#{STREET_NO_COMMAS},\s+#{CITY}$/,
+            /^#{BUILDING_NAME},\s+#{STREET_NO_COMMAS}$/,
+            /^#{NUMERIC_ONLY_BUILDING_NUM}?\s+#{STREET_NO_COMMAS},\s+#{CITY}$/,
+            /^#{UNIT_TYPE}\s+#{UNIT_NUM_NO_HYPHEN},\s+#{BUILDING_NAME},\s+#{CITY}$/,
+            /^#{CITY}$/,
+            /^#{NUMERIC_ONLY_BUILDING_NUM}?\s+#{STREET_NO_COMMAS}$/,
+            /^#{BUILDING_NAME}$/,
+            /^#{STREET_NO_COMMAS}$/,
+          ]
+        end
+      end
+    end
+  end
+end

data/app/models/atlas_engine/address_validation/es/datastore.rb

@@ -128,6 +128,9 @@ module AtlasEngine
             end
             Token::Sequence.new(tokens: tokens, raw_value: city_value)
           end
+        rescue Elasticsearch::Error => e
+          log_elasticsearch_error(error: e, method: "city_sequence")
+          Token::Sequence.new(tokens: [], raw_value: city_value)
         end
 
         sig { returns(T::Array[T::Hash[String, T.untyped]]) }
@@ -135,6 +138,9 @@ module AtlasEngine
           measure_es_validation_request_time(method: "full_address_candidates") do
             repository.search(query_builder.full_address_query)
           end
+        rescue Elasticsearch::Error => e
+          log_elasticsearch_error(error: e, method: "full_address_candidates")
+          []
         end
 
         sig { returns(T::Array[Token::Sequence]) }
@@ -154,6 +160,9 @@ module AtlasEngine
               end
             end
           end
+        rescue Elasticsearch::Error => e
+          log_elasticsearch_error(error: e, method: "street_sequence")
+          [Token::Sequence.new(tokens: [], raw_value: "")]
         end
 
         sig { params(candidates: T::Array[Candidate]).void }
@@ -165,6 +174,8 @@ module AtlasEngine
           end
 
           TermVectors.new(term_vectors_hashes: candidate_term_vectors, candidates: candidates).set_candidate_sequences
+        rescue Elasticsearch::Error => e
+          log_elasticsearch_error(error: e, method: "term_vectors")
         end
 
         sig { params(candidates: T::Array[Candidate]).returns(T::Hash[String, T.untyped]) }
@@ -190,6 +201,18 @@ module AtlasEngine
           )
         end
 
+        sig { params(error: Elasticsearch::Error, method: String).void }
+        def log_elasticsearch_error(error:, method:)
+          log_error("Elasticsearch error: #{error.message}")
+          StatsD.increment(
+            "AddressValidation.elasticsearch_error",
+            tags: {
+              country: country_code,
+              method:,
+            },
+          )
+        end
+
         sig { params(future: T.nilable(Concurrent::Promises::Future), method: String).void }
         def log_future_state_on_join(future:, method:)
           state = future&.state || :unsubmitted

data/app/models/atlas_engine/address_validation/validators/full_address/concern_builder.rb

@@ -109,10 +109,6 @@ module AtlasEngine
             concern = InvalidZipConcernBuilder.for(address, suggestion_ids)
             return concern if concern
 
-            if :province_code.in?(matched_components) && :city.in?(matched_components)
-              return UnknownZipForAddressConcernBuilder.new(address).build(suggestion_ids)
-            end
-
             build_default_concern
           end
 

data/app/models/atlas_engine/address_validation/validators/full_address/unmatched_field_concern_builder.rb

@@ -56,7 +56,7 @@ module AtlasEngine
 
           sig { returns(String) }
           def message
-            "Enter a valid #{COMPONENTS_TO_LABELS[component]} for #{valid_address_component_values.join(", ")}"
+            country.field(key: field_name).error(code: :unknown_for_address).to_s
           end
 
           sig { returns(Symbol) }

data/app/models/atlas_engine/address_validation/validators/predicates/country/valid_for_zip.rb

@@ -28,7 +28,7 @@ module AtlasEngine
             sig { params(suggestion: Suggestion).returns(Concern) }
             def build_concern(suggestion)
               Concern.new(
-                field_names: [:country, :zip],
+                field_names: [:country],
                 code: :country_invalid_for_zip,
                 type: T.must(Concern::TYPES[:error]),
                 type_level: 1,

data/app/models/atlas_engine/address_validation/validators/predicates/province/valid_for_country.rb

@@ -24,7 +24,7 @@ module AtlasEngine
             sig { returns(Concern) }
             def build_concern
               Concern.new(
-                field_names: [:province, :country],
+                field_names: [:province],
                 code: :province_invalid,
                 type: T.must(Concern::TYPES[:error]),
                 type_level: 3,

data/app/models/atlas_engine/address_validation/validators/predicates/street/building_number_in_address1.rb

@@ -24,7 +24,7 @@ module AtlasEngine
             sig { returns(Concern) }
             def build_concern
               Concern.new(
-                field_names: [:address1, :country],
+                field_names: [:address1],
                 code: :missing_building_number,
                 type: T.must(Concern::TYPES[:warning]),
                 type_level: 1,

data/app/models/atlas_engine/address_validation/validators/predicates/street/building_number_in_address1_or_address2.rb

@@ -22,7 +22,7 @@ module AtlasEngine
             sig { returns(Concern) }
             def build_concern
               Concern.new(
-                field_names: [:address1, :address2, :country],
+                field_names: [:address1, :address2],
                 code: :missing_building_number,
                 type: T.must(Concern::TYPES[:warning]),
                 type_level: 3,

data/app/models/atlas_engine/address_validation/validators/predicates/zip/valid_for_country.rb

@@ -26,7 +26,7 @@ module AtlasEngine
             sig { returns(Concern) }
             def build_concern
               Concern.new(
-                field_names: [:zip, :country],
+                field_names: [:zip],
                 code: :zip_invalid_for_country,
                 type: T.must(Concern::TYPES[:error]),
                 type_level: 3,

data/app/models/atlas_engine/address_validation/validators/predicates/zip/valid_for_province.rb

@@ -29,7 +29,7 @@ module AtlasEngine
             sig { returns(Concern) }
             def build_concern
               Concern.new(
-                field_names: [:zip, :country, :province],
+                field_names: [:zip],
                 code: :zip_invalid_for_province,
                 type: T.must(Concern::TYPES[:error]),
                 type_level: 3,

data/app/models/atlas_engine/elasticsearch/error.rb

@@ -0,0 +1,19 @@
+# typed: strict
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Elasticsearch
+    class Error < StandardError
+      extend T::Sig
+
+      sig { params(message: String, cause: T.nilable(Exception)).void }
+      def initialize(message, cause = nil)
+        super(message)
+        @cause = cause
+      end
+
+      sig { returns(T.nilable(Exception)) }
+      attr_reader :cause
+    end
+  end
+end

data/app/models/atlas_engine/elasticsearch/repository.rb

@@ -133,8 +133,8 @@ module AtlasEngine
         response = client.post(path, query, {})
 
         response.body
-      rescue ::Elastic::Transport::Transport::Error
-        { "hits" => { "hits" => [] } }
+      rescue ::Elastic::Transport::Transport::Error => e
+        raise Elasticsearch::Error.new(e.message, e)
       end
 
       sig { override.params(query: T::Hash[String, T.untyped]).returns(T::Hash[String, T.untyped]) }
@@ -143,8 +143,8 @@ module AtlasEngine
         response = client.post(path, query, {})
 
         response.body
-      rescue ::Elastic::Transport::Transport::Error
-        { "tokens" => [] }
+      rescue ::Elastic::Transport::Transport::Error => e
+        raise Elasticsearch::Error.new(e.message, e)
       end
 
       sig { override.params(query: T::Hash[String, T.untyped]).returns(T::Hash[String, T.untyped]) }
@@ -153,8 +153,8 @@ module AtlasEngine
         response = client.post(path, query, {})
 
         response.body
-      rescue ::Elastic::Transport::Transport::Error
-        { "docs" => [] }
+      rescue ::Elastic::Transport::Transport::Error => e
+        raise Elasticsearch::Error.new(e.message, e)
       end
 
       sig { override.params(id: T.any(String, Integer)).returns(T::Hash[String, T.untyped]) }
@@ -163,8 +163,8 @@ module AtlasEngine
         response = client.get(path, nil, {})
 
         response.body["_source"]
-      rescue ::Elastic::Transport::Transport::Error
-        {}
+      rescue ::Elastic::Transport::Transport::Error => e
+        raise Elasticsearch::Error.new(e.message, e)
       end
 
       sig { override.params(post_address: PostAddressData).returns(T::Hash[Symbol, T.untyped]) }

data/lib/atlas_engine/version.rb

@@ -2,5 +2,5 @@
 # frozen_string_literal: true
 
 module AtlasEngine
-  VERSION = "0.6.0"
+  VERSION = "0.8.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: atlas_engine
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Shopify
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-02-26 00:00:00.000000000 Z
+date: 2024-04-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: annex_29
@@ -305,8 +305,12 @@ files:
 - app/countries/atlas_engine/gb/country_profile.yml
 - app/countries/atlas_engine/gb/validation_transcriber/full_address_parser.rb
 - app/countries/atlas_engine/gb/validation_transcriber/parsed_address.rb
+- app/countries/atlas_engine/gg/address_importer/corrections/open_address/city_corrector.rb
+- app/countries/atlas_engine/gg/address_validation/validators/full_address/exclusions/city.rb
 - app/countries/atlas_engine/gg/address_validation/validators/full_address/restrictions/unsupported_city.rb
 - app/countries/atlas_engine/gg/country_profile.yml
+- app/countries/atlas_engine/gg/synonyms.yml
+- app/countries/atlas_engine/gg/validation_transcriber/address_parser.rb
 - app/countries/atlas_engine/ie/country_profile.yml
 - app/countries/atlas_engine/it/address_importer/corrections/open_address/city_corrector.rb
 - app/countries/atlas_engine/it/address_importer/corrections/open_address/province_corrector.rb
@@ -494,7 +498,6 @@ files:
 - app/models/atlas_engine/address_validation/validators/full_address/suggestion_builder.rb
 - app/models/atlas_engine/address_validation/validators/full_address/unknown_address_concern_builder.rb
 - app/models/atlas_engine/address_validation/validators/full_address/unknown_province_concern_builder.rb
-- app/models/atlas_engine/address_validation/validators/full_address/unknown_zip_for_address_concern_builder.rb
 - app/models/atlas_engine/address_validation/validators/full_address/unmatched_field_concern_builder.rb
 - app/models/atlas_engine/address_validation/validators/full_address/unsupported_script_result.rb
 - app/models/atlas_engine/address_validation/validators/full_address/zip_comparison.rb
@@ -530,6 +533,7 @@ files:
 - app/models/atlas_engine/country_repository.rb
 - app/models/atlas_engine/elasticsearch/client.rb
 - app/models/atlas_engine/elasticsearch/client_interface.rb
+- app/models/atlas_engine/elasticsearch/error.rb
 - app/models/atlas_engine/elasticsearch/repository.rb
 - app/models/atlas_engine/elasticsearch/repository_interface.rb
 - app/models/atlas_engine/elasticsearch/response.rb
@@ -593,7 +597,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.6
+rubygems_version: 3.5.7
 signing_key: 
 specification_version: 4
 summary: Address Validation API

data/app/models/atlas_engine/address_validation/validators/full_address/unknown_zip_for_address_concern_builder.rb

@@ -1,34 +0,0 @@
-# typed: true
-# frozen_string_literal: true
-
-module AtlasEngine
-  module AddressValidation
-    module Validators
-      module FullAddress
-        class UnknownZipForAddressConcernBuilder
-          extend T::Sig
-          attr_reader :address
-
-          sig { params(address: AbstractAddress).void }
-          def initialize(address)
-            @address = address
-          end
-
-          sig { params(suggestion_ids: T::Array[String]).returns(Concern) }
-          def build(suggestion_ids = [])
-            message = "Enter a valid ZIP for #{address.address1}, #{address.city}"
-
-            Concern.new(
-              code: :zip_inconsistent,
-              field_names: [:zip],
-              message: message,
-              type: T.must(Concern::TYPES[:warning]),
-              type_level: 3,
-              suggestion_ids: suggestion_ids,
-            )
-          end
-        end
-      end
-    end
-  end
-end