atlas_engine 0.7.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9948b64010b57506c90e07bf0fc324c7ad158112169375aa1ac1f7053850a9be
-  data.tar.gz: 872cd26b540c96af7a777b044a4af20cce6d1b7df5569f0b3cd7aaf999fb1847
+  metadata.gz: 2236db9cf72e66e7bec983fec0ce919b31e1a795e6509ac65e1f914e8cda1c2b
+  data.tar.gz: c4bb6b2c1c8c927abbdf22d66d7a198052ab772f9243ffbd4c02bccced29b4ea
 SHA512:
-  metadata.gz: d8ea33678c4c924fb202cecf027e4796892430c35394233c0a971474166b71a10881a6ddc2cc6056d7f66df6b225a63cfdafe18b5590327035d19f3f4e70cf99
-  data.tar.gz: 30d5b8b914d14f4fd6871228435383ff90499e25617af0c7dc5a6a5580fbf4005854f2189034712f4030001878be8c6ee42223e5b94a1890b2a14f8be80e0dc3
+  metadata.gz: a1ccff91cd2b8026fd568ba9aced9688741bf1fd107fd180a83447e9d62ce20750d1ba8d14efe690ab22db6d239dfc1af51edda770c00de31cf07bfc210dc8cf
+  data.tar.gz: 9ee3408a1926463cf05cef4ba3b835e71fb8f490b61033154ac47dc586e2eb8b20fa971c9ca8c23859cec7fd4f7f8e81370f506fdba16e081b6299595a517f05

data/README.md

@@ -167,17 +167,23 @@ The validation scope excludes zip because the zip was not successfully validated
 ## Rails App Installation
 
 ### Initial setup
-Add the engine to your gemfile
+* Add the engine to your gemfile
 ```
 gem "atlas_engine"
 ```
 
-Run the following commands to install the engine in your rails app
-
+* Run the following commands to install the engine in your rails app
 ```
 bundle lock
-bin/rails generate atlas_engine:install
+rails atlas_engine:install:migrations
+rails db:migrate
 ```
+* In `config/routes` mount AtlasEngine
+  * Adding the line `mount AtlasEngine::Engine => "/atlas_engine"
+`
+* In `app/assets/config/manifest.js`
+  * Adding the line `//= link atlas_engine/application.css`
+* Install [maintenance_tasks](https://github.com/Shopify/maintenance_tasks?tab=readme-ov-file#installation) - a dependency for Atlas Engine that is used to ingest country data.
 
 ### Updating to a newer version of the engine
 
@@ -290,12 +296,47 @@ level address validation, your app must have a populated elasticsearch index per
 to query.
 
 The data we use to power atlas engine validation is free open source data from the [open addresses](https://openaddresses.io/)
-project. The following guide demostrates how to ingest data with the dummy app, but the process is the same with
+project.
+
+### Supported countries
+At the moment, `atlas_engine` supports advanced address validation for the following countries.
+
+| Country/territory | Two-letter code | Locales  | Street | City | Postal Code | Province/State |
+|-------------------|-----------------|----------|--------|------|-------------|----------------|
+| Australia         | AU              |          |        | x    | x           | x              |
+| Austria           | AT              |          |        | x    | x           | x              |
+| Belgium           | BE              | fr,nl,de |        | x    | x           |                |
+| Bermuda           | BM              |          |        | x    | x           | x              |
+| Czechia           | CZ              |          |        | x    | x           |                |
+| Denmark           | DK              |          |        | x    | x           |                |
+| Faroe Islands     | FO              |          |        | x    | x           |                |
+| France            | FR              |          |        | x    | x           |                |
+| Italy             | IT              |          |        |      | x           |                |
+| Liechtenstein     | LI              |          |        | x    | x           | x              |
+| Luxembourg        | LU              | fr,lb    |        | x    | x           |                |
+| Netherlands       | NL              | nl       |        | x    | x           | x              |
+| Poland            | PL              |          |        | x    | x           | x              |
+| Portugal          | PT              |          |        | x    | x           | x              |
+| Slovenia          | SI              |          |        | x    | x           | x              |
+| South Korea       | KR              |          |        | x    | x           | x              |
+| Switzerland       | CH              | de,fr,it |        | x    | x           |                |
+| United States     | US              | en       | x      | x    | x           | x              |
+
+### Downloading and indexing instructions
+
+
+The following guide demonstrates how to ingest data with the dummy app, but the process is the same with
 the engine mounted into your own rails app.
 
 1. Go to the [open addresses](https://openaddresses.io/) download center, create an account, support the project, and
-download a GeoJSON+LD file for the country or region you wish to validate. For this example, we will be using the
-countrywide addresses data for Australia.
+download a GeoJSON+LD file for the country or region you wish to validate.
+
+Restrictions on the file:
+- Must be an `addresses` file, as opposed to a `buildings` or `parcels` file.
+- Must be gzipped (.gz format)
+- Datasets listed under the _Individual Sources_ section work fine. Those under _Data Collections_ must first be unzipped. The `addresses` geojson files within may then be gzipped and imported.
+
+For this example, we will be using the `au/countrywide` --> `addresses - country` data for Australia, in the GeoJSON+LD format.
 
 2. Once the file is downloaded, start your app with `rails s` and navigate to `http://localhost:3000/maintenance_tasks`
 (see [the github repo](https://github.com/Shopify/maintenance_tasks) for more information about maintenance_tasks).
@@ -309,7 +350,7 @@ records in our mysql database and has the following parameters:
 - **country_code: (required)** The ISO country code of the data we are ingesting.
 In this example, the country code of Australia is `AU`.
 
-- **geojson_file_path: (required)** The fully qualified path of the previously downloaded geojson data from open addresses.
+- **geojson_file_path: (required)** The fully qualified path of the previously downloaded geojson data from OpenAddresses. A comma-delimited list of fully-qualified paths is also accepted.
 
 - **locale: (optional)** The language of the data in the open addresses file.
 
@@ -319,8 +360,7 @@ In this example, the country code of Australia is `AU`.
 link for a more detailed view. Once the import status has updated from `in_progress` to `complete` we will have all of
 the raw open address data imported into our mysql database's `atlas_engine_post_addresses` table.
 
-6. Navigate back to `http://localhost:3000/maintenance_tasks` and click on the `Maintenance::AtlasEngine::ElasticsearchIndexCreateTask`. This task will ingest the data we have staged in mysql
-and use it to create documents in a new elasticsearch index which Atlas Engine will ultimately use for validation.
+6. Navigate back to `http://localhost:3000/maintenance_tasks` and click on the `Maintenance::AtlasEngine::ElasticsearchIndexCreateTask`. This task will ingest the data we have staged in mysql and use it to create documents in a new elasticsearch index which Atlas Engine will ultimately use for validation.
 
 7. The `ElasticsearchIndexCreateTask` includes the following parameters:
 
@@ -346,6 +386,26 @@ If unchecked, the created index will need to be activated manually.
 We may now use the `es` and `es_street` matching strategies with `AU` addresses. See [below](#elasticsearch-matching-strategy)
 for an example of its usage.
 
+#### Instructions for US import
+
+1. Go to the [open addresses](https://openaddresses.io/) download center and download the collection-us-{region}.zip
+files for each of the four regions (west, midwest, northeast, south).
+
+2. Run the US create state geojson script to create a statewide geojson.gz file for each state
+```
+bin/us_create_state_geojson execute /path/to/us_collection_zips /path/to/output_dir
+```
+
+3. Start your app with `rails s` and navigate to `http://localhost:3000/maintenance_tasks`. There is a task only used
+for the US import called `Maintenance::AtlasEngine::UsGeoJsonDirectoryImportTask`
+
+4. Parameterize the `UsGeoJsonDirectoryImportTask` with the output directory that contains all of the `{state}-statewide.geojson.gz` files created in step 2.
+
+5. Once properly parameterized, click run. The process will initialize a `country_import` and should succeed immediately.
+
+6. Navigate to `http://localhost:3000/country_imports` to track the progress of the country import. Once the import is complete
+and the US data is in mysql the rest of the process for creating the elasticsearch index and verifying should be the same as above.
+
 ## Elasticsearch Matching Strategy
 
 Once we have successfully created and activated an elasticsearch index using open address data, we may now use

data/app/countries/atlas_engine/gb/address_validation/es/query_builder.rb

@@ -16,7 +16,7 @@ module AtlasEngine
             ).void
           end
           def initialize(address, parsings, profile)
-            super(address, parsings, profile)
+            super
 
             @parsings = T.let(
               AtlasEngine::Gb::ValidationTranscriber::FullAddressParser

data/app/countries/atlas_engine/gg/address_importer/corrections/open_address/city_corrector.rb

@@ -0,0 +1,38 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Gg
+    module AddressImporter
+      module Corrections
+        module OpenAddress
+          class CityCorrector
+            class << self
+              extend T::Sig
+
+              CITY_ALIASES_MAPPING = {
+                "St. Sampson" => ["Saint Samsaon"],
+                "St. Saviour" => ["Saint-Sauveur", "Saint Sauveux"],
+                "St. Peter Port" => ["Saint-Pierre Port"],
+                "St. Andrew" => ["Saint Andri", "Saint-André-de-la-Pommeraye"],
+                "St. Pierre Du Bois" => ["St. Peter's", "St. Pierre"],
+                "Castel" => ["Lé Casté", "Sainte-Marie-du-Câtel"],
+                "Forest" => ["Le Fôret", "La Fouarêt"],
+                "Torteval" => ["Tortévas"],
+                "Vale" => ["Lé Vale", "Le Valle"],
+              }
+
+              sig { params(address: Hash).void }
+              def apply(address)
+                city = address[:city].first
+                if CITY_ALIASES_MAPPING.include?(city)
+                  address[:city] += CITY_ALIASES_MAPPING[city]
+                end
+              end
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/app/countries/atlas_engine/gg/country_profile.yml

@@ -6,6 +6,10 @@ validation:
   address_parser: AtlasEngine::Gg::ValidationTranscriber::AddressParser
   restrictions:
     - class: AtlasEngine::Gg::AddressValidation::Validators::FullAddress::Restrictions::UnsupportedCity
+ingestion:
+  correctors:
+      open_address:
+        - AtlasEngine::Gg::AddressImporter::Corrections::OpenAddress::CityCorrector
   exclusions:
     city:
       - AtlasEngine::Gg::AddressValidation::Validators::FullAddress::Exclusions::City

data/app/countries/atlas_engine/gg/synonyms.yml

@@ -0,0 +1,3 @@
+city_synonyms:
+- st, saint
+- ste, sainte

data/app/countries/atlas_engine/gg/validation_transcriber/address_parser.rb

@@ -10,16 +10,30 @@ module AtlasEngine
         CITY = %r{
           (?<city>
             st\.?\s?saviour[']?[s]?|
+            saint[-|\s]sauveur|
+            saint\s?sauveux|
             st\.?\s?sampson[']?[s]?|
+            saint\s?samsaon|
             st\.?\s?andrew[']?[s]?|
+            saint\s?andri|
+            saint[-|\s]andr[é|e][-|\s]de[-|\s]la[-|\s]pommeraye|
             st\.?\s?martin[']?[s]?|
             st\.?\s?peter[']?[s]?\s?port|
+            saint[-|\s]pierre\s?port|
             st\.?\s?peter[']?[s]?|
             st\.?\s?pierre\s?du\s?bois|
+            st\.?\s?pierre|
             vale|
+            l[é|e]\s?vale|
+            le\s?valle|
             torteval|
+            tort[é|e]vas|
             castel|
-            forest
+            l[é|e]\s?cast[é|e]|
+            sainte[-|\s]marie[-|\s]du[-|\s]c[â|a]tel|
+            forest|
+            le\s?f[ô|o]ret|
+            la\s?fouar[ê|e]t
           )
         }ix
 

data/app/countries/atlas_engine/si/address_importer/open_address/mapper.rb

@@ -10,7 +10,7 @@ module AtlasEngine
             params(feature: AtlasEngine::AddressImporter::OpenAddress::Feature).returns(T::Hash[Symbol, T.untyped])
           end
           def map(feature)
-            super(feature).merge(region4: feature["properties"]["district"])
+            super.merge(region4: feature["properties"]["district"])
           end
         end
       end

data/app/countries/atlas_engine/us/address_importer/open_address/filter.rb

@@ -0,0 +1,28 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Us
+    module AddressImporter
+      module OpenAddress
+        class Filter
+          extend T::Sig
+          include AtlasEngine::AddressImporter::OpenAddress::Filter
+
+          def initialize(country_import:); end
+
+          sig { override.params(feature: AtlasEngine::AddressImporter::OpenAddress::Feature).returns(T::Boolean) }
+          def filter(feature)
+            # Only consider features with lat lon geometry
+            geometry = feature["geometry"]
+            if geometry.present? && geometry["type"] == "Point"
+              return false if geometry["coordinates"].size > 2
+            end
+
+            true
+          end
+        end
+      end
+    end
+  end
+end

data/app/countries/atlas_engine/us/address_importer/open_address/mapper.rb

@@ -0,0 +1,66 @@
+# typed: true
+# frozen_string_literal: true
+
+module AtlasEngine
+  module Us
+    module AddressImporter
+      module OpenAddress
+        class Mapper < AtlasEngine::AddressImporter::OpenAddress::DefaultMapper
+          ORDINAL_REGEX = /\b(\d+)(?:\s+)(st|nd|rd|th)\b/i
+
+          sig do
+            params(feature: AtlasEngine::AddressImporter::OpenAddress::Feature).returns(T::Hash[Symbol, T.untyped])
+          end
+          def map(feature)
+            city, street, number, unit, postcode = feature["properties"].values_at(
+              "city",
+              "street",
+              "number",
+              "unit",
+              "postcode",
+            )
+            {
+              source_id: openaddress_source_id(feature),
+              locale: @locale,
+              country_code: @country_code,
+              province_code: @province_code, # region is inconsistently set, override with passed in province_code
+              # Omitted: region1..4
+              city: sanitize_city(city),
+              suburb: nil,
+              zip: postcode.first(5), # truncate zip+4 data
+              street: sanitize_street(street),
+              building_and_unit_ranges: housenumber_and_unit(number, unit),
+              latitude: geometry(feature)&.at(1),
+              longitude: geometry(feature)&.at(0),
+            }
+          end
+
+          private
+
+          sig { params(street: T.nilable(String)).returns(T.nilable(String)) }
+          def sanitize_street(street)
+            combine_ordinal_string(strip_extra_spaces(street&.downcase))&.titleize
+          end
+
+          sig { params(city: T.nilable(String)).returns(T::Array[String]) }
+          def sanitize_city(city)
+            return [] if city.nil?
+
+            city.split("/").map { |c| c.titleize.strip }
+          end
+
+          sig { params(text: T.nilable(String)).returns(T.nilable(String)) }
+          def combine_ordinal_string(text)
+            text&.gsub!(ORDINAL_REGEX) { "#{::Regexp.last_match(1)}#{::Regexp.last_match(2)}" }
+            text
+          end
+
+          sig { params(text: T.nilable(String)).returns(T.nilable(String)) }
+          def strip_extra_spaces(text)
+            text&.strip&.gsub(/\s+/, " ")
+          end
+        end
+      end
+    end
+  end
+end

data/app/countries/atlas_engine/us/country_profile.yml

@@ -1,10 +1,14 @@
 id: US
-validation:
-  enabled: true
-  default_matching_strategy: es_street
-  address_parser: AtlasEngine::ValidationTranscriber::AddressParserNorthAmerica
 ingestion:
   settings:
     number_of_shards: "7"
     min_zip_edge_ngram: "1"
     max_zip_edge_ngram: "10"
+  post_address_mapper:
+    open_address: AtlasEngine::Us::AddressImporter::OpenAddress::Mapper
+validation:
+  enabled: true
+  default_matching_strategy: es_street
+  address_parser: AtlasEngine::ValidationTranscriber::AddressParserNorthAmerica
+open_address:
+  filter: AtlasEngine::Us::AddressImporter::OpenAddress::Filter

data/app/countries/atlas_engine/us/jobs/address_importer/combined_import_job.rb

@@ -0,0 +1,120 @@
+# typed: false
+# frozen_string_literal: true
+
+require "zip"
+
+module AtlasEngine
+  module Us
+    module Jobs
+      module AddressImporter
+        class CombinedImportJob < ApplicationJob
+          include ::AtlasEngine::AddressImporter::ImportLogHelper
+          discard_on Exception
+          rescue_from(ArgumentError) { raise }
+
+          COUNTRY_CODE = "US"
+          LOCALE = "en"
+          US_STATES = [
+            "AK",
+            "AL",
+            "AR",
+            "AZ",
+            "CA",
+            "CO",
+            "CT",
+            "DC",
+            "DE",
+            "FL",
+            "GA",
+            "HI",
+            "IA",
+            "ID",
+            "IL",
+            "IN",
+            "KS",
+            "KY",
+            "LA",
+            "MA",
+            "MD",
+            "ME",
+            "MI",
+            "MN",
+            "MO",
+            "MS",
+            "MT",
+            "NC",
+            "ND",
+            "NE",
+            "NH",
+            "NJ",
+            "NM",
+            "NV",
+            "NY",
+            "OH",
+            "OK",
+            "OR",
+            "PA",
+            "RI",
+            "SC",
+            "SD",
+            "TN",
+            "TX",
+            "UT",
+            "VA",
+            "VT",
+            "WA",
+            "WI",
+            "WV",
+            "WY",
+          ]
+
+          def perform(geojson_directory:)
+            country_import = AtlasEngine::CountryImport.create!(country_code: COUNTRY_CODE)
+            country_import.start!
+
+            import_log_info(
+              country_import: country_import,
+              message: "Starting import for #{COUNTRY_CODE} from #{geojson_directory}",
+              notify: true,
+            )
+
+            jobs_to_run = job_list(geojson_directory, country_import)
+            first_job = jobs_to_run.shift
+            first_job[:job_name].perform_later(**first_job[:job_args].merge(followed_by: jobs_to_run))
+          end
+
+          private
+
+          def job_list(geojson_directory, country_import)
+            job_list = [
+              clear_records_job(country_import),
+            ]
+
+            US_STATES.each do |us_state|
+              job_list <<
+                {
+                  job_name: ::AtlasEngine::AddressImporter::OpenAddress::GeoJsonImportJob,
+                  job_args: {
+                    country_code: COUNTRY_CODE,
+                    province_code: us_state.upcase,
+                    locale: LOCALE,
+                    geojson_file_path: File.join(geojson_directory, "us-#{us_state.downcase}-statewide.geojson.gz"),
+                    country_import_id: country_import.id,
+                  },
+                }
+            end
+
+            job_list
+          end
+
+          def clear_records_job(country_import)
+            {
+              job_name: AtlasEngine::AddressImporter::ClearRecordsJob,
+              job_args: { country_code: COUNTRY_CODE, country_import_id: country_import.id },
+            }
+          end
+        end
+      end
+    end
+  end
+end

data/app/jobs/atlas_engine/address_importer/open_address/geo_json_import_job.rb

@@ -24,7 +24,7 @@ module AtlasEngine
         extend T::Sig
         include HandlesInterruption
         include PreparesGeoJsonFile
-        attr_reader :geojson_path, :country_import, :country_code, :loader, :transformer
+        attr_reader :geojson_path, :country_import, :country_code, :province_code, :loader, :transformer
 
         CHUNK_SIZE = 10_000
         REPORT_STEP = 5
@@ -36,10 +36,11 @@ module AtlasEngine
         def setup_and_download(&block)
           @loader = Loader.new
           @country_code = argument(:country_code)
+          @province_code = argument(:province_code)
           @geojson_path = Pathname.new(argument(:geojson_file_path))
           @locale = argument(:locale)&.downcase
           @country_import = CountryImport.find(argument(:country_import_id))
-          @transformer = Transformer.new(country_import: country_import, locale: @locale)
+          @transformer = Transformer.new(country_import: country_import, province_code: @province_code, locale: @locale)
 
           import_log_info(
             country_import: country_import,

data/app/models/atlas_engine/address_importer/open_address/default_mapper.rb

@@ -7,9 +7,10 @@ module AtlasEngine
       class DefaultMapper
         extend T::Sig
         include FeatureHelper
-        sig { params(country_code: String, locale: T.nilable(String)).void }
-        def initialize(country_code:, locale: nil)
+        sig { params(country_code: String, province_code: T.nilable(String), locale: T.nilable(String)).void }
+        def initialize(country_code:, province_code: nil, locale: nil)
           @country_code = country_code
+          @province_code = province_code
           @locale = locale
         end
 
@@ -27,7 +28,7 @@ module AtlasEngine
             source_id: openaddress_source_id(feature),
             locale: @locale,
             country_code: @country_code,
-            province_code: nil,
+            province_code: @province_code,
             region1: region,
             # Don't titleize. The sources have proper capitalization, and it's a problem for cities like
             # 's-Graveland, which would get titleized to "'S Graveland" which is wrong.

data/app/models/atlas_engine/address_importer/open_address/transformer.rb

@@ -8,11 +8,12 @@ module AtlasEngine
         extend T::Sig
         extend T::Helpers
 
-        def initialize(country_import:, locale: nil)
+        def initialize(country_import:, province_code: nil, locale: nil)
           @country_code = country_import.country_code
           @locale = locale
+          @province_code = province_code
           @mapper = CountryProfile.for(@country_code).ingestion.post_address_mapper("open_address").new(
-            country_code: @country_code, locale: @locale,
+            country_code: @country_code, province_code: @province_code, locale: @locale,
           )
           @corrector = AddressImporter::Corrections::Corrector.new(country_code: @country_code, source: "open_address")
           @validator = AddressImporter::Validation::Wrapper.new(

data/app/models/atlas_engine/address_validation/validator.rb

@@ -114,6 +114,8 @@ module AtlasEngine
         local_concerns = {}
         cache = Validators::Predicates::Cache.new(pipeline_address)
         @predicate_pipeline.pipeline.each do |config|
+          break if local_concerns[:country].present?
+
           local_concerns[config.field] = [] if local_concerns[config.field].nil?
           next if local_concerns[config.field].present?
 

data/app/models/atlas_engine/address_validation/validators/full_address/unmatched_field_concern_builder.rb

@@ -8,7 +8,7 @@ module AtlasEngine
         class UnmatchedFieldConcernBuilder
           extend T::Sig
           include ConcernFormatter
-          attr_reader :address, :component, :matched_components, :unmatched_field
+          attr_reader :address, :unmatched_component, :matched_components, :unmatched_field
 
           COMPONENTS_TO_LABELS = {
             zip: "ZIP",
@@ -31,9 +31,9 @@ module AtlasEngine
           end
           def initialize(unmatched_component, matched_components, address, unmatched_field = nil)
             @address = address
-            @component = unmatched_component
+            @unmatched_component = unmatched_component
             @matched_components = matched_components
-            @unmatched_field = unmatched_field
+            @unmatched_field = unmatched_field || unmatched_component
           end
 
           sig do
@@ -61,7 +61,7 @@ module AtlasEngine
 
           sig { returns(Symbol) }
           def code
-            "#{shortened_component_name}_inconsistent".to_sym
+            "#{unmatched_component_name}_inconsistent".to_sym
           end
 
           sig { returns(T::Array[Symbol]) }
@@ -69,21 +69,14 @@ module AtlasEngine
             [field_name]
           end
 
-          sig { returns(T::Array[String]) }
-          def valid_address_component_values
-            matched_components.last(2).map do |component|
-              component == :province_code ? province_name : address[component]
-            end
-          end
-
           sig { returns(Symbol) }
-          def shortened_component_name
-            SHORTENED_COMPONENT_NAMES[component] || component
+          def unmatched_component_name
+            SHORTENED_COMPONENT_NAMES[unmatched_component] || unmatched_component
           end
 
           sig { returns(Symbol) }
           def field_name
-            unmatched_field || shortened_component_name
+            SHORTENED_COMPONENT_NAMES[unmatched_field] || unmatched_field
           end
         end
       end

data/app/models/atlas_engine/address_validation/validators/predicates/city/present.rb

@@ -9,7 +9,6 @@ module AtlasEngine
           class Present < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return unless @cache.country.country?
               return if @cache.country.field(key: :city).autofill(locale: :en).present?
 
               build_concern if @address.city.blank?

data/app/models/atlas_engine/address_validation/validators/predicates/no_emojis.rb

@@ -8,8 +8,6 @@ module AtlasEngine
         class NoEmojis < Predicate
           sig { override.returns(T.nilable(Concern)) }
           def evaluate
-            return unless @cache.country.country?
-
             build_concern if contains_blocked_codepoints?(address.send(@field))
           end
 

data/app/models/atlas_engine/address_validation/validators/predicates/no_html_tags.rb

@@ -8,8 +8,6 @@ module AtlasEngine
         class NoHtmlTags < Predicate
           sig { override.returns(T.nilable(Concern)) }
           def evaluate
-            return unless @cache.country.country?
-
             build_concern if contains_html_tags(address.send(@field))
           end
 

data/app/models/atlas_engine/address_validation/validators/predicates/no_url.rb

@@ -8,8 +8,6 @@ module AtlasEngine
         class NoUrl < Predicate
           sig { override.returns(T.nilable(Concern)) }
           def evaluate
-            return unless @cache.country.country?
-
             build_concern if contains_url(address.send(@field))
           end
 

data/app/models/atlas_engine/address_validation/validators/predicates/not_exceed_max_length.rb

@@ -9,8 +9,6 @@ module AtlasEngine
           MAX_COMPONENT_LENGTH = 255
           sig { override.returns(T.nilable(Concern)) }
           def evaluate
-            return unless @cache.country.country?
-
             build_concern if address.send(@field).to_s.length > MAX_COMPONENT_LENGTH
           end
 

data/app/models/atlas_engine/address_validation/validators/predicates/phone/valid.rb

@@ -9,8 +9,6 @@ module AtlasEngine
           class Valid < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return if @address.country_code.blank?
-              return unless @cache.country.country?
               return if @address.phone.blank?
 
               phone = Worldwide::Phone.new(number: @address.phone, country_code: @address.country_code)

data/app/models/atlas_engine/address_validation/validators/predicates/province/exists.rb

@@ -9,7 +9,6 @@ module AtlasEngine
           class Exists < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return unless @cache.country.country?
               return if address.province_code.present? ||
                 country_has_no_provinces ||
                 @cache.country.province_optional?

data/app/models/atlas_engine/address_validation/validators/predicates/province/valid_for_country.rb

@@ -9,7 +9,6 @@ module AtlasEngine
           class ValidForCountry < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return unless @cache.country.country?
               return if address.province_code.blank?
               return if @cache.country.zones.none?(&:province?)
               return if @cache.country.hide_provinces_from_addresses

data/app/models/atlas_engine/address_validation/validators/predicates/street/building_number_in_address1.rb

@@ -9,8 +9,6 @@ module AtlasEngine
           class BuildingNumberInAddress1 < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return unless @cache.country.country?
-
               return unless @cache.country.building_number_required
               return if @cache.country.building_number_may_be_in_address2
 

data/app/models/atlas_engine/address_validation/validators/predicates/street/building_number_in_address1_or_address2.rb

@@ -9,7 +9,6 @@ module AtlasEngine
           class BuildingNumberInAddress1OrAddress2 < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return unless @cache.country.country?
               return unless @cache.country.building_number_required
               return unless @cache.country.building_number_may_be_in_address2
               return if contains_number?(T.must(@address.address1)) || contains_number?(@address.address2)

data/app/models/atlas_engine/address_validation/validators/predicates/street/present.rb

@@ -9,8 +9,6 @@ module AtlasEngine
           class Present < Predicate
             sig { override.returns(T.nilable(Concern)) }
             def evaluate
-              return unless @cache.country.country?
-
               build_concern if @address.address1.blank?
             end
 

data/app/models/atlas_engine/address_validation/validators/predicates/zip/zip_base.rb

@@ -11,7 +11,6 @@ module AtlasEngine
 
             sig { returns(T.nilable(T::Boolean)) }
             def concerning?
-              return false unless @cache.country.country?
               return false unless @cache.country.has_zip?
               return false unless @cache.country.zip_required?
 

data/app/tasks/maintenance/atlas_engine/geo_json_import_task.rb

@@ -12,9 +12,6 @@ module Maintenance
       # ISO3166 two-letter country code.
       attribute :country_code, :string
       validates :country_code, presence: true
-      # Filename to import. When running in staging or production, the worker expects to find
-      # this file in the relevant GCS bucket, configured in `config/storage/{environment}.yml`
-      # It must be placed under `openaddress/` with the same filename.
       attribute :geojson_file_path, :string
       attribute :locale, :string
 

data/app/tasks/maintenance/atlas_engine/us_geo_json_directory_import_task.rb

@@ -0,0 +1,20 @@
+# typed: false
+# frozen_string_literal: true
+
+module Maintenance
+  module AtlasEngine
+    class UsGeoJsonDirectoryImportTask < MaintenanceTasks::Task
+      include ::AtlasEngine::HandlesBlob
+
+      no_collection
+
+      attribute :geojson_directory, :string
+
+      def process
+        ::AtlasEngine::Us::Jobs::AddressImporter::CombinedImportJob.perform_later(
+          geojson_directory: geojson_directory.strip,
+        )
+      end
+    end
+  end
+end

data/lib/atlas_engine/version.rb

@@ -2,5 +2,5 @@
 # frozen_string_literal: true
 
 module AtlasEngine
-  VERSION = "0.7.0"
+  VERSION = "1.0.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: atlas_engine
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Shopify
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-03-14 00:00:00.000000000 Z
+date: 2024-05-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: annex_29
@@ -178,6 +178,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: sprockets-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: state_machines-activerecord
   requirement: !ruby/object:Gem::Requirement
@@ -305,9 +319,11 @@ files:
 - app/countries/atlas_engine/gb/country_profile.yml
 - app/countries/atlas_engine/gb/validation_transcriber/full_address_parser.rb
 - app/countries/atlas_engine/gb/validation_transcriber/parsed_address.rb
+- app/countries/atlas_engine/gg/address_importer/corrections/open_address/city_corrector.rb
 - app/countries/atlas_engine/gg/address_validation/validators/full_address/exclusions/city.rb
 - app/countries/atlas_engine/gg/address_validation/validators/full_address/restrictions/unsupported_city.rb
 - app/countries/atlas_engine/gg/country_profile.yml
+- app/countries/atlas_engine/gg/synonyms.yml
 - app/countries/atlas_engine/gg/validation_transcriber/address_parser.rb
 - app/countries/atlas_engine/ie/country_profile.yml
 - app/countries/atlas_engine/it/address_importer/corrections/open_address/city_corrector.rb
@@ -362,7 +378,10 @@ files:
 - app/countries/atlas_engine/si/validation_transcriber/address_parser.rb
 - app/countries/atlas_engine/tt/address_importer/open_address/mapper.rb
 - app/countries/atlas_engine/tt/country_profile.yml
+- app/countries/atlas_engine/us/address_importer/open_address/filter.rb
+- app/countries/atlas_engine/us/address_importer/open_address/mapper.rb
 - app/countries/atlas_engine/us/country_profile.yml
+- app/countries/atlas_engine/us/jobs/address_importer/combined_import_job.rb
 - app/countries/atlas_engine/us/synonyms.yml
 - app/graphql/atlas_engine/errors/locale_unsupported_error.rb
 - app/graphql/atlas_engine/schema.graphql
@@ -546,6 +565,7 @@ files:
 - app/models/atlas_engine/street.rb
 - app/tasks/maintenance/atlas_engine/elasticsearch_index_create_task.rb
 - app/tasks/maintenance/atlas_engine/geo_json_import_task.rb
+- app/tasks/maintenance/atlas_engine/us_geo_json_directory_import_task.rb
 - app/views/atlas_engine/connectivity/index.html.erb
 - app/views/atlas_engine/country_imports/index.html.erb
 - app/views/atlas_engine/country_imports/show.html.erb
@@ -595,7 +615,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.6
+rubygems_version: 3.5.10
 signing_key: 
 specification_version: 4
 summary: Address Validation API