geolocation_service 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 83121b2e17c3649b2f5f070dda4749a3efd7724f0a33de35ad857b7676df7727
-  data.tar.gz: 97bacb951601c4a2faf7042b3db24381a602aa9d35347142b745b8d46f0544fe
+  metadata.gz: 425afb60c0457918c20ce1dc77821fe51f72a5c3956baad224b2282628cf4ae7
+  data.tar.gz: 2b21fb3f283c1c9ae58d479c3ab1fa24ad8c989e1f741741f384f88f83c066f4
 SHA512:
-  metadata.gz: eef1ce0a4df76d0a350e99b0297f7171a2e5b5fb9011798cabe982f33ec3373163239c91c7c12bd5de3c2799e9975c7f43ce0c2bf197d7e5ad9b6f435ee57747
-  data.tar.gz: f8611c216add9ee4ab34dc8eb0c70006f71142ef777517825da5456927d14aa4c07e00276961e81d90be81140edd94ce7ce4ff20125769d39fc3b7a40568aef4
+  metadata.gz: f4fdd1994c0f4ba8fa758da9440a3510e679ea737d0431e1e312fffc34bfb925020a29b10a6574f14f82f63f8681a1b7fce541ce6012f1ba1ccd008c67afba77
+  data.tar.gz: 40636b1b37251ed3b00eed41246fc9cf238e5b78db80b1a5d3175ef432017876b0041c2a5f295591279c9cd77690f4657bbe3c9c20e6ead478d588e2dd82b147

data/README.md

@@ -1,10 +1,20 @@
-# GeolocationService
-Short description and motivation.
+# FindHotel Coding Challenge
 
-## Usage
-How to use my plugin.
+The FindHotel coding challenge consists of two parts, a library and a REST API application:
+
+1. A library with two main features:
+    * A service that parses the CSV file containing the raw data and persists it in a database;
+    * An interface to provide access to the geolocation data (model layer);
+2. A REST API that uses the aforementioned library to expose the geolocation data.
+
+This repository contains my solution to the library. You can find my solution to the REST API application here: https://github.com/jalerson/geolocation_api.
+
+## Geolocation Service
+
+The library was developed as a [Rails Engine gem](https://guides.rubyonrails.org/engines.html), which can be easily and seamlessly integrated into any Rails application.
+
+### Installation
 
-## Installation
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -12,17 +22,84 @@ gem 'geolocation_service'
 ```
 
 And then execute:
+
 ```bash
 $ bundle
 ```
 
-Or install it yourself as:
+Install the gem's migrations:
+
 ```bash
-$ gem install geolocation_service
+$ rails geolocation_service_engine:install:migrations
+```
+
+Execute pending migrations:
+
+```bash
+$ rake db:migrate
+```
+
+### Usage
+
+The gem provides four new models: `Ip`, `City`, `Country` and `Location`.
+
+![Models and associations](https://i.ibb.co/Dbb2nTH/geocoding-service-erd.png)
+
+In order to import data, you must use the `GeolocationService::Services::ImportBulkDataService`.
+
+```ruby
+GeolocationService::Services::ImportBulkDataService.call(file_path: 'path/to/data_file.csv')
+```
+
+The service returns a [Dry::Monad::Result](https://dry-rb.org/gems/dry-monads/result/) indicating the success or failure of the importing operation.
+
+```ruby
+result = GeolocationService::Services::ImportBulkDataService.call(file_path: 'path/to/data_file.csv')
+
+if result.success?
+  # do something...
+else
+  # do something else...
+end
 ```
 
-## Contributing
-Contribution directions go here.
+In a successful importing operation, the `result` will contain an instance of `GeolocationService::ImportResult`, which has:
+
+- `imported_records`: number of imported records
+- `invalid_records`: number of invalid records
+- `time_consumed`: time consumed in seconds
+
+In a failure importing operation, the `result` will contain an error/exception.
+
+```ruby
+result = GeolocationService::Services::ImportBulkDataService.call(file_path: 'path/to/data_file.csv')
+
+if result.success?
+  import_results = result.value!
+  Rails.logger.info "Records imported in #{import_results.time_consumed} seconds"
+else
+  error = result.failure
+  Rails.logger.error error.message
+end
+```
+
+### Design decisions
+
+The main guidance for design decisions in this project was: **provide the best importing performance while keeping the database normalized**.
+
+In order to achieve the best importing performance using a normalized database, experiments were conduct to seek for the best performance of (a) converting the CSV data to a format/representation that could be validated and stored into the database, (b) data validation and (c) actually store the data into the database. In each case, the alternatives considered were:
+
+**(a) CSV data conversion:** ActiveRecord instances, simple Ruby classes (no ActiveRecord), Arrays/Hashes and Structs.
+
+**(b) data validation:** [ActiveRecord validations](https://guides.rubyonrails.org/active_record_validations.html) or [contracts/schemas](https://dry-rb.org/gems/dry-validation/1.0/).
+
+**(c) store into the database**: in order to keep the performance as best as possible, the alternatives considered are limited to those which persist a set of records in a single `INSERT` statement: [activerecord-import gem](https://github.com/zdennis/activerecord-import) or writing and sending the SQL statements to the database.
+
+After several experiments with different combinations, the chosen approach is using Structs, contracts/schemas and send SQL statements directly to the database. This particular combination presented a great performance when importing one million records in approximately 4 minutes while avoiding duplicates and keeping a clean code.
+
+## Trade-offs
+
+In order to keep the importing performance as best as possible, two design decisions have consequences which users need to be aware of.
 
-## License
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+- **Memory usage:** when importing a set of records, the service will load all existing records in memory and also add new records. This way the service avoids creating duplicated records.
+- **id (primary key) needs to be manually set:** in order to guarantee the proper relationship constraints between records in the database when importing records, the `id` must be set manually in all tables, except `locations`.

data/lib/geolocation_service/services/import_bulk_data_service.rb

@@ -43,10 +43,10 @@ module GeolocationService::Services
 
       GeolocationService::ImportResult.new(
         imported_records: {
-          ip: @ip_count,
-          city: @city_count,
-          country: @country_count,
-          location: @new_records[:location].count
+          ip: @new_records[:ip].values.size,
+          city: @new_records[:city].values.size,
+          country: @new_records[:country].values.size,
+          location: @new_records[:location].size
         },
         invalid_records: @invalid_records,
         time_consumed: (Time.zone.now - start_time)
@@ -74,8 +74,8 @@ module GeolocationService::Services
     def build_city(row, country)
       return if row['city'].blank?
 
-      if validate(:city, name: row['city'], country_code: country[:id]).success?
-        new_city = Structs::CityStruct.new(@city_count, row['city'], country[:id])
+      if validate(:city, name: row['city'], country_id: country&.id).success?
+        new_city = Structs::CityStruct.new(@city_count, row['city'], country&.id)
         @new_records[:city][row['city'].downcase] = new_city
         @city_count += 1
         return new_city
@@ -117,9 +117,9 @@ module GeolocationService::Services
     end
 
     def load_new_records
-      @ip_count = 0
-      @city_count = 0
-      @country_count = 0
+      @ip_count = Ip.count == 0 ? 0 : Ip.last.id + 1
+      @city_count = City.count == 0 ? 0 : City.last.id + 1
+      @country_count = Country.count == 0 ? 0 : Country.last.id + 1
       @invalid_records = 0
       @new_records = {ip: {}, location: [], city: {}, country: {}}
     end

data/lib/geolocation_service/version.rb

@@ -1,3 +1,3 @@
 module GeolocationService
-  VERSION = '0.1.0'
+  VERSION = '0.1.2'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: geolocation_service
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.2
 platform: ruby
 authors:
 - Jalerson Lima
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-09-21 00:00:00.000000000 Z
+date: 2019-09-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails