address_tokens 0.1.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: dc428acad65025604ed184b2c5c243b48715fed99e00dc5b3bfee54208e169f5
+  data.tar.gz: 791c48d742010fc338cae65e03471796b98867dce09d07d3697adad638c150fc
+SHA512:
+  metadata.gz: afc4a676c93f9e0d2aa6be9431acc7cdebb0113c192e93cde33f4053d3cf405e09024be3ed211705d514a02a1e1f61395b40f91c444271dba7e228b6dbc00ec2
+  data.tar.gz: 5d49a5903fb85a89562a52151fd55aa394ec1528ea22558ccc6a4121e516c79ec7ef93c4a3aabdf56755ab3c6d3450e0a72e3ee280e6f65a3fc1fb2dc780a510

data/.gitignore

@@ -0,0 +1,8 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.5.0
+before_install: gem install bundler -v 1.16.1

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in address_tokens.gemspec
+gemspec

data/README.md

@@ -0,0 +1,150 @@
+# AddressTokens
+
+Gem to find tokens on address strings. Can identify address, city and state name
+and abbreviation.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'address_tokens'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install address_tokens
+
+## Usage
+
+Always hated when you get some address like
+
+```
+88 Colin P Kelly Jr St San Francisco, CA 94107
+```
+
+and don't know where is the address, where is the state, where is the city name?
+
+Ok, me too. So I made this class.
+
+We need state and city data to find out what the tokens are. To load the date,
+we'll need YAML files like [I have on this
+repo](https://github.com/taq/brstatescities). They follow formats like these:
+
+```
+# states.yml:
+---
+CA: California
+
+# cities.yml
+CA:
+- San Francisco
+```
+
+Finding by exact match is fast and piece of cake. The problem is where we have
+some different ways to write or abbreviate the city names like we have here on
+Brazil. For example, my city, São José do Rio Preto, can have some forms:
+
+1. São José do Rio Preto
+2. São José Rio Preto
+3. S. J. do Rio Preto
+4. S. J. R. Preto
+
+And keep going. But I think I could find some ways to find that.
+
+### Finding stuff
+
+First, we create a Finder object:
+
+```ruby
+finder = AddressTokens::Finder.new('88 Colin P Kelly Jr St San Francisco, CA 94107')
+```
+
+Load the cities and states:
+
+```ruby
+finder.load(:states, '/tmp/states.yml')
+finder.load(:cities, '/tmp/cities.yml')
+finder.find
+```
+
+And ask to find:
+
+```ruby
+matches = finder.find
+```
+
+We'll get something like this:
+
+```ruby
+p matches
+{
+   :state_abbr     => "CA", 
+   :state_name     => "California", 
+   :state_start_at => 36,
+   :city_name      => "San Francisco",
+   :city_string    => "San Francisco",
+   :city_start_at  => 23, 
+   :address        => "88 Colin P Kelly Jr St"
+}
+```
+The `start_at` values shows where the strings were found. The `city_string` is 
+the way the city name was found.
+
+### Custom options
+
+As we saw, the default city and states separator on USA is a comma (','), but on
+Brasil is a hyphen ('-'), so, **before asking to find the address**, we must
+change it:
+
+```ruby
+finder.state_separator = '-'
+```
+
+Using a Brazilian address:
+
+```ruby
+finder = AddressTokens::Finder.new('Rua Tratado de Tordesihas, 88, Pq. Estoril,
+S. J. do Rio Preto - SP')
+finder.state_separator = '-'
+finder.load(:states, '/tmp/states.yml')
+finder.load(:cities, '/tmp/cities.yml')
+finder.find
+```
+
+will return:
+
+```ruby
+{
+   :state_abbr       => "SP", 
+   :state_name       => "São Paulo", 
+   :state_start_at   => 63, 
+   :city_name        => "São José do Rio Preto", 
+   :city_string      => "S. J. do Rio Preto", 
+   :city_start_at    => 44, 
+   :address          => "Rua Tratado de Tordesihas, 88, Pq. Estoril,"
+}
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run
+`rake test` to run the tests. You can also run `bin/console` for an interactive
+prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To
+release a new version, update the version number in `version.rb`, and then run
+`bundle exec rake release`, which will create a git tag for the version, push
+git commits and tags, and push the `.gem` file to
+[rubygems.org](https://rubygems.org).
+
+Please run the tests using `rake test`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/taq/address_tokens.

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task :default => :test

data/address_tokens.gemspec

@@ -0,0 +1,30 @@
+
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "address_tokens/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "address_tokens"
+  spec.version       = AddressTokens::VERSION
+  spec.authors       = ["Eustaquio Rangel"]
+  spec.email         = ["taq@eustaquiorangel.com"]
+
+  spec.summary       = %q{Find address tokens on a string}
+  spec.description   = %q{Always want to find where address, city and state are on a string? Use this gem.}
+  spec.homepage      = "http://github.com/taq/address_tokens"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.16"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "minitest", "~> 5.0"
+  spec.add_development_dependency "i18n"
+
+  spec.signing_key = '/home/taq/.gemcert/gem-private_key.pem'
+  spec.cert_chain = ['gem-public_cert.pem']
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "address_tokens"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/address_tokens/version.rb

@@ -0,0 +1,3 @@
+module AddressTokens
+  VERSION = "0.1.3"
+end

data/lib/address_tokens.rb

@@ -0,0 +1,6 @@
+require 'address_tokens/version'
+require 'finder'
+require 'matcher'
+
+module AddressTokens
+end

data/lib/city_not_found.rb

@@ -0,0 +1,5 @@
+module AddressTokens
+  class CityNotFound < StandardError
+  end
+end
+

data/lib/finder.rb

@@ -0,0 +1,57 @@
+require 'yaml'
+require 'i18n'
+
+module AddressTokens
+  class Finder
+    attr_accessor :states, :cities, :state_separator, :string
+    attr_reader :city_tokens
+
+    def initialize(str)
+      raise Exception, 'String is null or empty' if str.strip.size < 1
+      @string, @states, @cities, = str, {}, {}
+      I18n.config.available_locales = :en
+      @state_separator = ','
+    end
+
+    def load(var, file)
+      load_file(var, file)
+    end
+
+    def find
+      raise ArgumentError, 'No states found' if @states.size == 0
+      raise ArgumentError, 'No cities found' if @cities.size == 0
+      transliterate_cities
+      Matcher.new(self).match(@string)
+    end
+
+    private
+
+    def load_file(var, file)
+      raise IOError, "File #{file} was not found" if !File.exist?(file)
+      data = YAML.load(File.read(file))
+      raise TypeError, "File #{file} is not a valid YAML file" if !data.kind_of?(Hash)
+      instance_variable_set("@#{var}", data)
+    end
+
+    def transliterate_cities
+      @city_tokens = {}
+
+      for state, cities in @cities
+        @city_tokens[state] = []
+        
+        for city in cities
+          tokens = []
+          city.gsub!(/\s{2,}/, ' ')
+          tokens << city
+
+          tdown = I18n.transliterate(city).downcase
+          tokens << tdown
+          
+          splitted = tdown.split
+          tokens << splitted if splitted.size > 1
+          @city_tokens[state] << tokens
+        end
+      end
+    end
+  end
+end

data/lib/matcher.rb

@@ -0,0 +1,90 @@
+module AddressTokens
+  class Matcher
+    def initialize(finder)
+      @finder = finder
+    end
+
+    def match(str)
+      state_info   = find_state(str)
+      city_info    = find_city(str, state_info)
+      address_info = find_address(str, city_info)
+
+      {
+        state_abbr:     state_info[:state],
+        state_name:     @finder.states[state_info[:state]],
+        state_start_at: state_info[:start_at],
+        city_name:      city_info[:city_name],
+        city_string:    city_info[:city_string],
+        city_start_at:  city_info[:start_at],
+        address:        address_info[:address]
+      }
+    end
+
+    private
+
+    def find_state(str)
+      last_char = str.rindex(@finder.state_separator)
+      token     = str[last_char .. -1]
+      token     = token.gsub(/\s{2,}/, ' ')
+      matches   = token.match(Regexp.new("#{@finder.state_separator}\s?(?<state>\\w+)"))
+      { state: matches[:state], start_at: last_char }
+    end
+
+    def find_city(str, state_info)
+      cities = @finder.city_tokens[state_info[:state]]
+      raise StateNotFound, "State #{state_info[:state]} not found on state data" if cities.nil?
+
+      without_state  = str[0 .. state_info[:start_at] - 1].strip.gsub(/\s{2,}/, ' ')
+      transliterated = I18n.transliterate(without_state).downcase
+
+      exact_or_trans = find_city_by_exact_or_trans(cities, without_state, transliterated)
+      return exact_or_trans if !exact_or_trans.nil?
+
+      tokenized = find_city_by_tokenized(cities, transliterated, str)
+      return tokenized if !tokenized.nil?
+
+      raise CityNotFound, "City not found"
+    end
+
+    def find_city_by_tokenized(cities, transliterated, str)
+      choices = []
+
+      cities.each do |city|
+        tokens      = transliterated.split
+        city_tokens = I18n.transliterate(city[0]).downcase.split
+
+        if tokens[-1] == city_tokens[-1]
+          first_tokens              = tokens.map         { |token| token[0] }.join
+          first_city_tokens         = city_tokens.map    { |token| token[0] }.join
+          first_shorten_city_tokens = city_tokens.select { |token| token.size > 2 }.map { |token| token[0] }.join
+
+          choices << [city[0], first_tokens, first_city_tokens]         if Regexp.new("#{first_city_tokens}$").match?         first_tokens
+          choices << [city[0], first_tokens, first_shorten_city_tokens] if Regexp.new("#{first_shorten_city_tokens}$").match? first_tokens
+        end
+      end
+
+      return nil if choices.size == 0
+
+      reversed = choices.sort_by { |choice| choice[2].size }.reverse
+      regex    = reversed[0][2].scan(/./).map { |char| "#{char}[\\p{Latin}\.]*\\s+"}.join.strip[0..-4]
+      matches = Regexp.new(regex, 'i').match(str)
+      { city_name: reversed[0][0], city_string: matches ? matches[0].strip : nil, start_at: matches ? str.index(matches[0].strip) : -1 }
+    end
+
+    def find_city_by_exact_or_trans(cities, without_state, transliterated)
+      cities.each do |city|
+        exact = Regexp.new("#{city[0]}$")
+        trans = Regexp.new("#{city[1]}$")
+
+        return { city_name: city[0], start_at: without_state.rindex(city[0]) , city_string: city[0] } if exact.match? without_state
+        return { city_name: city[0], start_at: transliterated.rindex(city[1]), city_string: without_state[transliterated.rindex(trans)..-1] } if trans.match? transliterated
+      end
+      nil
+    end
+
+    def find_address(str, city_info)
+      return { address: str[0 ... city_info[:start_at]].strip } if city_info[:start_at] > 0
+      { address: str.split(city_info[:city_string])[0].strip }
+    end
+  end
+end

data/lib/state_not_found.rb

@@ -0,0 +1,4 @@
+module AddressTokens
+  class StateNotFound < StandardError
+  end
+end

data.tar.gz.sig

@@ -0,0 +1,3 @@
+X�ٹ��d'
 ����N�,m��
+V�:��x��"Ɣ,��-V�@����0s�s/>������:^�ZlY*L�u��i���#S%
+�EɌq^s�y�X�ࢵ���T�Z[�EAr�:��

metadata

@@ -0,0 +1,137 @@
+--- !ruby/object:Gem::Specification
+name: address_tokens
+version: !ruby/object:Gem::Version
+  version: 0.1.3
+platform: ruby
+authors:
+- Eustaquio Rangel
+autorequire: 
+bindir: exe
+cert_chain:
+- |
+  -----BEGIN CERTIFICATE-----
+  MIIDjjCCAnagAwIBAgIBATANBgkqhkiG9w0BAQUFADBGMRgwFgYDVQQDDA9ldXN0
+  YXF1aW9yYW5nZWwxFTATBgoJkiaJk/IsZAEZFgVnbWFpbDETMBEGCgmSJomT8ixk
+  ARkWA2NvbTAeFw0xNzA2MDMyMjM3MjdaFw0xODA2MDMyMjM3MjdaMEYxGDAWBgNV
+  BAMMD2V1c3RhcXVpb3JhbmdlbDEVMBMGCgmSJomT8ixkARkWBWdtYWlsMRMwEQYK
+  CZImiZPyLGQBGRYDY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
+  pI58OaGELZWRnqvArECgbbOOU5ThDxGRoBz/91KaXysVAalPUVtEXQjsZSc6wOKb
+  a+m2vEZ6j1LWMfG1xKh07KN/EwcmUoG9uj/U+OcZ+M6YA+DSDBQJozXEUgKiD0e0
+  crZqrq+Qo9jg19lPvYcsFS4WumC3gtrx8Fe8K5JTEFR71Md3nnuBuKp1pouYTC3b
+  UQ2Xv1URDITKa3N3Kyu/6MJOYBJ2nvDKdEdMFbXxKiC/Njv7eIF+1iNWrzp8osgL
+  5LWITluL2n+/14QVDJehLRuh9Zg9WGT6FEji4PsmNQtZKwYpGGzNX5dDeYZcbhPk
+  ri9MK8ztMYBLkNzd4sbbPQIDAQABo4GGMIGDMAkGA1UdEwQCMAAwCwYDVR0PBAQD
+  AgSwMB0GA1UdDgQWBBR9bkdMJUe5Lyc1V8NQm1hbjPpKBzAkBgNVHREEHTAbgRll
+  dXN0YXF1aW9yYW5nZWxAZ21haWwuY29tMCQGA1UdEgQdMBuBGWV1c3RhcXVpb3Jh
+  bmdlbEBnbWFpbC5jb20wDQYJKoZIhvcNAQEFBQADggEBAF8KZx0njczbBRXzjGo0
+  mTsAOlB474ZqQjKN1zL/sGw8G4Q/UpeVEclBZrP9P9PUomYliGP38oYM4DHHMpyd
+  Qh9Wsou0ZJ5oD0O3nRraOVGFN7Azm05+xJ4fV1Zi8nsUdpF3za7s27Non9cYF4/2
+  iwJPWOjl/AxvAS2efkECSGbuZtuNKWrWYMH+aGbtavd5hfb8voGbTFYB4azbQQf+
+  ESf+WEicRlMQuQUn314wSFq1pq65S9GrZWMmSP5gc0kKfa51fOmIYmO3eSlmWG/p
+  G+/hO8DFcpmRPCD/YXu/rFkKHquizBGSfr1BR/es/HhfrIoT3BUN2uDUT02aaWmB
+  JMg=
+  -----END CERTIFICATE-----
+date: 2018-01-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: i18n
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Always want to find where address, city and state are on a string? Use
+  this gem.
+email:
+- taq@eustaquiorangel.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- Gemfile
+- README.md
+- Rakefile
+- address_tokens.gemspec
+- bin/console
+- bin/setup
+- lib/address_tokens.rb
+- lib/address_tokens/version.rb
+- lib/city_not_found.rb
+- lib/finder.rb
+- lib/matcher.rb
+- lib/state_not_found.rb
+homepage: http://github.com/taq/address_tokens
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.3
+signing_key: 
+specification_version: 4
+summary: Find address tokens on a string
+test_files: []