inci_score 1.2.1 → 2.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ffacec4022ea855b06a3eee2eae7813894ae11c8
-  data.tar.gz: c53b431b2a74d2f0156d7079f8f866a4326b4c7e
+  metadata.gz: 5970cfdecac8492dbfd510dce7a24488e543233c
+  data.tar.gz: fb5b1171f1fcab479e24b33dc7d7f37582b93741
 SHA512:
-  metadata.gz: 26538bc66e1aa5f845130e446a37d674b538173dcff2bbce5b6bc28c7e168979b44dba6b42ab8d80750b16bf92277d09c2eb9f31789a844485ab48601b13762a
-  data.tar.gz: ebc439f00b724c3a51bef74e4fac90c838c604a34150d9fd3324a90ed24b618f0092ecc9cb7ed0cb74cf2f20adefc872ef26d5c245800388c568bab003cced68
+  metadata.gz: 42624a99c66bc3fcfb53cff14ebe6a153b220901df9b9e5f49f3d8ec2c9378436cd7090446bb449fefc7320c8ff61fdd5375b551b015312858fac8b8bfa8b66c
+  data.tar.gz: 2f6bcc48dd8727a6b2b9665882cd3a809b849a7876a7a5303b7a8c3438c373fb3e25c28545889f70fcc14aa1724ea5febfbb5a78a555456ec99e44b5ca9de329

data/.travis.yml

@@ -1,5 +1,4 @@
 language: ruby
 rvm:
-  - 2.2.2
-  - 2.3.0
+  - 2.4.0
 before_install: gem install bundler -v 1.11.2

data/README.md

@@ -11,9 +11,13 @@
   * [Starting Puma](#starting-puma)
   * [Triggering a request](#triggering-a-request)
 * [CLI API](#cli-api)
-* [Performance](#performance)
+  * [Refresh catalog](#refresh-catalog)
+* [Benchmark](#benchmark)
   * [Levenshtein in C](#levenshtein-in-c)
-  * [Records](#records)
+  * [Platform](#platform)
+  * [Wrk](#wrk)
+  * [Results](#results)
+  * [Ruby 2.4](#ruby-2.4)
 
 ## Scope
 This gem computes the score of cosmetic components basing on the information provided by the [Biodizionario site](http://www.biodizionario.it/) by Fabrizio Zago.
@@ -75,8 +79,8 @@ The Web API exposes the *InciScore* library over HTTP via the [Puma](http://puma
 
 ### Starting Puma
 Simply start Puma via the *config.ru* file included in the repository by spawning how many workers as your current workstation supports:
-```
-bundle exec puma -w 7 -t 16:32 --preload
+```shell
+bundle exec puma -w 8 -t 0:2 --preload
 ```
 
 ### Triggering a request
@@ -84,7 +88,7 @@ The Web API responds with a JSON object representing the original *InciScore::Re
 
 You can pass the source string directly as a HTTP parameter:
 
-```
+```shell
 curl http://127.0.0.1:9292?src=aqua,dimethicone
 => {"components":{"aqua":0,"dimethicone":4},"unrecognized":[],"score":53.762874945799766,"valid":true}
 ```
@@ -92,8 +96,8 @@ curl http://127.0.0.1:9292?src=aqua,dimethicone
 ## CLI API
 You can collect INCI data by using the available binary:
 
-```
-inci_score "aqua,dimethicone,pej-10,noent"
+```shell
+inci_score --src="aqua,dimethicone,pej-10,noent"
 
 TOTAL SCORE:
         47.18034913243358
@@ -107,11 +111,41 @@ UNRECOGNIZED:
         noent
 ```
 
-## Performance
+### Refresh catalog
+When using CLI you have the option to fetch a fresh catalog from remote by specifyng a flag:
+```shell
+inci_score --fresh --src="aqua,dimethicone,pej-10,noent"
+```
+
+## Benchmark
+
+### Levenshtein in C
 I noticed the APIs slows down dramatically when dealing with unrecognized components to fuzzy match on.  
 I profiled the code by using the [benchmark-ips](https://github.com/evanphx/benchmark-ips) gem, finding the bottleneck was the pure Ruby implementation of the Levenshtein distance algorithm.  
 After some pointless optimization, i replaced this routine with a C implementation: i opted for the straightforward [Ruby Inline](https://github.com/seattlerb/rubyinline) library to call the C code straight from Ruby.  
 As a result i've got a 10x increment of the throughput, all without scarifying code readability.
 
-### Numbers
-I moved the benchmark numbers to the [Crystal porting](https://github.com/costajob/inci_score.cr) of the InciScore library, please look there.
+### Platform
+I registered these benchmarks with a MacBook PRO 15 mid 2015 having these specs:
+* OSX El Captain
+* 2,2 GHz Intel Core i7 (4 cores)
+* 16 GB 1600 MHz DDR3
+
+### Wrk
+As always i used [wrk](https://github.com/wg/wrk) as the loading tool.
+I measured each library three times, picking the best lap.  
+The following script command is used:
+
+```
+wrk -t 4 -c 100 -d 30s --timeout 2000 http://127.0.0.1:9292/?src=<list_of_ingredients>
+```
+
+### Results
+| Type               | Ingredients              | Throughput (req/s) | Latency in ms (avg/stdev/max) |
+| :----------------- | :----------------------- | -----------------: | ----------------------------: |
+| exact matching     | aqua,parfum,zeolite      |           48863.58 |               0.31/0.55/10.82 |
+
+## Ruby 2.4
+After upgrading to Ruby 2.4 i doubled the throughput of the matcher: i assume Ruby optimization to the [Hash access](#https://blog.heroku.com/ruby-2-4-features-hashes-integers-rounding) is the driving reason.  
+I also adopted the new #match? method to avoid creating a MatchData object when i am just checking for predicate.  
+In the end Ruby upgrade is a big deal for my gem, give it a try!

data/bin/inci_score

@@ -1,7 +1,7 @@
 #!/usr/bin/env ruby
+lib = File.expand_path("../../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
-require 'bundler/setup'
-require 'inci_score'
+require "inci_score"
 
-fail ArgumentError, "please specify at least a src argument" if ARGV.empty?
-puts InciScore::Computer.new(ARGV[0], InciScore::Catalog.fetch).call
+InciScore::CLI.new(args: ARGV.clone).call

data/inci_score.gemspec

@@ -14,7 +14,7 @@ Gem::Specification.new do |s|
   s.executables << "inci_score"
   s.require_paths = ["lib"]
   s.license = "MIT"
-  s.required_ruby_version = ">= 2.2.2"
+  s.required_ruby_version = ">= 2.4"
 
   s.add_runtime_dependency "nokogiri", "~> 1.6"
   s.add_runtime_dependency "puma", "~> 3"

data/lib/inci_score/api/app.rb

@@ -13,7 +13,7 @@ module InciScore
       def call(env)
         req = Rack::Request.new(env)
         src = req.params["src"]
-        json = src ? Computer.new(src, catalog).call.to_json : %q({"error": "no valid source"})
+        json = src ? Computer.new(src: src, catalog: catalog).call.to_json : %q({"error": "no valid source"})
         ['200', {'Content-Type' => 'application/json'}, [json]]
       end
     end

data/lib/inci_score/cli.rb

@@ -0,0 +1,44 @@
+require "optparse"
+require "inci_score/computer"
+
+module InciScore
+  class CLI
+    def initialize(args:, io: STDOUT, catalog: InciScore::Catalog.fetch)
+      @args = args
+      @io = io
+      @catalog = catalog
+      @src = nil
+      @fresh = nil
+    end
+
+    def call(computer_klass = Computer, fetcher = Fetcher.new)
+      parser.parse!(@args)
+      return @io.puts("Specify inci list as: --src='aqua, parfum, etc'") unless @src
+      @io.puts computer_klass.new(src: @src, catalog: catalog(fetcher)).call
+    end
+
+    private def parser
+      OptionParser.new do |opts|
+        opts.banner = %q{Usage: ./bin/inci_score --src='aqua, parfum, etc' --fresh}
+
+        opts.on("-sSRC", "--src=SRC", "The INCI list: 'aqua, parfum, etc'") do |src|
+          @src = src
+        end
+
+        opts.on("-f", "--fresh", "Fetch a fresh catalog from remote") do |fresh|
+          @fresh = fresh
+        end
+
+        opts.on("-h", "--help", "Prints this help") do
+          @io.puts opts
+          exit
+        end
+      end
+    end
+
+    private def catalog(fetcher)
+      return @catalog unless @fresh
+      fetcher.call
+    end
+  end
+end

data/lib/inci_score/computer.rb

@@ -7,9 +7,11 @@ module InciScore
   class Computer
     TOLERANCE = 30.0
 
-    def initialize(src, catalog)
+    def initialize(src:, catalog:, tolerance: TOLERANCE, rules: Normalizer::DEFAULT_RULES)
       @src = src
       @catalog = catalog
+      @tolerance = Float(tolerance)
+      @rules = rules
       @unrecognized = []
     end
 
@@ -20,17 +22,15 @@ module InciScore
                                  valid: valid?)
     end
 
-    private
-
-    def score
+    private def score
       Scorer.new(components.map(&:last)).call
     end
 
-    def ingredients
-      @ingredients ||= Normalizer.new(src: @src).call
+    private def ingredients
+      @ingredients ||= Normalizer.new(src: @src, rules: @rules).call
     end
 
-    def components
+    private def components
       @components ||= ingredients.map do |ingredient|
         Recognizer.new(ingredient, @catalog).call.tap do |component|
           @unrecognized << ingredient unless component
@@ -38,8 +38,8 @@ module InciScore
       end.compact
     end
 
-    def valid?
-      @unrecognized.size / (ingredients.size / 100.0) <= TOLERANCE
+    private def valid?
+      @unrecognized.size / (ingredients.size / 100.0) <= @tolerance
     end
   end
 end

data/lib/inci_score/{parser.rb → fetcher.rb}

@@ -1,7 +1,7 @@
 require 'nokogiri'
 
 module InciScore
-  class Parser
+  class Fetcher
     BIODIZIO_URI = 'http://www.biodizionario.it/biodizio.php'
     SEMAPHORES = %w[vv v g r rr]
     CSS_QUERY = 'table[width="751"] > tr > td img'

data/lib/inci_score/normalizer.rb

@@ -2,7 +2,7 @@ require 'inci_score/normalizer_rules'
 
 module InciScore
   class Normalizer
-    DEFAULT_RULES = Rules.constants - [:Base]
+    DEFAULT_RULES = [Rules::Replacer, Rules::Downcaser, Rules::Beheader, Rules::Separator, Rules::Tokenizer, Rules::Sanitizer, Rules::Desynonymizer]
 
     attr_reader :src
 
@@ -12,9 +12,9 @@ module InciScore
     end
 
     def call
-      @rules.reduce(@src) do |src, name|
-        rule = Rules.const_get(name).new(src)
-        src = rule.call
+      yield(@rules) if block_given?
+      @rules.reduce(@src) do |src, rule|
+        @src = rule.call(src)
       end
     end
   end

data/lib/inci_score/normalizer_rules.rb

@@ -1,73 +1,90 @@
 module InciScore
   class Normalizer
     module Rules
-      class Base
-        SEPARATOR = ','
+      SEPARATOR = ','
 
-        def initialize(src)
-          @src = src
-        end
+      module Replacer
+        extend self
 
-        def call
-          fail NotImplementedError
-        end
-      end
-
-      class Replacer < Base
         REPLACEMENTS = [
           [/\n+|\t+/, ' '],
           ['‘', "'"],
           ['—', '-'],
-          ['(', 'C'],
           ['_', ' '],
           ['~', '-'],
           ['|', 'l'],
           [' I ', '/']
         ]
 
-        def call
-          REPLACEMENTS.reduce(@src) do |src, replacement|
+        def call(src)
+          REPLACEMENTS.reduce(src) do |_src, replacement|
             invalid, valid = *replacement
-            src.index(invalid) ? src.gsub(invalid, valid) : src
+            _src.index(invalid) ? _src.gsub(invalid, valid) : _src
           end
         end
       end
 
-      class Downcaser < Base
-        def call 
-          @src.downcase
+      module Downcaser
+        extend self
+
+        def call(src)
+          src.downcase
         end
       end 
 
-      class Beheader < Base
+      module Beheader
+        extend self
+
         TITLE_SEP = ':'
         MAX_INDEX = 50
 
-        def call
-          sep_index = @src.index(TITLE_SEP)
-          return @src if !sep_index || sep_index > MAX_INDEX
-          @src[sep_index+1, @src.size]
+        def call(src)
+          sep_index = src.index(TITLE_SEP)
+          return src if !sep_index || sep_index > MAX_INDEX
+          src[sep_index+1, src.size]
         end
       end
 
-      class Separator < Base
+      module Separator
+        extend self
+
         SEPARATORS = ["; ", ". ", " ' ", " - ", " : "]
 
-        def call
-          SEPARATORS.reduce(@src) do |src, separator|
-            src = src.gsub(separator, SEPARATOR)
+        def call(src)
+          SEPARATORS.reduce(src) do |_src, separator|
+            _src = _src.gsub(separator, SEPARATOR)
           end
         end
       end 
 
-      class Tokenizer < Base
-        INVALID_CHARS = /[^\w\s-]/
+      module Tokenizer
+        extend self
+
+        def call(src)
+          src.split(SEPARATOR).map(&:strip)
+        end
+      end
+
+      module Sanitizer
+        extend self
+
+        INVALID_CHARS = /[^\/\(\)\w\s-]/
+
+        def call(src)
+          Array(src).map do |token|
+            token.gsub(INVALID_CHARS, '')
+          end.reject(&:empty?)
+        end
+      end
+
+      module Desynonymizer
+        extend self
+
+        SYNONYM = /\/.*/
 
-        def call
-          @src.split(SEPARATOR).map do |token|
-            token = token.sub(/\/.*/, '')
-            token = token.gsub(INVALID_CHARS, '')
-            token = token.strip
+        def call(src)
+          Array(src).map do |token|
+            token.sub(SYNONYM, '').strip
           end.reject(&:empty?)
         end
       end

data/lib/inci_score/recognizer.rb

@@ -2,7 +2,7 @@ require 'inci_score/recognizer_rules'
 
 module InciScore
   class Recognizer
-    DEFAULT_RULES = Rules.constants - [:Base]
+    DEFAULT_RULES = [Rules::Key, Rules::Levenshtein, Rules::Digits, Rules::Tokens]
 
     def initialize(src, catalog, rules = DEFAULT_RULES)
       @src = src
@@ -11,17 +11,13 @@ module InciScore
     end
 
     def call
-      @component = apply_rules
-      return [@component, @catalog[@component]] if @component
-    end 
-
-    private
-
-    def apply_rules
-      @rules.reduce(nil) do |component, name|
-        rule = Rules.const_get(name).new(@src, @catalog)
-        component || rule.call
+      @component = @rules.reduce(nil) do |component, rule|
+        break(component) if component
+        _rule = rule.new(@src, @catalog)
+        yield(rule) if block_given?
+        _rule.call
       end
-    end
+      [@component, @catalog[@component]] if @component
+    end 
   end
 end

data/lib/inci_score/recognizer_rules.rb

@@ -28,12 +28,12 @@ module InciScore
         def call
           size = @src.size
           initial = @src[0]
-          component, distance = @catalog.reduce([nil, size]) do |min, (component, _)|
-            next min unless component.start_with?(initial)
-            match = (n = component.index(ALTERNATE_SEP)) ? component[0, n] : component
+          component, distance = @catalog.reduce([nil, size]) do |min, (_component, _)|
+            next min unless _component.start_with?(initial)
+            match = (n = _component.index(ALTERNATE_SEP)) ? _component[0, n] : _component
             next min if match.size > (size + TOLERANCE)
             dist = @src.distance(match)
-            min = [component, dist] if dist < min[1]
+            min = [_component, dist] if dist < min[1]
             min
           end
           component unless distance > TOLERANCE || distance >= (size-1)
@@ -47,7 +47,7 @@ module InciScore
           return if @src.size < TOLERANCE
           digits = @src[0, MIN_MEANINGFUL]
           @catalog.detect do |component, _| 
-            component.match(/^#{Regexp::escape(digits)}/)
+            component.match?(/^#{Regexp::escape(digits)}/)
           end.to_a.first
         end
       end
@@ -58,7 +58,7 @@ module InciScore
         def call
           tokens.each do |token|
             @catalog.each do |component, _| 
-              return component if component.match(/\b#{Regexp.escape(token)}\b/)
+              return component if component.match?(/\b#{Regexp.escape(token)}\b/)
             end
           end
           nil

data/lib/inci_score/version.rb

@@ -1,3 +1,3 @@
 module InciScore
-  VERSION = "1.2.1"
+  VERSION = "2.0.1"
 end

data/lib/inci_score.rb

@@ -1,4 +1,5 @@
+require 'open-uri'
 require 'inci_score/version'
-require 'inci_score/parser'
+require 'inci_score/fetcher'
 require 'inci_score/catalog'
-require 'inci_score/computer'
+require 'inci_score/cli'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: inci_score
 version: !ruby/object:Gem::Version
-  version: 1.2.1
+  version: 2.0.1
 platform: ruby
 authors:
 - costajob
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-09-28 00:00:00.000000000 Z
+date: 2017-01-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
@@ -145,11 +145,12 @@ files:
 - lib/inci_score.rb
 - lib/inci_score/api/app.rb
 - lib/inci_score/catalog.rb
+- lib/inci_score/cli.rb
 - lib/inci_score/computer.rb
+- lib/inci_score/fetcher.rb
 - lib/inci_score/levenshtein.rb
 - lib/inci_score/normalizer.rb
 - lib/inci_score/normalizer_rules.rb
-- lib/inci_score/parser.rb
 - lib/inci_score/recognizer.rb
 - lib/inci_score/recognizer_rules.rb
 - lib/inci_score/response.rb
@@ -169,7 +170,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.2.2
+      version: '2.4'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
@@ -177,7 +178,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.6.8
 signing_key: 
 specification_version: 4
 summary: A library that computes the hazard of cosmetic products components, based