inci_score 2.5.1 → 3.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e60cb7510ccdae05853733d9da9aefac5fd96c92
-  data.tar.gz: 338c5331d203e29c7f3022a72867ea9eb23ff6df
+  metadata.gz: 3acb19f9ac522f6a50378ec318c6466ae9b67257
+  data.tar.gz: 3d46458825c9d1fb3ff74db0779b329b63c5f97a
 SHA512:
-  metadata.gz: 355b1ea92c7a1d7fd02f47ef8dd5aed401a3840d6e8a9b6f9e94e54dd5c2b1e4ad4406a379b02fc640ae8ec438c221dd8004ba08451bb2935b0e48188b76f3aa
-  data.tar.gz: 5749ed66ce6fcf83e277d21dd1d4f1e938d76fa77279c9a61c16760f006bb6f71a499289400614fb560d7f2d4fec13c1ee49e9b9662b2e9b3faa769037d938dd
+  metadata.gz: 9ab029ca732a8605502a7263fa3d7fc96ad20be2ca6aaf9b142e599746eae8216bd326ae318b92aed19a24e157456fe63dd99642d52f536c0a3ea0a4cd59e5bd
+  data.tar.gz: 6d23d330b4267abc0d2ba4241c52d0f55d94922bc1dbb776842e3266afd699f366ade1c648f2e76c3ba2a85db7a680aac59257f683d9d85b5fb2b9b352eae536

data/Gemfile.lock

@@ -0,0 +1,34 @@
+PATH
+  remote: .
+  specs:
+    inci_score (3.0.1)
+      RubyInline (~> 3)
+      puma (~> 3)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    RubyInline (3.12.4)
+      ZenTest (~> 4.3)
+    ZenTest (4.11.1)
+    benchmark-ips (2.7.2)
+    minitest (5.10.3)
+    puma (3.10.0)
+    rack (2.0.3)
+    rack-test (0.7.0)
+      rack (>= 1.0, < 3)
+    rake (10.5.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  benchmark-ips (~> 2)
+  bundler (~> 1.11)
+  inci_score!
+  minitest (~> 5.0)
+  rack-test (~> 0.6)
+  rake (~> 10.0)
+
+BUNDLED WITH
+   1.15.4

data/README.md

@@ -8,7 +8,6 @@
 * [API](#api)
   * [Unrecognized components](#unrecognized-components)
 * [CLI](#cli)
-  * [Refresh catalog](#refresh-catalog)
   * [HTTP server](#http-server)
     * [Triggering a request](#triggering-a-request)
   * [Getting help](#getting-help)
@@ -52,9 +51,7 @@ The API of the gem is pretty simple, you can open irb by *bundle console* and st
 
 ```ruby
 inci = InciScore::Computer.new(src: 'aqua, dimethicone').call
-=> #<InciScore::Response:0x000000029f8100 @components={"aqua"=>0, "dimethicone"=>4}, @score=53.762874945799766, @unrecognized=[], @valid=true>
-inci.score
-=> 53.762874945799766
+inci.score # 53.7629
 ```
 
 As you see the results are wrapped by an *InciScore::Response* object, this is useful when dealing with the CLI and HTTP interfaces (read below).
@@ -66,11 +63,8 @@ Is still possible to query the object for its state:
 
 ```ruby
 inci = InciScore::Computer.new(src: 'ingredients:aqua,noent1,noent2').call
-=> #<InciScore::Response:0x000000030c16d0 @components={"aqua"=>0}, @score=100.0, @unrecognized=["noent1", "noent2"], @valid=false>
-inci.valid
-=> false
-inci.unrecognized
-=> ["noent1", "noent2"]
+inci.valid # false
+inci.unrecognized # ["noent1", "noent2"]
 ```
 
 ## CLI
@@ -80,7 +74,7 @@ You can collect INCI data by using the available CLI interface:
 inci_score --src="ingredients: aqua, dimethicone, pej-10, noent"
 
 TOTAL SCORE:
-        47.18034913243358
+        47.1803
 VALID STATE:
         true
 COMPONENTS (hazard - name): 
@@ -91,12 +85,6 @@ UNRECOGNIZED:
         noent
 ```
 
-### Refresh catalog
-You also have the option to fetch a fresh catalog from www.biodizionario.it  by specifyng a flag:
-```shell
-inci_score --fresh --src="aqua, dimethicone"
-```
-
 ### HTTP server
 The CLI interface exposes a Web layer based on the [Puma](http://puma.io/) application server.  
 The HTTP server is started on the specified port by spawning as many workers as your current workstation supports:
@@ -111,17 +99,16 @@ You can pass the source string directly as a HTTP parameter (URI escaped):
 
 ```shell
 curl http://127.0.0.1:9292?src=aqua,dimethicone
-=> {"components":{"aqua":0,"dimethicone":4},"unrecognized":[],"score":53.762874945799766,"valid":true}
+=> {"components":{"aqua":0,"dimethicone":4},"unrecognized":[],"score":53.7629,"valid":true}
 ```
 
 ### Getting help
 You can get CLI interface help by:
 ```shell
-Usage: inci_score --src="aqua, parfum, etc" --fresh --precise
+Usage: inci_score --src="aqua, parfum, etc" --precise
     -s, --src=SRC                    The INCI list: "aqua, parfum, etc"
-    -f, --fresh                      Fetch a fresh catalog from remote
     -p, --precise                    Compute components more precisely (slower)
-        --http=PORT                  Start Puma server on the specified port
+        --http=PORT                  Start HTTP server on the specified port
     -h, --help                       Prints this help
 ```
 
@@ -140,13 +127,15 @@ I registered these benchmarks with a MacBook PRO 15 mid 2015 having these specs:
 * Ruby 2.4
 
 ### Wrk
-As always i used [wrk](https://github.com/wg/wrk) as the loading tool, executed on a dedicated workstation.
+As always i used [wrk](https://github.com/wg/wrk) as the loading tool.
 I measured the library three times, picking the best lap.  
 ```shell
-wrk -t 4 -c 100 -d 30s --timeout 2000 http://0.0.0.0:9292/?src=aqua,parfum,zeolithe
+wrk -t 4 -c 100 -d 30s --timeout 2000 "http://0.0.0.0:9292/?src=<source>&precise=true"
 ```
 
 ### Results
-| Throughput (req/s) | Latency (avg/stdev/max)   |
-| -----------------: | ------------------------: |
-|            2908.63 |  42.23ms/47.17ms/461.36ms |
+| Source                      | Throughput (req/s) |
+| --------------------------: | -----------------: |
+| aqua,parfum,zeolite         |          18784.21  |
+| agua,porfum,zeolithe        |           1087.88  |
+| agua/water,porfum/fragrance |           1599.47  |

data/config/catalog.yml

@@ -1,7 +1,7 @@
 ---
-water: 0
-parfum: 0
-fragrance: 0
+aqua: 0
+parfum: 3
+fragrance: 3
 phosphatidylcholine: 1
 1-naphthol: 4
 1,2,4-benzenetriacetate: 4
@@ -296,7 +296,6 @@ apricot kernel oil peg-6 esters: 3
 apricotamide dea: 3
 apricotamidopropyl betaine: 2
 apricotamidopropyl ethyldimonium ethosulfate: 2
-aqua: 0
 arachideth-20: 3
 arachidic acid: 1
 arachidonic acid: 1

data/config.ru

@@ -1,3 +1,3 @@
-require 'inci_score/app'
+require "inci_score/api"
 
-run InciScore::App
+run InciScore::Api

data/inci_score.gemspec

@@ -1,6 +1,6 @@
 lib = File.expand_path('../lib', __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require 'inci_score/version'
+require "inci_score/version"
 
 Gem::Specification.new do |s|
   s.name = "inci_score"
@@ -9,14 +9,13 @@ Gem::Specification.new do |s|
   s.email = ["costajob@gmail.com"]
   s.summary = %q{A library that computes the hazard of cosmetic products components, based on the Biodizionario data.}
   s.homepage = "https://github.com/costajob/inci_score.git"
-  s.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec|test|s|features)/}) }
+  s.files = %w(README.md Rakefile inci_score.gemspec Gemfile Gemfile.lock config.ru config/catalog.yml ext/levenshtein.c bin/inci_score) + Dir["{spec,lib}/**/*.rb"]
   s.bindir = "bin"
   s.executables << "inci_score"
   s.require_paths = ["lib"]
   s.license = "MIT"
   s.required_ruby_version = ">= 2.2.2"
 
-  s.add_runtime_dependency "nokogiri", "~> 1.6"
   s.add_runtime_dependency "puma", "~> 3"
   s.add_runtime_dependency "RubyInline", "~> 3"
 

data/lib/inci_score/{app.rb → api.rb}

@@ -2,7 +2,7 @@ require "rack"
 require "inci_score"
 
 module InciScore
-  module App
+  module Api
     extend self
 
     def catalog

data/lib/inci_score/cli.rb

@@ -9,35 +9,30 @@ module InciScore
       @io = io
       @catalog = catalog
       @src = nil
-      @fresh = nil
       @port = nil
       @precise = nil
     end
 
-    def call(server_klass: Server, computer_klass: Computer, fetcher: Fetcher.new)
+    def call(server_klass: Server, computer_klass: Computer)
       parser.parse!(@args)
       return server_klass.new(port: @port, preload: true).run if @port
       return @io.puts(%q{Specify inci list as: --src="aqua, parfum, etc"}) unless @src
-      @io.puts computer_klass.new(src: @src, catalog: catalog(fetcher), precise: @precise).call
+      @io.puts computer_klass.new(src: @src, catalog: @catalog, precise: @precise).call
     end
 
     private def parser
       OptionParser.new do |opts|
-        opts.banner = %q{Usage: inci_score --src="aqua, parfum, etc" --fresh --precise}
+        opts.banner = %q{Usage: inci_score --src="aqua, parfum, etc" --precise}
 
         opts.on("-sSRC", "--src=SRC", %q{The INCI list: "aqua, parfum, etc"}) do |src|
           @src = src
         end
 
-        opts.on("-f", "--fresh", "Fetch a fresh catalog from remote") do |fresh|
-          @fresh = fresh
-        end
-
         opts.on("-p", "--precise", "Compute components more precisely (slower)") do |precise|
           @precise = precise
         end
 
-        opts.on("--http=PORT", "Start Puma server on the specified port") do |port|
+        opts.on("--http=PORT", "Start HTTP server on the specified port") do |port|
           @port = port
         end
 
@@ -47,10 +42,5 @@ module InciScore
         end
       end
     end
-
-    private def catalog(fetcher)
-      return @catalog unless @fresh
-      fetcher.call
-    end
   end
 end

data/lib/inci_score/computer.rb

@@ -1,13 +1,19 @@
+require "inci_score/ingredient"
 require "inci_score/normalizer"
 require "inci_score/recognizer"
-require "inci_score/scorer"
 require "inci_score/response"
+require "inci_score/scorer"
 
 module InciScore
   class Computer
     TOLERANCE = 30.0
+    PERCENT = 100.0
 
-    def initialize(src:, catalog:, tolerance: TOLERANCE, rules: Normalizer::DEFAULT_RULES, precise: false)
+    def initialize(src:, 
+                   catalog: Catalog.fetch, 
+                   tolerance: TOLERANCE, 
+                   rules: Normalizer::DEFAULT_RULES, 
+                   precise: false)
       @src = src
       @catalog = catalog
       @tolerance = Float(tolerance)
@@ -17,18 +23,21 @@ module InciScore
     end
 
     def call
-      @response ||= Response.new(components: components.map(&:first),
+      @response ||= Response.new(components: components.map(&:name),
                                  unrecognized: @unrecognized,
                                  score: score,
                                  valid: valid?)
     end
 
     private def score
-      Scorer.new(components.map(&:last)).call
+      Scorer.new(components.map(&:hazard)).call
     end
 
     private def ingredients
-      @ingredients ||= Normalizer.new(src: @src, rules: @rules).call
+      @ingredients ||= begin
+                         tokens = Normalizer.new(src: @src, rules: @rules).call
+                         Ingredient.bulk(tokens)
+                       end
     end
 
     private def components
@@ -40,7 +49,7 @@ module InciScore
     end
 
     private def valid?
-      @unrecognized.size / (ingredients.size / 100.0) <= @tolerance
+      @unrecognized.size / (ingredients.size / PERCENT) <= @tolerance
     end
   end
 end

data/lib/inci_score/ingredient.rb

@@ -0,0 +1,45 @@
+module InciScore
+  class Ingredient
+    SLASH = "/"
+    SLASH_RULE = /(?<!ate)\//
+    PARENTHESIS = %w[( ) [ ]]
+    DETAILS_RULE = /(\(.+\)|\[.+\])/
+
+    def self.bulk(tokens)
+      tokens.map { |raw| new(raw) }
+    end
+
+    def initialize(raw)
+      @raw = raw
+      @tokens = raw.split(SLASH_RULE).map(&:strip)
+    end
+
+    def to_s
+      values.join(SLASH)
+    end
+
+    def values
+      @values ||= synonims.unshift(name).compact
+    end
+
+    private def name
+      return @tokens.first unless parenthesis?
+      @raw.sub(DETAILS_RULE, "").strip
+    end
+
+    private def synonims
+      @tokens[1, @tokens.size]
+    end
+
+    private def details
+      return unless parenthesis?
+      @raw.match(DETAILS_RULE)[1].delete(PARENTHESIS.join("|"))
+    end
+
+    private def parenthesis?
+      PARENTHESIS.each_slice(2).any? do |pair|
+        pair.all? { |p| @raw.index(p) }
+      end
+    end
+  end
+end

data/lib/inci_score/normalizer.rb

@@ -2,7 +2,7 @@ require "inci_score/normalizer_rules"
 
 module InciScore
   class Normalizer
-    DEFAULT_RULES = [Rules::Replacer, Rules::Downcaser, Rules::Beheader, Rules::Separator, Rules::Tokenizer, Rules::Sanitizer, Rules::Desynonymizer, Rules::Deparenthesizer, Rules::Uniquifier]
+    DEFAULT_RULES = [Rules::Replacer, Rules::Downcaser, Rules::Beheader, Rules::Separator, Rules::Tokenizer, Rules::Sanitizer, Rules::Uniquifier]
 
     attr_reader :src
 

data/lib/inci_score/normalizer_rules.rb

@@ -77,30 +77,6 @@ module InciScore
         end
       end
 
-      module Desynonymizer
-        extend self
-
-        SYNONYM = /\/.*/
-
-        def call(src)
-          Array(src).map do |token|
-            token.sub(SYNONYM, '').strip
-          end.reject(&:empty?)
-        end
-      end
-
-      module Deparenthesizer
-        extend self
-
-        PARENTHESIS = /\(.+?\)|\[.+?\]/
-
-        def call(src)
-          Array(src).map do |token|
-            token.sub(PARENTHESIS, '').strip
-          end.reject(&:empty?)
-        end
-      end
-
       module Uniquifier
         extend self
 

data/lib/inci_score/recognizer.rb

@@ -4,19 +4,37 @@ module InciScore
   class Recognizer
     DEFAULT_RULES = [Rules::Key, Rules::Levenshtein, Rules::Digits, Rules::Tokens]
 
-    def initialize(src, catalog, rules = DEFAULT_RULES)
-      @src = src
+    Component = Struct.new(:name, :hazard)
+
+    attr_reader :applied
+
+    def initialize(ingredient, catalog, rules = DEFAULT_RULES)
+      @ingredient = ingredient
       @catalog = catalog
       @rules = rules
+      @applied = []
     end
 
     def call(precise = false)
-      @component = @rules.reduce(nil) do |component, rule|
+      return if @ingredient.to_s.empty?
+      component = find_component(precise)
+      return unless component
+      Component.new(component, @catalog[component])
+    end 
+
+    private def find_component(precise)
+      @rules.reduce(nil) do |component, rule|
         break(component) if component
-        yield(rule) if block_given?
-        rule.call(@src, @catalog, precise)
+        applied << rule
+        apply(rule, precise)
       end
-      [@component, @catalog[@component]] if @component
-    end 
+    end
+
+    private def apply(rule, precise)
+      return rule.call(@ingredient.to_s, @catalog) unless precise
+      @ingredient.values.map do |value|
+        rule.call(value, @catalog)
+      end.find(&:itself)
+    end
   end
 end

data/lib/inci_score/recognizer_rules.rb

@@ -9,7 +9,7 @@ module InciScore
       module Key
         extend self
 
-        def call(src, catalog, precise = false)
+        def call(src, catalog)
           src if catalog.has_key?(src)
         end
       end
@@ -17,20 +17,24 @@ module InciScore
       module Levenshtein
         extend self
 
-        ALTERNATE_SEP  = "/"
+        Result = Struct.new(:name, :distance) do
+          def tolerable?(size)
+            distance < TOLERANCE && distance <= (size-1)
+          end
+        end
 
-        def call(src, catalog, precise = false)
+        def call(src, catalog)
           size = src.size
+          farthest = Result.new(nil, size)
           initial = src[0]
-          component, distance = catalog.reduce([nil, size]) do |min, (_component, _)|
-            next min unless precise || _component.start_with?(initial)
-            match = (n = _component.index(ALTERNATE_SEP)) ? _component[0, n] : _component
-            next min if match.size > (size + TOLERANCE)
-            dist = src.distance(match)
-            min = [_component, dist] if dist < min[1]
-            min
+          result = catalog.reduce(farthest) do |nearest, (component, _)|
+            next nearest unless component.start_with?(initial)
+            next nearest if component.size > (size + TOLERANCE)
+            d = src.distance(component)
+            nearest = Result.new(component, d) if d < nearest.distance
+            nearest
           end
-          component unless distance > TOLERANCE || distance >= (size-1)
+          result.name if result.tolerable?(size)
         end
       end
 
@@ -39,17 +43,10 @@ module InciScore
 
         MIN_MEANINGFUL = 7
 
-        def call(src, catalog, precise = false)
+        def call(src, catalog)
           return if src.size < TOLERANCE
-          digits = src[0, min_meaningful(precise)]
-          catalog.detect do |component, _| 
-            component.matches?(/^#{Regexp::escape(digits)}/)
-          end.to_a.first
-        end
-
-        def min_meaningful(precise)
-          return MIN_MEANINGFUL unless precise
-          MIN_MEANINGFUL + 2
+          digits = src[0, MIN_MEANINGFUL]
+          catalog.detect { |component, _| component.start_with?(digits) }.to_a.first
         end
       end
 
@@ -58,7 +55,7 @@ module InciScore
 
         UNMATCHABLE = %w[extract oil sodium acid sulfate]
       
-        def call(src, catalog, precise = false)
+        def call(src, catalog)
           tokens(src).each do |token|
             catalog.each do |component, _| 
               return component if component.matches?(/\b#{Regexp.escape(token)}\b/)

data/lib/inci_score/refinements.rb

@@ -1,4 +1,4 @@
-require 'inci_score/levenshtein' 
+require "inci_score/levenshtein"
 
 module InciScore
   module Refinements

data/lib/inci_score/response.rb

@@ -1,4 +1,4 @@
-require 'json'
+require "json"
 
 module InciScore
   class Response

data/lib/inci_score/score.rb

@@ -8,9 +8,7 @@ module InciScore
       @value = compute
     end
 
-    private
-
-    def compute
+    private def compute
       (@hazard - @weight).tap do |s|
         return 0.0 if s < 0
       end

data/lib/inci_score/scorer.rb

@@ -1,4 +1,4 @@
-require 'inci_score/score'
+require "inci_score/score"
 
 module InciScore
   class Scorer
@@ -12,33 +12,31 @@ module InciScore
 
     def call
       return 0 if @hazards.empty?
-      100 - avg * HAZARD_PERCENT
+      (100 - avg * HAZARD_PERCENT).round(4)
     end
 
-    private
-
-    def avg
+    private def avg
       avg_weighted / @size.to_f
     end
 
-    def avg_weighted
+    private def avg_weighted
       return @hazards.reduce(&:+) if same_hazard?
       weighted.reduce(0.0) do |acc,score| 
         acc += score.value
       end
     end
 
-    def same_hazard?
+    private def same_hazard?
       @hazards.uniq.size == 1
     end
 
-    def weighted
+    private def weighted
       @hazards.each_with_index.map do |h,i|
         Score.new(h, weight(i))
       end
     end
 
-    def weight(index)
+    private def weight(index)
       Math.log(index+1, @size * WEIGHT_FACTOR)
     end
   end

data/lib/inci_score/version.rb

@@ -1,3 +1,3 @@
 module InciScore
-  VERSION = "2.5.1"
+  VERSION = "3.0.3"
 end

data/lib/inci_score.rb

@@ -1,5 +1,4 @@
-require 'open-uri'
-require 'inci_score/version'
-require 'inci_score/fetcher'
-require 'inci_score/catalog'
-require 'inci_score/cli'
+require "inci_score/version"
+require "inci_score/catalog"
+require "inci_score/cli"
+require "inci_score/api"

data/spec/bench/levenshtein_bench.rb

@@ -0,0 +1,17 @@
+require "helper"
+
+s, t = "agua", "aqua"
+lev_ruby = InciScore::Levenshtein.new(s, t)
+lev_c = InciScore::LevenshteinC.new
+
+Benchmark.ips do |x|
+  x.report("levenshtein ruby") do
+    lev_ruby.call
+  end
+
+  x.report("levenshtein C") do
+    lev_c.call(s, s.size, t, t.size)
+  end
+
+  x.compare!
+end

data/spec/bench/normalizer_rules_bench.rb

@@ -0,0 +1,40 @@
+require "helper"
+
+replacer = InciScore::Normalizer::Rules::Replacer
+downcaser = InciScore::Normalizer::Rules::Downcaser
+beheader = InciScore::Normalizer::Rules::Beheader
+separator = InciScore::Normalizer::Rules::Separator
+tokenizer = InciScore::Normalizer::Rules::Tokenizer
+sanitizer = InciScore::Normalizer::Rules::Sanitizer
+uniquifier = InciScore::Normalizer::Rules::Uniquifier
+src = "‘INGREDIENTS‘:\n\nCOCO—BETANE,AQUA/WATER,DIMETHICONE"
+
+Benchmark.ips do |x|
+  x.report("replacer") do
+    replacer.call(src)
+  end
+
+  x.report("downcaser") do
+    downcaser.call(src)
+  end
+
+  x.report("beheader") do
+    beheader.call(src)
+  end
+
+  x.report("separator") do
+    separator.call(src)
+  end
+
+  x.report("tokenizer") do
+    tokenizer.call(src)
+  end
+
+  x.report("sanitizer") do
+    sanitizer.call(src)
+  end
+
+  x.report("uniquifier") do
+    uniquifier.call(src)
+  end
+end