food_ingredient_parser 1.0.0.pre.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7d20d204e60d22f23f7aa3a039450629c8f4c729
+  data.tar.gz: 127801cb5a00b81ca6abf244d92bfebb39ddde28
+SHA512:
+  metadata.gz: 3a79c07b4f193fcfbbd08f62fdd18407fc034ea036d5b4b5eb3278e449e337e1ee5e0ad5f468bab5f8e948746e99a6bcf5d686e13277b39dafb9c3b11d3ca531
+  data.tar.gz: 95c224f63934e3c601b62d0db9a7d8dc6070fe008816298e3e81f435751012d1bbf66d6270259c4f6e9252a9df6e6d0cf15081e8020e99853c0abb7af55b1a96

data/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2018 Questionmark
+Copyright (c) 2018 wvengen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,115 @@
+# Food ingredient parser
+
+Ingredients are listed on food products in various ways. This [Ruby](https://www.ruby-lang.org/)
+gem and program parses the ingredient text and returns a structured representation.
+
+## Installation
+
+```
+gem install food_ingredient_parser
+```
+
+This will also install the dependency [treetop](http://cjheath.github.io/treetop).
+If you want colored output for the test program, also install [pry](http://pryrepl.org/): `gem install pry`.
+
+## Example
+
+```ruby
+require 'food_ingredient_parser'
+
+s = "Water* 60%, suiker 30%, voedingszuren: citroenzuur, appelzuur, zuurteregelaar: E576/E577, " \
+    + "natuurlijke citroen-limoen aroma's 0,2%, zoetstof: steviolglycosiden, * = Biologisch. " \
+    + "E = door de E.U. goedgekeurde toevoeging."
+parser = FoodIngredientParser::Parser.new
+puts parser.parse(s).to_h.inspect
+```
+Results in
+```
+{
+  :contains=>[
+    {:name=>"Water", :amount=>"60%", :mark=>"*"},
+    {:name=>"suiker", :amount=>"30%"},
+    {:name=>"voedingszuren", :contains=>[
+      {:name=>"citroenzuur"}
+    ]},
+    {:name=>"appelzuur"},
+    {:name=>"zuurteregelaar", :contains=>[
+      {:name=>"E576"},
+      {:name=>"E577"}
+    ]},
+    {:name=>"natuurlijke citroen-limoen aroma's", :amount=>"0,2%"},
+    {:name=>"zoetstof", :contains=>[
+      {:name=>"steviolglycosiden"}
+    ]}
+  ],
+  :notes=>[
+    "* = Biologisch",
+    "E = door de E.U. goedgekeurde toevoeging"
+  ]
+}
+```
+
+## Test tool
+
+The executable `food_ingredient_parser` is available after installing the gem. If you're
+running this from the source tree, use `bin/food_ingredient_parser` instead.
+
+```
+$ food_ingredient_parser -h
+Usage: food_ingredient_parser [options] --file|-f <filename>
+       food_ingredient_parser [options] --string|-s <ingredients>
+
+    -f, --file FILE                  Parse all lines of the file as ingredient lists.
+    -s, --string INGREDIENTS         Parse specified ingredient list.
+    -q, --[no-]quiet                 Only show summary.
+    -p, --parsed                     Only show lines that were successfully parsed.
+    -e, --escape                     Escape newlines
+    -c, --[no-]color                 Use color
+    -n, --noresult                   Only show lines that had no result.
+    -v, --[no-]verbose               Show more data (parsed tree).
+        --version                    Show program version.
+    -h, --help                       Show this help
+
+$ food_ingredient_parser -v -s "tomato"
+"tomato"
+RootNode+Root3 offset=0, "tomato" (contains,notes):
+  SyntaxNode offset=0, ""
+  SyntaxNode offset=0, ""
+  SyntaxNode offset=0, ""
+  ListNode+List13 offset=0, "tomato" (contains):
+    SyntaxNode+List12 offset=0, "tomato" (ingredient):
+      SyntaxNode+Ingredient0 offset=0, "tomato":
+        SyntaxNode offset=0, ""
+        IngredientNode+IngredientSimpleWithAmount3 offset=0, "tomato" (ing):
+          IngredientNode+IngredientSimple5 offset=0, "tomato" (name):
+            SyntaxNode+IngredientSimple4 offset=0, "tomato" (word):
+              SyntaxNode offset=0, "tomato":
+                SyntaxNode offset=0, "t"
+                SyntaxNode offset=1, "o"
+                SyntaxNode offset=2, "m"
+                SyntaxNode offset=3, "a"
+                SyntaxNode offset=4, "t"
+                SyntaxNode offset=5, "o"
+              SyntaxNode offset=6, ""
+        SyntaxNode offset=6, ""
+      SyntaxNode offset=6, ""
+  SyntaxNode+Root2 offset=6, "":
+    SyntaxNode offset=6, ""
+    SyntaxNode offset=6, ""
+    SyntaxNode offset=6, ""
+  SyntaxNode offset=6, ""
+{:contains=>[{:name=>"tomato"}]}
+
+$ food_ingredient_parser -q -f data/test-cases
+parsed 35 (100.0%), no result 0 (0.0%)
+```
+
+If you want to use the output in (shell)scripts, the options `-e -c` may be quite useful.
+
+## Test data
+
+[`data/ingredient-samples-nl`](data/ingredient-samples-nl) contains about 150k
+real-world ingredient lists found on the Dutch market. Each line contains one ingredient
+list (newlines are encoded as `\n`, empty lines and those starting with `#` are ignored).
+Currently almost three quarter is recognized and parsed. We aim to reach at least 90%.
+

data/bin/food_ingredient_parser

@@ -0,0 +1,107 @@
+#!/usr/bin/env ruby
+#
+# Parser for food ingredient lists.
+#
+require 'optparse'
+
+$:.push(File.expand_path(File.dirname(__FILE__) + "/../lib"))
+require 'food_ingredient_parser'
+
+begin
+  require 'pry'
+  def pp(o, color: true)
+    if color
+      Pry::ColorPrinter.pp(o)
+    else
+      puts(o.inspect)
+    end
+  end
+rescue LoadError
+  # fallback without color printing
+  def pp(o, color: nil)
+    puts(o.inspect)
+  end
+end
+
+def colorize(color, s)
+  if color
+    "\e[#{color}m#{s}\e[0;22m"
+  else
+    s
+  end
+end
+
+def parse_single(s, parsed=nil, parser: nil, verbosity: 1, print: nil, escape: false, color: false)
+  parser ||= FoodIngredientParser::Parser.new
+  parsed ||= parser.parse(s)
+
+  return unless print.nil? || (parsed && print == :parsed) || (!parsed && print == :noresult)
+
+  puts colorize(color && "0;32", escape ? s.gsub("\n", "\\n") : s) if verbosity > 0
+
+  if parsed
+    puts(parsed.inspect) if verbosity > 1
+    pp(parsed.to_h, color: color) if verbosity > 0
+  else
+    puts "(no result: #{parser.failure_reason})" if verbosity > 0
+  end
+end
+
+def parse_file(path, parser: nil, verbosity: 1, print: nil, escape: false, color: false)
+  count_parsed = count_noresult = 0
+  File.foreach(path) do |line|
+    next if line =~ /^#/    # comment
+    next if line =~ /^\s*$/ # empty line
+
+    line = line.gsub('\\n', "\n").strip
+    parsed = parser.parse(line)
+    count_parsed += 1 if parsed
+    count_noresult += 1 unless parsed
+
+    parse_single(line, parsed, parser: parser, verbosity: verbosity, print: print, escape: escape, color: color)
+  end
+
+  pct_parsed   = 100.0 * count_parsed / (count_parsed + count_noresult)
+  pct_noresult = 100.0 * count_noresult / (count_parsed + count_noresult)
+  puts "parsed #{colorize(color && "1;32", count_parsed)} (#{pct_parsed.round(1)}%), no result #{colorize(color && "1;31", count_noresult)} (#{pct_noresult.round(1)}%)"
+end
+
+verbosity = 1
+files = []
+strings = []
+print = nil
+escape = false
+color = true
+OptionParser.new do |opts|
+  opts.banner = <<-EOF.gsub(/^    /, '')
+    Usage: #{$0} [options] --file|-f <filename>
+           #{$0} [options] --string|-s <ingredients>
+
+  EOF
+
+  opts.on("-f", "--file FILE", "Parse all lines of the file as ingredient lists.") {|f| files << f }
+  opts.on("-s", "--string INGREDIENTS", "Parse specified ingredient list.") {|s| strings << s }
+
+  opts.on("-q", "--[no-]quiet", "Only show summary.") {|q| verbosity = q ? 0 : 1 }
+  opts.on("-p", "--parsed", "Only show lines that were successfully parsed.") {|p| print = :parsed }
+  opts.on("-e", "--escape", "Escape newlines") {|e| escape = true }
+  opts.on("-c", "--[no-]color", "Use color") {|e| color = !!e }
+  opts.on("-n", "--noresult", "Only show lines that had no result.") {|p| print = :noresult }
+  opts.on("-v", "--[no-]verbose", "Show more data (parsed tree).") {|v| verbosity = v ? 2 : 1 }
+  opts.on(      "--version", "Show program version.") do
+    puts("food_ingredient_parser v#{FoodIngredientParser::VERSION}")
+    exit
+  end
+  opts.on("-h", "--help", "Show this help") do
+    puts(opts)
+    exit
+  end
+end.parse!
+
+if strings.any? || files.any?
+  parser = FoodIngredientParser::Parser.new
+  strings.each {|s| parse_single(s, parser: parser, verbosity: verbosity, print: print, escape: escape, color: color) }
+  files.each   {|f| parse_file(f, parser: parser, verbosity: verbosity, print: print, escape: escape, color: color) }
+else
+  STDERR.puts("Please specify one or more --file or --string arguments (see --help).")
+end

data/food_ingredient_parser.gemspec

@@ -0,0 +1,29 @@
+$:.unshift(File.expand_path(File.dirname(__FILE__) + '/lib'))
+require 'food_ingredient_parser/version'
+
+Gem::Specification.new do |s|
+  s.name             = 'food_ingredient_parser'
+  s.version          = FoodIngredientParser::VERSION
+  s.date             = '2018-08-09'
+  s.summary          = 'Parser for ingredient lists found on food products.'
+  s.authors          = ['wvengen']
+  s.email            = ['dev-ruby@willem.engen.nl']
+  s.homepage         = 'https://github.com/q-m/food-ingredient-parser-ruby'
+  s.license          = 'MIT'
+  s.description      = <<-EOD
+    Food products often contain an ingredient list of some sort. This parser
+    tries to recognize the syntax and returns a structured representation of the
+    food ingredients.
+  EOD
+  s.metadata         = {
+    'bug_tracker_uri' => 'https://github.com/q-m/food-ingredient-parser-ruby/issues',
+    'source_code_uri' => 'https://github.com/q-m/food-ingredient-parser-ruby',
+  }
+
+  s.files            = `git ls-files *.gemspec lib`.split("\n")
+  s.executables      = `git ls-files bin`.split("\n").map(&File.method(:basename))
+  s.extra_rdoc_files = ['README.md', 'LICENSE']
+  s.require_paths    = ['lib']
+
+  s.add_runtime_dependency 'treetop', '~> 1.6'
+end

data/lib/food_ingredient_parser/grammar/amount.treetop

@@ -0,0 +1,30 @@
+module FoodIngredientParser::Grammar
+  grammar Amount
+    include Common
+
+    rule amount
+      '(' ws* amount:simple_amount ws* ')' <AmountNode> /
+      '[' ws* amount:simple_amount ws* ']' <AmountNode> /
+      '{' ws* amount:simple_amount ws* '}' <AmountNode> /
+      amount:simple_amount                 <AmountNode>
+    end
+
+    rule simple_amount
+      ( (
+        'of which' / 'at least' /
+        'waarvan' / 'ten minste' / 'tenminste' / 'minimaal'
+      ) ws* )?
+      [<>]? ws*
+      simple_amount_quantity
+      ( ws+ (
+        'minimum' /
+        'minimaal' / 'minimum'
+      ) )?
+    end
+
+    rule simple_amount_quantity
+      number ( ws* '-' ws* number )? ws* ( '%' / 'g' / 'mg' / 'gram' / 'ml' )
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/common.treetop

@@ -0,0 +1,150 @@
+module FoodIngredientParser::Grammar
+  grammar Common
+
+    rule ws
+      " " / "\t"
+    end
+
+    rule newline
+      "\n"
+    end
+
+    rule char
+      [[:alnum:]] /
+      fraction /
+      [-/\'’+=_{}&] /
+      [®] /
+      [¿?] /                         # weird characters turning up in names (e.g. encoding issues)
+      [₁₂₃₄₅₆₇₈₉]                    # can occur with vitamins
+      #[A-Za-z0-9] / [-\'+/=?;!*#@$_%] / [#xC0-#xD6] / [#xD8-#xF6] / [#xF8-#x2FF] / [#x370-#x37D] / [#x37F-#x1FFF] / [#x200C-#x200D] / [#x2070-#x218F] / [#x2C00-#x2FEF] / [#x3001-#xD7FF] / [#xF900-#xFDCF] / [#xFDF0-#xFFFD] / [#x10000-#xEFFFF]
+    end
+
+    rule mark
+      # mark referencing a footnote
+      [¹²³⁴⁵ᵃᵇᶜᵈᵉᶠᵍªº] '⁾'? / '⁽' [¹²³⁴⁵ᵃᵇᶜᵈᵉᶠᵍªº] '⁾' / [†‡•°#^] / '*'+
+    end
+
+    rule digit
+      [0-9]
+    end
+
+    rule fraction
+      [½⅓⅔¼¾⅕⅖⅗⅘⅙⅚⅐⅛⅜⅝⅞⅑⅒]
+    end
+
+    rule number
+      digit+ [,.] digit+ / digit+ ws* fraction / fraction / digit+
+    end
+
+    rule word
+      abbrev / char+
+    end
+
+    rule word_nas
+      # word, but don't include the trailing '-' that may occure before an 'and'
+      abbrev / ( !andsep char )+
+    end
+
+    rule and
+      ( 'and' / 'en' / 'und' / '&' ) !char
+    end
+
+    # we want to match "a and b" but not "a- and bthing", this allows to avoid the latter
+    rule andsep
+      '-' ws+ and
+    end
+
+    rule abbrev
+      # These are listed explicitely to avoid incorrect interpretations, and allow missing trailing dots.
+      # To get an idea of what occurs (second one omits trailing dots):
+      #   cat data/ingredient-samples-nl | perl -ne '$_=lc($_); /\b(([a-z]\.)+[a-z]\.?)\W/ && print "$1\n"' | sort | uniq -c | sort -rn
+      #   cat data/ingredient-samples-nl | perl -ne '$_=lc($_); /\b(([a-z]\.)+[a-z])\W/ && print "$1\n"' | sort | uniq -c | sort -rn
+      # Finally, you can generate the full list using this command:
+      #   cat data/ingredient-samples-nl | perl -ne '$_=lc($_); /\b(([a-z]\.)+[a-z])\W/ && print "$1\n"' | sort | uniq | sed "s/^/'/;s/$/'i \//"
+      (
+        'a.o.p'i /
+        'b.g.a'i /
+        'b.o.b'i /
+        'c.a'i /
+        'c.i'i /
+        'd.e'i /
+        'd.m.v'i /
+        'd.o.c'i /
+        'd.o.p'i /
+        'd.s'i /
+        'e.a'i /
+        'e.g'i /
+        'e.u'i /
+        'f.i.l'i /
+        'f.o.s'i /
+        'i.a'i /
+        'i.d'i /
+        'i.e'i /
+        'i.g.m.e'i /
+        'i.g.p'i /
+        'i.m.v'i /
+        'i.o'i /
+        'i.v.m'i /
+        'l.s.l'i /
+        'n.a'i /
+        'n.b'i /
+        'n.o'i /
+        'n.v.t'i /
+        'o.a'i /
+        'o.b.v'i /
+        'p.d.o'i /
+        'p.g.i'i /
+        'q.s'i /
+        's.l'i /
+        's.s'i /
+        't.o.v'i /
+        'u.h.t'i /
+        'v.g'i /
+        'v.s'i /
+        'w.a'i /
+        'w.o'i /
+        'w.v'i /
+        # special words and abbreviations (not auto-generated)
+        'vit.'i /
+        'denat.'i /
+        'N°'i /
+        '°C'i /
+        # word combinations that should not be split (not auto-generated)
+        # @todo this really would benefit from matching known ingredients instead of hardcoding
+        ( 'oliën'i / 'olien'i / 'olië'i / 'olie'i ) ws+ and ws+ ( 'vetten'i / 'vet'i ) /
+        'palm'i ws+ and ws+ 'kokosvet'i /
+        color ( ws+ and ws+ color )+ /
+        color2 ( ws+ and ws+ color2 )+ /
+        'kruiden'i ws+ and ws+ 'specerijen'i /
+        'kruiden'i ws+ and ws+ 'specerij'i /
+        'specerijen'i ws+ and ws+ 'kruiden'i /
+        'vitamine'i 'n'i? ws+ and ws+ 'mineralen'i /
+        'lactose'i ws+ and ws+ 'melk'i ( 'eiwit'i 'en'i? )?  /
+        'granen'i ws+ and ws+ 'zaden'i /
+        'gekookt'i [eE]? ws+ and ws+ 'gemarineerd'i [eE]? /
+        'mono'i ws+ and ws+ 'diglyceriden'i /
+        'guarpitmeel'i ws+ and ws+ 'natriumalginaat'i /
+        'vlees'i ws+ and ws+ 'dierlijke bijproducten'i /
+        'vis'i ws+ and ws+ 'visbijproducten'i /
+        'glucose'i ws+ and ws+ 'fructosestroop'i /
+        'ijzeroxiden'i ws+ and ws+ 'hydroxiden'i /
+        char+ 'sap'i ws+ and ws+ 'overige vruchtensappen'i /
+        char* 'sap'i ( ws+ 'uit concentraat'i / ws+ 'uit sapconcentraat'i )? ws+ and ws+ 'vruchten'i? 'puree'i /
+        ( 'vit.'i / 'vitamine'i / 'vitamin' ) ws+ [a-zA-Z] [0-9]* ws+ and ws+ [a-zA-Z] [0-9]*
+      )
+      '.'? ![[:alpha:]]
+    end
+
+    rule color
+      # used for paprika, honey ("yellow and white honey") (nouns)
+      'red'i / 'green'i / 'yellow'i / 'white'i / 'black'i /
+      'rood'i / 'groen'i / 'geel'i / 'wit'i / 'zwart'i
+    end
+
+    rule color2
+      # adjective colors (can not occur together with noun colors in a list)
+      'rode'i / 'groene'i / 'gele'i / 'witte'i / 'zwarte'i
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/ingredient.treetop

@@ -0,0 +1,12 @@
+module FoodIngredientParser::Grammar
+  grammar Ingredient
+    include IngredientSimple
+    include IngredientNested
+    include IngredientColoned
+
+    rule ingredient
+      ws* ( ingredient_nested / ingredient_coloned / ingredient_simple_with_amount )
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/ingredient_coloned.treetop

@@ -0,0 +1,41 @@
+module FoodIngredientParser::Grammar
+  grammar IngredientColoned
+    include Common
+    include Amount
+    include IngredientSimple
+
+    rule ingredient_coloned
+      ing:ingredient_simple ws* ':' ws* amount:amount post:( ws* '}' )? !( ws* word )                <IngredientNode> /
+      ing:ingredient_simple ws* ':' post:( ws* '}' )? ws* contains:( ingredient_coloned_inner_list ) <NestedIngredientNode>
+    end
+
+    rule ingredient_coloned_inner_list
+      contains:( ingredient_coloned_simple_with_amount_and_nest ( ws+ and ws+ ingredient_coloned_simple_with_amount_and_nest )+ )  <ListNode> /
+      contains:( ingredient_coloned_simple_with_amount_and_nest ws* ( '/'+ ws* ingredient_coloned_simple_with_amount_and_nest )* ) <ListNode>
+    end
+
+    # @see IngredientSimple#ingredient_simple
+    rule ingredient_coloned_simple
+      name:( ingredient_coloned_word_nas ( andsep? ws+ !amount !and ingredient_coloned_word_nas )* ) ws? mark:mark <IngredientNode> /
+      name:( ingredient_coloned_word_nas ( andsep? ws+ !amount !and ingredient_coloned_word_nas )* )               <IngredientNode>
+    end
+
+    # @see IngredientSimple#ingredient_simple_with_amount
+    rule ingredient_coloned_simple_with_amount
+      pre:( '{' ws* )? amount:amount ws+ ing:ingredient_coloned_simple                   <IngredientNode> /
+      ing:ingredient_coloned_simple ws* amount:amount post:( ws* '}' )? (ws? mark:mark)? <IngredientNode> /
+      ing:ingredient_coloned_simple                                                      <IngredientNode>
+    end
+
+    rule ingredient_coloned_simple_with_amount_and_nest
+      ing:ingredient_coloned_simple_with_amount ws* '(' ws* contains:ingredient_coloned_simple_with_amount ws* ')' ( ws* '}' )? <NestedIngredientNode> /
+      ingredient_coloned_simple_with_amount
+    end
+
+    # @see Common#word
+    rule ingredient_coloned_word_nas
+      abbrev / ( !'/' !andsep char )+
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/ingredient_nested.treetop

@@ -0,0 +1,28 @@
+module FoodIngredientParser::Grammar
+  grammar IngredientNested
+    include Common
+    include Amount
+    include IngredientSimple
+
+    rule ingredient_nested
+      ( ing:ingredient_simple              ws* '(' contains:ingredient_nested_in ws* ')' ws? mark:mark ws* amount:amount <NestedIngredientNode> ) /
+      ( ing:ingredient_simple              ws* '(' contains:ingredient_nested_in ws* ')'               ws* amount:amount <NestedIngredientNode> ) /
+      ( ing:ingredient_simple_with_amount  ws* '(' contains:ingredient_nested_in ws* ')' ws? mark:mark                   <NestedIngredientNode> ) /
+      ( ing:ingredient_simple_with_amount  ws* '(' contains:ingredient_nested_in ws* ')'                                 <NestedIngredientNode> ) /
+      ( ing:ingredient_simple              ws* '[' contains:ingredient_nested_in ws* ']' ws? mark:mark ws* amount:amount <NestedIngredientNode> ) /
+      ( ing:ingredient_simple              ws* '[' contains:ingredient_nested_in ws* ']'               ws* amount:amount <NestedIngredientNode> ) /
+      ( ing:ingredient_simple_with_amount  ws* '[' contains:ingredient_nested_in ws* ']' ws? mark:mark                   <NestedIngredientNode> ) /
+      ( ing:ingredient_simple_with_amount  ws* '[' contains:ingredient_nested_in ws* ']'                                 <NestedIngredientNode> )
+    end
+
+    rule ingredient_nested_in
+      ( ingredient_nested_contains (ws* ':')? ws+ )? ws* contains:list ws* '.'?
+    end
+
+    rule ingredient_nested_contains
+      'contains'i /
+      'bevat'i
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/ingredient_simple.treetop

@@ -0,0 +1,21 @@
+module FoodIngredientParser::Grammar
+  grammar IngredientSimple
+    include Common
+    include Amount
+
+    rule ingredient_simple
+      name:( word_nas ( andsep? ws+ !amount !and word_nas )* ) ws? mark:mark <IngredientNode> /
+      name:( word_nas ( andsep? ws+ !amount !and word_nas )* )               <IngredientNode> /
+      # We've tried to omit 'and' from the ingredient, but if it doesn't work out, do it anyway.
+      name:( word_nas ( andsep? ws+ !amount      word_nas )* ) ws? mark:mark <IngredientNode> /
+      name:( word_nas ( andsep? ws+ !amount      word_nas )* )               <IngredientNode>
+    end
+
+    rule ingredient_simple_with_amount
+      pre:( '{' ws* )? amount:amount ws+ ing:ingredient_simple                   <IngredientNode> /
+      ing:ingredient_simple ws* amount:amount post:( ws* '}' )? (ws? mark:mark)? <IngredientNode> /
+      ing:ingredient_simple                                                      <IngredientNode>
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/list.treetop

@@ -0,0 +1,14 @@
+module FoodIngredientParser::Grammar
+  grammar List
+    include Common
+    include Ingredient
+
+    rule list
+      contains:(ingredient ( ws* '|' ws* ingredient )+ ( ws+ and ws+ ingredient )? ) <ListNode> /
+      contains:(ingredient ( ws* ';' ws* ingredient )+ ( ws+ and ws+ ingredient )? ) <ListNode> /
+      contains:(ingredient ( ws* ',' ws* ingredient )+ ( ws+ and ws+ ingredient )? ) <ListNode> /
+      contains:(ingredient ( ws* '.' ws* ingredient )+ ( ws+ and ws+ ingredient )? ) <ListNode> /
+      contains:(ingredient ( ws+ and ws+ ingredient )? )                             <ListNode>
+    end
+  end
+end

data/lib/food_ingredient_parser/grammar/list_coloned.treetop

@@ -0,0 +1,23 @@
+module FoodIngredientParser::Grammar
+  grammar ListColoned
+    include Common
+    include IngredientSimple
+    include Ingredient
+
+    rule list_coloned
+      contains:( ( list_coloned_ingredient ws* '.' ws* )+ list_coloned_ingredient? ) <ListNode> /
+      contains:( ( list_coloned_ingredient ws* ';' ws* )+ list_coloned_ingredient? ) <ListNode> /
+      contains:(   list_coloned_ingredient )                                         <ListNode>
+    end
+
+    rule list_coloned_inner_list
+      contains:( ingredient ( ws* ',' ws* ingredient )* ) <ListNode>
+    end
+
+    rule list_coloned_ingredient
+      ing:ingredient_simple_with_amount ws* ':' ws* amount:amount post:( ws* '}' )?                    <IngredientNode> /
+      ing:ingredient_simple_with_amount ws* ':' post:( ws* '}' )? ws* contains:list_coloned_inner_list <NestedIngredientNode>
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/list_newlined.treetop

@@ -0,0 +1,16 @@
+module FoodIngredientParser::Grammar
+  grammar ListNewlined
+    include Common
+    include IngredientSimple
+    include List
+
+    rule list_newlined
+      contains:( ( list_newlined_ingredient_nested ws* newline newline )* list_newlined_ingredient_nested ) <ListNode>
+    end
+
+    rule list_newlined_ingredient_nested
+      ws* ing:ingredient_simple ws* ':'? ws* newline contains:list ( ws* '.' )? <NestedIngredientNode>
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar/root.treetop

@@ -0,0 +1,51 @@
+module FoodIngredientParser::Grammar
+  grammar Root
+    include Common
+    include List
+    include ListColoned
+    include ListNewlined
+
+    rule root
+      '"'?
+      root_prefix? ws*
+      contains:( list_newlined / list_coloned / list )
+      notes:(
+        root_mark_sentences_in_list? ws*
+        ( ( '.' ws* newline* / '.'?  ws* newline+ ) ws* root_sentences? ws* )?
+      )
+      '"'?
+      <RootNode>
+    end
+
+    rule root_prefix
+      (
+        'ingredients'i / 'contains'i /
+        ('ingred'i [IÏiï] [EËeë] 'n'i ( 't'i 'en'i? 'declaratie'i? )? ) / 'bevat'i / 'dit zit er in'i / 'samenstelling'i /
+        'zutaten'i
+      )
+      ( ws* [:;.] ( ws* newline )? / ws* newline / ws ) ws*  # optional colon or other separator
+      "'"? ws*                                               # stray quote occurs sometimes
+    end
+
+    rule root_sentences
+      ( root_sentence ws* )+ root_sentence_open? / root_sentence_open
+    end
+
+    rule root_sentence
+      root_sentence_open ( '.' / ';' / newline+ )
+    end
+
+    rule root_sentence_open
+      ( word / ws / [,:()%] / '[' / ']' / mark )+ <NoteNode>
+    end
+
+    rule root_mark_sentences_in_list
+      ( ( ws* [,.] / ws ) ws* root_mark_sentence_in_list )+
+    end
+
+    rule root_mark_sentence_in_list
+      mark ws* root_sentence_open <NoteNode>
+    end
+
+  end
+end

data/lib/food_ingredient_parser/grammar.rb

@@ -0,0 +1,16 @@
+require 'treetop'
+require_relative 'nodes'
+
+# @todo find a way to auto-generate Ruby from Treetop files when building gem,
+#       see https://stackoverflow.com/q/37794587/2866660
+
+Treetop.load File.dirname(__FILE__) + '/grammar/common'
+Treetop.load File.dirname(__FILE__) + '/grammar/amount'
+Treetop.load File.dirname(__FILE__) + '/grammar/ingredient_simple'
+Treetop.load File.dirname(__FILE__) + '/grammar/ingredient_nested'
+Treetop.load File.dirname(__FILE__) + '/grammar/ingredient_coloned'
+Treetop.load File.dirname(__FILE__) + '/grammar/ingredient'
+Treetop.load File.dirname(__FILE__) + '/grammar/list'
+Treetop.load File.dirname(__FILE__) + '/grammar/list_coloned'
+Treetop.load File.dirname(__FILE__) + '/grammar/list_newlined'
+Treetop.load File.dirname(__FILE__) + '/grammar/root'

data/lib/food_ingredient_parser/nodes.rb

@@ -0,0 +1,69 @@
+require 'treetop/runtime'
+
+# Needs to be in grammar namespace so Treetop can find the nodes.
+module FoodIngredientParser::Grammar
+
+  # Treetop syntax node with our additions, use this as parent for all our own nodes.
+  class SyntaxNode < Treetop::Runtime::SyntaxNode
+    private
+
+    def to_a_deep(n, cls)
+      if n.is_a?(cls)
+        [n]
+      elsif n.nonterminal?
+        n.elements.map {|m| to_a_deep(m, cls) }.flatten(1).compact
+      end
+    end
+  end
+
+  # Root object, contains everything else.
+  class RootNode < SyntaxNode
+    def to_h
+      h = { contains: contains.to_a }
+      if notes && notes_ary = to_a_deep(notes, NoteNode)&.map(&:text_value)
+        h[:notes] = notes_ary if notes_ary.length > 0
+      end
+      h
+    end
+  end
+
+  # List of ingredients.
+  class ListNode < SyntaxNode
+    def to_a
+      to_a_deep(contains, IngredientNode).map(&:to_h)
+    end
+  end
+
+  # Ingredient
+  class IngredientNode < SyntaxNode
+    def to_h
+      h = {}
+      h.merge!(to_a_deep(ing, IngredientNode)&.first&.to_h || {}) if respond_to?(:ing)
+      h.merge!(to_a_deep(amount, AmountNode)&.first&.to_h || {}) if respond_to?(:amount)
+      h[:name] = name.text_value if respond_to?(:name)
+      h[:name] = pre.text_value + h[:name] if respond_to?(:pre)
+      h[:name] = h[:name] + post.text_value if respond_to?(:post)
+      h[:mark] = mark.text_value if respond_to?(:mark) && mark.text_value != ''
+      h
+    end
+  end
+
+  # Ingredient with containing ingredients.
+  class NestedIngredientNode < IngredientNode
+    def to_h
+      super.merge({ contains: to_a_deep(contains, IngredientNode).map(&:to_h) })
+    end
+  end
+
+  # Amount, specifying an ingredient.
+  class AmountNode < SyntaxNode
+    def to_h
+      { amount: amount.text_value }
+    end
+  end
+
+  # Note at the end of the ingredient list.
+  class NoteNode < SyntaxNode
+  end
+
+end

data/lib/food_ingredient_parser/parser.rb

@@ -0,0 +1,34 @@
+require_relative 'grammar'
+
+module FoodIngredientParser
+  class Parser
+
+    # Create a new food ingredient parser
+    # @return [FoodIngredientParser]
+    def initialize
+      @parser = Grammar::RootParser.new
+    end
+
+    # Parse food ingredient list text into a structured representation.
+    # @option clean [Boolean] pass +false+ to disable correcting frequently occuring issues
+    # @return [Hash] structured representation of food ingredients
+    def parse(s, clean: true)
+      s = clean(s) if clean
+      @parser.parse(s)
+    end
+
+    private
+
+    def clean(s)
+      s.gsub!("\u00ad", "")             # strip soft hyphen
+      s.gsub!("\u0092", "'")            # windows-1252 apostrophe - https://stackoverflow.com/a/15564279/2866660
+      s.gsub!("aÄs", "aïs")             # encoding issue for maïs
+      s.gsub!("ï", "ï")                # encoding issue
+      s.gsub!("ë", "ë")                # encoding issue
+      s.gsub!(/\A\s*"(.*)"\s*\z/, '\1') # enclosing double quotation marks
+      s.gsub!(/\A\s*'(.*)'\s*\z/, '\1') # enclosing single quotation marks
+      s
+    end
+
+  end
+end

data/lib/food_ingredient_parser/version.rb

@@ -0,0 +1,3 @@
+module FoodIngredientParser
+  VERSION = '1.0.0.pre.1'
+end

data/lib/food_ingredient_parser.rb

@@ -0,0 +1,2 @@
+require_relative 'food_ingredient_parser/version'
+require_relative 'food_ingredient_parser/parser'

metadata

@@ -0,0 +1,85 @@
+--- !ruby/object:Gem::Specification
+name: food_ingredient_parser
+version: !ruby/object:Gem::Version
+  version: 1.0.0.pre.1
+platform: ruby
+authors:
+- wvengen
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-08-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: treetop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+description: |2
+      Food products often contain an ingredient list of some sort. This parser
+      tries to recognize the syntax and returns a structured representation of the
+      food ingredients.
+email:
+- dev-ruby@willem.engen.nl
+executables:
+- food_ingredient_parser
+extensions: []
+extra_rdoc_files:
+- README.md
+- LICENSE
+files:
+- LICENSE
+- README.md
+- bin/food_ingredient_parser
+- food_ingredient_parser.gemspec
+- lib/food_ingredient_parser.rb
+- lib/food_ingredient_parser/grammar.rb
+- lib/food_ingredient_parser/grammar/amount.treetop
+- lib/food_ingredient_parser/grammar/common.treetop
+- lib/food_ingredient_parser/grammar/ingredient.treetop
+- lib/food_ingredient_parser/grammar/ingredient_coloned.treetop
+- lib/food_ingredient_parser/grammar/ingredient_nested.treetop
+- lib/food_ingredient_parser/grammar/ingredient_simple.treetop
+- lib/food_ingredient_parser/grammar/list.treetop
+- lib/food_ingredient_parser/grammar/list_coloned.treetop
+- lib/food_ingredient_parser/grammar/list_newlined.treetop
+- lib/food_ingredient_parser/grammar/root.treetop
+- lib/food_ingredient_parser/nodes.rb
+- lib/food_ingredient_parser/parser.rb
+- lib/food_ingredient_parser/version.rb
+homepage: https://github.com/q-m/food-ingredient-parser-ruby
+licenses:
+- MIT
+metadata:
+  bug_tracker_uri: https://github.com/q-m/food-ingredient-parser-ruby/issues
+  source_code_uri: https://github.com/q-m/food-ingredient-parser-ruby
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">"
+    - !ruby/object:Gem::Version
+      version: 1.3.1
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.13
+signing_key: 
+specification_version: 4
+summary: Parser for ingredient lists found on food products.
+test_files: []