attentive 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 96872811cbdc5d9db062e11cca2471dc4079a26d
-  data.tar.gz: 0df0336e5edf492c51048473605c69a7e9fe1886
+  metadata.gz: b5952398cd8d68e82b27a4e9be335b748b315845
+  data.tar.gz: 77f1e15ef3e6952861d110f3c5d7605664e2979f
 SHA512:
-  metadata.gz: 61869eae6aacb6c2a66aca38e55ef1ca927236d8239ad402c761744e88a8a9dad99006b5fec14648fe740d43df83a2678c45fad1403ce0f3444fa444261971c2
-  data.tar.gz: 4305e44d30e464ff734a72aa3326b9525776ba35f45330cbef4018d8541f754300bd41e4395531831cc628b6ba58ed7f1f7f26511f29d2bdfcd5f87680e3dec5
+  metadata.gz: 49de7e3eff7a8964082ffd35ac964497c5a7c706194f160fcb29bad5710c6977fc1d58ff86189d788cbe8290145575b07147cb35cafcbe16167b10607cd29063
+  data.tar.gz: b7692af23cbaa6b44467ab64c989883477ade8c1a9e26133c6c27e69168dbd132ba4e6dc7b4abbd8a8b5efeda09714a3138181e3e49290d16f449320f7e4d7d5

data/README.md

@@ -1,43 +1,196 @@
 # Attentive
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/attentive`. To experiment with that code, run `bin/console` for an interactive prompt.
+Attentive is a library for matching messages to natural-language listeners.
 
-TODO: Delete this and the text above, and describe your gem
 
 
+<br/>
+## Usage
 
-## Installation
+Its basic usage is like this:
+
+```ruby
+include Attentive
+
+listen_for "hi", context: { in: :any } do
+  puts "nice to meet you!"
+end
+
+hear! "hi!" # => "nice to meet you!"
+```
+
+In the snippet above,
+
+  1. We defined a [listener](#listeners) that is active in any [context](#contexts).
+  2. We received a message.
+  3. Attentive matched the message to our listener and invoked the block.
 
-Add this line to your application's Gemfile:
+
+<br/>
+#### Optional Characters
+
+You'll notice that we listened for `"hi"` but _heard_ `"hi!"`. Attentive treats punctuation and emojis as optional; but we can make them required by putting them in the listener:
 
 ```ruby
-gem 'attentive'
+listen_for "hi!", context: { in: :any } do
+  puts "nice to meet you!"
+end
+
+hear! "hi" # => nothing happened, the listener is expecting the exclamation mark
 ```
 
-And then execute:
+> It's best to leave all but the most necessary punctuation out of listeners.
 
-    $ bundle
 
-Or install it yourself as:
+<br/>
+#### Contractions and Abbreviations
 
-    $ gem install attentive
+Attentive understands contractions and abbreviations and can match those:
 
+```ruby
+listen_for "hi", context: { in: :any } do
+  puts "nice to meet you!"
+end
 
+hear! "hello!" # => "nice to meet you!"
+
+listen_for "what is for lunch", context: { in: :any } do
+  puts "HAMBURGERS!"
+end
+
+hear! "what's for lunch?" # => "HAMBURGERS!"
+```
+
+> Although you _can_ use contractions and abbreviations in listeners, it's a good habit not to. Attentive will not let you define listeners that use ambiguous contractions like `"where's"` (`"where's"` might be a contraction for `"where is"`, `"where does"`, or `"where has"`, or `"where was"`).
 
-## Usage
+
+<br/>
+#### Listeners
+
+Listeners are defined with three things:
+
+  1. One or more phrases
+  2. A set of [contexts](#contexts) where they're active
+  3. A block to be invoked when the listener is matched
+
+Here's an example of a listener that matches more than one phrase:
 
 ```ruby
-include Attentive
+listen_for "what is for lunch",
+           "what is for lunch {{date:core.date.relative.future}}",
+           "what is for lunch on {{date:core.date}}",
+           "show me the menu for {{date:core.date.relative.future}}",
+           "show me the menu for {{date:core.date}}" do
+  # ...
+end
+```
+
+(In the example above, the phrases `{{date:core.date.relative.future}}` and `{{date:core.date}}` are [entities](#entities): which we'll cover in a minute.)
+
+
+<br/>
+#### Contexts
+
+A listener can require that messages be heard in a certain context in order to be matched or it can ignore messages if they are heard in certain contexts.
 
-listen_for "hi there!" do
-  puts "heard greeting"
+The following is a listener that will only match messages heard in the "#general" channel and only then if the conversation is not "serious".
+
+```ruby
+listen_for "ouch", context: { in: %i{general}, not_in: %i{serious} } do
+  puts "On a scale of 1 to 10, how would you rate your pain?"
 end
 
-hear "hi, there :wink:"
+hear! "ouch" # => message has no context, listener isn't triggered
+hear! "ouch", contexts: %i{general} # => "On a scale of 1 to 10..."
+hear! "ouch", contexts: %i{general serious} # => listener ignores "serious" messages
 ```
 
+If you don't specify context requirements for listeners, Attentive requires `conversation` and prohibits `quotation` by default:
+
+```ruby
+# These two are the same:
+listen_for "ouch"
+listen_for "ouch", context: { in: %i{conversation}, not_in: %i{quotation} }
+```
+
+
+
+<br/>
+#### Entities
+
+Entities allow Attentive to match **_concepts_** rather than specific words.
+
+There are built-in entities like `core.date`, `core.number`, and `core.email` for recognizing dates, numbers, and email addresses (see [Core Entities](https://github.com/houston/attentive/wiki/Core-Entities) for a complete list); but you can also define entities for domain-specific concepts. For example:
+
+```ruby
+Attentive::Entity.define "deweys.menu.beers",
+  "Bell's Oberon",
+  "Rogue Dead Guy Ale",
+  "Schalfly Dry Hopped IPA",
+  "4 Hands Contact High",
+  "Scrimshaw Pilsner"
+```
+
+Now we can take drink orders:
+
+```ruby
+listen_for "I will have a pint of the {{deweys.menu.beers}}" do
+  puts "Good choice"
+end
+```
+
+> It is a good idea to namespace entities (i.e. `deweys.menu.beers`). Attentive's convention is to treat namespaces as a taxonomy for concepts.
+
+
+<br/>
+#### Regular Expressions
+
+As useful as enumerations are, entities can also be defined with regular expressions and with a block that converts the matched part of the message to a more useful value:
+
+```ruby
+# Usernames can be up to 21 characters long.
+# They can contain lowercase letters a to z
+# (without accents), and numbers 0 to 9.
+
+Attentive::Entity.define "slack.user", %q{(?<username>[a-z0-9]{1,21})} do |match|
+  Slack::User.find match["username"]
+end
+```
+
+> Whenever possible, though, prefer composing entities to using regular expressions.
+> For example:
+> ```ruby
+Attentive::Entity.define "core.date.relative.future",
+  "next {{core.date.wday}}"
+```
+> is better than:
+> ```ruby
+Attentive::Entity.define "core.date.relative.future",
+  "next (?<weekday>(:sun|mon|tues|wednes|thurs|fri|satur)day)"
+```
+
+
+
+<br/>
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'attentive'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install attentive
+
 
 
+<br/>
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -46,12 +199,14 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 
 
+<br/>
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/attentive.
+Bug reports and pull requests are welcome on GitHub at https://github.com/houston/attentive.
 
 
 
+<br/>
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/lib/attentive.rb

@@ -7,6 +7,14 @@ module Attentive
   # Default configuration
   self.invocations = ["@me".freeze]
 
+  # Default contexts that listeners will require
+  # a message to be heard in.
+  self.default_required_contexts = %i{conversation}
+
+  # Default contexts in which listeners will ignore messages.
+  self.default_prohibited_contexts = %i{quotation}
+
+
 
   # Attentive DSL
 
@@ -18,11 +26,22 @@ module Attentive
     listeners.listen_for(*args, &block)
   end
 
+  # Matches a message against all listeners
+  # and returns an array of matches
   def hear(message, params={})
     message = Attentive::Message.new(message, params) unless message.is_a?(Attentive::Message)
     listeners.hear message
   end
 
+  # Matches a message against all listeners
+  # and invokes the first listener that mathes
+  def hear!(message, params={})
+    hear(message, params).each do |match|
+      match.listener.call(match)
+      return
+    end
+  end
+
 end
 
 require "attentive/listener_collection"

data/lib/attentive/abbreviations.rb

@@ -1,3 +1,3 @@
 module Attentive
-  ABBREVIATIONS = {"bye"=>"goodbye", "gonna"=>"going to", "hi"=>"hello", "ol'"=>"old", "'sup"=>"what is up", "thanks"=>"thank you", "wanna"=>"want to", "mon"=>"monday", "tue"=>"tuesday", "tues"=>"tuesday", "wed"=>"wednesday", "thu"=>"thursday", "thur"=>"thursday", "thurs"=>"thursday", "fri"=>"friday", "sat"=>"saturday", "sun"=>"sunday"}.freeze
+  ABBREVIATIONS = {"bye"=>"goodbye", "gonna"=>"going to", "hi"=>"hello", "ol'"=>"old", "'sup"=>"what is up", "thanks"=>"thank you", "wanna"=>"want to", "mon"=>"monday", "tue"=>"tuesday", "tues"=>"tuesday", "wed"=>"wednesday", "thu"=>"thursday", "thur"=>"thursday", "thurs"=>"thursday", "fri"=>"friday", "sat"=>"saturday", "sun"=>"sunday", "jan"=>"january", "feb"=>"february", "mar"=>"march", "apr"=>"april", "jun"=>"june", "jul"=>"july", "aug"=>"august", "sep"=>"september", "sept"=>"september", "oct"=>"october", "nov"=>"november", "dec"=>"december"}.freeze
 end

data/lib/attentive/composite_entity.rb

@@ -9,10 +9,9 @@ module Attentive
       attr_accessor :entities
 
       def define(entity_name, *entities)
-        entity_klass = Class.new(Attentive::CompositeEntity)
-        entity_klass.token_name = entity_name
-        entity_klass.entities = entities.map { |entity| Entity[entity] }
-        Entity.register! entity_name, entity_klass
+        create! entity_name do |entity_klass|
+          entity_klass.entities = entities.map { |entity| Entity[entity] }
+        end
       end
     end
 
@@ -23,7 +22,7 @@ module Attentive
 
     def matches?(cursor)
       entities.each do |entity|
-        match = entity.matches?(cursor.dup)
+        match = entity.matches?(cursor)
         return match if match
       end
       false

data/lib/attentive/config.rb

@@ -2,6 +2,8 @@ module Attentive
   module Config
 
     attr_reader :invocations
+    attr_accessor :default_required_contexts
+    attr_accessor :default_prohibited_contexts
 
     def invocations=(*values)
       @invocations = values.flatten

data/lib/attentive/cursor.rb

@@ -8,21 +8,53 @@ module Attentive
     end
 
     def peek
-      tokens[pos]
+      tokens[pos] || EOF
     end
 
     def pop
-      @pos += 1
-      tokens[pos - 1]
+      peek.tap do
+        advance
+      end
+    end
+
+    def new_from_here
+      self.class.new(tokens[pos..-1])
     end
 
     def to_s
       tokens[pos..-1].join
     end
 
+    def inspect
+      "|#{(tokens[0...pos] || []).join}\e[7m#{tokens[pos]}\e[0m#{(tokens[(pos + 1)..-1] || []).join}|"
+    end
+
     def offset
       peek.pos
     end
 
+    def advance(n=1)
+      @pos += n
+    end
+
+    def eof?
+      @pos == @tokens.length
+    end
+
+
+
+  private
+
+    class Eof
+      def whitespace?
+        false
+      end
+
+      def eof?
+        true
+      end
+    end
+    EOF = Eof.new.freeze
+
   end
 end

data/lib/attentive/entities/core.rb

@@ -0,0 +1,3 @@
+require "attentive/entities/core/number"
+require "attentive/entities/core/date"
+require "attentive/entities/core/email"

data/lib/attentive/entities/core/date.rb

@@ -0,0 +1,6 @@
+require "attentive/entities/core/date/month"
+require "attentive/entities/core/date/wday"
+require "attentive/entities/core/date/relative"
+
+Attentive::CompositeEntity.define "core.date",
+  "core.date.relative"

data/lib/attentive/entities/core/date/month.rb

@@ -0,0 +1,7 @@
+require "attentive/entity"
+require "date"
+
+month_names = Date::MONTHNAMES.compact.map(&:downcase)
+Attentive::Entity.define "core.date.month", *month_names do |match|
+  month_names.index(match.phrase) + 1
+end

data/lib/attentive/entities/core/date/relative.rb

@@ -0,0 +1,6 @@
+require "attentive/entities/core/date/relative/past"
+require "attentive/entities/core/date/relative/future"
+
+Attentive::CompositeEntity.define "core.date.relative",
+  "core.date.relative.future",
+  "core.date.relative.past"

data/lib/attentive/entities/core/date/relative/future.rb

@@ -0,0 +1,27 @@
+require "attentive/entity"
+require "date"
+
+Attentive::Entity.define "core.date.relative.future",
+    "today",
+    "tomorrow",
+    "{{core.date.wday}}",
+    "next {{core.date.wday}}" do |match|
+
+  today = Date.today
+
+  if match.matched?("core.date.wday")
+    wday = match["core.date.wday"]
+    days_until_wday = wday - today.wday
+    days_until_wday += 7 if days_until_wday < 0
+    date = today + days_until_wday
+
+    date += 7 if match.to_s.start_with?("next")
+    date
+  else
+    case match.to_s
+    when "today" then today
+    when "tomorrow" then today + 1
+    else raise NotImplementedError, "Unrecognized match: #{match.to_s}"
+    end
+  end
+end

data/lib/attentive/entities/core/date/relative/past.rb

@@ -0,0 +1,24 @@
+require "attentive/entity"
+require "date"
+
+Attentive::Entity.define "core.date.relative.past",
+    "today",
+    "yesterday",
+    "{{core.date.wday}}",
+    "last {{core.date.wday}}" do |match|
+
+  today = Date.today
+
+  if match.matched?("core.date.wday")
+    wday = match["core.date.wday"]
+    days_since_wday = today.wday - wday
+    days_since_wday += 7 if days_since_wday < 0
+    today - days_since_wday
+  else
+    case match.to_s
+    when "today" then today
+    when "yesterday" then today - 1
+    else raise NotImplementedError, "Unrecognized match: #{match.to_s}"
+    end
+  end
+end

data/lib/attentive/entities/core/date/wday.rb

@@ -0,0 +1,7 @@
+require "attentive/entity"
+require "date"
+
+day_names = Date::DAYNAMES.map(&:downcase)
+Attentive::Entity.define "core.date.wday", *day_names do |match|
+  day_names.index(match.phrase)
+end

data/lib/attentive/entities/core/email.rb

@@ -0,0 +1,8 @@
+require "attentive/entity"
+
+# Email regex asserts that there are no @ symbols or whitespaces in either the
+# localpart or the domain, and that there is a single @ symbol separating the
+# localpart and the domain.
+Attentive::Entity.define "core.email", %q{(?<email>[^@\s]+@[^@\s]+)} do |match|
+  match["email"]
+end

data/lib/attentive/entities/core/number.rb

@@ -0,0 +1,8 @@
+require "attentive/entities/core/number/integer"
+require "attentive/entities/core/number/float"
+require "attentive/entities/core/number/positive"
+require "attentive/entities/core/number/negative"
+
+Attentive::CompositeEntity.define "core.number",
+  "core.number.float",
+  "core.number.integer"

data/lib/attentive/entities/core/number/float.rb

@@ -0,0 +1,6 @@
+require "attentive/entities/core/number/float/positive"
+require "attentive/entities/core/number/float/negative"
+
+Attentive::CompositeEntity.define "core.number.float",
+  "core.number.float.positive",
+  "core.number.float.negative"

data/lib/attentive/entities/core/number/float/negative.rb

@@ -0,0 +1,6 @@
+require "attentive/entity"
+require "bigdecimal"
+
+Attentive::Entity.define "core.number.float.negative", %q{(?<float>\-[\d,]+\.\d+)} do |match|
+  BigDecimal.new(match["float"].gsub(",", ""))
+end

data/lib/attentive/entities/core/number/float/positive.rb

@@ -0,0 +1,6 @@
+require "attentive/entity"
+require "bigdecimal"
+
+Attentive::Entity.define "core.number.float.positive", %q{(?<float>[\d,]+\.\d+)} do |match|
+  BigDecimal.new(match["float"].gsub(",", ""))
+end

data/lib/attentive/entities/core/number/integer.rb

@@ -0,0 +1,6 @@
+require "attentive/entities/core/number/integer/positive"
+require "attentive/entities/core/number/integer/negative"
+
+Attentive::CompositeEntity.define "core.number.integer",
+  "core.number.integer.positive",
+  "core.number.integer.negative"

data/lib/attentive/entities/core/number/integer/negative.rb

@@ -0,0 +1,5 @@
+require "attentive/entity"
+
+Attentive::Entity.define "core.number.integer.negative", %q{(?<integer>\-\d+)} do |match|
+  match["integer"].gsub(",", "").to_i
+end

data/lib/attentive/entities/core/number/integer/positive.rb

@@ -0,0 +1,5 @@
+require "attentive/entity"
+
+Attentive::Entity.define "core.number.integer.positive", %q{(?<integer>[\d,]+)} do |match|
+  match["integer"].gsub(",", "").to_i
+end

data/lib/attentive/entities/core/number/negative.rb

@@ -0,0 +1,6 @@
+require "attentive/entities/core/number/integer/negative"
+require "attentive/entities/core/number/float/negative"
+
+Attentive::CompositeEntity.define "core.number.negative",
+  "core.number.float.negative",
+  "core.number.integer.negative"

data/lib/attentive/entities/core/number/positive.rb

@@ -0,0 +1,6 @@
+require "attentive/entities/core/number/integer/positive"
+require "attentive/entities/core/number/float/positive"
+
+Attentive::CompositeEntity.define "core.number.positive",
+  "core.number.float.positive",
+  "core.number.integer.positive"

data/lib/attentive/entity.rb

@@ -12,25 +12,44 @@ module Attentive
       attr_accessor :token_name
 
       def [](entity_name)
+        entity_name = entity_name.to_sym
         @entities.fetch(entity_name)
       rescue KeyError
         raise Attentive::UndefinedEntityError.new("Undefined Entity #{entity_name.inspect}")
       end
 
       def define(entity_name, *phrases, &block)
-        entity_klass = Class.new(Attentive::Entity)
-        entity_klass.token_name = entity_name
-        entity_klass.phrases = phrases.map do |phrase|
-          Attentive::Tokenizer.tokenize(phrase, entities: true, regexps: true, ambiguous: false)
+        create! entity_name do |entity_klass|
+          entity_klass.phrases = phrases.map do |phrase|
+            Attentive::Tokenizer.tokenize(phrase, entities: true, regexps: true, ambiguous: false)
+          end
+          entity_klass.send :define_method, :_value_from_match, &block if block_given?
         end
-        entity_klass.send :define_method, :_value_from_match, &block
-        register! entity_name, entity_klass
+      end
+
+      def undefine(entity_name)
+        entity_symbol = entity_name.to_sym
+        unregister! entity_symbol
+      end
+
+    protected
+
+      def create!(entity_name)
+        entity_symbol = entity_name.to_sym
+        entity_klass = Class.new(self)
+        entity_klass.token_name = entity_symbol
+        yield entity_klass
+        Entity.register! entity_symbol, entity_klass
       end
 
       def register!(entity_name, entity_klass)
-        # TODO: raise already registered error
+        raise ArgumentError, "Entity #{entity_name.inspect} has already been defined" if @entities.key?(entity_name)
         @entities[entity_name] = entity_klass
       end
+
+      def unregister!(entity_name)
+        @entities.delete entity_name
+      end
     end
 
 
@@ -46,7 +65,11 @@ module Attentive
     end
 
     def to_s
-      "{{#{variable_name}:#{self.class.token_name}}}"
+      if variable_name.to_s == self.class.token_name.to_s
+        "{{#{self.class.token_name}}}"
+      else
+        "{{#{variable_name}:#{self.class.token_name}}}"
+      end
     end
 
     def entity?
@@ -55,15 +78,19 @@ module Attentive
 
     def matches?(cursor)
       self.class.phrases.each do |phrase|
-        cursor_copy = cursor.dup
+        cursor_copy = cursor.new_from_here
         match = Attentive::Matcher.new(phrase, cursor_copy).match!
         if match
-          cursor.instance_variable_set :@pos, cursor_copy.pos
+          cursor.advance cursor_copy.pos
           return { variable_name => _value_from_match(match) }
         end
       end
       false
     end
 
+    def _value_from_match(match)
+      match.to_s
+    end
+
   end
 end

data/lib/attentive/listener.rb

@@ -8,10 +8,10 @@ module Attentive
 
     def initialize(listeners, phrases, options, callback)
       context_options = options.fetch(:context, {})
-      @required_contexts = context_options.fetch(:in, %i{conversation})
+      @required_contexts = context_options.fetch(:in, Attentive.default_required_contexts)
       @required_contexts = [] if @required_contexts == :any
       @required_contexts = Set[*@required_contexts]
-      @prohibited_contexts = context_options.fetch(:not_in, %i{quotation})
+      @prohibited_contexts = context_options.fetch(:not_in, Attentive.default_prohibited_contexts)
       @prohibited_contexts = Set[*@prohibited_contexts]
 
       @listeners = listeners

data/lib/attentive/matcher.rb

@@ -2,16 +2,21 @@ require "attentive/match"
 
 module Attentive
   class Matcher
-    attr_reader :phrase, :cursor, :pos
+    attr_reader :phrase, :message, :cursor
 
-    def initialize(phrase, cursor, params={})
+    def initialize(phrase, message, params={})
       @phrase = phrase
-      @cursor = cursor
-      @pos = params.fetch(:pos, 0)
+      @cursor = Cursor.new(phrase, params.fetch(:pos, 0))
+      @message = message
       @match_params = params.each_with_object({}) { |(key, value), new_hash| new_hash[key] = value if %i{listener message}.member?(key) }
-      @pos += 1 while phrase[pos] && phrase[pos].whitespace?
       @match_data = {}
       @state = :matching
+
+      cursor.pop while cursor.peek.whitespace?
+    end
+
+    def pos
+      cursor.pos
     end
 
     def matching?
@@ -23,57 +28,41 @@ module Attentive
     end
 
     def match!
-      while token = cursor.peek
+      until (token = message.peek).eof?
         if token.ambiguous?
-          unless match_subphrase!(token.possibilities)
+          unless match_any!(token.possibilities)
             @state = :mismatch
             break
           end
-          @pos += 1 while phrase[pos] && phrase[pos].whitespace?
+          cursor.pop while cursor.peek.whitespace?
 
-        elsif match_data = phrase[pos].matches?(cursor)
-          if match_data.is_a?(MatchData)
-            new_character_index = cursor.offset + match_data.to_s.length
-            @match_data.merge! Hash[match_data.names.zip(match_data.captures)]
-
-            # Advance the cursor to the first token after the regexp match
-            cursor_pos = cursor.tokens.index { |token| token.pos >= new_character_index }
-            cursor_pos = cursor.tokens.length unless cursor_pos
-            cursor.instance_variable_set :@pos, cursor_pos
-            @pos += 1
-          else
-            @match_data.merge!(match_data) unless match_data == true
-            @pos += 1
-          end
-          @pos += 1 while phrase[pos] && phrase[pos].whitespace?
+        elsif match_data = cursor.peek.matches?(message)
+          @match_data.merge!(match_data) unless match_data == true
+          cursor.pop
+          cursor.pop while cursor.peek.whitespace?
           @state = :found
 
-
           # -> This is the one spot where we instantiate a Match
-          return Attentive::Match.new(phrase, @match_params.merge(match_data: @match_data)) if pos == phrase.length
+          return Attentive::Match.new(phrase, @match_params.merge(match_data: @match_data)) if cursor.eof?
 
         elsif !token.skippable?
           @state = :mismatch
           break
         end
 
-        cursor.pop
-        break unless cursor.peek
-        while cursor.peek.whitespace?
-          cursor.pop
-          break unless cursor.peek
-        end
+        message.pop
+        message.pop while message.peek.whitespace?
       end
 
       nil
     end
 
-    def match_subphrase!(subphrases)
-      subphrases.each do |subphrase|
-        matcher = Matcher.new(phrase, Cursor.new(subphrase), pos: pos)
+    def match_any!(messages)
+      messages.each do |message|
+        matcher = Matcher.new(phrase[pos..-1], Cursor.new(message))
         matcher.match!
         unless matcher.mismatch?
-          @pos = matcher.pos
+          cursor.advance matcher.pos
           return true
         end
       end

data/lib/attentive/token.rb

@@ -26,10 +26,18 @@ module Attentive
       false
     end
 
+    def eof?
+      false
+    end
+
     def matches?(cursor)
       self == cursor.peek
     end
 
+    def inspect
+      "<#{self.class.name ? self.class.name.split("::").last : "Entity"} #{to_s.inspect}>"
+    end
+
   end
 
 
@@ -50,6 +58,10 @@ module Attentive
       string
     end
 
+    def length
+      string.length
+    end
+
     def ==(other)
       self.class == other.class && self.string == other.string
     end

data/lib/attentive/tokenizer.rb

@@ -8,141 +8,187 @@ require "attentive/errors"
 
 module Attentive
   class Tokenizer
-    extend Attentive::Tokens
-
-    # Splits apart words and punctuation,
-    # treats apostrophes and dashes as a word-characters,
-    # trims each fragment of whitepsace
-    # SPLITTER = /\s*([\w'-]+)\s*/.freeze
-    SPLITTER = /(\n|{{|}}|\s+|\.{2,}|[^\s\w'@-])/.freeze
-    PUNCTUATION = /^\W+$/.freeze
-    WHITESPACE = /^\s+$/.freeze
-    ENTITY_START = "{{".freeze
-    ENTITY_END = "}}".freeze
-    REGEXP_START = "(".freeze
-    REGEXP_END = ")".freeze
-    REGEXP_ESCAPE = "\\".freeze
+    include Attentive::Tokens
 
+    attr_reader :message, :chars, :options
 
-    def self.split(message)
-      Attentive::Text.normalize(message).split(SPLITTER).reject(&:empty?)
+    def self.tokenize(message, options={})
+      self.new(message, options).tokenize
     end
 
 
-    def self.tokenize(message, options={})
-      match_entities = options.fetch(:entities, false)
-      match_regexps = options.fetch(:regexps, false)
-      fail_if_ambiguous = !options.fetch(:ambiguous, true)
-      strings = split(message)
-      tokens = []
+
+    def initialize(message, options={})
+      @message = Attentive::Text.normalize(message)
+      @chars = self.message.each_char.to_a
+      @options = options
+    end
+
+
+
+    def tokenize
       i = 0
-      pos = 0
-      while i < strings.length
-        string = strings[i]
-        case string
-        when ""
-          # do nothing
-
-        when WHITESPACE
-          tokens << whitespace(string, pos: pos)
-
-        when ":"
-          if strings[i + 2] == ":"
-            tokens << emoji(strings[i + 1], pos: pos)
-            pos += strings[i + 1].length + 1
-            i += 2
-          else
-            tokens << punctuation(":", pos: pos)
-          end
+      tokens = []
+      while i < chars.length
+        char = chars[i]
 
-        when ENTITY_START
-          if match_entities
-            j = i + 1
-            found_entity = false
-            while j < strings.length
-              if strings[j] == ENTITY_END
-                entity = strings[(i + 1)...j] # e.g. ["variable-name", ":" "entity-type"]
-                tokens << entity(*entity.join.split(":").reverse, pos: pos)
-                i = j + 1
-                pos += entity.join.length + 4
-                found_entity = true
-                break
-              end
-              j += 1
-            end
-            next if found_entity
-          end
-          tokens << punctuation(ENTITY_START, pos: pos)
-
-        when REGEXP_START
-          if match_regexps && strings[i + 1] == "?"
-            j = i + 2
-            found_regexp = false
-            parens = 1
-            inside_square_bracket = false
-            while j < strings.length
-              if strings[j] == "[" && strings[j - 1] != REGEXP_ESCAPE
-                inside_square_bracket = true
-              elsif strings[j] == "]" && strings[j - 1] != REGEXP_ESCAPE
-                inside_square_bracket = false
-              end
-
-              unless inside_square_bracket
-                if strings[j] == REGEXP_START && strings[j - 1] != REGEXP_ESCAPE
-                  parens += 1
-                elsif strings[j] == REGEXP_END && strings[j - 1] != REGEXP_ESCAPE
-                  parens -= 1
-                end
-
-                if parens == 0
-                  tokens << regexp(strings[i..j].join, pos: pos)
-                  pos += strings[i..j].join.length + 2
-                  i = j + 1
-                  found_regexp = true
-                  break
-                end
-              end
-              j += 1
-            end
-            next if found_regexp
-          end
-          tokens << punctuation(REGEXP_START, pos: pos)
+        if EMOJI_START === char && string = match_emoji_at(i)
+          tokens << emoji(string, pos: i)
+          i += string.length + 2
+
+        elsif ENTITY_START === char && string = match_entity_at(i)
+          tokens << entity(*string.split(":").reverse, pos: i)
+          i += string.length + 4
+
+        elsif REGEXP_START === char && string = match_regexp_at(i)
+          tokens << regexp(string, pos: i)
+          i += string.length
 
-        when PUNCTUATION
-          tokens << punctuation(string, pos: pos)
+        elsif WHITESPACE === char && string = match_whitespace_at(i)
+          tokens << whitespace(string, pos: i)
+          i += string.length
 
-        when *Attentive.invocations
-          tokens << invocation(string, pos: pos)
+        elsif NUMBER_START === char && string = match_number_at(i)
+          tokens << word(string, pos: i)
+          i += string.length
+
+        elsif PUNCTUATION === char # =~ /\W/
+          tokens << punctuation(char, pos: i)
+          i += 1
 
         else
-          if replace_with = Attentive::ABBREVIATIONS[string]
-            tokens.concat tokenize(replace_with, options)
+          string = match_word_at(i)
+          if Attentive.invocations.member?(string)
+            tokens << invocation(string, pos: i)
+
+          elsif replace_with = Attentive::ABBREVIATIONS[string]
+            tokens.concat self.class.tokenize(replace_with, options)
 
           elsif expands_to = Attentive::CONTRACTIONS[string]
             possibilities = expands_to.map do |possibility|
-              tokenize(possibility, options)
+              self.class.tokenize(possibility, options)
             end
 
             if possibilities.length == 1
               tokens.concat possibilities[0]
             else
-              tokens << any_of(possibilities, pos: pos)
+              tokens << any_of(string, possibilities, pos: i)
             end
+
           else
-            tokens << word(string, pos: pos)
+            tokens << word(string, pos: i)
           end
+          i += string.length
         end
+      end
+
+      fail_if_ambiguous!(message, tokens) if fail_if_ambiguous?
+      Attentive::Phrase.new(tokens)
+    end
+
+
 
-        i += 1
-        pos += string.length
+    def match_emoji_at(i)
+      emoji = ""
+      while (i += 1) < chars.length
+        return if_present?(emoji) if EMOJI_END === chars[i]
+        return false if WHITESPACE === chars[i]
+        emoji << chars[i]
       end
+      false
+    end
 
-      fail_if_ambiguous!(message, tokens) if fail_if_ambiguous
+    def match_entity_at(i)
+      return false unless match_entities?
+      return false unless chars[i += 1] == "{"
+      entity = ""
+      while (i += 1) < chars.length
+        return if_present?(entity) if ["}", "}"] == chars[i, 2]
+        return false unless ENTITY === chars[i]
+        entity << chars[i]
+      end
+      false
+    end
 
-      Attentive::Phrase.new(tokens)
+    def match_regexp_at(i)
+      return false unless match_regexps?
+      return false unless chars[i += 1] == "?"
+      regexp = "(?"
+      parens = 1
+      inside_square_bracket = false
+      while (i += 1) < chars.length
+        regexp << chars[i]
+        next if chars[i - 1] == "\\"
+        inside_square_bracket = true if chars[i] == "["
+        inside_square_bracket = false if chars[i] == "]"
+        next if inside_square_bracket
+        parens += 1 if chars[i] == "("
+        parens -= 1 if chars[i] == ")"
+        return if_present?(regexp) if parens == 0
+      end
+      false
+    end
+
+    def match_whitespace_at(i)
+      whitespace = chars[i]
+      while (i += 1) < chars.length
+        break unless WHITESPACE === chars[i]
+        whitespace << chars[i]
+      end
+      whitespace
+    end
+
+    def match_number_at(i)
+      return false if CONDITIONAL_NUMBER_START === chars[i] && !(NUMBER === chars[i + 1])
+      number = chars[i]
+      while (i += 1) < chars.length
+        break unless NUMBER === chars[i] || (CONDITIONAL_NUMBER === chars[i] && NUMBER === chars[i + 1])
+        number << chars[i]
+      end
+      number
+    end
+
+    def match_word_at(i)
+      word = chars[i]
+      while (i += 1) < chars.length
+        break unless WORD === chars[i]
+        word << chars[i]
+      end
+      word
+    end
+
+    def if_present?(string)
+      string.empty? ? false : string
+    end
+
+
+
+    def match_entities?
+      options.fetch(:entities, false)
     end
 
-    def self.fail_if_ambiguous!(phrase, tokens)
+    def match_regexps?
+      options.fetch(:regexps, false)
+    end
+
+    def fail_if_ambiguous?
+      !options.fetch(:ambiguous, true)
+    end
+
+    WHITESPACE = /\s/.freeze
+    PUNCTUATION = /[^\s\w'@-]/.freeze
+    EMOJI_START = ":".freeze
+    EMOJI_END = ":".freeze
+    ENTITY_START = "{".freeze
+    ENTITY = /[a-z0-9\.\-:]/.freeze
+    REGEXP_START = "(".freeze
+    NUMBER_START = /[\d\.\-]/.freeze
+    CONDITIONAL_NUMBER_START = /[\.\-]/.freeze
+    NUMBER = /\d/.freeze
+    CONDITIONAL_NUMBER = /[\.,]/.freeze
+    WORD = /[\w'\-@]/.freeze
+
+    def fail_if_ambiguous!(phrase, tokens)
       ambiguous_token = tokens.find(&:ambiguous?)
       return unless ambiguous_token
 
@@ -159,5 +205,4 @@ end
 require "attentive/entity"
 require "attentive/composite_entity"
 
-require "attentive/entities/integer"
-require "attentive/entities/relative_date"
+require "attentive/entities/core"

data/lib/attentive/tokens.rb

@@ -1,8 +1,8 @@
 module Attentive
   module Tokens
 
-    def any_of(possibilities, pos: nil)
-      Attentive::Tokens::AnyOf.new possibilities, pos
+    def any_of(string, possibilities, pos: nil)
+      Attentive::Tokens::AnyOf.new string, possibilities, pos
     end
 
     def emoji(string, pos: nil)

data/lib/attentive/tokens/any_of.rb

@@ -2,12 +2,12 @@ require "attentive/token"
 
 module Attentive
   module Tokens
-    class AnyOf < Token
+    class AnyOf < StringToken
       attr_reader :possibilities
 
-      def initialize(possibilities, pos)
+      def initialize(string, possibilities, pos)
+        super string, pos
         @possibilities = possibilities
-        super pos
       end
 
       def ==(other)

data/lib/attentive/tokens/regexp.rb

@@ -15,11 +15,28 @@ module Attentive
       end
 
       def matches?(cursor)
-        regexp.match(cursor.to_s)
+        # Compare the original, untokenized, message to the regular expression
+        match_data = regexp.match(cursor.to_s)
+        return false unless match_data
+
+        # Find the first token following the match
+        new_character_index = cursor.offset + match_data.to_s.length
+        cursor_pos = cursor.tokens.index { |token| token.pos >= new_character_index }
+        cursor_pos = cursor.tokens.length unless cursor_pos
+
+        # If the match ends in the middle of a token, treat it as a mismatch
+        match_end_token = cursor.tokens[cursor_pos - 1]
+        return false if match_end_token.pos + match_end_token.length > new_character_index
+
+        # Advance the cursor to the first token after the regexp match
+        cursor.advance cursor_pos - cursor.pos
+
+        # Return the MatchData as a hash
+        Hash[match_data.names.zip(match_data.captures)]
       end
 
       def to_s
-        regexp.inspect[1...-1]
+        regexp.inspect
       end
 
     end

data/lib/attentive/version.rb

@@ -1,3 +1,3 @@
 module Attentive
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: attentive
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Bob Lail
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-04-27 00:00:00.000000000 Z
+date: 2016-05-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thread_safe
@@ -158,8 +158,23 @@ files:
 - lib/attentive/config.rb
 - lib/attentive/contractions.rb
 - lib/attentive/cursor.rb
-- lib/attentive/entities/integer.rb
-- lib/attentive/entities/relative_date.rb
+- lib/attentive/entities/core.rb
+- lib/attentive/entities/core/date.rb
+- lib/attentive/entities/core/date/month.rb
+- lib/attentive/entities/core/date/relative.rb
+- lib/attentive/entities/core/date/relative/future.rb
+- lib/attentive/entities/core/date/relative/past.rb
+- lib/attentive/entities/core/date/wday.rb
+- lib/attentive/entities/core/email.rb
+- lib/attentive/entities/core/number.rb
+- lib/attentive/entities/core/number/float.rb
+- lib/attentive/entities/core/number/float/negative.rb
+- lib/attentive/entities/core/number/float/positive.rb
+- lib/attentive/entities/core/number/integer.rb
+- lib/attentive/entities/core/number/integer/negative.rb
+- lib/attentive/entities/core/number/integer/positive.rb
+- lib/attentive/entities/core/number/negative.rb
+- lib/attentive/entities/core/number/positive.rb
 - lib/attentive/entity.rb
 - lib/attentive/errors.rb
 - lib/attentive/listener.rb

data/lib/attentive/entities/integer.rb

@@ -1,5 +0,0 @@
-require "attentive/entity"
-
-Attentive::Entity.define :integer, %q{(?<integer>\d+)} do |match|
-  match["integer"].to_i
-end

data/lib/attentive/entities/relative_date.rb

@@ -1,44 +0,0 @@
-require "attentive/entity"
-require "date"
-
-weekday_regexp = "(?<weekday>sunday|monday|tuesday|wednesday|thursday|friday|saturday)"
-Attentive::Entity.define :"relative-date",
-    "today",
-    "tomorrow",
-    "yesterday",
-    weekday_regexp,
-    "next #{weekday_regexp}",
-    "last #{weekday_regexp}" do |match|
-
-  today = Date.today
-
-  next_wday = lambda do |wday|
-    days_until_wday = wday - today.wday
-    days_until_wday += 7 if days_until_wday < 0
-    today + days_until_wday
-  end
-
-  if match.matched?("weekday")
-    date = case weekday = match["weekday"]
-    when /^sun/ then next_wday[0]
-    when /^mon/ then next_wday[1]
-    when /^tue/ then next_wday[2]
-    when /^wed/ then next_wday[3]
-    when /^thu/ then next_wday[4]
-    when /^fri/ then next_wday[5]
-    when /^sat/ then next_wday[6]
-    else raise NotImplementedError, "Unrecognized weekday: #{weekday.inspect}"
-    end
-
-    date += 7 if match.to_s.start_with?("next")
-    date -= 7 if match.to_s.start_with?("last")
-    date
-  else
-    case match.to_s
-    when "today" then today
-    when "tomorrow" then today + 1
-    when "yesterday" then today - 1
-    else raise NotImplementedError, "Unrecognized match: #{match.to_s}"
-    end
-  end
-end