string-eater 0.1.0

data/LICENSE

@@ -0,0 +1,24 @@
+Copyright (c) 2012 Dan Swain
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+

data/README.md

@@ -0,0 +1,133 @@
+# String Eater
+
+A fast ruby string tokenizer.  It eats strings and dumps tokens.
+
+## License
+
+String Eater is released under the 
+[MIT license](http://en.wikipedia.org/wiki/MIT_License). 
+See the LICENSE file.
+
+## Requirements
+
+String Eater probably only works in Ruby 1.9.2+ with MRI.  It's been
+tested with Ruby 1.9.3p194. 
+
+String Eater uses a C extension, so it will only work on Ruby
+implemenatations that provide support for C extensions.
+
+## Installation
+
+We'll publish this gem soon, but for now you can clone and install as
+
+    git clone git://github.com/dantswain/string-eater.git 
+    cd string-eater 
+    rake install
+
+If you are working on a system where you need to `sudo gem install`
+you can do
+
+    rake gem 
+    sudo gem install string-eater
+
+As always, you can `rake -T` to find out what other rake tasks we have
+provided.
+
+## Basic Usage
+
+Suppose we want to tokenize a string that contains address information
+for a person and is consistently formatted like
+
+    Last Name, First Name | Street address, City, State, Zip
+
+Suppose we only want to extract the last name, city, and state.
+
+To do this using String Eater, create a subclass of
+`StringEater::Tokenizer` like this
+
+    require 'string-eater' 
+    
+    class PersonTokenizer < StringEater::Tokenizer 
+      add_field :last_name 
+      look_for ", "
+      add_field :first_name, :extract => false
+      look_for " | "
+      add_field :street_address, :extract => false 
+      look_for ", " 
+      add_field :city
+      look_for ", " 
+      add_field :state 
+      look_for ", " 
+    end
+
+Note the use of `:extract => false` to specify fields that are important
+to the structure of the line but that we don't necessarily need to
+extract.
+
+Then, we can tokenize the string like this:
+
+    tokenizer = PersonTokenizer.new
+    string = "Flinstone, Fred | 301 Cobblestone Way, Bedrock, NA, 00000" 
+    tokenizer.tokenize! string
+
+    puts tokenizer.last_name # => "Flinestone" 
+    puts tokenizer.city      # => "Bedrock" 
+    puts tokenizer.state     # => "NA"
+
+We can also do something like this:
+
+    tokenizer.tokenize!(string) do |tokens| 
+      puts "The #{tokens[:last_name]}s live in #{tokens[:city]}"
+    end
+
+For another example, see `examples/nginx.rb`, which defines an
+[nginx](http://nginx.org) log line tokenizer.
+
+## Implementation
+
+There are actually three tokenizer algorithms provided here.  The
+three algorithms should be interchangeable.
+
+1. `StringEater::CTokenizer` - A C extension implementation.  The
+   fastest of the three.  This is the default implementation for
+   `StringEater::Tokenizer`.
+
+2. `StringEater::RubyTokenizer` - A pure-Ruby implementation.  This is
+   a slightly different implementation of the algorithm - an
+   implementation that is faster on Ruby than a translation of the C
+   algorithm.  Probably not as fast (or not much faster) than using
+   Ruby regular expressions. 
+
+3. `StringEater::RubyTokenizerEachChar` - A pure-Ruby implementation.
+   This is essentially the same as the C implementation, but written
+   in pure Ruby.  It uses `String#each_char` and is therefore VERY
+   SLOW!  It provides a good way to hack the algorithm, though.
+
+The main algorithm works by finding the start and end points of tokens
+in a string.  The search is done incrementally (i.e., loop through the
+string and look for each sequence of characters).  The algorithm is
+"lazy" in the sense that only the required tokens are copied for
+output ("extracted").
+
+## Performance
+
+Soon I'll add some code here to run your own benchmarks.
+
+I've run my own benchmarks comparing String Eater to some code that does the
+same task (both tokenizing nginx log lines) using Ruby regular expressions.  So
+far, String Eater is about 200% faster; able to process over 100,000 lines per
+second on my laptop vs less than 50,000 lines per second for the regular
+expression version.  I'm working to further optimize the String Eater code.
+
+## Contributing
+
+The usual github process applies here:
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+You can also contribute to the author's ego by letting him know that
+you find String Eater useful ;)

data/Rakefile

@@ -0,0 +1,33 @@
+require 'rake/clean'
+
+desc "Run rspec spec/ (compile if needed)"
+task :test => :compile do
+  sh "rspec spec/"
+end
+
+so_ext = RbConfig::CONFIG['DLEXT']
+ext_dir = "ext/string-eater"
+ext_file = ext_dir + "/c_tokenizer_ext.#{so_ext}"
+
+file ext_file => Dir.glob("ext/string-eater/*{.rb,.c}") do
+  Dir.chdir("ext/string-eater") do
+    ruby "extconf.rb"
+    sh "make"
+  end
+end
+
+desc "Create gem"
+task :gem => "string-eater.gemspec" do
+  sh "gem build string-eater.gemspec"
+end
+
+desc "Install using 'gem install'"
+task :install => :gem do
+  sh "gem install string-eater"
+end
+
+desc "Compile the extension"
+task :compile => ext_file
+
+CLEAN.include('ext/**/*{.o,.log,.so,.bundle}')
+CLEAN.include('ext/**/Makefile')

data/examples/address.rb

@@ -0,0 +1,35 @@
+# once the gem is installed, you don't need this
+$: << File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
+$: << File.expand_path(File.join(File.dirname(__FILE__), '..', 'ext/string-eater'))
+
+# this is the example from the README
+require 'string-eater'
+
+class PersonTokenizer < StringEater::Tokenizer 
+  add_field :last_name 
+  look_for ", "
+  add_field :first_name, :extract => false
+  look_for " | "
+  add_field :street_address, :extract => false 
+  look_for ", " 
+  add_field :city
+  look_for ", " 
+  add_field :state 
+  look_for ", " 
+end
+
+if __FILE__ == $0
+  tokenizer = PersonTokenizer.new
+  puts tokenizer.describe_line
+
+  string = "Flinstone, Fred | 301 Cobblestone Way, Bedrock, NA, 00000" 
+  tokenizer.tokenize! string
+
+  puts tokenizer.last_name # => "Flinestone" 
+  puts tokenizer.city      # => "Bedrock" 
+  puts tokenizer.state     # => "NA"
+
+  tokenizer.tokenize!(string) do |tokens| 
+    puts "The #{tokens[:last_name]}s live in #{tokens[:city]}"
+  end
+end

data/examples/nginx.rb

@@ -0,0 +1,70 @@
+# once the gem is installed, you don't need this
+$: << File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
+$: << File.expand_path(File.join(File.dirname(__FILE__), '..', 'ext/string-eater'))
+
+require 'string-eater'
+
+class NginxLogTokenizer < StringEater::CTokenizer
+  add_field :ip
+  look_for " - "
+  add_field :remote_user, :extract => false
+  look_for " ["
+  add_field :timestamp, :extract => false
+  look_for "] \""
+  add_field :request
+  look_for "\" "
+  add_field :status_code
+  look_for " "
+  add_field :bytes_sent, :extract => false
+  look_for " \""
+  add_field :referrer_url
+  look_for "\" \""
+  add_field :user_agent
+  look_for "\" \""
+  add_field :compression, :extract => false
+  look_for "\" "
+  add_field :remainder
+
+  def status_code
+    @extracted_tokens[:status_code].to_i
+  end
+
+  def request_verb
+    @extracted_tokens[:request_verb]
+  end
+
+  def request_url
+    @extracted_tokens[:request_url]
+  end
+
+  def do_extra_parsing
+    return unless @extracted_tokens[:request]
+    request_parts = @extracted_tokens[:request].split
+    if request_parts.size == 3
+      @extracted_tokens[:request_verb] = request_parts[0]
+      @extracted_tokens[:request_url] = request_parts[1]
+    end
+  end
+end
+
+if __FILE__ == $0
+  tokenizer = NginxLogTokenizer.new
+  puts tokenizer.describe_line
+
+  str = '73.80.217.212 - - [01/Aug/2012:09:14:25 -0500] "GET /this_is_a_url HTTP/1.1" 304 152 "http://referrer.com" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)" "-" "there could be" other "stuff here"'
+
+  puts "input string: " + str
+  puts "Tokens: "
+
+  # use a block to work with the extracted tokens
+  tokenizer.tokenize!(str) do |tokens|
+    tokens.each do |token|
+      puts "\t" + token.inspect
+    end
+  end
+
+  # use the token's name as a method to get its value
+  puts tokenizer.ip
+  puts tokenizer.status_code
+  puts tokenizer.request_verb
+end

data/ext/string-eater/c-tokenizer.c

@@ -0,0 +1,141 @@
+#include <ruby.h>
+
+/* not used in production - useful for debugging */
+#define puts_inspect(var) \
+  ID inspect = rb_intern("inspect"); \
+  VALUE x = rb_funcall(var, inspect, 0); \
+  printf("%s\n", StringValueCStr(x));
+
+static VALUE rb_cCTokenizer;
+static VALUE rb_mStringEater;
+
+static VALUE tokenize_string(VALUE self, 
+    VALUE string,
+    VALUE tokens_to_find_indexes,
+    VALUE tokens_to_find_strings,
+    VALUE tokens_to_extract_indexes,
+    VALUE tokens_to_extract_names)
+{
+  const char* input_string = StringValueCStr(string);
+  VALUE extracted_tokens = rb_hash_new();
+  VALUE curr_token;
+  unsigned int curr_token_ix;
+  long n_tokens_to_find = RARRAY_LEN(tokens_to_find_indexes);
+  size_t str_len = strlen(input_string);
+  size_t ix;
+  char c;
+  char looking_for;
+  size_t looking_for_len;
+  size_t looking_for_ix = 0;
+  long find_ix = 0;
+  const char*  looking_for_token;
+  unsigned int n_tokens = (unsigned int)RARRAY_LEN(rb_iv_get(self, "@tokens"));
+
+  size_t startpoint = 0;
+
+  long n_tokens_to_extract = RARRAY_LEN(tokens_to_extract_indexes);
+  long last_token_extracted_ix = 0;
+
+  long next_token_to_extract_ix = NUM2UINT(rb_ary_entry(tokens_to_extract_indexes, last_token_extracted_ix));
+
+  curr_token = rb_ary_entry(tokens_to_find_strings, find_ix);
+  curr_token_ix = NUM2UINT(rb_ary_entry(tokens_to_find_indexes, find_ix));
+  looking_for_token = StringValueCStr(curr_token);
+  looking_for_len = strlen(looking_for_token);
+  looking_for = looking_for_token[looking_for_ix];
+
+  for(ix = 0; ix < str_len; ix++)
+  {
+    c = input_string[ix];
+    if(c == looking_for)
+    {
+      if(looking_for_ix == 0)
+      {
+        /* entering new token */
+        if(curr_token_ix > 0)
+        {
+          /* extract, if necessary */
+          if((curr_token_ix - 1) == next_token_to_extract_ix)
+          {
+            last_token_extracted_ix++;
+            if(last_token_extracted_ix < n_tokens_to_extract)
+            {
+              next_token_to_extract_ix = NUM2UINT(rb_ary_entry(tokens_to_extract_indexes, last_token_extracted_ix));
+            }
+            else
+            {
+              next_token_to_extract_ix = -1;
+            }
+            rb_hash_aset(extracted_tokens,
+                rb_ary_entry(tokens_to_extract_names, curr_token_ix - 1),
+                rb_usascii_str_new(input_string + startpoint,
+                  ix - startpoint));
+          }
+        }
+        startpoint = ix;
+      }
+      if(looking_for_ix >= looking_for_len - 1)
+      {
+        /* leaving token */
+        if(curr_token_ix >= n_tokens-1)
+        {
+          break;
+        }
+        else
+        {
+          startpoint = ix + 1;
+        }
+
+
+        /* next token */
+        find_ix++;
+        if(find_ix >= n_tokens_to_find)
+        {
+          /* done! */
+          break;
+        }
+
+        curr_token = rb_ary_entry(tokens_to_find_strings, find_ix);
+        curr_token_ix = NUM2UINT(rb_ary_entry(tokens_to_find_indexes, find_ix));
+        looking_for_token = StringValueCStr(curr_token);
+        looking_for_len = strlen(looking_for_token);
+        looking_for_ix = 0;
+      }
+      else
+      {
+        looking_for_ix++;
+      }
+      looking_for = looking_for_token[looking_for_ix];
+    }
+  }
+
+  ix = str_len;
+  curr_token_ix = n_tokens - 1;
+
+  if(curr_token_ix == next_token_to_extract_ix)
+  {
+    rb_hash_aset(extracted_tokens,
+        rb_ary_entry(tokens_to_extract_names, curr_token_ix),
+        rb_usascii_str_new(input_string + startpoint,
+          ix - startpoint));
+  }
+
+  return extracted_tokens;
+}
+
+void finalize_c_tokenizer_ext(VALUE unused)
+{
+  /* free memory, etc */
+}
+
+void Init_c_tokenizer_ext(void)
+{
+  rb_mStringEater = rb_define_module("StringEater");
+  rb_cCTokenizer = rb_define_class_under(rb_mStringEater, 
+      "CTokenizer", rb_cObject);
+
+  rb_define_method(rb_cCTokenizer, "ctokenize!", tokenize_string, 5);
+
+  /* set the callback for when the extension is unloaded */
+  rb_set_end_proc(finalize_c_tokenizer_ext, 0);
+}

data/ext/string-eater/extconf.rb

@@ -0,0 +1,2 @@
+require 'mkmf'
+create_makefile('c_tokenizer_ext')

data/lib/c-tokenizer.rb

@@ -0,0 +1,93 @@
+require 'c_tokenizer_ext'
+
+class StringEater::CTokenizer
+  def self.tokens
+    @tokens ||= []
+  end
+
+  def self.add_field name, opts={}
+    self.tokens << StringEater::Token::new_field(name, opts)
+    define_method(name) {@extracted_tokens[name]}
+  end
+
+  def self.look_for tokens
+    self.tokens << StringEater::Token::new_separator(tokens)
+  end
+
+  def initialize
+    refresh_tokens
+  end
+
+  def tokens
+    @tokens
+  end
+
+  def refresh_tokens
+    @tokens = self.class.tokens
+    tokens_to_find = tokens.each_with_index.map do |t, i|
+      [i, t.string] if t.string
+    end.compact
+
+    @tokens_to_find_indexes = tokens_to_find.map{|t| t[0]}
+    @tokens_to_find_strings = tokens_to_find.map{|t| t[1]}
+
+    tokens_to_extract = tokens.each_with_index.map do |t, i|
+      [i, t.name] if t.extract?
+    end.compact
+
+    @tokens_to_extract_indexes = tokens_to_extract.map{|t| t[0]}
+    @tokens_to_extract_names = tokens.map{|t| t.name}
+  end
+
+  def describe_line
+    tokens.inject("") do |desc, t|
+      desc << (t.string || t.name.to_s || "xxxxxx")
+    end
+  end
+
+  def do_extra_parsing
+  end
+
+  def tokenize! string, &block
+    @string = string
+    @extracted_tokens ||= {}
+    @extracted_tokens.clear
+
+    tokens.first.breakpoints[0] = 0
+
+    @extracted_tokens = ctokenize!(@string, 
+                                   @tokens_to_find_indexes,
+                                   @tokens_to_find_strings,
+                                   @tokens_to_extract_indexes,
+                                   @tokens_to_extract_names)
+
+    # extra parsing hook
+    do_extra_parsing
+
+    if block_given?
+      yield @extracted_tokens
+    end
+
+    # return self for chaining
+    self
+  end 
+  
+  private
+
+  def set_token_startpoint ix, startpoint
+    @tokens[ix].breakpoints[0] = startpoint
+  end
+
+  def get_token_startpoint ix
+    @tokens[ix].breakpoints[0]
+  end
+
+  def set_token_endpoint ix, endpoint
+    @tokens[ix].breakpoints[1] = endpoint
+  end
+
+  def extract_token? ix
+    @tokens[ix].extract?
+  end
+
+end

data/lib/ruby-tokenizer-each-char.rb

@@ -0,0 +1,145 @@
+# this tokenizer is very slow, but it illustrates the
+# basic idea of the C tokenizer
+class StringEater::RubyTokenizerEachChar
+
+  def self.tokens
+    @tokens ||= []
+  end
+
+  def self.combined_tokens
+    @combined_tokens ||= []
+  end
+
+  def self.add_field name, opts={}
+    self.tokens << StringEater::Token::new_field(name, opts)
+    define_method(name) {@extracted_tokens[name]}
+  end
+
+  def self.look_for tokens
+    self.tokens << StringEater::Token::new_separator(tokens)
+  end
+
+  def self.combine_fields opts={}
+    from_token_index = self.tokens.index{|t| t.name == opts[:from]}
+    to_token_index = self.tokens.index{|t| t.name == opts[:to]}
+    self.combined_tokens << [opts[:as], from_token_index, to_token_index]
+    define_method(opts[:as]) {@extracted_tokens[opts[:as]]}
+  end
+
+  def tokens
+    @tokens ||= self.class.tokens
+  end
+
+  def combined_tokens
+    @combined_tokens ||= self.class.combined_tokens
+  end
+
+  def refresh_tokens
+    @combined_tokens = nil
+    @tokens = nil
+    tokens
+  end
+
+  def describe_line
+    tokens.inject("") do |desc, t|
+      desc << (t.string || t.name.to_s || "xxxxxx")
+    end
+  end
+
+  def find_breakpoints string
+    tokenize!(string) unless @string == string
+    tokens.inject([]) do |bp, t|
+      bp << t.breakpoints
+      bp
+    end.flatten.uniq
+  end
+
+  def tokenize! string, &block
+    @string = string
+    @extracted_tokens ||= {}
+    @extracted_tokens.clear
+    @tokens_to_find ||= tokens.each_with_index.map do |t, i| 
+      [i, t.string] if t.string
+    end.compact
+    @tokens_to_extract_indeces ||= tokens.each_with_index.map do |t, i|
+      i if t.extract?
+    end.compact
+
+    tokens.first.breakpoints[0] = 0
+
+    find_index = 0
+
+    curr_token = @tokens_to_find[find_index]
+    curr_token_index = curr_token[0]
+    curr_token_length = curr_token[1].length
+    looking_for_index = 0
+    looking_for = curr_token[1][looking_for_index]
+
+    counter = 0
+    string.each_char do |c|
+      if c == looking_for
+        if looking_for_index == 0
+          # entering new token
+          if curr_token_index > 0
+            t = tokens[curr_token_index - 1]
+            t.breakpoints[1] = counter
+            if t.extract?
+              @extracted_tokens[t.name] = string[t.breakpoints[0]...t.breakpoints[1]]
+            end
+          end
+          tokens[curr_token_index].breakpoints[0] = counter
+        end
+        if looking_for_index >= (curr_token_length - 1)
+          # leaving token
+          tokens[curr_token_index].breakpoints[1] = counter
+
+          if curr_token_index >= tokens.size-1
+            # we're done!
+            break
+          else
+            tokens[curr_token_index + 1].breakpoints[0] = counter + 1
+          end
+
+          # next token
+          find_index += 1
+          if find_index >= @tokens_to_find.length
+            # we're done!
+            break
+          end
+          curr_token = @tokens_to_find[find_index]
+          curr_token_index = curr_token[0]
+          curr_token_length = curr_token[1].length
+          looking_for_index = 0
+        else
+          looking_for_index += 1
+        end
+      end
+      looking_for = curr_token[1][looking_for_index]
+      counter += 1
+    end
+
+    last_token = tokens.last
+    last_token.breakpoints[1] = string.length
+
+    if last_token.extract?
+      @extracted_tokens[last_token.name] = string[last_token.breakpoints[0]..last_token.breakpoints[1]]
+    end
+
+    combined_tokens.each do |combiner|
+      name = combiner[0]
+      from = @tokens[combiner[1]].breakpoints[0]
+      to = @tokens[combiner[2]].breakpoints[1]
+      @extracted_tokens[name] = string[from...to]
+    end
+
+    if block_given?
+      yield @extracted_tokens
+    end
+
+    # return self for chaining
+    self
+  end
+
+end
+
+

data/lib/ruby-tokenizer.rb

@@ -0,0 +1,98 @@
+# this tokenizer is fairly fast, but not necessarily faster than regexps
+class StringEater::RubyTokenizer
+  def self.tokens
+    @tokens ||= []
+  end
+
+  def self.combined_tokens
+    @combined_tokens ||= []
+  end
+
+  def self.add_field name, opts={}
+    self.tokens << StringEater::Token::new_field(name, opts)
+    define_method(name) {@extracted_tokens[name]}
+  end
+
+  def self.look_for tokens
+    self.tokens << StringEater::Token::new_separator(tokens)
+  end
+
+  def self.combine_fields opts={}
+    from_token_index = self.tokens.index{|t| t.name == opts[:from]}
+    to_token_index = self.tokens.index{|t| t.name == opts[:to]}
+    self.combined_tokens << [opts[:as], from_token_index, to_token_index]
+    define_method(opts[:as]) {@extracted_tokens[opts[:as]]}
+  end
+
+  def tokens
+    @tokens ||= self.class.tokens
+  end
+
+  def combined_tokens
+    @combined_tokens ||= self.class.combined_tokens
+  end
+
+  def refresh_tokens
+    @combined_tokens = nil
+    @tokens = nil
+    tokens
+  end
+
+  def describe_line
+    tokens.inject("") do |desc, t|
+      desc << (t.string || t.name.to_s || "xxxxxx")
+    end
+  end
+
+  def find_breakpoints(string)
+    @literal_tokens ||= tokens.select{|t| t.string}
+    @breakpoints ||= Array.new(2*@literal_tokens.size + 2)
+    @breakpoints[0] = 0
+    @breakpoints[-1] = string.length
+    start_point = 0
+    @literal_tokens.each_with_index do |t, i|
+      @breakpoints[2*i+1], start_point = find_end_of(t, string, start_point)
+      @breakpoints[2*i+2] = start_point
+    end
+    @breakpoints
+  end
+
+  def tokenize! string, &block
+    @extracted_tokens ||= {}
+    @extracted_tokens.clear
+    @tokens_to_extract ||= tokens.select{|t| t.extract?}
+
+    find_breakpoints(string)
+    last_important_bp = [@breakpoints.length, tokens.size].min
+    (0...last_important_bp).each do |i|
+      tokens[i].breakpoints = [@breakpoints[i], @breakpoints[i+1]]
+    end
+
+    @tokens_to_extract.each do |t|
+      @extracted_tokens[t.name] = string[t.breakpoints[0]...t.breakpoints[1]]
+    end
+
+    combined_tokens.each do |combiner|
+      name = combiner[0]
+      from = @tokens[combiner[1]].breakpoints[0]
+      to = @tokens[combiner[2]].breakpoints[1]
+      @extracted_tokens[name] = string[from...to]
+    end
+
+    if block_given?
+      yield @extracted_tokens
+    end
+
+    # return self for chaining
+    self
+  end
+
+  protected
+
+  def find_end_of token, string, start_at
+    start = string.index(token.string, start_at+1) || string.length
+    [start, [start + token.string.length, string.length].min]
+  end
+
+end
+

data/lib/string-eater.rb

@@ -0,0 +1,10 @@
+module StringEater
+  autoload :Token, 'token'
+  autoload :RubyTokenizer, 'ruby-tokenizer'
+  autoload :RubyTokenizerEachCHar, 'ruby-tokenizer-each-char'
+  autoload :CTokenizer, 'c-tokenizer'
+
+  autoload :VERSION, 'version'
+
+  class Tokenizer < CTokenizer; end
+end

data/lib/token.rb

@@ -0,0 +1,26 @@
+class StringEater::Token
+  attr_accessor :name, :string, :opts, :breakpoints, :children
+
+  def initialize
+    @opts = {}
+    @breakpoints = [nil,nil]
+  end
+
+  def extract?
+    @opts[:extract]
+  end
+
+  def self.new_field(name, opts)
+    t = new
+    t.name = name
+    t.opts = {:extract => true}.merge(opts)
+    t
+  end
+
+  def self.new_separator(string)
+    t = new
+    t.string = string
+    t
+  end
+
+end

data/lib/version.rb

@@ -0,0 +1,9 @@
+module StringEater
+  module VERSION 
+    MAJOR = 0
+    MINOR = 1
+    PATCH = 0
+    PRE   = nil
+    STRING = [MAJOR, MINOR, PATCH, PRE].compact.join('.')
+  end
+end

data/spec/nginx_spec.rb

@@ -0,0 +1,27 @@
+require 'spec_helper'
+require 'string-eater'
+
+$: << File.expand_path(File.join(File.dirname(__FILE__), '..', 'examples'))
+
+require 'nginx'
+
+describe NginxLogTokenizer do
+  before(:each) do
+    @tokenizer = NginxLogTokenizer.new
+    @str = '73.80.217.212 - - [01/Aug/2012:09:14:25 -0500] "GET /this_is_a_url HTTP/1.1" 304 152 "http://referrer.com" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)" "-" "there could be" other "stuff here"'
+   end
+
+   {
+      :ip => "73.80.217.212",
+      :request => "GET /this_is_a_url HTTP/1.1",
+      :status_code => 304,
+      :referrer_url => "http://referrer.com",
+      :user_agent => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)",
+      :remainder => "\"there could be\" other \"stuff here\"",
+   }.each_pair do |token,val|
+        it "should find the right value for #{token}" do
+          @tokenizer.tokenize!(@str).send(token).should == val
+        end
+      end
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1 @@
+$LOAD_PATH.concat %w[./lib ./ext/string-eater]

data/spec/string_eater_spec.rb

@@ -0,0 +1,133 @@
+require 'spec_helper'
+require 'string-eater'
+
+TestedClass = StringEater::CTokenizer
+
+describe StringEater do
+  it "should have a version" do
+    StringEater::VERSION::STRING.split(".").size.should >= 3
+  end
+end
+
+# normal use
+class Example1 < TestedClass
+  add_field :first_word
+  look_for " "
+  add_field :second_word, :extract => false
+  look_for "|"
+  add_field :third_word
+end
+
+describe Example1 do
+
+  before(:each) do
+    @tokenizer = Example1.new
+    @str1 = "foo bar|baz"
+    @first_word1 = "foo"
+    @third_word1 = "baz"
+    @bp1 = [0, 3,4,7,8,11]
+  end
+
+  describe "find_breakpoints" do
+    it "should return an array of the breakpoints" do
+      @tokenizer.find_breakpoints(@str1).should == @bp1 if @tokenizer.respond_to?(:find_breakpoints)
+    end
+  end
+
+  describe "tokenize!" do
+    it "should return itself" do
+      @tokenizer.tokenize!(@str1).should == @tokenizer
+    end
+
+    it "should set the first word" do
+      @tokenizer.tokenize!(@str1).first_word.should == "foo"
+    end
+
+    it "should set the third word" do
+      @tokenizer.tokenize!(@str1).third_word.should == "baz"
+    end
+
+    it "should not set the second word" do
+      @tokenizer.tokenize!(@str1).second_word.should be_nil
+    end
+
+    it "should yield a hash of tokens if a block is given" do
+      @tokenizer.tokenize!(@str1) do |tokens|
+        tokens[:first_word].should == "foo"
+      end
+    end
+
+    it "should return everything to the end of the line for the last token" do
+      s = "c defg asdf | foo , baa"
+      @tokenizer.tokenize!("a b|#{s}").third_word.should == s
+    end
+
+  end
+
+end
+
+# an example where we ignore after a certain point in the string
+class Example2 < TestedClass
+  add_field :first_word, :extract => false
+  look_for " "
+  add_field :second_word
+  look_for " "
+  add_field :third_word, :extract => false
+  look_for "-"
+end
+
+describe Example2 do
+
+  before(:each) do
+    @tokenizer = Example2.new
+    @str1 = "foo bar baz-"
+    @second_word1 = "bar"
+  end
+
+  describe "tokenize!" do
+    it "should find the token when there is extra stuff at the end of the string" do
+      @tokenizer.tokenize!(@str1).second_word.should == @second_word1
+    end
+  end
+
+end
+
+# CTokenizer doesn't do combine_fields because
+#  writing out breakpoints is a significant slow-down
+if TestedClass.respond_to?(:combine_fields)
+  # an example where we combine fields
+  class Example3 < TestedClass
+    add_field :first_word, :extract => false
+    look_for " \""
+    add_field :part1, :extract => false
+    look_for " "
+    add_field :part2
+    look_for " "
+    add_field :part3, :extract => false
+    look_for "\""
+
+    combine_fields :from => :part1, :to => :part3, :as => :parts
+  end
+
+  describe Example3 do
+    before(:each) do
+      @tokenizer = Example3.new
+      @str1 = "foo \"bar baz bang\""
+      @part2 = "baz"
+      @parts = "bar baz bang"
+    end
+
+    it "should extract like normal" do
+      @tokenizer.tokenize!(@str1).part2.should == @part2
+    end
+
+    it "should ignore like normal" do
+      @tokenizer.tokenize!(@str1).part1.should be_nil
+    end
+
+    it "should extract the combined field" do
+      @tokenizer.tokenize!(@str1).parts.should == @parts
+    end
+
+  end
+end

metadata

@@ -0,0 +1,66 @@
+--- !ruby/object:Gem::Specification
+name: string-eater
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Dan Swain
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-08-20 00:00:00.000000000 Z
+dependencies: []
+description: Fast string tokenizer. Nom strings.
+email:
+- dan@simpli.fi
+executables: []
+extensions:
+- ext/string-eater/extconf.rb
+extra_rdoc_files: []
+files:
+- lib/c-tokenizer.rb
+- lib/ruby-tokenizer-each-char.rb
+- lib/ruby-tokenizer.rb
+- lib/string-eater.rb
+- lib/token.rb
+- lib/version.rb
+- ext/string-eater/extconf.rb
+- ext/string-eater/c-tokenizer.c
+- spec/nginx_spec.rb
+- spec/spec_helper.rb
+- spec/string_eater_spec.rb
+- examples/address.rb
+- examples/nginx.rb
+- LICENSE
+- Rakefile
+- README.md
+homepage: http://github.com/simplifi/string-eater
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+- ext/string-eater
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Fast string tokenizer.  Nom strings.
+test_files:
+- spec/nginx_spec.rb
+- spec/spec_helper.rb
+- spec/string_eater_spec.rb