stringtree 0.1.0

data/.gitignore

@@ -0,0 +1,7 @@
+*.gem
+*.rbc
+*.tokens.txt
+.bundle
+.config
+coverage
+doc/

data/.rspec

@@ -0,0 +1 @@
+--color

data/.travis.yml

@@ -0,0 +1,2 @@
+language: ruby
+sudo: false

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,13 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion.
+
+Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
+
+This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)

data/Gemfile

@@ -0,0 +1,3 @@
+source "https://rubygems.org"
+
+gemspec

data/Gemfile.lock

@@ -0,0 +1,78 @@
+PATH
+  remote: .
+  specs:
+    stringtree (0.1.0)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    coveralls (0.8.2)
+      json (~> 1.8)
+      rest-client (>= 1.6.8, < 2)
+      simplecov (~> 0.10.0)
+      term-ansicolor (~> 1.3)
+      thor (~> 0.19.1)
+    diff-lcs (1.2.5)
+    docile (1.1.5)
+    domain_name (0.5.24)
+      unf (>= 0.0.5, < 1.0.0)
+    http-cookie (1.0.2)
+      domain_name (~> 0.5)
+    json (1.8.3)
+    mime-types (2.6.1)
+    netrc (0.10.3)
+    rake (10.4.2)
+    rdoc (4.2.0)
+      json (~> 1.4)
+    rest-client (1.8.0)
+      http-cookie (>= 1.0.2, < 2.0)
+      mime-types (>= 1.16, < 3.0)
+      netrc (~> 0.7)
+    rspec (3.3.0)
+      rspec-core (~> 3.3.0)
+      rspec-expectations (~> 3.3.0)
+      rspec-mocks (~> 3.3.0)
+    rspec-core (3.3.2)
+      rspec-support (~> 3.3.0)
+    rspec-expectations (3.3.1)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.3.0)
+    rspec-mocks (3.3.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.3.0)
+    rspec-support (3.3.0)
+    sdoc (0.4.1)
+      json (~> 1.7, >= 1.7.7)
+      rdoc (~> 4.0)
+    simplecov (0.10.0)
+      docile (~> 1.1.0)
+      json (~> 1.8)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.0)
+    simplecov-rcov (0.2.3)
+      simplecov (>= 0.4.1)
+    term-ansicolor (1.3.2)
+      tins (~> 1.0)
+    thor (0.19.1)
+    tins (1.6.0)
+    unf (0.1.4)
+      unf_ext
+    unf_ext (0.0.7.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.10)
+  coveralls
+  rake (~> 10.0)
+  rdoc (~> 4.1)
+  rspec (~> 3.1)
+  rspec-mocks (~> 3.1)
+  sdoc (~> 0.4)
+  simplecov (~> 0.10.0)
+  simplecov-rcov (~> 0.2)
+  stringtree!
+
+BUNDLED WITH
+   1.10.5

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Tom Cully
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,95 @@
+# stringtree-ruby
+
+[![Build Status](https://travis-ci.org/tomdionysus/stringtree-ruby.svg?branch=master)](https://travis-ci.org/tomdionysus/stringtree-ruby)
+[![Coverage Status](https://coveralls.io/repos/tomdionysus/stringtree-ruby/badge.svg?branch=master&service=github)](https://coveralls.io/github/tomdionysus/stringtree-ruby?branch=master)
+[![Gem Version](https://badge.fury.io/rb/stringtree.svg)](http://badge.fury.io/rb/stringtree)
+[![Dependency Status](https://gemnasium.com/tomdionysus/stringtree.svg)](https://gemnasium.com/tomdionysus/stringtree)
+[![Gem Downloads](http://ruby-gem-downloads-badge.herokuapp.com/stringtree?color=brightgreen)](http://ruby-gem-downloads-badge.herokuapp.com/stringtree?color=brightgreen)
+
+StringTree is a fast forward-only tokeniser and partial string matcher, that is, it can:
+
+* Load a dictionary of arbitarty size and record count - e.g. an actual dictionary, an english word list - where each record is associated with a key - e.g. a numeric identifier for the word.
+* Parse an arbitary data string in a single pass, finding and storing instances of each item in the dictionary and storing their offsets and associated keys.
+* Host a set of strings in such a way as to efficiently match partial input strings against the dictionary
+
+This has become my 'hello world' over the years with any new language. I use it to get to know a language, as implementing it correctly involves many of the usual concepts needed get started coding from the hip (syntax, grammar, classes, public/private instance vars, statics, pass-by value/pass-by-reference etc.) not to mention usual code support skills like how to set up unit tests for this language and environment, etc.
+
+## Installation
+
+```ruby
+gem install stringtree
+```
+
+## Implementation
+
+StringTree is based on multidimensional binary trees - Each node in the tree has its usual left/right references to its children, but also an optional 'down' reference, which would refer to the root of the binary tree representing the next character in the string from this point. This version also has an 'up' reference on the node which can be faster for iterating backward through a set of trees.
+
+From a CS point of view, Tree is a specific implementation of an n-dimensional trie.
+
+## Limitations
+
+* Tree is not intended and should not be used as a key/value store. There are much faster algorithims for storing such data. 
+* Tree should not be used in place of regular expressions. Tree is good when the total number and/or size of the tokens to be found is either unknown or dynamic.
+* Tree is essentially a byte tokeniser and is case-sensitive. It would be trivial to extend it to search in an case insensitive manner, however.
+
+## Applications
+
+Tree has the following applications:
+
+* Spellchecking
+* Virus scanning
+* Partial string matching, e.g. autocomplete
+* Tokenizing, e.g. Matching Bank Transaction data to Businesses from reference/particular/code fields, etc.
+
+## Environment
+
+The demo is tested and runs under Ruby 1.9.3, and will probably work under earlier and later versions but this is not guaranteed. The unit tests require rspec also:
+
+```bash
+bundle install
+```
+
+## Specs
+
+Specs are rspec, as follows:
+
+```bash
+rspec
+```
+
+## Documentation
+
+Regenerate the Documentation with `rake`:
+
+```bash
+rake rdoc
+```
+
+## Demo
+
+```bash
+bundle exec ruby examples/demo.rb
+```
+
+This will generate two files:
+
+* hamlet.tokens.txt
+* warandpeace.tokens.txt
+
+Each of which contain a set of lines as such:
+
+28: platform 41003 (8)
+
+Where 28 is the offset where the token was found, 'platform' is the token itself, 41003 is the id of the token (in this case the line number of the dictionary file), and (8) is the length of the token.
+
+The demo code then goes into a console, which will do partial searches within the dictionary. Type a partial word and press enter, and the demo will show all words in the dictionary that start with the partial entered.
+Type 'exit' to finish the demo.
+
+## Code of Conduct
+
+The StringTree project is committed to the [Contributor Covenant](http://contributor-covenant.org). Please read [CODE_OF_CONDUCT.md] before making any contributions or comments.
+
+## References
+
+* http://www.ruby-doc.org
+* http://en.wikipedia.org/wiki/Trie

data/Rakefile

@@ -0,0 +1,13 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require 'rdoc/task'
+
+RDoc::Task.new do |rd|
+  rd.main = "README.rdoc"
+  rd.rdoc_files.include("README.md", "lib/**/*.rb")
+  rd.rdoc_dir = "doc"
+end
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/examples/demo.rb

@@ -0,0 +1,82 @@
+require 'stringtree'
+
+class Main
+  def main(argc)
+    puts "StringTree Ruby Demo"
+
+    begin
+      root = File.dirname(__FILE__)
+      @st = StringTree::Tree.new
+      puts "Loading Dictionary..."
+      @count = 0
+      File.open("#{root}/dictionary.txt", "r") do |infile|
+        while line = infile.gets
+          line = line.strip
+          @st.add(line, @count+=1)
+          @st.add(line.upcase, @count+=1)
+        end
+      end
+
+      t=Time.now
+      puts "Optimizing Dictionary..."
+      @st.optimize
+      puts "Optimized #{@count} entries in "+sprintf("%.2f",(Time.now-t)*1000)+"ms"
+
+      puts "-Loading Hamlet-------------------"
+      load "#{root}/hamlet.txt"
+      do_match "hamlet.tokens.txt"
+
+      puts "--Loading War And Peace-----------"
+      load "#{root}/warandpeace.txt"
+      do_match "warandpeace.tokens.txt"
+
+      do_partial_matching
+    rescue Exception => e
+      puts "StringTree demo has encountered an error (#{e.message}). Please run again."
+    end
+  end
+
+  def load(file_name)
+    @contents = File.open(file_name,'rb').read
+  end
+
+  def do_match(output_file_name=nil)
+    t=Time.now
+    puts "Matching #{@count} entries in #{@contents.length} bytes..."
+    list = []
+    @st.match_all(@contents) { |match| list << match }
+    interval = (Time.now-t)*1000
+
+    puts list.length.to_s+" entities matched in "+sprintf("%.2f",interval)+"ms ("+sprintf("%.2f", list.length / interval * 1000)+" entities/s)"
+
+    unless output_file_name.nil?
+      puts "Writing File #{output_file_name}"
+      File.open(output_file_name,'wb') do |f|
+        list.each do |item|
+          f.write item.offset.to_s + ": #{item.node} #{item.node.value} (#{item.node.length})\n"
+        end
+      end
+    end
+  end
+
+  def do_partial_matching
+    puts "-Partial Matching -----------------"
+    puts "-Type 'exit' to quit"
+    while true do
+      print "> "
+      x = gets.chomp
+      return if x == 'exit'
+      partials = @st.partials(x)
+      str = ""
+      unless partials.nil?
+        partials.collect { |partial| str += "#{partial} " }
+      else
+        str =  "No Matches Found for '#{x}'"
+      end
+      puts str
+    end
+  end
+end
+
+@main = Main.new
+@main.main(0)