wordtriez 0.0.1

data/readme.md

@@ -0,0 +1,223 @@
+## Triez
+
+[![Build Status](https://travis-ci.org/luikore/triez.png)](https://travis-ci.org/luikore/triez)
+[![Code Climate](https://codeclimate.com/github/luikore/triez.png)](https://codeclimate.com/github/luikore/triez)
+[![Gem Version](https://badge.fury.io/rb/triez.png)](http://badge.fury.io/rb/triez)
+
+Pragmatic [tries](http://en.wikipedia.org/wiki/Trie) for Ruby, spelled in lolcat.
+
+It is fast, memory efficient, unicode aware, prefix searchable, and enchanced with prefix/suffix/substring keys.
+
+The backend of *triez* is a cache oblivious data structure: the [HAT trie](https://github.com/dcjones/hat-trie) (In fact it is a [modified version](https://github.com/luikore/hat-trie) for improved functionality). HAT trie is generally faster and more memory efficient than [double array](http://linux.thai.net/~thep/datrie/datrie.html) or [burst trie](http://ww2.cs.mu.oz.au/~jz/fulltext/acmtois02.pdf).
+
+## Requirement
+
+- CRuby 1.9 / 2.0
+- `g++` or `clang`
+
+## Install
+
+``` bash
+gem ins triez
+```
+
+## Synopsis
+
+``` ruby
+require 'triez'
+
+# create triez
+t = Triez.new
+
+# the above code is equivalent to :int64 for :value_type and 0 for :default
+t = Triez.new value_type: :int64
+
+# more flexible with object type [*see note below]
+t = Triez.new value_type: :object
+
+# get the value type
+t.value_type
+
+# set a different default value
+t = Triez.new value_type: :object, default: 'hello'
+
+# insert or change value
+t['key'] = 100
+
+# insert a key with default value
+t << 'key'
+
+# batch change values under all suffices/prefices/substrings of a key
+t.change_all(:suffix, 'key') {|old_value| ...calculate new value }
+t.change_all(:prefix, 'key') {|old_value| ...calculate new value }
+# enumerates all occurences of substrings of the key
+t.change_all(:substring, 'key') {|old_value| ...calculate new value }
+
+# size of inserted keys
+t.size
+
+# search with exact match
+t.has_key? 'key'
+t['key']
+
+# prefixed search (iterate over values under a prefix), available options are:
+# - limit: max items, `nil` means no limit
+# - sort: whether iterate in alphabetic order, default is true
+t.search_with_prefix(prefix, limit: 10, sort: true) do |suffix, value|
+  ...
+end
+
+# if no block given, an array in the form of [[suffix, value]] is returned
+t.search_with_prefix('prefix')
+
+# enumerate all keys and values in the order of binary collation
+t.each do |key, value|
+  ...
+end
+
+# iterate stored keys which are prefices of a given string, from shallow to deep
+t.walk string do |k, v|
+  ...
+end
+```
+
+\* Note: By default, *triez* store signed integers within 64bits, you can use them as weights, counts or database IDs. In case you need to store arbitrary object in a node, use `value_type: :object`:
+
+``` ruby
+t = Triez.new value_type: :object
+t['Tom'] = {name: 'Tom', sex: 'Female'}
+t['Tree'] = [:leaf, :trunk, :root]
+```
+
+## Examples
+
+**Prefix based autocompletion**:
+
+``` ruby
+require 'triez'
+words = %w[readme, rot, red, rah, rasterization]
+t = Triez.new
+words.each do |word|
+  t[word] = 1
+end
+t.search_with_prefix 're' do |suffix|
+  puts "candidate: re#{suffix}"
+end
+```
+
+The output:
+
+```bash
+candidate: readme
+candidate: red
+```
+
+---
+
+**Efficient [full text search](https://en.wikipedia.org/wiki/Full_text_search) with a [suffix tree](https://en.wikipedia.org/wiki/Suffix_tree)**:
+
+``` ruby
+require 'triez'
+sequences = {
+  'ACTGAAAAAAACTG' => 1,
+  'ATACGGTCCA' => 2,
+  'GCTTGTACGT' => 3
+}
+t = Triez.new
+
+# build suffix tree
+sequences.each do |seq, id|
+  t.change_all(:suffix, seq){id}
+end
+
+t.search_with_prefix 'CGGT' do |_, id|
+  puts id #=> 2
+end
+```
+
+The searching time is linear to the length of the substring. You may also be interested in the example of a simple [full text search server](https://github.com/luikore/triez/tree/master/examples/full-text-search-server) with *triez*.
+
+---
+
+**Solve the [longest common substring problem](https://en.wikipedia.org/wiki/Longest_common_substring_problem)**:
+
+``` ruby
+# coding: utf-8
+require 'triez'
+sentences = %w[
+  万塘路一锅鸡
+  去文二路一锅鸡吃饭
+  来一锅鸡顶盒
+  一锅鸡胗
+]
+
+# value is bitset representing id of the sentence
+# in ruby we can use integers of arbitrary length as bitsets
+t = Triez.new value_type: :object, default: 0
+
+sentences.each_with_index do |sentence, i|
+  elem = 1 << i
+  t.change_all :substring, sentence do |v|
+    # union
+    v | elem
+  end
+end
+
+# longest common substring
+lcs = ''
+
+# find the key tagged with universe
+universe = (1 << sentences.size) - 1
+t.each do |k, v|
+  lcs = k if k.size > lcs.size and v == universe
+end
+
+puts lcs #=> 一锅鸡
+```
+
+## Benchmark
+
+Here's a benchmark on
+
+```ruby
+ruby 1.9.3p374 (2013-01-15 revision 38858) [x86_64-darwin12.2.1]
+2.3 GHz Intel Core i7
+```
+
+The test data are 3 milion titles of wikipedia articles (from http://dumps.wikimedia.org/enwiki/20121101/)
+
+```
+thing/backend           | memory  | insertion time | 3 M query
+------------------------|---------|----------------|----------
+hash/linked hash        | 340.2 M |    4.369 s     | 0.2800 s
+fast_trie/double array* | 155.6 M |    130.7 s     | 0.4359 s
+triez/HAT trie          | 121.7 M |    3.872 s     | 0.3472 s
+```
+
+Note: `fast_trie/double array` -> https://github.com/tyler/trie
+
+## Caveats
+
+- The `sort` option in prefixed search orders keys with binary [collation](https://en.wikipedia.org/wiki/Collation), but string comparison in Ruby is with unicode codepoint collation.
+- For some rare case of many threads modifying the same trie, you may need a mutex.
+- If you still feel memory not enough, you may consider [MARISA-trie](https://code.google.com/p/marisa-trie/) (note that MARISA is immutable), or a database.
+
+## Development
+
+``` bash
+git clone git://github.com/luikore/triez.git
+cd triez
+rake glob_src
+rake
+```
+
+To update vendor lib and re-compile:
+
+``` bash
+rake glob_src
+rake
+```
+
+## Note
+
+Although HAT trie uses MurMurHash3 instead of SipHash in Ruby, It is still safe under hashDoS because bucket size is limited.

data/test/triez_test.rb

@@ -0,0 +1,225 @@
+# coding: utf-8
+require "test/unit"
+require_relative "../lib/triez"
+
+GC.stress
+
+class TriezTest < Test::Unit::TestCase
+  def test_init_type_options
+    t = Triez.new value_type: :int64
+    assert_equal :int64, t.value_type
+    t = Triez.new value_type: :object
+    assert_equal :object, t.value_type
+    t = Triez.new
+    assert_equal :int64, t.value_type
+
+    assert_raise ArgumentError do
+      Triez.new value_type: :string
+    end
+    assert_raise ArgumentError do
+      Triez.new invalid_option: :int64
+    end
+  end
+
+  def test_hat_trie
+    t = Triez.new value_type: :object
+
+    v1 = (1 << 40)
+    v2 = (1 << 141)
+    t['万塘路一锅鸡'] = v1
+    t['万塘路'] = v2
+    assert_equal v1, t['万塘路一锅鸡']
+    assert_equal v2, t['万塘路']
+    assert_equal nil, t['万']
+    assert_equal false, t.has_key?('万')
+    assert_equal true, t.has_key?('万塘路')
+
+    assert_equal v1, t.delete('万塘路一锅鸡')
+    assert_equal nil, t['万塘路一锅鸡']
+    assert_equal v2, t['万塘路']
+
+    a = t.search_with_prefix ''
+    assert_equal [['万塘路', v2]], a
+
+    t['马当路'] = 3
+    a = t.search_with_prefix '万塘'
+    assert_equal [['路', v2]], a
+  end
+
+  def test_insertion_and_search_on_many_keys
+    t = Triez.new
+    as = ('A'..'z').to_a
+    bs = ('一'..'百').to_a
+    as.each do |a|
+      # 10k chars to ensure burst
+      bs.each do |b|
+        t[a + b] = 0
+      end
+    end
+    assert_equal as.size * bs.size, t.size
+
+    a = t.search_with_prefix 'a'
+    assert_equal bs.to_a, a.map(&:first).sort
+
+    a = []
+    t.search_with_prefix 'b', sort: true, limit: 3 do |k, v|
+      a << k
+    end
+    assert_equal 3, a.size
+    assert_equal a, a.sort
+  end
+
+  def test_each_and_raise
+    t = Triez.new
+    t['abcd'] = 0
+    t['abc'] = 1
+
+    assert_raise NameError do
+      t.each do |k, v|
+        raise NameError, k
+      end
+    end
+
+    assert_raise ArgumentError do
+      t.each
+    end
+  end
+
+  def test_append
+    t = Triez.new
+    ('a'..'z').each do |c|
+      t << c
+    end
+    assert_equal 26, t.size
+    assert_equal 0, t['c']
+    assert_equal true, t.has_key?('c')
+  end
+
+  def test_full_text_search
+    sequences = {
+      'ACTGAAAAAAACTG' => 1,
+      'ATACGGTCCA' => 2,
+      'GCTTGTACGT' => 3
+    }
+    t = Triez.new
+    sequences.each do |seq, id|
+      t.change_all(:suffix, seq){ id }
+    end
+    assert_equal 2, t.search_with_prefix('CGGT').map(&:last).flatten.first
+  end
+
+  def test_nul_char_in_keys
+    t = Triez.new
+    t["a\0b"] = 1
+    assert_equal 1, t["a\0b"]
+    assert_equal 1, t.size
+    assert_equal 0, t["a"]
+  end
+
+  def test_change_all_with_prefix
+    default = 10
+    t = Triez.new default: default
+    t['regexp'] = 1
+    t['readme'] = 2
+    t.change_all :prefix, 'readme' do |v|
+      v += 4
+    end
+    assert_equal 'readme'.size + 1, t.size
+    assert_equal 6, t['readme']
+    assert_equal default + 4, t['read']
+    assert_equal 1, t['regexp']
+  end
+
+  def test_change_all_with_suffix
+    t = Triez.new
+    t['regexp'] = 1
+    t['exp'] = 2
+    t['reg'] = 3
+    t.change_all :suffix, 'regexp' do |v|
+      v += 4
+    end
+    assert_equal 5, t['regexp']
+    assert_equal 6, t['exp']
+    assert_equal 3, t['reg']
+    assert_equal 'regexp'.size + 1, t.size
+  end
+
+  def test_change_all_with_substring
+    t = Triez.new value_type: :object
+    t.change_all :substring, 'abc' do
+      1
+    end
+
+    keys = []
+    t.each do |k, v|
+      keys << k
+    end
+    assert_equal %w[a b c ab bc abc].sort, keys.sort
+  end
+
+  def test_walk
+    urls = %w[
+      /users/
+      /users/12/edit
+      /posts
+    ]
+    t = Triez.new value_type: :object
+    urls.each_with_index do |url, i|
+      t[url] = i.to_s
+    end
+
+    assert_equal [%w'/users/ 0'], t.walk('/users/12/delete').to_a
+    assert_equal [%w'/users/ 0', %w'/users/12/edit 1'], t.walk('/users/12/edit').to_a
+    assert_equal [%w'/users/ 0', %w'/users/12/edit 1'], t.walk('/users/12/edit/3').to_a
+
+    assert_raise TypeError do
+      t.walk :'/post' do
+      end
+    end
+
+    t.walk '' do |k, v|
+      assert_equal [nil, nil], [k, v]
+    end
+
+    # try to trigger rb_gc_mark(), it can stuck if hattrie_iter_next() not called properly
+    100000.times{ 'a' + 'b' }
+  end
+
+  def test_solve_longest_common_substring
+    sentences = %w[
+      万塘路一锅鸡
+      文二路一锅鸡
+      来一锅鸡顶盒
+      一锅鸡胗
+    ]
+
+    # value is bitset representing id of the sentence
+    # in ruby we can use integers of arbitrary length as bitsets
+    t = Triez.new value_type: :object, default: 0
+
+    sentences.each_with_index do |sentence, i|
+      elem = 1 << i
+      t.change_all :substring, sentence do |v|
+        # union
+        v | elem
+      end
+    end
+
+    # longest common substring
+    lcs = ''
+    universe = (1 << sentences.size) - 1
+    t.each do |k, v|
+      lcs = k if (k.size > lcs.size and v == universe)
+    end
+    assert_equal '一锅鸡', lcs
+  end
+
+  def test_should_not_segfault_when_search_with_prefix
+    t = Triez.new
+    # bursts when 16384
+    16_385.times{ |i| t["a#{i}"] = i }
+    expected_postfices = 16_385.times.map &:to_s
+    actual_postfices = t.search_with_prefix("a").map(&:first)
+    assert_equal expected_postfices.sort, actual_postfices.sort
+  end
+end

metadata

@@ -0,0 +1,67 @@
+--- !ruby/object:Gem::Specification
+name: wordtriez
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Zete Lui
+- Duane Johnson
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-09-21 00:00:00.000000000 Z
+dependencies: []
+description: fast, efficient, unicode aware HAT trie with prefix / suffix support.
+email: 
+executables: []
+extensions:
+- ext/extconf.rb
+extra_rdoc_files: []
+files:
+- copying
+- changes
+- readme.md
+- lib/wordtriez.rb
+- test/triez_test.rb
+- ext/triez.cc
+- ext/common.h
+- ext/extconf.rb
+- ext/hat-trie/ahtable.c
+- ext/hat-trie/ahtable.h
+- ext/hat-trie/common.h
+- ext/hat-trie/hat-trie.c
+- ext/hat-trie/hat-trie.h
+- ext/hat-trie/misc.c
+- ext/hat-trie/misc.h
+- ext/hat-trie/murmurhash3.c
+- ext/hat-trie/murmurhash3.h
+- ext/hat-trie/pstdint.h
+- ext/hat-trie/text.c
+- ext/hat-trie/text.h
+homepage: https://github.com/canadaduane/triez
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.9.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: fast, efficient, unicode aware HAT trie with prefix / suffix support
+test_files: []
+has_rdoc: false