rroonga 5.1.1 → 6.0.0

data/doc/text/release.md

@@ -0,0 +1,135 @@
+# Release
+
+This document describes how to release Rroonga.
+
+## 1. Summarize the changes for this release
+
+### Show the commits since the latest release
+
+    % git log -p --reverse <the latest release version>..HEAD
+
+For example:
+
+    % git log -p --reverse 4.0.3..HEAD
+
+Or
+
+[Commits on GitHub](https://github.com/ranguba/rroonga/commits/master)
+
+### Extract the commits related to users
+
+#### Including
+
+* Changed specifications
+* Added new features
+* ...
+
+#### Excluding
+
+* Refactorings
+* Tests
+* ...
+
+### Categorize
+
+* Improvements: new features, improved behavior, ...
+* Fixes: fixed bugs, ...
+
+### Thanks
+
+If a contributor name is in a commit log, create "Thanks" group and write the name to the group.
+
+### Template for a new release for news.textile
+
+    ## X.X.X: YYYY-MM-DD {#version-x-x-x}
+
+    ### Improvements
+
+    * ...
+
+    ### Fixes
+
+    * ...
+
+    ### Thanks
+
+    * ...
+
+## 2. Translate
+
+### 2-1. Update PO
+
+Update PO files (e.g.: ja.po) by the following command:
+
+    $ rake reference:po:update
+
+### 2-2. Edit PO
+
+Then, edit PO files:
+
+* Edit msgid and msgstr.
+* Search fuzzy.
+  * Edit if necessary.
+  * Then, remove fuzzy.
+
+## 3. Upload gem to RubyGems.org
+
+### 3-1. For Linux
+
+    $ rake release
+
+### 3-2. For Windows
+
+    $ rake build:windows
+    $ gem push pkg/rroonga-<the latest release version>-x86-mingw32.gem
+    $ gem push pkg/rroonga-<the latest release version>-x64-mingw32.gem
+
+### 3-3. Bump version
+
+    $ $EDITOR ext/groonga/rb-grn.h
+
+## 4. Update ranguba.org
+
+### 4-1. Update the latest version and release date
+
+Update version and release date in ranguba.org (index.html、ja/index.html).
+
+First, clone ranguba.org repository by the following command:
+
+    $ cd ..
+    $ git clone git@github.com:ranguba/ranguba.org.git
+    $ cd ranguba.org
+
+Or
+
+    $ cd ../ranguba.org
+    $ git pull
+
+Second, update the latest version and release date:
+
+    $ $EDITOR _config.yml
+
+### 4-2. Update reference manual
+
+First, copy references to reference html directory in ranguba.org:
+
+    $ cd ../rroonga
+    $ rake release:references:upload
+
+Second, commit and push the html directory:
+
+    $ cd ../ranguba.org
+    $ git add .
+    $ git commit
+    $ git push
+
+## 5. Announce in mailing lists
+
+* groonga-talk (English)
+  * e.g.: http://sourceforge.net/p/groonga/mailman/message/33144993/
+* groonga-dev (Japanese)
+  * e.g.: http://sourceforge.jp/projects/groonga/lists/archive/dev/2014-December/003014.html
+* ruby-talk (English) (optional)
+  * e.g.: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/428992
+* ruby-list (Japanese) (optional)
+  * e.g.: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/50025

data/doc/text/tutorial.md

@@ -0,0 +1,402 @@
+# Tutorial
+
+This page introduce how to use Rroonga via a simple application making.
+
+## Install
+
+You can install Rroonga in your compter with RubyGems.
+
+    % sudo gem install rroonga
+
+## Create Database
+
+Let's create database for simple bookmark application.
+Please execute irb with loading Rroonga with this command:
+
+    % irb --simple-prompt -r groonga
+    >>
+
+Then, try to create database in a file.
+
+    >> Groonga::Database.create(:path => "/tmp/bookmark.db")
+    => #<Groonga::Database ...>
+
+From now, the created database is used implicitly.
+You don't have to be aware of it after you created a database first.
+
+## Define table
+
+Groonga supports 4 types of tables.
+
+Groonga::Hash
+:   Hash table. It manages records via each primary key. It supports
+    very quickly exact match search.
+
+Groonga::PatriciaTrie
+:   Patricia Trie. It supports some search such as predictive search and
+    common prefix search, but it provides a little slowly exact match search
+    than Groonga::Hash. It provides cursor to take records in ascending
+    or descending order.
+
+Groonga::DoubleArrayTrie
+:   Double Array Trie. It requires large spaces rather than other
+    tables, but it can update key without ID change. It provides exract
+    match search, predictive search and common prefix search and cursor
+    like Groonga::PatriciaTrie.
+
+Groonga::Array
+:   Array. It doesn't have primary keys. It manages records by ID.
+
+Now, you use Groonga::Hash and create the table named `Items`. The type
+of its primary key is String.
+
+    >> Groonga::Schema.create_table("Items", :type => :hash)
+    => [...]
+
+You have `Items` table by this code.
+You can refer the defined table with Groonga.[] like below:
+
+    >> items = Groonga["Items"]
+    => #<Groonga::Hash ...>
+
+You can treat it like Hash.
+For example, let's type `items.size` to get the number of records in
+the table.
+
+    >> items.size
+    => 0
+
+## Add records
+
+Let's add records to `Items` table.
+
+    >> items.add("http://en.wikipedia.org/wiki/Ruby")
+    => #<Groonga::Record ...>
+    >> items.add("http://www.ruby-lang.org/")
+    => #<Groonga::Record ...>
+
+Please check the number of records. It increases from 0 to 2.
+
+    >> items.size
+    => 2
+
+If you can get record by primary key, type like below:
+
+    >> items["http://en.wikipedia.org/wiki/Ruby"]
+    => #<Groonga::Record ...>
+
+## Full text search
+
+Let's add item's title to full text search.
+
+first, you add the `Text` type column "`title`" to `Items` table.
+
+    >> Groonga::Schema.change_table("Items") do |table|
+    ?>     table.text("title")
+    >>   end
+    => [...]
+
+Defined columns is named as `#{TABLE_NAME}.#{COLUMN_NAME}`.
+You can refer them with {Groonga.[]} as same as tables.
+
+    >> title_column = Groonga["Items.title"]
+    => #<Groonga::VariableSizeColumn ...>
+
+
+Secondly, let's add the table containing terms from splited from texts.
+Then you define the `Terms` for it.
+
+    >> Groonga::Schema.create_table("Terms",
+    ?>                              :type => :patricia_trie,
+    ?>                              :normalizer => :NormalizerAuto,
+    ?>                              :default_tokenizer => "TokenBigram")
+
+You specify `:default_tokenzier => "TokenBigram"` for "Tokenizer" in
+the above code.
+"Tokenizer" is the object to split terms from texts. The default value
+for it is none.
+Full text search requires a tokenizer, so you specify "Bigram", a type
+of N-gram.
+Full text search with N-gram uses splited N characters and their
+position in texts. "N" in N-gram specifies the number of each terms.
+Groonga supports Unigram (N=1), Bigram (N=2) and Trigram (N=3).
+
+You also specify `:normalizer => :NormalizerAuto` to search texts with
+ignoring the case.
+
+Now, you ready table for terms, so you define the index of
+`Items.tiltle` column.
+
+    >> Groonga::Schema.change_table("Terms") do |table|
+    ?>     table.index("Items.title")
+    >>   end
+    => [...]
+
+You may feel a few unreasonable code. The index of `Items` table's
+column is defined as the column in `Terms`.
+
+When a record is added to `Items`, groonga adds records associated
+each terms in it to `Terms` automatically.
+
+
+`Terms` is a few particular table, but you can add some columns to term
+table such as `Terms` and manage many attributes of each terms. It is
+very useful to process particular search.
+
+Now, you finished table definition.
+Let's put some values to `title` of each record you added before.
+
+    >> items["http://en.wikipedia.org/wiki/Ruby"].title = "Ruby"
+    => "Ruby"
+    >> items["http://www.ruby-lang.org/"].title = "Ruby Programming Language"
+    "Ruby Programming Language"
+
+Now, you can do full text search like above:
+
+    >> ruby_items = items.select {|record| record.title =~ "Ruby"}
+    => #<Groonga::Hash ..., normalizer: (nil)>
+
+Groonga returns the search result as Groonga::Hash.
+Keys in this hash table is records of hitted `Items`.
+
+    >> ruby_items.collect {|record| record.key.key}
+    => ["http://en.wikipedia.org/wiki/Ruby", "http://www.ruby-lang.org/"]
+
+In above example, you get records in `Items` with `record.key`, and
+keys of them with `record.key.key`.
+
+You can access a refered key in records briefly with `record["_key"]`.
+
+    >> ruby_items.collect {|record| record["_key"]}
+    => ["http://en.wikipedia.org/wiki/Ruby", "http://www.ruby-lang.org/"]
+
+## Improve the simple bookmark application
+
+Let's try to improve this simple application a little. You can create
+bookmark application for multi users and they can comment to each
+bookmarks.
+
+First, you add tables for users and for comments like below:
+
+!http://qwik.jp/senna/senna2.files/rect4605.png!
+
+Let's add the table for users, `Users`.
+
+    >> Groonga::Schema.create_table("Users", :type => :hash) do |table|
+    ?>     table.text("name")
+    >>   end
+    => [...]
+
+
+Next, let's add the table for comments as `Comments`.
+
+    >> Groonga::Schema.create_table("Comments") do |table|
+    ?>     table.reference("item")
+    >>   table.reference("author", "Users")
+    >>   table.text("content")
+    >>   table.time("issued")
+    >>   end
+    => [...]
+
+Then you define the index of `content` column in `Comments` for full
+text search.
+
+    >> Groonga::Schema.change_table("Terms") do |table|
+    ?>     table.index("Comments.content")
+    >>   end
+    => [...]
+
+You finish table definition by above code.
+
+Secondly, you add some users to `Users`.
+
+    >> users = Groonga["Users"]
+    => #<Groonga::Hash ...>
+    >> users.add("alice", :name => "Alice")
+    => #<Groonga::Record ...>
+    >> users.add("bob", :name => "Bob")
+    => #<Groonga::Record ...>
+
+Now, let's write the process to bookmark by a user.
+You assume that the user, `moritan`, bookmark a page including
+infomation related Ruby.
+
+First, you check if the page has been added `Items` already.
+
+    >> items.has_key?("http://www.ruby-doc.org/")
+    => false
+
+The page hasn't been added, so you add it to `Items`.
+
+    >> items.add("http://www.ruby-doc.org/",
+    ?>           :title => "Ruby-Doc.org: Documenting the Ruby Language")
+=> #<Groonga::Record ...>
+
+Next, you add the record to `Comments`. This record contains this page
+as its `item` column.
+
+    >> require "time"
+    => true
+    >> comments = Groonga["Comments"]
+    => #<Groonga::Array ...>
+    >> comments.add(:item => "http://www.ruby-doc.org/",
+    ?>              :author => "alice",
+    ?>              :content => "Ruby documents",
+    ?>              :issued => Time.parse("2010-11-20T18:01:22+09:00"))
+    => #<Groonga::Record ...>
+
+## Define methods for this process
+
+For usefull, you define methods for above processes.
+
+    >> @items = items
+    => #<Groonga::Hash ...>
+    >> @comments = comments
+    => #<Groonga::Array ...>
+    >> def add_bookmark(url, title, author, content, issued)
+    >>   item = @items[url] || @items.add(url, :title => title)
+    >>   @comments.add(:item => item,
+    ?>                 :author => author,
+    ?>                 :content => content,
+    ?>                 :issued => issued)
+    >>   end
+    => nil
+
+You assign `items` and `comments` to each instance variable, so you can
+use them in `add_bookmark` method.
+
+`add_bookmark` executes processes like below:
+
+* Check if the record associated the page exists in `Items` table.
+* If not, add the record to it.
+* Add the record to `Comments` table.
+
+With this method, lets bookmark some pages.
+
+    >> add_bookmark("https://rubygems.org/",
+    ?>              "RubyGems.org | your community gem host", "alice", "Ruby gems",
+    ?>              Time.parse("2010-10-07T14:18:28+09:00"))
+    => #<Groonga::Record ...>
+    >> add_bookmark("http://ranguba.org/",
+    ?>              "Fulltext search by Ruby with groonga - Ranguba", "bob",
+    ?>              "Ruby groonga fulltextsearch",
+    ?>              Time.parse("2010-11-11T12:39:59+09:00"))
+    => #<Groonga::Record ...>
+    >> add_bookmark("http://www.ruby-doc.org/",
+    ?>              "ruby-doc", "bob", "ruby documents",
+    ?>              Time.parse("2010-07-28T20:46:23+09:00"))
+    => #<Groonga::Record ...>
+
+## Full text search part 2
+
+Let's do full text search for added records.
+
+    >> records = comments.select do |record|
+    ?>     record["content"] =~ "Ruby"
+    >>   end
+    => #<Groonga::Hash ...>
+    >> records.each do |record|
+    ?>     comment = record
+    >>   p [comment.id,
+    ?>       comment.issued,
+    ?>       comment.item.title,
+    ?>       comment.author.name,
+    ?>       comment.content]
+    >>   end
+    [1, 2010-11-20 18:01:22 +0900, "Ruby-Doc.org: Documenting the Ruby Language", "Alice", "Ruby documents"]
+    [2, 2010-10-07 14:18:28 +0900, "RubyGems.org | your community gem host", "Alice", "Ruby gems"]
+    [3, 2010-11-11 12:39:59 +0900, "Fulltext search by Ruby with groonga - Ranguba", "Bob", "Ruby groonga fulltextsearch"]
+    [4, 2010-07-28 20:46:23 +0900, "Ruby-Doc.org: Documenting the Ruby Language", "Bob", "ruby documents"]
+
+You can access the columns with the same name method as each them.
+These methods suport to access the complex data type.
+(In usually RDB, you should namage JOIN tables, `Items`, `Comments`,
+`Users`.)
+
+The search is finished when the first sentence in this codes. The
+results of this search is the object as records set.
+
+    >> records
+    #<Groonga::Hash ..., size: <4>>
+
+You can arrange this records set before output.
+For example, sort these records in the descending order by date.
+
+    >> records.sort([{:key => "issued", :order => "descending"}]).each do |record|
+    ?>     comment = record
+    >>   p [comment.id,
+    ?>       comment.issued,
+    ?>       comment.item.title,
+    ?>       comment.author.name,
+    ?>       comment.content]
+    >>   end
+    [1, 2010-11-20 18:01:22 +0900, "Ruby-Doc.org: Documenting the Ruby Language", "Alice", "Ruby documents"]
+    [2, 2010-11-11 12:39:59 +0900, "Fulltext search by Ruby with groonga - Ranguba", "Bob", "Ruby groonga fulltextsearch"]
+    [3, 2010-10-07 14:18:28 +0900, "RubyGems.org | your community gem host", "Alice", "Ruby gems"]
+    [4, 2010-07-28 20:46:23 +0900, "Ruby-Doc.org: Documenting the Ruby Language", "Bob", "ruby documents"]
+    => [...]
+
+Let's group the result by each item for easy view.
+
+    >> records.group("item").each do |record|
+    ?>     item = record.key
+    >>   p [record.n_sub_records,
+    ?>       item.key,
+    ?>       item.title]
+    >>   end
+    [2, "http://www.ruby-doc.org/", "Ruby-Doc.org: Documenting the Ruby Language"]
+    [1, "https://rubygems.org/", "RubyGems.org | your community gem host"]
+    [1, "http://ranguba.org/", "Fulltext search by Ruby with groonga - Ranguba"]
+    => nil
+
+`n_sub_records` is the number of records in each group.
+It is similar value as count() function of a query including "GROUP
+BY" in SQL.
+
+## more complex search
+
+Now, you challenge the more useful search.
+
+You should calcurate goodness of fit of search explicitly.
+
+You can use `Items.title` and `Comments.content` as search targets now.
+`Items.title` is the a few reliable information taken from each
+original pages. On the other hands, `Comments.content` is the less
+reliable information because this depends on users of bookmark
+application.
+
+Then, you search records with this policy:
+
+* Search item matched `Items.title` or `Comments.content`.
+* Add 10 times heavier weight to socres of each record matched
+  `Items.title` than ones of `Comments.comment`.
+* If multi `comment` of one item are matched keyword, specify the sum
+  of scores of each `coments` as score of the item.
+
+On this policy, you try to type below:
+
+    >> ruby_comments = @comments.select {|record| record.content =~ "Ruby"}
+    => #<Groonga::Hash ..., size: <4>
+    >> ruby_items = @items.select do |record|
+    ?>     target = record.match_target do |match_record|
+    ?>       match_record.title * 10
+    >>     end
+    >>   target =~ "Ruby"
+    >>   end
+    #<Groonga::Hash ..., size: <4>>
+
+You group the results of *ruby_comments* in each item and union
+*ruby_items* .
+
+    >> ruby_items = ruby_comments.group("item").union!(ruby_items)
+    #<Groonga::Hash ..., size: <5>>
+    >> ruby_items.sort([{:key => "_score", :order => "descending"}]).each do |record|
+    >>   p [record.score, record.title]
+    >> end
+    [22, "Ruby-Doc.org: Documenting the Ruby Language"]
+    [11, "Fulltext search by Ruby with groonga - Ranguba"]
+    [10, "Ruby Programming Language"]
+    [10, "Ruby"]
+    [1, "RubyGems.org | your community gem host"]
+
+Then, you get the result.

data/ext/groonga/rb-grn-column.c

@@ -2,6 +2,7 @@
 /* vim: set sts=4 sw=4 ts=8 noet: */
 /*
   Copyright (C) 2009-2015  Kouhei Sutou <kou@clear-code.com>
+  Copyright (C) 2016  Masafumi Yokoyama <yokoyama@clear-code.com>
 
   This library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
@@ -709,44 +710,72 @@ rb_grn_column_with_weight_p(VALUE self)
 }
 
 /*
- * Return indexes on `column` which can execute `operator`.
- * @since 1.0.9
- * @return [Array<index_column>] Indexes on `column` which can execute `operator`.
- * @overload indexes(operator=Groonga::Operator::MATCH)
- *   @param [Groonga::Operator::XXX] operator
+ * Return indexes on `column`. If operator is specified, indexes that
+ * can executes the operator are only returned. Otherwise, all indexes
+ * are returned.
+ *
+ * Index means that index column and section.
+ *
+ * @overload find_indexes(options={})
+ *   @param options [::Hash] The name and value pairs.
+ *      Omitted names are initialized as the default value.
+ *   @option options :operator (nil) The operator that should be
+ *      executable by indexes. `nil` means that all operators.
+ *   @return [Array<Groonga::Index>] Target indexes on `column`.
+ *
+ * @since 6.0.0
  */
 static VALUE
-rb_grn_column_get_indexes (int argc, VALUE *argv, VALUE self)
+rb_grn_column_find_indexes (int argc, VALUE *argv, VALUE self)
 {
+    VALUE rb_options;
+    VALUE rb_operator;
     grn_ctx *context;
     grn_obj *column;
     grn_index_datum *index_data = NULL;
     int i, n_indexes;
-    grn_operator operator = GRN_OP_MATCH;
-    VALUE rb_operator, rb_indexes;
+    VALUE rb_indexes;
 
-    rb_scan_args(argc, argv, "01", &rb_operator);
+    rb_scan_args(argc, argv, "01", &rb_options);
+    rb_grn_scan_options(rb_options,
+                        "operator", &rb_operator,
+                        NULL);
 
     rb_grn_column_deconstruct(SELF(self), &column, &context,
-                             NULL, NULL,
-                             NULL, NULL, NULL);
+                              NULL, NULL,
+                              NULL, NULL, NULL);
 
-    if (!NIL_P(rb_operator)) {
+    if (NIL_P(rb_operator)) {
+        n_indexes = grn_column_get_all_index_data(context, column, NULL, 0);
+        if (n_indexes == 0)
+            return rb_ary_new();
+
+        index_data = xmalloc(sizeof(grn_index_datum) * n_indexes);
+        n_indexes = grn_column_get_all_index_data(context, column,
+                                                  index_data, n_indexes);
+    } else {
+        grn_operator operator;
         operator = RVAL2GRNOPERATOR(rb_operator);
+        n_indexes = grn_column_find_index_data(context, column, operator,
+                                               NULL, 0);
+        if (n_indexes == 0)
+            return rb_ary_new();
+
+        index_data = xmalloc(sizeof(grn_index_datum) * n_indexes);
+        n_indexes = grn_column_find_index_data(context, column, operator,
+                                               index_data, n_indexes);
     }
 
-    rb_indexes = rb_ary_new();
-    n_indexes = grn_column_find_index_data(context, column, operator, NULL, 0);
-    if (n_indexes == 0)
-        return rb_indexes;
-
-    index_data = xmalloc(sizeof(grn_index_datum) * n_indexes);
-    n_indexes = grn_column_find_index_data(context, column, operator,
-                                           index_data, n_indexes);
+    rb_indexes = rb_ary_new_capa(n_indexes);
     for (i = 0; i < n_indexes; i++) {
-        VALUE rb_index;
-        rb_index = GRNOBJECT2RVAL(Qnil, context, index_data[i].index, GRN_FALSE);
-        rb_ary_push(rb_indexes, rb_index);
+        VALUE rb_index_column;
+        VALUE rb_section;
+        rb_index_column = GRNOBJECT2RVAL(Qnil,
+                                         context,
+                                         index_data[i].index,
+                                         GRN_FALSE);
+        rb_section = UINT2NUM(index_data[i].section);
+        rb_ary_push(rb_indexes, rb_grn_index_new(rb_index_column, rb_section));
         grn_obj_unlink(context, index_data[i].index);
     }
     xfree(index_data);
@@ -812,7 +841,8 @@ rb_grn_init_column (VALUE mGrn)
     rb_define_method(rb_cGrnColumn, "with_weight?",
                      rb_grn_column_with_weight_p, 0);
 
-    rb_define_method(rb_cGrnColumn, "indexes", rb_grn_column_get_indexes, -1);
+    rb_define_method(rb_cGrnColumn, "find_indexes",
+                     rb_grn_column_find_indexes, -1);
 
     rb_define_method(rb_cGrnColumn, "rename", rb_grn_column_rename, 1);