groonga 0.0.5 → 0.0.6

data/NEWS.ja.rdoc

@@ -1,5 +1,18 @@
 = お知らせ
 
+== 0.0.6: 2009-07-31
+
+* groonga 0.1.1対応
+* ドキュメントの修正 [id:mat_aki]
+* Groonga::Table#selectでのg式対応
+* APIの追加
+  * Groonga::Table#union!
+  * Groonga::Table#intersect!
+  * Groonga::Table#differene!
+  * Groonga::Table#merge!
+* tar.gzも提供 [id:m_seki]
+* メモリリークの修正
+
 == 0.0.3: 2009-07-18
 
 * [#26145] Groonga::TableKeySupport#has_key?の追加 [Tasuku SUENAGA]

data/NEWS.rdoc

@@ -1,5 +1,18 @@
 = NEWS
 
+== 0.0.6: 2009-07-31
+
+* Supported groonga 0.1.1.
+* Fixed documents [id:mat_aki]
+* Supported groonga expression for searching.
+* Added API
+  * Groonga::Table#union!
+  * Groonga::Table#intersect!
+  * Groonga::Table#differene!
+  * Groonga::Table#merge!
+* Provided tar.gz [id:m_seki]
+* Fixed memory leaks
+
 == 0.0.3: 2009-07-18
 
 * [#26145] Added Groonga::TableKeySupport#has_key? [Tasuku SUENAGA]

data/README.ja.rdoc

@@ -32,7 +32,7 @@ pkg-config.rbはrcairoに付属しているもので、これはRubyライセ
 == 依存ソフトウェア
 
 * Ruby >= 1.8 （1.9.1対応）
-* groonga >= 0.0.8
+* groonga >= 0.1.1
 
 == インストール
 
@@ -54,3 +54,4 @@ http://lists.sourceforge.jp/mailman/listinfo/groonga-dev
 * グニャラくん: バグレポートしてくれました。
 * にくさん: バグレポートしてくれました。
 * daraさん: テストを書いてくれました。
+* id:mat_akiさん: チュートリアルのバグを教えてくれました。

data/README.rdoc

@@ -33,7 +33,7 @@ license/GPL for details.
 == Dependencies
 
 * Ruby >= 1.8 (including 1.9.1)
-* groonga >= 0.0.8
+* groonga >= 0.1.1
 
 == Install
 
@@ -55,3 +55,4 @@ http://rubyforge.org/mailman/listinfo/groonga-users-en
 * Tasuku SUENAGA: sent bug reports.
 * niku: sent bug reports.
 * dara: wrote tests.
+* id:mat_aki: sent bug reports.

data/Rakefile

@@ -119,7 +119,6 @@ Hoe.spec('groonga') do |_project|
   description = cleanup_white_space(entries[entries.index("Description") + 1])
   project.summary, project.description, = description.split(/\n\n+/, 3)
 
-  project.need_tar = false
   project.remote_rdoc_dir = "groonga"
 end
 

data/TUTORIAL.ja.rdoc

@@ -187,9 +187,9 @@ http://qwik.jp/senna/senna2.files/rect4605.png
 
 次に、<tt><comments></tt>テーブルを追加します。
 
-  >> comments = Groonga::Hash.create(:name => "<comments>",
-                                     :key_type => "<shorttext>")
-  => #<Groonga::Hash ...>
+  >> comments = Groonga::Array.create(:name => "<comments>",
+                                      :key_type => "<shorttext>")
+  => #<Groonga::Array ...>
   >> comments.define_column("item", items)
   => #<Groonga::FixSizeColumn ..>
   >> comments.define_column("author", users)
@@ -347,4 +347,49 @@ http://qwik.jp/senna/senna2.files/rect4605.png
 
 == 少し複雑な検索
 
-つづく。。。
+↓はまだ動かない！！！
+
+さらに実用的な検索について考えてみましょう。
+
+ブックマークが大量に蓄積されるに従って、より的確に適合度を算
+出する必要性に迫られます。
+
+今のところ検索対象として利用できるのは<tt><items>.title</tt>
+と<tt><comments>.content</tt>ですが、<tt><items>.title</tt>は
+元ページから得られるやや信頼できる情報なのに対して、
+<tt><comments>.content</tt>はブックマークユーザが任意に設定で
+きる情報で、やや信憑性に乏しいと言えます。しかし、再現率を確
+保するためにはユーザのコメントも是非対象に含めたいところです。
+
+そこで、以下のようなポリシーで検索を行うことにします。
+
+* <tt><items>.title</tt>か<tt><comments>.content</tt>のいずれ
+  かにマッチするitemを検索する。
+* ただし、<tt><items>.title</tt>にマッチしたレコードはスコア
+  を10倍重み付けする。
+* 同一のitemに対して、キーワードにマッチする<tt>comment</tt>
+  が複数存在した場合は、それぞれの<tt>comment</tt>のスコアの
+  和を、該当するitemのスコアとする。
+
+以下のようにして、commentとitemとそれぞれに対する検索結果を求
+めます。
+
+  >> ruby_comments = @comments.select {|record| record["content"] =~ "Ruby"}
+  #<Groonga::Hash ..., size: <2>>
+  >> ruby_items = @items.select("*W1:50 title:%Ruby")
+  #<Groonga::Hash ..., size: <2>>
+
+_ruby_comments_の結果をitem毎にグループ化し、_ruby_items_と
+unionして出力します。
+
+  >> ruby_items = ruby_comments.group([".item"]).union!(ruby_items)
+  #<Groonga::Hash ..., size: <4>>
+  >> ruby_items.sort([{:key => ".:score", :order => "descendant"}]).each do |record|
+  >>   p [record[".:score"], record[".title"]]
+  >> end
+  [1, "るびま"]
+  [1, "オブジェクトスクリプト言語Ruby"]
+  [1, "Ruby"]
+  [1, "ラングバ"]
+
+これで目的の結果が得られました。（FIXME: 得られていない！）

data/benchmark/read-write-many-small-items.rb

@@ -55,10 +55,11 @@ begin
 
   require 'groonga'
   @database = Groonga::Database.create
+  value_type = Groonga::Type.new("Text8", :size => 8)
 
   item("groonga: Hash: memory") do
     @hash = Groonga::Hash.create(:key_type => "<shorttext>",
-                                 :value_size => 8)
+                                 :value_type => value_type)
     values.each do |value|
       @hash[value] = value
       @hash[value]
@@ -67,7 +68,7 @@ begin
 
   item("groonga: Trie: memory") do
     @hash = Groonga::PatriciaTrie.create(:key_type => "<shorttext>",
-                                         :value_size => 8)
+                                         :value_type => value_type)
     values.each do |value|
       @hash[value] = value
       @hash[value]
@@ -77,7 +78,7 @@ begin
   @hash_file = Tempfile.new("groonga-hash")
   item("groonga: Hash: file") do
     @hash = Groonga::Hash.create(:key_type => "<shorttext>",
-                                 :value_size => 8,
+                                 :value_type => value_type,
                                  :path => @hash_file.path)
     values.each do |value|
       @hash[value] = value
@@ -88,7 +89,7 @@ begin
   trie_file = Tempfile.new("groonga-trie")
   item("groonga: Trie: file") do
     @hash = Groonga::PatriciaTrie.create(:key_type => "<shorttext>",
-                                         :value_size => 8,
+                                         :value_type => value_type,
                                          :path => trie_file.path)
     values.each do |value|
       @hash[value] = value

data/benchmark/write-many-small-items.rb

@@ -54,10 +54,11 @@ begin
 
   require 'groonga'
   @database = Groonga::Database.create
+  value_type = Groonga::Type.new("Text8", :size => 8)
 
   item("groonga: Hash: memory") do
     @hash = Groonga::Hash.create(:key_type => "<shorttext>",
-                                 :value_size => 8)
+                                 :value_type => value_type)
     values.each do |value|
       @hash[value] = value
     end
@@ -65,7 +66,7 @@ begin
 
   item("groonga: Trie: memory") do
     @hash = Groonga::PatriciaTrie.create(:key_type => "<shorttext>",
-                                         :value_size => 8)
+                                         :value_type => value_type)
     values.each do |value|
       @hash[value] = value
     end
@@ -74,7 +75,7 @@ begin
   @hash_file = Tempfile.new("groonga-hash")
   item("groonga: Hash: file") do
     @hash = Groonga::Hash.create(:key_type => "<shorttext>",
-                                 :value_size => 8,
+                                 :value_type => value_type,
                                  :path => @hash_file.path)
     values.each do |value|
       @hash[value] = value
@@ -84,7 +85,7 @@ begin
   trie_file = Tempfile.new("groonga-trie")
   item("groonga: Trie: file") do
     @hash = Groonga::PatriciaTrie.create(:key_type => "<shorttext>",
-                                         :value_size => 8,
+                                         :value_type => value_type,
                                          :path => trie_file.path)
     values.each do |value|
       @hash[value] = value

data/example/bookmark.rb

@@ -14,6 +14,12 @@ rescue LoadError
   require "groonga"
 end
 
+require 'time'
+
+# Groonga::Logger.register(:level => :debug) do |level, time, title, message, location|
+#   p [level, time, title, message, location]
+# end
+
 $KCODE = "UTF-8"
 Groonga::Context.default_options = {:encoding => :utf8}
 
@@ -43,10 +49,13 @@ p(terms = Groonga::Hash.create(:name => "<terms>",
                                :default_tokenizer => "<token:bigram>"))
 p terms.define_index_column("item_title", items,
                             :persistent => persistent,
+                            :with_weight => true,
+                            :with_section => true,
+                            :with_position => true,
                             :source => "<items>.title")
 
 p items.find("http://ja.wikipedia.org/wiki/Ruby")["title"] = "Ruby"
-p items.find("http://www.ruby-lang.org/")["title"] = "オブジェクトスクリプト言語Ruby"
+p items.find("http://www.ruby-lang.org/")["title"] = "オブジェクト指向スクリプト言語Ruby"
 
 p(users = Groonga::Hash.create(:name => "<users>",
                                :key_type => "<shorttext>",
@@ -63,6 +72,9 @@ p comments.define_column("issued", "<time>")
 
 p terms.define_index_column("comment_content", comments,
                             :persistent => persistent,
+                            :with_weight => true,
+                            :with_section => true,
+                            :with_position => true,
                             :source => "<comments>.content")
 
 p users.add("moritan", :name => "モリタン")
@@ -96,6 +108,15 @@ p add_bookmark("http://d.hatena.ne.jp/higepon/20070815/1187192864",
 p add_bookmark("http://practical-scheme.net/docs/cont-j.html",
                "なんでも継続", "taporobo", "トランポリン LISP continuation",
                1187568692)
+p add_bookmark("http://jp.rubyist.net/managinze",
+               "るびま", "moritan", "Ruby ドキュメント",
+               Time.now)
+p add_bookmark("http://jp.rubyist.net/managinze",
+               "るびま", "taporobo", "Ruby 雑誌",
+               Time.now)
+p add_bookmark("http://groonga.rubyforge.org/",
+               "ラングバ", "moritan", "Ruby groonga",
+               Time.parse("2009-07-19"))
 
 
 records = comments.select do |record|
@@ -128,3 +149,11 @@ records.group([".item"]).each do |record|
      item.key,
      item[".title"]]
 end
+
+p ruby_comments = @comments.select {|record| record["content"] =~ "Ruby"}
+p ruby_items = @items.select("*W1:50 title:%Ruby")
+
+p ruby_items = ruby_comments.group([".item"]).union!(ruby_items)
+ruby_items.sort([{:key => ".:score", :order => "descending"}]).each do |record|
+  p [record[".:score"], record[".title"]]
+end

data/example/index-html.rb

@@ -0,0 +1,79 @@
+#!/usr/bin/env ruby
+
+if ARGV.size < 2 or ARGV.find {|option| option == "-h" or option == "--help"}
+  puts "Usage: #{$0} DATABASE_FILE [FILE_OR_DIRECTORY ...]"
+  exit
+end
+
+require 'pathname'
+
+base_directory = Pathname(__FILE__).dirname + ".."
+$LOAD_PATH.unshift((base_directory + "ext").to_s)
+$LOAD_PATH.unshift((base_directory + "lib").to_s)
+
+require 'rubygems'
+require 'groonga'
+require 'nokogiri'
+
+database_file, *targets = ARGV
+
+database_file = Pathname(database_file)
+database_directory = database_file.dirname
+database_directory.mkpath unless database_directory.exist?
+
+if database_file.exist?
+  Groonga::Database.open(database_file.to_s)
+else
+  Groonga::Database.create(:path => database_file.to_s)
+  Groonga::Schema.define do |schema|
+    schema.create_table("documents") do |table|
+      table.string("title")
+      table.text("content")
+      table.string("path")
+      table.time("last-modified")
+    end
+
+    schema.create_table("terms",
+                        :type => :patricia_trie,
+                        :key_normalize => true,
+                        :default_tokenizer => "TokenBigram") do |table|
+      table.index("documents.title")
+      table.index("documents.content")
+    end
+  end
+end
+
+documents = Groonga::Context.default["documents"]
+
+targets.each do |target|
+  target = Pathname(target)
+  target.find do |path|
+    throw :prune if path.basename.to_s == ".svn"
+    if path.file? and path.extname == ".html"
+      path.open do |html|
+        values = {:path => path.relative_path_from(target).to_s}
+        _documents = documents.select do |record|
+          record["path"] == values[:path]
+        end
+        if _documents.size.zero?
+          document = documents.add
+        else
+          document = _documents.to_a[0].key
+        end
+
+        html_document = Nokogiri::HTML(html)
+        html_document.css("title").each do |title|
+          values[:title] = title.text
+        end
+        html_document.css("body").each do |body|
+          values[:content] = body.text
+        end
+        values["last-modified"] = path.mtime
+
+        values.each do |key, value|
+          document[key] = value
+        end
+      end
+    end
+  end
+end

data/example/search/config.ru

@@ -0,0 +1,182 @@
+# -*- mode: ruby; coding: utf-8 -*-
+
+base_dir = File.join(File.dirname(__FILE__), "..", "..")
+$LOAD_PATH.unshift(File.join(base_dir, "ext"))
+$LOAD_PATH.unshift(File.join(base_dir, "lib"))
+
+require 'rubygems'
+require 'rack'
+require 'groonga'
+
+use Rack::CommonLogger
+use Rack::Static, :urls => ["/css", "/images"], :root => "public"
+
+Groonga::Database.new("data/database")
+
+class Searcher
+  include Rack::Utils
+
+  def initialize
+    @documents = Groonga::Context.default["documents"]
+  end
+
+  def call(env)
+    request = Rack::Request.new(env)
+    response = Rack::Response.new
+    response["Content-Type"] = "text/html"
+
+    response.write(<<-EOH)
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja" lang="ja">
+<head>
+  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
+  <meta name="robot" content="noindex,nofollow" />
+  <title>groongaで全文検索</title>
+  <link rel="stylesheet" href="css/groonga.css" type="text/css" media="all" />
+</head>
+<body>
+<div class="header">
+  <h1>groongaで全文検索</h1>
+</div>
+
+<div class="content">
+EOH
+
+    render_search_box(request, response)
+    render_search_result(request, response)
+
+    response.write(<<-EOF)
+</div>
+
+<div class="footer">
+<p class="powered-by-groonga">
+  <a href="http://groonga.org/">groonga</a>
+  #{Groonga::VERSION.join('.')}を使っています。
+</p>
+<p class="powered-by-ruby-groonga">
+  <a href="http://groonga.rubyforge.org/">Ruby/groonga</a>
+  #{Groonga::BINDINGS_VERSION.join('.')}を使っています。
+</p>
+</div>
+
+</body>
+</html>
+EOF
+    response.to_a
+  end
+
+  private
+  def query(request)
+    request['query'] || ''
+  end
+
+  def words(request)
+    query(request).split
+  end
+
+  def render_search_box(request, response)
+    response.write(<<-EOF)
+<form method="get" action="./">
+  <p>
+    <a href="."><img src="images/mini-groonga.png" alt="groonga" /></a>
+    <input name="query" type="text" value="#{escape_html(query(request))}" />
+    <input type="submit" value="検索" />
+  </p>
+</form>
+EOF
+  end
+
+  def render_search_result(request, response)
+    _words = words(request)
+    if _words.empty?
+      records = []
+      response.write(<<-EOS)
+  <div class='search-summary'>
+    <p>Rubyでgroonga使って全文検索</p>
+  </div>
+EOS
+    else
+      offset = 0
+      options = {}
+      before = Time.now
+      records = @documents.select do |record|
+        expression = nil
+        _words.each do |word|
+          sub_expression = record["content"] =~ word
+          if expression.nil?
+            expression = sub_expression
+          else
+            expression &= sub_expression
+          end
+        end
+        expression
+      end
+      total_records = records.size
+      records = records.sort([[".:score", "descending"],
+                              [".last-modified", "descending"]],
+                             :limit => 20)
+      elapsed = Time.now - before
+
+      response.write(<<-EOS)
+  <div class='search-summary'>
+    <p>
+      <span class="keyword">#{escape_html(query(request))}</span>の検索結果:
+      <span class="total-entries">#{total_records}</span>件中
+      <span class="display-range">
+        #{total_records.zero? ? 0 : offset + 1}
+        -
+        #{offset + records.size}
+      </span>
+      件（#{elapsed}秒）
+    </p>
+  </div>
+EOS
+    end
+
+    response.write("  <div class='records'>\n")
+    records.each do |record|
+      render_record(request, response, record)
+    end
+    response.write("  </div>\n")
+  end
+
+  def render_record(request, response, record)
+    response.write("    <div class='record'>\n")
+    href = escape_html(record['.path'])
+    title = escape_html(record['.title'])
+    last_modified = escape_html(record['.last-modified'].iso8601)
+    score = record.score
+    response.write("      <h2><a href='#{href}'>#{title}</a>(#{score})</h2>\n")
+    render_snippet(request, response, record)
+    response.write(<<-EOM)
+      <p class="metadata">
+        <span class="url">#{unescape(href)}</span>
+        -
+        <span class="last-modified">#{last_modified}</span>
+      </p>
+EOM
+    response.write("    </div>\n")
+  end
+
+  def render_snippet(request, response, record)
+    open_tag = "<span class=\"keyword\">"
+    close_tag = "</span>"
+    snippet = Groonga::Snippet.new(:width => 100,
+                                   :default_open_tag => open_tag,
+                                   :default_close_tag => close_tag,
+                                   :html_escape => true,
+                                   :normalize => true)
+    words(request).each do |word|
+      snippet.add_keyword(word)
+    end
+    separator = "\n<span class='separator'>...</span>\n"
+    response.write(<<-EOS)
+      <p class="snippet">
+        #{snippet.execute(record[".content"]).join(separator)}
+      </p>
+EOS
+  end
+end
+
+run Searcher.new

data/example/search/public/css/groonga.css

@@ -0,0 +1,122 @@
+body
+{
+  background-color: #fff;
+  color: #000;
+}
+
+div.header
+{
+  display: none;
+}
+
+div.search-box
+{
+  width: 100%;
+  text-align: center;
+}
+
+div.re-search
+{
+}
+
+div.search-summary
+{
+  font-size: small;
+  border-top: 1px solid #fc3;
+  background-color: #ffd;
+}
+
+div.search-summary p
+{
+  padding: 3px;
+  margin: 0;
+}
+
+div.records
+{
+  font-size: smaller;
+  padding-top: 10px;
+  padding-left: 30px;
+  margin-right: 20%;
+  margin-bottom: 30px;
+}
+
+div.records h2
+{
+  padding: 0;
+  margin-bottom: 5px;
+}
+
+div.records p
+{
+}
+
+p.snippet
+{
+  padding: 0;
+  margin: 0;
+}
+
+p.snippet span.separator
+{
+  font-weight: bold;
+}
+
+span.keyword,
+span.total-entries,
+span.display-range
+{
+  font-weight: bold;
+}
+
+div.pagination
+{
+  text-align: center;
+  margin-bottom: 20px;
+}
+
+p.metadata
+{
+  padding: 0;
+  margin: 0;
+  margin-top: 5px;
+}
+
+span.url
+{
+  color: green;
+}
+
+pre
+{
+  overflow: auto;
+  margin: 20px 30px;
+  padding: 10px;
+  border-top: 1px solid #fc3;
+  background-color: #ffd;
+}
+
+a img
+{
+  border-style: none;
+}
+
+ul.sources li
+{
+  display: inline;
+}
+
+div.footer
+{
+  border-top: 1px solid #fc3;
+  background-color: #ffd;
+  margin-top: 5px;
+  text-align: right;
+}
+
+div.footer p
+{
+  font-size: small;
+  padding: 0;
+  margin: 0;
+}