natto 0.9.3 → 0.9.4

data/CHANGELOG

@@ -1,7 +1,18 @@
 ## CHANGELOG
 
-- __2012/02/xx__: 0.9.3 release.
-    - Updating structures, tests following the latest release of MeCab 0.993a
+- __2012/02/26__: 0.9.4 release.
+    - Exposing the underlying FFI pointer as @tagger in Natto::MeCab
+    - Exposing the underlying FFI pointer as @pointer in Natto::MeCabNode
+    - Fixed node formatting behavior for both regular and N-Best node parsing
+    - Adding is_nor?, is_unk?, is_bos?, is_eos?, is_eon? to Natto::MeCabNode 
+    - Adding is_sysdic?, is_usrdic?, is_unkdic? to Natto::DictionaryInfo 
+    - Adding readnodes to allow for method-chaining on list of parsed nodes
+    - Adding readlines to allow for method-chaining on list of string output
+    - Updating README to include description of DevKit install for RubyInstaller
+    - Refactored the codebase, introducing more tests
+
+- __2012/02/14__: 0.9.3 release.
+    - Updating structures, tests following the latest release of MeCab 0.993
     - Ending support for 0.98 MeCab.
     - Support for MeCab 0.993 in Cygwin is still not confirmed.
 

data/README.md

@@ -2,7 +2,7 @@
 A Tasty Ruby Binding with MeCab
 
 ## What is natto?
-natto combines the [Ruby programming language](http://www.ruby-lang.org/) with [MeCab](http://mecab.sourceforge.net/), the part-of-speech and morphological analyzer for the Japanese language.
+natto combines the [Ruby programming language](http://www.ruby-lang.org/) with [MeCab](http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html), the part-of-speech and morphological analyzer for the Japanese language.
 
 natto is a gem bridging Ruby and MeCab using FFI (foreign function interface). No compilation is necessary, as natto is _not_ a C extension. natto will run on CRuby (mri/yarv) and JRuby (jvm) equally well. natto will also run on Windows, Unix/Linux, and Mac.
 
@@ -15,8 +15,22 @@ natto requires the following:
 -  [ffi _0.6.3 or greater_](http://rubygems.org/gems/ffi)
 -  Ruby _1.8.7 or greater_
 
-## Installation
+## Installation on *NIX/Mac/Cygwin
 Install natto with the following gem command:
+
+    gem install natto
+
+This will automatically install the [ffi](http://rubygems.org/gems/ffi) rubygem, which is what natto uses to bind to the <tt>mecab</tt> library.
+
+## Installation on Windows 
+However, if you are using a CRuby on Windows, then you will first need to install the [RubyInstaller Development Kit (DevKit)](https://github.com/oneclick/rubyinstaller/wiki/Development-Kit), which is a MSYS/MinGW based toolkit than enables your Windows Ruby installation to build many of the native C/C++ extensions available, including <tt>ffi</tt>.
+
+1. Download the latest release for RubyInstaller for Windows platforms and the corresponding DevKit from the [RubyInstaller for Windows downloads page](http://rubyinstaller.org/downloads/).
+2. After installing RubyInstaller for Windows, double-click on the DevKit-tdm installer <tt>.exe</tt>, and expand the contents to an appropriate location, for example <tt>C:\devkit</tt>.
+3. Open a command window under <tt>C:\devkit</tt>, and execute: <tt>ruby dk.rb init</tt>. This will locate all known ruby installations, and add them to <tt>C:\devkit\config.yml</tt>.
+4. Next, execute: <tt>ruby dk.rb install</tt>, which will add the DevKit to all of the installed rubies listed in your <tt>C:\devkit\config.yml</tt>.
+5. Now you should be able to install and build the <tt>ffi</tt> rubygem correctly on your Windows-installed ruby, so you can install <tt>natto with</tt>: 
+
     gem install natto
 
 ## Configuration
@@ -24,12 +38,19 @@ Install natto with the following gem command:
 -  In case of <tt>LoadError</tt>, please set the <tt>MECAB_PATH</tt> environment variable to the exact name/path to your <tt>mecab</tt> library.
 
 e.g., for bash on UNIX/Linux
+
     export MECAB_PATH=/usr/local/lib/libmecab.so
+
 e.g., on Windows
+
     set MECAB_PATH=C:\Program Files\MeCab\bin\libmecab.dll
+
 e.g., for Cygwin
+
     export MECAB_PATH=cygmecab-1
+
 e.g., from within a Ruby program
+
     ENV['MECAB_PATH']=/usr/local/lib/libmecab.so
 
 ## Usage
@@ -38,15 +59,16 @@ e.g., from within a Ruby program
 
     nm = Natto::MeCab.new
     => #<Natto::MeCab:0x28d30748 
-         @ptr=#<FFI::Pointer address=0x28a97d50>, \
+         @tagger=#<FFI::Pointer address=0x28a97d50>, \
          @options={}, \
-         @dicts=[#<Natto::DictionaryInfo:0x28d3061c 
-                 filename="/usr/local/lib/mecab/dic/ipadic/sys.dic", 
-                 charset="utf8">], 
-         @version="0.992">
+         @dicts=[#<Natto::DictionaryInfo:0x28d3061c \
+                 type="0", \
+                 filename="/usr/local/lib/mecab/dic/ipadic/sys.dic", \
+                 charset="utf8">], \
+         @version="0.993">
 
     puts nm.version
-    => "0.992" 
+    => "0.993" 
 
     sysdic = nm.dicts.first
 
@@ -55,22 +77,25 @@ e.g., from within a Ruby program
 
     puts sysdic.charset
     => "utf8" 
-
-    nm.parse('暑い日にはもってこいの一品ですね。') do |n|
+    
+    nm.parse('ピンチの時には必ずヒーローが現れる。') do |n|
       puts "#{n.surface}\t#{n.feature}"
     end
-    暑い    形容詞,自立,*,*,形容詞・アウオ段,基本形,暑い,アツイ,アツイ
-    日      名詞,非自立,副詞可能,*,*,*,日,ヒ,ヒ
-    に      助詞,格助詞,一般,*,*,*,に,ニ,ニ
-    は      助詞,係助詞,*,*,*,*,は,ハ,ワ
-    もってこい 名詞,一般,*,*,*,*,もってこい,モッテコイ,モッテコイ
-    の      助詞,連体化,*,*,*,*,の,ノ,ノ
-    一品    名詞,一般,*,*,*,*,一品,イッピン,イッピン
-    です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
-    ね      助詞,終助詞,*,*,*,*,ね,ネ,ネ
-    。      終助詞記号,句点,*,*,*,*,。,。,。
-            BOS/EOS,*,*,*,*,*,*,*,*
-    => nil
+    ピンチ      名詞,一般,*,*,*,*,ピンチ,ピンチ,ピンチ
+    の          助詞,連体化,*,*,*,*,の,ノ,ノ
+    時          名詞,非自立,副詞可能,*,*,*,時,トキ,トキ 
+    に          助詞,格助詞,一般,*,*,*,に,一般ニ,ニ
+    は          助詞,係助詞,*,*,*,*,は,ハ,ワ
+    必ず        副詞,助詞類接続,*,*,*,*,必ず,カナラズ,カナラズ
+    ヒーロー    名詞,一般,*,*,*,*,ヒーロー,ヒーローー,ヒーロー
+    が          助詞,格助詞,一般,*,*,*,が,ガ,ガ
+    現れる      動詞,自立,*,*,一段,基本形,現れる,アラワレル,アラワレル
+    。         記号,句点,*,*,*,*,。,。,。句点
+               BOS/EOS,*,*,*,*,*,*,*,*
+
+
+## Learn more 
+- You can read more about natto on the [project Wiki](https://bitbucket.org/buruzaemon/natto/wiki/Home).
 
 ## Contributing to natto
 -  Use [mercurial](http://mercurial.selenic.com/) and [check out the latest code at bitbucket](https://bitbucket.org/buruzaemon/natto/src/) to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.

data/lib/natto.rb

@@ -1,13 +1,14 @@
 # coding: utf-8
 require 'rubygems' if RUBY_VERSION.to_f < 1.9
 require 'natto/binding'
+require 'natto/utils'
 
 module Natto 
   require 'ffi'
   require 'optparse'
 
-  # <tt>MeCab</tt> is a wrapper class for the <tt>mecab</tt> parser.
-  # Options to the <tt>mecab</tt> parser are passed in as a string
+  # <tt>MeCab</tt> is a wrapper class for the <tt>mecab</tt> tagger.
+  # Options to the <tt>mecab</tt> tagger are passed in as a string
   # (MeCab command-line style) or as a Ruby-style hash at
   # initialization.
   #
@@ -16,32 +17,42 @@ module Natto
   #     require 'rubygems' if RUBY_VERSION.to_f < 1.9
   #     require 'natto'
   #
-  #     nm = Natto::MeCab.new('-O chasen2')
+  #     nm = Natto::MeCab.new('-Ochasen')
   #     => #<Natto::MeCab:0x28d3bdc8 \
-  #          @ptr=#<FFI::Pointer address=0x28afb980>, \
-  #          @options={:output_format_type=>"chasen2"}, \
-  #          @dicts=[#<Natto::DictionaryInfo:0x289a1f14 \
+  #          @tagger=#<FFI::Pointer address=0x28afb980>, \
+  #          @options={:output_format_type=>"chasen"},  \
+  #          @dicts=[#<Natto::DictionaryInfo:0x289a1f14  \
+  #                    type="0", \
   #                    filename="/usr/local/lib/mecab/dic/ipadic/sys.dic", \
   #                    charset="utf8">], \
   #          @version="0.993">
   #
-  #     nm.parse('ネバネバの組み合わせ美味しいです。') do |n| 
+  #     nm.parse('凡人にしか見えねえ風景ってのがあるんだよ。') do |n| 
   #       puts "#{n.surface}\t#{n.feature}" 
-  #     end
-  #
-  #     ネバネバ        名詞,サ変接続,*,*,*,*,ネバネバ,ネバネバ,ネバネバ
-  #     の              助詞,連体化,*,*,*,*,の,ノ,ノ
-  #     組み合わせ      名詞,一般,*,*,*,*,組み合わせ,クミアワセ,クミアワセ
-  #     美味しい        形容詞,自立,*,*,形容詞・イ段,基本形,美味しい,オイシイ,オイシイ
-  #     です            助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
-  #     。              記号,句点,*,*,*,*,。,。,。
+  #     end 
+  #     凡人   名詞,一般,*,*,*,*,凡人,ボンジン,ボンジン
+  #     に     助詞,格助詞,一般,*,*,*,に,ニ,ニ 
+  #     しか   助詞,係助詞,*,*,*,*,しか,シカ,シカ 
+  #     見え   動詞,自立,*,*,一段,未然形,見える,ミエ,ミエ
+  #     ねえ   助動詞,*,*,*,特殊・ナイ,音便基本形,ない,ネエ,ネー 
+  #     風景   名詞,一般,*,*,*,*,風景,フウケイ,フーケイ
+  #     って   助詞,格助詞,連語,*,*,*,って,ッテ,ッテ
+  #     の     名詞,非自立,一般,*,*,*,の,ノ,ノ 
+  #     が     助詞,格助詞,一般,*,*,*,が,ガ,ガ
+  #     ある   動詞,自立,*,*,五段・ラ行,基本形,ある,アル,アル 
+  #     ん     名詞,非自立,一般,*,*,*,ん,ン,ン
+  #     だ     助動詞,*,*,*一般,特殊・ダ,基本形,だ,ダ,ダ
+  #     よ     助詞,終助詞,*,*,*,*,よ,ã¨,ヨ
+  #     。     記号,句点,*,*,*,*,。,。,。
+  #            BOS/EOS,*,*,*,*,*,*,*,*BOS
   #
   class MeCab
     include Natto::Binding
+    include Natto::Utils
 
-    attr_reader :options, :dicts, :version
+    attr_reader :tagger, :options, :dicts, :version
 
-    # Mapping of mecab short-style configuration options to the <tt>mecab</tt> parser.
+    # Mapping of mecab short-style configuration options to the <tt>mecab</tt> tagger.
     # See the <tt>mecab</tt> help for more details. 
     SUPPORTED_OPTS = { '-r' => :rcfile, 
                        '-d' => :dicdir, 
@@ -69,7 +80,7 @@ module Natto
     # - :rcfile --  resource file
     # - :dicdir --  system dicdir
     # - :userdic --  user dictionary
-    # - :lattice_level --  lattice information level (integer, default 0)
+    # - :lattice_level --  lattice information level (DEPRECATED)
     # - :output_format_type --  output format type (wakati, chasen, yomi, etc.)
     # - :all_morphs --  output all morphs (default false)
     # - :nbest --  output N best results (integer, default 1), requires lattice level >= 1
@@ -90,24 +101,30 @@ module Natto
     # e.g.<br/>
     #
     #     nm = Natto::MeCab.new(:node_format=>'%m¥t%f[7]¥n')
-    #     => #<Natto::MeCab:0x28d2ae10 @ptr=#<FFI::Pointer address=0x28a97980>, \
-    #          @options={:node_format=>"%m¥t%f[7]¥n"}, \
-    #          @dicts=[#<Natto::DictionaryInfo:0x28d2a85c \
+    #     => #<Natto::MeCab:0x28d2ae10 
+    #          @tagger=#<FFI::Pointer address=0x28a97980>, \
+    #          @options={:node_format=>"%m¥t%f[7]¥n"},     \
+    #          @dicts=[#<Natto::DictionaryInfo:0x28d2a85c  \
+    #                    type="0", \
     #                    filename="/usr/local/lib/mecab/dic/ipadic/sys.dic" \
     #                    charset="utf8">], \
     #          @version="0.993">
-    #
-    #     puts nm.parse('簡単で美味しくて良いですよね。')
-    #     簡単       カンタン
-    #     で         デ
-    #     美味しくて オイシクテ
-    #     良い       ヨイ
-    #     です       デス
-    #     よ         ヨ
-    #     ね         ネ
-    #     。
+    # 
+    #     puts nm.parse('才能とは求める人間に与えられるものではない。')
+    #     才能    サイノウ
+    #     と      ト
+    #     は      ハ
+    #     求      モトメル
+    #     人間    ニンゲン
+    #     に      ニ
+    #     与え    アタエ
+    #     られる  ラレル
+    #     もの    モノ
+    #     で      デ
+    #     は      ハ
+    #     ない    ナイ
+    #     。      。
     #     EOS
-    #     => nil
     #
     # @param [Hash or String]
     # @raise [MeCabError] if <tt>mecab</tt> cannot be initialized with the given <tt>options</tt>
@@ -118,48 +135,87 @@ module Natto
       @dicts = []
 
       opt_str = self.class.build_options_str(@options)
-      @ptr = self.mecab_new2(opt_str)
-      raise MeCabError.new("Could not initialize MeCab with options: '#{opt_str}'") if @ptr.address == 0x0
+      @tagger = self.mecab_new2(opt_str)
+      raise MeCabError.new("Could not initialize MeCab with options: '#{opt_str}'") if @tagger.address == 0x0
 
-      # set mecab parsing options
-      self.mecab_set_theta(@ptr, @options[:theta]) if @options[:theta]
-      self.mecab_set_lattice_level(@ptr, @options[:lattice_level]) if @options[:lattice_level]
-      self.mecab_set_all_morphs(@ptr, 1) if @options[:all_morphs]
+      self.mecab_set_theta(@tagger, @options[:theta]) if @options[:theta]
+      self.mecab_set_lattice_level(@tagger, @options[:lattice_level]) if @options[:lattice_level]
+      self.mecab_set_all_morphs(@tagger, 1) if @options[:all_morphs]
        
       # Set mecab parsing implementations for N-best and regular parsing,
       # for both parsing as string and yielding a node object
       # N-Best parsing implementations
       if @options[:nbest] && @options[:nbest] > 1
         # nbest parsing require lattice level >= 1
-        self.mecab_set_lattice_level(@ptr, (@options[:lattice_level] || 1))
-        @parse_tostr = lambda { |str| 
-          self.mecab_nbest_init(@ptr, str) 
-          return self.mecab_nbest_sparse_tostr(@ptr, @options[:nbest], str) || 
-                raise(MeCabError.new(self.mecab_strerror(@ptr))) 
-        } 
-        @parse_tonode = lambda { |str| 
-          self.mecab_nbest_init(@ptr, str) 
-          return self.mecab_nbest_next_tonode(@ptr) 
-        }
+        self.mecab_set_lattice_level(@tagger, (@options[:lattice_level] || 1))
+        @parse_tostr = lambda do |str| 
+          return self.mecab_nbest_sparse_tostr(@tagger, @options[:nbest], str) || 
+                raise(MeCabError.new(self.mecab_strerror(@tagger))) 
+        end 
+        @parse_tonodes = lambda do |str| 
+          nodes = []
+          if @options[:nbest] && @options[:nbest] > 1
+            self.mecab_nbest_init(@tagger, str) 
+            n = self.mecab_nbest_next_tonode(@tagger)
+            raise(MeCabError.new(self.mecab_strerror(@tagger))) if n.nil? || n.address==0x0
+            nlen = @options[:nbest]
+            nlen.times do
+              s = str.bytes.to_a
+              while n && n.address != 0x0
+                mn = Natto::MeCabNode.new(n)
+                if mn.is_nor?
+                  slen, sarr = mn.length, []
+                  slen.times { sarr << s.shift }
+                  surf = sarr.pack('C*')
+                  mn.surface = self.class.force_enc(surf)
+                  if @options[:output_format_type] || @options[:node_format]
+                    mn.feature = self.class.force_enc(self.mecab_format_node(@tagger, n)) 
+                  end
+                end
+                nodes << mn if !mn.is_bos?
+                n = mn.next
+              end
+              n = self.mecab_nbest_next_tonode(@tagger)
+            end
+          end
+          return nodes
+        end
       else
         # default parsing implementations
-        @parse_tostr = lambda { |str|
-          return self.mecab_sparse_tostr(@ptr, str) || raise(MeCabError.new(self.mecab_strerror(@ptr))) 
-        }
-        @parse_tonode = lambda { |str| return self.mecab_sparse_tonode(@ptr, str) }
+        @parse_tostr = lambda do |str|
+          return self.mecab_sparse_tostr(@tagger, str) || 
+                raise(MeCabError.new(self.mecab_strerror(@tagger))) 
+        end
+        @parse_tonodes = lambda do |str| 
+          nodes = []
+          n = self.mecab_sparse_tonode(@tagger, str) 
+          raise(MeCabError.new(self.mecab_strerror(@tagger))) if n.nil? || n.address==0x0
+          mn = Natto::MeCabNode.new(n)
+          n = mn.next if mn.next.address!=0x0 && mn.is_bos?
+          s = str.bytes.to_a
+          while n && n.address!=0x0
+            mn = Natto::MeCabNode.new(n)
+            if mn.is_nor?
+              slen, sarr = mn.length, []
+              slen.times { sarr << s.shift }
+              surf = sarr.pack('C*')
+              mn.surface = self.class.force_enc(surf)
+            end
+            nodes << mn 
+            n = mn.next
+          end
+          return nodes
+        end
       end
 
-      # set ref to dictionaries
-      @dicts << Natto::DictionaryInfo.new(Natto::Binding.mecab_dictionary_info(@ptr))
+      @dicts << Natto::DictionaryInfo.new(Natto::Binding.mecab_dictionary_info(@tagger))
       while @dicts.last.next.address != 0x0
         @dicts << Natto::DictionaryInfo.new(@dicts.last.next)
       end
 
-      # set ref to mecab version string
       @version = self.mecab_version
 
-      # set Proc for freeing mecab pointer
-      ObjectSpace.define_finalizer(self, self.class.create_free_proc(@ptr))
+      ObjectSpace.define_finalizer(self, self.class.create_free_proc(@tagger))
     end
     
     # Parses the given string <tt>str</tt>. If a block is passed to this method,
@@ -167,47 +223,48 @@ module Natto
     #
     # @param [String] str
     # @return parsing result from <tt>mecab</tt>
-    # @raise [MeCabError] if the <tt>mecab</tt> parser cannot parse the given string <tt>str</tt>
+    # @raise [MeCabError] if the <tt>mecab</tt> tagger cannot parse the given string <tt>str</tt>
     # @see MeCabNode
     def parse(str)
       if block_given?
-        m_node_ptr = @parse_tonode.call(str)
-        head = Natto::MeCabNode.new(m_node_ptr) 
-        if head && head[:next].address != 0x0
-          node = Natto::MeCabNode.new(head[:next])
-          i = 0
-          while node.nil? == false
-            if node.length > 0
-              node.surface = str.bytes.to_a()[i, node.length].pack('C*')
-            end
-            yield node
-            if node[:next].address != 0x0
-              i += node.length
-              node = Natto::MeCabNode.new(node[:next])
-            else
-              break
-            end
-          end
-        end
+        nodes = @parse_tonodes.call(str)
+        nodes.each {|n| yield n }
       else
-        result = @parse_tostr.call(str)
-        result.force_encoding(Encoding.default_external) if result.respond_to?(:encoding) && result.encoding!=Encoding.default_external
-        result
+        self.class.force_enc(@parse_tostr.call(str))
       end
     end
 
-    # Returns human-readable details for the wrapped <tt>mecab</tt> parser.
+    # Parses the given string <tt>str</tt>, and returns
+    # a list of <tt>mecab</tt> nodes.
+    # @param [String] str
+    # @return [Array] of parsed <tt>mecab</tt> nodes.
+    # @raise [MeCabError] if the <tt>mecab</tt> tagger cannot parse the given string <tt>str</tt>
+    # @see MeCabNode
+    def readnodes(str)
+      @parse_tonodes.call(str)
+    end
+
+    # Parses the given string <tt>str</tt>, and returns
+    # a list of <tt>mecab</tt> result strings.
+    # @param [String] str
+    # @return [Array] of parsed <tt>mecab</tt> result strings.
+    # @raise [MeCabError] if the <tt>mecab</tt> tagger cannot parse the given string <tt>str</tt>
+    def readlines(str)
+      self.class.force_enc(@parse_tostr.call(str)).lines.to_a
+    end
+
+    # Returns human-readable details for the wrapped <tt>mecab</tt> tagger.
     # Overrides <tt>Object#to_s</tt>.
     #
     # - encoded object id
-    # - FFI pointer to <tt>mecab</tt> object
+    # - underlying FFI pointer to the <tt>mecab</tt> tagger
     # - options hash
     # - list of dictionaries
     # - MeCab version
     #
-    # @return [String] encoded object id, FFI pointer, options hash, list of dictionaries, and MeCab version
+    # @return [String] encoded object id, underlying FFI pointer, options hash, list of dictionaries, and MeCab version
     def to_s
-      %(#{super.chop} @ptr=#{@ptr.to_s}, @options=#{@options.inspect}, @dicts=#{@dicts.to_s}, @version="#{@version.to_s}">)
+      %(#{super.chop} @tagger=#{@tagger}, @options=#{@options.inspect}, @dicts=#{@dicts.to_s}, @version="#{@version.to_s}">)
     end
 
     # Overrides <tt>Object#inspect</tt>.
@@ -284,10 +341,10 @@ module Natto
     end
 
     # Returns a string-representation of the options to
-    # be passed in the construction of <tt>mecab</tt>.
+    # be passed in the construction of the <tt>mecab</tt> tagger.
     #
     # @param [Hash] options 
-    # @return [String] representation of the options to the <tt>mecab</tt> parser
+    # @return [String] representation of the options to the <tt>mecab</tt> tagger
     def self.build_options_str(options={})
       opt = []
       SUPPORTED_OPTS.values.each do |k|
@@ -352,7 +409,16 @@ module Natto
   #
   #     puts sysdic.charset
   #     => "utf8"
+  # 
+  #     puts sysdic.is_sysdic?
+  #     => true
   class DictionaryInfo < MeCabStruct
+    # System dictionary.
+    SYS_DIC = 0
+    # User dictionary.
+    USR_DIC = 1
+    # Unknown dictionary.
+    UNK_DIC = 2
 
     layout  :filename, :string,
             :charset,  :string,
@@ -381,12 +447,13 @@ module Natto
     # Overrides <tt>Object#to_s</tt>.
     #
     # - encoded object id
+    # - dictionary type
     # - full-path dictionary filename
     # - dictionary charset
     #
-    # @return [String] encoded object id, dictionary filename, and charset
+    # @return [String] encoded object id, type, dictionary filename, and charset
     def to_s
-      %(#{super.chop} filename="#{self.filename}", charset="#{self.charset}">)
+      %(#{super.chop} type="#{self.type}", filename="#{self.filename}", charset="#{self.charset}">)
     end
     
     # Overrides <tt>Object#inspect</tt>.
@@ -396,6 +463,24 @@ module Natto
     def inspect
       self.to_s
     end
+
+    # Returns <tt>true</tt> if this is a system dictionary.
+    # @return [Boolean]
+    def is_sysdic?
+      self.type == SYS_DIC
+    end
+
+    # Returns <tt>true</tt> if this is a user dictionary.
+    # @return [Boolean]
+    def is_usrdic?
+      self.type == USR_DIC
+    end
+
+    # Returns <tt>true</tt> if this is a unknown dictionary type.
+    # @return [Boolean]
+    def is_unkdic?
+      self.type == UNK_DIC
+    end
   end
 
   # <tt>MeCabNode</tt> is a wrapper for the structure holding
@@ -430,61 +515,57 @@ module Natto
   # - :cost
   #
   # <h2>Usage</h2>
-  # An instance of <tt>MeCabNode</tt> is yielded to a block
-  # used with <tt>MeCab#parse</tt>. Each resulting node is
-  # yielded to the block passed in, where the above-mentioned
-  # node attributes may be accessed.
+  # An instance of <tt>MeCabNode</tt> is yielded to the block
+  # used with <tt>MeCab#parse</tt>, where the above-mentioned
+  # node attributes may be accessed by name.
   #
   #     nm = Natto::MeCab.new
   #
-  #     nm.parse('めかぶの使い方がわからなくて困ってました。') do |n| 
-  #       puts "#{n.surface}¥t#{n.cost}" 
+  #     nm.parse('卓球なんて死ぬまでの暇つぶしだよ。') do |n| 
+  #       puts "#{n.surface}\t#{n.cost}" if n.is_nor? 
   #     end
-  #
-  #     め      7961
-  #     かぶ    19303
-  #     の      25995
-  #     使い方  29182
-  #     が      28327
-  #     わから  33625
-  #     なく    34256
-  #     て      36454
-  #     困っ    43797
-  #     て      42178
-  #     まし    46708
-  #     た      46111
-  #     。      42677
-  #             41141
-  #     => nil
+  #     卓球     2874
+  #     な       4398
+  #     死ぬ     9261
+  #     まで     9386
+  #     の       10007
+  #     暇つぶし 13324
+  #     だ       15346
+  #     よ       14396
+  #     。       10194
   #
   # It is also possible to use the <tt>Symbol</tt> for the
   # <tt>mecab</tt> node member to index into the 
   # <tt>FFI::Struct</tt> layout associative array like so:
   #     
-  #     nm.parse('納豆に乗っけて頂きます！') {|n| puts n[:feature] }
-  #
-  #     名詞,一般,*,*,*,*,納豆,ナットウ,ナットー
-  #     助詞,格助詞,一般,*,*,*,に,ニ,ニ
-  #     動詞,自立,*,*,一段,連用形,乗っける,ノッケ,ノッケ
-  #     助詞,接続助詞,*,*,*,*,て,テ,テ
-  #     動詞,非自立,*,*,五段・カ行イ音便,連用形,頂く,イタダキ,イタダキ
-  #     助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
-  #     記号,一般,*,*,*,*,！,！,！
+  #     nm.parse('あいつ笑うと結構可愛い顔してんよ。') {|n| puts n[:feature] }
+  #     名詞,代名詞,一般,*,*,*,あいつ,アイツ,アイツ
+  #     動詞,自立,*,*,五段・ワ行促音便,基本形,笑う,ワラウ,ワラウ
+  #     助詞,接続助詞,*,*,*,*,と,ト,ト
+  #     副詞,一般,*,*,*,*,結構,ケッコウ,ケッコー
+  #     形容詞,自立,*,*,形容詞・イ段,基本形,可愛い,カワイイ,カワイイ
+  #     名詞,一般,*,*,*,*,顔,カオ,カオ
+  #     動詞,自立,*,*,サ変・スル,連用形,する,シ,シ
+  #     動詞,非自立,*,*,一段,体言接続特殊,てる,テン,テン
+  #     助詞,終助詞,*,*,*,*,よ,ヨ,ヨ
+  #     記号,句点,*,*,*,*,。,。,。
   #     BOS/EOS,*,*,*,*,*,*,*,*
-  #     => nil
   #
   class MeCabNode < MeCabStruct
+    include Natto::Utils
+
     attr_accessor :surface, :feature
+    attr_reader   :pointer
 
-    # Normal <tt>mecab</tt> node.
+    # Normal <tt>mecab</tt> node defined in the dictionary.
     NOR_NODE = 0
-    # Unknown <tt>mecab</tt> node.
+    # Unknown <tt>mecab</tt> node not defined in the dictionary.
     UNK_NODE = 1
-    # Beginning-of-string <tt>mecab</tt> node.
+    # Virtual node representing the beginning of the sentence.
     BOS_NODE = 2
-    # End-of-string <tt>mecab</tt> node.
+    # Virutual node representing the end of the sentence.
     EOS_NODE = 3
-    # End-of-NBest <tt>mecab</tt> node list.
+    # Virtual node representing the end of an N-Best <tt>mecab</tt> node list.
     EON_NODE = 4
 
     layout  :prev,            :pointer,
@@ -525,39 +606,30 @@ module Natto
     end
 
     # Initializes this node instance.
-    # Sets the <ttMeCab</tt> feature value for this node.
+    # Sets the <tt>MeCab</tt> feature value for this node.
     #
     # @param [FFI::Pointer]
     def initialize(ptr)
       super(ptr)
+      @pointer = ptr
 
       if self[:feature]
-        @feature = self[:feature]
-        @feature.force_encoding(Encoding.default_external) if @feature.respond_to?(:encoding) && @feature.encoding!=Encoding.default_external
+        @feature = self.class.force_enc(self[:feature])
       end
     end
      
-    # Sets the morpheme surface value for this node.
-    #
-    # @param [String] 
-    def surface=(str)
-      if str && self[:length] > 0
-        @surface = str
-        @surface.force_encoding(Encoding.default_external) if @surface.respond_to?(:encoding) && @surface.encoding!=Encoding.default_external
-      end
-    end
-
     # Returns human-readable details for the <tt>mecab</tt> node.
     # Overrides <tt>Object#to_s</tt>.
     #
     # - encoded object id
-    # - stat 
+    # - underlying FFI pointer to MeCab Node 
+    # - stat (node type: NOR, UNK, BOS/EOS, EON)
     # - surface 
     # - feature
     #
-    # @return [String] encoded object id, stat, surface, and feature 
+    # @return [String] encoded object id, underlying FFI pointer, stat, surface, and feature 
     def to_s
-      %(#{super.chop} stat=#{self[:stat]}, surface="#{self.surface}", feature="#{self.feature}">)
+      %(#{super.chop} @pointer=#{@pointer}, stat=#{self[:stat]}, @surface="#{self.surface}", @feature="#{self.feature}">)
     end
 
     # Overrides <tt>Object#inspect</tt>.
@@ -567,5 +639,35 @@ module Natto
     def inspect
       self.to_s
     end
+
+    # Returns <tt>true</tt> if this is a normal <tt>mecab</tt> node found in the dictionary.
+    # @return [Boolean]
+    def is_nor?
+      self.stat == NOR_NODE
+    end
+
+    # Returns <tt>true</tt> if this is an unknown <tt>mecab</tt> node not found in the dictionary.
+    # @return [Boolean]
+    def is_unk?
+      self.stat == UNK_NODE
+    end
+   
+    # Returns <tt>true</tt> if this is a virtual <tt>mecab</tt> node representing the beginning of the sentence.
+    # @return [Boolean]
+    def is_bos?
+      self.stat == BOS_NODE
+    end
+   
+    # Returns <tt>true</tt> if this is a virtual <tt>mecab</tt> node representing the end of the sentence.
+    # @return [Boolean]
+    def is_eos?
+      self.stat == EOS_NODE 
+    end
+   
+    # Returns <tt>true</tt> if this is a virtual <tt>mecab</tt> node representing the end of the node list.
+    # @return [Boolean]
+    def is_eon?
+      self.stat == EON_NODE
+    end
   end
 end