natto 0.9.9 → 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: dda6a3b2c795f26a54307637aeaa14b44f2035f0
+  data.tar.gz: 53b0944ea5822e7e3ab2e38bb07e0bc44d8336de
+SHA512:
+  metadata.gz: aa4d39e5e0b2a11449a55f7c002a257444af14d1e534316ede57f5d2aedb64d95e9b00f5f77396f6701068e4203873856c91fea0fbe07d9ddf949e2913aa5a99
+  data.tar.gz: db01e6275b7348af0c58adcf0f7c7f62a29549bea5cff29c3e15e001550deebd8c10c66d0c88ce2ed6d9715d5d0681c4b2e3a554becd4c8d73f506777fe83287

data/CHANGELOG

@@ -1,5 +1,18 @@
 ## CHANGELOG
 
+- __2015/04/14__: 1.0.0 release.
+    - Issue 36: Fixed @param documentation
+    - Issue 37: README and bullet points under Automatic Configuration
+    - Issue 38: Updated URLs in documentation to point to Ruby 2.2.1
+    - Issue 39: Make refs to MeCab and Tagger consistent in docs
+    - Issue 40: Use new Model- and Lattice-based C APIs internally
+    - Issue 45: Add support for feature constraint parsing
+    - Issue 48: Put in a guard to prevent partial parsing of text that does not end with a new-line char
+    - Issue 50: Update all references to Natto::MeCab in documentation to reflect new internal structure
+    - Issue 52: Downloads and license badges for README.md & API docs
+    - Issue 55: Node parsing with --all-morphs option missing surface values
+
+
 - __2015/03/31__: 0.9.9 release.
     - Issue 21/34: Implemented boundary constraint parsing.
     - Issue 26: Removing deprecated methods parse_as_nodes, parse_as_strings, readnodes and readlines.

data/README.md

@@ -1,4 +1,4 @@
-# natto [![Gem Version](https://badge.fury.io/rb/natto.svg)](http://badge.fury.io/rb/natto) [![Build Status](https://travis-ci.org/buruzaemon/natto.svg?branch=master)](https://travis-ci.org/buruzaemon/natto) 
+# natto [![Gem Version](https://badge.fury.io/rb/natto.svg)](https://rubygems.org/gems/natto) [![Build Status](https://travis-ci.org/buruzaemon/natto.svg?branch=master)](https://travis-ci.org/buruzaemon/natto) [![Gem Downloads](https://img.shields.io/gem/dt/natto.svg)](https://rubygems.org/gems/natto) [![Gem License](https://img.shields.io/badge/license-BSD-blue.svg)]() 
 A Tasty Ruby Binding with MeCab
 
 ## What is natto?
@@ -47,6 +47,7 @@ However, if you are using a CRuby on Windows, then you will first need to instal
 
 ## Automatic Configuration
 No explicit configuration should be necessary, as natto will try to locate the MeCab library based upon its runtime environment.
+
 - On OS X and \*nix, it will query `mecab-config --libs` 
 - On Windows, it will query the Windows Registry to determine where `libmecab.dll` is installed
 
@@ -78,14 +79,16 @@ Instantiate a reference to the MeCab library, and display some details:
     require 'natto'
 
     nm = Natto::MeCab.new
-    => #<Natto::MeCab:0x28d30748 
-         @tagger=#<FFI::Pointer address=0x28a97d50>, \
-         @libpath="/usr/local/lib/libmecab.so", \
-         @options={}, \
-         @dicts=[#<Natto::DictionaryInfo:0x28d3061c \
+    => #<Natto::MeCab:0x00000803633ae8
+         @model=#<FFI::Pointer address=0x000008035d4640>,             \
+         @tagger=#<FFI::Pointer address=0x00000802b07c90>,            \
+         @lattice=#<FFI::Pointer address=0x00000803602f80>,           \
+         @libpath="/usr/local/lib/libmecab.so",                       \
+         @options={},                                                 \
+         @dicts=[#<Natto::DictionaryInfo:0x000008036337c8             \
                  @filepath="/usr/local/lib/mecab/dic/ipadic/sys.dic", \
-                 charset=utf8, \
-                 type=0>] \
+                 charset=utf8,                                        \
+                 type=0>]                                             \
          @version=0.996>
     
     puts nm.version
@@ -167,7 +170,7 @@ nodes:
 
 For more complex parsing, such as that for natural language
 processing tasks, it is far more efficient to use `enum_parse` to
-obtain an [`Enumerator`](http://ruby-doc.org/core-2.2.0/Enumerator.html)
+obtain an [`Enumerator`](http://ruby-doc.org/core-2.2.1/Enumerator.html)
 to iterate over the resulting `MeCabNode` instances. An `Enumerator`
 yields each `MeCabNode` instance without first materializing all
 instances at once, thus being more efficient.
@@ -226,11 +229,13 @@ back to the beginning, and then iterate over it.
 ----
 
 [Partial parsing](http://taku910.github.io/mecab/partial.html) allows you to
-pass hints to MeCab on how to tokenize morphemes when parsing. With boundary
-constraint parsing, you can specify either
+pass hints to MeCab on how to tokenize morphemes when parsing. Most useful are
+boundary constraint parsing and feature constraint parsing.
+
+With boundary constraint parsing, you can specify either
 a [Regexp](http://ruby-doc.org/core-2.2.1/Regexp.html) or
 [String](http://ruby-doc.org/core-2.2.1/String.html) to tell MeCab where the
-boundaries of a morpheme should be. Use the new `boundary_constraints` keyword.
+boundaries of a morpheme should be. Use the `boundary_constraints` keyword.
 For hints on tokenization, please see
 [String#scan](http://ruby-doc.org/core-2.2.1/String.html#method-i-scan)
 
@@ -242,6 +247,7 @@ the resulting `MeCabNode` feature attribute to extract:
 - `%s` - node `stat` status value, 1 is `unknown`
 
 Note that any such morphemes captured will have node `stat` status of unknown.
+Also note that MeCab will tag such nodes as a noun.
 
     nm = Natto::MeCab.new('-F%m,\s%f[0],\s%s')
 
@@ -268,6 +274,37 @@ Note that any such morphemes captured will have node `stat` status of unknown.
     ヒーロー見参, 名詞, 1
     ！, 記号, 0
 
+With feature constraint parsing, you can provide instructions to MeCab on
+what feature to use for a matching morpheme. Use the `feature_constraints`
+keyword to pass in a hash mapping a specific morpheme key (String)
+to a corresponding feature (String).
+
+    # we re-use nm and text from above
+
+    nm.options
+    => {:node_format=>"%m,\\s%f[0],\\s%s"}
+
+    mapping = {"ヒーロー見参"=>"その他"}
+
+    nm.enum_parse(text, feature_constraints: mapping).each do |n|
+      puts n.feature if !(n.is_bos? || n.is_eos?)
+    end
+
+    # ヒーロー見参 will be treated as a single morpheme mapping to その他 
+    心, 名詞, 0
+    の, 助詞, 0
+    中, 名詞, 0
+    で, 助詞, 0
+    3, 名詞, 1
+    回, 名詞, 0
+    唱え, 動詞, 0
+    、, 記号, 0
+    ヒーロー見参, その他, 1
+    ！, 記号, 0
+    ヒーロー見参, その他, 1
+    ！, 記号, 0
+    ヒーロー見参, その他, 1
+    ！, 記号, 0
 
 
 ## Learn more 

data/lib/natto.rb

@@ -1,3 +1,4 @@
+# coding: utf-8
 require 'natto/natto'
 
 # Copyright (c) 2015, Brooke M. Fujita.

data/lib/natto/binding.rb

@@ -2,16 +2,12 @@
 module Natto
 
   # Module `Binding` encapsulates methods and behavior 
-  # which are made available via `FFI` bindings to 
-  # `mecab`.
+  # which are made available via `FFI` bindings to MeCab.
   module Binding
     require 'ffi'
     require 'rbconfig'
     extend FFI::Library
 
-    # String name for the environment variable used by 
-    # `Natto` to indicate the absolute pathname
-    # to the `mecab` library.
     MECAB_PATH = 'MECAB_PATH'.freeze
     
     # @private
@@ -19,89 +15,91 @@ module Natto
       base.extend(ClassMethods)
     end
 
-    # Returns the absolute pathname to the `mecab` library based on 
+    # Returns the absolute pathname to the MeCab library based on 
     # the runtime environment.
-    # 
-    # @return [String] absolute pathname to the `mecab` library
+    # @return [String] absolute pathname to the MeCab library
     # @raise [LoadError] if the library cannot be located
     def self.find_library
-      return File.absolute_path(ENV[MECAB_PATH]) if ENV[MECAB_PATH]
-
-      host_os = RbConfig::CONFIG['host_os']
-
-      if host_os =~ /mswin|mingw/i
-        require 'win32/registry'
-        begin
-          base = nil
-          Win32::Registry::HKEY_CURRENT_USER.open('Software\MeCab') do |r|
-            base = r['mecabrc'].split('etc').first
-          end
-          lib = File.join(base, 'bin/libmecab.dll')
-          File.absolute_path(lib)
-        rescue
-          raise LoadError, "Please set #{MECAB_PATH} to the full path to libmecab.dll"
-        end
+      if ENV[MECAB_PATH]
+        File.absolute_path(ENV[MECAB_PATH])
       else
-        require 'open3'
-        if host_os =~ /darwin/i
-          ext = 'dylib'
+        host_os = RbConfig::CONFIG['host_os']
+
+        if host_os =~ /mswin|mingw/i
+          require 'win32/registry'
+          begin
+            base = nil
+            Win32::Registry::HKEY_CURRENT_USER.open('Software\MeCab') do |r|
+              base = r['mecabrc'].split('etc').first
+            end
+            lib = File.join(base, 'bin/libmecab.dll')
+            File.absolute_path(lib)
+          rescue
+            raise LoadError, "Please set #{MECAB_PATH} to the full path to libmecab.dll"
+          end
         else
-          ext = 'so'
-        end
+          require 'open3'
+          if host_os =~ /darwin/i
+            ext = 'dylib'
+          else
+            ext = 'so'
+          end
 
-        begin
-          base, lib = nil, nil
-          cmd = 'mecab-config --libs'
-          Open3.popen3(cmd) do |stdin,stdout,stderr|
-            toks = stdout.read.split
-            base = toks[0][2..-1]
-            lib  = toks[1][2..-1]
+          begin
+            base, lib = nil, nil
+            cmd = 'mecab-config --libs'
+            Open3.popen3(cmd) do |stdin,stdout,stderr|
+              toks = stdout.read.split
+              base = toks[0][2..-1]
+              lib  = toks[1][2..-1]
+            end
+            File.absolute_path(File.join(base, "lib#{lib}.#{ext}"))
+          rescue
+            raise LoadError, "Please set #{MECAB_PATH} to the full path to libmecab.#{ext}"
           end
-          File.absolute_path(File.join(base, "lib#{lib}.#{ext}"))
-        rescue
-          raise LoadError, "Please set #{MECAB_PATH} to the full path to libmecab.#{ext}"
         end
       end
     end
 
     ffi_lib find_library
 
-    # C interface
-    attach_function :mecab_new2, [:string], :pointer
+    # Model interface
+    attach_function :mecab_model_new2, [:string], :pointer
+    attach_function :mecab_model_destroy, [:pointer], :void
+    attach_function :mecab_model_new_tagger, [:pointer], :pointer
+    attach_function :mecab_model_new_lattice, [:pointer], :pointer
+    attach_function :mecab_model_dictionary_info, [:pointer], :pointer
+
+    # Tagger interface
+    attach_function :mecab_destroy, [:pointer], :void
     attach_function :mecab_version, [], :string
     attach_function :mecab_strerror, [:pointer],:string
-    attach_function :mecab_destroy, [:pointer], :void
-    attach_function :mecab_set_partial, [:pointer, :int], :void 
-    attach_function :mecab_set_theta, [:pointer, :float], :void
-    attach_function :mecab_set_lattice_level, [:pointer, :int], :void 
-    attach_function :mecab_set_all_morphs, [:pointer, :int], :void
-    attach_function :mecab_sparse_tostr, [:pointer, :string], :string
-    attach_function :mecab_sparse_tonode, [:pointer, :string], :pointer
-    attach_function :mecab_nbest_init, [:pointer, :string], :int
-    attach_function :mecab_nbest_sparse_tostr, [:pointer, :int, :string], :string
-    attach_function :mecab_nbest_next_tonode, [:pointer], :pointer
     attach_function :mecab_format_node, [:pointer, :pointer], :string
-    attach_function :mecab_dictionary_info, [:pointer], :pointer
 
-    attach_function :mecab_lattice_new, [], :pointer
+    # Lattice interface
     attach_function :mecab_lattice_destroy, [:pointer], :void
     attach_function :mecab_lattice_clear, [:pointer], :void
     attach_function :mecab_lattice_is_available, [:pointer], :int
-    attach_function :mecab_lattice_get_bos_node, [:pointer], :pointer
+    attach_function :mecab_lattice_strerror, [:pointer], :string
+
+    attach_function :mecab_lattice_get_sentence, [:pointer], :string
     attach_function :mecab_lattice_set_sentence, [:pointer, :string], :void
     attach_function :mecab_lattice_get_size, [:pointer], :int
-    attach_function :mecab_lattice_set_z, [:pointer, :float], :void
     attach_function :mecab_lattice_set_theta, [:pointer, :float], :void
-    attach_function :mecab_lattice_next, [:pointer], :int
+    attach_function :mecab_lattice_set_z, [:pointer, :float], :void
     attach_function :mecab_lattice_get_request_type, [:pointer], :int
     attach_function :mecab_lattice_add_request_type, [:pointer, :int], :void
     attach_function :mecab_lattice_set_request_type, [:pointer, :int], :void
-    attach_function :mecab_lattice_tostr, [:pointer], :string
-    attach_function :mecab_lattice_nbest_tostr, [:pointer, :int], :string
     attach_function :mecab_lattice_get_boundary_constraint, [:pointer, :int], :int
     attach_function :mecab_lattice_set_boundary_constraint, [:pointer, :int, :int], :void
+    attach_function :mecab_lattice_get_feature_constraint, [:pointer, :int], :string
+    attach_function :mecab_lattice_set_feature_constraint, [:pointer, :int, :int, :string], :void
+
     attach_function :mecab_parse_lattice, [:pointer, :pointer], :int
-    attach_function :mecab_lattice_strerror, [:pointer], :string
+    attach_function :mecab_lattice_next, [:pointer], :int
+    attach_function :mecab_lattice_tostr, [:pointer], :string
+    attach_function :mecab_lattice_nbest_tostr, [:pointer, :int], :string
+    attach_function :mecab_lattice_get_bos_node, [:pointer], :pointer
 
     # @private
     module ClassMethods
@@ -110,75 +108,45 @@ module Natto
         Natto::Binding.find_library
       end
 
-      # ----------------------------------------
-      def mecab_new2(options_str)
-        Natto::Binding.mecab_new2(options_str)
-      end
-      
-      def mecab_version
-        Natto::Binding.mecab_version
-      end
-
-      def mecab_strerror(tptr)
-        Natto::Binding.mecab_strerror(tptr)
+      # Model interface ------------------------
+      def mecab_model_new2(opts_str)
+        Natto::Binding.mecab_model_new2(opts_str)
       end
 
-      def mecab_destroy(tptr)
-        Natto::Binding.mecab_destroy(tptr)
+      def mecab_model_destroy(mptr)
+        Natto::Binding.mecab_model_destroy(mptr)
       end
 
-      def mecab_set_partial(tptr, ll)
-        Natto::Binding.mecab_set_partial(tptr, ll)
-      end
-      
-      def mecab_set_theta(tptr, t)
-        Natto::Binding.mecab_set_theta(tptr, t)
+      def mecab_model_new_tagger(mptr)
+        Natto::Binding.mecab_model_new_tagger(mptr)
       end
 
-      def mecab_set_lattice_level(tptr, ll)
-        Natto::Binding.mecab_set_lattice_level(tptr, ll)
-      end
-      
-      def mecab_set_all_morphs(tptr, am)
-        Natto::Binding.mecab_set_all_morphs(tptr, am)
+      def mecab_model_new_lattice(mptr)
+        Natto::Binding.mecab_model_new_lattice(mptr)
       end
 
-      def mecab_sparse_tostr(tptr, str)
-        Natto::Binding.mecab_sparse_tostr(tptr, str)
-      end
-      
-      def mecab_sparse_tonode(tptr, str)
-        Natto::Binding.mecab_sparse_tonode(tptr, str)
-      end
-
-      def mecab_nbest_next_tonode(tptr)
-        Natto::Binding.mecab_nbest_next_tonode(tptr)
+      def mecab_model_dictionary_info(mptr)
+        Natto::Binding.mecab_model_dictionary_info(mptr)
       end
 
-      def mecab_nbest_init(tptr, str)
-        Natto::Binding.mecab_nbest_init(tptr, str)
+      # Tagger interface -----------------------
+      def mecab_destroy(tptr)
+        Natto::Binding.mecab_destroy(tptr)
       end
 
-      def mecab_nbest_sparse_tostr(tptr, n, str)
-        Natto::Binding.mecab_nbest_sparse_tostr(tptr, n, str)
+      def mecab_version
+        Natto::Binding.mecab_version
       end
 
-      def mecab_nbest_next_tonode(tptr)
-        Natto::Binding.mecab_nbest_next_tonode(tptr)
+      def mecab_strerror(tptr)
+        Natto::Binding.mecab_strerror(tptr)
       end
 
       def mecab_format_node(tptr, nptr)
         Natto::Binding.mecab_format_node(tptr, nptr)
       end
-      
-      def mecab_dictionary_info(tptr)
-        Natto::Binding.mecab_dictionary_info(tptr)
-      end
-      
-      def mecab_lattice_new()
-        Natto::Binding.mecab_lattice_new()
-      end
 
+      # Lattice interface ----------------------
       def mecab_lattice_destroy(lptr)
         Natto::Binding.mecab_lattice_destroy(lptr)
       end
@@ -191,10 +159,14 @@ module Natto
         Natto::Binding.mecab_lattice_is_available(lptr)
       end
 
-      def mecab_lattice_get_bos_node(lptr)
-        Natto::Binding.mecab_lattice_get_bos_node(lptr)
+      def mecab_lattice_strerror(lptr)
+        Natto::Binding.mecab_lattice_strerror(lptr)
       end
-    
+
+      def mecab_lattice_get_sentence(lptr)
+        Natto::Binding.mecab_lattice_get_sentence(lptr)
+      end
+
       def mecab_lattice_set_sentence(lptr, str)
         Natto::Binding.mecab_lattice_set_sentence(lptr, str)
       end
@@ -202,39 +174,27 @@ module Natto
       def mecab_lattice_get_size(lptr)
         Natto::Binding.mecab_lattice_get_size(lptr)
       end
-    
-      def mecab_lattice_set_z(lptr, z)
-        Natto::Binding.mecab_lattice_set_z(lptr, z)
-      end
-
+   
       def mecab_lattice_set_theta(lptr, t)
         Natto::Binding.mecab_lattice_set_theta(lptr, t)
       end
 
-      def mecab_lattice_next(lptr)
-        Natto::Binding.mecab_lattice_next(lptr)
+      def mecab_lattice_set_z(lptr, z)
+        Natto::Binding.mecab_lattice_set_z(lptr, z)
       end
- 
+
       def mecab_lattice_get_request_type(lptr)
         Natto::Binding.mecab_lattice_get_request_type(lptr)
       end
       
-      def mecab_lattice_add_request_type(lptr, rtype)
-        Natto::Binding.mecab_lattice_add_request_type(lptr, rtype)
+      def mecab_lattice_add_request_type(lptr, rt)
+        Natto::Binding.mecab_lattice_add_request_type(lptr, rt)
       end
 
       def mecab_lattice_set_request_type(lptr, rtype)
         Natto::Binding.mecab_lattice_set_request_type(lptr, rtype)
       end
 
-      def mecab_lattice_tostr(lptr)
-        Natto::Binding.mecab_lattice_tostr(lptr)
-      end
-
-      def mecab_lattice_nbest_tostr(lptr, n)
-        Natto::Binding.mecab_lattice_nbest_tostr(lptr, n)
-      end
-
       def mecab_lattice_get_boundary_constraint(lptr, pos)
         Natto::Binding.mecab_lattice_get_boundary_constraint(lptr, pos)
       end
@@ -243,12 +203,35 @@ module Natto
         Natto::Binding.mecab_lattice_set_boundary_constraint(lptr, pos, btype)
       end
 
+      def mecab_lattice_get_feature_constraint(lptr, bpos)
+        Natto::Binding.mecab_lattice_get_feature_constraint(lptr, bpos)
+      end
+
+      def mecab_lattice_set_feature_constraint(lptr, bpos, epos, feat)
+        Natto::Binding.mecab_lattice_set_feature_constraint(lptr, 
+                                                            bpos,
+                                                            epos,
+                                                            feat)
+      end
+
       def mecab_parse_lattice(tptr, lptr)
         Natto::Binding.mecab_parse_lattice(tptr, lptr)
       end
 
-      def mecab_lattice_strerror(lptr)
-        Natto::Binding.mecab_lattice_strerror(lptr)
+      def mecab_lattice_next(lptr)
+        Natto::Binding.mecab_lattice_next(lptr)
+      end
+ 
+      def mecab_lattice_tostr(lptr)
+        Natto::Binding.mecab_lattice_tostr(lptr)
+      end
+
+      def mecab_lattice_nbest_tostr(lptr, n)
+        Natto::Binding.mecab_lattice_nbest_tostr(lptr, n)
+      end
+
+      def mecab_lattice_get_bos_node(lptr)
+        Natto::Binding.mecab_lattice_get_bos_node(lptr)
       end
     end
   end