rmmseg-cpp-traditional 0.0.2

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in rmmseg-cpp-traditional.gemspec
+gemspec

data/History.txt

@@ -0,0 +1,21 @@
+== 0.2.9 / 2011-09-10
+
+* Fix GC-related bugs in Ruby C extension.
+
+== 0.2.8 / 2010-03-22
+
+* Minor release, fixed building bugs in Ruby 1.9.
+
+== 0.2.7 / 2008-09-17
+
+* Fix various stupid bugs (typo) that cause problems under MacOSX.
+
+== 0.2.6 / 2008-08-14
+
+* Fix the problem that hang up when required (OS: MacOSX-Leopard)
+
+== 0.2.5 / 2008-06-07
+
+* Created homepage on Rubyforge.
+* Gem can be installed directly from Rubyforge now.
+* Added rdoc for native extensions.

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 MichaelHsu
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Manifest.txt

@@ -0,0 +1,43 @@
+History.txt
+Manifest.txt
+README
+Rakefile
+bin/rmmseg
+data/chars.dic
+data/words.dic
+ext/rmmseg/algor.cpp
+ext/rmmseg/algor.h
+ext/rmmseg/chunk.h
+ext/rmmseg/dict.cpp
+ext/rmmseg/dict.h
+ext/rmmseg/extconf.rb
+ext/rmmseg/memory.cpp
+ext/rmmseg/memory.h
+ext/rmmseg/rmmseg.cpp
+ext/rmmseg/rules.h
+ext/rmmseg/token.h
+ext/rmmseg/word.h
+lib/rmmseg.rb
+lib/rmmseg/dictionary.rb
+lib/rmmseg/ferret.rb
+misc/convert.rb
+misc/ferret_example.rb
+misc/homepage.erb
+misc/homepage.html
+spec/rmmseg_spec.rb
+spec/spec_helper.rb
+tasks/ann.rake
+tasks/bones.rake
+tasks/gem.rake
+tasks/git.rake
+tasks/homepage.rake
+tasks/manifest.rake
+tasks/notes.rake
+tasks/post_load.rake
+tasks/rdoc.rake
+tasks/rubyforge.rake
+tasks/setup.rb
+tasks/spec.rake
+tasks/svn.rake
+tasks/test.rake
+test/test_rmmseg.rb

data/README

@@ -0,0 +1,111 @@
+rmmseg-cpp
+    by pluskid
+    http://rmmseg-cpp.rubyforge.org
+
+== DESCRIPTION:
+
+rmmseg-cpp is a high performance Chinese word segmentation utility for
+Ruby. It features full "Ferret":http://ferret.davebalmain.com/ integration
+as well as support for normal Ruby program usage.
+
+rmmseg-cpp is a re-written of the original
+RMMSeg(http://rmmseg.rubyforge.org/) gem in C++. RMMSeg is written
+in pure Ruby. Though I tried hard to tweak RMMSeg, it just consumes
+lots of memory and the segmenting process is rather slow.
+
+The interface is almost identical to RMMSeg but the performance is
+much better. This gem is always preferable in production
+use. However, if you want to understand how the MMSEG segmenting
+algorithm works, the source code of RMMSeg is a better choice than
+this.
+
+== FEATURES/PROBLEMS:
+
+* Runs fast and the memory consumption is small.
+* Support user customized dictionaries.
+* Easy Ferret integration.
+
+== SYNOPSIS:
+
+=== A simple script
+
+rmmseg-cpp provides a simple script, which can read the text from
+standard input and print the segmented result to standard output. Try
+<tt>rmmseg -h</tt> for help on the options.
+
+=== In a normal Ruby program
+
+To use rmmseg-cpp in normal Ruby program, first load the package and
+init by loading the dictionaries:
+
+  require 'rubygems'
+  require 'rmmseg'
+
+  RMMSeg::Dictionary.load_dictionaries
+
+If you want to add customized dictionaries, append them to 
++RMMSeg::Dictionary.dictionaries+ before calling +load_dictionaries+.
+The formats of chars.dic and  words.dic are NOT the same:
+
+* For chars.dic, each line contains freq, a space, and then the character
+* For words.dic, each line contains length, a space, and then the word.
+
+Note length mean the length of the word, i.e. the number of characters
+of the word, not number of bytes. WARNING: there should be a newline at
+the end of every dictionary file.
+
+Then create a +Algorithm+ object and call +next_token+ until got a
++nil+:
+
+  algor = RMMSeg::Algorithm.new(text)
+  loop do
+    tok = algor.next_token
+    break if tok.nil?
+    puts "#{tok.text} [#{tok.start}..#{tok.end}]"
+  end
+
+=== With Ferret
+
+To use rmmseg-cpp with Ferret, just use the analyzer provided:
+
+  analyzer = RMMSeg::Ferret::Analyzer.new { |tokenizer|
+    Ferret::Analysis::LowerCaseFilter.new(tokenizer)
+  }
+
+  index = Ferret::Index::Index.new(:analyzer => analyzer)
+
+See <tt>misc/ferret_example.rb</tt> for a complete example.
+
+== REQUIREMENTS:
+
+* ruby 1.8.x
+* g++
+
+== INSTALL:
+
+* sudo gem install rmmseg-cpp
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2008 FIXME (different license?)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,29 @@
+# Rmmseg::Cpp::Traditional
+
+TODO: Write a gem description
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'rmmseg-cpp-traditional'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install rmmseg-cpp-traditional
+
+## Usage
+
+TODO: Write usage instructions here
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,19 @@
+# -*- Ruby -*-
+
+load 'tasks/setup.rb'
+
+ensure_in_path 'lib'
+require 'rmmseg'
+
+task :default => 'spec:run'
+
+PROJ.name = 'rmmseg-cpp'
+PROJ.version = '0.2.9'
+PROJ.authors = 'pluskid'
+PROJ.email = 'pluskid@gmail.com'
+PROJ.url = 'http://rmmseg-cpp.rubyforge.org'
+PROJ.rubyforge.name = 'rmmseg-cpp'
+
+PROJ.spec.opts << '--color'
+
+# EOF

data/bin/rmmseg

@@ -0,0 +1,63 @@
+#!/usr/bin/env ruby
+
+require File.expand_path(
+    File.join(File.dirname(__FILE__), '..', 'lib', 'rmmseg'))
+
+require 'getoptlong'
+
+def print_usage
+  puts <<EOF
+#{__FILE__}  Segment Chinese text. Read from stdin and print to stdout.
+
+Options:
+  -h
+  --help      Print this message
+
+  -s
+  --separator Select the separator of the segmented text. Default is
+              space.
+EOF
+  exit 0
+end
+
+separator = " "
+
+optparser = GetoptLong.new
+optparser.set_options(["-s", "--separator", GetoptLong::REQUIRED_ARGUMENT],
+                      ["-h", "--help", GetoptLong::NO_ARGUMENT])
+
+loop do
+  begin
+    opt, arg = optparser.get
+    break if not opt
+
+    case opt
+    when "-h"
+      print_usage
+      
+    when "-s"
+      separator = arg
+    end
+    
+  rescue => err
+    puts err
+    exit 1
+  end
+end
+
+RMMSeg::Dictionary.load_dictionaries
+algor = RMMSeg::Algorithm.new(STDIN.read)
+tok = algor.next_token
+unless tok.nil?
+  print tok.text
+
+  loop do
+    tok = algor.next_token
+    break if tok.nil?
+    print separator
+    print tok.text
+  end
+  puts
+end
+
+# EOF