scalpel 0.2

data/LICENSE

@@ -0,0 +1,18 @@
+Scalpel - A fast and accurate sentence segmentation tool for Ruby.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+This license also applies to the included Stanford CoreNLP files.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Author: Louis-Antoine Mullie (louis.mullie@gmail.com). Copyright 2012.

data/README.md

@@ -0,0 +1,40 @@
+[![Build Status](https://secure.travis-ci.org/louismullie/scalpel.png)](http://travis-ci.org/#!/louismullie/scalpel)
+
+**What?**
+
+Scalpel is the result of my unability to find a simple and elegant solution to sentence segmentation in Ruby. Machine learning approaches - both unsupervised ([punkt-segmenter](https://github.com/lfcipriani/punkt-segmenter)) and supervised ( [tactful_tokenizer](https://github.com/SlyShy/Tactful_Tokenizer)) - depend on proper domain-specific training to work well. Stanford's tokenize-first group-later method ([stanford-core-nlp](https://github.com/louismullie/stanford-core-nlp)) do not work so well in the face of ill-formatted content. Finally, extensive rule-based methods ([srx-english](https://github.com/apohllo/srx-english)) are very accurate but suffer from poor performance. 
+
+Scalpel is based on a very simple principle that reduces the complexity of performing sentence segmentation. The idea is that it is simpler and more efficient to find occurrences of periods that do __not__ indicate the end of a sentence, rather than those who do. These occurrences are temporarily replaced by "placeholder" characters, and sentence splitting is subsequently performed. The placeholder characters are then replaced by the original characters.
+
+**How?**
+
+    gem install scalpel
+
+```ruby
+require 'scalpel'
+Scalpel.cut("some text")
+```
+
+**Why?**
+
+![Benchmark](http://www.louismullie.com/ruby/scalpel/comparison.png)
+
+* Scalpel is 80x faster than the only other method with 100% accuracy.
+* Scalpel is 5x faster than the next fastest-running segmentation tool.
+* Scalpel loads twice as fast as the next fastest-loading segmentation tool.
+
+A few notes on the benchmark: 
+
+* As with any benchmark, your mileage may vary, take it with a grain of salt, etc.
+* Loading time for the Punkt segmenter is dependent on the size of the model.
+* Stanford segmenter that was used is the one in DocumentPreprocessor, via Rjb.
+
+The text that was used is the following:
+
+```ruby
+"For years, people in the U.A.E.R. have accepted murky air, tainted waters and scarred landscapes as the unavoidable price of the country’s meteoric economic growth. But public dissent over environmental issues has been growing steadily in the communist nation, and now seems to be building the foundations of a fledgling green movement! In July alone, two separate demonstrations made international news when they turned violent after about 1.5 minutes... These recent successes come after a slew of ever-larger and more violent green protests over the past few years, as the environmentalist Dr. Jeung of China’s growth becomes harder to ignore.Some ask: “Are demonstrations are evidence of the public anger and frustration at opaque environmental management and decision-making?” Others yet say: \"Should we be scared about these 'protests'?\""
+```
+
+**Contributing**
+
+Feel free to fork the project and send me a pull request!

data/lib/scalpel.rb

@@ -0,0 +1,74 @@
+# encoding: utf-8
+#
+# Sentence segmentation based on a set of predefined
+# rules that handle a large number of usage cases of
+# sentence enders. The idea is to remove all cases of
+# .!? being used for other purposes than marking a
+# full stop before naively segmenting the text.
+class Scalpel
+
+  # Current version.
+  VERSION = '0.2'
+
+  # Segment a text using the Scalpel algorithm.
+  # This will eventually be ported to a gem.
+  def self.cut(text)
+
+    # Get a copy of the string.
+    text = text.to_s
+    # Remove composite abbreviations.
+    text.gsub!('et al.', '&&&')
+    # Remove suspension points.
+    text.gsub!('...', '&;&.')
+    # Remove floating point numbers.
+    text.gsub!(/([0-9]+)\.([0-9]+)/) { $1 + '&@&' + $2 }
+    # Remove abbreviations.
+    text.gsub!(/(?:[A-Za-z]\.){2,}/) { |abbr| abbr.gsub('.', '&-&') }
+    # Remove titles.
+    text.gsub!(/[A-Z][a-z]{1,2}\./) { |title| title.gsub('.', '&*&') }
+    # Unstick sentences from each other.
+    text.gsub!(/([^.?!]\.|\!|\?)([^\s"'])/) { $1 + ' ' + $2 }
+    # Remove sentence enders next to quotes.
+    text.gsub!(/'([.?!])\s?"/) { '&^&' + $1 }
+    text.gsub!(/'([.?!])\s?”/) { '&*&' + $1 }
+    text.gsub!(/([.?!])\s?”/) { '&=&' + $1 }
+    text.gsub!(/([.?!])\s?'"/) { '&,&' + $1 }
+    text.gsub!(/([.?!])\s?'/) { '&%&' + $1 }
+    text.gsub!(/([.?!])\s?"/) { '&$&' + $1 }
+    # Split on any sentence ender.
+    sentences = text.split(/([.!?])/)
+    new_sents = []
+    # Join the obtaine slices.
+    sentences.each_slice(2) do |slice|
+      new_sents << slice.join('')
+    end
+    # Repair the damage we've done.
+    results = []
+    new_sents.each do |sentence|
+      # Skip whitespace zones.
+      next if sentence.strip == ''
+      # Repair composite abbreviations.
+      sentence.gsub!('&&&', 'et al.')
+      # Repair abbreviations.
+      sentence.gsub!('&-&', '.')
+      # Repair titles.
+      sentence.gsub!('&*&', '.')
+      # Repair suspension points.
+      sentence.gsub!('&;&.', '...')
+      # Repair floats.
+      sentence.gsub!(/([0-9]+)&@&([0-9]+)/) { $1 + '.' + $2 }
+      # Repair quotes with sentence enders
+      sentence.gsub!(/&=&([.!?])/) { $1 + '”' }
+      sentence.gsub!(/&,&([.!?])/) { $1 + "'\"" }
+      sentence.gsub!(/&%&([.!?])/) { $1 + "'" }
+      sentence.gsub!(/&\^&([.?!])/) { "'" + $1 + '"' }
+      sentence.gsub!(/&\*&([.?!])/) { "'" + $1 + '”' }
+      sentence.gsub!(/&$&([.!?])/) { $1 + '"' }
+      results << sentence.strip
+    end
+
+    results
+
+  end
+
+end

metadata

@@ -0,0 +1,51 @@
+--- !ruby/object:Gem::Specification
+name: scalpel
+version: !ruby/object:Gem::Version
+  version: '0.2'
+  prerelease: 
+platform: ruby
+authors:
+- Louis Mullie
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-08-15 00:00:00.000000000 Z
+dependencies: []
+description: ! ' Scalpel is a sentence segmentation tool for Ruby. It allows you to
+  split a text into an array of sentences. It is simple, lightweight, blazing fast
+  and does not require any domain-specific training. It works well even in the face
+  of ill-formatted texts. '
+email:
+- louis.mullie@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/scalpel.rb
+- README.md
+- LICENSE
+homepage: https://github.com/louismullie/scalpel
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: A fast and accurate rule-based sentence segmentation tool for Ruby.
+test_files: []