text-hyphen 1.0.2 → 1.2

data/History.rdoc

@@ -0,0 +1,54 @@
+== 1.2 / 2011.07.17
+* Major Enhancements:
+  * This release supports Ruby 1.9 with UTF-8 encodings. The language files are
+    duplicated for both Ruby 1.8 and 1.9 and the correct version is loaded
+    based on RUBY\_VERSION.
+* Minor Enhancements:
+  * Making Hungarian and Norwegian language files act like the German language
+    files (both of these languages have two alternative hyphenation tables).
+  * Added a Malasy language file that should work correctly.
+  * Cleaned up the documentation.
+* Bug Fixes:
+  * Fixed 9807 and 28128 (previously noted as not reproducible; quality
+    reproduction cases were found).
+* Changes:
+  * Relicensing the core library to the MIT license and attempting to clarify
+    the license situation.
+  * Renaming hyphen to ruby-hyphen.
+  * This is the final release compatible with Ruby 1.8, first release
+    compatible with Ruby 1.9.
+* Test Coverage:
+  * Tested with all major Ruby releases except IronRuby.
+    * jruby-1.6.3 (1.8.7): passes all tests.
+    * jruby-1.6.3 (1.9.2): fails one UTF-8 related test (see JRUBY-5927).
+    * macruby-0.10 (1.9.2): passes all tests. Segfault with "rake test" (see
+      https://www.macruby.org/trac/ticket/1362 for more information).
+    * maglev-ruby-0.9 (1.8.7): passes all tests.
+    * rubinius-1.2.5 (1.8.7): passes all tests.
+    * ree-1.8.7-2011.03 (1.8.7): passes all tests.
+    * MRI-1.8.7-p352 (1.8.7): passes all tests.
+    * MRI-1.9.2-p290 (1.9.2): passes all tests.
+
+== 1.0.2 / 2011.02.09
+* Moved to 'hoe' and GitHub.
+* Preparing for 2.0 which will be Ruby 1.9-only for UTF-8.
+* Fixing German support (RubyForge 28498):
+  * Choosing 'de' as a language will load 'de1'. Choosing 'de1' or 'de2' will
+    load properly now, but they will be reported with an ISO language code of
+    'de' (new optional #isocode attribute on a language definition that will
+    override the #iso_language setting of a Text::Hyphen instance if set).
+  * Both 'de1' and 'de2' can be loaded simultaneously now, but the first one
+    loaded will claim the Text::Hyphen::Language::DE constant.
+* Added test cases for bugs:
+  * RubyForge 9807 (cannot reproduce)
+  * RubyForge 28128 (cannot reproduce)
+  * RubyForge 28498
+
+== 1.0.1
+* Minor modification to the RubyGem release of Text::Hyphen to enable the
+  hyphen command-line program.
+
+== 1.0.0
+* Initial version based on TeX::Hyphen 0.4.0 (some changes have been
+  backported to TeX::Hyphen 0.5.0). 
+* Incorporated many hyphenation pattern files from CTAN.

data/License.rdoc

@@ -0,0 +1,159 @@
+== License
+
+Licensing for Text::Hyphen is unfortunately complex because of the various
+copyrights and licenses of the source hyphenation files that have been
+converted to Ruby format. Some of these files are available only under the TeX
+license and others are available only under the GNU GPL while others are public
+domain. Each language file has these licenses embedded within the file. Please
+consult each file's license to ensure that it is compatible with your
+application.
+
+The Text::Hyphen library software, the application ruby-hyphen, and the library
+(gem) as a compilation is licensed under the terms of the MIT license. The
+files in this distribution covered by this license are in the list below called
+"Library Files".
+
+Individual language hyphenation files (in the list called "Language Files") are
+maintained under the license described in the language file itself; the
+copyright for these original files is held by the original authors; any
+mistakes made in conversion of these files to Ruby is attributable to the
+contributors of the Text::Hyphen package only. If license information is not
+present in a given Language File, it should be considered under the terms of
+TeX.
+
+=== Library License
+* Copyright Austin Ziegler, 2004–2011.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+The copyright on the Text::Hyphen application/library and the Ruby
+translations of hyphenation files belongs to Austin Ziegler. All other
+copyrights on original versions still stand; Text::Hyphen is a derivative work
+of these and other projects.
+
+=== Library Files
+Note that while this list appears to include language files, these are "loader"
+files only and do not contain the hyphenation patterns themselves.
+
+* lib/text-hyphen.rb
+* lib/text/hyphen.rb
+* lib/text/hyphen/language.rb
+* lib/text/hyphen/language/ca.rb
+* lib/text/hyphen/language/cs.rb
+* lib/text/hyphen/language/da.rb
+* lib/text/hyphen/language/de.rb
+* lib/text/hyphen/language/de1.rb
+* lib/text/hyphen/language/de2.rb
+* lib/text/hyphen/language/en_uk.rb
+* lib/text/hyphen/language/en_us.rb
+* lib/text/hyphen/language/es.rb
+* lib/text/hyphen/language/et.rb
+* lib/text/hyphen/language/eu.rb
+* lib/text/hyphen/language/fi.rb
+* lib/text/hyphen/language/fr.rb
+* lib/text/hyphen/language/ga.rb
+* lib/text/hyphen/language/hr.rb
+* lib/text/hyphen/language/hsb.rb
+* lib/text/hyphen/language/hu.rb
+* lib/text/hyphen/language/hu1.rb
+* lib/text/hyphen/language/hu2.rb
+* lib/text/hyphen/language/ia.rb
+* lib/text/hyphen/language/id.rb
+* lib/text/hyphen/language/is.rb
+* lib/text/hyphen/language/it.rb
+* lib/text/hyphen/language/la.rb
+* lib/text/hyphen/language/mn.rb
+* lib/text/hyphen/language/ms.rb
+* lib/text/hyphen/language/nl.rb
+* lib/text/hyphen/language/no.rb
+* lib/text/hyphen/language/no1.rb
+* lib/text/hyphen/language/no2.rb
+* lib/text/hyphen/language/pl.rb
+* lib/text/hyphen/language/pt.rb
+* lib/text/hyphen/language/sv.rb
+* test/data/bug_9807_latin1.rb
+* test/data/bug_9807_utf-8.rb
+* test/test_bugs.rb
+* test/test_text_hyphen.rb
+* text-hyphen.gemspec
+* History.rdoc
+* License.rdoc
+* README.rdoc
+* Rakefile
+* bin/ruby-hyphen
+
+=== Language Files
+* lib/text/hyphen/language/1.8/ca.rb
+* lib/text/hyphen/language/1.8/cs.rb
+* lib/text/hyphen/language/1.8/da.rb
+* lib/text/hyphen/language/1.8/de1.rb
+* lib/text/hyphen/language/1.8/de2.rb
+* lib/text/hyphen/language/1.8/en_uk.rb
+* lib/text/hyphen/language/1.8/en_us.rb
+* lib/text/hyphen/language/1.8/es.rb
+* lib/text/hyphen/language/1.8/et.rb
+* lib/text/hyphen/language/1.8/eu.rb
+* lib/text/hyphen/language/1.8/fi.rb
+* lib/text/hyphen/language/1.8/fr.rb
+* lib/text/hyphen/language/1.8/ga.rb
+* lib/text/hyphen/language/1.8/hr.rb
+* lib/text/hyphen/language/1.8/hsb.rb
+* lib/text/hyphen/language/1.8/hu1.rb
+* lib/text/hyphen/language/1.8/hu2.rb
+* lib/text/hyphen/language/1.8/ia.rb
+* lib/text/hyphen/language/1.8/id.rb
+* lib/text/hyphen/language/1.8/is.rb
+* lib/text/hyphen/language/1.8/it.rb
+* lib/text/hyphen/language/1.8/la.rb
+* lib/text/hyphen/language/1.8/mn.rb
+* lib/text/hyphen/language/1.8/nl.rb
+* lib/text/hyphen/language/1.8/no1.rb
+* lib/text/hyphen/language/1.8/no2.rb
+* lib/text/hyphen/language/1.8/pl.rb
+* lib/text/hyphen/language/1.8/pt.rb
+* lib/text/hyphen/language/1.8/sv.rb
+* lib/text/hyphen/language/1.9/ca.rb
+* lib/text/hyphen/language/1.9/cs.rb
+* lib/text/hyphen/language/1.9/da.rb
+* lib/text/hyphen/language/1.9/de1.rb
+* lib/text/hyphen/language/1.9/de2.rb
+* lib/text/hyphen/language/1.9/en_uk.rb
+* lib/text/hyphen/language/1.9/en_us.rb
+* lib/text/hyphen/language/1.9/es.rb
+* lib/text/hyphen/language/1.9/et.rb
+* lib/text/hyphen/language/1.9/eu.rb
+* lib/text/hyphen/language/1.9/fi.rb
+* lib/text/hyphen/language/1.9/fr.rb
+* lib/text/hyphen/language/1.9/ga.rb
+* lib/text/hyphen/language/1.9/hr.rb
+* lib/text/hyphen/language/1.9/hsb.rb
+* lib/text/hyphen/language/1.9/hu1.rb
+* lib/text/hyphen/language/1.9/hu2.rb
+* lib/text/hyphen/language/1.9/ia.rb
+* lib/text/hyphen/language/1.9/id.rb
+* lib/text/hyphen/language/1.9/is.rb
+* lib/text/hyphen/language/1.9/it.rb
+* lib/text/hyphen/language/1.9/la.rb
+* lib/text/hyphen/language/1.9/mn.rb
+* lib/text/hyphen/language/1.9/nl.rb
+* lib/text/hyphen/language/1.9/no1.rb
+* lib/text/hyphen/language/1.9/no2.rb
+* lib/text/hyphen/language/1.9/pl.rb
+* lib/text/hyphen/language/1.9/pt.rb
+* lib/text/hyphen/language/1.9/sv.rb

data/Manifest.txt

@@ -1,14 +1,71 @@
 .autotest
-COPYING.txt
-History.txt
-LICENCE.txt
+History.rdoc
+License.rdoc
 Manifest.txt
-README.txt
+README.rdoc
 Rakefile
-bin/hyphen
+bin/ruby-hyphen
 lib/text-hyphen.rb
 lib/text/hyphen.rb
 lib/text/hyphen/language.rb
+lib/text/hyphen/language/1.8/ca.rb
+lib/text/hyphen/language/1.8/cs.rb
+lib/text/hyphen/language/1.8/da.rb
+lib/text/hyphen/language/1.8/de1.rb
+lib/text/hyphen/language/1.8/de2.rb
+lib/text/hyphen/language/1.8/en_uk.rb
+lib/text/hyphen/language/1.8/en_us.rb
+lib/text/hyphen/language/1.8/es.rb
+lib/text/hyphen/language/1.8/et.rb
+lib/text/hyphen/language/1.8/eu.rb
+lib/text/hyphen/language/1.8/fi.rb
+lib/text/hyphen/language/1.8/fr.rb
+lib/text/hyphen/language/1.8/ga.rb
+lib/text/hyphen/language/1.8/hr.rb
+lib/text/hyphen/language/1.8/hsb.rb
+lib/text/hyphen/language/1.8/hu1.rb
+lib/text/hyphen/language/1.8/hu2.rb
+lib/text/hyphen/language/1.8/ia.rb
+lib/text/hyphen/language/1.8/id.rb
+lib/text/hyphen/language/1.8/is.rb
+lib/text/hyphen/language/1.8/it.rb
+lib/text/hyphen/language/1.8/la.rb
+lib/text/hyphen/language/1.8/mn.rb
+lib/text/hyphen/language/1.8/nl.rb
+lib/text/hyphen/language/1.8/no1.rb
+lib/text/hyphen/language/1.8/no2.rb
+lib/text/hyphen/language/1.8/pl.rb
+lib/text/hyphen/language/1.8/pt.rb
+lib/text/hyphen/language/1.8/sv.rb
+lib/text/hyphen/language/1.9/ca.rb
+lib/text/hyphen/language/1.9/cs.rb
+lib/text/hyphen/language/1.9/da.rb
+lib/text/hyphen/language/1.9/de1.rb
+lib/text/hyphen/language/1.9/de2.rb
+lib/text/hyphen/language/1.9/en_uk.rb
+lib/text/hyphen/language/1.9/en_us.rb
+lib/text/hyphen/language/1.9/es.rb
+lib/text/hyphen/language/1.9/et.rb
+lib/text/hyphen/language/1.9/eu.rb
+lib/text/hyphen/language/1.9/fi.rb
+lib/text/hyphen/language/1.9/fr.rb
+lib/text/hyphen/language/1.9/ga.rb
+lib/text/hyphen/language/1.9/hr.rb
+lib/text/hyphen/language/1.9/hsb.rb
+lib/text/hyphen/language/1.9/hu1.rb
+lib/text/hyphen/language/1.9/hu2.rb
+lib/text/hyphen/language/1.9/ia.rb
+lib/text/hyphen/language/1.9/id.rb
+lib/text/hyphen/language/1.9/is.rb
+lib/text/hyphen/language/1.9/it.rb
+lib/text/hyphen/language/1.9/la.rb
+lib/text/hyphen/language/1.9/mn.rb
+lib/text/hyphen/language/1.9/nl.rb
+lib/text/hyphen/language/1.9/no1.rb
+lib/text/hyphen/language/1.9/no2.rb
+lib/text/hyphen/language/1.9/pl.rb
+lib/text/hyphen/language/1.9/pt.rb
+lib/text/hyphen/language/1.9/sv.rb
 lib/text/hyphen/language/ca.rb
 lib/text/hyphen/language/cs.rb
 lib/text/hyphen/language/da.rb
@@ -25,6 +82,7 @@ lib/text/hyphen/language/fr.rb
 lib/text/hyphen/language/ga.rb
 lib/text/hyphen/language/hr.rb
 lib/text/hyphen/language/hsb.rb
+lib/text/hyphen/language/hu.rb
 lib/text/hyphen/language/hu1.rb
 lib/text/hyphen/language/hu2.rb
 lib/text/hyphen/language/ia.rb
@@ -33,12 +91,16 @@ lib/text/hyphen/language/is.rb
 lib/text/hyphen/language/it.rb
 lib/text/hyphen/language/la.rb
 lib/text/hyphen/language/mn.rb
+lib/text/hyphen/language/ms.rb
 lib/text/hyphen/language/nl.rb
+lib/text/hyphen/language/no.rb
 lib/text/hyphen/language/no1.rb
 lib/text/hyphen/language/no2.rb
 lib/text/hyphen/language/pl.rb
 lib/text/hyphen/language/pt.rb
 lib/text/hyphen/language/sv.rb
+test/data/bug_9807_latin1.rb
+test/data/bug_9807_utf-8.rb
 test/test_bugs.rb
 test/test_text_hyphen.rb
 text-hyphen.gemspec

data/README.rdoc

@@ -0,0 +1,69 @@
+= text-hyphen
+
+== Description
+
+Text::Hyphen is a Ruby library to hyphenate words in various languages using
+Ruby-fied versions of TeX hyphenation patterns. It will properly hyphenate
+various words according to the rules of the language the word is written in.
+The algorithm is based on that of the TeX typesetting system by Donald E.
+Knuth.
+
+This is originally based on the Perl implementation of
+{TeX::Hyphen}[http://search.cpan.org/author/JANPAZ/TeX-Hyphen-0.140/lib/TeX/Hyphen.pm]
+and the {Ruby port}[http://rubyforge.org/projects/text-format]. The language
+hyphenation pattern files are based on the sources available from
+{CTAN}[http://www.ctan.org] as of 2004.12.19 and have been manually translated
+by Austin Ziegler.
+
+This release is 1.2. This is a major release providing both Ruby 1.8.7 and Ruby
+1.9.2 support. This is the last major release supporting Ruby 1.8 interpreters.
+Future versions will only work with Ruby 1.9 or later interpreters.
+
+== Where
+
+* {RubyForge}[http://rubyforge.org/projects/text-format/]
+* {RubyGems}[https://rubygems.org/gems/text-hyphen]
+* {GitHub}[https://github.com/halostatue/text-hyphen/]
+* {RDoc}[http://rdoc.info/github/halostatue/text-hyphen/master/frames]
+
+== Synopsis
+
+  require 'text/hyphen'
+  hh = Text::Hyphen.new(:language => 'en_us', :left => 2, :right => 2)
+  # Defaults to the above
+  hh = TeX::Hyphen.new
+
+  word = "representation"
+  points = hyp.hyphenate(word)  #=> [3, 5, 8, 10]
+  puts hyp.visualize(word)      #=> rep-re-sen-ta-tion
+
+Text::Hyphen is truly multilingual, with 29 languages or language variants
+supported. As an example, consider the difference between the following:
+
+  require 'text/hyphen'
+  # Using left and right minimum values of 0 ensures that you will see all
+  # possible hyphenation points, not just those that meet the minimum width
+  # requirements.
+  en = Text::Hyphen.new(:left => 0, :right => 0)
+  fr = Text::Hyphen.new(:language => "fr", :left => 0, :right => 0)
+
+  puts en.visualise("organiser")      #=> or-gan-iser
+  puts fr.visualise("organiser")      #=> or-ga-ni-ser
+
+As you can see, the hyphenation is distinct between the two hyphenators.
+Additional improvements over TeX::Hyphen include thread safety (except for
+debug control) and support for UTF-8 under Ruby 1.9.
+
+== Install
+  gem install text-hyphen
+
+== Developers
+
+After checking out the source, run:
+
+  $ rake newb
+
+This task will install any missing dependencies, run the tests/specs,
+and generate the RDoc.
+
+:include: License.rdoc

data/Rakefile

@@ -1,19 +1,23 @@
-# -*- ruby -*-
+# -*- ruby encoding: utf-8 -*-
 
 require 'rubygems'
 require 'hoe'
-require 'rubyforge'
 
 Hoe.plugin :doofus
 Hoe.plugin :gemspec
 Hoe.plugin :git
 Hoe.plugin :rubyforge
 
-Hoe.spec 'text-hyphen' do
+Hoe.spec 'text-hyphen' do |spec|
   developer('Austin Ziegler', 'austin@rubyforge.org')
+
   self.rubyforge_name = 'text-format'
+  spec.remote_rdoc_dir = 'text-hyphen/rdoc'
+  spec.rsync_args << ' --exclude=statsvn/'
 
-  self.spec_extras[:required_ruby_version] = '< 1.9'
+  spec.history_file = 'History.rdoc'
+  spec.readme_file = 'README.rdoc'
+  spec.extra_rdoc_files = FileList["*.rdoc"].to_a
 
   self.extra_dev_deps << ['hoe-doofus', '~> 1.0']
   self.extra_dev_deps << ['hoe-gemspec', '~> 1.0']

data/lib/text-hyphen.rb

@@ -1 +1,2 @@
+# -*- ruby encoding: utf-8 -*-
 require 'text/hyphen'

data/lib/text/hyphen.rb

@@ -1,95 +1,26 @@
-module Text; end
-
-# = Introduction
-# Text::Hyphen -- hyphenate words using modified versions of TeX hyphenation
-# patterns.
-#
-# == Usage
-#   require 'text/hyphen'
-#   hh = Text::Hyphen.new(:language => 'en_us', :left => 2, :right => 2)
-#     # Defaults to the above
-#   hh = TeX::Hyphen.new
-#
-#   word = "representation"
-#   points = hyp.hyphenate(word)    #=> [3, 5, 8, 10]
-#   puts hyp.visualize(word)        #=> rep-re-sen-ta-tion
-#   
-#   en = Text::Hyphen.new(:left => 0, :right => 0)
-#   fr = Text::Hyphen.new(:language = "fr", :left => 0, :right => 0)
-#   puts en.visualise("organiser")  #=> or-gan-iser
-#   puts fr.visualise("organiser")  #=> or-ga-ni-ser
-#
-# == Description
-# Creates a new Hyphen object and loads the language patterns into memory.
-# The hyphenator can then be asked for the hyphenation of a word. If no
-# language is specified, then the language en_us (EN_US) is used by default.
-#
-# Copyright::   Copyright (c) 2004 - 2005 Austin Ziegler
-# Version::     1.0.2
-# Based On::    <tt>TeX::Hyphen</tt> 0.4 Copyright (c) 2003 - 2004
-#               Martin DeMello and Austin Ziegler, in turn based on
-#               Perl's <tt>TeX::Hyphen</tt>
-#               [http://search.cpan.org/author/JANPAZ/TeX-Hyphen-0.140/lib/TeX/Hyphen.pm]
-#               Copyright (c) 1997 - 2002 Jan Pazdziora
-#
-# == Licence
-# Licensing for Text::Hyphen is unfortunately complex because of the various
-# copyrights and licences of the source hyphenation files. Some of these
-# files are available only under the TeX licence and others are available
-# only under the GNU GPL while others are public domain. Each language file
-# has these licences embedded within the file. Please consult each file's
-# licence to ensure that it is compatible with your application.
-#
-# The copyright on the Text::Hyphen application/library and the Ruby
-# translations of hyphenation files belongs to Austin Ziegler. All other
-# copyrights on original versions still stand; Text::Hyphen is a derivative
-# work of these and other projects.
-#
-# === Application and Compilation Licences
-# Text::Hyphen, the application/library is licensed under the same terms as
-# Ruby. Note that this specifically refers to the contents of bin/hyphen,
-# lib/text/hyphen.rb, and lib/text/hyphen/language.rb.
-#
-# Individual language hyphenation files are NOT licensed under these terms,
-# but under the following MIT-style licence and the original hyphenation
-# pattern licenses. The copyright for the original TeX hyphenation files is
-# held by the original authors; any mistakes in conversion of these files to
-# Ruby is attributable to the contributors to the Text::Hyphen package only.
-#
-# The compilation package Text::Hyphen is licensed under the same terms as
-# Ruby.
-#
-# === Blanket Language Hyphenation File Licence
-# Permission is hereby granted, free of charge, to any person obtaining a
-# copy of this software and associated documentation files (the "Software"),
-# to deal in the Software without restriction, including without limitation
-# the rights to use, copy, modify, merge, publish, distribute, sublicense,
-# and/or sell copies of the Software, and to permit persons to whom the
-# Software is furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-# DEALINGS IN THE SOFTWARE.
+# -*- ruby encoding: utf-8 -*-
+module Text # :nodoc:
+end
+
+# An object that knows how to perform hyphenation based on the TeX
+# hyphenation algorithm with pattern files. Each object is constructed with
+# a specific language's hyphenation patterns.
 class Text::Hyphen
   DEBUG   = false
-  VERSION = '1.0.2'
+  VERSION = '1.2'
 
   DEFAULT_MIN_LEFT  = 2
   DEFAULT_MIN_RIGHT = 2
 
   # No fewer than this number of letters will show up to the left of the
-  # hyphen. This overrides the default specified in the language.
+  # hyphen. The initial value for this will be specified by the language;
+  # setting this value will override the language's defaults.
   attr_accessor :left
+
   # No fewer than this number of letters will show up to the right of the
   # hyphen. This overrides the default specified in the language.
   attr_accessor :right
+
   # The name of the language to be used in hyphenating words. This will be a
   # two or three character ISO 639 code, with the two character form being
   # the canonical resource name. This will load the language hyphenation
@@ -101,14 +32,15 @@ class Text::Hyphen
   # Minimal transformations will be performed on the language code provided,
   # such that any dashes are converted to underscores (e.g., 'en-us' becomes
   # 'en_us') and all characters are regularised. Resource names will be
-  # downcased and class names will be upcased (e.g., 'Pt' for the Portuguese
-  # language becomes 'pt' and 'PT', respectively).
+  # downcased and class names will be converted to uppercase (e.g., 'Pt' for
+  # the Portuguese language becomes 'pt' and 'PT', respectively).
   #
   # The language may also be specified as an instance of
   # Text::Hyphen::Language.
   attr_accessor :language
+
   undef :language=
-  def language=(lang)
+  def language=(lang) #:nodoc:
     require 'text/hyphen/language' unless defined?(Text::Hyphen::Language)
     if lang.kind_of? Text::Hyphen::Language
       @iso_language = lang.to_s.split(%r{::}o)[-1].downcase
@@ -119,13 +51,27 @@ class Text::Hyphen
     end
     @iso_language
   end
+
   # Returns the language's ISO 639 ID, e.g., "en_us" or "pt".
-  attr_reader   :iso_language
+  attr_reader :iso_language
 
-  # The following initializations are equivalent:
+  # Creates a hyphenation object with the options requested. The options
+  # available are:
   #
-  #   hyp = TeX::Hyphenate.new(:language => "EU")
-  #   hyp = TeX::Hyphenate.new { |h| h.language = "EU" }
+  # language::  The language to perform hyphenation with. See #language and
+  #             #iso_language.
+  # left::      The minimum number of characters to the left of a
+  #             hyphenation point. See #left.
+  # right::     The minimum number of characters to the right of a
+  #             hyphenation point. See #right.
+  #
+  # The options can be provided either as hashed parameters or set as
+  # methods in an initialization block. The following initializations are
+  # all equivalent:
+  #
+  #   hyp = Text::Hyphenate.new(:language => 'en_us')
+  #   hyp = Text::Hyphenate.new(language: 'en_us') # under Ruby 1.9
+  #   hyp = Text::Hyphenate.new { |h| h.language = 'en_us' }
   def initialize(options = {}) # :yields self:
     @iso_language = options[:language]
     @left         = options[:left]
@@ -147,16 +93,16 @@ class Text::Hyphen
 
     load_language
 
-    @left       ||= DEFAULT_MIN_LEFT
-    @right      ||= DEFAULT_MIN_RIGHT
+    @left  ||= DEFAULT_MIN_LEFT
+    @right ||= DEFAULT_MIN_RIGHT
   end
 
-  # Returns a list of places where the word can be divided, as
+  # Returns an array of character positions where a word can be hyphenated.
   #
-  #   hyp.hyphenate('representation')
+  #   hyp.hyphenate('representation') #=> [3, 5, 8 10]
   #
-  # returns [3, 5, 8, 10]. If the word has been hyphenated previously, it
-  # will be returned from a per-instance cache.
+  # Because hyphenation can be expensive, if the word has been hyphenated
+  # previously, it will be returned from a per-instance cache.
   def hyphenate(word)
     word = word.downcase
     $stderr.puts "Hyphenating #{word}" if DEBUG
@@ -164,28 +110,32 @@ class Text::Hyphen
     res = @language.exceptions[word]
     return @cache[word] = make_result_list(res) if res
 
-    result = [0] * (word.split(//).size + 1)
-    rightstop = word.split(//).size - @right
+    letters = word.scan(@language.scan_re)
+    $stderr.puts letters.inspect if DEBUG
+    word_size = letters.size
+
+    result = [0] * (word_size + 1)
+    right_stop = word_size - @right
 
     updater = Proc.new do |hash, str, pos|
       if hash.has_key?(str)
         $stderr.print "#{pos}: #{str}: #{hash[str]}" if DEBUG
-        hash[str].split(//).each_with_index do |cc, ii|
+        hash[str].scan(@language.scan_re).each_with_index do |cc, ii|
           cc = cc.to_i
           result[ii + pos] = cc if cc > result[ii + pos]
         end
-        $stderr.print ": #{result}\n" if DEBUG
+        $stderr.print ": #{result.inspect}\n" if DEBUG
       end
     end
 
       # Walk the word
-    (0..rightstop).each do |pos|
-      restlength = word.length - pos
-      (1..restlength).each do |length|
-        substr = word[pos, length]
+    (0..right_stop).each do |pos|
+      rest_length = word_size - pos
+      (1..rest_length).each do |length|
+        substr = letters[pos, length].join('')
         updater[@language.hyphen, substr, pos]
         updater[@language.start, substr, pos] if pos.zero?
-        updater[@language.stop, substr, pos] if (length == restlength)
+        updater[@language.stop, substr, pos] if (length == rest_length)
       end
     end
 
@@ -196,23 +146,23 @@ class Text::Hyphen
     @cache[word] = make_result_list(result)
   end
 
-  # Returns a visualization of the hyphenation points, so:
+  # Returns a visualization of the hyphenation points.
   #
-  #   hyp.visualize('representation')
+  #   hyp.visualize('representation') #=> rep-re-sen-ta-tion
   #
-  # returns <tt>rep-re-sen-ta-tion</tt>, at least for English patterns. If
-  # the word has been visualised previously, it will be returned from a
-  # per-instance cache.
+  # Because hyphenation can be expensive, if the word has been visualised
+  # previously, it will be returned from a per-instance cache.
   def visualise(word)
     return @vcache[word] if @vcache.has_key?(word)
     w = word.dup
-    hyphenate(w).each_with_index do |pos, n| 
+    hyphenate(w).each_with_index do |pos, n|
       w[pos.to_i + n, 0] = '-' if pos != 0
     end
     @vcache[word] = w
   end
   alias visualize visualise
 
+  # Clears the per-instance hyphenation and visualization caches.
   def clear_cache!
     @cache.clear
     @vcache.clear
@@ -229,7 +179,8 @@ class Text::Hyphen
     end
   end
 
-  # Returns statistics
+  # Returns a string describing the structure of the patterns for the
+  # language of this hyphenation object.
   def stats
     _b = @language.both.size
     _s = @language.start.size
@@ -254,7 +205,7 @@ EOS
   def updateresult(hash, str, pos)
     if hash.has_key?(str)
       STDERR.print "#{pos}: #{str}: #{hash[str]}" if DEBUG
-      hash[str].split('').each_with_index do |c, i| 
+      hash[str].scan(@language.scan_re).each_with_index do |c, i|
         c = c.to_i
         @result[i + pos] = c if c > @result[i + pos]
       end
@@ -287,4 +238,16 @@ EOS
     @iso_language
   end
   private :load_language
+
+  # Resolves a file for cleaner loading from a hyphenation loader file.
+  def self.require_real_hyphenation_file(loader) # :nodoc:
+    p = File.dirname(loader)
+    f = File.basename(loader)
+    v = if RUBY_VERSION < "1.9.1"
+          "1.8"
+        else
+          "1.9"
+        end
+    require File.join(p, v, f)
+  end
 end