latex-decode 0.0.12-java

data/README.md

@@ -0,0 +1,63 @@
+LaTeX::Decode
+=============
+
+LaTeX::Decode is a Ruby gem to convert LaTeX input to Unicode. Its original
+use was as an input filter for [BibTeX-Ruby](http://rubygems.org/gems/bibtex-ruby)
+but can be used independently to decode LaTeX. Many of the patterns used by
+this Ruby gem are based on François Charette's equivalent Perl module
+[LaTeX::Decode](https://github.com/fc7/LaTeX-Decode).
+
+Quickstart
+----------
+
+    $ [sudo] gem install bibtex-ruby
+    $ irb
+    >> require 'latex/decode'
+    >> LaTeX.decode "dipl\\^{o}me d'\\'{e}tudes sup\\'erieures"
+    => "diplôme d'études supérieures"
+
+Compatibility
+-------------
+
+Unicode handling is one of the major differences between Ruby 1.8 and newer
+version; LaTeX::Decode; nevertheless, we try to support 1.8 as best as possible.
+
+Issues
+------
+
+Please use the tracker of the project's
+[Github repository](https://github.com/inukshuk/latex-decode) to report any
+issues. When describing intended behaviour, please use the extremely simple
+syntax of the Cucumber features used by LaTeX::Decode; for instance, you could
+describe the example above as:
+
+    Feature: Decode LaTeX accents
+      As a hacker who works with LaTeX
+      I want to be able to decode LaTeX accents
+  
+      Scenario: A French sentence
+        When I decode the string "dipl\\^{o}me d'\\'{e}tudes sup\\'erieures"
+        Then the result should be "diplôme d'études supérieures"
+
+Credits
+-------
+
+Kudos and thanks to all [contributors](https://github.com/inukshuk/latex-decode/contributors)
+who have made LaTeX::Decode possible!
+
+Copyright (C) 2011 [Sylvester Keil](sylvester.keil.or.at)
+
+Copyright (C) 2010 François Charette
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

data/features/brackets.feature

@@ -0,0 +1,11 @@
+Feature: Decode LaTeX umlauts
+  As a hacker who works with LaTeX
+  I want to be able to remove brackets around single characters
+
+  Scenario: Single character in curly brackets
+    When I decode the string '{a}'
+    Then the result should be 'a'
+
+  Scenario: German umlauts in curly brackets
+    When I decode the string '{\"A}{\"o}{\"u}'
+    Then the result should be 'Äöü'

data/features/diacritics.feature

@@ -0,0 +1,40 @@
+Feature: Decode LaTeX diacritics
+	As a hacker who works with LaTeX
+	I want to be able to decode LaTeX diacritics
+	
+	Scenario Outline: LaTeX to Unicode transformation
+		When I decode the string '<latex>'
+		Then the result should be '<unicode>'
+
+	Scenarios: Diacritics
+		| latex   | unicode | description                                  |
+		| \\\`{o} | ò       | grave accent                                 |
+		| \\\'{o} | ó       | acute accent                                 |
+		| \\^{o}  | ô       | circumflex                                   |
+		| \\"{o}  | ö       | umlaut or dieresis                           |
+		| \\H{o}  | ő       | long Hungarian umlaut (double acute)         |
+		| \\~{o}  | õ       | tilde                                        |
+		| \\c{c}  | ç       | cedilla                                      |
+		| \\c c   | ç       |                                              |
+		| \\c cb  | çb      |                                              |
+		| \\c {cb}| \\c cb  |                                              |
+		| \\c C   | Ç       |                                              |
+		| {\\c c} | ç       |                                              |
+		| \\k{a}  | ą       | ogonek                                       |
+		| \\l     | ł       | l with stroke                                |
+		| \\L     | Ł       | l with stroke                                |
+		| \\={o}  | ō       | macron accent (a bar over the letter)        |
+		| \\b{o}  | o̱      | bar under the letter                         |
+		| \\.{o}  | ȯ       | dot over the letter                          |
+		| \\d{u}  | ụ       | dot under the letter                         |
+		| \\r{a}  | å       | ring over the letter                         |
+		| \\u{o}  | ŏ       | breve over the letter                        |
+		| \\v{s}  | š       | caron/hacek ("v") over the letter            |
+		| \\t{oo} | o͡o     | "tie" (inverted u) over the two letters      |
+		| \\aa    | å       | ring over the letter a                       |
+		| \\AA    | Å       | ring over the letter A                       |
+		| \\o     | ø       | slashed o                                    |
+		| \\O     | Ø       | slashed O                                    |
+		| \\ae    | æ       | ae                                           |
+		| \\AE    | Æ       | AE                                           |
+	

data/features/non-latex.feature

@@ -0,0 +1,15 @@
+Feature: Keep non-LaTeX markup as it is
+  As a hacker who works with LaTeX
+  I want to be preserve non-LaTeX text as it is
+
+  Scenario: Standalone escape characters
+    When I decode the string '\\'
+    Then the result should be '\\'
+
+  Scenario: Common markup in Regular Expressions
+    When I decode the string '.*'
+    Then the result should be '.*'
+    When I decode the string '^x$'
+    Then the result should be '^x$'
+    When I decode the string '\\\\2'
+    Then the result should be '\\\\2'

data/features/punctuation.feature

@@ -0,0 +1,46 @@
+Feature: Decode LaTeX punctuation directives
+  As a hacker who works with LaTeX
+  I want to be able to decode LaTeX punctuation marks
+
+  Scenario Outline: LaTeX to Unicode transformation
+    When I decode the string '<latex>'
+    Then the result should be '<unicode>'
+
+  Scenarios: Punctuation macros
+    | latex                | unicode |
+    | \\textendash         | –       |
+    | \\textemdash         | —       |
+    | \\textquoteleft      | ‘       |
+    | \\textquoteright     | ’       |
+    | \\quotesinglbase     | ‚       |
+    | \\textquotedblleft   | “       |
+    | \\textquotedblright  | ”       |
+    | \\quotedblbase       | „       |
+    | \\dag                | †       |
+    | \\ddag               | ‡       |
+    | \\textbullet         | •       |
+    | \\dots               | …       |
+    | \\textperthousand    | ‰       |
+    | \\textpertenthousand | ‱       |
+    | \\guilsinglleft      | ‹       |
+    | \\guilsinglright     | ›       |
+    | \\textreferencemark  | ※       |
+    | \\textinterrobang    | ‽       |
+    | \\textoverline       | ‾       |
+    | \\langle             | ⟨       |
+    | \\rangle             | ⟩       |
+
+
+  Scenarios: Punctuation symbols
+    | latex   | unicode | description |
+    | -       | -       | hyphen      |
+    | --      | –       | en-dash     |
+    | ---     | —       | em-dash     |
+    | \\~{}   | ~       | tilde       |
+
+  Scenarios: Quotation marks
+    | latex   | unicode | description         |
+    | ``      | “       | left double quotes  |
+    | ''      | ”       | right double quotes |
+    | `       | ‘       | left single quotes  |
+    | '       | ’       | right single quotes |

data/features/special_characters.feature

@@ -0,0 +1,20 @@
+Feature: Decode LaTeX special characters
+  As a hacker who works with LaTeX
+  I want to be able to decode a few special characters which are escaped by LaTeX
+
+  Scenario Outline: LaTeX to Unicode transformation
+    When I decode the string '<latex>'
+    Then the result should be '<unicode>'
+
+  Scenarios: Special characters
+    | latex                | unicode |
+    | \\\&                 | &       |
+    | \\#                  | #       |
+    | \\$                  | $       |
+    | \\%                  | %       |
+    | \\{                  | {       |
+    | \\}                  | }       |
+    | \\_                  | _       |
+    | \\textasciitilde{}   | ~       |
+    | \\textbackslash{}    | \\      |
+    | \\textasciicircum{}  | ^       |

data/features/step_definitions/latex.rb

@@ -0,0 +1,7 @@
+When /^I decode the string ('|")(.*)\1$/ do |quote,string|
+  @result = LaTeX.decode(string)
+end
+
+Then /^the result should be ('|")(.*)\1$/ do |quote,value|
+  @result.should == value
+end

data/features/support/env.rb

@@ -0,0 +1 @@
+require 'latex/decode'

data/features/umlauts.feature

@@ -0,0 +1,11 @@
+Feature: Decode LaTeX umlauts
+	As a hacker who works with LaTeX
+	I want to be able to decode LaTeX umlauts
+	
+  Scenario: Lowercase german umlauts
+    When I decode the string '\"a\"o\"u'
+    Then the result should be 'äöü'
+
+  Scenario: Uppercase german umlauts
+    When I decode the string '\"A\"O\"U'
+    Then the result should be 'ÄÖÜ'

data/latex-decode.gemspec

@@ -0,0 +1,39 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib/', __FILE__)
+$:.unshift lib unless $:.include?(lib)
+
+require 'latex/decode/version'
+
+Gem::Specification.new do |s|
+  s.name        = 'latex-decode'
+  s.version     = LaTeX::Decode::VERSION.dup
+  s.authors     = ['Sylvester Keil']
+  s.email       = ['http://sylvester.keil.or.at']
+  s.homepage    = 'http://github.com/inukshuk/latex-decode'
+  s.summary     = 'Decodes LaTeX to Unicode.'
+  s.description = 'Decodes strings formatted in LaTeX to equivalent Unicode strings.'
+  s.license     = 'GPL-3'
+
+  if RUBY_PLATFORM =~ /java/
+    s.platform    = 'java'
+  else
+    s.add_dependency('unicode', '~> 0.4')
+    s.platform    = 'ruby'
+  end
+
+  s.add_development_dependency('rake', '~> 0.8')
+  s.add_development_dependency('bundler', '~> 1.0')
+  s.add_development_dependency('rdoc', '~> 3.6')
+  s.add_development_dependency('rspec', '~> 2.6')
+  s.add_development_dependency('cucumber', "~> 1.0")
+
+  s.files        = `git ls-files`.split("\n")
+  s.test_files   = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables  = []
+  s.require_path = 'lib'
+
+  s.rdoc_options     = %w{--line-numbers --inline-source --title "LaTeX-Decode\ Documentation" --main README.md --webcvs=http://github.com/inukshuk/latex-decode/tree/master/}
+  s.extra_rdoc_files = %w{README.md LICENSE}
+end
+
+# vim: syntax=ruby

data/lib/latex/decode.rb

@@ -0,0 +1,49 @@
+#--
+# LaTeX::Decode
+# Copyright (C) 2011 Sylvester Keil <sylvester.keil.or.at>
+# Copyright (C) 2010 François Charette
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#++
+
+require 'latex/decode/version'
+require 'latex/decode/compatibility'
+require 'latex/decode/base'
+
+require 'latex/decode/accents'
+require 'latex/decode/diacritics'
+require 'latex/decode/punctuation'
+require 'latex/decode/symbols'
+
+module LaTeX
+  
+  class << self
+    def decode (string)
+      return string unless string.respond_to?(:to_s)
+
+      string = string.is_a?(String) ? string.dup : string.to_s
+      
+      Decode::Base.normalize(string)
+      
+      Decode::Accents.decode!(string)
+      Decode::Diacritics.decode!(string)
+      Decode::Punctuation.decode!(string)
+      Decode::Symbols.decode!(string)
+      
+      Decode::Base.strip_braces(string)
+      
+      LaTeX.normalize_C(string)
+    end
+  end
+end

data/lib/latex/decode/accents.rb

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+
+module LaTeX
+  module Decode
+    
+    class Accents < Decoder
+      @map = Hash[*%W{
+        `  \u0300
+        '  \u0301
+        ^  \u0302
+        ~  \u0303
+        =  \u0304
+        .  \u0307
+        '' \u0308
+        "  \u0308  
+      }.map { |s| LaTeX.to_unicode(s) }].freeze
+
+      @patterns = [
+        ruby_18 {
+          /\\(#{ map.keys.map { |k| Regexp.escape(k) }.join('|') })\{([[:alpha:]]+)\}/ou
+        } ||
+        ruby_19 {
+          /\\(#{ map.keys.map { |k| Regexp.escape(k) }.join('|') })\{(\p{L}\p{M}*)\}/ou
+        },
+        ruby_18 {
+          /\\(#{ map.keys.map { |k| Regexp.escape(k) }.join('|') })([[:alpha:]])/ou          
+        } ||
+        ruby_19 {
+          /\\(#{ map.keys.map { |k| Regexp.escape(k) }.join('|') })(\p{L}\p{M}*)/ou
+        }
+      ].freeze
+      
+    end
+
+  end
+end

data/lib/latex/decode/base.rb

@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+
+module LaTeX
+  module Decode
+        
+    class Decoder
+      class << self
+        attr_reader :patterns, :map
+    
+        def inherited (base)
+          subclasses << base
+        end
+        
+        def subclasses
+          @subclasses ||= []
+        end
+        
+        def decode (string)
+          decode!(string.dup)
+        end
+        
+        def decode! (string)
+          puts name unless patterns
+          patterns.each do |pattern|
+            string.gsub!(pattern) { |m| [$2,map[$1],$3].compact.join }
+          end
+          string
+        end
+      end
+    end
+    
+    module Base
+      
+      module_function
+      
+      def normalize (string)
+        string.gsub!(/\\(?:i|j)\b/) { |m| m == '\\i' ? 'ı' : 'ȷ' }
+
+				# \foo\ bar -> \foo{} bar
+        string.gsub!(/(\\[a-zA-Z]+)\\(\s+)/, '\1{}\2')
+
+				# Aaaa\o, -> Aaaa\o{},
+        string.gsub!(/([^{]\\\w)([;,.:%])/, '\1{}\2')
+
+				# \c cb -> \c{cb}
+        string.gsub!(/(\\[^\sij&#\$\{\}_~%])\s+([[:alpha:]]+)\b/i, '\1{\2}')
+				
+        string
+      end
+      
+      def strip_braces (string)
+        string.gsub!(/(^|[^\\])([\{\}]+)/, '\1')
+        string.gsub!(/\\(\{|\})/, '\1')
+        string
+      end
+      
+    end
+    
+  end
+end

data/lib/latex/decode/compatibility.rb

@@ -0,0 +1,59 @@
+
+if RUBY_VERSION < "1.9"
+  $KCODE = 'U'
+  
+  module LaTeX
+    def self.to_unicode (string)
+      string.gsub(/\\?u([\da-f]{4})/i) { |m| [$1.to_i(16)].pack('U') }
+    end
+  end
+    
+  def ruby_18; yield; end
+  def ruby_19; false; end
+else  
+
+  module LaTeX
+    def self.to_unicode (string); string; end
+  end
+
+  def ruby_18; false; end
+  def ruby_19; yield; end
+end
+
+if RUBY_PLATFORM == 'java'
+  require 'java'
+  
+  # Use the Java native Unicode normalizer
+  module LaTeX
+    def self.normalize_C(string)
+      java.text.Normalizer.normalize(string, java.text.Normalizer::Form::NFC).to_s
+    end
+  end
+  
+else
+
+	begin
+  	require 'unicode'
+
+    # Use the Unicode gem
+    module LaTeX
+      def self.normalize_C(string)
+        Unicode::normalize_C(string)
+      end
+    end
+	rescue LoadError
+    begin
+      require 'active_support/multibyte/chars'
+      
+      # Use ActiveSupport's normalizer
+      module LaTeX
+        def self.normalize_C(string)
+          ActiveSupport::Multibyte::Chars.new(string).normalize(:c).to_s
+        end
+      end
+    rescue LoadError
+  		fail "Failed to load unicode normalizer: please gem install unicode (or active_support)"
+    end
+	end
+
+end