abroad 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 83e4de2f30d6fd16a710df079d0bc64f3f76da4c
+  data.tar.gz: 9afb235066836270d5eace94db8d5a815ed2c538
+SHA512:
+  metadata.gz: aa9ccf3bee54d01500b3ecf996d8dcaeed5ae6fb0a27de1adc7d1a31c63797ccd1083432ee5d03132358aba65f7a721282a25947beda6bdf063c17b47bcedd0d
+  data.tar.gz: 5db0947ad41a62c6d3efeeb5ed11e7705fee237eeeebd6e524b7d77418a95b87d5788fac61479f36786f952e2ba85f22fde845c4eff870284d5e91670a237230

data/README.md

@@ -0,0 +1,142 @@
+[![Build Status](https://travis-ci.org/camertron/abroad.svg?branch=master)](https://travis-ci.org/camertron/abroad)
+
+abroad
+====================
+
+A set of tools for serializing and extracting strings to and from a number of localization file formats. Currently supported formats are:
+
+1. YAML, both plain and Rails-style
+2. Android XML
+3. JSON key/value
+
+Adding additional extractors and serializers is straightforward; skip to the bottom of this document to learn more.
+
+## Installation
+
+`gem install abroad`, or add it to your Gemfile.
+
+Then, somewhere in your project:
+
+```ruby
+require 'abroad'
+```
+
+## Introduction
+
+Most application frameworks specify a way to localize (i.e. translate) UI phrases and other content. Usually this is done via flat, static files that map strings written in a source langage to translations written in any number of target languages. In Ruby on Rails, this is done via YAML files stored in the config/locales directory. Each file contains a series of nested key/value pairs, where the key is a machine-readable, globally unique identifier and the value is a human-readable bit of text meant to be displayed to users of the application. English strings go in config/locales/en.yml, Spanish strings go in config/locales/es.yml and so on. Both en.yml and es.yml contain the same set of keys, but different (translated) values for those keys.
+
+Localization file formats are usually based on some standard format like YAML, but often extended in unique ways specific to the framework or platform. Interpreting these files can be difficult because of the various edge cases and platform-specific expectations. If you ever find yourself needing to parse or write out compatible files, consider using well-tested tools like the ones in this project.
+
+## Usage
+
+Abroad provides extractors for reading keys and values from localization files, and serializers for writing them out. The usage for each is slightly different.
+
+### Extractors
+
+Let's say you're working with this Rails YAML file:
+
+```yaml
+en:
+  welcome:
+    message: hello
+  goodbye:
+    message: goodbye
+```
+
+To extract strings from this file, try something like this:
+
+```ruby
+Abroad.extractor('yaml/rails').open('/path/to/en.yml') do |extractor|
+  extractor.extract_each do |key, string|
+    # on first iteration, key == 'welcome.message', string == 'hello'
+    # on second iteration, key == 'goodbye.message', string == 'goodbye'
+  end
+end
+```
+
+The `Abroad.extractor` method returns a registered extractor class, or `nil` if the extractor can't be found. Extractor classes respond to `open`, `from_stream`, and `from_string`, and can be called with or without a block. If passed a block, the file or stream will be automatically closed when the block terminates. If you choose to not pass a block, you're responsible for calling `close` yourself.
+
+Here's an example with all the steps broken down:
+
+```ruby
+extractor_klass = Abroad.extractor('yaml/rails')
+extractor = extractor_klass.open('/path/to/en.yml')
+extractor.extract_each do |key, string|
+  ...
+end
+extractor.close
+```
+
+The `extract_each` method on extractor instances returns an enumerable, which means you have access to all the wonderful `Enumerable` methods like `map`, `inject`, etc:
+
+```ruby
+Abroad.extractor('yaml/rails').open('/path/to/en.yml') do |extractor|
+  extractor.extract_each.with_object({}) do |(key, string), result|
+    result[key] = string
+  end
+end
+```
+
+To get a list of all available extractors, use the `extractors` method:
+
+```ruby
+Abroad.extractors  # => ["yaml/rails", "xml/android", ...]
+```
+
+### Serializers
+
+While extractors pull strings out of localization files, serializers write them back in. Serializers conform to a similar interface, but offer different methods to write content out to the stream:
+
+```ruby
+Abroad.serializer('yaml/rails').open('/path/to/es.yml') do |serializer|
+  serializer.write_key_value('welcome.message', 'hola')
+  serializer.write_key_value('goodbye.message', 'adios')
+end
+```
+
+In addition to `write_key_value`, serializer instances respond to the `write_raw` method, which is capable of writing raw text to the underlying stream. You might use this method if you needed to write a comment to the file or maybe a preamble at the beginning.
+
+Serializer classes respond to `from_stream` in addition to `open`. Both methods can be called with or without a block. If passed a block, the file or stream will be automatically closed when the block terminates. If you choose to not pass a block, you're responsible for calling `close` yourself.
+
+Here's an example with all the steps broken down:
+
+```ruby
+serializer_klass = Abroad.serializer('yaml/rails')
+serializer = serializer_klass.open('/path/to/es.yml')
+serializer.write_key_value('welcome.message', 'hola')
+serializer.write_key_value('goodbye.message', 'adios')
+serializer.close
+```
+
+To get a list of all available serializers, use the `serializers` method:
+
+```ruby
+Abroad.serializers  # => ["yaml/rails", "xml/android", ...]
+```
+
+### Writing Your Own
+
+Conformant _extractors_ should inherit from `Abroad::Extractors::Extractor` and need to define the method `extract_each`. See lib/abroad/extractors/extractor.rb for a quick look at the interface. Methods that raise `NotImplementedError`s are ones you need to define in your subclass.
+
+Conformant _serializers_ should inherit from `Abroad::Serializers::Serializer` and need to define the `write_key_value` and `write_raw` methods. See lib/abroad/serializers/serializer.rb for a quick look at the interface. Methods that raise `NotImplementedError`s are ones you need to define in your subclass.
+
+Once you've finished writing your extractor or serializer, register it with Abroad:
+
+```ruby
+Abroad::Extractors.register('strings/ios', Strings::IosExtractor)
+Abroad::Serializers.register('strings/ios', Strings::IosSerializer)
+```
+
+The first argument to the `register` method is called the serializer or extractor's _id_. The id can really be anything you want, but Abroad has established a convention of format/framework. The format is the underlying file format (eg. json, yaml, xml, etc), and the framework is the platform or application framework you're targeting (eg. iOS, Android, Rails, Django, etc). This makes it easy to avoid writing "one size fits all" classes. For example, it would be straightforward to add support for Chrome's localization file format, which is just json with a special structure. We might register an extractor with the id json/chrome instead of trying to retrofit our existing json extractor with Chrome-specific functionality.
+
+## Requirements
+
+This project has no external requirements.
+
+## Running Tests
+
+`bundle exec rake` or `bundle exec rspec` should do the trick.
+
+## Authors
+
+* Cameron C. Dutro: http://github.com/camertron

data/abroad.gemspec

@@ -0,0 +1,25 @@
+$:.unshift File.join(File.dirname(__FILE__), 'lib')
+require 'abroad/version'
+
+Gem::Specification.new do |s|
+  s.name     = 'abroad'
+  s.version  = ::Abroad::VERSION
+  s.authors  = ['Cameron Dutro']
+  s.email    = ['camertron@gmail.com']
+  s.homepage = "https://github.com/camertron/abroad"
+
+  s.description = s.summary = 'A set of parsers and serializers for dealing with localization file formats.'
+
+  s.platform = Gem::Platform::RUBY
+  s.has_rdoc = true
+
+  s.add_dependency 'htmlentities', '~> 4.0'
+  s.add_dependency 'json-stream', '~> 0.0'
+  s.add_dependency 'json-write-stream', '~> 1.0'
+  s.add_dependency 'nokogiri', '~> 1.0'
+  s.add_dependency 'xml-write-stream', '~> 1.0'
+  s.add_dependency 'yaml-write-stream', '~> 1.0'
+
+  s.require_path = 'lib'
+  s.files = Dir['{lib,spec}/**/*', 'README.md', 'abroad.gemspec']
+end

data/lib/abroad/extractors/extractor.rb

@@ -0,0 +1,46 @@
+require 'stringio'
+
+module Abroad
+  module Extractors
+
+    class Extractor
+      attr_reader :stream
+
+      class << self
+        def from_stream(stream)
+          extractor = new(stream)
+
+          if block_given?
+            yield(extractor).tap do
+              extractor.close
+            end
+          else
+            extractor
+          end
+        end
+
+        def from_string(string, &block)
+          from_stream(StringIO.new(string), &block)
+        end
+
+        def open(file, &block)
+          from_stream(File.open(file, 'r'), &block)
+        end
+      end
+
+      def initialize(stream)
+        @stream = stream
+      end
+
+      def extract_each
+        raise NotImplementedError,
+          'expected to be implemented in derived classes'
+      end
+
+      def close
+        stream.close
+      end
+    end
+
+  end
+end

data/lib/abroad/extractors/json/json_extractor.rb

@@ -0,0 +1,17 @@
+module Abroad
+  module Extractors
+    module Json
+
+      class JsonExtractor < Extractor
+        def extract_each(&block)
+          if block_given?
+            each_entry(&block)
+          else
+            to_enum(__method__)
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/abroad/extractors/json/key_value_extractor.rb

@@ -0,0 +1,38 @@
+require 'json/stream'
+
+module Abroad
+  module Extractors
+    module Json
+
+      class KeyValueExtractor < JsonExtractor
+        private
+
+        def each_entry
+          open_obj_count = 0
+          open_array_count = 0
+          key = nil
+
+          parser = ::JSON::Stream::Parser.new.tap do |parser|
+            parser.key { |key_str| key = key_str }
+
+            parser.value do |value_str|
+              if block_given? && open_array_count.zero? && open_obj_count == 1
+                yield key, value_str
+              end
+            end
+
+            parser.start_object { open_obj_count += 1 }
+            parser.end_object { open_obj_count -= 1 }
+            parser.start_array { open_array_count += 1 }
+            parser.end_array { open_array_count -= 1 }
+          end
+
+          parser << stream.read
+        rescue ::JSON::Stream::ParserError => e
+          raise Abroad::SyntaxError, "Syntax error in json: #{e.message}"
+        end
+      end
+
+    end
+  end
+end

data/lib/abroad/extractors/json.rb

@@ -0,0 +1,8 @@
+module Abroad
+  module Extractors
+    module Json
+      autoload :KeyValueExtractor, 'abroad/extractors/json/key_value_extractor'
+      autoload :JsonExtractor,     'abroad/extractors/json/json_extractor'
+    end
+  end
+end

data/lib/abroad/extractors/xml/android_extractor.rb

@@ -0,0 +1,113 @@
+require 'nokogiri'
+require 'ext/htmlentities/android_xml_decoder'
+
+module Abroad
+  module Extractors
+    module Xml
+
+      class AndroidExtractor < XmlExtractor
+        private
+
+        def each_entry(&block)
+          doc = parse
+          each_string_entry(doc, &block)
+          each_array_entry(doc, &block)
+          each_plural_entry(doc, &block)
+        end
+
+        def each_string_entry(doc)
+          doc.xpath('//string').each do |node|
+            yield name_from(node), text_from(node)
+          end
+        end
+
+        def each_array_entry(doc)
+          doc.xpath('//string-array').each do |array|
+            prefix = name_from(array)
+
+            array.xpath('item').each_with_index do |item, idx|
+              yield "#{prefix}.#{idx}", text_from(item)
+            end
+          end
+        end
+
+        def each_plural_entry(doc)
+          doc.xpath('//plurals').each do |plurals|
+            prefix = name_from(plurals)
+
+            plurals.xpath('item').each do |item|
+              quantity = item.attributes['quantity'].value
+              yield "#{prefix}.#{quantity}", text_from(item)
+            end
+          end
+        end
+
+        def text_from(node)
+          builder = Nokogiri::XML::Builder.new do |builder|
+            builder.root do
+              node.children.each do |child|
+                serialize(child, builder)
+              end
+            end
+          end
+
+          # safe to call `strip` after `to_xml` because any string that
+          # needs leading or trailing whitespace preserved should be wrapped
+          # in double quotes
+          unescape(
+            strip_enclosing_quotes(
+              builder.doc.xpath('/root/node()').to_xml.strip
+            )
+          )
+        end
+
+        def serialize(node, builder)
+          if node.text?
+            builder.text(unescape(node.text))
+          else
+            builder.send("#{node.name}_", node.attributes) do
+              node.children.each do |child|
+                serialize(child, builder)
+              end
+            end
+          end
+        end
+
+        def name_from(node)
+          if attribute = node.attributes['name']
+            attribute.value
+          end
+        end
+
+        def unescape(text)
+          text = text
+            .gsub("\\'", "'")
+            .gsub('\\"', '"')
+            .gsub("\\n", "\n")
+            .gsub("\\r", "\r")
+            .gsub("\\t", "\t")
+
+          coder.decode(text)
+        end
+
+        def coder
+          @coder ||= HTMLEntities::AndroidXmlDecoder.new
+        end
+
+        def strip_enclosing_quotes(text)
+          quote = case text[0]
+            when "'", '"'
+              text[0]
+          end
+
+          if quote
+            text.gsub(/\A#{quote}(.*)#{quote}\z/) { $1 }
+          else
+            text
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/abroad/extractors/xml/xml_extractor.rb

@@ -0,0 +1,28 @@
+require 'nokogiri'
+
+module Abroad
+  module Extractors
+    module Xml
+
+      class XmlExtractor < Extractor
+        def extract_each(&block)
+          if block_given?
+            each_entry(&block)
+          else
+            to_enum(__method__)
+          end
+        end
+
+        private
+
+        def parse
+          Nokogiri::XML(stream) do |config|
+            # don't allow network connections
+            config.options = Nokogiri::XML::ParseOptions::NONET
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/abroad/extractors/xml.rb

@@ -0,0 +1,8 @@
+module Abroad
+  module Extractors
+    module Xml
+      autoload :AndroidExtractor, 'abroad/extractors/xml/android_extractor'
+      autoload :XmlExtractor,     'abroad/extractors/xml/xml_extractor'
+    end
+  end
+end

data/lib/abroad/extractors/yaml/dotted_key_extractor.rb

@@ -0,0 +1,40 @@
+module Abroad
+  module Extractors
+    module Yaml
+
+      class DottedKeyExtractor < YamlExtractor
+        def extract_each(&block)
+          if block_given?
+            walk(parse, [], &block)
+          else
+            to_enum(__method__)
+          end
+        end
+
+        private
+
+        def walk(obj, cur_path, &block)
+          case obj
+            when Hash
+              obj.each_pair do |key, val|
+                walk(val, cur_path + [key], &block)
+              end
+            when Array
+              obj.each_with_index do |val, idx|
+                walk(val, cur_path + [idx.to_s], &block)
+              end
+            else
+              yield scrub_path(cur_path).join('.'), obj
+          end
+        end
+
+        private
+
+        def scrub_path(path)
+          path  # no-op
+        end
+      end
+
+    end
+  end
+end

data/lib/abroad/extractors/yaml/jruby_compat.rb

@@ -0,0 +1,170 @@
+require 'yaml'
+
+java_import 'java.lang.StringBuilder'
+
+module Abroad
+  module Extractors
+    module Yaml
+      # For some reason, Psych in MRI thinks it's valid yaml syntax to escape
+      # single quotes (surprise, it's not). This class is a performant way to
+      # identify and correct such invalid escaping. Be warned, it's a fairly
+      # naïve implementation.
+      #
+      # NOTE: this technique will only work with YAML emitted in block mode,
+      # i.e. yaml that uses indentation and newlines for each key/value pair
+      # and array element
+      class JRubyCompat
+
+        UNKNOWN_ESCAPE_MSG = "found unknown escape character '(39)"
+        UNEXPECTED_EOS_MSG = 'found unexpected end of stream while scanning a quoted scalar'
+
+        class << self
+          def clean(yaml_content)
+            # convert to java string for performance reasons (don't have to
+            # coerce/convert from java to ruby and back again)
+            # https://github.com/jruby/jruby/wiki/ImprovingJavaIntegrationPerformance#pre-coerce-values-used-repeatedly
+            yaml_content = yaml_content.to_java
+            ranges = identify_ranges(yaml_content)
+            reconstruct(yaml_content, ranges)
+          end
+
+          private
+
+          # given a list of range/substitution pairs, reconstruct the yaml_content
+          # string so it contains the substitutions
+          def reconstruct(yaml_content, ranges)
+            builder = StringBuilder.new
+            index = 0
+
+            ranges.each do |range|
+              builder.append(yaml_content, index, range.first.first)
+              builder.append(range.last)
+              index = range.first.last
+            end
+
+            builder.append(yaml_content, index, yaml_content.length)
+            builder.toString
+          end
+
+          # Find the problem areas in yaml_content. Returns a list of range/substitution
+          # pairs of the form [[start, finish], substitution]
+          def identify_ranges(yaml_content)
+            each_line_indices(yaml_content).each_with_object([]) do |(start, finish), ranges|
+              if range = clean_line(yaml_content, start, finish)
+                ranges << range
+              end
+            end
+          end
+
+          def clean_line(yaml_content, start, finish)
+            unless array_element?(yaml_content, start, finish)
+              value_start, value_finish = find_value(yaml_content, start, finish)
+
+              if value_start && double_quoted?(yaml_content, value_start, value_finish)
+                clean_value(yaml_content, value_start, value_finish)
+              end
+            else
+              clean_value(yaml_content, start, finish)
+            end
+          end
+
+          # if the yaml parser (snakeyaml in jruby's case) can parse the value,
+          # return nil. If not, attempt to fix the problem by returning
+          # a range/substitution pair.
+          def clean_value(yaml_content, start, finish)
+            value = yaml_content.substring(start, finish)
+
+            begin
+              YAML.load(value)
+              nil
+            rescue Psych::SyntaxError => e
+              if error_type = error_type_for(e)
+                fix(error_type, yaml_content, start, finish, value)
+              else
+                raise e
+              end
+            end
+          end
+
+          # naïvely try to fix the problem by replacing escaped single quotes
+          def clean_string(value)
+            value.gsub("\\'", "'")
+          end
+
+          def fix(error_type, yaml_content, start, finish, value)
+            # don't do anything with error type for now
+            [[start, finish], clean_string(value)]
+          end
+
+          def error_type_for(e)
+            if e.message.include?(UNKNOWN_ESCAPE_MSG)
+              :unknown_escape_sequence
+            elsif e.message.include?(UNEXPECTED_EOS_MSG)
+              :unexpected_eos
+            else
+              :unknown
+            end
+          end
+
+          # rather naïvely isolates the value in a yaml key/value pair by searching
+          # for the first colon
+          def find_value(yaml_content, start, finish)
+            value_start = yaml_content.indexOf(':', start)
+
+            if value_start <= finish
+              [value_start + 1, finish]
+            else
+              [nil, nil]
+            end
+          end
+
+          # is this chunk of text double-quoted?
+          def double_quoted?(yaml_content, start, finish)
+            starts_with?('"'.ord, yaml_content, start, finish)
+          end
+
+          # is this line of yaml an array element, i.e. does it begin with "-"?
+          def array_element?(yaml_content, start, finish)
+            starts_with?('-'.ord, yaml_content, start, finish)
+          end
+
+          def starts_with?(charcode, yaml_content, start, finish)
+            (start..finish).each do |index|
+              return false if index >= yaml_content.length
+              unless yaml_content.charAt(index) == 32
+                break yaml_content.charAt(index) == charcode
+              end
+            end
+          end
+
+          # Iterates over the yaml string and yields ranges that encapsulate
+          # each "line". Expects lines to be delimited by newlines ("\n").
+          def each_line_indices(yaml_content)
+            if block_given?
+              index = 0
+
+              loop do
+                next_index = yaml_content.indexOf("\n", index)
+
+                if next_index == -1
+                  if index < yaml_content.length
+                    yield index + 1, yaml_content.length
+                  end
+
+                  break
+                else
+                  yield index, next_index + 1
+                end
+
+                index = next_index + 1
+              end
+            else
+              to_enum(__method__, yaml_content)
+            end
+          end
+        end
+
+      end
+    end
+  end
+end

data/lib/abroad/extractors/yaml/rails_extractor.rb

@@ -0,0 +1,15 @@
+module Abroad
+  module Extractors
+    module Yaml
+
+      class RailsExtractor < DottedKeyExtractor
+        private
+
+        def scrub_path(path)
+          path[1..-1]
+        end
+      end
+
+    end
+  end
+end