i18nliner 0.0.1

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2013 Jon Jensen
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,320 @@
+# I18nliner
+
+yay readme-driven development!
+
+## TODO
+
+* inferred placeholders (instance vars and methods)
+* ERB pre-processor
+  * wrapper inference
+  * helper/placeholder extraction
+* rake tasks
+  * dump
+  * diff
+  * import
+
+====
+
+I18nliner is I18n made simple.
+
+No .yml files. Inline defaults. Optional keys. Inferred interpolation values.
+Wrappers and blocks, so your templates look template-y and your translations
+HTML-free.
+
+## TL;DR
+
+I18nliner lets you do stuff like this:
+
+    t "Ohai %{@user.name}, my default translation is right here in the code. " +
+      "Inferred keys and placeholder values, oh my!"
+
+and even this:
+
+    <%= t do %>
+      Hey <%= amigo %>!
+      Although I am <%= link_to "linking to something", random_path %> and
+      have some <strong>bold text</strong>, the translators will see
+      <strong><em>absolutely no markup</em></strong> and will only have a
+      single string to translate :o
+    <% end %>
+
+## Installation
+
+Add the following to your Gemfile:
+
+    gem 'i18nliner'
+
+## Features
+
+### No more en.yml
+
+Instead of maintaining .yml files and doing stuff like this:
+
+    I18n.t :account_page_title
+
+Forget the .yml and just do:
+
+    I18n.t :account_page_title, "My Account"
+
+Regular I18n options follow the (optional) default translation, so you can do
+the usual stuff (placeholders, etc.).
+
+#### Okay, but don't the translators need en.yml?
+
+Sure, but *you* don't need to write it. Just run:
+
+    rake i18nliner:dump
+
+This extracts all default translations from your codebase, merges them with any
+other ones (from rails or pre-existing .yml files), and outputs them to
+`config/locales/generated/en.yml` (or rather, `"#{I18n.default_locale}.yml"`).
+
+### It's okay to lose your keys
+
+Why waste time coming up with keys that are less descriptive than the default
+translation? I18nliner makes keys optional, so you can just do this:
+
+    I18n.t "My Account"
+
+I18nliner will create a [unique key](CONFIG.md) based on the translation (e.g.
+`:my_account`), so you don't have to.
+
+This can actually be a **good thing**, because when the `en` changes, the key
+changes, which means you know you need to get it retranslated (instead of
+letting a now-inaccurate translation hang out indefinitely). Whether you want
+to show "[ missing translation ]" or the `en` value in the meantime is up to
+you.
+
+### Inferred Interpolation Values
+
+Interpolation values may be inferred by I18nliner if not provided. So long as
+it's an instance variable or method (or chain), you don't need to specify its
+value. So this:
+
+    <p>
+      <%= t "Hello, %{user}. This request was a %{request_method}.", 
+            :user => @user.name,
+            :request_method => request.method
+      %>
+    </p>
+
+Can just be this:
+
+    <p>
+      <%= t "Hello, %{@user.name}. This request was a %{request.method}." %>
+    </p>
+
+Note that local variables cannot be inferred.
+
+### Wrappers and Blocks
+
+#### The Problem
+
+Suppose you have something like this in your ERB:
+
+    <p>
+      You can <%= link_to "lead", new_discussion_path %> a new discussion or
+      <%= link_to "join", discussion_search_path %> an existing one.
+    </p>
+
+You might try something like this:
+
+    <p>
+      <%= t "You can %{lead} a new discussion or %{join} an existing one.",
+            :lead => link_to(t("lead"), new_discussion_path),
+            :join => link_to(t("join"), discussion_search_path)
+      %>
+    </p>
+
+This is not great, because:
+
+1. There are three strings to translate.
+2. When translating the verbs, the translator has no context for where it's
+   being used... Is "lead" a verb or a noun?
+3. Translators have their hands somewhat tied as far as what is inside the
+   links and what is not.
+
+So you might try this instead:
+
+    <p>
+      <%= t :discussion_html,
+            "You can <a href="%{lead_url}">lead</a> a new discussion or " +
+            "<a href="%{join_url}">join</a> an existing one.",
+            :lead_url => new_discussion_path,
+            :join_url => discussion_search_path
+      %>
+    </p>
+
+This isn't much better, because now you have HTML in your translations. If you
+want to add a class to the link, you have to go update all the translations.
+A translator could accidentally break your page (or worse, cross-site script
+it).
+
+So what do you do?
+
+#### Wrappers
+
+I18nliner lets you specify wrappers, so you can keep HTML out the translations,
+while still just having a single string needing translation:
+
+    <p>
+      <%= t "You can *lead* a new discussion or **join** an existing one.",
+            :wrappers => [
+              link_to('\1', new_discussion_path),
+              link_to('\1', discussion_search_path)
+            ]
+      %>
+    </p>
+
+Default delimiters are increasing numbers of asterisks, but you can specify
+any string as a delimiter by using a hash rather than an array.
+
+#### Blocks
+
+But wait, there's more!
+
+Perhaps you want your templates to look like, well, templates. Try this:
+
+    <p>
+      <%= t do %>
+        Welcome to the internets, <%= user.name %>
+      <% end %>
+    </p>
+
+Or even this:
+
+    <p>
+      <%= t do %>
+        <b>Ohai <%= user.name %>,</b>
+        you can <%= link_to "lead", new_discussion_path %> a new discussion or
+        <%= link_to "join", discussion_search_path %> an existing one.
+      <% end %>
+    </p>
+
+In case you're curious about the man behind the curtain, I18nliner adds an ERB
+pre-processor that turns the second example into something like this right
+before it hits ERB:
+
+    <p>
+      <%= t :some_unique_key,
+            "*Ohai %{user_name}*, you can **lead** a new discussion or ***join*** an existing one.",
+            :user_name => @user.name,
+            :wrappers => [
+              '<b>\1</b>',
+              link_to('\1', new_discussion_path),
+              link_to('\1', discussion_search_path)
+            ]
+      %>
+    </p>
+
+In other words, it will infer wrappers from your (balanced) markup and
+[`link_to` calls](INFERRED_WRAPPERS.md), and will create placeholders for any
+other ERB expressions. ERB statements (e.g.
+`<% if some_condition %>...`) are *not* supported inside block translations, with
+the notable exception of nested translations, e.g.
+
+    <%= t do %>
+      Be sure to
+      <a href="/account/" title="<%= t do %>Account Settings<% end %>">
+        set up your account
+      </a>.
+    <% end %>
+
+#### HTML Safety
+
+I18nliner ensures translations, interpolated values, and wrappers all play
+nicely (and safely) when it comes to HTML escaping. If any translation,
+interpolated value, or wrapper is HTML-safe, everything else will be HTML-
+escaped.
+
+### Inline Pluralization Support
+
+Pluralization can be tricky, but [I18n gives you some flexibility](http://guides.rubyonrails.org/i18n.html#pluralization).
+I18nliner brings this inline with a default translation hash, e.g.
+
+    t({:one => "There is one light!", :other => "There are %{count} lights!"},
+      :count => picard.visible_lights.count)
+
+Note that the :count interpolation value needs to be explicitly set when doing
+pluralization. On the ERB side, I18nliner enhances pluralize to take a block,
+so you can do this:
+
+    <%= pluralize picard.visible_lights.count do %>
+      <% one do %>There is one light!<% end %>
+      <% other do %>There are <%= count %> lights!<% end %>
+    <% end %>
+
+If you just want to pluralize a single word, there's a shortcut:
+
+    t "person", :count => users.count
+
+This is equivalent to:
+
+    t({:one => "1 person", :other => "%{count} people"},
+      :count => users.count)
+
+I18nliner uses the pluralize helper to determine the default one/other values,
+so if your `I18n.default_locale` is something other than English, you may need
+to [add some inflections](https://gist.github.com/838188).
+
+## Rake Tasks
+
+### i18nliner:check
+
+Ensures that there are no problems with your translate calls (e.g. missing
+interpolation values, reusing a key for a different translation, etc.). **Go
+add this to your Jenkins/Travis tasks.**
+
+### i18nliner:dump
+
+Does an i18nliner:check, and then extracts all default translations from your
+codebase, merges them with any other ones (from rails or pre-existing .yml
+files), and outputs them to `config/locales/generated/en.yml`.
+
+### i18nliner:diff
+
+Does an i18nliner:dump and creates a diff from a previous one (path or git
+commit hash). This is useful if you only want to see what has changed since a
+previous release of your app.
+
+### i18nliner:import
+
+Imports a translated yml file. Ensures that all placeholders and wrappers are
+present.
+
+#### .i18nignore and more
+
+By default, the check and dump tasks will look for inline translations in any
+.rb or .erb files. You can tell it to always skip certain
+files/directories/patterns by creating a .i18nignore file. The syntax is the
+same as [.gitignore](http://www.kernel.org/pub/software/scm/git/docs/gitignore.html),
+though it supports
+[a few extra things](https://github.com/jenseng/globby#compatibility-notes).
+
+If you only want to check a particular file/directory/pattern, you can set the
+environment variable `ONLY` when you run the command, e.g.
+
+    rake i18nliner:check ONLY=/app/**/user*
+
+## Compatibility
+
+I18nliner is backwards compatible with I18n, so you can add it to an
+established (and already internationalized) Rails app. Your existing
+translation calls, keys and yml files will still just work without
+modification.
+
+I18nliner requires at least Ruby 1.9.3 and Rails 3.
+
+## What about JavaScript/Handlebars?
+
+Coming soon. I18nliner was inspired by some [I18n extensions](https://github.com/instructure/canvas-lms/tree/master/lib/i18n_extraction)
+I did in [Canvas-LMS](https://github.com/instructure/canvas-lms). While
+it also has the JavaScript/Handlebars equivalents, they are tightly
+coupled to Canvas-LMS and are written in Ruby. So basically we're talking
+a full reimplementation in JavaScript using
+[esprima](http://esprima.org/) instead of the [shameful, brittle, regexy hack](http://cdn.memegenerator.net/instances/400x/24091937.jpg)
+that is js_extractor.
+
+## License
+
+Copyright (c) 2013 Jon Jensen, released under the MIT license

data/Rakefile

@@ -0,0 +1,9 @@
+require 'rake'
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+task :default => :spec

data/lib/i18nliner/call_helpers.rb

@@ -0,0 +1,69 @@
+require 'iconv'
+require 'zlib'
+
+module I18nliner
+  module CallHelpers
+    ALLOWED_PLURALIZATION_KEYS = [:zero, :one, :few, :many, :other]
+    REQUIRED_PLURALIZATION_KEYS = [:one, :other]
+
+    def normalize_key(key, scope, receiver)
+      key = key.to_s
+      scope.normalize_key(key)
+    end
+
+    def normalize_default(default, translate_options = {})
+      default = infer_pluralization_hash(default, translate_options)
+      default.strip! if default.is_a?(String)
+      default
+    end
+
+    def infer_pluralization_hash(default, translate_options)
+      return default unless default.is_a?(String) &&
+                            default =~ /\A[\w\-]+\z/ &&
+                            translate_options.include?(:count)
+      {:one => "1 #{default}", :other => "%{count} #{default.pluralize}"}
+    end
+
+    def infer_key(default, translate_options = {})
+      default = default[:other].to_s if default.is_a?(Hash)
+      keyify(normalize_default(default, translate_options))
+    end
+
+    def keyify_underscored(string)
+      Iconv.iconv('ascii//translit//ignore', 'utf-8', string).
+        to_s.
+        downcase.
+        gsub(/[^a-z0-9_\.]+/, '_').
+        gsub(/\A_|_\z/, '')[0..50]
+    end
+
+    def keyify_underscored_crc32(string)
+      checksum = Zlib.crc32(string.size.to_s + ":" + string).to_s(16) 
+      keyify_underscored(string) + "_#{checksum}"
+    end
+
+    def keyify(string)
+      case I18nliner.inferred_key_format
+      when :underscored       then keyify_underscored(string)
+      when :underscored_crc32 then keyify_underscored_crc32(string)
+      else                         string
+      end
+    end
+
+    # Possible translate signatures:
+    #
+    # key [, options]
+    # key, default_string [, options]
+    # key, default_hash, options
+    # default_string [, options]
+    # default_hash, options
+    def key_provided?(scope, receiver, key_or_default = nil, default_or_options = nil, maybe_options = nil, *others)
+      return false if key_or_default.is_a?(Hash)
+      return true if key_or_default.is_a?(Symbol)
+      return true if default_or_options.is_a?(String)
+      return true if maybe_options
+      return true if I18nliner.look_up(normalize_key(key_or_default, scope, receiver))
+      false
+    end
+  end
+end

data/lib/i18nliner/commands/basic_formatter.rb

@@ -0,0 +1,13 @@
+module I18nliner
+  module Commands
+    module BasicFormatter
+      def red(text)
+        text
+      end
+
+      def green(text)
+        text
+      end
+    end
+  end
+end

data/lib/i18nliner/commands/check.rb

@@ -0,0 +1,59 @@
+module I18nliner
+  module Commands
+    class Check < GenericCommand
+      attr_reader :translations
+
+      def initialize(options)
+        super
+        @errors = []
+        @translations = TranslationHash.new(I18nliner.manual_translations)
+      end
+
+      def processors
+        @processors ||= I18nliner::Processors.all.map do |klass|
+          klass.new :only => @options[:only],
+                    :translations => @translations,
+                    :checker => method(:check_file)
+        end
+      end
+
+      def check_files
+        processors.each &:check_files
+      end
+
+      def check_file(file)
+        print green(".") if yield file
+      rescue SyntaxError, StandardError, ExtractionError
+        @errors << "#{$!}\n#{file}"
+        print red("F")
+      end
+
+      def failure
+        @errors.size > 0
+      end
+
+      def print_summary
+        translation_count = processors.sum(&:translation_count)
+        file_count = processors.sum(&:file_count)
+
+        print "\n\n"
+
+        @errors.each_with_index do |error, i|
+          puts "#{i+1})"
+          puts red(error)
+          print "\n"
+        end
+
+        print "Finished in #{Time.now - @start} seconds\n\n"
+        summary = "#{file_count} files, #{translation_count} strings, #{@errors.size} failures"
+        puts failure ? red(summary) : green(summary)
+      end
+
+      def run
+        check_files
+        print_summary
+        raise "check command encountered errors" if failure
+      end
+    end
+  end
+end

data/lib/i18nliner/commands/color_formatter.rb

@@ -0,0 +1,13 @@
+module I18nliner
+  module Commands
+    module ColorFormatter
+      def red(text)
+        "\e[31m#{text}\e[0m"
+      end
+
+      def green(text)
+        "\e[32m#{text}\e[0m"
+      end
+    end
+  end
+end

data/lib/i18nliner/commands/dump.rb

@@ -0,0 +1,8 @@
+module I18nliner
+  module Commands
+    class Dump < GenericCommand
+      def run
+      end
+    end
+  end
+end

data/lib/i18nliner/commands/generic_command.rb

@@ -0,0 +1,17 @@
+module I18nliner
+  module Commands
+    class GenericCommand
+      include BasicFormatter
+
+      def initialize(options)
+        @options = options
+        @start = Time.now
+        extend ColorFormatter if $stdout.tty?
+      end
+
+      def self.run(options)
+        new(options).run
+      end
+    end
+  end
+end

data/lib/i18nliner/errors.rb

@@ -0,0 +1,29 @@
+module I18nliner
+  class ExtractionError < StandardError
+    def initialize(line, detail = nil)
+      @line = line
+      @detail = detail
+    end
+
+    def to_s
+      error = self.class.name.humanize.sub(/ error\z/, '')
+      error = "#{error} on line #{@line}"
+      @detail ?
+        error + " (got #{@detail.inspect})" :
+        error
+    end
+  end
+
+  class InvalidSignatureError < ExtractionError; end
+  class MissingDefaultError < ExtractionError; end
+  class AmbiguousKeyError < ExtractionError; end
+  class InvalidPluralizationKeyError < ExtractionError; end
+  class MissingPluralizationKeyError < ExtractionError; end
+  class InvalidPluralizationDefaultError < ExtractionError; end
+  class MissingCountValueError < ExtractionError; end
+  class MissingInterpolationValueError < ExtractionError; end
+  class InvalidOptionsError < ExtractionError; end
+  class InvalidOptionKeyError < ExtractionError; end
+  class KeyAsScopeError < ExtractionError; end
+  class KeyInUseError < ExtractionError; end
+end

data/lib/i18nliner/extractors/abstract_extractor.rb

@@ -0,0 +1,33 @@
+module I18nliner
+  module Extractors
+    module AbstractExtractor
+      def initialize(options = {})
+        @scope = options[:scope] || ''
+        @translations = TranslationHash.new(options[:translations] || {})
+        @total = 0
+        super()
+      end
+
+      def look_up(key)
+        @translations[key]
+      end
+
+      def add_translation(full_key, default)
+        @total += 1
+        @translations[full_key] = default
+      end
+
+      def total_unique
+        @translations.total_size
+      end
+
+      def self.included(base)
+        base.instance_eval do
+          attr_reader :total
+          attr_accessor :translations, :scope
+        end
+      end
+    end
+  end
+end
+

data/lib/i18nliner/extractors/ruby_extractor.rb

@@ -0,0 +1,102 @@
+module I18nliner
+  module Extractors
+    class RubyExtractor < SexpProcessor
+      TRANSLATE_CALLS = [:t, :translate]
+      attr_reader :current_line
+
+      def initialize(sexps, scope)
+        @sexps = sexps
+        @scope = scope
+        super()
+      end
+
+      def each_translation(&block)
+        @block = block
+        process(@sexps)
+      end
+
+      def process_call(exp)
+        exp.shift
+        receiver = process(exp.shift)
+        receiver = receiver.last if receiver
+        method = exp.shift
+  
+        if extractable_call?(receiver, method)
+          @current_line = exp.line
+
+          # convert s-exps into literals where possible 
+          args = process_arguments(exp)
+
+          process_translate_call(receiver, method, args)
+        else
+          # even if this isn't a translate call, its arguments might contain
+          # one
+          process exp.shift until exp.empty?
+        end
+  
+        s
+      end
+
+     protected
+
+      def extractable_call?(receiver, method)
+        TRANSLATE_CALLS.include?(method) && (receiver.nil? || receiver == :I18n)
+      end
+
+      def process_translate_call(receiver, method, args)
+        call = TranslateCall.new(@scope, @current_line, receiver, method, args)
+        call.translations.each &@block
+      end
+
+     private
+
+      def process_arguments(args)
+        values = []
+        while arg = args.shift
+          values << evaluate_expression(arg)
+        end
+        values
+      end
+  
+      def evaluate_expression(exp)
+        if exp.sexp_type == :lit || exp.sexp_type == :str
+          exp.shift
+          return exp.shift
+        end
+        return string_from(exp) if string_concatenation?(exp)
+        return hash_from(exp) if exp.sexp_type == :hash
+        process(exp)
+        UnsupportedExpression
+      end
+  
+      def string_concatenation?(exp)
+        exp.sexp_type == :call &&
+        exp[2] == :+ &&
+        exp.last &&
+        exp.last.sexp_type == :str
+      end
+  
+      def string_from(exp)
+        exp.shift
+        lhs = exp.shift
+        exp.shift
+        rhs = exp.shift
+        if lhs.sexp_type == :str
+          lhs.last + rhs.last
+        elsif string_concatenation?(lhs)
+          string_from(lhs) + rhs.last
+        else
+          return UnsupportedExpression
+        end
+      end
+
+      def hash_from(exp)
+        exp.shift
+        values = exp.map{ |e| evaluate_expression(e) }
+        Hash[*values]
+      end
+    end
+
+    class UnsupportedExpression; end
+  end
+end