rosette-core 1.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1e9c9e17cffceee3657ed1dce1aba43c06e66873
+  data.tar.gz: 4b0170aa075e3ba248ccfe49dc8d042fb6578d35
+SHA512:
+  metadata.gz: c4fc6828341aafeed4ccfbd42034337c001ff0562f234d29f2d00fc296b10af0e74ee4fded6b8cb0f3498d7f5932a1f9edcff715f86af618936242947ff04788
+  data.tar.gz: 415b95e177ab76034b84f3262fa93c3d5c02c3b18552e2a9040dc06653036d1958b143ccd495ce376e12454f2a9f2c4c37fe5f2d616afed66ab1b6fea215d951

data/Gemfile

@@ -0,0 +1,26 @@
+source "https://rubygems.org"
+
+gemspec
+
+ruby '2.0.0', engine: 'jruby', engine_version: '1.7.15'
+
+gem 'rosette-datastore-memory', github: 'rosette-proj/rosette-datastore-memory'
+
+group :development, :test do
+  gem 'activemodel', '>= 3.2.20'
+  gem 'expert', '~> 1.0'
+  gem 'pry-nav'
+  gem 'rake'
+  gem 'repo-fixture'
+end
+
+group :development do
+  gem 'yard', '~> 0.8.0'
+end
+
+group :test do
+  gem 'codeclimate-test-reporter', require: nil
+  gem 'simplecov'
+  gem 'rspec'
+  gem 'rosette-test-helpers', github: 'rosette-proj/rosette-test-helpers'
+end

data/History.txt

@@ -0,0 +1,3 @@
+== 1.0.0
+
+* Birthday!

data/README.md

@@ -0,0 +1,94 @@
+rosette-core
+========
+
+[![Build Status](https://travis-ci.org/rosette-proj/rosette-core.svg?branch=master)](https://travis-ci.org/rosette-proj/rosette-core.svg?branch=master) [![Code Climate](https://codeclimate.com/github/rosette-proj/rosette-core/badges/gpa.svg)](https://codeclimate.com/github/rosette-proj/rosette-core) [![Test Coverage](https://codeclimate.com/github/rosette-proj/rosette-core/badges/coverage.svg)](https://codeclimate.com/github/rosette-proj/rosette-core/coverage)
+
+## Installation
+
+`gem install rosette-core`
+
+## Usage
+
+```ruby
+require 'rosette/core'
+```
+
+## Intro
+
+This repository contains the core classes for the Rosette internationalizaton framework. Full documentation can be found on [rubydoc.info](http://www.rubydoc.info/github/rosette-proj/rosette-core).
+
+Generally, this library is required by other projects like [rosette-server](https://github.com/rosette-proj/rosette-server), which means that most likely you'll make use of it indirectly. Please refer to the documentation that accompanies these other projects to set up and use Rosette.
+
+## Major Components
+
+What follows is a list of the major components in rosette-core. It's not meant to be an exhaustive list, but should give a decent overview of what's going on.
+
+### Commands
+
+Rosette commands are designed to mimic git operations. For example, use the `DiffCommand` to show the added/removed/changed phrases between two git refs. Use the `StatusCommand` to compute the status of a git ref i.e. the percent translated per locale. Commands all follow the builder pattern, meaning you create a blank instance, call setter methods for required fields, then call the `#execute` method. Here's an example for the `ShowCommand`:
+
+```ruby
+Rosette::Core::Commands::ShowCommand.new(rosette_config)
+  .set_repo_name('my_awesome_repo')
+  .set_ref('master')
+  .execute
+```
+
+### Git Classes
+
+Rosette manages git repos by leveraging the open-source [jGit library](https://eclipse.org/jgit), which is a full git implementation for the JVM. The `Repo` and `DiffFinder` classes wrap jGit and provide a set of methods for interacting with git repositories in a slightly more Ruby-ish way. Specifically, `Repo` offers generic methods like `#parents_of` and `#read_object_bytes`, while `DiffFinder` finds diffs between git refs. Here's an example of creating a `Repo` instance and calculating a diff:
+
+```ruby
+repo = Rosette::Core::Repo.from_path('/path/to/my_repo/.git')
+repo.diff('master', 'my_branch')  # you can optionally specify a list of paths as well
+```
+
+### Resolvers
+
+The resolvers provide a way of looking up class constants for a number of Rosette's sub-components using slash-separated strings. For example, the `ExtractorId` class will resolve the string 'xml/android' into `Rosette::Extractors::XmlExtractor::AndroidExtractor`. Resolvers exist for extractors, integrations, pre-processors, and serializers.
+
+```ruby
+Rosette::Core::ExtractorId.resolve('xml/android')
+Rosette::Core::SerializerId.resolve('yaml/rails')
+```
+
+### Snapshot Classes
+
+Snapshots are Ruby hashes of file paths to git commit ids. The idea of the snapshot is central to Rosette's git model in that Rosette uses them to know when files that contain translatable content last changed. Rather than storing phrases for every file for every commit, Rosette only stores content when files change. Rosette can use a snapshot to gather a complete list of phrases for each commit in the repository.
+
+You probably won't have to take snapshots manually, but if you do, here's an example:
+
+```ruby
+Rosette::Core::SnapshotFactory.new
+  .set_repo_config(repo_config)
+  .set_start_commit_id('abc123')
+  .set_paths('path/to/snapshot')  # looks at all paths in repo by default
+  .take_snapshot
+```
+
+### Queue Classes
+
+You can choose to process new commits by placing them in a queue (perhaps in combination with a github webhook or a plain 'ol git hook). The queuing logic and various queue stages all live in rosette-core. The stages are:
+
+1. Fetch/pull the repo
+2. Extract phrases for the given commit, store in datastore
+3. Push phrases to TMS (translation management system) for translation
+4. Finalize the TMS submission (perform any necessary clean-up)
+
+If you're not using a queue to process new commits, you'll have to process them some other way to stay current.
+
+### Interfaces
+
+rosette-core contains a few base classes that serve as interfaces for implementations that live in other gems. For example, the [rosette-extractor-yaml gem](https://github.com/rosette-proj/rosette-extractor-yaml) defines the `YamlExtractor` class, which inherits from `Rosette::Core::Extractor`. Other interfaces include `Rosette::Tms::Repository`, `Rosette::Serializers::Serializer`, `Rosette::Preprocessors::Preprocessor`, and more.
+
+## Requirements
+
+All Rosette components only run under jRuby. Java dependencies are managed via the [expert gem](https://github.com/camertron/expert). Run `bundle exec expert install` to install Java dependencies.
+
+## Running Tests
+
+`bundle`, then `bundle exec expert install`, then `bundle exec rspec`.
+
+## Authors
+
+* Cameron C. Dutro: http://github.com/camertron

data/Rakefile

@@ -0,0 +1,18 @@
+# encoding: UTF-8
+
+require 'rubygems' unless ENV['NO_RUBYGEMS']
+
+require 'bundler'
+require 'rspec/core/rake_task'
+require 'rubygems/package_task'
+
+require './lib/rosette/core'
+
+Bundler::GemHelper.install_tasks
+
+task :default => :spec
+
+desc 'Run specs'
+RSpec::Core::RakeTask.new do |t|
+  t.pattern = './spec/**/*_spec.rb'
+end

data/lib/rosette/core.rb

@@ -0,0 +1,110 @@
+# encoding: UTF-8
+
+require 'logger'
+require 'rosette/core/errors'
+
+java_import java.lang.System
+
+# Rosette is a modular internationalization platform written in Ruby.
+module Rosette
+  # Get the current Rosette logger. Defaults to a logger that logs to STDOUT.
+  #
+  # @return [#info, #warning, #error] The current logger.
+  def self.logger
+    @logger ||= Logger.new(STDOUT)
+  end
+
+  # Set the Rosette logger.
+  #
+  # @param [#info, #warning, #error] new_logger The new logger.
+  # @return [void]
+  def self.logger=(new_logger)
+    @logger = new_logger
+  end
+
+  # Get the current Rosette environment name. Defaults to "development".
+  #
+  # @return [String] The current environment name.
+  def self.env
+    @env || 'development'
+  end
+
+  # Set the Rosette environment name.
+  #
+  # @param [String] new_env The new environment name.
+  # @return [void]
+  def self.env=(new_env)
+    @env = new_env
+  end
+
+  # Constructs a new Rosette configurator object and yields it to the block.
+  #
+  # @return [Configurator] The configurator object that was yielded.
+  # @yield [configuration]
+  # @yieldparam configuration [Configurator]
+  def self.build_config
+    configuration = Rosette::Core::Configurator.new
+    yield configuration
+    configuration.apply_integrations(configuration)
+    configuration
+  end
+
+  # Namespace for all Rosette core classes.
+  module Core
+    # The default encoding for all of Rosette.
+    DEFAULT_ENCODING = Encoding::UTF_8
+
+    autoload :Configurator,                  'rosette/core/configurator'
+
+    autoload :StringUtils,                   'rosette/core/string_utils'
+
+    autoload :Ref,                           'rosette/core/git/ref'
+    autoload :Repo,                          'rosette/core/git/repo'
+    autoload :DiffFinder,                    'rosette/core/git/diff_finder'
+
+    autoload :Snapshot,                      'rosette/core/snapshots/snapshot'
+    autoload :SnapshotFactory,               'rosette/core/snapshots/snapshot_factory'
+    autoload :CachedSnapshotFactory,         'rosette/core/snapshots/cached_snapshot_factory'
+    autoload :HeadSnapshotFactory,           'rosette/core/snapshots/head_snapshot_factory'
+    autoload :CachedHeadSnapshotFactory,     'rosette/core/snapshots/cached_head_snapshot_factory'
+    autoload :SnapshotFilterable,            'rosette/core/snapshots/snapshot_filterable'
+    autoload :RepoConfigPathFilter,          'rosette/core/snapshots/repo_config_path_filter'
+
+    autoload :Extractor,                     'rosette/core/extractor/extractor'
+    autoload :StaticExtractor,               'rosette/core/extractor/static_extractor'
+    autoload :Phrase,                        'rosette/core/extractor/phrase'
+    autoload :Translation,                   'rosette/core/extractor/translation'
+    autoload :CommitLog,                     'rosette/core/extractor/commit_log'
+    autoload :CommitLogStatus,               'rosette/core/extractor/commit_log_status'
+    autoload :ExtractorConfig,               'rosette/core/extractor/extractor_config'
+    autoload :PathMatcherFactory,            'rosette/core/path_matcher_factory'
+    autoload :TranslationsPathConfig,        'rosette/core/extractor/translations_path_config'
+    autoload :RepoConfig,                    'rosette/core/extractor/repo_config'
+    autoload :SerializerConfig,              'rosette/core/extractor/serializer_config'
+    autoload :CommitProcessor,               'rosette/core/extractor/commit_processor'
+    autoload :Locale,                        'rosette/core/extractor/locale'
+
+    autoload :PhraseIndexPolicy,             'rosette/core/extractor/phrase/phrase_index_policy'
+    autoload :PhraseToHash,                  'rosette/core/extractor/phrase/phrase_to_hash'
+
+    autoload :TranslationToHash,             'rosette/core/extractor/translation/translation_to_hash'
+
+    autoload :ErrorReporter,                 'rosette/core/error_reporters/error_reporter'
+    autoload :NilErrorReporter,              'rosette/core/error_reporters/nil_error_reporter'
+    autoload :PrintingErrorReporter,         'rosette/core/error_reporters/printing_error_reporter'
+    autoload :BufferedErrorReporter,         'rosette/core/error_reporters/buffered_error_reporter'
+    autoload :RaisingErrorReporter,          'rosette/core/error_reporters/raising_error_reporter'
+
+    autoload :Validators,                    'rosette/core/validators'
+
+    autoload :Resolver,                      'rosette/core/resolvers/resolver'
+    autoload :ExtractorId,                   'rosette/core/resolvers/extractor_id'
+    autoload :IntegrationId,                 'rosette/core/resolvers/integration_id'
+    autoload :SerializerId,                  'rosette/core/resolvers/serializer_id'
+    autoload :PreprocessorId,                'rosette/core/resolvers/preprocessor_id'
+
+    autoload :Commands,                      'rosette/core/commands'
+    autoload :BranchUtils,                   'rosette/core/branch_utils'
+    autoload :TranslationStatus,             'rosette/core/translation_status'
+  end
+end

data/lib/rosette/core/branch_utils.rb

@@ -0,0 +1,152 @@
+# encoding: UTF-8
+
+require 'digest/sha1'
+
+module Rosette
+  module Core
+
+    # Utility methods that work at the branch level.
+    class BranchUtils
+      class << self
+
+        # Computes the status for an array of commit logs. Determining the
+        # aggregate status of more than one commit log means choosing the least
+        # far-along status from the list. In other words, if the list contains
+        # two commit logs with FETCHED and FINALIZED statuses respectively, this
+        # method will return FETCHED because that's the least advanced status in
+        # the list. A list of commit logs is only as advanced as its 'weakest',
+        # least advanced element.
+        #
+        # @param [Array<CommitLog>] commit_logs The list of commit logs to
+        #   derive the status from.
+        # @return [PhraseStatus]
+        def derive_status_from(commit_logs)
+          ps = Rosette::DataStores::PhraseStatus
+          entry = commit_logs.min_by { |entry| ps.index(entry.status) }
+
+          if entry
+            entry.status
+          else
+            # since FINALIZED commit_logs aren't considered, the absence of any
+            # commit logs (i.e. a nil entry) indicates the branch is fully
+            # processed
+            Rosette::DataStores::PhraseStatus::FINALIZED
+          end
+        end
+
+        # Computes the phrase count for the given list of commit logs. This is
+        # simply the sum of all the commit log's phrase counts.
+        #
+        # @param [Array<CommitLog>] commit_logs The list of commit logs to
+        #   compute the phrase count from.
+        # @return [Fixnum]
+        def derive_phrase_count_from(commit_logs)
+          commit_logs.inject(0) { |sum, entry| sum + entry.phrase_count }
+        end
+
+        # Computes the status for each individual locale contained in the
+        # given list of commit logs. Locale statuses are hashes that contain a
+        # count of the number of translated phrases for that locale as well as
+        # the percent translated.
+        #
+        # @param [Array<CommitLog>] commit_logs The list of commit logs to
+        #   compute locale statuses from.
+        # @param [String] repo_name The name of the repo these commit logs came
+        #   from.
+        # @param [Object] datastore The datastore to query for locale data.
+        # @param [Fixnum] phrase_count The aggregate number of phrases for the
+        #   list of commit logs. If +nil+, the +derive_phrase_count+ method
+        #   will be called to fill in this parameter.
+        # @return [Hash] the locale status hash. Contains locale codes as
+        #   top-level keys and hashes as values. Each locale hash contains the
+        #   keys +:translated_count+ and +:percent_translated+.
+        def derive_locale_statuses_from(commit_logs, repo_name, datastore, phrase_count = nil)
+          phrase_count ||= derive_phrase_count_from(commit_logs)
+
+          locale_statuses = commit_logs.each_with_object({}) do |commit_log, ret|
+            locale_entries = datastore.commit_log_locales_for(
+              repo_name, commit_log.commit_id
+            )
+
+            locale_entries.each do |locale_entry|
+              ret[locale_entry.locale] ||= { translated_count: 0 }
+
+              ret[locale_entry.locale].tap do |locale_hash|
+                locale_hash[:translated_count] += locale_entry.translated_count
+              end
+            end
+          end
+
+          add_translation_percentages(locale_statuses, phrase_count)
+        end
+
+        # Adds any missing locale keys to a hash of locale statuses.
+        #
+        # @param [Array<Locale>] all_locales The complete list of locales that
+        #   +locale_statuses+ should contain.
+        # @param [Hash] locale_statuses The locale statuses hash such as is
+        #   returned by +derive_locale_statuses_from+.
+        # @return [Hash] a copy of +locale_statuses+ that contains entries for
+        #   all the locales in +all_locales+. Any locale entries missing from
+        #   +locale_statuses+ will have been filled in with blank data, i.e.
+        #   +:translated_count+ will be 0 and +:percent_translated+ will be 0.0.
+        def fill_in_missing_locales(all_locales, locale_statuses)
+          all_locales.each_with_object({}) do |locale, ret|
+            if found_locale_status = locale_statuses[locale.code]
+              ret[locale.code] = found_locale_status
+            else
+              ret[locale.code] = {
+                percent_translated: 0.0,
+                translated_count: 0
+              }
+            end
+          end
+        end
+
+        def derive_branch_name(commit_id, repo)
+          refs = repo.refs_containing(commit_id).map(&:getName)
+
+          if refs.include?('refs/remotes/origin/master')
+            'refs/remotes/origin/master'
+          else
+            filter_refs(refs).first
+          end
+        end
+
+        private
+
+        def filter_refs(refs)
+          refs.each_with_object([]) do |ref, ret|
+            ret << ref if valid_ref?(ref)
+          end
+        end
+
+        def valid_ref?(ref_text)
+          ref = Rosette::Core::Ref.parse(ref_text)
+          ref && ref.remote? && ref.name != 'master' && ref.name != 'HEAD'
+        end
+
+        def add_translation_percentages(locale_statuses, phrase_count)
+          locale_statuses.each_with_object({}) do |(locale, locale_status), ret|
+            ret[locale] = locale_status.merge(
+              percent_translated: percentage(
+                locale_status.fetch(:translated_count, 0) || 0,
+                phrase_count || 0
+              )
+            )
+          end
+        end
+
+        def percentage(dividend, divisor)
+          if divisor > 0
+            (dividend.to_f / divisor.to_f).round(2)
+          else
+            0.0
+          end
+        end
+
+      end
+    end
+
+  end
+end

data/lib/rosette/core/commands.rb

@@ -0,0 +1,139 @@
+# encoding: UTF-8
+
+require 'digest/sha1'
+
+module Rosette
+  module Core
+
+    # Namespace for all Rosette commands.
+    module Commands
+
+      autoload :Errors,                   'rosette/core/commands/errors'
+
+      autoload :CommitCommand,            'rosette/core/commands/git/commit_command'
+      autoload :DiffBaseCommand,          'rosette/core/commands/git/diff_base_command'
+      autoload :DiffCommand,              'rosette/core/commands/git/diff_command'
+      autoload :ShowCommand,              'rosette/core/commands/git/show_command'
+      autoload :StatusCommand,            'rosette/core/commands/git/status_command'
+      autoload :FetchCommand,             'rosette/core/commands/git/fetch_command'
+      autoload :SnapshotCommand,          'rosette/core/commands/git/snapshot_command'
+      autoload :RepoSnapshotCommand,      'rosette/core/commands/git/repo_snapshot_command'
+      autoload :WithRepoName,             'rosette/core/commands/git/with_repo_name'
+      autoload :WithRef,                  'rosette/core/commands/git/with_ref'
+      autoload :WithRefs,                 'rosette/core/commands/git/with_refs'
+      autoload :WithNonMergeRef,          'rosette/core/commands/git/with_non_merge_ref'
+      autoload :WithSnapshots,            'rosette/core/commands/git/with_snapshots'
+      autoload :DiffEntry,                'rosette/core/commands/git/diff_entry'
+
+      autoload :WithLocale,               'rosette/core/commands/translations/with_locale'
+      autoload :ExportCommand,            'rosette/core/commands/translations/export_command'
+      autoload :TranslationLookupCommand, 'rosette/core/commands/translations/translation_lookup_command'
+
+      autoload :EnqueueCommitCommand,     'rosette/core/commands/queuing/enqueue_commit_command'
+      autoload :RequeueCommitCommand,     'rosette/core/commands/queuing/requeue_commit_command'
+
+      # Base class for all Rosette commands.
+      #
+      # @!attribute [r] configuration
+      #   @return [Configurator] Rosette configuration.
+      class Command
+        attr_reader :configuration
+
+        class << self
+          # Validates a single field.
+          #
+          # @param [Symbol] field The field to validate.
+          # @param [Hash] validator_hash The hash of options for this
+          #   validation. For now, should just contain +:type+ which contains
+          #   to a symbol corresponding to the type of validator to use.
+          # @return [void]
+          def validate(field, validator_hash)
+            validators[field] << instantiate_validator(validator_hash)
+          end
+
+          # A hash of all the currently configured validators.
+          #
+          # @return [Hash<Symbol, Array<Validator>>] The hash of fields to
+          #   valdiator instances.
+          def validators
+            @validators ||= Hash.new { |hash, key| hash[key] = [] }
+          end
+
+          private
+
+          def instantiate_validator(validator_hash)
+            validator_type = validator_hash.fetch(:type)
+            validator_class_for(validator_type).new(validator_hash)
+          end
+
+          def validator_class_for(validator_type)
+            klass = "#{Rosette::Core::StringUtils.camelize(validator_type.to_s)}Validator"
+            if Rosette::Core::Validators.const_defined?(klass)
+              Rosette::Core::Validators.const_get(klass)
+            else
+              raise TypeError, "couldn't find #{validator_type} validator"
+            end
+          end
+        end
+
+        # Creates a new command instance.
+        #
+        # @param [Configurator] configuration The Rosette configuration to use.
+        def initialize(configuration)
+          @configuration = configuration
+        end
+
+        # Gets the hash of current error messages.
+        #
+        # @return [Hash<Symbol, Array<String>>] The hash of current messages.
+        #   The hash's keys are field names and the values are arrays of error
+        #   messages for that field.
+        def messages
+          @messages ||= {}
+        end
+
+        # Returns true if the command's validators all pass, false otherwise.
+        #
+        # @raise [NotImplementedError]
+        def valid?
+          raise NotImplementedError, 'please use a Command subclass.'
+        end
+
+        protected
+
+        def datastore
+          configuration.datastore
+        end
+
+        def path_digest(paths)
+          Digest::MD5.hexdigest(paths.join)
+        end
+      end
+
+      # Base class for all of Rosette's git-based commands.
+      class GitCommand < Command
+        # Returns true if the command's validators all pass, false otherwise.
+        # After this method is finished executing, the +messages+ hash will
+        # have been populated with error messages per field.
+        #
+        # @return [Boolean]
+        def valid?
+          self.class.validators.all? do |name, validators|
+            validators.all? do |validator|
+              valid = validator.valid?(send(name), repo_name, configuration)
+              messages[name] = validator.messages unless valid
+              valid
+            end
+          end
+        end
+
+        protected
+
+        def get_repo(name)
+          configuration.get_repo(name)
+        end
+      end
+
+    end
+  end
+end