phraseapp_updater 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 9b417ad418dde918054346c6ae0ab08666a5d205
+  data.tar.gz: 8c76240084044ebee4a3a602293914760e72e8f0
+SHA512:
+  metadata.gz: bdac23c33192d626f2cfee46bd0481f5efd327742fb0ea2f1e64e4a1db04e06b29ba423156af235627522726c97c8c060cbd8288f1884dc8077cebe79576cee4
+  data.tar.gz: 304f52b5b55c1c1c0b2e445e8e72a5563b83d984723d8d0265b9ccc5005d45f0e7db2120df799fb5d42b6797b368f2f755671daf72008cfec40f54e5a496728a

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.3
+before_install: gem install bundler -v 1.12.5

data/CODE_OF_CONDUCT.markdown

@@ -0,0 +1,50 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer at kev@bibo.com.ph. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/
+

data/Gemfile

@@ -0,0 +1,5 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in phraseapp_updater.gemspec
+gemspec
+

data/LICENSE.txt

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Kevin Griffin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.markdown

@@ -0,0 +1,129 @@
+# PhraseAppUpdater
+
+[![Build Status](https://travis-ci.org/iknow/phraseapp_updater.svg?branch=master)](https://travis-ci.org/iknow/phraseapp_updater)
+
+**Version** 0.1.0
+
+This is a tool for performing three-way merges of [PhraseApp](https://phraseapp.com) locale data with locale data commited to your application.
+
+Our current workflow has localizers working on a `master` project on
+PhraseApp. This regularly gets pulled into the `master` branch of our
+application and released. This branch is for "maintenance" localizations:
+ongoing translations of existing locale keys.
+
+However, we also introduce, remove, and change locale data by merging in
+feature branches to `master`. When we do this, we want to update the
+`master` PhraseApp project with the data newly-commited to our `master`
+branch. PhraseApp provides [APIs](https://phraseapp.com/docs/api/v2/) and a [Ruby gem](https://github.com/phrase/phraseapp-ruby) for accessing
+them, but the API only allows either a) completely overwriting
+PhraseApp's data or b) reapplying PhraseApp's data on top of the
+uploaded data.
+
+What we want instead is a three way merge where the uploaded data wins
+on conflict. Non-conflicting changes on PhraseApp are preserved, while
+changes on both sides take the committed data. The result of the merge
+is then sent to PhraseApp, keeping it up-to-date with the newest commit
+of `master`.
+
+This is especially important when removing keys. Imagine we have the
+following, no-longer useful key:
+
+```json
+unused:
+  one: An unused
+```
+
+On PhraseApp, we've added another plural form:
+
+
+```json
+unused:
+  one: An unused
+  zero: No unused's
+```
+
+And in our feature branch, we remove it. The result we want is that the
+key completely disappears, instead of getting a result like either of
+the above.
+
+## Installation
+
+This gem provides a command line interface for performing the
+merge and uploading the result to PhraseApp. To use it, install the gem:
+
+`gem install phraseapp_updater`
+
+You may also use this gem programatically from your own application.
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'phraseapp_updater'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install phraseapp_updater
+
+## Usage
+
+CLI
+---
+
+`phraseapp_updater` operates on two directories and your PhraseApp API
+data. The two directories should contain the previous revision of your
+locale files and the latest revision of the same files. These will be
+used in the merge with the files on PhraseApp.
+
+The main command is the the `push_changes` command:
+
+```
+phraseapp_updater push_changes --new_locales_path="/data/previous", --previous_locales_path="/data/new" --phraseapp_api_key="yourkey" --phraseapp_project_id="projectid"
+```
+
+The arguments provided to the command can also be specified as shell
+variables:
+
+```
+PA_NEW_LOCALES_PATH
+PA_PREVIOUS_LOCALES_PATH
+PA_API_KEY
+PA_PROJECT_ID
+```
+
+Ruby
+---
+
+`PhraseAppUpdater.push` is analogous to the command line version:
+
+```ruby
+PhraseAppUpdater.push("api_key", "project_id", "previous/path", "current/path")
+```
+
+## Future Improvements
+
+If you'd like to contribute, these would be very helpful!
+
+* Expose the changed files on the command line
+* Implement other `LocaleFile`s with `parse` for non-JSON types
+* Checking if PhraseApp files changed during execution before upload, to reduce the race condition window
+* More specs for the API and shell
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. When everything is working, make a pull request.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/iknow/phraseapp_updater. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,7 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+

data/bin/console

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "phraseapp_updater"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "pry"
+Pry.start
+

data/bin/phraseapp_updater

@@ -0,0 +1,117 @@
+#!/usr/bin/env ruby
+require 'thor'
+require 'phraseapp_updater'
+
+class PhraseAppUpdaterCLI < Thor
+  desc "push", "Update PhraseApp project by merging changes from locale file and PhraseApp"
+  option :new_locales_path,                type: :string
+  option :previous_locales_path,           type: :string
+  option :phraseapp_api_key,               type: :string
+  option :phraseapp_project_id,            type: :string
+  option :store_results_path,              type: :string
+  option :store_phraseapp_originals_path,  type: :string
+
+  def push
+    new_locales_path = options.fetch(:new_locales_path, ENV["PA_NEW_LOCALES_PATH"])
+    if new_locales_path.to_s.empty?
+      raise RuntimeError.new("Must provide a path to the locale files to upload. --new_locales_path or PA_NEW_LOCALES_PATH")
+    end
+
+    unless File.readable?(new_locales_path) && File.directory?(new_locales_path)
+      raise RuntimeError.new("Path to locales is not a readable directory: #{new_locales_path}")
+    end
+
+    previous_locales_path = options.fetch(:previous_locales_path, ENV["PA_PREVIOUS_LOCALES_PATH"])
+    if previous_locales_path.to_s.empty?
+      raise RuntimeError.new("Must provide a path to the locale files to upload. --previous_locales_path or PA_PREVIOUS_LOCALES_PATH")
+    end
+
+    unless File.readable?(previous_locales_path) && File.directory?(previous_locales_path)
+      raise RuntimeError.new("Path to locales is not a readable directory: #{previous_locales_path}")
+    end
+
+    phraseapp_api_key = options.fetch(:phraseapp_api_key, ENV["PA_API_KEY"])
+    if phraseapp_api_key.to_s.empty?
+      raise RuntimeError.new("Must provide Phraseapp API key. --phraseapp_api_key or PA_API_KEY")
+    end
+
+    phraseapp_project_id = options.fetch(:phraseapp_project_id, ENV["PA_PROJECT_ID"])
+    if phraseapp_project_id.to_s.empty?
+      raise RuntimeError.new("Must provide Phraseapp project ID. --phraseapp_project_id or PA_PROJECT_ID")
+    end
+
+    store_results_path = options.fetch(:store_results_path, ENV["PA_store_results_path"]).to_s
+
+    if !store_results_path.empty? && (!File.writable?(store_results_path) || !File.directory?(store_results_path))
+      raise RuntimeError.new("Path to store results is not a writable directory: #{store_results_path}")
+    end
+
+    store_phraseapp_originals_path = options.fetch(:store_phraseapp_originals_path, ENV["PA_store_phraseapp_originals_path"]).to_s
+
+    if !store_phraseapp_originals_path.empty? && (!File.writable?(store_phraseapp_originals_path) || !File.directory?(store_phraseapp_originals_path))
+      raise RuntimeError.new("Path to store PhraseApp originals is not a writable directory: #{store_phraseapp_originals_path}")
+    end
+
+    begin
+      result = PhraseAppUpdater.push(phraseapp_api_key, phraseapp_project_id, previous_locales_path, new_locales_path)
+
+      unless store_results_path.empty?
+        result.resolved_files.each do |file|
+          path = "#{store_results_path.chomp("/")}/#{file.name}.json"
+          File.write(path, file.content)
+        end
+      end
+
+      unless store_phraseapp_originals_path.empty?
+        result.resolved_files.each do |file|
+          path = "#{store_phraseapp_originals_path.chomp("/")}/#{file.name}.json"
+          File.write(path, file.content)
+        end
+      end
+    rescue StandardError => e
+      # Like a bad API key
+      # Raise more specific errors and handle
+      raise e
+    end
+  end
+
+
+  desc "pull", "Pulls data from PhraseApp for deployment."
+  option :fallback_path,         type: :string,  required: true
+  option :destination_path,      type: :string
+  option :phraseapp_api_key,     type: :string
+  option :phraseapp_project_id,  type: :string
+
+  def pull
+    fallback_path = options[:fallback_path]
+    unless File.readable?(fallback_path) && File.directory?(fallback_path)
+      raise RuntimeError.new("fallback_path directory is not a readable directory: #{fallback_path}")
+    end
+
+    destination_path = options.fetch(:destination_path, fallback_path)
+
+    unless File.writable?(fallback_path) && File.directory?(fallback_path)
+      raise RuntimeError.new("destination directory is not a writable directory: #{fallback_path}")
+    end
+
+    phraseapp_api_key = options.fetch(:phraseapp_api_key, ENV["PA_API_KEY"])
+    if phraseapp_api_key.to_s.empty?
+      raise RuntimeError.new("Must provide Phraseapp API key. --phraseapp_api_key or PA_API_KEY")
+    end
+
+    phraseapp_project_id = options.fetch(:phraseapp_project_id, ENV["PA_PROJECT_ID"])
+    if phraseapp_project_id.to_s.empty?
+      raise RuntimeError.new("Must provide Phraseapp project ID. --phraseapp_project_id or PA_PROJECT_ID")
+    end
+
+    files = PhraseAppUpdater.pull(phraseapp_api_key, phraseapp_project_id, fallback_path)
+
+    files.each do |file|
+      path = "#{destination_path.chomp("/")}/#{file.name}.json"
+      File.write(path, file.content)
+    end
+  end
+end
+
+PhraseAppUpdaterCLI.start(ARGV)
+

data/bin/setup

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here
+

data/lib/phraseapp_updater/differ.rb

@@ -0,0 +1,145 @@
+require 'set'
+require 'hashdiff'
+require 'deep_merge'
+
+class PhraseAppUpdater
+  class Differ
+    SEPARATOR = "~~~"
+    using IndexBy
+
+    class << self
+      # Resolution strategy is that primary always wins in the event of a conflict
+      def resolve_diffs(primary:, secondary:, secondary_deleted_prefixes:)
+        primary   = primary.index_by   { |op, path, from, to| path }
+        secondary = secondary.index_by { |op, path, from, to| path }
+
+        # As well as explicit conflicts, we want to make sure that deletions or
+        # incompatible type changes to a `primary` key prevent addition of child
+        # keys in `secondary`. Because input hashes are flattened, it's never
+        # possible for a given path and its prefix to be in the same input.
+        # For example, in:
+        #
+        # primary   = [["+", "a", 1]]
+        # secondary = [["+", "a.b", 2]]
+        #
+        # the secondary change is impossible to perform on top of the primary, and
+        # must be blocked.
+        #
+        # This applies in reverse: prefixes of paths in `p` need to be available
+        # as hashes, so must not appear as terminals in `s`:
+        #
+        # primary   = [["+", "a.b", 2]]
+        # secondary = [["+", "a",   1]]
+        primary_prefixes = primary.keys.flat_map { |p| path_prefixes(p) }.to_set
+
+        # Remove conflicting entries from secondary, recording incompatible
+        # changes.
+        path_conflicts = []
+        secondary.delete_if do |path, diff|
+          if primary_prefixes.include?(path) || primary.keys.any? { |pk| path.start_with?(pk) }
+            path_conflicts << path unless primary.has_key?(path) && diff == primary[path]
+            true
+          else
+            false
+          end
+        end
+
+        # For all path conflicts matching secondary_deleted_prefixes, additionally
+        # remove other changes with the same prefix.
+        prefix_conflicts = secondary_deleted_prefixes.select do |prefix|
+          path_conflicts.any? { |path| path.start_with?(prefix) }
+        end
+
+        secondary.delete_if do |path, diff|
+          prefix_conflicts.any? { |prefix| path.start_with?(prefix) }
+        end
+
+        primary.values + secondary.values
+      end
+
+      def apply_diffs(hash, diffs)
+        deep_compact!(HashDiff.patch!(hash, diffs))
+      end
+
+      def resolve!(original:, primary:, secondary:)
+        # To appropriately cope with type changes on either sides, flatten the
+        # trees before calculating the difference and then expand afterwards.
+        f_original  = flatten(original)
+        f_primary   = flatten(primary)
+        f_secondary = flatten(secondary)
+
+        primary_diffs   = HashDiff.diff(f_original, f_primary)
+        secondary_diffs = HashDiff.diff(f_original, f_secondary)
+
+        # However, flattening discards one critical piece of information: when we
+        # have deleted or clobbered an entire prefix (subtree) from the original,
+        # we want to consider this deletion atomic. If any of the changes is
+        # cancelled, they must all be. Motivating example:
+        #
+        # original:  { word: { one: "..", "many": ".." } }
+        # primary:   { word: { one: "..", "many": "..", "zero": ".." } }
+        # secondary: { word: ".." }
+        # would unexpectedly result in { word: { zero: ".." } }.
+        #
+        # Additionally calculate subtree prefixes that were deleted in `secondary`:
+        secondary_deleted_prefixes =
+          HashDiff.diff(original, secondary, delimiter: SEPARATOR).lazy
+          .select { |op, path, from, to| (op == "-" || op == "~") && from.is_a?(Hash) && !to.is_a?(Hash) }
+          .map    { |op, path, from, to| path }
+          .to_a
+
+
+        resolved_diffs = resolve_diffs(primary:   primary_diffs,
+                                       secondary: secondary_diffs,
+                                       secondary_deleted_prefixes: secondary_deleted_prefixes)
+        HashDiff.patch!(f_original, resolved_diffs)
+
+        expand(f_original)
+      end
+
+
+      # Prefer everything in current except deletions,
+      # which are restored from previous if available
+      def restore_deletions(current, previous)
+        current.deep_merge(previous)
+      end
+
+      private
+
+      def flatten(hash, prefix = nil, acc = {})
+        hash.each do |k, v|
+          k = "#{prefix}#{SEPARATOR}#{k}" if prefix
+          if v.is_a?(Hash)
+            flatten(v, k, acc)
+          else
+            acc[k] = v
+          end
+        end
+        acc
+      end
+
+      def expand(flat_hash)
+        flat_hash.each_with_object({}) do |(key, value), root|
+          path = key.split(SEPARATOR)
+          leaf_key = path.pop
+          leaf = path.inject(root) do |node, path_key|
+            node[path_key] ||= {}
+          end
+          raise ArgumentError.new("Type conflict in flattened hash expand: expected no key at #{key}") if leaf.has_key?(leaf_key)
+          leaf[leaf_key] = value
+        end
+      end
+
+      def path_prefixes(path_string)
+        path = path_string.split(SEPARATOR)
+        parents = []
+        path.inject do |acc, el|
+          parents << acc
+          "#{acc}#{SEPARATOR}#{el}"
+        end
+        parents
+      end
+    end
+  end
+end
+

data/lib/phraseapp_updater/index_by.rb

@@ -0,0 +1,11 @@
+class PhraseAppUpdater
+  module IndexBy
+    refine Array do
+      def index_by(&block)
+        each_with_object({}) do |value, hash|
+          hash[yield(value)] = value
+        end
+      end
+    end
+  end
+end

data/lib/phraseapp_updater/locale_file.rb

@@ -0,0 +1,41 @@
+require 'multi_json'
+require 'oj'
+
+# We're working with pure JSON, not
+# serialized Ruby objects
+Oj.default_options = {mode: :strict}
+
+class PhraseAppUpdater
+  class LocaleFile
+    attr_reader :name, :content, :parsed_content
+
+    def self.from_hash(name, hash)
+      new(name, MultiJson.dump(hash))
+    end
+
+    def initialize(name, content)
+      @name           = name
+      @content        = content
+      @parsed_content = parse(@content)
+      format_content!
+    end
+
+    def to_s
+      "#{name}, #{content[0,20]}..."
+    end
+
+    private
+
+    def parse(content)
+      MultiJson.load(content)
+    rescue MultiJson::ParseError => e
+      raise ArgumentError.new("Provided content was not valid JSON")
+    end
+
+    def format_content!
+      # Add indentation for better diffs
+      @content = MultiJson.dump(MultiJson.load(@content), pretty: true)
+    end
+  end
+end
+

data/lib/phraseapp_updater/locale_file_loader.rb

@@ -0,0 +1,18 @@
+require 'phraseapp_updater/locale_file'
+
+class PhraseAppUpdater
+  class LocaleFileLoader
+    def self.load(filename)
+      unless File.readable?(filename) && File.file?(filename)
+        raise RuntimeError.new("Couldn't read localization file at #{filename}")
+      end
+
+      LocaleFile.new(File.basename(filename).chomp(".json"), File.read(filename))
+    end
+
+    def self.filenames(locale_directory)
+      Dir["#{locale_directory}/*.json"]
+    end
+  end
+end
+

data/lib/phraseapp_updater/phraseapp_api.rb

@@ -0,0 +1,163 @@
+require 'phraseapp-ruby'
+require 'phraseapp_updater/locale_file'
+require 'thread'
+
+class PhraseAppUpdater
+  class PhraseAppAPI
+    def initialize(api_key, project_id)
+      @client     = PhraseApp::Client.new(PhraseApp::Auth::Credentials.new(token: api_key))
+      @project_id = project_id
+    end
+
+    def download_locales
+      # This is a paginated API, however the maximum page size of 100
+      # is well above our expected locale size,
+      # so we take the first page only for now
+      phraseapp_request { @client.locales_list(@project_id, 1, 100) }.map do |pa_locale|
+        Locale.new(pa_locale)
+      end
+    end
+
+    def download_files(locales, skip_unverified)
+      threaded_request(locales) do |locale|
+        puts "Downloading file for #{locale}"
+        download_file(locale, skip_unverified)
+      end.map do |locale, file_contents|
+        LocaleFile.new(locale.name, file_contents)
+      end
+    end
+
+    def upload_files(locale_files)
+      threaded_request(locale_files) do |locale_file|
+        puts "Uploading #{locale_file}"
+        upload_file(locale_file)
+      end.map { |locale_file, upload_id | upload_id }
+    end
+
+    def remove_keys_not_in_uploads(upload_ids)
+      threaded_request(upload_ids) do |upload_id|
+        puts "Removing keys not in upload #{upload_id}"
+        remove_keys_not_in_upload(upload_id)
+      end
+    end
+
+    def download_file(locale, skip_unverified)
+      download_params = PhraseApp::RequestParams::LocaleDownloadParams.new
+
+      download_params.file_format                  = "nested_json"
+      download_params.skip_unverified_translations = skip_unverified
+
+      phraseapp_request { @client.locale_download(@project_id, locale.id, download_params) }
+    end
+
+    def upload_file(locale_file)
+      upload_params = create_upload_params(locale_file.name)
+
+      # The PhraseApp gem only accepts a filename to upload,
+      # so we need to write the file out and pass it the path
+      Tempfile.create(["#{locale_file.name}", ".json"]) do |f|
+        f.write(locale_file.content)
+        f.close
+
+        upload_params.file = f.path
+        phraseapp_request { @client.upload_create(@project_id, upload_params) }.id
+      end
+    end
+
+    def remove_keys_not_in_upload(upload_id)
+      delete_params   = PhraseApp::RequestParams::KeysDeleteParams.new
+      delete_params.q = "unmentioned_in_upload:#{upload_id}"
+
+      begin
+        phraseapp_request { @client.keys_delete(@project_id, delete_params) }
+      rescue RuntimeError => e
+        # PhraseApp will accept but mark invalid uploads, however the gem
+        # returns the same response in both cases. If we call this API
+        # with the ID of an upload of a bad file, it will fail.
+        # This usually occurs when sending up an empty file, which is
+        # a case we can ignore. However, it'd be better to have a way
+        # to detect a bad upload and find the cause.
+      end
+    end
+
+    private
+
+    def phraseapp_request(&block)
+      res, err = block.call
+
+      unless err.nil?
+        if err.respond_to?(:error)
+          error = err.error
+        else
+          error = err.errors.join("|")
+        end
+
+        raise RuntimeError.new(error)
+      end
+
+      res
+    end
+
+    # PhraseApp allows two concurrent connections at a time.
+    THREAD_COUNT = 2
+
+    def threaded_request(worklist, &block)
+      queue   = worklist.inject(Queue.new, :push)
+      threads = []
+
+      THREAD_COUNT.times do
+        threads << Thread.new do
+          Thread.current[:result] = {}
+
+          begin
+            while work = queue.pop(true) do
+              Thread.current[:result][work] = block.call(work)
+            end
+          rescue ThreadError => e
+            Thread.exit
+          end
+
+        end
+      end
+
+      threads.each(&:join)
+
+      threads.each_with_object({}) do |thread, results|
+        results.merge!(thread[:result])
+      end
+    end
+
+    def create_upload_params(locale_name)
+      upload_params = PhraseApp::RequestParams::UploadParams.new
+      upload_params.file_encoding       = "UTF-8"
+      upload_params.file_format         = "nested_json"
+      upload_params.locale_id           = locale_name
+      upload_params.skip_unverification = false
+      upload_params.update_translations = true
+      upload_params.tags                = [generate_upload_tag]
+      upload_params
+    end
+
+    def generate_upload_tag
+      "file_merge_upload_#{Time.now.strftime('%Y%m%d%H%M%S')}"
+    end
+
+    class Locale
+      attr_reader :id, :name, :default
+      def initialize(phraseapp_locale)
+        @name    = phraseapp_locale.name
+        @id      = phraseapp_locale.id
+        @default = phraseapp_locale.default
+      end
+
+      def default?
+        default
+      end
+
+      def to_s
+        "#{name} : #{id}"
+      end
+    end
+  end
+end
+

data/lib/phraseapp_updater/version.rb

@@ -0,0 +1,3 @@
+class PhraseAppUpdater
+  VERSION = "0.1.0"
+end

data/lib/phraseapp_updater.rb

@@ -0,0 +1,110 @@
+require 'phraseapp_updater/version'
+require 'phraseapp_updater/index_by'
+require 'phraseapp_updater/differ'
+require 'phraseapp_updater/locale_file'
+require 'phraseapp_updater/locale_file_loader'
+require 'phraseapp_updater/phraseapp_api'
+
+class PhraseAppUpdater
+  using IndexBy
+
+  def self.push(phraseapp_api_key, phraseapp_project_id, previous_locales_path, new_locales_path)
+    phraseapp_api         = PhraseAppAPI.new(phraseapp_api_key, phraseapp_project_id)
+    phraseapp_locales     = phraseapp_api.download_locales
+
+    phraseapp_files, (previous_locale_files, new_locale_files) =
+      load_files(phraseapp_api, phraseapp_locales, false, previous_locales_path, new_locales_path)
+
+    new_locale_files = new_locale_files.index_by(&:name)
+    phraseapp_files  = phraseapp_files.index_by(&:name)
+
+    resolved_files = previous_locale_files.map do |previous_locale_file|
+      new_locale_file = new_locale_files.fetch(previous_locale_file.name)
+      phraseapp_file  = phraseapp_files.fetch(previous_locale_file.name)
+
+      resolved_content = Differ.resolve!(original: previous_locale_file.parsed_content,
+                                         primary: new_locale_file.parsed_content,
+                                         secondary: phraseapp_file.parsed_content)
+
+      LocaleFile.from_hash(previous_locale_file.name, resolved_content)
+    end
+
+    # Upload all of the secondary languages first,
+    # so that the missing keys in them get filled in
+    # with blanks on PhraseApp by the default locale.
+    # If we do the clean up after uploading the default
+    # locale file, these blanks will get cleaned because
+    # they're not mentioned in the secondary locale files.
+
+    default_locale_file = find_default_locale_file(phraseapp_locales, resolved_files)
+
+    resolved_files.delete(default_locale_file)
+
+    changed_files = resolved_files.select do |file|
+      file.parsed_content != phraseapp_files[file.name].parsed_content
+    end
+
+    upload_ids = phraseapp_api.upload_files(changed_files)
+    phraseapp_api.remove_keys_not_in_uploads(upload_ids)
+
+    upload_id = phraseapp_api.upload_file(default_locale_file)
+    phraseapp_api.remove_keys_not_in_upload(upload_id)
+
+    LocaleFileUpdates.new(phraseapp_files.values, changed_files + [default_locale_file])
+  end
+
+  def self.pull(phraseapp_api_key, phraseapp_project_id, fallback_locales_path)
+    phraseapp_api         = PhraseAppAPI.new(phraseapp_api_key, phraseapp_project_id)
+    phraseapp_locales     = phraseapp_api.download_locales
+
+    phraseapp_files, (fallback_files,) =
+      load_files(phraseapp_api, phraseapp_locales, true, fallback_locales_path)
+
+    fallback_files = fallback_files.index_by(&:name)
+
+    phraseapp_files.map do |phraseapp_file|
+      new_content = Differ.restore_deletions(phraseapp_file.parsed_content,
+                                             fallback_files[phraseapp_file.name].parsed_content)
+      LocaleFile.from_hash(phraseapp_file.name, new_content)
+    end
+  end
+
+  def self.find_default_locale_file(locales, files)
+    default_locale = locales.find(&:default?)
+
+    default_locale_file = files.find do |file|
+      file.name == default_locale.name
+    end
+  end
+
+  def self.load_files(phraseapp_api, phraseapp_locales, skip_unverified, *paths)
+    file_groups = paths.map do |path|
+       LocaleFileLoader.filenames(path).map { |l| LocaleFileLoader.load(l) }
+    end
+
+    phraseapp_files = phraseapp_api.download_files(phraseapp_locales, skip_unverified)
+
+    file_name_lists = [*file_groups, phraseapp_files].map do |files|
+      files.map(&:name).to_set
+    end
+
+    # If we don't have the exact same locales for all of the sources, we can't diff them
+    unless file_name_lists.uniq.size == 1
+      message = "Number of files differs. This tool does not yet support adding\
+      or removing langauges: #{file_name_lists}"
+      raise RuntimeError.new(message)
+    end
+
+    return [phraseapp_files, file_groups]
+  end
+
+  class LocaleFileUpdates
+    attr_reader :original_phraseapp_files, :resolved_files
+
+    def initialize(original_phraseapp_files, resolved_files)
+      @original_phraseapp_files = original_phraseapp_files
+      @resolved_files           = resolved_files
+    end
+  end
+end
+

data/phraseapp_updater.gemspec

@@ -0,0 +1,34 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'phraseapp_updater/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "phraseapp_updater"
+  spec.version       = PhraseAppUpdater::VERSION
+  spec.authors       = ["Kevin Griffin"]
+  spec.email         = ["kev@bibo.com.ph"]
+
+  spec.summary       = %q{A three-way differ for PhraseApp projects.}
+  spec.description   = %q{A tool for merging data on PhraseApp with local changes (usually two git revisions)}
+  spec.homepage      = "https://app.engoo.com"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "bin"
+  spec.executables   = ["phraseapp_updater"]
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "thor", "~> 0.19"
+  spec.add_dependency "phraseapp-ruby", "~> 1.3"
+  spec.add_dependency "hashdiff", "~> 0.3"
+  spec.add_dependency "multi_json", "~> 1.12"
+  spec.add_dependency "oj", "~> 2.18"
+  spec.add_dependency "deep_merge", "~> 1.1"
+
+  spec.add_development_dependency "bundler", "~> 1.12"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "pry", "~> 0.10"
+end
+

metadata

@@ -0,0 +1,205 @@
+--- !ruby/object:Gem::Specification
+name: phraseapp_updater
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Kevin Griffin
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2017-02-15 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.19'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.19'
+- !ruby/object:Gem::Dependency
+  name: phraseapp-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: hashdiff
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+- !ruby/object:Gem::Dependency
+  name: multi_json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.12'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.12'
+- !ruby/object:Gem::Dependency
+  name: oj
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.18'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.18'
+- !ruby/object:Gem::Dependency
+  name: deep_merge
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.12'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.12'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.10'
+description: A tool for merging data on PhraseApp with local changes (usually two
+  git revisions)
+email:
+- kev@bibo.com.ph
+executables:
+- phraseapp_updater
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- CODE_OF_CONDUCT.markdown
+- Gemfile
+- LICENSE.txt
+- README.markdown
+- Rakefile
+- bin/console
+- bin/phraseapp_updater
+- bin/setup
+- lib/phraseapp_updater.rb
+- lib/phraseapp_updater/differ.rb
+- lib/phraseapp_updater/index_by.rb
+- lib/phraseapp_updater/locale_file.rb
+- lib/phraseapp_updater/locale_file_loader.rb
+- lib/phraseapp_updater/phraseapp_api.rb
+- lib/phraseapp_updater/version.rb
+- phraseapp_updater.gemspec
+homepage: https://app.engoo.com
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: A three-way differ for PhraseApp projects.
+test_files: []