trove 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a501c7fded530b2dc537d412eae735c19cfa25f04e2e21dd8d6861ae42b2a271
+  data.tar.gz: 0cf407da8dce52a446e053a7e03791622ecd303a6514bdf551ce12bd31ebe788
+SHA512:
+  metadata.gz: 4ad4fa70071eb4745046938a9b042b7df87eaa85964bc4d9a71fa90837335e40d8195f9de7ea27d70963c83a4facd163a1542527f7f88a5ffeb6aec8b4b71649
+  data.tar.gz: 73b4037bf7a50b498ef6d6bb640f2d62e237c7a1effba0aa298c9a54cf8d56edbd6efd539bdfc0768e1e6e1c85d3e8d2f524c60f5b996469881b6d67a66f9d1d

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## 0.1.0 (2020-10-30)
+
+- First release

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Andrew Kane
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,237 @@
+# Trove
+
+:fire: Deploy machine learning models in Ruby (and Rails)
+
+Works great with [XGBoost](https://github.com/ankane/xgboost), [Torch.rb](https://github.com/ankane/torch.rb), [fastText](https://github.com/ankane/fastText), and many other gems
+
+## Installation
+
+Add this line to your application’s Gemfile:
+
+```ruby
+gem 'trove'
+```
+
+And run:
+
+```sh
+bundle install
+trove init
+```
+
+And [configure your storage](#storage) in `.trove.yml`.
+
+## Storage
+
+### Amazon S3
+
+Create a bucket and enable object versioning.
+
+Next, set up your AWS credentials. You can use the [AWS CLI](https://github.com/aws/aws-cli):
+
+```sh
+pip install awscli
+aws configure
+```
+
+Or environment variables:
+
+```sh
+export AWS_ACCESS_KEY_ID=...
+export AWS_SECRET_ACCESS_KEY=...
+export AWS_REGION=...
+```
+
+IAM users need:
+
+- `s3:GetObject` and `s3:GetObjectVersion` to pull files
+- `s3:PutObject` to push files
+- `s3:ListBucket` and `s3:ListBucketVersions` to list files and versions
+- `s3:DeleteObject` and `s3:DeleteObjectVersion` to delete files
+
+Here’s an example policy:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "Trove",
+            "Effect": "Allow",
+            "Action": [
+                "s3:GetObject",
+                "s3:GetObjectVersion",
+                "s3:PutObject",
+                "s3:ListBucket",
+                "s3:ListBucketVersions",
+                "s3:DeleteObject",
+                "s3:DeleteObjectVersion"
+            ],
+            "Resource": [
+                "arn:aws:s3:::my-bucket",
+                "arn:aws:s3:::my-bucket/trove/*"
+            ]
+        }
+    ]
+}
+```
+
+If your production servers only need to pull files, only give them `s3:GetObject` and `s3:GetObjectVersion` permissions.
+
+## How It Works
+
+Git is great for code, but it’s not ideal for large files like models. Instead, we use an object store like Amazon S3 to store and version them.
+
+Trove creates an `trove` directory for you to use as a workspace. Files in the directory are ignored by Git, but can be pushed and pulled from the object store. By default, files are tracked in `.trove.yml` to make it easy to deploy specific versions with code changes.
+
+## Getting Started
+
+Use the `trove` directory to save and load models.
+
+```ruby
+# training code
+model.save_model("trove/model.bin")
+
+# prediction code
+model = FastText.load_model("trove/model.bin")
+```
+
+When a model is ready, push it to the object store with:
+
+```sh
+trove push model.bin
+```
+
+And commit the changes to `.trove.yml`. The model is now ready to be deployed.
+
+## Deployment
+
+We recommend pulling files during the build process.
+
+- [Heroku and Dokku](#heroku-and-dokku)
+- [Docker](#docker)
+
+Make sure your storage credentials are available in the build environment.
+
+### Heroku and Dokku
+
+Add to your `Rakefile`:
+
+```ruby
+Rake::Task["assets:precompile"].enhance do
+  Trove.pull
+end
+```
+
+This will pull files at the very end of the asset precompile. Check the build output for:
+
+```text
+remote:        Pulling model.bin...
+remote:        Asset precompilation completed (30.00s)
+```
+
+### Docker
+
+And add to your `Dockerfile`:
+
+```Dockerfile
+RUN bundle exec trove pull
+```
+
+## Commands
+
+Push a file
+
+```sh
+trove push model.bin
+```
+
+Pull all files in `.trove.yml`
+
+```sh
+trove pull
+```
+
+Pull a specific file (uses the version in `.trove.yml` if present)
+
+```sh
+trove pull model.bin
+```
+
+Pull a specific version of a file
+
+```sh
+trove pull model.bin --version 123
+```
+
+Delete a file
+
+```sh
+trove delete model.bin
+```
+
+List files
+
+```sh
+trove list
+```
+
+List versions
+
+```sh
+trove versions model.bin
+```
+
+## Ruby API
+
+You can use the Ruby API in addition to the CLI.
+
+```ruby
+Trove.push(filename)
+Trove.pull
+Trove.pull(filename)
+Trove.pull(filename, version: version)
+Trove.delete(filename)
+Trove.list
+Trove.versions(filename)
+```
+
+This makes it easy to perform operations from code, iRuby notebooks, and the Rails console.
+
+## Automated Training
+
+By default, Trove tracks files in `.trove.yml` so you can deploy specific versions with `trove pull`. However, this functionality is entirely optional. Disable it with:
+
+```yml
+vcs: false
+```
+
+This is useful if you want to automate training or build more complex workflows.
+
+## History
+
+View the [changelog](https://github.com/ankane/trove/blob/master/CHANGELOG.md)
+
+## Contributing
+
+Everyone is encouraged to help improve this project. Here are a few ways you can help:
+
+- [Report bugs](https://github.com/ankane/trove/issues)
+- Fix bugs and [submit pull requests](https://github.com/ankane/trove/pulls)
+- Write, clarify, or fix documentation
+- Suggest or add new features
+
+To get started with development:
+
+```sh
+git clone https://github.com/ankane/trove.git
+cd trove
+bundle install
+
+export AWS_ACCESS_KEY_ID=...
+export AWS_SECRET_ACCESS_KEY=...
+export AWS_REGION=...
+export S3_BUCKET=my-bucket
+
+bundle exec rake test
+```

data/exe/trove

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+# handle interrupts
+trap("SIGINT") { abort }
+
+require "trove"
+require "trove/cli"
+
+begin
+  Trove::CLI.start
+rescue => e
+  raise e if ENV["DEBUG"]
+  abort e.message
+end

data/lib/trove.rb

@@ -0,0 +1,162 @@
+# stdlib
+require "digest/md5"
+require "yaml"
+
+# modules
+require "trove/utils"
+require "trove/version"
+
+module Trove
+  # storage
+  module Storage
+    autoload :S3, "trove/storage/s3"
+  end
+
+  # methods
+  class << self
+    # TODO use flock to prevent multiple concurrent downloads
+    def pull(filename = nil, version: nil)
+      if filename
+        pull_file(filename, version: version)
+      else
+        raise ArgumentError, "Specify filename for version" if version
+
+        (config["files"] || []).each do |file|
+          pull_file(file["name"], version: file["version"], all: true)
+        end
+      end
+    end
+
+    # could use upload_file method for multipart uploads over a certain size
+    # but multipart uploads have extra cost and cleanup, so keep it simple for now
+    def push(filename)
+      src = File.join(root, filename)
+      raise "File not found" unless File.exist?(src)
+
+      info = storage.info(filename)
+      upload = info.nil?
+      unless upload
+        version = info[:version]
+        if modified?(src, info)
+          upload = true
+        else
+          stream.puts "Already up-to-date"
+        end
+      end
+
+      if upload
+        stream.puts "Pushing #{filename}..." unless stream.tty?
+        resp = storage.upload(src, filename) do |current_size, total_size|
+          Utils.progress(stream, filename, current_size, total_size)
+        end
+        version = resp[:version]
+      end
+
+      if vcs?
+        # add files to yaml if needed
+        files = (config["files"] ||= [])
+
+        # find file
+        file = files.find { |f| f["name"] == filename }
+        unless file
+          file = {"name" => filename}
+          files << file
+        end
+
+        # update version
+        file["version"] = version
+
+        File.write(".trove.yml", config.to_yaml.sub(/\A---\n/, ""))
+      end
+
+      {
+        version: version
+      }
+    end
+
+    def delete(filename)
+      storage.delete(filename)
+    end
+
+    def list
+      storage.list
+    end
+
+    def versions(filename)
+      storage.versions(filename)
+    end
+
+    private
+
+    def pull_file(filename, version: nil, all: false)
+      dest = File.join(root, filename)
+
+      if !version
+        file = (config["files"] || []).find { |f| f["name"] == filename }
+        version = file["version"] if file
+      end
+
+      download = !File.exist?(dest)
+      unless download
+        info = storage.info(filename, version: version)
+        if info.nil? || modified?(dest, info)
+          download = true
+        else
+          stream.puts "Already up-to-date" unless all
+        end
+      end
+
+      if download
+        stream.puts "Pulling #{filename}..." unless stream.tty?
+        storage.download(filename, dest, version: version) do |current_size, total_size|
+          Utils.progress(stream, filename, current_size, total_size)
+        end
+      end
+
+      download
+    end
+
+    def modified?(src, info)
+      Digest::MD5.file(src).hexdigest != info[:md5]
+    end
+
+    # TODO test file not found
+    def config
+      @config ||= begin
+        begin
+          YAML.load_file(".trove.yml")
+        rescue Errno::ENOENT
+          raise "Config not found"
+        end
+      end
+    end
+
+    def root
+      @root ||= config["root"] || "trove"
+    end
+
+    def storage
+      @storage ||= begin
+        uri = URI.parse(config["storage"])
+
+        case uri.scheme
+        when "s3"
+          Storage::S3.new(
+            bucket: uri.host,
+            prefix: uri.path[1..-1]
+          )
+        else
+          raise "Invalid storage provider: #{uri.scheme}"
+        end
+      end
+    end
+
+    def vcs?
+      config.key?("vcs") ? config["vcs"] : true
+    end
+
+    def stream
+      $stderr
+    end
+  end
+end

data/lib/trove/cli.rb

@@ -0,0 +1,113 @@
+require "thor"
+
+module Trove
+  class CLI < Thor
+    include Thor::Actions
+
+    desc "init", "Initialize a project"
+    def init
+      create_file "trove/.keep", ""
+
+      if File.exist?(".gitignore")
+        contents = <<~EOS
+
+          # Ignore Trove storage
+          /trove/*
+          !/trove/.keep
+        EOS
+        unless File.read(".gitignore").include?(contents)
+          append_to_file(".gitignore", contents)
+        end
+      else
+        say "Check in trove/.keep and ignore trove/*"
+      end
+
+      create_file ".trove.yml", <<~EOS
+        storage: s3://my-bucket/trove
+      EOS
+    end
+
+    desc "push FILENAME", "Push a file"
+    def push(filename)
+      Trove.push(filename)
+    end
+
+    desc "pull [FILENAME]", "Pull files"
+    option :version
+    def pull(filename = nil)
+      Trove.pull(filename, version: options[:version])
+    end
+
+    desc "delete FILENAME", "Delete a file"
+    def delete(filename = nil)
+      Trove.delete(filename)
+    end
+
+    desc "list", "List files"
+    def list
+      say table(
+        Trove.list,
+        [:filename, :size, :updated_at]
+      )
+    end
+
+    desc "version", "Show the current version"
+    def version
+      say Trove::VERSION
+    end
+
+    desc "versions FILENAME", "List versions"
+    def versions(filename)
+      say table(
+        Trove.versions(filename),
+        [:version, :size, :updated_at]
+      )
+    end
+
+    private
+
+    def table(data, columns)
+      columns.each do |c|
+        if c == :size
+          data.each { |r| r[c] = Utils.human_size(r[c]) }
+        elsif c == :updated_at
+          data.each { |r| r[c] = "#{time_ago(r[c])} ago" }
+        elsif c == :version
+          data.each { |r| r[c] ||= "<none>" }
+        end
+      end
+      column_names = columns.map { |c| c.to_s.sub(/_at\z/, "").upcase }
+      widths = columns.map.with_index { |c, i| [column_names[i].size, data.map { |r| r[c].to_s.size }.max].max }
+
+      output = String.new("")
+      str = widths.map { |w| "%-#{w}s" }.join("     ") + "\n"
+      output << str % column_names
+      data.each do |row|
+        output << str % columns.map { |c| row[c] }
+      end
+      output
+    end
+
+    def time_ago(time)
+      diff = (Time.now - time).round
+
+      if diff < 60
+        pluralize(diff, "second")
+      elsif diff < 60 * 60
+        pluralize((diff / 60.0).floor, "minute")
+      elsif diff < 60 * 60 * 24
+        pluralize((diff / (60.0 * 60)).floor, "hour")
+      else
+        pluralize((diff / (60.0 * 60 * 24)).floor, "day")
+      end
+    end
+
+    def pluralize(value, str)
+      "#{value} #{value == 1 ? str : "#{str}s"}"
+    end
+
+    def self.exit_on_failure?
+      true
+    end
+  end
+end

data/lib/trove/storage/s3.rb

@@ -0,0 +1,120 @@
+require "aws-sdk-s3"
+require "fileutils"
+
+module Trove
+  module Storage
+    class S3
+      attr_reader :bucket, :prefix
+
+      def initialize(bucket:, prefix: nil)
+        @bucket = bucket
+        @prefix = prefix
+      end
+
+      def download(filename, dest, version: nil)
+        current_size = 0
+        total_size = nil
+
+        # TODO better path
+        tmp = "#{Dir.tmpdir}/trove-#{Time.now.to_f}"
+        begin
+          File.open(tmp, "wb") do |file|
+            options = {bucket: bucket, key: key(filename)}
+            options[:version_id] = version if version
+            client.get_object(**options) do |chunk, headers|
+              file.write(chunk)
+
+              current_size += chunk.bytesize
+              total_size ||= headers["content-length"].to_i
+              yield current_size, total_size
+            end
+          end
+          FileUtils.mv(tmp, dest)
+        ensure
+          # delete file if interrupted
+          File.unlink(tmp) if File.exist?(tmp)
+        end
+      rescue Aws::S3::Errors::ServiceError
+        raise "File not found"
+      end
+
+      def upload(src, filename, &block)
+        on_chunk_sent = lambda do |_, current_size, total_size|
+          block.call(current_size, total_size)
+        end
+        resp = nil
+        File.open(src, "rb") do |file|
+          resp = client.put_object(bucket: bucket, key: key(filename), body: file, on_chunk_sent: on_chunk_sent)
+        end
+        {version: resp.version_id}
+      end
+
+      # etag isn't always MD5, but low likelihood of match if not
+      # could alternatively add sha256 to metadata
+      def info(filename, version: nil)
+        options = {bucket: bucket, key: key(filename)}
+        options[:version_id] = version if version
+        resp = client.head_object(**options)
+        {
+          version: resp.version_id,
+          md5: resp.etag.gsub('"', "")
+        }
+      rescue Aws::S3::Errors::ServiceError
+        nil
+      end
+
+      def delete(filename, version: nil)
+        options = {bucket: bucket, key: key(filename)}
+        options[:version_id] = version if version
+        client.delete_object(**options)
+        true
+      rescue Aws::S3::Errors::ServiceError
+        false
+      end
+
+      def list
+        files = []
+        options = {bucket: bucket}
+        options[:prefix] = prefix if prefix
+        client.list_objects_v2(**options).each do |response|
+          response.contents.each do |object|
+            filename = prefix ? object.key[(prefix.size + 1)..-1] : object.key
+            files << {
+              filename: filename,
+              size: object.size,
+              updated_at: object.last_modified
+            }
+          end
+        end
+        files
+      end
+
+      def versions(filename)
+        versions = []
+        object_key = key(filename)
+        client.list_object_versions(bucket: bucket, prefix: object_key).each do |response|
+          response.versions.each do |version|
+            next if version.key != object_key
+
+            versions << {
+              version: version.version_id == "null" ? nil : version.version_id,
+              size: version.size,
+              updated_at: version.last_modified
+            }
+          end
+        end
+        versions
+      end
+
+      private
+
+      def client
+        @client ||= Aws::S3::Client.new
+      end
+
+      def key(filename)
+        prefix ? "#{prefix}/#{filename}" : filename
+      end
+    end
+  end
+end

data/lib/trove/utils.rb

@@ -0,0 +1,33 @@
+module Trove
+  module Utils
+    # TODO improve performance
+    def self.human_size(size)
+      if size < 2**10
+        units = "B"
+      elsif size < 2**20
+        size /= (2**10).to_f
+        units = "KB"
+      elsif size < 2**30
+        size /= (2**20).to_f
+        units = "MB"
+      else
+        size /= (2**30).to_f
+        units = "GB"
+      end
+
+      round = size < 9.95 ? 1 : 0
+      "#{size.round(round)}#{units}"
+    end
+
+    def self.progress(stream, filename, current_size, total_size)
+      return unless stream.tty?
+
+      width = 50
+      progress = (100.0 * current_size / total_size).floor
+      completed = (width / 100.0 * progress).round
+      remaining = width - completed
+      stream.print "\r#{filename} [#{"=" * completed}#{" " * remaining}] %3s%% %11s " % [progress, "#{Utils.human_size(current_size)}/#{Utils.human_size(total_size)}"]
+      stream.print "\n" if current_size == total_size
+    end
+  end
+end

data/lib/trove/version.rb

@@ -0,0 +1,3 @@
+module Trove
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,122 @@
+--- !ruby/object:Gem::Specification
+name: trove
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Andrew Kane
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2020-10-31 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk-s3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5'
+description:
+email: andrew@chartkick.com
+executables:
+- trove
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- exe/trove
+- lib/trove.rb
+- lib/trove/cli.rb
+- lib/trove/storage/s3.rb
+- lib/trove/utils.rb
+- lib/trove/version.rb
+homepage: https://github.com/ankane/trove
+licenses:
+- MIT
+metadata: {}
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '2.5'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.1.4
+signing_key:
+specification_version: 4
+summary: Deploy machine learning models in Ruby (and Rails)
+test_files: []