rstreet 1.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5c9c064930f22deb7fb0a3fab8b8323509bcc533
+  data.tar.gz: e429f6c7e6ac79f70bed6253eb741cf9308007ef
+SHA512:
+  metadata.gz: 8960137d9358d4e36fe35cc9020c48027be18c870be28585689dbed2ebe9d589d762a859b37bfa01814f5acc92540d2e7737c21000df306d08b38bb4b122530e
+  data.tar.gz: 1209e3e3483d148cc166287b5fba2bd42cb92c11bc1593c952e0082d0629e7ebd866d320cb61a3c362c72e543ce287517fc17e77600ea4e320b5ef58acad87a2

data/.env.example

@@ -0,0 +1,3 @@
+S3_BUCKET=asdf
+AWS_ACCESS_KEY_ID=qwer
+AWS_SECRET_ACCESS_KEY=zxcv

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in rstreet.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 jaketrent
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,163 @@
+# Rstreet
+
+Smart Uploader for AWS S3 buckets.
+
+Generates a manifest file upon first upload;
+subsequent uploads only send changed files.  Especially useful as a deploy tool
+for uploading static websites to S3. Benefits of using Street:
+
+* Upload only changed files; reduces **PUT** requests.
+* GZip Compressed **manifest** file stored on S3 is very small.
+* Mime-type lookup on upload to facilitate proper browser handling.
+* Non-destructive *(mostly)*.  Works with existing buckets and data. See *Manifest
+  File* section below for more details.
+
+Ruby port of [tgroshon's](https://github.com/tgroshon) [street.js](https://github.com/tgroshon/street.js)
+
+## tl;dr
+
+* Install with `gem install rstreet` to get the `rstreet` cli tool.
+* Export **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to your environment.
+* Run `rstreet -e -b <Target S3 Bucket> path/to/upload/dir`
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'rstreet'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install rstreet
+
+## Command Line Options
+
+A helpful interface into Street is using the command line tool `rstreet`.  The
+command line tool requires a path to a directory that you want to upload to S3 as
+the last argument.
+Run `rstreet --help` to see a list of options.
+
+#### -e, --load-env, Load Environment Variables ####
+
+Rstreet can load environment variables declared in a `.env` file of the current
+working directory.  With this option, Rstreet searches for 3 AWS variables:
+
+1. **AWS_ACCESS_KEY_ID**
+2. **AWS_SECRET_ACCESS_KEY**
+3. **S3_BUCKET**
+
+Instead of using Environment Variables, you can some or all of these through
+the command line options `--aws-key`, `--aws-secret`, and `--bucket`
+respectively.
+
+*Example*: `rstreet -e path/to/upload/dir`
+
+#### -b, --bucket [bucket], S3 Destination Bucket ####
+
+The Amazon S3 bucket to be the destination of your uploaded files.
+
+*Example*: `rstreet -b <S3 bucket name> path/to/upload/dir`
+
+#### -k, --aws-key [key], AWS Access Key Id ####
+
+The AWS Access Key Id to be used for authenticating the S3 session.  User must
+have `PUT` and `GET` permissions on the bucket.
+
+*Example*: `street -k <AWS Access Key Id> path/to/upload/dir`
+
+#### -s, --aws-secret [secret], AWS Secret Access Key ####
+
+The AWS Secret Access Key associated with the AWS Access Key Id you are using.
+
+*Example*: `rstreet -s <AWS Secret Key> path/to/upload/dir`
+
+#### -v, --verbose, Run with expanded messages ####
+
+Shows the number of files uploaded, but not much else.  This option will do more
+in the future.
+
+*Example*: `rstreet -v /path/to/upload/dir`
+
+#### -n, --dry-run, Run but do not upload ####
+
+Best if used in conjuction with the `--verbose` option.
+
+*Example*:
+
+`rstreet -n path/to/upload/dir`
+
+`rstreet -nv path/to/upload/dir`
+
+`rstreet -n path/to/upload/dir`
+
+## Run Programmatically
+
+If you want to incorporate the Rstreet uploader into another program, that's
+also possible.  Perhaps you'd like to use it as a part of an API that can upload
+to S3 or your own build script.  Usage is simple.  It takes a hash of options that
+ match the commandline interface.  However, note that the names of the options are slightly
+ different.
+
+```ruby
+require "rstreet"
+
+options = {
+  src: 'path/to/upload/dir',      # Path to directory to upload.
+  dry_run: false,                 # Disable upload mechanism.
+  s3_bucket: 'bucketname.com',    # Name of S3 Bucket to upload to.
+  aws_key: 'AWS Access Key Id',   # AWS Access Key Id for authentication w/ S3.
+  aws_secret: 'AWS Secret Key',   # AWS Secret Key for authentication w/ S3.
+  load_env: true,                 # Load Environment Variables with 'dotenv'.
+  verbose: true,                  # Trigger extra messages.
+}
+
+Rstreet::Uploader.new(options).run
+```
+
+You can make this part of a [Grunt](http://gruntjs.com/) task or your own,
+standalone deploy script.  Dealer's choice!
+
+## Manifest File
+
+The `manifest.json.gz` file is a GZipped JSON file that maps S3 Object Keys (file
+names) to MD5 Hashes.
+
+The `manifest.json.gz` file is generated each time Street is run.  On the first run
+of Street on a specific directory, the `manifest.json.gz` file is written to that
+directory (called the *upload directory*), and all files it found are uploaded to
+S3.  The next time Street is run, a **new manifest** is generated locally, the
+**old manifest** is pulled down from S3, and the two files are compared to
+search for differences.  The **new manifest** is only written if files have
+been changed.  If files have been changed, the new `manifest.json.gz` and changed
+files are then uploaded to S3 where they replace the objects of the same name.
+
+This approach has two important effects that you should be aware of:
+
+1. **\*IMPORTANT\*** Upon first run of Street, all files in the directory
+   and a new `manifest.json.gz` file will be uploaded and replace S3 objects with
+   the same name.
+
+2. A `manifest.json.gz` file on S3 does *not* hold information on all the objects in
+   that bucket; only on objects that Street has specifically uploaded.
+
+Keep these in mind as you work.
+
+## Setup AWS Access Policies
+
+Make sure that the credentials that you're providing to the tool have sufficient access to read and write to the bucket you're attempting to access.
+
+For an easy default, attach the managed access control policy "AmazonS3FullAccess" to your user/group.
+
+## Contributing
+
+1. Fork it ( https://github.com/[jaketrent]/rstreet/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/bin/rstreet

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+require "rstreet"
+require "opts_parser"
+
+options = Rstreet::OptsParser.parse(ARGV)
+chatter = Rstreet::Uploader.new(options)
+chatter.run
+

data/lib/bucket_communicator.rb

@@ -0,0 +1,42 @@
+require "s3"
+
+require "manifest_builder"
+
+module Rstreet
+  class BucketCommunicator
+    def initialize(s3_bucket, dry_run)
+      @s3, @dry_run = S3.new(s3_bucket), dry_run
+    end
+
+    def pull_manifest(manifest_uploadable)
+      manifest_file = @s3.get_file(manifest_uploadable)
+      if manifest_file
+        ManifestBuilder.new(manifest_file).read
+      else
+        ManifestBuilder.empty_manifest
+      end
+    end
+
+    def upload(uploadables)
+      # TODO: refactor into sep procs
+      uploadables.map do |u|
+        if dry_run?
+          puts "dry run upload: #{u.name}"
+        else
+          begin
+            @s3.put_file(u)
+            puts "uploaded:        #{u.name}"
+          rescue StandardError => e
+            puts "error uploading: #{u.name}", e
+          end
+        end
+      end
+    end
+
+    private
+
+    def dry_run?
+      @dry_run
+    end
+  end
+end

data/lib/files_reader.rb

@@ -0,0 +1,20 @@
+require 'set'
+
+module Rstreet
+  class FilesReader
+    def self.read_all(dir)
+      dir = File.realpath dir
+      # TODO: print if verbose
+      # puts "Found files in #{@dir}:"
+
+      files = Dir.glob("#{dir}/**/*").to_set
+      dirs = Dir.glob("#{dir}/**/*/").map { |d| d[0..-2] }.to_set
+
+      # puts files.inspect
+      # puts ""
+      # puts dirs.inspect
+
+      files.subtract(dirs).to_a
+    end
+  end
+end

data/lib/manifest_builder.rb

@@ -0,0 +1,48 @@
+require "json"
+
+module Rstreet
+  # TODO: name something more general to manifests (does more than builds)
+  class ManifestBuilder
+    attr_reader :manifest
+
+    def initialize(manifest_file)
+      @manifest = {}
+      @manifest_file = manifest_file
+    end
+
+    def self.empty_manifest
+      {}
+    end
+
+    def read
+      gz = Zlib::GzipReader.new(StringIO.new(@manifest_file))
+      @manifest = JSON.parse(gz.read).to_h
+      gz.close
+      @manifest
+    end
+
+    def write()
+      File.open(@manifest_file, "w") do |f|
+        gz = Zlib::GzipWriter.new(f)
+        gz.write @manifest.to_json.to_s
+        gz.close
+      end
+    end
+
+    def add(uploadable)
+      @manifest[uploadable.name] = uploadable.hash
+    end
+
+    def diff(other_manifest)
+      # TODO: determine if this is the diff algorithm we want
+      # TODO: in the future, account for deletions and delete objects from bucket
+      # 8 [ruby-2.1.5]::rstreet]>a.to_set
+      # => #<Set: {[:a, 1], [:b, 2], [:c, 3]}>
+      #     9 [ruby-2.1.5]::rstreet]>a.to_set.difference b.to_set
+      # => #<Set: {[:b, 2], [:c, 3]}>
+      #     10 [ruby-2.1.5]::rstreet]>b.to_set.difference a.to_set
+      # => #<Set: {[:b, 4]}>
+      @manifest.to_set.difference(other_manifest.to_set).to_h.keys
+    end
+  end
+end

data/lib/opts_parser.rb

@@ -0,0 +1,47 @@
+require "optparse"
+require "ostruct"
+
+module Rstreet
+  class OptsParser
+    def self.parse(args)
+      options = OpenStruct.new
+      options.src = args[-1]
+
+      opt_parser = OptionParser.new do |opts|
+        opts.banner = "Usage: rstreet [options] <src_dir>"
+        opts.separator ""
+        opts.separator "Options:"
+
+        # TODO: DRY up
+        opts.on("-e", "--load-env",
+                "Load environment from .env file") do |load_env|
+          options.load_env = true
+        end
+        opts.on("-b b", "--bucket b",
+                "S3 destination bucket") do |s3_bucket|
+          options.s3_bucket = s3_bucket
+        end
+        opts.on("-k k", "--aws-key k",
+                "AWS Access Key ID") do |aws_key|
+          options.aws_key = aws_key
+        end
+        opts.on("-s s", "--aws-secret s",
+                "AWS Secret Access Key") do |aws_secret|
+          options.aws_secret = aws_secret
+        end
+        opts.on("-v", "--verbose",
+                "Run with expanded messages") do |verbose|
+          options.verbose = true
+        end
+        opts.on("-n", "--dry-run",
+                "Run but do not upload") do |dry_run|
+          options.dry_run = true
+        end
+
+      end
+
+      opt_parser.parse!(args)
+      options
+    end
+  end
+end

data/lib/rstreet.rb

@@ -0,0 +1,58 @@
+require "aws-sdk"
+require "dotenv"
+
+require "bucket_communicator"
+require "rstreet/version"
+require "uploadable_collector"
+
+module Rstreet
+  class Uploader
+    def initialize(options)
+      @options = options
+      load_env if should_load_env?
+      config_aws
+      validate_options
+    end
+
+    def run
+      collector = UploadableCollector.new(@options.src)
+      uploadables = collector.collect
+
+      bucket_communicator = BucketCommunicator.new(@options.s3_bucket, @options.dry_run)
+      old_manifest = bucket_communicator.pull_manifest(collector.manifest_uploadable)
+      current_manifest_builder = collector.manifest_builder
+      diff = current_manifest_builder.diff(old_manifest)
+
+      to_upload = collector.find_uploadables(diff)
+      bucket_communicator.upload(to_upload)
+    end
+
+    private
+
+    def config_aws
+      AWS.config(access_key_id: @options.aws_key, secret_access_key: @options.aws_secret)
+    end
+
+    def load_env
+      Dotenv.load
+      @options.s3_bucket ||= ENV["S3_BUCKET"]
+      @options.aws_key ||= ENV["AWS_ACCESS_KEY_ID"]
+      @options.aws_secret ||= ENV["AWS_SECRET_ACCESS_KEY"]
+    end
+
+    def should_load_env?
+      @options.load_env
+    end
+
+    def validate_options
+      # TODO: DRY'ify
+      raise ArgumentError, "Specify a source directory in options.src" if @options.src.nil?
+
+      raise ArgumentError, "Specify an S3 Bucket in an env var called S3_BUCKET or options.s3_bucket" if @options.s3_bucket.nil?
+
+      raise ArgumentError, "Specify an AWS Access Key Id in an env var called AWS_ACCESS_KEY_ID or options.aws_key" if @options.aws_key.nil?
+
+      raise ArgumentError, "Specify an AWS Secret Access Key in env var called AWS_SECRET_ACCESS_KEY or options.aws_secret" if @options.aws_secret.nil?
+    end
+  end
+end

data/lib/rstreet/version.rb

@@ -0,0 +1,3 @@
+module Rstreet
+  VERSION = "1.0.1"
+end

data/lib/s3.rb

@@ -0,0 +1,19 @@
+require "aws-sdk"
+
+module Rstreet
+  class S3
+    def initialize(s3_bucket)
+      @s3 = AWS::S3.new
+      @s3_bucket = s3_bucket
+    end
+
+    def get_file(uploadable)
+      @s3.buckets[@s3_bucket].objects[uploadable.name].read
+    rescue AWS::S3::Errors::NoSuchKey
+    end
+
+    def put_file(uploadable)
+      @s3.buckets[@s3_bucket].objects[uploadable.name].write(file: uploadable.path, acl: :authenticated_read)
+    end
+  end
+end

data/lib/uploadable.rb

@@ -0,0 +1,24 @@
+require "mime/types"
+
+module Rstreet
+  class Uploadable
+    attr_reader :name, :path, :hash
+
+    def initialize(name, path)
+      @name, @path = name, path
+      @hash = gen_hash || ""
+      @content_type = gen_content_type || "application/octet-stream"
+    end
+
+    private
+
+    def gen_hash
+      Digest::MD5.file(@path).hexdigest if File.file?(@path)
+    end
+
+    def gen_content_type
+      MIME::Types.type_for(@path).first.content_type
+    rescue
+    end
+  end
+end

data/lib/uploadable_collector.rb

@@ -0,0 +1,72 @@
+require "files_reader"
+require "manifest_builder"
+require "uploadable"
+
+module Rstreet
+  class UploadableCollector
+
+    # TODO: Shouldn't need to get this for the pull_manifest in rstreet.rb
+    # Instead, consider some class method with the default file path info
+    attr_reader :manifest_builder, :manifest_uploadable
+
+    def initialize(src)
+      @src = File.realpath src
+      # TODO: get rid of cache and consolidate inside of manifest builder
+      @uploadables_cache = {}
+      init_manifest_builder
+    end
+
+    def collect
+      files = FilesReader.read_all(@src)
+      uploadables = convert_files_to_uploadables files
+      uploadables = add_manifest_to_uploadables uploadables
+      save_uploadables_in_manifest uploadables
+      uploadables
+    end
+
+    # TODO: be more consistent with name, file, path var names
+    def find_uploadables(file_names)
+      @uploadables_cache.select { |k, v| file_names.include? k }.values
+    end
+
+    private
+
+    def save_uploadables_in_manifest(uploadables)
+      uploadables.each do |u|
+        @manifest_builder.add u
+        @uploadables_cache[u.name] = u
+      end
+      @manifest_builder.write
+    end
+
+    def init_manifest_builder
+      manifest_file = File.join(@src, ".manifest.json.gz")
+      @manifest_uploadable = convert_to_uploadable(manifest_file)
+      @manifest_builder = ManifestBuilder.new manifest_file
+    end
+
+    def convert_files_to_uploadables(files)
+      files.map do |file|
+        convert_to_uploadable(file)
+      end
+    end
+
+    def add_manifest_to_uploadables(uploadables)
+      uploadables << @manifest_uploadable if uploadables.any?
+      uploadables
+    end
+
+    def convert_to_uploadable(file)
+      Uploadable.new(format_uploadable_name(file), file)
+    end
+
+    def format_uploadable_name(file)
+      file.sub(ensure_ends_with_separator(@src), "")
+    end
+
+    def ensure_ends_with_separator(path)
+      path.end_with?(File::SEPARATOR) ? path : "#{path}#{File::SEPARATOR}"
+    end
+
+  end
+end

data/rstreet.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'rstreet/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "rstreet"
+  spec.version       = Rstreet::VERSION
+  spec.authors       = ["jaketrent"]
+  spec.email         = ["trent.jake@gmail.com"]
+  spec.summary       = %q{Smart uploader for S3}
+  spec.description   = %q{Smart s3 uploader.  Generates a manifest file on first upload.  Minimizes future uploads by only sending changed files.}
+  spec.homepage      = "https://github.com/jaketrent/rstreet"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "dotenv", "~> 1.0.2"
+  spec.add_development_dependency "mime-types", "~> 2.4.3"
+  spec.add_development_dependency "aws-sdk", "~> 2"
+end

metadata

@@ -0,0 +1,134 @@
+--- !ruby/object:Gem::Specification
+name: rstreet
+version: !ruby/object:Gem::Version
+  version: 1.0.1
+platform: ruby
+authors:
+- jaketrent
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-02-22 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: dotenv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.0.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.0.2
+- !ruby/object:Gem::Dependency
+  name: mime-types
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.4.3
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.4.3
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+description: Smart s3 uploader.  Generates a manifest file on first upload.  Minimizes
+  future uploads by only sending changed files.
+email:
+- trent.jake@gmail.com
+executables:
+- rstreet
+extensions: []
+extra_rdoc_files: []
+files:
+- ".env.example"
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/rstreet
+- lib/bucket_communicator.rb
+- lib/files_reader.rb
+- lib/manifest_builder.rb
+- lib/opts_parser.rb
+- lib/rstreet.rb
+- lib/rstreet/version.rb
+- lib/s3.rb
+- lib/uploadable.rb
+- lib/uploadable_collector.rb
+- rstreet.gemspec
+homepage: https://github.com/jaketrent/rstreet
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.3
+signing_key: 
+specification_version: 4
+summary: Smart uploader for S3
+test_files: []
+has_rdoc: