s3browser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: a2af7ec1f09ccd3a5a9108b4db4cd02fff8d55bc
+  data.tar.gz: 37bc0b82fc54b2716711388437e688d32db12e06
+SHA512:
+  metadata.gz: bfe118747868fdb3e74470a4b9d9551c97a988fc2efbb1f3b038d3c8604053384fb1df67e701b013a64d674605370acec46c53049463566e7d1bf2edbda4d664
+  data.tar.gz: 0b807d66b7fa7dc2f97cf2ba658d1dcb9a643c9c1cf0c91a6d965314032652c7548f93210a2fcbbdae7b26cbad6d5dc0e4c0e12cee016773f939a9a5273521e4

data/.gitignore

@@ -0,0 +1,11 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+.env
+.vagrant

data/.ruby-version

@@ -0,0 +1 @@
+2.3.0

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in s3browser.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 TODO: Write your name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,164 @@
+# S3Browser
+
+The S3Browser is a simple wrapper around Amazon's [S3 Service](https://aws.amazon.com/s3/).
+Apart from listing files and managing, S3 doesn't give you a lot of functionality.
+
+This wrapper gives you a couple of killer functions:
+
+* [x] Search
+* [x] Sorting
+* [x] Automatically update file information
+* [x] Upload files
+* [x] Download files
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 's3browser'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install s3browser
+
+## Usage
+
+Here's an example config.ru for booting S3Browser::Server in your choice of Rack server:
+
+```ruby
+# config.ru
+require 's3browser/server'
+run S3Browser::Server
+```
+
+You can mount S3Browser to existing Rack (Sinatra) application as well:
+
+```ruby
+# config.ru
+require 'your_app'
+
+require 's3browser/server'
+run Rack::URLMap.new('/' => Sinatra::Application, '/s3browser' => S3Browser::Server)
+```
+
+To run S3Browser rake files, require the tasks file:
+
+```ruby
+# Rakefile
+require 's3browser/gem_tasks'
+```
+
+Run the fetcher
+
+```bash
+bundle exec rake s3browser:fetch
+```
+
+Run the server
+
+```bash
+bundle exec rake s3browser:server
+```
+
+Run the worker
+```
+bundle exec s3browser-worker
+```
+
+This will use the default config file `shoryuken-config.yml` generated by the setup process. You can override this and add any shoryuken options you want:
+
+```
+bundle exec s3browser-worker -C config.yml -L /var/log/shoryuken.log -P ./tmp/shoryuken.pid 
+```
+
+## Real time updates
+
+S3 buckets can be configured to send [Event Notifications](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html)
+when certain events happen in a bucket.
+
+### Setup
+
+Run `bundle exec rake s3browser:setup` to ensure the correct ENV variables are set, and that your AWS setup is correct.
+
+It will:
+
+1. Record the neccesary ENV variables:
+  * AWS_ACCESS_KEY_ID
+  * AWS_SECRET_ACCESS_KEY
+  * AWS_REGION
+  * AWS_S3_BUCKET
+  * AWS_SQS_QUEUE
+2. Create the SQS queue, with the correct permissions
+3. Create the S3 bucket, with the correct notification configuration
+
+To do the setup, the IAM profile you're using to run the script should have the following policy in place:
+
+```javascript
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "S3BrowserFullAccess",
+            "Effect": "Allow",
+            "Action": [
+                "s3:GetObject",
+                "s3:GetObjectVersion",
+                "s3:ListBucket",
+                "s3:PutObject",
+                "s3:CreateBucket",
+                "s3:PutBucketNotification"
+            ],
+            "Resource": [
+                "arn:aws:s3:::*",
+                "arn:aws:s3:::*/*"
+            ]
+        },
+        {
+            "Sid": "Stmt1460660107000",
+            "Effect": "Allow",
+            "Action": [
+                "sqs:DeleteMessage",
+                "sqs:ReceiveMessage",
+                "sqs:CreateQueue",
+                "sqs:GetQueueUrl",
+                "sqs:GetQueueAttributes",
+                "sqs:SetQueueAttributes"
+            ],
+            "Resource": [
+                "arn:aws:sqs:*"
+            ]
+        }
+    ]
+}
+```
+
+After setting up the queue and the bucket, a policy **without** the following actions can be used:
+
+* s3:CreateBucket
+* s3:PutBucketNotification
+* sqs:CreateQueue
+* sqs:GetQueueUrl
+* sqs:GetQueueAttributes
+* sqs:SetQueueAttributes
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/jrgns/s3browser.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,3 @@
+require 'bundler/gem_tasks'
+require 's3browser/gem_tasks'
+task :default => :spec

data/Vagrantfile

@@ -0,0 +1,53 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure(2) do |config|
+  config.vm.box = "ubuntu/trusty64"
+
+  config.vm.network "forwarded_port", guest: 9200, host: 9200
+  config.vm.network "forwarded_port", guest: 9292, host: 9292
+
+  config.vm.provider "virtualbox" do |vb|
+    # Customize the amount of memory on the VM:
+    vb.memory = "2048"
+  end
+
+  config.vm.provision "shell", privileged: false, inline: <<-SHELL
+    # Repositories
+    wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
+    echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
+    sudo add-apt-repository ppa:webupd8team/java
+    sudo apt-get update
+    sudo apt-get upgrade -y
+
+    # Java
+    echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
+    echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
+    sudo apt-get install -y oracle-java8-installer
+
+    # Elasticsearch
+    sudo apt-get install -y elasticsearch
+    sudo service elasticsearch stop
+    echo "ES_HEAP_SIZE=1g" | sudo tee -a /etc/default/elasticsearch
+
+    # Dependencies / Utilities
+    sudo apt-get install -y screen curl git build-essential
+
+    # Ruby
+    if [ ! -f /home/vagrant/.rvm/scripts/rvm ]
+    then
+        gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
+        \\curl -sSL https://get.rvm.io | bash
+    fi
+    source /home/vagrant/.rvm/scripts/rvm
+
+    # Gems
+    cd /vagrant
+    rvm use $(cat .ruby-version) --install
+    gem install bundler --no-rdoc --no-ri
+    bundle install
+
+    # Services
+    sudo service elasticsearch start
+  SHELL
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "s3browser"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+bundle exec rake s3browser:setup

data/exe/s3browser-worker

@@ -0,0 +1,25 @@
+#!/usr/bin/env ruby
+
+begin
+  require 'shoryuken/cli'
+  require 's3browser/worker'
+rescue LoadError
+  require 'rubygems'
+  require 'shoryuken/cli'
+  require 's3browser/worker'
+end
+
+unless ARGV.include?('-C')
+  ARGV << '-C'
+  ARGV << 'shoryuken-config.yml'
+end
+
+# This comes from the shoryuken CLI
+begin
+  Shoryuken::CLI.instance.run(ARGV)
+rescue => e
+  raise e if $DEBUG
+  STDERR.puts e.message
+  STDERR.puts e.backtrace.join("\n")
+  exit 1
+end

data/lib/s3browser.rb

@@ -0,0 +1,5 @@
+require "s3browser/version"
+
+module S3Browser
+  # Your code goes here...
+end

data/lib/s3browser/fetch.rb

@@ -0,0 +1,43 @@
+require 'aws-sdk'
+require 'time'
+require 's3browser/store'
+
+module S3Browser
+  class Fetch
+    class Store < S3Browser::Store
+      plugin :es
+      plugin :images
+    end
+
+    def run
+      s3.list_objects(bucket: bucket).contents.map do |object|
+        info = s3.head_object({
+          bucket: bucket,
+          key: object.key
+        })
+
+        info = info.to_h.merge(object.to_h)
+        store.add bucket, info
+      end
+    end
+
+    private
+    def bucket
+      @bucket ||= ENV['AWS_S3_BUCKET']
+    end
+
+    private
+    def s3
+      @s3 ||= Aws::S3::Client.new
+    end
+
+    private
+    def store
+      @store ||= Store.new('s3browser')
+    end
+  end
+end
+
+if $0 == __FILE__
+  S3Browser::Fetch.new.run
+end

data/lib/s3browser/gem_tasks.rb

@@ -0,0 +1,158 @@
+require 'rake'
+require 'rake/tasklib'
+require 's3browser'
+require 'yaml'
+
+module S3Browser
+  class GemTasks < ::Rake::TaskLib
+    include ::Rake::DSL if defined?(::Rake::DSL)
+
+    def install_tasks
+      require 'dotenv/tasks'
+
+      namespace :s3browser do
+        desc 'Run the web server for S3Browser'
+        task :server do
+          require 's3browser/server'
+          app = S3Browser::Server
+          app.set :environment, :production
+          app.set :bind, '0.0.0.0'
+          app.set :port, 9292
+          app.run!
+        end
+
+        desc 'Fetch and store all the S3 objects'
+        task :fetch do
+          require 's3browser/fetch'
+          S3Browser::Fetch.new.run
+        end
+
+        desc 'Set up the S3Browser'
+        task :setup => :dotenv do
+          require 'highline'
+          require 'json'
+          require 'aws-sdk'
+          require 'time'
+          require 'logger'
+          # Aws.config.update(logger: Logger.new($stdout), log_level: :debug, log_formatter: Aws::Log::Formatter.colored)
+
+
+          if File.exist?('.env') == false || cli.agree('Do you want to update your .env file (y/n)?') { |q| q.default = 'n' }
+            setup_env
+          end
+
+          if cli.agree('Should we set up SQS notification on the S3 bucket for you (y/n)?') { |q| q.default = 'y' }
+            setup_bucket
+            setup_sqs
+          end
+
+          if cli.agree('Do you want to generate a Shoryuken worker config (y/n)?') { |q| q.default = 'y' }
+            setup_worker_config
+          end
+        end
+      end
+    end
+
+    def setup_env
+      envs = {}
+      envs[:AWS_ACCESS_KEY_ID] = cli.ask('What is your AWS Access Key ID') { |q| q.validate = /^\w+$/; q.default = ENV['AWS_ACCESS_KEY_ID'] if ENV['AWS_ACCESS_KEY_ID'] }
+      envs[:AWS_SECRET_ACCESS_KEY] = cli.ask('What is your AWS Secret Access Key') { |q| q.validate = /^[\w\/\+]+$/; q.default = ENV['AWS_SECRET_ACCESS_KEY'] if ENV['AWS_SECRET_ACCESS_KEY'] }
+      envs[:AWS_REGION] = cli.ask('What AWS region should the service be located in') { |q| q.validate = /^[a-z]{2}\-[a-z]+\-\d$/; q.default = ENV['AWS_REGION'] if ENV['AWS_REGION'] }
+      envs[:AWS_S3_BUCKET] = cli.ask('What is the name of the S3 bucket to use') { |q| q.validate = /^[^ ]+$/; ; q.default = ENV['AWS_S3_BUCKET'] if ENV['AWS_S3_BUCKET'] }
+      envs[:AWS_SQS_QUEUE] = cli.ask('What is the name of the SQS queue to use') { |q| q.validate = /^[^ ]+$/; ; q.default = ENV['AWS_SQS_QUEUE'] if ENV['AWS_SQS_QUEUE'] }
+
+      envs_string = envs.map {|k,v| "#{k}=#{v}"}.join("\n") + "\n"
+
+      cli.say 'This is the proposed .env file:'
+      cli.say envs_string + "\n"
+
+      write_file = cli.agree 'Are you happy with the settings (y/n)? If yes, your current .env file will be overwritten'
+
+      if write_file
+        cli.say 'Writing .env file'
+        File.open('.env', 'w') { |file| file.write('# export $(cat .env | grep -v ^# | xargs)' + "\n" + envs_string) }
+        Dotenv.load! # Reload the .env files
+      else
+        cli.say 'Skipping the .env file'
+      end
+    end
+
+    def setup_bucket
+      # Ensure that the bucket exists
+      s3.create_bucket({
+        bucket: ENV['AWS_S3_BUCKET']
+      })
+      cli.say "Created the S3 bucket: #{ENV['AWS_S3_BUCKET']}"
+    rescue
+      cli.say "Bucket already exists: #{ENV['AWS_S3_BUCKET']}"
+    end
+
+    def setup_sqs
+      # Create the Queue
+      resp = sqs.create_queue(
+        queue_name: ENV['AWS_SQS_QUEUE']
+      )
+      queue_url = resp.to_h[:queue_url]
+      cli.say "Created queue: #{queue_url}"
+
+      # Replace the access policy on the queue to allow the bucket to write to it
+      queue_arn = sqs.get_queue_attributes(queue_url: queue_url, attribute_names: ['QueueArn']).attributes['QueueArn']
+      sqs.set_queue_attributes(
+        queue_url: queue_url,
+        attributes: {
+          'Policy' => sqs_policy(ENV['AWS_S3_BUCKET'], queue_arn)
+        }
+      )
+      cli.say 'Created the correct queue policy'
+
+      # Ensure that the bucket pushes notifications to the queue
+      s3.put_bucket_notification_configuration({
+        bucket: ENV['AWS_S3_BUCKET'],
+        notification_configuration: {
+          queue_configurations: [
+            {
+              id: "S3BrowserNotification",
+              queue_arn: queue_arn,
+              events: ['s3:ObjectCreated:*','s3:ObjectRemoved:*']
+            }
+          ]
+        }
+      })
+      cli.say 'Set the bucket to push notifications to SQS'
+    end
+
+    def setup_worker_config
+      config = {
+        'concurrency': 1,
+        'delay': 300,
+        'queues': [ ENV['AWS_SQS_QUEUE'] ]
+      }
+
+      cli.say 'Writing shoryuken-config.yml'
+      File.open('shoryuken-config.yml', 'w') { |file| file.write(config.to_yaml) }
+    end
+
+    def sqs
+      @sqs ||= Aws::SQS::Client.new
+    end
+
+    def s3
+      @s3 ||= Aws::S3::Client.new
+    end
+
+    def cli
+      @cli ||= HighLine.new
+    end
+
+    def sqs_policy(bucket_name, queue_arn)
+      folder = File.expand_path File.dirname(__FILE__)
+      policy = JSON.parse(File.read("#{folder}/policy.json"))
+      policy['Id'] = Time.now.strftime('%Y%m%dT%H%M%S')
+      policy['Statement'][0]['Condition']['ArnLike']['aws:SourceArn'] = "arn:aws:s3:*:*:#{bucket_name}"
+      policy['Statement'][0]['Resource'] = queue_arn
+      JSON.generate(policy)
+    end
+  end
+end
+
+S3Browser::GemTasks.new.install_tasks