paperclip-gcs 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1f1a924ccfce1e00e1528f8dd388dc410f4b9810
+  data.tar.gz: 1df9b5cc7b0ca186322ef4a3f02564bcf00ffa11
+SHA512:
+  metadata.gz: 91ef9146b4d9c3531f8b19c0ff54a226a7ab3c6fd10cb6ca0735f8e13b5a5b31fafad1b94962dc2c2a46e9ca9b60b21b410a1063abdd6079bd15c1b69b2c3656
+  data.tar.gz: 42fd00c5d874ac04e83919f1b6cbb5c0751dc2b3df6936dd8f3dd4ca54918039503281ca3970c625b82cad98bc144d5bfffeb26b4735ed47c4eb17dd537a0aef

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+before_install: gem install bundler -v 1.15.0

data/Gemfile

@@ -0,0 +1,4 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in paperclip-gcs.gemspec
+gemspec

data/README.md

@@ -0,0 +1,217 @@
+# paperclip-gcs
+
+paperclip-gcs is a Paperclip storage driver for storing files in a Google Cloud Storage.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'paperclip-gcs'
+```
+
+And then execute:
+
+``` shell
+$ bundle
+```
+
+Or install it yourself as:
+
+``` shell
+$ gem install paperclip-gcs
+```
+
+## Usage
+
+The GCS storage engine has been developed to work as similarly to S3 storage configuration as is possible. This gem can be configured in a Paperclip initializer as follows:
+
+``` ruby
+Paperclip::Attachment.default_options[:storage] = :gcs
+Paperclip::Attachment.default_options[:gcs_bucket] = "your-bucket"
+Paperclip::Attachment.default_options[:url] = ":gcs_path_url"
+Paperclip::Attachment.default_options[:path] = ":class/:attachment/:id/:style/:filename"
+Paperclip::Attachment.default_options[:gcs_credentials] = {
+    project: ENV["GCS_PROJECT"],
+    keyfile: ENV["GCS_KEYFILE"],
+}
+
+```
+
+Or, at the level of the model such as in the following example:
+
+``` ruby
+has_attached_file :avatar,
+  storage: :gcs,
+  gcs_bucket: "your-bucket",
+  gcs_credentials: {
+    project: "your-project",
+    keyfile: "path/to/your/keyfile",
+  }
+```
+
+See also http://www.rubydoc.info/gems/paperclip/Paperclip/Storage/S3
+
+## Configuration
+
+### gcs_bucket
+
+GCS bucket name.
+
+### gcs_credentials
+
+You can provide the project and credential information to connect to the Storage service, or if you are running on Google Compute Engine this configuration is taken care of for you.
+
+#### project
+
+Project identifier for GCS. Project are discovered in the following order:
+
+* Specify project in `project`
+* Discover project in environment variables `STORAGE_PROJECT`, `GOOGLE_CLOUD_PROJECT`, `GCLOUD_PROJECT`
+* Discover GCE credentials
+
+#### keyfile
+
+Path of GCS service account credentials JSON file. Credentials are discovered in the following order:
+
+* Specify credentials path in `keyfile`
+* Discover credentials path in environment variables `GOOGLE_CLOUD_KEYFILE`, `GCLOUD_KEYFILE`
+* Discover credentials JSON in environment variables `GOOGLE_CLOUD_KEYFILE_JSON`, `GCLOUD_KEYFILE_JSON`
+* Discover credentials file in the Cloud SDK's path
+* Discover GCE credentials
+
+#### bucket (optional)
+
+Here you can specify the GCS bucket name. If `gcs_bucket` also has a bucket specification, the value of` gcs_bucket` will be used.
+
+### gcs_options
+
+#### retries
+
+Number of times to retry requests on server error.
+
+#### timeout
+
+Default timeout to use in requests.
+
+### gcs_protocol
+
+The protocol for the URLs generated to your GCS assets. Can be either 'http', 'https', or an empty string to generate protocol-relative URLs. Defaults to empty string.
+
+### gcs_host_alias
+
+The fully-qualified domain name (FQDN) that is the alias to the GCS domain of your bucket. Used with the :gcs_alias_url url interpolation. See the link in the url entry for more information about GCS domains and buckets.
+
+### gcs_host_name
+
+If you are using a bucket in a custom domain, write host_name.
+
+### gcs_encryption_key
+
+You can also choose to provide your own AES-256 key for server-side encryption. See also [Customer-supplied encryption keys](https://cloud.google.com/storage/docs/encryption#customer-supplied).
+
+### gcs_metadata
+
+User provided web-safe keys and arbitrary string values that will returned with requests for the file as "x-goog-meta-" response headers.
+
+You can set metadata on a per style bases by doing the following:
+
+``` ruby
+gcs_metadata: {
+  thumb: { "foo" => "bar" }
+}
+```
+
+Or globally:
+
+``` ruby
+gcs_metadata: { "foo" => "bar" }
+```
+
+### gcs_permissions
+
+Permission for the object in GCS. Acceptable values are:
+
+* `auth_read`       - File owner gets OWNER access, and allAuthenticatedUsers get READER access.
+* `owner_full`      - File owner gets OWNER access, and project team owners get OWNER access.
+* `owner_read`      - File owner gets OWNER access, and project team owners get READER access.
+* `private`         - File owner gets OWNER access.
+* `project_private` - File owner gets OWNER access, and project team members get access according to their roles.
+* `public_read`     - File owner gets OWNER access, and allUsers get READER access.
+
+Default is nil (bucket default object ACL). See also [official document](https://cloud.google.com/storage/docs/access-control/lists).
+
+You can set permissions on a per style bases by doing the following:
+
+``` ruby
+gcs_permissions: {
+  thumb: :public_read
+}
+```
+
+Or globally:
+
+``` ruby
+gcs_permissions: :public_read
+```
+
+### gcs_storage_class
+
+Storage class of the file. Acceptable values are:
+
+* `dra`            - Durable Reduced Availability
+* `nearline`       - Nearline Storage
+* `coldline`       - Coldline Storage
+* `multi_regional` - Multi-Regional Storage
+* `regional`       - Regional Storage
+* `standard`       - Standard Storage
+
+You can set storage class on a per style bases by doing the following:
+
+``` ruby
+gcs_storage_class: {
+  thumb: :multi_regional
+}
+```
+
+Or globally:
+
+``` ruby
+gcs_storage_class: :multi_regional
+```
+
+### Interpolates
+
+#### :gcs_alias_url
+
+``` ruby
+"#{attachment.gcs_protocol}//#{attachment.gcs_host_alias}/#{attachment.path(style)}"
+```
+
+#### :gcs_path_url
+
+``` ruby
+"#{attachment.gcs_protocol}//#{attachment.gcs_host_name}/#{attachment.gcs_bucket_name}/#{attachment.path(style)}"
+```
+
+#### :gcs_domain_url
+
+``` ruby
+"#{attachment.gcs_protocol}//#{attachment.gcs_bucket_name}.#{attachment.gcs_host_name}/#{attachment.path(style)}"
+```
+
+#### :asset_host
+
+``` ruby
+"#{attachment.path(style)}"
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/daichirata/paperclip-gcs.

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "paperclip/gcs"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/paperclip-gcs.rb

@@ -0,0 +1 @@
+require "paperclip/storage/gcs"

data/lib/paperclip/gcs/version.rb

@@ -0,0 +1,5 @@
+module Paperclip
+  module Gcs
+    VERSION = "0.1.0"
+  end
+end

data/lib/paperclip/storage/gcs.rb

@@ -0,0 +1,164 @@
+require "google-cloud-storage"
+
+require "paperclip/gcs/version"
+require "paperclip/storage/gcs/bucket_repository"
+require "paperclip/storage/gcs/client_repository"
+require "paperclip/storage/gcs/credentials_resolver"
+
+module Paperclip
+  module Storage
+    module Gcs
+      DEFAULT_GCS_HOST_NAME = "storage.googleapis.com"
+
+      def self.extended(base)
+        base.instance_eval do
+          @gcs_options        = @options[:gcs_options] || {}
+          @gcs_credentials    = @options[:gcs_credentials]
+          @gcs_bucket_name    = @options[:gcs_bucket]
+          @gcs_protocol       = @options[:gcs_protocol]
+          @gcs_host_alias     = @options[:gcs_host_alias]
+          @gcs_host_name      = @options[:gcs_host_name]
+          @gcs_encryption_key = @options[:gcs_encryption_key]
+          @gcs_metadata       = normalize_style(@options[:gcs_metadata])
+          @gcs_permissions    = normalize_style(@options[:gcs_permissions])
+          @gcs_storage_class  = normalize_style(@options[:gcs_storage_class])
+
+          unless @options[:url].to_s.match(%r{\A:gcs_(alias|path|domain)_url\z}) || @options[:url] == ":asset_host".freeze
+            @options[:path] = path_option.gsub(/:url/, @options[:url]).sub(%r{\A:rails_root/public/system}, "".freeze)
+            @options[:url] = ":gcs_path_url".freeze
+          end
+        end
+
+        Paperclip.interpolates(:gcs_alias_url) do |attachment, style|
+          "#{attachment.gcs_protocol}//#{attachment.gcs_host_alias}/#{attachment.path(style)}"
+        end unless Paperclip::Interpolations.respond_to?(:gcs_alias_url)
+
+        Paperclip.interpolates(:gcs_path_url) do |attachment, style|
+          "#{attachment.gcs_protocol}//#{attachment.gcs_host_name}/#{attachment.gcs_bucket_name}/#{attachment.path(style)}"
+        end unless Paperclip::Interpolations.respond_to?(:gcs_path_url)
+
+        Paperclip.interpolates(:gcs_domain_url) do |attachment, style|
+          "#{attachment.gcs_protocol}//#{attachment.gcs_bucket_name}.#{attachment.gcs_host_name}/#{attachment.path(style)}"
+        end unless Paperclip::Interpolations.respond_to?(:gcs_domain_url)
+
+        Paperclip.interpolates(:asset_host) do |attachment, style|
+          "#{attachment.path(style)}"
+        end unless Paperclip::Interpolations.respond_to?(:asset_host)
+      end
+
+      def expiring_url(time = 3600, style = default_style)
+        if file_path = path(style)
+          gcs_bucket.signed_url(file_path, expires: time)
+        else
+          url(style)
+        end
+      end
+
+      def exists?(style = default_style)
+        if original_filename
+          gcs_bucket.find_file(path(style))
+        else
+          false
+        end
+      rescue Google::Cloud::Error
+        false
+      end
+
+      def flush_writes
+        @queued_for_write.each do |style, file|
+          log("saving #{path(style)}")
+
+          opts = {
+            content_type: file.content_type,
+            encryption_key: gcs_encryption_key,
+            acl: gcs_permissions(style),
+            storage_class: gcs_storage_class(style),
+            metadata: gcs_metadata(style),
+          }
+          gcs_bucket.upload_file(file.path, path(style), opts)
+        end
+        after_flush_writes
+        @queued_for_write = {}
+      end
+
+      def flush_deletes
+        @queued_for_delete.each do |path|
+          begin
+            log("deleting #{path}")
+
+            gcs_bucket.file(path).delete
+          rescue Google::Cloud::Error
+            # Ignore this.
+          end
+        end
+        @queued_for_delete = []
+      end
+
+      def copy_to_local_file(style, local_dest_path)
+        log("copying #{path(style)} to local file #{local_dest_path}")
+
+        gcs_bucket.file(path(style)).download(local_dest_path, encryption_key: gcs_encryption_key)
+      rescue Google::Cloud::Error => e
+        warn("#{e} - cannot copy #{path(style)} to local file #{local_dest_path}")
+        false
+      end
+
+      def gcs_protocol
+        unwrap_proc(@gcs_protocol, self)
+      end
+
+      def gcs_host_alias
+        unwrap_proc(@gcs_host_alias, self)
+      end
+
+      def gcs_host_name
+        unwrap_proc(@gcs_host_name, self) || DEFAULT_GCS_HOST_NAME
+      end
+
+      def gcs_bucket_name
+        (unwrap_proc(@gcs_bucket_name, self) || gcs_credentials[:bucket]) or
+          raise ArgumentError, "missing required :gcs_bucket option"
+      end
+
+      private
+
+      def normalize_style(opts)
+        opts = { default: opts } unless opts.respond_to?(:merge)
+        opts.merge(default: opts[:default])
+      end
+
+      def unwrap_proc(obj, *args)
+        obj.respond_to?(:call) ? obj.call(*args) : obj
+      end
+
+      def gcs_encryption_key
+        unwrap_proc(@gcs_encryption_key, self)
+      end
+
+      def gcs_permissions(style = default_style)
+        unwrap_proc(@gcs_permissions[style] || @gcs_permissions[:default], self, style)
+      end
+
+      def gcs_storage_class(style = default_style)
+        unwrap_proc(@gcs_storage_class[style] || @gcs_storage_class[:default], self, style)
+      end
+
+      def gcs_metadata(style = default_style)
+        unwrap_proc(@gcs_metadata[style] || @gcs_metadata[:default], self, style)
+      end
+
+      def gcs_credentials
+        @_gcs_credentials ||= CredentialsResolver.resolve(unwrap_proc(@gcs_credentials, self))
+      end
+
+      def gcs_client
+        @_gcs_client ||= ClientRepository.find(gcs_credentials.slice(:project, :keyfile).merge(
+                                                 @gcs_options.slice(:scope, :retries, :timeout)))
+      end
+
+      def gcs_bucket
+        @_gcs_bucket ||= BucketRepository.find(gcs_client, gcs_bucket_name)
+      end
+    end
+  end
+end

data/lib/paperclip/storage/gcs/bucket_repository.rb

@@ -0,0 +1,25 @@
+module Paperclip
+  module Storage
+    module Gcs
+      class BucketRepository
+        include Singleton
+
+        CACHE_KEY = self.class.name.freeze
+
+        def self.find(client, bucket_name)
+          instance.find(client, bucket_name)
+        end
+
+        def find(client, bucket_name)
+          buckets[[client, bucket_name]] ||= client.bucket(bucket_name)
+        end
+
+        private
+
+        def buckets
+          Thread.current[CACHE_KEY] ||= {}
+        end
+      end
+    end
+  end
+end

data/lib/paperclip/storage/gcs/client_repository.rb

@@ -0,0 +1,29 @@
+module Paperclip
+  module Storage
+    module Gcs
+      class ClientRepository
+        include Singleton
+
+        CACHE_KEY = self.class.name.freeze
+
+        def self.find(config)
+          instance.find(config)
+        end
+
+        def find(config)
+          clients[config] ||= Google::Cloud.storage(
+            config[:project],
+            config[:keyfile],
+            config.slice(:scope, :retries, :timeout)
+          )
+        end
+
+        private
+
+        def clients
+          Thread.current[CACHE_KEY] ||= {}
+        end
+      end
+    end
+  end
+end

data/lib/paperclip/storage/gcs/credentials_resolver.rb

@@ -0,0 +1,29 @@
+module Paperclip
+  module Storage
+    module Gcs
+      module CredentialsResolver
+        module_function
+
+        def resolve(credentials)
+          cred = case credentials
+                 when File
+                   YAML.load(ERB.new(File.read(credentials.path)).result)
+                 when String, Pathname
+                   YAML.load(ERB.new(File.read(credentials)).result)
+                 when Hash
+                   credentials
+                 when NilClass
+                   {}
+                 else
+                   raise ArgumentError, ":gcs_credentials is not a path, file, nor a hash"
+                 end
+          (cred.stringify_keys[env] || cred).symbolize_keys
+        end
+
+        def env
+          (defined?(Rails) ? Rails.env : nil).to_s
+        end
+      end
+    end
+  end
+end

data/paperclip-gcs.gemspec

@@ -0,0 +1,28 @@
+# coding: utf-8
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "paperclip/gcs/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "paperclip-gcs"
+  spec.version       = Paperclip::Gcs::VERSION
+  spec.authors       = ["Daichi HIRATA"]
+  spec.email         = ["daichirata@gmail.com"]
+  spec.summary       = "Extends Paperclip with Google Cloud Storage"
+  spec.description   = "Extends Paperclip with Google Cloud Storage"
+  spec.homepage      = "https://github.com/daichirata/paperclip-gcs"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_runtime_dependency "paperclip", ">= 4.0"
+  spec.add_runtime_dependency "google-cloud-storage", "~> 1.0"
+
+  spec.add_development_dependency "bundler", "~> 1.15"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "minitest", "~> 5.0"
+end

metadata

@@ -0,0 +1,127 @@
+--- !ruby/object:Gem::Specification
+name: paperclip-gcs
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Daichi HIRATA
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2017-06-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: paperclip
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: google-cloud-storage
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.15'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.15'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+description: Extends Paperclip with Google Cloud Storage
+email:
+- daichirata@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- Gemfile
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/paperclip-gcs.rb
+- lib/paperclip/gcs/version.rb
+- lib/paperclip/storage/gcs.rb
+- lib/paperclip/storage/gcs/bucket_repository.rb
+- lib/paperclip/storage/gcs/client_repository.rb
+- lib/paperclip/storage/gcs/credentials_resolver.rb
+- paperclip-gcs.gemspec
+homepage: https://github.com/daichirata/paperclip-gcs
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Extends Paperclip with Google Cloud Storage
+test_files: []