active_storage_encryption 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9c79ba1ae2b68a8d4e76c9d02092e426a41ddd414a968a082121cab647c2e239
-  data.tar.gz: ee60709d1843c4a814c52496a4fffcdec44b36bd3a48f18513fb0b56ba4a0c05
+  metadata.gz: 1f97954b59908255d0d2e09943e111ca33af803ca47d3a9af06a44e23064f725
+  data.tar.gz: 700a8909d6492c8a8a8633c24a15418428c5e740d15adeef2cfcb15dcf872ac9
 SHA512:
-  metadata.gz: 2d73fa7ea374c47f4fea531791db5962d469c8e1a95e9e7f9b0f64b9904e1b5948fd298b788b2a6c93a2e8beff0d58fde6ba24aa00c3c852b316232780a4ba4e
-  data.tar.gz: 7a2427499a85bec52196dfbdf304193618febcf7c3ceb0eb0944a01452683dbbb2ff2304a716ae3694fb9f55f6683c337b4d19189f67ce034abe8a58e093a0f5
+  metadata.gz: 5eef06c92277075e3b5b9c5b4421f6b8352956b48f29b442d70b777ce737326f0470a728903ae1a42994792deb2d026b30b7497ea01c9ed723965e2e03c0d0d9
+  data.tar.gz: d94575af669034bccf27f41de435c44219f247d71c8d19e3c24d27e0af5301e55e53afccd141e1c5047921880836380aa414f628f4602cafbce82831ceb35d5a

data/README.md

@@ -1,6 +1,4 @@
-## What does this do?
-
-This library will enable the use of per-blob encryption keys with ActiveStorage. It enables file encryption with a separate encryption key generated for every `ActiveStorage::Blob`. It uses [CSEK](https://cloud.google.com/storage/docs/encryption/using-customer-supplied-keys) on Google Cloud, [SSE-C](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html#specifying-s3-c-encryption) on AWS, and [block_cipher_kit](https://rubygems.org/gems/block_cipher_kit) for files on disk to add a layer of encryption to every uploaded file. Every `Blob` gets its own, random encryption key.
+This library enables use of per-blob encryption keys with ActiveStorage, with a separate encryption key for every `Blob`. To implement encryption, it enables the use of  [CSEK](https://cloud.google.com/storage/docs/encryption/using-customer-supplied-keys) on Google Cloud, [SSE-C](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html#specifying-s3-c-encryption) on AWS, and [block_cipher_kit](https://rubygems.org/gems/block_cipher_kit) for files on disk.
 
 During streaming download, either the cloud provider or a Rails controller will decrypt the requested chunk of the file as it gets served to the client.
 
@@ -185,10 +183,31 @@ When we stream through the controller, we encrypt the token instead of just sign
 
 ## Direct uploads
 
+Our recommended setup is to have your encrypted ActiveStorage service as an additional service configuration in `service.yml`:
+
+```yaml
+development:
+  public: true
+  service: Disk
+  root: <%= Rails.root.join("storage") %>
+
+encrypted_disk:
+  service: EncryptedDisk
+  root: <%= Rails.root.join("storage", "encrypted") %>
+  private_url_policy: stream
+
+```
+
+To upload into a named service that is non-default, you will need to use a different method for generating your presigned upload URL, as the standard Rails controller that creates the `Blob` records prior to upload does not allow you to set it. If you follow the [officlal Rails guide](https://guides.rubyonrails.org/active_storage_overview.html#direct-uploads) you will need to use a different URL helper for generating a URL to create blobs, which _does_ accommodate the service name. The URL you need to use is `create_encrypted_blob_direct_upload_url` (instead of `rails_direct_uploads_url`).
+
+This is not going to be necessary once the corresponding [Rails issue](https://github.com/rails/rails/issues/38940) will get addressed.
+
 All supported services accept the encryption key as part of the headers for the PUT direct upload. The provided Service implementations will generate the correct headers for you. However, your upload client _must_ use the headers provided to you by the server, and not invent its own. The standard ActiveStorage JS bundles honor those headers - but if you use your own uploader you will need to ensure it honors and forwards the headers too.
 
 We also configure the services to generate _and_ require the `Content-MD5` header. The client doing the PUT will need to precompute the MD5 for the object before starting the PUT request or getting a PUT url (as the checksum gets signed by the cloud SDK or by our `EncryptedDiskService`). This is so that any data transmission errors can be intercepted early (we know of one case where a bug in Ruby, combined with a particular HTTP client library, led to bytes getting uploaded out of order).
 
+Note that the encryption headers may require amending your CORS configuration - see the documentation per service regarding that.
+
 ## Security considerations / notes
 
 * It is imperative that you let the server generate the encryption key, and not generate one on the client. Where possible, we add the headers for the encryption key to the signed URL parameters, so client generated keys will deliberately _not_ function. Letting the client generate the keys can lead to key reuse (unintentional or - worse - intentional).

data/Rakefile

@@ -7,8 +7,6 @@ APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
 load "rails/tasks/engine.rake"
 load "rails/tasks/statistics.rake"
 
-require "bundler/gem_tasks"
-
 task :format do
   `bundle exec standardrb --fix`
   `bundle exec magic_frozen_string_literal .`

data/config/routes.rb

@@ -2,6 +2,6 @@
 
 ActiveStorageEncryption::Engine.routes.draw do
   put "/blob/:token", to: "encrypted_blobs#update", as: "encrypted_blob_put"
-  get "/blob/:token/*filename(.:format)", to: "encrypted_blobs#show", as: "encrypted_blob_streaming_get"
   post "/blob/direct-uploads", to: "encrypted_blobs#create_direct_upload", as: "create_encrypted_blob_direct_upload"
+  get "/blob/:token/*filename(.:format)", to: "encrypted_blob_proxy#show", as: "encrypted_blob_streaming_get"
 end

data/gemfiles/rails_7.gemfile.lock

@@ -1,10 +1,11 @@
 PATH
   remote: ..
   specs:
-    active_storage_encryption (0.1.0)
+    active_storage_encryption (0.2.0)
       activestorage
       block_cipher_kit (>= 0.0.4)
       rails (>= 7.2.2.1)
+      serve_byte_range (~> 1.0)
 
 GEM
   remote: https://rubygems.org/
@@ -151,6 +152,8 @@ GEM
     net-smtp (0.5.1)
       net-protocol
     nio4r (2.7.4)
+    nokogiri (1.18.3-arm64-darwin)
+      racc (~> 1.4)
     nokogiri (1.18.3-x86_64-darwin)
       racc (~> 1.4)
     nokogiri (1.18.3-x86_64-linux-gnu)
@@ -227,6 +230,9 @@ GEM
       rubocop-ast (>= 1.31.1, < 2.0)
     ruby-progressbar (1.13.0)
     securerandom (0.4.1)
+    serve_byte_range (1.0.0)
+      rack (>= 1.0)
+    sqlite3 (2.6.0-arm64-darwin)
     sqlite3 (2.6.0-x86_64-darwin)
     sqlite3 (2.6.0-x86_64-linux-gnu)
     standard (1.45.0)
@@ -258,6 +264,8 @@ GEM
     zeitwerk (2.7.2)
 
 PLATFORMS
+  arm64-darwin-21
+  arm64-darwin-24
   x86_64-darwin
   x86_64-linux
 

data/gemfiles/rails_8.gemfile.lock

@@ -1,10 +1,11 @@
 PATH
   remote: ..
   specs:
-    active_storage_encryption (0.1.0)
+    active_storage_encryption (0.2.0)
       activestorage
       block_cipher_kit (>= 0.0.4)
       rails (>= 7.2.2.1)
+      serve_byte_range (~> 1.0)
 
 GEM
   remote: https://rubygems.org/
@@ -151,6 +152,8 @@ GEM
     net-smtp (0.5.1)
       net-protocol
     nio4r (2.7.4)
+    nokogiri (1.18.3-arm64-darwin)
+      racc (~> 1.4)
     nokogiri (1.18.3-x86_64-darwin)
       racc (~> 1.4)
     nokogiri (1.18.3-x86_64-linux-gnu)
@@ -227,6 +230,9 @@ GEM
       rubocop-ast (>= 1.31.1, < 2.0)
     ruby-progressbar (1.13.0)
     securerandom (0.4.1)
+    serve_byte_range (1.0.0)
+      rack (>= 1.0)
+    sqlite3 (2.6.0-arm64-darwin)
     sqlite3 (2.6.0-x86_64-darwin)
     sqlite3 (2.6.0-x86_64-linux-gnu)
     standard (1.45.0)
@@ -258,6 +264,8 @@ GEM
     zeitwerk (2.7.2)
 
 PLATFORMS
+  arm64-darwin-21
+  arm64-darwin-24
   x86_64-darwin
   x86_64-linux
 

data/lib/active_storage_encryption/encrypted_blob_proxy_controller.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require "serve_byte_range"
+
+# This controller is analogous to the ActiveStorage::ProxyController
+class ActiveStorageEncryption::EncryptedBlobProxyController < ActionController::Base
+  include ActiveStorage::SetCurrent
+
+  class InvalidParams < StandardError
+  end
+
+  DEFAULT_BLOB_STREAMING_DISPOSITION = "inline"
+
+  self.etag_with_template_digest = false
+  skip_forgery_protection
+
+  # Streams the decrypted contents of an encrypted blob
+  def show
+    params = read_params_from_token_and_headers_for_get
+    service = lookup_service(params[:service_name])
+    raise InvalidParams, "#{service.name} does not allow private URLs" if service.private_url_policy == :disable
+
+    # Test the encryption key beforehand, so that the exception does not get raised when serving the actual body
+    service.download_chunk(params[:key], 0..0, encryption_key: params[:encryption_key])
+
+    stream_blob(service:,
+      key: params[:key],
+      encryption_key: params[:encryption_key],
+      blob_byte_size: params[:blob_byte_size],
+      filename: params[:filename],
+      disposition: params[:disposition] || DEFAULT_BLOB_STREAMING_DISPOSITION,
+      type: params[:content_type])
+  rescue ActiveStorage::FileNotFoundError
+    head :not_found
+  rescue InvalidParams, ActiveStorageEncryption::StreamingTokenInvalidOrExpired, ActiveSupport::MessageEncryptor::InvalidMessage, ActiveStorageEncryption::IncorrectEncryptionKey
+    head :forbidden
+  end
+
+  private
+
+  def read_params_from_token_and_headers_for_get
+    token_str = params.require(:token)
+
+    # The token params for GET / private_url download are encrypted, as they contain the object encryption key.
+    token_params = ActiveStorageEncryption.token_encryptor.decrypt_and_verify(token_str, purpose: :encrypted_get).symbolize_keys
+    encryption_key = Base64.decode64(token_params.fetch(:encryption_key))
+    service = lookup_service(token_params.fetch(:service_name))
+
+    # To be more like cloud services: verify presence of headers, if we were asked to (but this is optional)
+    if service.private_url_policy == :require_headers
+      b64_encryption_key = request.headers["x-active-storage-encryption-key"]
+      raise InvalidParams, "x-active-storage-encryption-key header is missing" if b64_encryption_key.blank?
+      raise InvalidParams, "Incorrect encryption key supplied via header" unless Rack::Utils.secure_compare(Base64.decode64(b64_encryption_key), encryption_key)
+    end
+
+    {
+      key: token_params.fetch(:key),
+      service_name: token_params.fetch(:service_name),
+      disposition: token_params.fetch(:disposition),
+      content_type: token_params.fetch(:content_type),
+      encryption_key: Base64.decode64(token_params.fetch(:encryption_key)),
+      blob_byte_size: token_params.fetch(:blob_byte_size)
+    }
+  end
+
+  def lookup_service(name)
+    service = ActiveStorage::Blob.services.fetch(name) { ActiveStorage::Blob.service }
+    raise InvalidParams, "No ActiveStorage default service defined and service #{name.inspect} was not found" unless service
+    raise InvalidParams, "#{service.name} is not providing file encryption" unless service.try(:encrypted?)
+    service
+  end
+
+  def stream_blob(service:, key:, blob_byte_size:, encryption_key:, filename:, disposition:, type:)
+    # The ActiveStorage::ProxyController buffers the entire response into memory
+    # when serving multipart byte ranges, which is extremely inefficient. We use our own thing
+    # which can actually stream from the Service directly, using byte ranges. This limits the
+    # amount of data buffered to 5 megabytes. There can be a better scheme with pagewise caching
+    # in tempfiles, but that's for later.
+    streaming_proc = ->(client_requested_range, response_io) {
+      chunk_size = 5.megabytes
+      client_requested_range.begin.step(client_requested_range.end, chunk_size) do |subrange_start|
+        chunk_end = subrange_start + chunk_size - 1
+        subrange_end = (chunk_end > client_requested_range.end) ? client_requested_range.end : chunk_end
+        range_on_service = subrange_start..subrange_end
+        response_io.write(service.download_chunk(key, range_on_service, encryption_key:))
+      end
+    }
+
+    # A few header things for streaming:
+    # 1. We need to ensure Rack::ETag does not suddenly start buffering us, for that either
+    # the ETag header or the Last-Modified header must be set. We set an ETag from the blob key,
+    # so nothing to do here.
+    # 2. Disable buffering for both nginx and Google Load Balancer, see
+    # https://cloud.google.com/appengine/docs/flexible/how-requests-are-handled?tab=python#x-accel-buffering
+    response.headers["X-Accel-Buffering"] = "no"
+    # 3. Make sure Rack::Deflater does not touch our response body either, see
+    # https://github.com/felixbuenemann/xlsxtream/issues/14#issuecomment-529569548
+    response.headers["Content-Encoding"] = "identity"
+
+    # Range requests use ETags to ensure that if a client goes to download a range of a resource
+    # it has already has some data of, it either gets the full resource - if it changed - or
+    # the bytes the client requested. An ActiveStorage blob never changes once it has been uploaded -
+    # it stays on the service "just as it was" until it gets deleted, so we can reliably use the key
+    # of the blob as the ETag.
+    blob_etag = key.inspect # Strong ETags must be quoted
+    status, headers, ranges_body = ServeByteRange.serve_ranges(request.env,
+      resource_size: blob_byte_size,
+      etag: blob_etag, # TODO
+      resource_content_type: type,
+      &streaming_proc)
+
+    response.status = status
+    headers.each { |(header, value)| response.headers[header] = value }
+    self.response_body = ranges_body
+  end
+end

data/lib/active_storage_encryption/encrypted_blobs_controller.rb

@@ -3,10 +3,6 @@
 class ActiveStorageEncryption::EncryptedBlobsController < ActionController::Base
   include ActiveStorage::SetCurrent
 
-  # Below similar to ActiveStorage::Streaming but ActionController::Live is meh.
-  include ActionController::DataStreaming
-  include ActionController::Live
-
   class InvalidParams < StandardError
   end
 
@@ -32,24 +28,6 @@ class ActiveStorageEncryption::EncryptedBlobsController < ActionController::Base
     head :unprocessable_entity
   end
 
-  # Streams the decrypted contents of an encrypted blob
-  def show
-    params = read_params_from_token_and_headers_for_get
-    service = lookup_service(params[:service_name])
-    raise InvalidParams, "#{service.name} does not allow private URLs" if service.private_url_policy == :disable
-
-    key = params[:key]
-    encryption_key = params[:encryption_key]
-
-    send_stream(filename: params[:filename], disposition: params[:disposition] || DEFAULT_BLOB_STREAMING_DISPOSITION, type: params[:content_type]) do |stream|
-      service.download(key, encryption_key: encryption_key) do |chunk|
-        stream.write chunk
-      end
-    end
-  rescue InvalidParams, ActiveStorageEncryption::StreamingTokenInvalidOrExpired, ActiveSupport::MessageEncryptor::InvalidMessage, ActiveStorageEncryption::IncorrectEncryptionKey
-    head :forbidden
-  end
-
   # Creates a Blob record with a random encryption key and returns the details for PUTing it
   # This is only necessary because in Rails there is some disagreement regarding the service_name parameter.
   # See https://github.com/rails/rails/issues/38940
@@ -111,35 +89,6 @@ class ActiveStorageEncryption::EncryptedBlobsController < ActionController::Base
     }
   end
 
-  def read_params_from_token_and_headers_for_get
-    token_str = params.require(:token)
-
-    # The token params for GET / private_url download are encrypted, as they contain the object encryption key.
-    token_params = ActiveStorageEncryption.token_encryptor.decrypt_and_verify(token_str, purpose: :encrypted_get).symbolize_keys
-    encryption_key = Base64.decode64(token_params.fetch(:encryption_key))
-
-    service = lookup_service(token_params.fetch(:service_name))
-
-    # To be more like cloud services: verify presence of headers, if we were asked to (but this is optional)
-    if service.private_url_policy == :require_headers
-      b64_encryption_key = request.headers["x-active-storage-encryption-key"]
-      raise InvalidParams, "x-active-storage-encryption-key header is missing" if b64_encryption_key.blank?
-      raise InvalidParams, "Incorrect encryption key supplied via header" unless Rack::Utils.secure_compare(Base64.decode64(b64_encryption_key), encryption_key)
-    end
-
-    # Verify the SHA of the encryption key
-    encryption_key_b64sha = Digest::SHA256.base64digest(encryption_key)
-    raise InvalidParams, "Incorrect encryption key supplied via token" unless Rack::Utils.secure_compare(encryption_key_b64sha, token_params.fetch(:encryption_key_sha256))
-
-    {
-      key: token_params.fetch(:key),
-      encryption_key: encryption_key,
-      service_name: token_params.fetch(:service_name),
-      disposition: token_params.fetch(:disposition),
-      content_type: token_params.fetch(:content_type)
-    }
-  end
-
   def lookup_service(name)
     service = ActiveStorage::Blob.services.fetch(name) { ActiveStorage::Blob.service }
     raise InvalidParams, "#{service.name} is not providing file encryption" unless service.try(:encrypted?)

data/lib/active_storage_encryption/encrypted_s3_service.rb

@@ -183,6 +183,7 @@ class ActiveStorageEncryption::EncryptedS3Service < ActiveStorage::Service::S3Se
       sse_options_for_presigned_url.delete(:sse_customer_key)
 
       options_for_super = options.merge(sse_options_for_presigned_url) # The "rest" kwargs for super are the `client_options`
+      options_for_super.delete(:blob_byte_size) # This is not a valid S3 option
       super(key, **options_for_super)
     end
   end

data/lib/active_storage_encryption/overrides.rb

@@ -138,6 +138,7 @@ module ActiveStorageEncryption
             key, expires_in: expires_in, filename: ActiveStorage::Filename.wrap(filename || self.filename),
             encryption_key: encryption_key,
             content_type: content_type_for_serving, disposition: forced_disposition_for_serving || disposition,
+            blob_byte_size: byte_size,
             **options
           )
         else

data/lib/active_storage_encryption/private_url_policy.rb

@@ -18,14 +18,15 @@ module ActiveStorageEncryption::PrivateUrlPolicy
     @private_url_policy
   end
 
-  def private_url_for_streaming_via_controller(key, expires_in:, filename:, content_type:, disposition:, encryption_key:)
+  def private_url_for_streaming_via_controller(key, blob_byte_size:, expires_in:, filename:, content_type:, disposition:, encryption_key:)
     if private_url_policy == :disable
       raise ActiveStorageEncryption::StreamingDisabled, <<~EOS
         Requested a signed GET URL for #{key.inspect} on service #{name}. This service
         has disabled presigned URLs (private_url_policy: disable), you have to use `Blob#download` instead.
       EOS
     end
-
+    # This method requires the "blob_byte_size" because it is needed for HTTP ranges (you need to know the range of a resource),
+    # The ActiveStorage::ProxyController retrieves the blob from the DB for that, but we can embed it right in the token.
     content_disposition = content_disposition_with(type: disposition, filename: filename)
     verified_key_with_expiration = ActiveStorageEncryption.token_encryptor.encrypt_and_sign(
       {
@@ -34,6 +35,7 @@ module ActiveStorageEncryption::PrivateUrlPolicy
         encryption_key_sha256: Digest::SHA256.base64digest(encryption_key),
         content_type: content_type,
         service_name: name,
+        blob_byte_size: blob_byte_size,
         encryption_key: Base64.strict_encode64(encryption_key)
       },
       expires_in: expires_in,

data/lib/active_storage_encryption/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ActiveStorageEncryption
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/active_storage_encryption.rb

@@ -6,6 +6,7 @@ require "active_storage_encryption/engine"
 module ActiveStorageEncryption
   autoload :PrivateUrlPolicy, __dir__ + "/active_storage_encryption/private_url_policy.rb"
   autoload :EncryptedBlobsController, __dir__ + "/active_storage_encryption/encrypted_blobs_controller.rb"
+  autoload :EncryptedBlobProxyController, __dir__ + "/active_storage_encryption/encrypted_blob_proxy_controller.rb"
   autoload :EncryptedDiskService, __dir__ + "/active_storage_encryption/encrypted_disk_service.rb"
   autoload :EncryptedMirrorService, __dir__ + "/active_storage_encryption/encrypted_mirror_service.rb"
   autoload :EncryptedS3Service, __dir__ + "/active_storage_encryption/encrypted_s3_service.rb"