gcs_put 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9d4c77b40e96536400be6f6bcff74265d7e175252afc19a6bea7d59dea9bd880
+  data.tar.gz: e983657f638cad02d7657293e4dd051b5e16667c4b1e1d480da656a3b7700240
+SHA512:
+  metadata.gz: 3b48f9d1fba5c38dab1f428c28065765579fb639c57af52c285f4e6d314e67d392dfa0e4bb04ec7ac48d0428e34e310bd77b4289e3ae54edaac09dd689b584bc
+  data.tar.gz: 45577085211d0da00ab42da1416a88c90baf02273b8b01427dc2da42932d63586cf75ef75a6d841ea19c4faf55d80fa6d3f139498cc37f7186cee3a90fe78b06

data/.gitignore

@@ -0,0 +1,9 @@
+# Reference code from another project, must not ship
+/tmp/
+/pkg/
+/vendor/bundle/
+/.bundle/
+/Gemfile.lock
+.DS_Store
+/.yardoc/
+/doc/

data/.standard.yml

@@ -0,0 +1,3 @@
+ruby_version: 3.2
+ignore:
+  - 'tmp/**/*'

data/.yardopts

@@ -0,0 +1,8 @@
+--markup markdown
+--markup-provider redcarpet
+--no-private
+lib/**/*.rb
+-
+README.md
+CHANGELOG.md
+LICENSE

data/CHANGELOG.md

@@ -0,0 +1,4 @@
+## 0.1.0
+
+- Initial release: `ByteChunker`, `ResumableUpload` with `with_gcs_file`, `with_signed_post_url` and `with_session_url`, also aliased on `GCSPut`, and the IAM-backed signer for workload identity
+- Pluggable transports, with `Net::HTTP` and Faraday included

data/Gemfile

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Julik Tarkhanov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,139 @@
+# gcs_put
+
+Streaming uploads to Google Cloud Storage when you do not know the size in advance.
+
+The official `google-cloud-storage` gem can only upload things it can measure up front. It sends the total size when it opens the resumable session, and builds a `Content-Range` with that total on every chunk. Hand it a pipe or a socket and it has nothing to measure. GCS itself has supported resumable uploads of unknown size for years. This gem uses that: it gives you a writable object, chops what you write into 256 KiB-aligned chunks, and PUTs them into a resumable upload session as you go. Generating a zip, a CSV export or a transcode straight into a bucket works without a temp file.
+
+## Installation
+
+```ruby
+gem "gcs_put"
+```
+
+## Usage
+
+```ruby
+require "gcs_put"
+
+storage = Google::Cloud::Storage.new
+bucket = storage.bucket("my-bucket")
+gcs_file = bucket.file("exports/report.csv.gz", skip_lookup: true) # does not need to exist
+
+GCSPut.with_gcs_file(gcs_file, content_type: "application/gzip") do |io|
+  gz = Zlib::GzipWriter.new(io)
+  rows.each { |row| gz.write(row.to_csv) }
+  gz.finish
+end
+```
+
+The object yielded to the block responds to `write`, `<<` and `close`, so anything that writes to an IO can write to it, including `IO.copy_stream`, `Zlib::GzipWriter` and `ZipKit::Streamer`. The last chunk is sent when the block returns, or earlier if something closes the object, and the block form returns the total number of bytes uploaded. If the block raises, nothing is finalized and the session simply expires after a week.
+
+The factories live on `GCSPut::ResumableUpload` and are aliased on `GCSPut` for brevity. Everything except the file is optional. The content type defaults to `binary/octet-stream`, HTTP goes through `Net::HTTP`, and chunks are 5 MB.
+
+Without a block you get the upload back to drive by hand:
+
+```ruby
+upload = GCSPut.with_gcs_file(gcs_file)
+upload.write(bytes)
+upload.finish # => total bytes
+```
+
+### Chunk size
+
+Every chunk except the last is held in memory and must be a multiple of 256 KiB. The default is 5 MB. Larger chunks mean fewer requests:
+
+```ruby
+GCSPut.with_gcs_file(gcs_file, chunk_size: 32 * 1024 * 1024)
+```
+
+### Starting from a session URL
+
+The session URL is just a string, and once you have it uploading needs no Google credentials at all. So a web process can sign and start the session while a worker does the upload:
+
+```ruby
+upload = GCSPut.with_gcs_file(gcs_file)
+session_url = upload.session_url
+
+# Elsewhere, no SDK needed
+GCSPut.with_session_url(session_url) do |io|
+  io.write(bytes)
+end
+```
+
+If you have a signed POST URL from somewhere else, one signed for `POST` with the `x-goog-resumable: start` header, start from that instead:
+
+```ruby
+GCSPut.with_signed_post_url(signed_post_url) do |io|
+  io.write(bytes)
+end
+```
+
+If the URL was signed, or the session started, with a content type other than the default, pass the same `content_type:`.
+
+The chunker is also usable on its own, for anything that needs evenly sized pieces:
+
+```ruby
+chunker = GCSPut::ByteChunker.new(chunk_size: 1024) { |bytes, is_last| ... }
+chunker << data
+chunker.finish
+```
+
+### Transports
+
+HTTP goes through a small transport object. The default uses `Net::HTTP` with one persistent connection and needs nothing installed. To use Faraday instead, pass a transport wrapping your own connection, so you get to pick the adapter, timeouts and instrumentation:
+
+```ruby
+conn = Faraday.new { |f| f.options.timeout = 120 }
+transport = GCSPut::Transport::Faraday.new(conn)
+
+GCSPut.with_gcs_file(gcs_file, transport: transport) { |io| ... }
+GCSPut.with_session_url(session_url, transport: transport)
+```
+
+Without an argument the Faraday transport makes a default connection. The `raise_error` middleware is tolerated. Timeouts and connection errors for `Net::HTTP` can be set on its transport too:
+
+```ruby
+GCSPut::Transport::NetHTTP.new(open_timeout: 10, read_timeout: 120)
+```
+
+Anything responding to `put(uri, body, headers)`, `post(uri, body, headers)` and `close` works as a transport. The verbs must return something with `status`, `body` and a case-insensitive `[]` for headers, and raise `GCSPut::TransientError` for failures worth retrying. Every chunk carries a `Content-MD5`, so an adapter that mangles request bodies fails loudly at upload time rather than quietly at download time. [httpx 1.4.0 did exactly that](https://gitlab.com/os85/httpx/-/issues/338).
+
+### Retries
+
+A chunk which fails with a connection error or a 5xx is not resent from the start. The session is asked how many bytes it has, and only the remainder goes out again. The same happens when GCS answers a PUT with a `Range` header showing it took fewer bytes than were sent. A chunk is given up on after 5 attempts, configurable via `max_attempts:`. A 4xx fails immediately.
+
+## Permissions
+
+Starting a session needs a signed POST URL, and signing needs a private key.
+
+**With a service account JSON key** (`GOOGLE_APPLICATION_CREDENTIALS` or `credentials:` on `Google::Cloud::Storage.new`) the key is local, the SDK signs with it, and the only permission needed is `storage.objects.create` on the bucket. Nothing else to do.
+
+**Under workload identity** on GCE, GKE, Cloud Run or Cloud Functions there is no private key on the machine. The gem detects this and asks the IAM Credentials API to sign on the service account's behalf via `signBlob`. For that call to be allowed, the service account must hold the `roles/iam.serviceAccountTokenCreator` role **on itself**:
+
+```sh
+gcloud iam service-accounts add-iam-policy-binding SA_EMAIL \
+  --member="serviceAccount:SA_EMAIL" \
+  --role="roles/iam.serviceAccountTokenCreator"
+```
+
+Without it, `with_gcs_file` fails with a permission error from the IAM API, not from Cloud Storage, which is confusing the first time. The background is in [google-cloud-ruby#13307](https://github.com/googleapis/google-cloud-ruby/issues/13307). The IAM Credentials API must also be enabled on the project.
+
+Extra options such as `expires:`, or an `issuer:` and `signer:` of your own, go in `signed_url_options:` and are passed through to `gcs_file.signed_url`.
+
+## Running the tests
+
+```sh
+bundle exec rake
+```
+
+Unit tests stub HTTP and need nothing. The live tests upload to a real bucket called `gcs_put_test_bucket`, or whatever `GCS_PUT_TEST_BUCKET` names, using the credentials and project the SDK finds on its own: `GOOGLE_APPLICATION_CREDENTIALS` and `GOOGLE_CLOUD_PROJECT`, or `gcloud auth application-default login`. Without a usable configuration or bucket they skip. Objects they create are deleted afterwards.
+
+## Resources
+
+- [Performing resumable uploads](https://cloud.google.com/storage/docs/performing-resumable-uploads)
+- [Signed URLs with resumable uploads](https://cloud.google.com/storage/docs/access-control/signed-urls#signing-resumable)
+- Google's own [stream_upload.rb gist](https://gist.github.com/frankyn/9a5344d1b19ed50ebbf9f15f0ff92032), which this gem started from
+
+## License
+
+MIT

data/Rakefile

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+require "standard/rake"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task :format do
+  `bundle exec standardrb --fix`
+  `bundle exec magic_frozen_string_literal .`
+end
+
+task :generate_typedefs do
+  # Types from other gems (URI, Faraday, Google::Cloud::Storage) are not resolvable for sord, hence the untyped fallback
+  `bundle exec sord --replace-errors-with-untyped rbi/gcs_put.rbi`
+  `bundle exec sord --replace-errors-with-untyped sig/gcs_put.rbs`
+end
+
+# When building the gem, generate typedefs beforehand so that they get included
+Rake::Task["build"].enhance(["generate_typedefs"])
+
+task default: [:test, :standard, :generate_typedefs]

data/gcs_put.gemspec

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require_relative "lib/gcs_put/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "gcs_put"
+  spec.version = GCSPut::VERSION
+  spec.authors = ["Julik Tarkhanov"]
+  spec.email = ["me@julik.nl"]
+  spec.license = "MIT"
+  spec.summary = "Streaming resumable uploads to Google Cloud Storage without knowing the size in advance."
+  spec.description = "Gives you a writable IO-like object which streams what you write into it to a GCS object using " \
+    "the resumable upload protocol over a signed URL. The size of the upload does not need to be known beforehand."
+
+  spec.homepage = "https://github.com/julik/gcs_put"
+  # The homepage link on rubygems.org only appears if you add homepage_uri. Just spec.homepage is not enough.
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/main/CHANGELOG.md"
+
+  spec.required_ruby_version = ">= 3.2.0" # google-cloud-storage wants 3.2
+
+  spec.metadata["allowed_push_host"] = "https://rubygems.org"
+  spec.files = `git ls-files -z`.split("\x0").reject do |path|
+    path.start_with?("tmp/", "test/", ".github/") || File.basename(path) == "Gemfile.lock"
+  end
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "google-cloud-storage", "~> 1.40"
+
+  spec.add_development_dependency "minitest"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "webmock"
+  spec.add_development_dependency "faraday"
+  spec.add_development_dependency "magic_frozen_string_literal"
+  spec.add_development_dependency "standard", ">= 1.35.1"
+  spec.add_development_dependency "yard", "~> 0.9"
+  spec.add_development_dependency "sord"
+  # redcarpet is needed for yard to render Github Flavored Markdown
+  spec.add_development_dependency "redcarpet"
+end

data/lib/gcs_put/byte_chunker.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+# Chops an arbitrary stream of writes into evenly sized chunks. Every chunk except the last
+# will be exactly `chunk_size` bytes, and the last one can be anything from 0 bytes up to and
+# including `chunk_size`. A chunk which fills up exactly is held back until `finish` so that
+# a stream ending on a chunk boundary still delivers that chunk flagged as the last one.
+#
+#   chunker = ByteChunker.new(chunk_size: 3) { |bytes, is_last| puts [bytes, is_last].inspect }
+#   chunker << "ab" << "cdefg"  # => ["abc", false], ["def", false]
+#   chunker.finish              # => ["g", true]
+class GCSPut::ByteChunker
+  # @param chunk_size[Integer] the size that every chunk except the last must have
+  # @yield [bytes, is_last] a binary String and whether this is the final chunk
+  def initialize(chunk_size:, &delivery_proc)
+    raise ArgumentError, "chunk_size must be positive" unless chunk_size.to_i > 0
+    @chunk_size = chunk_size.to_i
+    # A mutable String with preallocated capacity beats a StringIO here - the buffer
+    # gets reused for the whole life of the chunker and never reallocates
+    @buf = String.new(encoding: Encoding::BINARY, capacity: @chunk_size * 2)
+    @delivery_proc = delivery_proc.to_proc
+  end
+
+  # @param bin_str[String] the bytes to append
+  # @return [self]
+  def <<(bin_str)
+    @buf << bin_str.b
+    deliver_full_chunks
+    self
+  end
+
+  # @param bin_str[String] the bytes to append
+  # @return [Integer] the number of bytes appended, like `IO#write`
+  def write(bin_str)
+    self << bin_str
+    bin_str.bytesize
+  end
+
+  # Delivers whatever is left in the buffer as the last chunk. The last chunk
+  # gets delivered even when empty - it is what closes the upload
+  #
+  # @return [void]
+  def finish
+    deliver_full_chunks
+    # Hand out a copy - the receiver may hold on to it, and we reuse the buffer
+    @delivery_proc.call(@buf.byteslice(0, @buf.bytesize), _is_last = true)
+    @buf.clear
+    nil
+  end
+
+  private
+
+  # @return [void]
+  def deliver_full_chunks
+    while @buf.bytesize > @chunk_size
+      @delivery_proc.call(@buf.byteslice(0, @chunk_size), _is_last = false)
+      @buf.replace(@buf.byteslice(@chunk_size..))
+    end
+  end
+end

data/lib/gcs_put/resumable_upload.rb

@@ -0,0 +1,227 @@
+# frozen_string_literal: true
+
+# A writable object which streams what you write into a GCS resumable upload session using
+# ranged PUTs, see https://cloud.google.com/storage/docs/performing-resumable-uploads#chunked-upload
+# You do not need to know the size of the output in advance. You do need to `finish` the upload,
+# since it is the last PUT (with the total size filled in) which closes the GCS object.
+#
+#   gcs_file = bucket.file("upload.bin", skip_lookup: true)
+#   GCSPut.with_gcs_file(gcs_file) do |io|
+#     io.write("Hello resumable")
+#     20.times { io.write(Random.bytes(1024 * 1024)) }
+#   end
+#
+# Every chunk is retried from the byte offset GCS reports as persisted, so a chunk which
+# only partially made it over the wire gets topped up rather than resent from the start.
+#
+# Starting the session needs a signed POST URL, and signing needs a private key. Under
+# workload identity there is no key, so we fall back to the IAM signBlob API - which means
+# the service account must hold `roles/iam.serviceAccountTokenCreator` on itself. See
+# https://github.com/googleapis/google-cloud-ruby/issues/13307
+class GCSPut::ResumableUpload
+  extend Forwardable
+
+  # @!method write(bin_str)
+  #   Appends bytes to the upload, sending out a chunk whenever one fills up
+  #   @param bin_str[String]
+  #   @return [Integer] the number of bytes appended, like `IO#write`
+  # @!method <<(bin_str)
+  #   Appends bytes to the upload, sending out a chunk whenever one fills up
+  #   @param bin_str[String]
+  #   @return [self]
+  def_delegators :@chunker, :write, :<<
+
+  # @return [String] the session URL, valid for a week and usable from any process
+  attr_reader :session_url
+
+  # @return [Integer] the number of bytes GCS has confirmed as persisted so far
+  attr_reader :bytes_persisted
+
+  # Signs a session start URL for the object, starts the session and returns the upload.
+  # With a block, yields the upload, finishes it once the block returns and returns the total size.
+  #
+  # @param gcs_file[Google::Cloud::Storage::File] the object to upload into, does not need to exist yet
+  # @param content_type[String] the content type of the resulting object
+  # @param transport[#put, #post, #close] see `GCSPut::Transport`
+  # @param signed_url_options[Hash] passed to `gcs_file.signed_url`, see `Signer.url_issuer_and_signer`
+  # @param options[Hash] see {#initialize}
+  # @yield [GCSPut::ResumableUpload] the upload to write into
+  # @return [GCSPut::ResumableUpload, Integer] the upload, or the total size when given a block
+  def self.with_gcs_file(gcs_file, content_type: "binary/octet-stream", transport: GCSPut::Transport::NetHTTP.new, signed_url_options: {}, **options, &blk)
+    signed_url_options = GCSPut::Signer.url_issuer_and_signer.merge(signed_url_options)
+    signed_post_url = gcs_file.signed_url(method: "POST", content_type: content_type, headers: {"x-goog-resumable" => "start"}, **signed_url_options)
+    with_signed_post_url(signed_post_url, content_type: content_type, transport: transport, **options, &blk)
+  end
+
+  # Starts a session from a signed POST URL (one with `x-goog-resumable: start` among its signed headers)
+  # and returns the upload, see https://cloud.google.com/storage/docs/performing-resumable-uploads#initiate-session
+  # With a block, yields the upload, finishes it once the block returns and returns the total size.
+  #
+  # @param signed_post_url[String]
+  # @param content_type[String] must match the content type the URL was signed with
+  # @param transport[#put, #post, #close] see `GCSPut::Transport`
+  # @param options[Hash] see {#initialize}
+  # @yield [GCSPut::ResumableUpload] the upload to write into
+  # @return [GCSPut::ResumableUpload, Integer] the upload, or the total size when given a block
+  def self.with_signed_post_url(signed_post_url, content_type: "binary/octet-stream", transport: GCSPut::Transport::NetHTTP.new, **options, &blk)
+    response = transport.post(URI(signed_post_url), "", {"Content-Type" => content_type, "x-goog-resumable" => "start"})
+    unless response.status == 201
+      raise GCSPut::UploadFailed.new("Session start responded with HTTP #{response.status}: #{response.body}", response: response)
+    end
+    session_url = response["Location"] or raise GCSPut::UploadFailed.new("Session start did not return a Location header", response: response)
+    with_session_url(session_url, content_type: content_type, transport: transport, **options, &blk)
+  end
+
+  # Wraps an already started session. With a block, yields the upload, finishes it once
+  # the block returns and returns the total size.
+  #
+  # @param session_url[String] the `Location` returned by the session start
+  # @param options[Hash] see {#initialize}
+  # @yield [GCSPut::ResumableUpload] the upload to write into
+  # @return [GCSPut::ResumableUpload, Integer] the upload, or the total size when given a block
+  def self.with_session_url(session_url, **options)
+    upload = new(session_url, **options)
+    return upload unless block_given?
+    yield(upload)
+    upload.finish
+  end
+
+  # Prefer the `with_*` factories. This does no HTTP by itself, the first request goes out
+  # once a chunk fills up or `finish` gets called
+  #
+  # @param session_url[String] the `Location` returned by the session start
+  # @param chunk_size[Integer] must be a multiple of 256 KiB
+  # @param content_type[String] must match the content type the session was started with
+  # @param max_attempts[Integer] how many times a single chunk may be sent before giving up
+  # @param transport[#put, #post, #close] see `GCSPut::Transport`
+  def initialize(session_url, chunk_size: GCSPut::DEFAULT_CHUNK_SIZE, content_type: "binary/octet-stream", max_attempts: 5, transport: GCSPut::Transport::NetHTTP.new)
+    unless (chunk_size % GCSPut::CHUNK_SIZE_UNIT).zero?
+      raise ArgumentError, "chunk_size of #{chunk_size} is not a multiple of #{GCSPut::CHUNK_SIZE_UNIT}"
+    end
+
+    @session_url = session_url
+    @session_uri = URI(session_url)
+    @content_type = content_type
+    @max_attempts = max_attempts
+    @transport = transport
+    @bytes_persisted = 0
+    @finished = false
+    @chunker = GCSPut::ByteChunker.new(chunk_size: chunk_size) { |bytes, is_last| upload_chunk(bytes, is_last) }
+  end
+
+  # Sends the remaining buffered bytes as the final chunk and closes the GCS object.
+  # Also available as `close` so that writers which close their underlying IO, like
+  # `Zlib::GzipWriter`, finish the upload for you
+  #
+  # @return [Integer] the total number of bytes uploaded
+  def finish
+    return @bytes_persisted if @finished
+    @chunker.finish
+    @finished = true
+    @bytes_persisted
+  ensure
+    @transport.close
+  end
+  alias_method :close, :finish
+
+  private
+
+  # @param chunk[String]
+  # @param is_last[Boolean]
+  # @return [void]
+  def upload_chunk(chunk, is_last)
+    chunk_start = @bytes_persisted
+    chunk_end = chunk_start + chunk.bytesize
+    total = is_last ? chunk_end : "*"
+    attempts = 0
+
+    loop do
+      attempts += 1
+      if @bytes_persisted < chunk_start
+        raise GCSPut::UploadFailed, "GCS reports #{@bytes_persisted} bytes persisted but we already discarded everything before #{chunk_start}"
+      end
+      body = chunk.byteslice(@bytes_persisted - chunk_start, chunk.bytesize)
+
+      begin
+        response = put_bytes(body, from: @bytes_persisted, total: total)
+        failure = "HTTP #{response.status}"
+      rescue GCSPut::TransientError => e
+        response = nil
+        failure = e.message
+      end
+
+      case response&.status.to_i
+      when 200, 201
+        @bytes_persisted = chunk_end
+        return
+      when 308
+        @bytes_persisted = persisted_offset_from(response)
+        return if @bytes_persisted == chunk_end && !is_last
+        # GCS took fewer bytes than we sent, or wants the sized finalizing PUT - send the rest
+      when 500..599, 408, 429, 0
+        if sync_with_session_finds_it_finalized?
+          @bytes_persisted = chunk_end
+          return
+        end
+      else
+        raise GCSPut::UploadFailed.new("Chunk PUT responded with HTTP #{response.status}: #{response.body}", response: response)
+      end
+
+      if attempts >= @max_attempts
+        raise GCSPut::UploadFailed, "Gave up on chunk at offset #{chunk_start} after #{attempts} attempts, last failure: #{failure}"
+      end
+    end
+  end
+
+  # @param body[String]
+  # @param from[Integer]
+  # @param total[Integer, String] the total size, or "*" while still unknown
+  # @return [GCSPut::Transport::Response]
+  def put_bytes(body, from:, total:)
+    content_range = if body.empty?
+      "bytes */#{total}"
+    else
+      "bytes #{from}-#{from + body.bytesize - 1}/#{total}"
+    end
+    headers = {
+      "Content-Length" => body.bytesize.to_s,
+      "Content-Range" => content_range,
+      "Content-Type" => @content_type,
+      # Lets GCS reject a chunk mangled in transit instead of us discovering it at download time
+      "Content-MD5" => Digest::MD5.base64digest(body)
+    }
+    @transport.put(@session_uri, body, headers)
+  end
+
+  # Asks GCS how much of the upload it has and updates `bytes_persisted` accordingly, see
+  # https://cloud.google.com/storage/docs/performing-resumable-uploads#status-check
+  # A status check which fails itself just leaves us with what we already knew
+  #
+  # @return [Boolean] whether the session turned out to be finalized already
+  def sync_with_session_finds_it_finalized?
+    response = @transport.put(@session_uri, "", {"Content-Length" => "0", "Content-Range" => "bytes */*", "Content-Type" => @content_type})
+    case response.status
+    when 308
+      @bytes_persisted = persisted_offset_from(response)
+      false
+    when 200, 201
+      true
+    when 500..599, 408, 429
+      false
+    else
+      raise GCSPut::UploadFailed.new("Session status check responded with HTTP #{response.status}: #{response.body}", response: response)
+    end
+  rescue GCSPut::TransientError
+    false
+  end
+
+  # The `Range` header is "bytes=0-N" with N being the last persisted byte, and absent if nothing persisted yet
+  #
+  # @param response[GCSPut::Transport::Response]
+  # @return [Integer]
+  def persisted_offset_from(response)
+    range = response["Range"]
+    return 0 unless range
+    range[/\d+\z/].to_i + 1
+  end
+end

data/lib/gcs_put/signer.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+# Supplies the `issuer` and `signer` that `Google::Cloud::Storage::File#signed_url` needs
+# when there is no private key around to sign with
+module GCSPut::Signer
+  # When running on GCE, GKE, Cloud Run etc. under a service account there is no private key
+  # on the box to sign the URL with. The SDK then needs an `issuer` (the account email) and a
+  # `signer` lambda which asks the IAM credentials API to sign for us. For that to be allowed the
+  # service account must have `roles/iam.serviceAccountTokenCreator` on itself.
+  # Lifted from https://github.com/googleapis/google-cloud-ruby/issues/13307#issuecomment-1894546343
+  #
+  # @return [Hash] either `{issuer:, signer:}` or an empty hash when not on compute engine
+  def self.url_issuer_and_signer
+    require "google/cloud/env"
+    env = Google::Cloud.env
+    return {} unless env.compute_engine?
+
+    issuer = env.lookup_metadata("instance", "service-accounts/default/email")
+    {issuer: issuer, signer: iam_signer_for(issuer)}
+  end
+
+  # @param service_account_email[String]
+  # @return [Proc] a lambda which takes the string to sign and returns the signature
+  def self.iam_signer_for(service_account_email)
+    lambda do |string_to_sign|
+      require "google/apis/iamcredentials_v1"
+      require "googleauth"
+
+      iam_client = Google::Apis::IamcredentialsV1::IAMCredentialsService.new
+      iam_client.authorization = Google::Auth.get_application_default(["https://www.googleapis.com/auth/iam"])
+      request = Google::Apis::IamcredentialsV1::SignBlobRequest.new(payload: string_to_sign)
+      response = iam_client.sign_service_account_blob("projects/-/serviceAccounts/#{service_account_email}", request)
+      response.signed_blob
+    end
+  end
+end

data/lib/gcs_put/transport/faraday.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "faraday"
+
+# Uses a Faraday connection, so you can pick the adapter, timeouts and instrumentation yourself.
+# Check the bytes arrive intact with whatever adapter you choose - we send a `Content-MD5` with
+# every chunk precisely because httpx 1.4.0 used to mangle request bodies,
+# see https://gitlab.com/os85/httpx/-/issues/338
+class GCSPut::Transport::Faraday
+  # What Faraday raises when the connection, rather than the request, is at fault
+  TRANSIENT_ERRORS = [::Faraday::ConnectionFailed, ::Faraday::TimeoutError, ::Faraday::SSLError].freeze
+
+  # @param connection[Faraday::Connection, nil] a connection of your own, or `nil` to get a default one
+  def initialize(connection = nil)
+    @owns_connection = connection.nil?
+    @connection = connection || ::Faraday.new
+  end
+
+  # @param uri[URI::Generic]
+  # @param body[String]
+  # @param headers[Hash{String => String}]
+  # @return [GCSPut::Transport::Response]
+  # @raise [GCSPut::TransientError] on connection errors and timeouts
+  def put(uri, body, headers)
+    request(:put, uri, body, headers)
+  end
+
+  # @param uri[URI::Generic]
+  # @param body[String]
+  # @param headers[Hash{String => String}]
+  # @return [GCSPut::Transport::Response]
+  # @raise [GCSPut::TransientError] on connection errors and timeouts
+  def post(uri, body, headers)
+    request(:post, uri, body, headers)
+  end
+
+  # A connection you passed in is yours to close, we only close the one we made
+  #
+  # @return [void]
+  def close
+    @connection.close if @owns_connection && @connection.respond_to?(:close)
+  end
+
+  private
+
+  def request(verb, uri, body, headers)
+    response = @connection.run_request(verb, uri.to_s, body, headers)
+    GCSPut::Transport::Response.new(response.status, response.headers, response.body)
+  rescue *TRANSIENT_ERRORS => e
+    raise GCSPut::TransientError, "#{e.class}: #{e.message}"
+  rescue ::Faraday::Error => e
+    # The `raise_error` middleware turns 4xx and 5xx into exceptions, we want the response back
+    raise unless e.response
+    GCSPut::Transport::Response.new(e.response_status, e.response_headers, e.response_body)
+  end
+end