oximg 0.17.0 → 0.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9da49f93c106126b40a39d491a9794c044b4d28f1f84b752bf4be5095664cbf3
-  data.tar.gz: f218f4d6179d4b71bdb8c2bfbcfb4232e3a2bdd517355f2c8ccd8a91bddd0d4b
+  metadata.gz: 16e1f0079f0e556b9dd2bf4200f3e5667bb6904d1ab6597a7189dee077c731bc
+  data.tar.gz: cf19c0196c9636544db5c4318170b262a6ba9d57529776ef7fd0b13faa77e521
 SHA512:
-  metadata.gz: 89f0dbf9d871e86a0d954071370275469c091122b1901c5240f0d829899cc4a5fb24121d8643cced7a51a7d490a2b4728ddd38aec7fa4f68b473f6cf5fc282e4
-  data.tar.gz: 410ecc05ad827faad5c46cdd80efbfcc9cc9c6c168bb9028f98cd5cca8975a157b1307c16857d3c0347d6368e9d674b2b06a4ff144bfef6e614a4183caa2ba0f
+  metadata.gz: c0cf987fde6720add95242c3c5e12a21f395b27c2c32cc7ab8e52c9203d02162a3823315a68d4bb69719c13d831573256b87d7acf2e140fe6d97c30f18d46e30
+  data.tar.gz: a8d0316e4b3770e02c1f4347d6f9e514ba24a85b02903ea43b12a3ef01cbd875c851bbe2fc2ad634493ae601be810cf918d40407e3065d591da8562c57394c88

data/README.md

@@ -18,6 +18,7 @@ gem "oximg"
 ```ruby
 Oximg.resize("in.jpg", "out.jpg", width: 750)
 Oximg.resize("in.jpg", "out.webp", width: 750, height: 500, quality: 80)
+Oximg.strip("upload.jpg", "clean.jpg")
 Oximg.probe("in.jpg")
 #=> {content_type: "image/jpeg", format: :jpeg, width: 4000, height: 3000}
 ```
@@ -55,10 +56,76 @@ on failure, carrying the binary's own message and exit status. Paths are
 expanded before they reach the CLI, so an uploaded file called `-q.jpg`
 cannot be read as a flag.
 
+## Stripping metadata
+
+```ruby
+Oximg.strip(source, destination, icc: nil)
+```
+
+Removes EXIF (GPS, camera, timestamps), XMP, IPTC and comments from a
+JPEG, PNG or WebP **without re-encoding**. Decoded pixels are identical,
+and it costs a fraction of a `0 x 0` resize in time and memory. This
+replaces `saver(strip: true)` (libvips) or `-strip` (ImageMagick). A
+rotated photo keeps an EXIF block holding only its orientation, so it
+still displays upright. `icc: :strip` also drops the colour profile.
+The default is `:keep`, or `OXIMG_ICC`. Dropping a CMYK JPEG's profile
+raises `Oximg::ProcessingError`, as do GIF and AVIF sources.
+
 Everything the server tunes through `OXIMG_*` environment variables
 applies here too, and is validated the same fail-closed way: a typo'd
 knob is a startup error, never a silent default.
 
+## Probing
+
+```ruby
+Oximg.probe("in.jpg")                          # a path
+Oximg.probe(Pathname("in.jpg"))                # also a path
+Oximg.probe(upload.tempfile)                   # anything with #path
+Oximg.probe(File.open("in.jpg", "rb"))         # an open File
+Oximg.probe(StringIO.new(bytes))               # an IO: bytes in memory
+#=> {content_type: "image/jpeg", format: :jpeg, width: 4000, height: 3000}
+```
+
+`probe` reads the header only and decodes no pixels. The dimensions are
+the stored ones, before any EXIF rotation. A String is always a path,
+so bytes already in memory go in as a `StringIO`. An open `File` is
+read from its start, its position untouched, and it is the file you
+hold that is read, even if its name has since been replaced, renamed,
+or was relative to a directory the process has left. A closed or
+write-only `File`, which cannot be read through, is read by its name.
+Other IO — a `StringIO`, a pipe, a socket — is read from its current
+position through `#read(n)` and left where the read stopped.
+
+The gem does that reading itself, in Ruby, between calls into the
+library, so `#read` behaves as it would anywhere: whatever it raises
+comes out of `probe` unchanged, and a `Timeout.timeout` or a
+`Thread#kill` lands in it as usual.
+
+With the probe library, `probe` runs in-process: oximg's own probe,
+compiled into a small library (`ext/`) the gem calls through
+[`ffi`](https://github.com/ffi/ffi). It reads only as much of the source
+as the header takes, typically the first 64 KiB, and costs microseconds
+where a spawn costs milliseconds. `resize` is not affected and still
+runs in a child process, which keeps a full decode's memory and crashes
+out of your process.
+
+The platform gems bundle the library beside the executable (x86_64 and
+aarch64 Linux, glibc and musl, and arm64 macOS), so they also depend
+on `ffi`; beyond that it needs only libc and libgcc_s, which Ruby
+itself needs, and not libstdc++.
+`OXIMG_PROBE_LIB` points at another one; like `OXIMG_BIN`, one that
+does not load raises rather than falling back. To build it from this
+checkout, run `rake compile` here: it leaves the library under the
+repo's `target/oximg-probe/release/`.
+
+Without the library — the plain-Ruby gem, or a bundled one that cannot
+load — `probe` spawns `oximg probe` instead, copying an IO to a
+tempfile first. The answer is the same; only the cost differs.
+
+```ruby
+Oximg.in_process_probe?  #=> true on a platform gem
+```
+
 ## Where the binary comes from
 
 `Oximg.executable` resolves, first hit wins:
@@ -71,14 +138,14 @@ knob is a startup error, never a silent default.
 
 ```ruby
 Oximg.available?  #=> true
-Oximg.executable  #=> "/…/gems/oximg-0.16.0-arm64-darwin/exe/oximg"
-Oximg.version     #=> "0.16.0"   # the binary's version, not the gem's
+Oximg.executable  #=> "/…/gems/oximg-0.18.0-arm64-darwin/exe/oximg"
+Oximg.version     #=> "0.18.0"   # the binary's version, not the gem's
 ```
 
-The gem shells out to that binary — one process per image, which is
+`resize` shells out to that binary — one process per image, which is
 noise next to the encode itself and invisible inside the background job
-where variants are generated anyway. An in-process native extension is a
-later option; it would not change the API above.
+where variants are generated anyway. Only `probe`, where the spawn was
+the whole cost, runs in-process (see [Probing](#probing)).
 
 ## Rails
 

data/lib/oximg/native.rb

@@ -0,0 +1,179 @@
+# frozen_string_literal: true
+
+require "rbconfig"
+
+module Oximg
+  # The in-process header read: oximg's own probe, compiled as a small
+  # C library (rubygem/oximg/ext) and called through the +ffi+ gem, so
+  # Oximg.probe costs microseconds instead of a process spawn.
+  #
+  # Only the header read runs in-process. It reads a bounded prefix of
+  # the source and decodes no pixels; resize stays a child process, for
+  # the memory and crash isolation a full decode wants.
+  #
+  # Resolution order, first hit wins:
+  #
+  #   1. +OXIMG_PROBE_LIB+ — an explicit path always wins, and a wrong
+  #      one raises rather than falling back, as +OXIMG_BIN+ does.
+  #   2. the library bundled in this gem, beside this file — platform
+  #      gems ship one. If it cannot load (no +ffi+ gem, a missing
+  #      system library), probe falls back to spawning the executable:
+  #      slower, same answer.
+  #
+  # The plain-Ruby gem ships no library and always spawns.
+  module Native
+    LIBRARY = "liboximg_probe.#{RbConfig::CONFIG["SOEXT"]}"
+
+    # Populated by the platform gems at package time; absent in the
+    # plain-Ruby gem.
+    BUNDLED = File.expand_path("native/#{LIBRARY}", __dir__)
+
+    # The library's oximg_probe_abi_version this code was written for.
+    ABI_VERSION = 1
+
+    # Statuses (ext/src/lib.rs).
+    OK = 0
+    READ = 3
+
+    # The byte count READ gives for "to the end".
+    READ_ALL = 2**64 - 1
+
+    # Room for an error message; the library cuts a longer one at a
+    # character boundary.
+    MESSAGE_BYTES = 1024
+
+    class << self
+      # True when the library is loaded and probe runs in-process.
+      def available?
+        !binding_module.nil?
+      end
+
+      # Header of the file at +path+ (already expanded).
+      def probe_path(path)
+        # Copied off the Ruby heap: the call runs without the GVL, while
+        # other threads may run GC.
+        bytes = FFI::MemoryPointer.from_string(path)
+        call { |dims, msg| binding_module.oximg_probe_path(bytes, path.bytesize, dims, msg, MESSAGE_BYTES) }
+      end
+
+      # Header of the open, readable regular +file+, read through its
+      # descriptor from the start with positioned reads: what it holds,
+      # whatever its name leads to by now, its position untouched.
+      def probe_file(file)
+        call { |dims, msg| binding_module.oximg_probe_fd(file.fileno, dims, msg, MESSAGE_BYTES) }
+      end
+
+      # Header of +io+, read from its current position through
+      # IO#read(n) — a pipe, a socket, a StringIO. Only what the header
+      # needs is read, and the IO is left where the read stopped.
+      #
+      # The library decides how much to read and the reading happens
+      # here, between calls, so the IO's code never runs under the
+      # library's frames: whatever it raises, a Timeout or a Thread#kill
+      # included, leaves the way it would leave any Ruby method.
+      def probe_io(io)
+        prefix = String.new(encoding: Encoding::BINARY)
+        eof = false
+        loop do
+          want = nil
+          result = call do |dims, msg|
+            status = binding_module.oximg_probe_step(prefix, prefix.bytesize, eof ? 1 : 0, dims, msg, MESSAGE_BYTES)
+            want = dims.read_uint64 if status == READ
+            status
+          end
+          return result unless want
+
+          eof = read_to(io, prefix, want)
+        end
+      end
+
+      # Forgets the loaded library. For tests, and for a process that
+      # changes OXIMG_PROBE_LIB after boot.
+      def reset!
+        remove_instance_variable(:@binding_module) if instance_variable_defined?(:@binding_module)
+      end
+
+      private
+
+      # Reads +io+ onto +prefix+ until it holds +want+ bytes; true if the
+      # source ended first. READ_ALL asks for everything, which is read
+      # a MiB at a time: a length that large would be allocated up
+      # front, and an IO need only answer #read(n).
+      def read_to(io, prefix, want)
+        while want == READ_ALL || prefix.bytesize < want
+          chunk = io.read(want == READ_ALL ? 1 << 20 : want - prefix.bytesize)
+          return true if chunk.nil? || chunk.empty?
+
+          prefix << chunk.b
+        end
+        false
+      end
+
+      def call
+        dims = FFI::MemoryPointer.new(:uint64, 2)
+        msg = FFI::MemoryPointer.new(:char, MESSAGE_BYTES)
+        status = yield(dims, msg)
+        return if status == READ
+
+        text = msg.read_string.force_encoding(Encoding::UTF_8)
+        raise ProcessingError.new(text, 1) unless status == OK
+
+        width, height = dims.read_array_of_uint64(2)
+        {content_type: text, format: CONTENT_TYPE_FORMATS[text], width: width, height: height}
+      end
+
+      # nil when there is no library to use. Memoized either way, so a
+      # gem without one does not look again on every probe.
+      def binding_module
+        return @binding_module if instance_variable_defined?(:@binding_module)
+
+        @binding_module = load_library
+      end
+
+      def load_library
+        configured = ENV["OXIMG_PROBE_LIB"]
+        if configured.nil? || configured.strip.empty?
+          return nil unless File.file?(BUNDLED)
+
+          begin
+            open_library(BUNDLED)
+          rescue LoadError
+            nil
+          end
+        else
+          unless File.file?(configured)
+            raise LoadError, "OXIMG_PROBE_LIB is set to #{configured.inspect}, which is not a file"
+          end
+
+          # Absolute, so the file loaded is the one just checked. Where
+          # dlopen looks for a bare name depends on the platform (glibc
+          # never looks in the working directory; dyld does, after other
+          # places). absolute_path, unlike expand_path, leaves "~" alone,
+          # as File.file? did.
+          open_library(File.absolute_path(configured))
+        end
+      end
+
+      def open_library(path)
+        require "ffi"
+        mod = Module.new do
+          extend FFI::Library
+          ffi_lib path
+          attach_function :oximg_probe_abi_version, [], :uint32
+          # A path or descriptor read blocks on the filesystem, so it runs
+          # without the GVL. A step only parses bytes already in memory and
+          # keeps it, which also keeps the prefix String where it is.
+          attach_function :oximg_probe_path, %i[pointer size_t pointer pointer size_t], :int, blocking: true
+          attach_function :oximg_probe_fd, %i[int pointer pointer size_t], :int, blocking: true
+          attach_function :oximg_probe_step, %i[buffer_in size_t uint8 pointer pointer size_t], :int
+        end
+        abi = mod.oximg_probe_abi_version
+        unless abi == ABI_VERSION
+          raise LoadError, "#{path} speaks probe ABI #{abi}; this gem speaks #{ABI_VERSION}"
+        end
+
+        mod
+      end
+    end
+  end
+end

data/lib/oximg/version.rb

@@ -7,5 +7,5 @@ module Oximg
   # names a binary it does not carry. `Oximg.version` reports what the
   # resolved executable actually is, which can differ when it comes
   # from PATH.
-  VERSION = "0.17.0"
+  VERSION = "0.18.0"
 end

data/lib/oximg.rb

@@ -1,7 +1,10 @@
 # frozen_string_literal: true
 
+require "fcntl"
+require "tempfile"
 require_relative "oximg/version"
 require_relative "oximg/binary"
+require_relative "oximg/native"
 
 # Ruby bindings for oximg (https://github.com/oximg/oximg):
 # high-performance image compression and resizing, in-process work for a
@@ -9,6 +12,7 @@ require_relative "oximg/binary"
 #
 #   Oximg.resize("in.jpg", "out.jpg", width: 750)
 #   Oximg.resize("in.jpg", "out.webp", width: 750, height: 500, quality: 80)
+#   Oximg.strip("upload.jpg", "clean.jpg")  # drop EXIF/GPS, XMP, IPTC; no re-encode
 #   Oximg.probe("in.jpg")  #=> {content_type: "image/jpeg", format: :jpeg, ...}
 #
 # Rails/ActiveStorage integration, and URL building for a remote oximg
@@ -41,6 +45,9 @@ module Oximg
   # mozjpeg's fast/small trade differently.
   PRESETS = %i[jpegli fast small].freeze
 
+  # What `strip` may do with an embedded colour profile.
+  ICC_MODES = %i[keep strip].freeze
+
   CONTENT_TYPE_FORMATS = {
     "image/jpeg" => :jpeg,
     "image/png" => :png,
@@ -106,14 +113,70 @@ module Oximg
       output
     end
 
+    # Removes metadata — EXIF (GPS, camera, timestamps), XMP, IPTC,
+    # comments — from a JPEG, PNG or WebP without re-encoding, and
+    # writes the result to +destination+. Decoded pixels are identical;
+    # a rotated photo keeps only its EXIF orientation, so it still
+    # displays upright. +icc:+ is :keep or :strip (default: the
+    # binary's OXIMG_ICC, else keep); dropping a CMYK JPEG's profile is
+    # refused with a ProcessingError. Returns the destination path.
+    #
+    # This is what <tt>saver(strip: true)</tt> / <tt>-strip</tt> did
+    # under libvips or ImageMagick, minus the re-encode.
+    def strip(source, destination, icc: nil)
+      output = expand(destination, "destination")
+      Binary.run(*strip_argv(source, output, icc: icc))
+      output
+    end
+
+    # The argv `strip` would run, minus the executable; public for the
+    # same reason as resize_argv.
+    def strip_argv(source, destination, icc: nil)
+      argv = ["strip", expand(source, "source"), expand(destination, "destination")]
+      argv.push("--icc", token(icc, ICC_MODES, "icc")) unless icc.nil?
+      argv
+    end
+
     # Header-only inspection — no pixels are decoded. Returns
     # <tt>{content_type:, format:, width:, height:}</tt>, where the
     # dimensions are the stored ones (an EXIF rotation is not applied).
+    #
+    # +source+ is a path — a String, a Pathname, or anything with #path
+    # such as a Tempfile — an open File, or a stream read from its
+    # current position through #read(n): a StringIO, a pipe, a socket.
+    # A String is always a path, so bytes already in memory go in as
+    # <tt>StringIO.new(bytes)</tt>. An open File on a regular file is
+    # read from its start, its position untouched, and it is the file
+    # it holds that is read, whatever its name leads to by now. A closed
+    # or write-only File, which cannot be read through, is read by its
+    # name.
+    #
+    # With the probe library this runs in-process and reads only as much
+    # of the source as the header takes (see in_process_probe?). Without it,
+    # it spawns <tt>oximg probe</tt>, which reads the whole source, and an
+    # IO is copied to a tempfile first. The answer is the same.
     def probe(source)
-      out, = Binary.run("probe", expand(source, "source"))
+      return probe_open_file(source) if open_regular_file?(source)
+
+      if stream?(source)
+        return Native.available? ? Native.probe_io(source) : probe_spawned_io(source)
+      end
+
+      path = expand(source, "source")
+      return Native.probe_path(path) if Native.available?
+
+      out, = Binary.run("probe", path)
       parse_probe(out)
     end
 
+    # True when probe runs in-process, through the probe library (see
+    # Native); false when it spawns the executable. Either way
+    # probe answers the same, and only its cost differs — this is for a
+    # boot-time log, or a test that must know which path it took.
+    def in_process_probe?
+      Native.available?
+    end
+
     # The argv `resize` would run, minus the executable. Public because
     # it is the honest way to see what a call turns into — and the way
     # to test that without a binary present.
@@ -133,6 +196,80 @@ module Oximg
 
     private
 
+    def open_regular_file?(source)
+      source.is_a?(File) && !source.closed? && source.stat.file?
+    end
+
+    # An open File is read through its descriptor, not by the name it
+    # was opened under: the name may lead to another file by now
+    # (replaced, renamed, or relative to a working directory the
+    # process has since left). Positioned reads leave its position
+    # alone. A write-only File cannot be read that way, and is read by
+    # its name.
+    def probe_open_file(file)
+      readable = (file.fcntl(Fcntl::F_GETFL) & Fcntl::O_ACCMODE) != Fcntl::O_WRONLY
+      return Native.probe_file(file) if readable && Native.available?
+
+      # A File made from a descriptor (File.for_fd) has no name.
+      name = file.path.to_s.empty? ? nil : File.expand_path(file.path)
+      if name && (!readable || File.identical?(file, name))
+        return Native.probe_path(name) if Native.available?
+
+        out, = Binary.run("probe", name)
+        return parse_probe(out)
+      end
+      raise IOError, "a write-only File with no name cannot be probed" unless readable
+
+      # The executable reads names only, and this one has moved on: it
+      # gets a copy, made with positioned reads.
+      Tempfile.create("oximg-probe") do |copy|
+        copy.binmode
+        offset = 0
+        loop do
+          chunk = file.pread(1 << 20, offset)
+          copy.write(chunk)
+          offset += chunk.bytesize
+        rescue EOFError
+          break
+        end
+        copy.flush
+        out, = Binary.run("probe", copy.path)
+        parse_probe(out)
+      end
+    end
+
+    # Whether probe reads +source+ through #read rather than opening it
+    # by name. Every IO answers #path since Ruby 3.2 — nil for a pipe,
+    # "" for a socket, "<STDIN>" for STDIN — so for an IO that proves
+    # nothing: an IO is a stream, except a closed File, which has
+    # nothing left to read but its name (an open one on a regular file
+    # never gets here). Anything else keeps the duck typing: #to_path
+    # or #path (a Pathname, a Tempfile, an upload) is a path.
+    def stream?(source)
+      case source
+      when String then false
+      when File then !source.closed?
+      when IO then true
+      else !source.respond_to?(:to_path) && !source.respond_to?(:path) && source.respond_to?(:read)
+      end
+    end
+
+    # The executable reads paths only, so without the library an IO
+    # goes through a tempfile. It is read through #read(n) alone, as the
+    # library reads it, so probe takes the same IO either way
+    # (IO.copy_stream would also pass #read a buffer).
+    def probe_spawned_io(io)
+      Tempfile.create("oximg-probe") do |file|
+        file.binmode
+        while (chunk = io.read(1 << 20)) && !chunk.empty?
+          file.write(chunk)
+        end
+        file.flush
+        out, = Binary.run("probe", file.path)
+        parse_probe(out)
+      end
+    end
+
     # The hash `probe` builds from the line the CLI printed. Separate
     # from `probe` so the parser can be tested against every line the
     # CLI can print — the still form, each loop spelling — without a
@@ -153,7 +290,11 @@ module Oximg
     # would otherwise reach the CLI's argument parser as a flag, and
     # the caller rarely controls what an uploaded file is called.
     def expand(path, name)
-      path = path.path if path.respond_to?(:path)
+      if path.respond_to?(:path)
+        path = path.path
+      elsif path.respond_to?(:to_path)
+        path = path.to_path
+      end
       path = path.to_s
       raise ArgumentError, "#{name} path is empty" if path.strip.empty?
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oximg
 version: !ruby/object:Gem::Version
-  version: 0.17.0
+  version: 0.18.0
 platform: ruby
 authors:
 - oximg contributors
@@ -20,6 +20,7 @@ files:
 - README.md
 - lib/oximg.rb
 - lib/oximg/binary.rb
+- lib/oximg/native.rb
 - lib/oximg/version.rb
 homepage: https://github.com/oximg/oximg
 licenses: