progeny 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e16d4fb0220a00c09eba23fc3b5f43cf72facb831a8d70f8afb2e4c48507b964
+  data.tar.gz: 6f852ef9c5fb2f0dafd6a153cc9045b5d37021f189e7da18d2ec137b7d9d3338
+SHA512:
+  metadata.gz: 2f808a8bcdae021ab23ef31ad18df2c3d39b359f9b3752bc5924e143a00757359e64ff0586c617b1e8908cec1bb5e0f9b1076834edce27595e1733f76ddcdbc6
+  data.tar.gz: 97bbb478f5c1ad1fabd55491784da44562d28d909be6f9c67586643f3e740fc1189e298cb2ea05a3ae46b0726a299f2d1cdba0307e19ffec31c83e3a41e7d049

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+## [Unreleased]
+
+## [0.1.0] - 2023-05-03
+
+- Initial release. Forked `posix-spawn` gem.
+
+### Changed
+
+- `Progeny::Command#new` and `Progeny::Command#build` only take actual `Hash`
+  objects as the `options` argument. We will not call `#to_hash` on the object
+  like `POSIX::Spawn::Child` did.
+

data/LICENSE.txt

@@ -0,0 +1,23 @@
+Copyright (c) 2011 by Ryan Tomayko <r@tomayko.com>
+                  and Aman Gupta <aman@tmm1.net>
+Copyright (c) 2023 by Luan Vieira <luanv@me.com>
+
+Permission  is  hereby granted, free of charge, to any person ob-
+taining a copy of  this  software  and  associated  documentation
+files  (the "Software"), to deal in the Software without restric-
+tion, including without limitation the rights to use, copy, modi-
+fy, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is  fur-
+nished to do so, subject to the following conditions:
+
+The  above  copyright  notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF  ANY  KIND,
+EXPRESS  OR  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE  AND  NONIN-
+FRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER  IN  AN
+ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN  THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,196 @@
+# progeny
+
+Spawn child processes with simple string based standard input/output/error
+stream handling in an easy-to-use interface.
+
+## Progeny::Command
+
+The `Progeny::Command` class includes logic for executing child processes and
+reading/writing from their standard input, output, and error streams. It's
+designed to take all input in a single string and provides all output as single
+strings and is therefore not well-suited to streaming large quantities of data
+in and out of commands. That said, it has some benefits:
+
+ - **Simple** - requires little code for simple stream input and capture.
+ - **Internally non-blocking** (using `select(2)`) - handles all pipe hang cases
+   due to exceeding `PIPE_BUF` limits on one or more streams.
+ - **Uses Ruby under the hood** - It leverages Ruby's `Open3#popen3` and `Process.spawn`
+   behind the scenes so it's widely supported and consistently getting performance updates.
+
+`Progeny::Command` takes the [standard `Process::spawn`
+arguments](https://ruby-doc.org/current/Process.html#method-c-spawn) when
+instantiated, and runs the process to completion after writing all input and
+reading all output:
+
+```ruby
+require 'progeny'
+child = Progeny::Command.new('git', '--help')
+```
+
+Retrieve process output written to stdout / stderr, or inspect the process's
+exit status:
+
+```ruby
+child.out
+# => "usage: git [--version] [--exec-path[=GIT_EXEC_PATH]]\n ..."
+child.err
+# => ""
+child.status
+# => #<Process::Status: pid=80718,exited(0)>
+```
+
+Use the `:input` option to write data on the new process's stdin immediately
+after spawning:
+
+```ruby
+child = Progeny::Command.new('bc', :input => '40 + 2')
+child.out
+# => "42\n"
+```
+
+Additional options can be used to specify the maximum output size (`:max`) and
+time of execution (`:timeout`) before the child process is aborted. See the
+`Progeny::Command` docs for more info.
+
+#### Reading Partial Results
+
+`Progeny::Command` spawns the process immediately when instantiated.
+As a result, if it is interrupted by an exception (either from reaching the
+maximum output size, the time limit, or another factor), it is not possible to
+access the `out` or `err` results because the constructor did not complete.
+
+If you want to get the `out` and `err` data that was available when the process
+was interrupted, use the `Progeny::Command` alternate form to create the child
+without immediately spawning the process.  Call `exec!` to run the command at a
+place where you can catch any exceptions:
+
+```ruby
+child = Progeny::Command.build('git', 'log', :max => 100)
+begin
+  child.exec!
+rescue Progeny::MaximumOutputExceeded
+  # limit was reached
+end
+child.out
+# => "commit fa54abe139fd045bf6dc1cc259c0f4c06a9285bb\n..."
+```
+
+Please note that when the `MaximumOutputExceeded` exception is raised, the
+actual combined `out` and `err` data may be a bit longer than the `:max`
+value due to internal buffering.
+
+## Why fork posix-spawn
+
+This gem is a fork of the
+[`posix-spawn`](https://github.com/rtomayko/posix-spawn) gem. Originally,
+`posix-spawn` was developed as a higher-performance alternative to Ruby's
+built-in `Process.spawn` method. It achieved this by utilizing the
+[`posix_spawn()`](https://man7.org/linux/man-pages/man3/posix_spawn.3.html)
+system call, which resulted in significant performance improvements, as
+demonstrated in the benchmarks below.
+
+However, the performance advantage of `posix-spawn` diminished with the release
+of Ruby 2.2. In this version, Ruby transitioned from using the `fork()` system
+call to [`vfork()`](https://man7.org/linux/man-pages/man2/vfork.2.html), which
+creates a new process without copying the parent process's page tables, leading
+to enhanced performance.
+
+The following benchmarks illustrate the performance comparison:
+
+- Performance comparison at the time of `posix-spawn` creation (Ruby used `fork` + `exec`):
+![image](https://user-images.githubusercontent.com/20481048/235776984-048669d9-8949-4bf8-90e8-632984ae0516.png)
+
+Source: [`posix-spawn` README](https://github.com/rtomayko/posix-spawn/blob/313f2abd4b9b5e737615178e0b353114481b9ab8/README.md#benchmarks)
+
+- Current performance comparison (Ruby's built-in functionality is now more performant):
+![image](https://user-images.githubusercontent.com/20481048/235777021-7e7afb19-0f73-41f1-bbc9-9bc952581c4d.png)
+
+Source: Generated with the script in [this gist](https://gist.github.com/luanzeba/a1ebe2497ed4e2fb6491fd1780a52440) on a Debian 11 (bullseye) x86_64 machine.
+
+For that reason, we decided to delete all of the custom spawn implementations
+in the original gem: `POSIX::Spawn#spawn`, `POSIX::Spawn#popen4`,
+`Kernel#system`, and <code>Kernel#\`</code>.
+
+However, we didn't want to completely remove our use of `posix-spawn` because
+we really enjoy the interface provided by `POSIX::Spawn::Child`. That's how
+`progeny` came to be. It maintains all of the functionality provided by
+`POSIX::Spawn::Child` under a new namespace: `Progeny::Command`.
+
+## How to migrate from `posix-spawn` to `progeny`
+
+1. Remove all usage of `POSIX::Spawn` as a Mixin.
+Progeny does not include a Mixin so if you're including `POSIX::Spawn` in any
+classes like so:
+```ruby
+require 'posix/spawn'
+
+class YourSpawnerClass
+  include POSIX::Spawn
+
+  # [...]
+end
+```
+
+You will need to remove the include statements and replace any use of `#spawn`
+with Ruby's native `Process.spawn` and  `#popen4` with Open3's `#popen3`.
+```diff
+- require 'posix/spawn'
++ require 'open3'
+
+class YourSpawnerClass
+- include POSIX::Spawn
+
+  def speak(message)
+-   pid = spawn('echo', message)
++   pid = Process.spawn('echo', message)
+    Process::waitpid(pid)
+  end
+
+  def calculate(expression)
+-   pid, in, out, err = popen4('bc')
++   in, out, err, wait_thr = Open3.popen3('bc')
++   pid = wait_thr[:pid] # if pid is needed
+    in.write(expression)
+    in.close
+    out.read
+  ensure
+    [in, out, err].each { |io| io.close if !io.closed? }
+-   Process::waitpid(pid)
+-   $?
++   wait_thr.value
+  end
+end
+```
+
+2. Find and replace in Gemfile
+```diff
+- gem 'posix-spawn'
++ gem 'progeny'
+```
+3. Find and replace `POSIX::Spawn::Child` with `Progeny::Command` and any `POSIX::Spawn` exceptions with `Progeny`
+```diff
+class GitDiff
+  def compare(from_sha, to_sha)
+-   child = POSIX::Spawn::Child.new("git", "diff #{from_sha}..#{to_sha}")
++   child = Progeny::Command.new("git", "diff #{from_sha}..#{to_sha}")
+    child.out
+  end
+
+  def compare_to_remote_head
+-   child = POSIX::Spawn::Child.build('git', 'diff origin/main HEAD')
++   child = Progeny::Command.build('git', 'diff origin/main HEAD')
+    begin
+      child.exec!
+-   rescue POSIX::Spawn::MaximumOutputExceeded
++   rescue Progeny::MaximumOutputExceeded
+      # limit was reached
+    end
+    child.out
+  end
+end
+```
+
+4. Confirm all is working as expected
+`bundle install` and make sure your tests are passing. If you encounter any
+issues feel free to open an issue or a PR.
+

data/lib/progeny/command.rb

@@ -0,0 +1,272 @@
+require "open3"
+
+module Progeny
+  # Progeny::Command includes logic for executing child processes and
+  # reading/writing from their standard input, output, and error streams. It's
+  # designed to take all input in a single string and provides all output
+  # (stderr and stdout) as single strings and is therefore not well-suited
+  # to streaming large quantities of data in and out of commands.
+  #
+  # Create and run a process to completion:
+  #
+  #   >> child = Progeny::Command.new('git', '--help')
+  #
+  # Retrieve stdout or stderr output:
+  #
+  #   >> child.out
+  #   => "usage: git [--version] [--exec-path[=GIT_EXEC_PATH]]\n ..."
+  #   >> child.err
+  #   => ""
+  #
+  # Check process exit status information:
+  #
+  #   >> child.status
+  #   => #<Process::Status: pid=80718,exited(0)>
+  #
+  # To write data on the new process's stdin immediately after spawning:
+  #
+  #   >> child = Progeny::Command.new('bc', :input => '40 + 2')
+  #   >> child.out
+  #   "42\n"
+  #
+  # To access output from the process even if an exception was raised:
+  #
+  #   >> child = Progeny::Command.build('git', 'log', :max => 1000)
+  #   >> begin
+  #   ?>   child.exec!
+  #   ?> rescue Progeny::MaximumOutputExceeded
+  #   ?>   # just so you know
+  #   ?> end
+  #   >> child.out
+  #   "... first 1000 characters of log output ..."
+
+  ##
+  # Exception raised when the total number of bytes output on the command's
+  # stderr and stdout streams exceeds the maximum output size (:max option).
+  class MaximumOutputExceeded < StandardError
+  end
+
+  # Exception raised when timeout is exceeded.
+  class TimeoutExceeded < StandardError
+  end
+
+  class Command
+    # Spawn a new process, write all input and read all output, and wait for
+    # the program to exit. Supports the standard spawn interface:
+    #   new([env], command, [argv1, ...], [options])
+    #
+    # The following options are supported in addition to the standard
+    # Process.spawn options:
+    #
+    #   :input   => str      Write str to the new process's standard input.
+    #   :timeout => int      Maximum number of seconds to allow the process
+    #                        to execute before aborting with a TimeoutExceeded
+    #                        exception.
+    #   :max     => total    Maximum number of bytes of output to allow the
+    #                        process to generate before aborting with a
+    #                        MaximumOutputExceeded exception.
+    #   :pgroup_kill => bool Boolean specifying whether to kill the process
+    #                        group (true) or individual process (false, default).
+    #                        Setting this option true implies :pgroup => true.
+    #
+    # Returns a new Command instance whose underlying process has already
+    # executed to completion. The out, err, and status attributes are
+    # immediately available.
+    def initialize(*args)
+      if args.last.is_a?(Hash)
+        options = args.pop.dup
+      else
+        options = {}
+      end
+
+      if args.first.is_a?(Hash)
+        @env = args.shift
+      else
+        @env = {}
+      end
+      @env.merge!(options.delete(:env)) if options.key?(:env)
+      @argv = args
+      @options = options.dup
+      @input = @options.delete(:input)
+      @timeout = @options.delete(:timeout)
+      @max = @options.delete(:max)
+      if @options.delete(:pgroup_kill)
+        @pgroup_kill = true
+        @options[:pgroup] = true
+      end
+      @options.delete(:chdir) if @options[:chdir].nil?
+      exec! if !@options.delete(:noexec)
+    end
+
+    # Set up a new process to spawn, but do not actually spawn it.
+    #
+    # Invoke this just like the normal constructor to set up a process
+    # to be run.  Call `exec!` to actually run the child process, send
+    # the input, read the output, and wait for completion.  Use this
+    # alternative way of constructing a Progency::Command if you want
+    # to read any partial output from the child process even after an
+    # exception.
+    #
+    #   child = Progency::Command.build(... arguments ...)
+    #   child.exec!
+    #
+    # The arguments are the same as the regular constructor.
+    #
+    # Returns a new Command instance but does not run the underlying process.
+    def self.build(*args)
+      options =
+        if args.last.is_a?(Hash)
+          args.pop.dup
+        else
+          {}
+        end
+      new(*(args + [{ :noexec => true }.merge(options)]))
+    end
+
+    # All data written to the child process's stdout stream as a String.
+    attr_reader :out
+
+    # All data written to the child process's stderr stream as a String.
+    attr_reader :err
+
+    # A Process::Status object with information on how the child exited.
+    attr_reader :status
+
+    # Total command execution time (wall-clock time)
+    attr_reader :runtime
+
+    # The pid of the spawned child process. This is unlikely to be a valid
+    # current pid since Command#exec! doesn't return until the process finishes
+    # and is reaped.
+    attr_reader :pid
+
+    # Determine if the process did exit with a zero exit status.
+    def success?
+      @status && @status.success?
+    end
+
+    # Execute command, write input, and read output. This is called
+    # immediately when a new instance of this object is created, or
+    # can be called explicitly when creating the Command via `build`.
+    def exec!
+      stdin, stdout, stderr, wait_thread = Open3.popen3(@env, *(@argv + [@options]))
+      @pid = wait_thread[:pid]
+
+      # async read from all streams into buffers
+      read_and_write(@input, stdin, stdout, stderr, @timeout, @max)
+
+      # wait_thr.value waits for the termination of the process and returns exit status
+      @status = wait_thread.value
+    rescue Object
+      [stdin, stdout, stderr].each { |fd| fd.close rescue nil }
+      if @status.nil?
+        if !@pgroup_kill
+          ::Process.kill('TERM', pid) rescue nil
+        else
+          ::Process.kill('-TERM', pid) rescue nil
+        end
+        @status = wait_thread.value rescue nil
+      end
+      raise
+    ensure
+      # let's be absolutely certain these are closed
+      [stdin, stdout, stderr].each { |fd| fd.close rescue nil }
+    end
+
+    private
+    # Maximum buffer size for reading
+    BUFSIZE = (32 * 1024)
+
+    # Start a select loop writing any input on the child's stdin and reading
+    # any output from the child's stdout or stderr.
+    #
+    # input   - String input to write on stdin. May be nil.
+    # stdin   - The write side IO object for the child's stdin stream.
+    # stdout  - The read side IO object for the child's stdout stream.
+    # stderr  - The read side IO object for the child's stderr stream.
+    # timeout - An optional Numeric specifying the total number of seconds
+    #           the read/write operations should occur for.
+    #
+    # Returns an [out, err] tuple where both elements are strings with all
+    #   data written to the stdout and stderr streams, respectively.
+    # Raises TimeoutExceeded when all data has not been read / written within
+    #   the duration specified in the timeout argument.
+    # Raises MaximumOutputExceeded when the total number of bytes output
+    #   exceeds the amount specified by the max argument.
+    def read_and_write(input, stdin, stdout, stderr, timeout=nil, max=nil)
+      max = nil if max && max <= 0
+      @out, @err = '', ''
+
+      # force all string and IO encodings to BINARY under 1.9 for now
+      if @out.respond_to?(:force_encoding) and stdin.respond_to?(:set_encoding)
+        [stdin, stdout, stderr].each do |fd|
+          fd.set_encoding('BINARY', 'BINARY')
+        end
+        @out.force_encoding('BINARY')
+        @err.force_encoding('BINARY')
+        input = input.dup.force_encoding('BINARY') if input
+      end
+
+      timeout = nil if timeout && timeout <= 0.0
+      @runtime = 0.0
+      start = Time.now
+
+      readers = [stdout, stderr]
+      writers =
+        if input
+          [stdin]
+        else
+          stdin.close
+          []
+        end
+      slice_method = input.respond_to?(:byteslice) ? :byteslice : :slice
+      t = timeout
+
+      while readers.any? || writers.any?
+        ready = IO.select(readers, writers, readers + writers, t)
+        raise TimeoutExceeded if ready.nil?
+
+        # write to stdin stream
+        ready[1].each do |fd|
+          begin
+            boom = nil
+            size = fd.write_nonblock(input)
+            input = input.send(slice_method, size..-1)
+          rescue Errno::EPIPE => boom
+          rescue Errno::EAGAIN, Errno::EINTR
+          end
+          if boom || input.bytesize == 0
+            stdin.close
+            writers.delete(stdin)
+          end
+        end
+
+        # read from stdout and stderr streams
+        ready[0].each do |fd|
+          buf = (fd == stdout) ? @out : @err
+          begin
+            buf << fd.readpartial(BUFSIZE)
+          rescue Errno::EAGAIN, Errno::EINTR
+          rescue EOFError
+            readers.delete(fd)
+            fd.close
+          end
+        end
+
+        # keep tabs on the total amount of time we've spent here
+        @runtime = Time.now - start
+        if timeout
+          t = timeout - @runtime
+          raise TimeoutExceeded if t < 0.0
+        end
+
+        # maybe we've hit our max output
+        if max && ready[0].any? && (@out.size + @err.size) > max
+          raise MaximumOutputExceeded
+        end
+      end
+
+      [@out, @err]
+    end
+  end
+end

data/lib/progeny/version.rb

@@ -0,0 +1,3 @@
+module Progeny
+  VERSION = '0.1.0'
+end

data/lib/progeny.rb

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+require_relative "progeny/version"
+require_relative "progeny/command"

metadata

@@ -0,0 +1,81 @@
+--- !ruby/object:Gem::Specification
+name: progeny
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Luan Vieira
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2023-05-03 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+description: Spawn child processes without managing IO streams, zombie processes and
+  other details.
+email:
+- luanv@me.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/progeny.rb
+- lib/progeny/command.rb
+- lib/progeny/version.rb
+homepage: https://github.com/luanzeba/progeny
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/luanzeba/progeny
+  source_code_uri: https://github.com/luanzeba/progeny
+  changelog_uri: https://github.com/luanzeba/progeny/blob/main/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.10
+signing_key:
+specification_version: 4
+summary: A popen3 wrapper with a nice interface and extra options.
+test_files: []