uringmachine 0.23.1 → 0.24.0

data/grant-2025/interim-report.md

@@ -0,0 +1,130 @@
+# Interim Report for Ruby Association Grant Program 2025
+
+## Project Summary 
+
+Io_uring is a relatively new Linux API, permitting the invocation of Linux
+system calls asynchronously. UringMachine is a gem that brings low-level access
+to the io_uring interface to Ruby programs, and permits not only asynchronous
+I/O on files and sockets, but also timeouts, futex wait/wake, statx etc, with
+support for fiber-based concurrency. This project will work to enhance
+UringMachine to include a fiber scheduler implementation for usage with the
+standard Ruby I/O classes, to have builtin support for SSL, to support more
+io_uring ops such as writev, splice, fsync, mkdir, fadvise, etc.
+
+## Progress Report
+
+As of the present date, I have worked on the following:
+
+### Improvements to the Ruby `Fiber::Scheduler` interface
+
+- [PR](https://github.com/ruby/ruby/pull/15213) to expose
+  `rb_process_status_new` internal Ruby C API
+  (https://bugs.ruby-lang.org/issues/21704). This is needed in order to allow
+  FiberScheduler implementations to instantiate `Process::Status` objects in the
+  `#process_wait` hook. This PR is still pending a decision by the Ruby core team.
+
+- [PR](https://github.com/ruby/ruby/pull/15385) to cleanup FiberScheduler and
+  fiber state in a forked process (https://bugs.ruby-lang.org/issues/21717).
+  This was merged into Ruby 4.0.
+
+- [PR](https://github.com/ruby/ruby/pull/15609) to invoke FiberScheduler
+  `io_write` hook on IO flush (https://bugs.ruby-lang.org/issues/21789). This
+  was merged into Ruby 4.0.
+
+- Found an issue while implementing the `#io_pwrite` hook, which resulted in a
+  [PR](https://github.com/ruby/ruby/pull/15428) submitted by Samuel Williams,
+  and merged into Ruby 4.0.
+
+- Worked with Samuel Williams on how to implement the `#io_close` hook, which
+  resulted in a [PR](https://github.com/ruby/ruby/pull/15434) submitted by
+  Samuel and merged into Ruby 4.0.
+
+- [PR](https://github.com/ruby/ruby/pull/15865) to add socket I/O hooks to the
+  FiberScheduler interface (https://bugs.ruby-lang.org/issues/21837). This PR is
+  currently in draft phase.
+
+### UringMachine `Fiber::Scheduler` Implementation
+
+- I developed a [full
+  implementation](https://github.com/digital-fabric/uringmachine/blob/main/lib/uringmachine/fiber_scheduler.rb)
+  of the `Fiber::Scheduler` interface using UringMachine, with methods for *all*
+  hooks:
+
+  - `#scheduler_close`
+  - `#fiber`, `#yield`
+  - `#blocking_operation_wait`, `#block`, `#unblock`, `#fiber_interrupt`
+  - `#kernel_sleep`, `#timeout_after`
+  - `#io_read`, `#io_write`, `#io_pread`, `#io_pwrite`, `#io_close`
+  - `#io_wait`, `#io_select`
+  - `#process_wait` (relies on the `rb_process_status_new` PR)
+  - `#address_resolve`
+
+- Wrote [extensive
+  tests](https://github.com/digital-fabric/uringmachine/blob/main/test/test_fiber_scheduler.rb)
+  for the UringMachine fiber scheduler.
+
+### Improvements to UringMachine
+
+- Improved various internal aspects of the C-extension: performance and
+  correctness of mutex and queue implementations.
+
+- Added support for accepting instances of `IO::Buffer` as buffer for the
+  various I/O operations, in order to facilitate the `Fiber::Scheduler`
+  implementation.
+
+- Added various methods for working with processes:
+
+  - `UringMachine#waitid`
+  - `UringMachine.pidfd_open`
+  - `UringMachine.pidfd_send_signal`
+
+- Added detailed internal metrics.
+
+- Added support for vectorized write/send using io_uring: `UringMachine#writev`
+  and `UringMachine#sendv`
+
+- Added support for `SQPOLL` mode - this io_uring mode lets us avoid entering
+  the kernel when submitting I/O operations as the kernel is busy polling the SQ
+  ring.
+
+- Added support for sidecar mode: an auxiliary thread is used to enter the
+  kernel and wait for CQE's (I/O operation completion entries), letting the Ruby
+  thread avoid entering the kernel in order to wait for CQEs.
+
+### Benchmarking
+
+- I did extensive benchmarking comparing different solutions for performing
+  concurrent I/O in Ruby:
+
+  - Using normal Ruby threads
+  - Using Samuel's [Async](https://github.com/socketry/async/) gem which
+    implements a `Fiber::Scheduler`
+  - Using the UringMachine `Fiber::Scheduler`
+  - Using the UringMachine low-level API
+
+- The benchmarks simulate different kinds of workloads:
+
+  - Writing and reading from pipes
+  - Writing and reading from sockets
+  - Doing CPU-bound work synchronized by mutex
+  - Doing I/O-bound work synchronized by mutex
+  - Pushing and pulling items from queues
+  - Running queries on a PostgreSQL database
+
+- The results are here: https://github.com/digital-fabric/uringmachine/blob/main/benchmark/README.md
+
+### Pending Work
+
+Before the end of the grant work period I intend to do the following:
+
+- I already started work on SSL integration. I intend to contribute changes to
+  `ruby/openssl` to add support for custom BIO that will use the underlying
+  socket for performing I/O (currently the Ruby openssl implementation
+  completely bypasses the Ruby I/O layer in order to send/recv to sockets). This
+  will allow integration with the `Fiber::Scheduler` interface.
+
+- Add support for automatic buffer management for performing mutlishot read/recv
+  using io_uring's registered buffers feature.
+
+- Add some more low-level methods for performing I/O operations supported by
+  io_uring: splice, fsync, mkdir, fadvise etc.

data/grant-2025/journal.md

@@ -455,16 +455,180 @@ Ruby I/O layer. Some interesting warts in the Ruby `IO` implementation:
   [Extralite](https://github.com/digital-fabric/extralite/)): normally, using an
   actor interface, or protected by a mutex. I'll try to follow up with a
   benchmark measuring concurrent access to SQLite DBs, similar to the PG one.
-  
+
   Another interesting benchmark I found was one for resolving DNS addresses
   using Ruby's builtin `Addrinfo` API, the bundled `resolv` gem, and a basic DNS
   resolver included in UringMachine (I totally forgot I made one). Here too, I'd
   like to add a benchmark to measure how these different solutions do in a
   highly concurrent scenario.
-  
+
 - Thanks to one of these old benchmarks I made a change that more than doubled
   the performance of `UM#snooze`. What this method does is it adds the current
   fiber to the end of the runqueue, and yields control to the next fiber in the
   runqueue, or to process available CQE's. This method is useful for testing,
   but also for yielding control periodically when performing CPU-bound work, in
   order to keep the application responsive and improve latency.
+
+# 2025-12-14
+
+- Changed how `struct um_op`s are allocated. This struct is used to represent
+  any io_uring operation. It is also used to represent runqueue entries. Now,
+  for most I/O operations, this struct is stack-allocated. But when a new fiber
+  is scheduled, or when using the `#timeout` or any of the `#xxx_async` methods,
+  like `#close_async` or `#write_async`, we need to use a heap-allocated
+  `um_op`, because we don't control its lifetime. In order to minimize
+  allocations, once a `um_op` is done with (it's been pulled out of the
+  runqueue, or its corresponding CQE has been processed), it is put on a
+  freelist in order to be reused when needed. Previously, when the freelist was
+  empty, UringMachine would just allocate a new one using `malloc`. Now
+  UringMachine allocates a array of 256 structs at once and puts all of them on
+  the freelist.
+- Implemented the vectorized versions of `#write` and `#send`, so now one can
+  use `#writev` and `#sendv` to send multiple buffers at once. This could be
+  very useful for situations like sending an HTTP response, which is made of a
+  headers part and a body part. Also, `#writev` and `#sendv` are guaranteed to
+  write/send the entirety of the given buffers, unlike `#write` and `#send`
+  which can do partial write/send (for `#send` you can specify the
+  `UM::MSG_WAITALL` flag) to guarantee a complete send.
+- With the new built-in `Set` class and its new [C
+  API](https://github.com/ruby/ruby/pull/13735), I've switched the internal
+  `pending_fibers` holding fibers waiting for an operation to complete, from a
+  hash to a set.
+
+# 2025-12-15
+
+- Working more with benchmarks, it has occurred to me that with the current
+  design of UringMachine, whenever we check for I/O completions (which is also
+  the moment when we make I/O submissions to the kernel), we leave some
+  performance on the table. This is because when we call `io_uring_submit` or
+  `io_uring_wait_cqes`, we make a blocking system call (namely,
+  `io_uring_enter`), and correspondigly we release the GVL.
+
+  What this means is that while we're waiting for the system call to return, the
+  GVL is available for another Ruby thread to do CPU-bound work. Normally when
+  there's a discussion about concurrency in Ruby, there's this dichotomy: it's
+  either threads or fibers. But as described above, even when using fibers and
+  io_uring for concurrent I/O, we still need to enter the kernel periodically in
+  order to submit operations and process completions. So this is an opportunity
+  to yield the GVL to a different thread, which can run some Ruby code while the
+  first thread is waiting for the system call to return.
+
+  With that in mind, I modified the benchmark code to see what would happen if
+  we run two UringMachine instances on two separate threads. The results are
+  quite interesting: splitting the work load between two UringMachine instances
+  running on separate threads, we get a marked improvement in performance.
+  Depending on the benchmark, we get even better performance if we increase the
+  thread count to 4.
+
+  But, as we increase the thread count, we eventually hit diminishing returns
+  and risk actually having worse performance than with just a single thread. So,
+  at least for the workloads I tested (including a very primitive HTTP/1.1
+  server), the sweet spot is between 2 and 4 threads.
+
+  One thing I have noticed though, is that while the pure UM version (i.e. using
+  the UM low-level API) gets a boost from running on multiple threads, the UM
+  fiber scheduler actually can perform worse. This is also the case for the
+  Async fiber scheduler, so this might have to do with the fact that the Ruby IO
+  class does a lot of work behind the scenes, including locking write mutexes
+  and other stuff that's done when the IO is closed. This is still to be
+  investigated...
+
+# 2025-12-16
+
+- Added `UM#accept_into_queue`, which accepts incoming socket connections in a
+  loop and pushes them to the given queue.
+
+- Improved error handling in the fiber scheduler, and added more tests. There
+  are now about 4.2KLoC of test code, with 255 test cases and 780 assertions. And
+  that's without all the tests that depend on the
+  [`rb_process_new`](https://github.com/ruby/ruby/pull/15213) API, the PR for
+  which is currently still not merged.
+
+- Added a test mode to UringMachine that affects runqueue processing, without
+  impacting performance under normal conditions.
+
+# 2025-12-17
+
+- I noticed that the fiber scheduler `#io_write` was not being called on
+  `IO#flush` or when closing an IO with buffered writes. So any time the IO
+  write buffer needs to be flushed, instead of calling the `#io_write` hook, the
+  Ruby I/O layer would just run this on a worker thread by calling the
+  `#blocking_operation_wait` hook. I've made a
+  [PR](https://github.com/ruby/ruby/pull/15609) to fix this.
+
+# 2025-12-18
+
+- Added a [PR](https://github.com/ruby/ruby/pull/15629) to update Ruby NEWS with
+  changes to the FiberScheduler interface.
+
+- I did some more verification work on the fiber scheduler implementation. I
+  added more tests and improved error handling in read/write hooks.
+
+- Made some small changes to fiber scheduling. I added a test mode which peeks
+  at CQEs on each snooze, in order to facilitate testing.
+
+# 2025-12-20
+
+- Did some more work on benchmarks, and added provisory GVL time measurement.
+
+- Implemented sidecar mode - the basic idea is that UringMachine starts an
+  auxiliary thread that loops entering the kernel with a call to
+  `io_uring_enter` in order to make CQEs available. On return from the system
+  call, it signals through a futex that ready CQEs can be processed. 
+
+  On fiber switch, the next fiber to run is shifted from the runqueue. If the
+  runqueue is empty, the UringMachine will wait for the signal, and then process
+  all CQEs. The idea is that in a single threaded environment, under high enough
+  I/O load,  we don't need to release the GVL in order to process ready CQEs,
+  and thus we can better saturate the CPU.
+
+# 2025-12-26
+
+- Finished up the sidecar mode implementation. I did some preliminary benchmarks
+  and this mode does provide a small performance benefit, depending on the
+  context. But for the moment, I consider this mode experimental.
+
+# 2026-01-07
+
+- In the last week I've been working on implementing a buffer pool  
+  with automatic buffer manangement. I've been contemplating the design for a
+  few weeks already, and after the vacation has decided the idea is solid enough
+  for me to start writing some code. But let me back up and explain what I'm
+  trying to achieve.
+
+  The io_uring interface includes a facility for setting up buffer rings. The
+  idea is that the application provides buffers to the kernel, which uses those
+  buffers for reading or receiving repeatedly from an fd, letting the
+  application know with each CQE which buffer was used and with how much data.
+  This is particularly useful when dealing with bursts of incoming data.
+  
+  The application initiates multishot read/recv operations on each connection,
+  and the kernel has at its disposition a pool of application-provided buffers
+  it can use whenever a chunk of data is read / received. So the kernel consumes
+  those buffers as needed, and fills them with data when it becomes available.
+  Those data will be processed by the application at some later time when it's
+  ready to process CQEs. The application will then add the consumed buffers back
+  to the buffer ring, making them available to the kernel again.
+
+  Multiple buffer rings may be registered by the application, each with a set
+  maxmimum number of buffers and with a buffer group id (`bgid`). The buffers
+  added to a buffer ring may be of any size. Each buffer in a buffer ring also
+  has an id (`bid`). So buffers are identified by the tuple `[bgid, bid]`. When
+  submitting a multishot read/recv operation, we indicate the buffer group id
+  (`bgid`), letting the kernel know which buffer ring to use. The kernel then
+  generates CQEs (completion queue entries) which contain the id of the buffer
+  that contains the data (`bid`). Crucially, a single buffer ring may be used in
+  multiple concurrent multishot read/recv operations on different fd's.
+
+  In addition,on recent kernels io_uring is capable of partially consuming
+  buffers, which prevents wasting buffer space. When a buffer ring is set up for
+  [partial buffer
+  consumption](https://www.man7.org/linux/man-pages/man3/io_uring_setup_buf_ring.3.html),
+  each CQE relating to a multishot read/recv operation will also have a flag
+  telling the application [whether the buffer will be further
+  used](https://www.man7.org/linux/man-pages/man3/io_uring_prep_recv.3.html)
+  beyond the amount of data readily available. Each completion of a given buffer
+  ID will continue where the previous one left off. So it's great that buffer
+  space can be used fully by the kernel, but the application is required to keep
+  track of a "cursor" for each buffer.
+  

data/grant-2025/tasks.md

@@ -12,29 +12,39 @@
         https://unixism.net/loti/tutorial/sq_poll.html
   - [v] Add `UM.socketpair`
 
-  - [ ] Add more metrics
+  - [v] Add more metrics
     - [v] runqueue depth
     - [v] number of pending fibers
     - [v] ops: transient count, free count
     - [v] total fiber switches, total waiting for CQEs
-    - [ ] watermark: ops_pending, ops_unsubmitted, ops_runqueue, ops_free, ops_transient
-          (only in profile mode)
-  - [ ] Performance tuning parameters
-    - [ ] max fiber switches before processing CQEs
-    - [ ] max fiber switches before submitting unsubmitted SQEs
-    - [ ] measure switches since last submitting / last CQE processing
-
-  - [ ] Better buffer management buffer rings
+
+  - [v] Make writev automatically complete partial writes
+
+  - [ ] Add inotify API
+
+    https://www.man7.org/linux/man-pages/man7/inotify.7.html
+
+  - [ ] Better buffer management
     - [v] Add `UM#sendv` method (see below)
     - [v] Benchmark `#sendv` vs `#send_bundle` (in concurrent situation)
+    - [v] Support for `IO::Buffer`?
     - [ ] Benchmark `#read_each` vs `#read` (in concurrent situation)
-    - [ ] Support for `IO::Buffer`? How's the API gonna look like?
-    - [ ] Some higher-level abstraction for managing a *pool* of buffer rings
-
-  - [ ] Add some way to measure fiber CPU time.
-        https://github.com/socketry/async/issues/428
-
-- [ ] UringMachine Fiber::Scheduler implementation
+    - [ ] Implement automatic buffer pool:
+      - [ ] Automatic buffer allocation,registration and management.
+      - [ ] Support for partial buffer consumption.
+      - [ ] Data processing through a rewritten stream implementation.
+
+  - [v] Sidecar mode
+    - [v] Convert `UM#initialize` to take kwargs
+      - [v] `:size` - SQ entries
+      - [v] `:sqpoll` - sqpoll mode
+      - [v] `:sidecar` - sidecar mode
+    - [v] Sidecar implementation
+      - [v] sidecar thread
+      - [v] futex handling
+      - [v] submission logic
+
+- [v] UringMachine Fiber::Scheduler implementation
   - [v] Check how scheduler interacts with `fork`.
   - [v] Implement `process_wait` (with `rb_process_status_new`)
   - [v] Implement `fiber_interrupt` hook
@@ -97,7 +107,7 @@
     - [v] pipes: multiple pairs of fibers - reader / writer
     - [v] sockets: echo server + many clients
 
-  - [ ] Benchmarks
+  - [v] Benchmarks
     - [v] UM queue / Ruby queue (threads) / Ruby queue with UM fiber scheduler
 
           N groups where each group has M producers and O consumers accessing the same queue.

data/lib/uringmachine/fiber_scheduler.rb

@@ -13,10 +13,11 @@ class UringMachine
     # Initializes a new worker pool.
     #
     # @return [void]
-    def initialize
+    def initialize(max_workers = Etc.nprocessors)
+      @max_workers = max_workers
       @pending_count = 0
       @worker_count = 0
-      @max_workers = Etc.nprocessors
+
       @worker_mutex = UM::Mutex.new
       @job_queue = UM::Queue.new
       @workers = []
@@ -52,7 +53,7 @@ class UringMachine
 
     # @return [void]
     def run_worker_thread
-      machine = UM.new(4)
+      machine = UM.new(size: 4)
       loop do
         q, op = machine.shift(@job_queue)
         @pending_count += 1
@@ -76,7 +77,7 @@ class UringMachine
   class FiberScheduler
 
     # The blocking operation thread pool is shared by all fiber schedulers.
-    @@blocking_operation_thread_pool = BlockingOperationThreadPool.new
+    DEFAULT_THREAD_POOL = BlockingOperationThreadPool.new
 
     # UringMachine instance associated with scheduler.
     attr_reader :machine
@@ -92,8 +93,9 @@ class UringMachine
     #
     # @param machine [UringMachine, nil] UringMachine instance
     # @return [void]
-    def initialize(machine = nil)
+    def initialize(machine = nil, thread_pool = DEFAULT_THREAD_POOL)
       @machine = machine || UM.new
+      @thread_pool = thread_pool
       @fiber_map = ObjectSpace::WeakMap.new
       @thread = Thread.current
     end
@@ -107,7 +109,8 @@ class UringMachine
     # the fiber map, scheduled on the scheduler machine, and started before this
     # method returns (by calling snooze).
     #
-    # @param block [Proc] fiber block @return [Fiber]
+    # @param block [Proc] fiber block
+    # @return [Fiber]
     def fiber(&block)
       fiber = Fiber.new(blocking: false) { @machine.run(fiber, &block) }
 
@@ -145,7 +148,7 @@ class UringMachine
     # @param op [callable] blocking operation
     # @return [void]
     def blocking_operation_wait(op)
-      @@blocking_operation_thread_pool.process(@machine, op)
+      @thread_pool.process(@machine, op)
     end
 
     # Blocks the current fiber by yielding to the machine. This hook is called
@@ -188,7 +191,6 @@ class UringMachine
     # Yields to the next runnable fiber.
     def yield
       @machine.snooze
-      # @machine.yield
     end
 
     # Waits for the given io to become ready.
@@ -198,7 +200,6 @@ class UringMachine
     # @param timeout [Number, nil] optional timeout
     # @param return
     def io_wait(io, events, timeout = nil)
-      # p(io_wait: io, events:)
       timeout ||= io.timeout
       if timeout
         @machine.timeout(timeout, Timeout::Error) {
@@ -243,7 +244,7 @@ class UringMachine
       length = buffer.size if length == 0
 
       if (timeout = io.timeout)
-				@machine.timeout(timeout, Timeout::Error) do
+        @machine.timeout(timeout, Timeout::Error) do
           @machine.read(io.fileno, buffer, length, offset)
         rescue Errno::EINTR
           retry
@@ -253,6 +254,8 @@ class UringMachine
       end
     rescue Errno::EINTR
       retry
+    rescue Errno => e
+      -e.errno
     end
 
     # Reads from the given IO at the given file offset
@@ -267,7 +270,7 @@ class UringMachine
       length = buffer.size if length == 0
 
       if (timeout = io.timeout)
-				@machine.timeout(timeout, Timeout::Error) do
+        @machine.timeout(timeout, Timeout::Error) do
           @machine.read(io.fileno, buffer, length, offset, from)
         rescue Errno::EINTR
           retry
@@ -277,6 +280,8 @@ class UringMachine
       end
     rescue Errno::EINTR
       retry
+    rescue Errno => e
+      -e.errno
     end
 
     # Writes to the given IO.
@@ -287,12 +292,11 @@ class UringMachine
     # @param offset [Integer] write offset
     # @return [Integer] bytes written
     def io_write(io, buffer, length, offset)
-      # p(io_write: io, length:, offset:, timeout: io.timeout)
       length = buffer.size if length == 0
       buffer = buffer.slice(offset) if offset > 0
 
       if (timeout = io.timeout)
-				@machine.timeout(timeout, Timeout::Error) do
+        @machine.timeout(timeout, Timeout::Error) do
           @machine.write(io.fileno, buffer, length)
         rescue Errno::EINTR
           retry
@@ -302,6 +306,8 @@ class UringMachine
       end
     rescue Errno::EINTR
       retry
+    rescue Errno => e
+      -e.errno
     end
 
     # Writes to the given IO at the given file offset.
@@ -313,12 +319,11 @@ class UringMachine
     # @param offset [Integer] buffer offset
     # @return [Integer] bytes written
     def io_pwrite(io, buffer, from, length, offset)
-      # p(io_pwrite: io, from:, length:, offset:, timeout: io.timeout)
       length = buffer.size if length == 0
       buffer = buffer.slice(offset) if offset > 0
 
       if (timeout = io.timeout)
-				@machine.timeout(timeout, Timeout::Error) do
+        @machine.timeout(timeout, Timeout::Error) do
           @machine.write(io.fileno, buffer, length, from)
         rescue Errno::EINTR
           retry
@@ -328,6 +333,8 @@ class UringMachine
       end
     rescue Errno::EINTR
       retry
+    rescue Errno => e
+      -e.errno
     end
 
     # Closes the given fd.
@@ -335,8 +342,9 @@ class UringMachine
     # @param fd [Integer] file descriptor
     # @return [Integer] file descriptor
     def io_close(fd)
-      # p(io_close: fd)
       @machine.close_async(fd)
+    rescue Errno => e
+      -e.errno
     end
 
     if UM.method_defined?(:waitid_status)
@@ -366,17 +374,17 @@ class UringMachine
     #
     # @param hostname [String] hostname to resolve
     # @return [Array<Addrinfo>] array of resolved addresses
-		def address_resolve(hostname)
-			Resolv.getaddresses(hostname)
-		end
-
-		# Run the given block with a timeout.
-		#
-		# @param duration [Number] timeout duration
-		# @param exception [Class] exception Class
-		# @param message [String] exception message
-		# @param block [Proc] block to run
-		# @return [any] block return value
+    def address_resolve(hostname)
+      Resolv.getaddresses(hostname)
+    end
+
+    # Run the given block with a timeout.
+    #
+    # @param duration [Number] timeout duration
+    # @param exception [Class] exception Class
+    # @param message [String] exception message
+    # @param block [Proc] block to run
+    # @return [any] block return value
     def timeout_after(duration, exception, message, &block)
       @machine.timeout(duration, exception, &block)
     end

data/lib/uringmachine/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class UringMachine
-  VERSION = '0.23.1'
+  VERSION = '0.24.0'
 end

data/lib/uringmachine.rb

@@ -7,10 +7,8 @@ require 'uringmachine/dns_resolver'
 UM = UringMachine
 
 class UringMachine
-  @@fiber_map = {}
-
   def fiber_map
-    @@fiber_map
+    @fiber_map ||= {}
   end
 
   class Terminate < Exception
@@ -20,13 +18,13 @@ class UringMachine
     fiber = klass.new { |v| run_block_in_fiber(block, fiber, v) }
     self.schedule(fiber, value)
 
-    @@fiber_map[fiber] = fiber
+    fiber_map[fiber] = fiber
   end
 
   def run(fiber, &block)
     run_block_in_fiber(block, fiber, nil)
     self.schedule(fiber, nil)
-    @@fiber_map[fiber] = fiber
+    fiber_map[fiber] = fiber
   end
 
   def join(*fibers)
@@ -97,7 +95,7 @@ class UringMachine
   ensure
     fiber.mark_as_done
     # cleanup
-    @@fiber_map.delete(fiber)
+    fiber_map.delete(fiber)
     self.notify_done_listeners(fiber)
 
     # switch away to a different fiber

data/test/helper.rb

@@ -62,17 +62,22 @@ class UMBaseTest < Minitest::Test
 
   def setup
     @machine = UM.new
+    @machine.test_mode = true
   end
 
   def teardown
     return if !@machine
 
-    pending_fibers = @machine.pending_fibers
-    raise "leaked fibers: #{pending_fibers}" if pending_fibers.size > 0
-    
+    # pending_fibers = @machine.pending_fibers
+    # raise "leaked fibers: #{pending_fibers}" if pending_fibers.size > 0
+
     GC.start
   end
 
+  def scheduler_calls_tally
+    @scheduler.calls.map { it[:sym] }.tally
+  end
+
   def assign_port
     @@port_assign_mutex ||= Mutex.new
     @@port_assign_mutex.synchronize do

data/test/test_fiber.rb

@@ -222,6 +222,22 @@ class WaitFibersTest < UMBaseTest
     res = machine.await_fibers(f)
     assert_equal 1, res
   end
+
+  def test_await_fibers_terminate
+    f1 = machine.spin { machine.sleep(1) }
+    f2 = machine.spin { machine.sleep(1) }
+    done = false
+    a = machine.spin do
+      machine.await_fibers([f1, f2])
+    rescue UM::Terminate
+      done = true
+    end
+
+    machine.snooze
+    machine.schedule(a, UM::Terminate.new)
+    machine.join(a)
+    assert_equal true, done
+  end
 end
 
 class ScopeTest < UMBaseTest