polyphony 1.3 → 1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 537c28ec35d12c724a5f5ef4117ada1641d0b49b780757965615665bddbe995a
-  data.tar.gz: 5c11137a7a7dc6774ec815ec8b38791752f1beb939311b5968f0d0b242c7f8ad
+  metadata.gz: 8ec9aae8d4c83ff8dd3332187fdd12fc598596dc4bbab11eaad6400e61769684
+  data.tar.gz: a42cedfc33172dbacc4f61e16fa0a6fce3eb772e78e8829a2dfe744601c77b1c
 SHA512:
-  metadata.gz: 912c4a3cea0a183350c939e588bba70656fcd14a07af6bd5912e3c2208cbb286150477d050482b030c494b0f87f21da7ec350c50f5bfee8e563bc21211143adc
-  data.tar.gz: c72b01fcef19a525cef49b277f0f1fe949956499be51f34f75fe1e87fc6efe9222a43ac96d282460610c3a82989934a0f661e78a27736120117c1ef391036dd6
+  metadata.gz: 2f6c4a70a56c854dfaa707baeac665ed04adcf04a0f67c25419b2de50d367b155ed6aad3e6bb3dcdc6391af08f4a4dafc3afcadecc499f26cb4b5a06a5d0c69e
+  data.tar.gz: e8b74bd9dbec6a0d05e5510514ccff97792e04c0742e85624087eb92fe55d195b553288c0e3132ab468ba7a28a921ead91ce20ce26d6dab93f5b5a467f5f61bd

data/.yardopts

@@ -18,6 +18,7 @@
 ./ext/polyphony
 -
 docs/readme.md
+docs/installation.md
 docs/overview.md
 docs/tutorial.md
 docs/cancellation.md

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+## 1.5 2023-07-28
+
+- Refactor backend_await in io_uring backend
+- Fix calling `Timeout.timeout` with `nil` or `0` (#114)
+- Rework support for io_uring multishot accept
+- Combine SQE submission and waiting for CQE into a single syscall
+- Use io_uring for closing a `Polyphony::Pipe`, removing call to `close()`
+
+## 1.4 2023-07-01
+
+- Implement concurrent `IO#close`
+- Improve docs
+- Use only positional arguments in `IO#read` and `IO#readpartial` (#109 @floriandejonckheere)
+
 ## 1.3 2023-06-23
 
 - Improve cancellation doc page

data/README.md

@@ -23,58 +23,24 @@
 ## What is Polyphony?
 
 Polyphony is a library for building concurrent applications in Ruby. Polyphony
-harnesses the power of [Ruby fibers](https://ruby-doc.org/core-2.5.1/Fiber.html)
-to provide a cooperative, sequential coroutine-based concurrency model. Under
-the hood, Polyphony uses
-[io_uring](https://unixism.net/loti/what_is_io_uring.html) or
+harnesses the power of [Ruby fibers](https://rubyapi.org/3.2/o/fiber) to provide
+a cooperative, sequential coroutine-based concurrency model. Under the hood,
+Polyphony uses [io_uring](https://unixism.net/loti/what_is_io_uring.html) or
 [libev](https://github.com/enki/libev) to maximize I/O performance.
 
 ## Features
 
-* Co-operative scheduling of concurrent tasks using Ruby fibers.
-* High-performance event reactor for handling I/O events and timers.
-* Natural, sequential programming style that makes it easy to reason about
-  concurrent code.
-* Abstractions and constructs for controlling the execution of concurrent code:
-  supervisors, cancel scopes, throttling, resource pools etc.
-* Code can use native networking classes and libraries, growing support for
-  third-party gems such as `pg` and `redis`.
-* Use stdlib classes such as `TCPServer`, `TCPSocket` and
-  `OpenSSL::SSL::SSLSocket`.
-* Competitive performance and scalability characteristics, in terms of both
-  throughput and memory consumption.
-
-## Installing
-
-### System Requirements
-
-In order to use Polyphony you need to have:
-
-- Linux or MacOS (support for Windows will come at a later stage)
-- Ruby (MRI) 3.1 or newer
-
-### Installing the Polyphony Gem
-
-Add this line to your application's Gemfile:
-
-```ruby
-gem 'polyphony'
-```
-
-And then execute:
-
-```bash
-$ bundle
-```
-
-Or install it yourself as:
-
-```bash
-$ gem install polyphony
-```
+* Ruby fibers as the main unit of concurrency.
+* [Structured concurrency](https://en.wikipedia.org/wiki/Structured_concurrency)
+  coupled with robust exception handling.
+* Message passing between fibers, even across threads!
+* High-performance I/O using the core Ruby I/O classes and
+  [io_uring](https://unixism.net/loti/what_is_io_uring.html) with support for
+  [advanced I/O patterns](docs/advanced-io.md).
 
 ## Usage
 
+- [Installation](docs/installation.md)
 - [Overview](docs/overview.md)
 - [Tutorial](docs/tutorial.md)
 - [All About Cancellation: How to Stop Concurrent Operations](docs/cancellation.md)

data/TODO.md

@@ -5,6 +5,10 @@
   - if `io_uring_get_sqe` returns null, call `io_uring_submit`, (snooze fiber)?
     and try again
 
+- closing and shutdown:
+  - `Pipe_free()` - can we use the backend to close the pipe fds?
+  - Implement `BasicSocket#shutdown`, add `Backend_shutdown` API.
+
 - Tracing:
   - Emit events on I/O ops, e.g.:
     - [:op_read_submit, id, io, len]
@@ -19,16 +23,7 @@
 - More tight loops
   - `IO#gets_loop`, `Socket#gets_loop`, `OpenSSL::Socket#gets_loop` (medium effort)
 
-- Add support for `close` to io_uring backend
-
-## Roadmap for Polyphony 1.1
-
-- io_uring
-  - Use playground.c to find out why we when submitting and waiting for
-    completion in single syscall signals seem to be blocked until the syscall
-    returns. Is this a bug in io_uring/liburing?
-
------------------------------------------------------
+## Roadmap for Polyphony 2
 
 - allow backend selection at runtime?
 - Debugging

data/docs/installation.md

@@ -0,0 +1,30 @@
+# @title Installation
+
+# Installation
+
+## System Requirements
+
+In order to use Polyphony you need to have:
+
+- Linux or MacOS (support for Windows will come at a later stage)
+- Ruby (MRI) 3.1 or newer
+
+## Installing the Polyphony Gem
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'polyphony'
+```
+
+And then execute:
+
+```bash
+$ bundle
+```
+
+Or install it yourself as:
+
+```bash
+$ gem install polyphony
+```

data/docs/readme.md

@@ -25,58 +25,24 @@
 ## What is Polyphony?
 
 Polyphony is a library for building concurrent applications in Ruby. Polyphony
-harnesses the power of [Ruby fibers](https://ruby-doc.org/core-2.5.1/Fiber.html)
-to provide a cooperative, sequential coroutine-based concurrency model. Under
-the hood, Polyphony uses
-[io_uring](https://unixism.net/loti/what_is_io_uring.html) or
+harnesses the power of [Ruby fibers](https://rubyapi.org/3.2/o/fiber) to provide
+a cooperative, sequential coroutine-based concurrency model. Under the hood,
+Polyphony uses [io_uring](https://unixism.net/loti/what_is_io_uring.html) or
 [libev](https://github.com/enki/libev) to maximize I/O performance.
 
 ## Features
 
-* Co-operative scheduling of concurrent tasks using Ruby fibers.
-* High-performance event reactor for handling I/O events and timers.
-* Natural, sequential programming style that makes it easy to reason about
-  concurrent code.
-* Abstractions and constructs for controlling the execution of concurrent code:
-  supervisors, cancel scopes, throttling, resource pools etc.
-* Code can use native networking classes and libraries, growing support for
-  third-party gems such as `pg` and `redis`.
-* Use stdlib classes such as `TCPServer`, `TCPSocket` and
-  `OpenSSL::SSL::SSLSocket`.
-* Competitive performance and scalability characteristics, in terms of both
-  throughput and memory consumption.
-
-## Installing
-
-### System Requirements
-
-In order to use Polyphony you need to have:
-
-- Linux or MacOS (support for Windows will come at a later stage)
-- Ruby (MRI) 3.1 or newer
-
-### Installing the Polyphony Gem
-
-Add this line to your application's Gemfile:
-
-```ruby
-gem 'polyphony'
-```
-
-And then execute:
-
-```bash
-$ bundle
-```
-
-Or install it yourself as:
-
-```bash
-$ gem install polyphony
-```
+* Ruby fibers as the main unit of concurrency.
+* [Structured concurrency](https://en.wikipedia.org/wiki/Structured_concurrency)
+  coupled with robust exception handling.
+* Message passing between fibers, even across threads!
+* High-performance I/O using the core Ruby I/O classes and
+  [io_uring](https://unixism.net/loti/what_is_io_uring.html) with support for
+  {file:/docs/advanced-io.md advanced I/O patterns}.
 
 ## Usage
 
+- {file:/docs/installation.md       Installation}
 - {file:/docs/overview.md           Overview}
 - {file:/docs/tutorial.md           Tutorial}
 - {file:/docs/cancellation.md       All About Cancellation: How to Stop Concurrent Operations}

data/examples/core/stages.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+require 'bundler/setup'
+require 'polyphony'
+
+# Based on the design of Elixir's GenStage
+
+class Producer
+  def initialize(mod, *a, **b)
+    extend(mod)
+    setup(*a, **b)
+    @_fiber = spin do
+      receive_loop do |msg|
+        case msg[:kind]
+        when :demand
+          items = handle_demand(msg[:limit])
+          msg[:peer] << items
+        end
+      end
+    end
+  end
+
+  def <<(msg)
+    @_fiber << msg
+  end
+end
+
+module Counter
+  def setup(counter = 0)
+    @counter = counter
+  end
+
+  def handle_demand(demand)
+    events = (@counter...@counter + demand).to_a
+    @counter += demand
+    events
+  end
+end
+
+counter = Producer.new(Counter, 0)
+
+class Consumer
+  def initialize(mod, *a, **b)
+    extend(mod)
+    setup(*a, **b) if respond_to?(:setup)
+    @_fiber = spin do
+      while true
+        items = get_items
+        handle_items(items)
+      end
+    end
+
+    @max_demand = 10
+    @min_demand = 5
+  end
+
+  def subscribe(upstream)
+    @upstream = upstream
+  end
+
+  private
+
+  def get_items
+    send_demand(@max_demand) if !@sent_demand
+    items = receive
+    send_demand(@min_demand)
+    items
+  end
+
+  def send_demand(demand)
+    if @upstream
+      @upstream << { peer: Fiber.current, kind: :demand, limit: demand }
+      @sent_demand = true
+    else
+      sleep 0.1
+    end
+  end
+end
+
+module Printer
+  def handle_items(items)
+    sleep 1
+    puts "got: #{items.join(' ')}"
+  end
+end
+
+# counter << { peer: Fiber.current, kind: :demand, limit: 10 }
+# r = receive
+
+# p r: r
+
+# counter << { peer: Fiber.current, kind: :demand, limit: 10 }
+# r = receive
+# p r: r
+
+printer = Consumer.new(Printer)
+printer.subscribe(counter)
+
+sleep

data/examples/pipes/http_server.rb

@@ -4,33 +4,63 @@ require 'bundler/inline'
 
 gemfile do
   source 'https://rubygems.org'
-  gem 'h1p'
-  gem 'polyphony', path: '.'
+  gem 'h1p', path: '../h1p'
+  # gem 'polyphony', path: '.'
 end
 
-require 'polyphony'
+# require 'polyphony'
 require 'h1p'
 
-def handle_client(conn)
-  spin do
-    parser = H1P::Parser.new(conn, :server)
+module ::Kernel
+  def trace(*args)
+    STDOUT << format_trace(args)
+  end
 
-    while true # assuming persistent connection
-      headers = parser.parse_headers
-      break unless headers
+  def format_trace(args)
+    if args.first.is_a?(String)
+      if args.size > 1
+        format("%s: %p\n", args.shift, args)
+      else
+        format("%s\n", args.first)
+      end
+    else
+      format("%p\n", args.size == 1 ? args.first : args)
+    end
+  end
 
-      parser.read_body unless parser.complete?
+  def monotonic_clock
+    ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+  end
+end
 
+def handle_client(conn)
+  Thread.new do
+    reader = proc do |len, buf, buf_pos|
+      trace(len:, buf:, buf_pos:)
+      s = conn.readpartial(len)
+      buf ? (buf << s) : +s
+    rescue EOFError
+      nil
+    end
+    parser = H1P::Parser.new(reader, :server)
+    # parser = H1P::Parser.new(conn, :server)
+    while (headers = parser.parse_headers)
+      parser.read_body unless parser.complete?
       conn << "HTTP/1.1 200 OK\r\nContent-Length: 14\r\n\r\nHello, world!\n"
     end
-  rescue Errno::ECONNRESET
+  rescue Errno::ECONNRESET, Errno::EPIPE
     # ignore
   rescue H1P::Error
     puts 'Got invalid request, closing connection...'
   ensure
+    parser = nil
     conn.close rescue nil
   end
 end
 
 puts "Serving HTTP on port 1234..."
-TCPServer.new('0.0.0.0', 1234).accept_loop { |c| handle_client(c) }
+s = TCPServer.new('0.0.0.0', 1234)
+while true
+  c = s.accept
+  handle_client(c)
+end

data/examples/pipes/http_server2.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require 'bundler/inline'
+
+gemfile do
+  source 'https://rubygems.org'
+  gem 'http_parser.rb'
+  gem 'polyphony', path: '.'
+end
+
+require 'polyphony'
+require 'http_parser.rb'
+
+def handle_client(conn)
+  spin do
+    parser = Http::Parser.new
+    done = false
+    headers = nil
+    parser.on_headers_complete = proc do |h|
+      headers = h
+      headers[':method'] = parser.http_method
+      headers[':path'] = parser.request_url
+    end
+    parser.on_message_complete = proc { done = true }
+
+    while true # assuming persistent connection
+      conn.read_loop do |msg|
+        parser << msg
+        break if done
+      end
+  
+      conn << "HTTP/1.1 200 OK\r\nContent-Length: 14\r\n\r\nHello, world!\n"
+      done = false
+      headers = nil
+    end
+  rescue Errno::ECONNRESET, Errno::EPIPE
+    # ignore
+  ensure
+    parser = nil
+    conn.close rescue nil
+  end
+end
+
+puts "Serving HTTP on port 1234..."
+TCPServer.new('0.0.0.0', 1234).accept_loop { |c| handle_client(c) }

data/ext/polyphony/backend_common.c

@@ -458,7 +458,7 @@ VALUE Backend_stats(VALUE self) {
 
 VALUE Backend_verify_blocking_mode(VALUE self, VALUE io, VALUE blocking) {
   io_verify_blocking_mode(io, rb_io_descriptor(io), blocking);
-  return self;
+  return io;
 }
 
 void backend_setup_stats_symbols(void) {

data/ext/polyphony/backend_common.h

@@ -2,9 +2,14 @@
 #define BACKEND_COMMON_H
 
 #include <sys/types.h>
+#ifdef POLYPHONY_WINDOWS
+#include <winsock2.h>
+#else
 #include <arpa/inet.h>
 #include <netinet/in.h>
 #include <netdb.h>
+#include <sys/socket.h>
+#endif
 
 #include "ruby.h"
 #include "ruby/io.h"