async 2.27.0 → 2.27.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4c9d36d758f8a197b7c00d3d2e4d83d280cfd8ddf4fb60bba1d7b9e95b64ec47
-  data.tar.gz: c0c19dcc509563b18a9385c1ff07a982054e6e984c9ed9c1fd715ee027c74f09
+  metadata.gz: c419b1abab6e535bd3f68c7de489ad0086e236f54b513d959f0b5d81373039e2
+  data.tar.gz: d995214a8ac8539172ffe669563cdaaca34925b7da97939095d2f48f22628ead
 SHA512:
-  metadata.gz: b936e5c17d8e9e2eec7f3d9b3d2d4b00b5a16359cfb267a036f72fb254b53aeb234f8b9368ba1cd2ba41010438dd8da120621005bc1791fc554ee62647e46ed1
-  data.tar.gz: 808b0ce51e2bfa4a28d01126d59627409e6a25b37ee8e9e1f8146c138fcef995ab693e699f9c3ece326e234100a397c19e917a55c9d1e34dd797d26531e411f5
+  metadata.gz: 79cf05bc46450f20fdbf524f7bc1886564b26cadcbf731c3186c3f1dbec93ad51f15bc042591b77a790d0647a768103533b2eae8ff732ef5350e3ad2b866ffe0
+  data.tar.gz: 7663314ae8ad34d6c7d647ce8065268cc91754828d474044e3b2c5a6c80bbc9f8467b12fb90af6d4f3d5db6c0a14cc74735ce32ffa37998b2d9c261c5cc4a876

data/agent.md

@@ -30,6 +30,10 @@ This guide explains how to test and monitor documentation coverage in your Ruby
 
 This guide covers documentation practices and pragmas supported by the Decode gem for documenting Ruby code. These pragmas provide structured documentation that can be parsed and used to generate A...
 
+#### [Setting Up RBS Types and Steep Type Checking for Ruby Gems](.context/decode/types.md)
+
+This guide covers the process for establishing robust type checking in Ruby gems using RBS and Steep, focusing on automated generation from source documentation and proper validation.
+
 ### sus
 
 A fast and scalable test runner.
@@ -45,3 +49,15 @@ There are two types of mocking in sus: `receive` and `mock`. The `receive` match
 #### [Shared Test Behaviors and Fixtures](.context/sus/shared.md)
 
 Sus provides shared test contexts which can be used to define common behaviours or tests that can be reused across one or more test files.
+
+### sus-fixtures-agent-context
+
+Test fixtures for running in Async.
+
+#### [Getting Started](.context/sus-fixtures-agent-context/getting-started.md)
+
+This guide explains how to use the `sus-fixtures-agent-context` gem to test agent contexts.
+
+#### [GitHub Actions](.context/sus-fixtures-agent-context/github-actions.md)
+
+This guide explains how to integrate the `sus-fixtures-agent-context` gem with GitHub Actions for testing agent contexts.

data/context/best-practices.md

@@ -0,0 +1,188 @@
+# Best Practices
+
+This guide gives an overview of best practices for using Async.
+
+## Use a top-level `Sync` to denote the root of your program
+
+`Async{}` has two uses: it creates an event loop if one doesn't exist, and it creates a task which runs asynchronously with respect to the parent scope. However, the top level `Async{}` block will be synchronous because it creates the event loop. In some programs, you do not care about executing asynchronously, but you still want your code to run in an event loop. `Sync{}` exists to do this efficiently.
+
+```ruby
+require 'async'
+
+class Packages
+	def initialize(urls)
+		@urls = urls
+	end
+	
+	def fetch
+		# A common use case is to make functions which appear synchronous, but internally use asynchronous execution:
+		Sync do |task|
+			@urls.map do |url|
+				task.async do
+					fetch(url)
+				end
+			end.map(&:wait)
+		end
+	end
+end
+```
+
+`Sync{...}` is semantically equivalent to `Async{}.wait`, but it is more efficient. It is the preferred way to run code in an event loop at the top level of your program or to ensure some code runs in an event loop without creating a new task. The name `Sync` means "Synchronous Async", indicating that it runs synchronously with respect to the outer scope, but still allows for asynchronous execution within it.
+
+### Current Task
+
+In some scenarios, it can be invalid to call a method outside of an event loop, for example a top level `Async{...}` can block forever, which might be unexpected.
+
+```ruby
+def wait(queue)
+	Async do
+		queue.pop
+	end
+end
+```
+
+You can force callers of a method to only call the method within an asynchronous context by using a keyword argument `parent: Async::Task.current`. If no task is present, this will raise an exception.
+
+```ruby
+def wait(queue, parent: Async::Task.current)
+	parent.async do
+		queue.pop
+	end
+end
+```
+
+This expresses the intent to the caller that this method should only be invoked from within an asynchonous task. In addition, it allows the caller to substitute other parent objects, like semaphores or barriers, which can be useful for managing concurrency.
+
+## Use barriers to manage unbounded concurrency
+
+Barriers provide a way to manage an unbounded number of tasks.
+
+```ruby
+Async do
+	barrier = Async::Barrier.new
+	
+	items.each do |item|
+		barrier.async do
+			process(item)
+		end
+	end
+	
+	# Process the tasks in order of completion:
+	barrier.wait do |task|
+		result = task.wait
+		# Do something with result.
+
+		# If you don't want to wait for any more tasks you can break:
+		break
+	end
+	
+	# Or just wait for all tasks to finish:
+	barrier.wait # May raise an exception if a task failed.
+ensure
+	# Stop all outstanding tasks in the barrier:
+	barrier&.stop
+end
+```
+
+## Use a semaphore to limit the number of concurrent tasks
+
+Semaphores allow you to limit the level of concurrency to a fixed number of tasks:
+
+```ruby
+Async do |task|
+	barrier = Async::Barrier.new
+	semaphore = Async::Semaphore.new(4, parent: barrier)
+	
+	# Since the semaphore.async may block, we need to run the work scheduling in a child task:
+	task.async do
+		items.each do |item|
+			semaphore.async do
+				process(item)
+			end
+		end
+	end
+	
+	# Wait for all the work to complete:
+	barrier.wait
+ensure
+	# Stop all outstanding tasks in the barrier:
+	barrier&.stop
+end
+```
+
+In general, the barrier should be the root of your task hierarchy, and the semaphore should be a child of the barrier. This allows you to manage the lifetime of all tasks created by the semaphore, and ensures that all tasks are stopped when the barrier is stopped.
+
+### Idler
+
+Idlers are like semaphores but with a limit defined by current processor utilization. In other words, an idler will do work up to a specific ratio of idle/busy time in the scheduler, and try to maintain that.
+
+```ruby
+Async do
+	# Create an idler that will aim for a load average of 80%:
+	idler = Async::Idler.new(0.8)
+	
+	# Some list of work to be done:
+	work.each do |work|
+		idler.async do
+			# Do the work:
+			work.call
+		end
+	end
+end
+```
+
+The idler will try to schedule as much work such that the load of the scheduler stays at around 80% saturation.
+
+## Use queues to share data between tasks
+
+Queues allow you to share data between tasks without the risk of data corruption or deadlocks.
+
+```ruby
+Async do |task|
+	queue = Async::Queue.new
+	
+	reader = task.async do
+		while chunk = socket.gets
+			queue.push(chunk)
+		end
+	end
+		# After this point, we won't be able to add items to the queue, and popping items will eventually result in nil once all items are dequeued:
+		queue.close
+	end
+	
+	# Process items from the queue:
+	while line = queue.pop
+		process(line)
+	end
+end
+```
+
+The above program may have unbounded memory use, so it can be a good idea to use a limited queue with back-pressure:
+
+```ruby
+Async do |task|
+	queue = Async::LimitedQueue.new(8)
+	
+	# Everything else is the same from the queue example, except that the pushing onto the queue will block once 8 items are buffered.
+end
+```
+
+## Use timeouts for operations that might block forever
+
+General timeouts can be imposed by using `task.with_timeout(duration)`.
+
+```ruby
+Async do |task|
+	# This will raise an Async::TimeoutError after 1 second:
+	task.with_timeout(1) do |timeout|
+		# Timeout#duration= can be used to adjust the duration of the timeout.
+		# Timeout#cancel can be used to cancel the timeout completely.
+		
+		sleep 10
+	end
+end
+```
+
+It can be especially important to impose timeouts when processing user-provided data.
+
+## 

data/context/debugging.md

@@ -0,0 +1,63 @@
+# Debugging
+
+This guide explains how to debug issues with programs that use Async.
+
+## Debugging Techniques
+
+### Debugging with `puts`
+
+The simplest way to debug an Async program is to use `puts` to print messages to the console. This is useful for understanding the flow of your program and the values of variables. However, it can be difficult to use `puts` to debug programs that use asynchronous code, as the output may be interleaved. To prevent this, wrap it in `Fiber.blocking{}`:
+
+```ruby
+require 'async'
+
+Async do
+	3.times do |i|
+		sleep i
+		Fiber.blocking{puts "Slept for #{i} seconds."}
+	end
+end
+```
+
+Using `Fiber.blocking{}` prevents any context switching until the block is complete, ensuring that the output is not interleaved and that flow control is strictly sequential. You should not use `Fiber.blocking{}` in production code, as it will block the reactor.
+
+### Debugging with IRB
+
+You can use IRB to debug your Async program. In some cases, you will want to stop the world and inspect the state of your program. You can do this by wrapping `binding.irb` inside a `Fiber.blocking{}` block:
+
+```ruby
+Async do
+	3.times do |i|
+		sleep i
+		# The event loop will stop at this point and you can inspect the state of your program.
+		Fiber.blocking{binding.irb}
+	end
+end
+```
+
+If you don't use `Fiber.blocking{}`, the event loop will continue to run and you will end up with three instances of `binding.irb` running.
+
+### Debugging with `Async::Debug`
+
+The `async-debug` gem provides a visual debugger for Async programs. It is a powerful tool that allows you to inspect the state of your program and see the hierarchy of your program:
+
+```ruby
+require 'async'
+require 'async/debug'
+
+Sync do
+	debugger = Async::Debug.serve
+	
+	3.times do
+		Async do |task|
+			while true
+				duration = rand
+				task.annotate("Sleeping for #{duration} second...")
+				sleep(duration)
+			end
+		end
+	end
+end
+```
+
+When you run this program, it will start a web server on `http://localhost:9000`. You can open this URL in your browser to see the state of your program.

data/context/getting-started.md

@@ -0,0 +1,177 @@
+# Getting Started
+
+This guide shows how to add async to your project and run code asynchronously.
+
+Async is a Ruby library that provides asynchronous programming capabilities using fibers and a fiber scheduler. It allows you to write non-blocking, concurrent code that's easy to understand and maintain.
+
+## Installation
+
+Add the gem to your project:
+
+~~~ bash
+$ bundle add async
+~~~
+
+## Core Concepts
+
+`async` has several core concepts:
+
+- A {ruby Async::Task} instance which captures your sequential computations.
+- A {ruby Async::Reactor} instance which implements the fiber scheduler interface and event loop.
+- A {ruby Fiber} is an object which executes user code with cooperative concurrency, i.e. you can transfer execution from one fiber to another and back again.
+
+### What is a scheduler?
+
+A scheduler is an interface which manages the execution of fibers. It is responsible for intercepting blocking operations and redirecting them to an event loop.
+
+### What is an event loop?
+
+An event loop is part of the implementation of a scheduler which is responsible for waiting for events to occur, and waking up fibers when they are ready to run.
+
+### What is a selector?
+
+A selector is part of the implementation of an event loop which is responsible for interacting with the operating system and waiting for specific events to occur. This is often referred to as "select"ing ready events from a set of file descriptors, but in practice has expanded to encompass a wide range of blocking operations.
+
+### What is a reactor?
+
+A reactor is a specific implementation of the scheduler interface, which includes an event loop and selector, and is responsible for managing the execution of fibers.
+
+## Creating an Asynchronous Task
+
+The main entry point for creating tasks is the {ruby Kernel#Async} method. Because this method is defined on `Kernel`, it's available in all parts of your program.
+
+~~~ ruby
+require 'async'
+
+Async do |task|
+	puts "Hello World!"
+end
+~~~
+
+A {ruby Async::Task} runs using a {ruby Fiber} and blocking operations e.g. `sleep`, `read`, `write` yield control until the operation can complete. When a blocking operation yields control, it means another fiber can execute, giving the illusion of simultaneous execution.
+
+### When should I use `Async`?
+
+You should use `Async` when you desire explicit concurrency in your program. That means you want to run multiple tasks at the same time, and you want to be able to wait for the results of those tasks.
+
+- You should use `Async` when you want to perform network operations concurrently, such as HTTP requests or database queries.
+- You should use `Async` when you want to process independent requests concurrently, such as a web server.
+- You should use `Async` when you want to handle multiple connections concurrently, such as a chat server.
+
+You should consider the boundary around your program and the request handling. For example, one task per operation, request or connection, is usually appropriate.
+
+### Waiting for Results
+
+Similar to a promise, {ruby Async::Task} produces results. In order to wait for these results, you must invoke {ruby Async::Task#wait}:
+
+``` ruby
+require 'async'
+
+task = Async do
+	rand
+end
+
+puts "The number was: #{task.wait}"
+```
+
+## Creating a Fiber Scheduler
+
+The first (top level) async block will also create an instance of {ruby Async::Reactor} which is a subclass of {ruby Async::Scheduler} to handle the event loop. You can also do this directly using {ruby Fiber.set_scheduler}:
+
+~~~ ruby
+require 'async/scheduler'
+
+scheduler = Async::Scheduler.new
+Fiber.set_scheduler(scheduler)
+
+Fiber.schedule do
+	1.upto(3) do |i|
+		Fiber.schedule do
+			sleep 1
+			puts "Hello World"
+		end
+	end
+end
+~~~
+
+## Synchronous Execution in an existing Fiber Scheduler
+
+Unless you need fan-out, map-reduce style concurrency, you can actually use a slightly more efficient {ruby Kernel::Sync} execution model. This method will run your block in the current event loop if one exists, or create an event loop if not. You can use it for code which uses asynchronous primitives, but itself does not need to be asynchronous with respect to other tasks.
+
+```ruby
+require 'async/http/internet'
+
+def fetch(url)
+	Sync do
+		internet = Async::HTTP::Internet.new
+		return internet.get(url).read
+	end
+end
+
+# At the level of your program, this method will create an event loop:
+fetch(...)
+
+Sync do
+	# The event loop already exists, and will be reused:
+	fetch(...)
+end
+```
+
+In other words, `Sync{...}` is very similar in behaviour to `Async{...}.wait`, but significantly more efficient.
+
+## Enforcing Embedded Execution
+
+In some methods, you may want to implement a fan-out or map-reduce. That requires a parent scheduler. There are two ways you can do this:
+
+```ruby
+def fetch_all(urls, parent: Async::Task.current)
+	urls.map do |url|
+		parent.async do
+			fetch(url)
+		end
+	end.map(&:wait)
+end
+```
+
+or:
+
+```ruby
+def fetch_all(urls)
+	Sync do |parent|
+		urls.map do |url|
+			parent.async do
+				fetch(url)
+			end
+		end.map(&:wait)
+	end
+end
+```
+
+The former allows you to inject the parent, which could be a barrier or semaphore, while the latter will create a new parent scheduler if one does not exist. In both cases, you guarantee that the map operation will be executed in the parent task (of some sort).
+
+## Compatibility
+
+The Fiber Scheduler interface is compatible with most pure Ruby code and well-behaved C code. For example, you can use {ruby Net::HTTP} for performing concurrent HTTP requests:
+
+```ruby
+urls = [...]
+
+Async do
+	# Perform several concurrent requests:
+	responses = urls.map do |url|
+		Async do
+			Net::HTTP.get(url)
+		end
+	end.map(&:wait)
+end
+```
+
+Unfortunately, some libraries do not integrate well with the fiber scheduler: either they are blocking, processor bound, or use thread locals for execution state. To use these libraries, you may be able to use a background thread.
+
+```ruby
+Async do
+	result = Thread.new do
+		# Code which is otherwise unsafe...
+	end.value # Wait for the result of the thread, internally non-blocking.
+end
+```

data/context/index.yaml

@@ -0,0 +1,29 @@
+# Automatically generated context index for Utopia::Project guides.
+# Do not edit then files in this directory directly, instead edit the guides and then run `bake utopia:project:agent:context:update`.
+---
+getting-started:
+  title: Getting Started
+  order: 1
+  description: This guide shows how to add async to your project and run code asynchronously.
+scheduler:
+  title: Scheduler
+  order: 2
+  description: This guide gives an overview of how the scheduler is implemented.
+tasks:
+  title: Asynchronous Tasks
+  order: 3
+  description: This guide explains how asynchronous tasks work and how to use them.
+best-practices:
+  title: Best Practices
+  order: 5
+  description: This guide gives an overview of best practices for using Async.
+debugging:
+  title: Debugging
+  order: 6
+  description: This guide explains how to debug issues with programs that use Async.
+thread-safety:
+  title: Thread safety
+  order: 10
+  description: This guide explains thread safety in Ruby, focusing on fibers and threads,
+    common pitfalls, and best practices to avoid problems like data corruption, race
+    conditions, and deadlocks.

data/context/scheduler.md

@@ -0,0 +1,109 @@
+# Scheduler
+
+This guide gives an overview of how the scheduler is implemented.
+
+## Overview
+
+The {ruby Async::Scheduler} uses an event loop to execute tasks. When tasks are waiting on blocking operations like IO, the scheduler will use the operating system's native event system to wait for the operation to complete. This allows the scheduler to efficiently handle many tasks.
+
+### Tasks
+
+Tasks are the building blocks of concurrent programs. They are lightweight and can be scheduled by the event loop. Tasks can be nested, and the parent task is used to determine the current reactor. Tasks behave like promises, in the sense you can wait on them to complete, and they might fail with an exception.
+
+~~~ ruby
+require 'async'
+
+def sleepy(duration, task: Async::Task.current)
+	task.async do |subtask|
+		subtask.annotate "I'm going to sleep #{duration}s..."
+		sleep duration
+		puts "I'm done sleeping!"
+	end
+end
+
+def nested_sleepy(task: Async::Task.current)
+	task.async do |subtask|
+		subtask.annotate "Invoking sleepy 5 times..."
+		5.times do |index|
+			sleepy(index, task: subtask)
+		end
+	end
+end
+
+Async do |task|
+	task.annotate "Invoking nested_sleepy..."
+	subtask = nested_sleepy
+	
+	# Print out all running tasks in a tree:
+	task.print_hierarchy($stderr)
+	
+	# Kill the subtask
+	subtask.stop
+end
+~~~
+
+### Thread Safety
+
+Most methods of the reactor and related tasks are not thread-safe, so you'd typically have [one reactor per thread or process](https://github.com/socketry/async-container).
+
+### Embedding Schedulers
+
+{ruby Async::Scheduler#run} will run until the reactor runs out of work to do. To run a single iteration of the reactor, use {ruby Async::Scheduler#run_once}.
+
+~~~ ruby
+require 'async'
+
+Console.logger.debug!
+reactor = Async::Scheduler.new
+
+# Run the reactor for 1 second:
+reactor.async do |task|
+	sleep 1
+	puts "Finished!"
+end
+
+while reactor.run_once
+	# Round and round we go!
+end
+~~~
+
+You can use this approach to embed the reactor in another event loop. For some integrations, you may want to specify the maximum time to wait to {ruby Async::Scheduler#run_once}.
+
+### Stopping a Scheduler
+
+{ruby Async::Scheduler#stop} will stop the current scheduler and all children tasks.
+
+### Fiber Scheduler Integration
+
+In order to integrate with native Ruby blocking operations, the {ruby Async::Scheduler} uses a {ruby Fiber::Scheduler} interface.
+
+```ruby
+require 'async'
+
+scheduler = Async::Scheduler.new
+Fiber.set_scheduler(scheduler)
+
+Fiber.schedule do
+	puts "Hello World!"
+end
+```
+
+## Design
+
+### Optimistic vs Pessimistic Scheduling
+
+There are two main strategies for scheduling tasks: optimistic and pessimistic. An optimistic scheduler is usually greedy and will try to execute tasks as soon as they are scheduled using a direct transfer of control flow. A pessimistic scheduler will schedule tasks into the event loop ready list and will only execute them on the next iteration of the event loop.
+
+```ruby
+Async do
+	puts "Hello "
+	
+	Async do
+		puts "World"
+	end
+	
+	puts "!"
+end
+```
+
+An optimstic scheduler will print "Hello World!", while a pessimistic scheduler will print "Hello !World". In practice you should not design your code to rely on the order of execution, but it's important to understand the difference. It is an unspecifed implementation detail of the scheduler.