wasmtime 0.4.1 → 5.0.0

data/ext/cargo-vendor/tokio-1.24.2/docs/reactor-refactor.md

@@ -0,0 +1,276 @@
+# Refactor I/O driver
+
+Describes changes to the I/O driver for the Tokio 0.3 release.
+
+## Goals
+
+* Support `async fn` on I/O types with `&self`.
+* Refine the `Registration` API.
+
+### Non-goals
+
+* Implement `AsyncRead` / `AsyncWrite` for `&TcpStream` or other reference type.
+
+## Overview
+
+Currently, I/O types require `&mut self` for `async` functions. The reason for
+this is the task's waker is stored in the I/O resource's internal state
+(`ScheduledIo`) instead of in the future returned by the `async` function.
+Because of this limitation, I/O types limit the number of wakers to one per
+direction (a direction is either read-related events or write-related events).
+
+Moving the waker from the internal I/O resource's state to the operation's
+future enables multiple wakers to be registered per operation. The "intrusive
+wake list" strategy used by `Notify` applies to this case, though there are some
+concerns unique to the I/O driver.
+
+## Reworking the `Registration` type
+
+While `Registration` is made private (per #2728), it remains in Tokio as an
+implementation detail backing I/O resources such as `TcpStream`. The API of
+`Registration` is updated to support waiting for an arbitrary interest set with
+`&self`. This supports concurrent waiters with a different readiness interest.
+
+```rust
+struct Registration { ... }
+
+// TODO: naming
+struct ReadyEvent {
+    tick: u32,
+    ready: mio::Ready,
+}
+
+impl Registration {
+    /// `interest` must be a super set of **all** interest sets specified in
+    /// the other methods. This is the interest set passed to `mio`.
+    pub fn new<T>(io: &T, interest: mio::Ready) -> io::Result<Registration>
+        where T: mio::Evented;
+
+    /// Awaits for any readiness event included in `interest`. Returns a
+    /// `ReadyEvent` representing the received readiness event.
+    async fn readiness(&self, interest: mio::Ready) -> io::Result<ReadyEvent>;
+
+    /// Clears resource level readiness represented by the specified `ReadyEvent`
+    async fn clear_readiness(&self, ready_event: ReadyEvent);
+```
+
+A new registration is created for a `T: mio::Evented` and a `interest`. This
+creates a `ScheduledIo` entry with the I/O driver and registers the resource
+with `mio`.
+
+Because Tokio uses **edge-triggered** notifications, the I/O driver only
+receives readiness from the OS once the ready state **changes**. The I/O driver
+must track each resource's known readiness state. This helps prevent syscalls
+when the process knows the syscall should return with `EWOULDBLOCK`.
+
+A call to `readiness()` checks if the currently known resource readiness
+overlaps with `interest`. If it does, then the `readiness()` immediately
+returns. If it does not, then the task waits until the I/O driver receives a
+readiness event.
+
+The pseudocode to perform a TCP read is as follows.
+
+```rust
+async fn read(&self, buf: &mut [u8]) -> io::Result<usize> {
+    loop {
+        // Await readiness
+        let event = self.readiness(interest).await?;
+
+        match self.mio_socket.read(buf) {
+            Ok(v) => return Ok(v),
+            Err(ref e) if e.kind() == WouldBlock => {
+                self.clear_readiness(event);
+            }
+            Err(e) => return Err(e),
+        }
+    }
+}
+```
+
+## Reworking the `ScheduledIo` type
+
+The `ScheduledIo` type is switched to use an intrusive waker linked list. Each
+entry in the linked list includes the `interest` set passed to `readiness()`.
+
+```rust
+#[derive(Debug)]
+pub(crate) struct ScheduledIo {
+    /// Resource's known state packed with other state that must be
+    /// atomically updated.
+    readiness: AtomicUsize,
+
+    /// Tracks tasks waiting on the resource
+    waiters: Mutex<Waiters>,
+}
+
+#[derive(Debug)]
+struct Waiters {
+    // List of intrusive waiters.
+    list: LinkedList<Waiter>,
+
+    /// Waiter used by `AsyncRead` implementations.
+    reader: Option<Waker>,
+
+    /// Waiter used by `AsyncWrite` implementations.
+    writer: Option<Waker>,
+}
+
+// This struct is contained by the **future** returned by `readiness()`.
+#[derive(Debug)]
+struct Waiter {
+    /// Intrusive linked-list pointers
+    pointers: linked_list::Pointers<Waiter>,
+
+    /// Waker for task waiting on I/O resource
+    waiter: Option<Waker>,
+
+    /// Readiness events being waited on. This is
+    /// the value passed to `readiness()`
+    interest: mio::Ready,
+
+    /// Should not be `Unpin`.
+    _p: PhantomPinned,
+}
+```
+
+When an I/O event is received from `mio`, the associated resources' readiness is
+updated and the waiter list is iterated. All waiters with `interest` that
+overlap the received readiness event are notified. Any waiter with an `interest`
+that does not overlap the readiness event remains in the list.
+
+## Cancel interest on drop
+
+The future returned by `readiness()` uses an intrusive linked list to store the
+waker with `ScheduledIo`. Because `readiness()` can be called concurrently, many
+wakers may be stored simultaneously in the list. If the `readiness()` future is
+dropped early, it is essential that the waker is removed from the list. This
+prevents leaking memory.
+
+## Race condition
+
+Consider how many tasks may concurrently attempt I/O operations. This, combined
+with how Tokio uses edge-triggered events, can result in a race condition. Let's
+revisit the TCP read function:
+
+```rust
+async fn read(&self, buf: &mut [u8]) -> io::Result<usize> {
+    loop {
+        // Await readiness
+        let event = self.readiness(interest).await?;
+
+        match self.mio_socket.read(buf) {
+            Ok(v) => return Ok(v),
+            Err(ref e) if e.kind() == WouldBlock => {
+                self.clear_readiness(event);
+            }
+            Err(e) => return Err(e),
+        }
+    }
+}
+```
+
+If care is not taken, if between `mio_socket.read(buf)` returning and
+`clear_readiness(event)` is called, a readiness event arrives, the `read()`
+function could deadlock. This happens because the readiness event is received,
+`clear_readiness()` unsets the readiness event, and on the next iteration,
+`readiness().await` will block forever as a new readiness event is not received.
+
+The current I/O driver handles this condition by always registering the task's
+waker before performing the operation. This is not ideal as it will result in
+unnecessary task notification.
+
+Instead, we will use a strategy to prevent clearing readiness if an "unseen"
+readiness event has been received. The I/O driver will maintain a "tick" value.
+Every time the `mio` `poll()` function is called, the tick is incremented. Each
+readiness event has an associated tick. When the I/O driver sets the resource's
+readiness, the driver's tick is packed into the atomic `usize`.
+
+The `ScheduledIo` readiness `AtomicUsize` is structured as:
+
+```
+| shutdown | generation |  driver tick | readiness |
+|----------+------------+--------------+-----------|
+|   1 bit  |   7 bits   +    8 bits    +  16 bits  |
+```
+
+The `shutdown` and `generation` components exist today.
+
+The `readiness()` function returns a `ReadyEvent` value. This value includes the
+`tick` component read with the resource's readiness value. When
+`clear_readiness()` is called, the `ReadyEvent` is provided. Readiness is only
+cleared if the current `tick` matches the `tick` included in the `ReadyEvent`.
+If the tick values do not match, the call to `readiness()` on the next iteration
+will not block and the new `tick` is included in the new `ReadyToken.`
+
+TODO
+
+## Implementing `AsyncRead` / `AsyncWrite`
+
+The `AsyncRead` and `AsyncWrite` traits use a "poll" based API. This means that
+it is not possible to use an intrusive linked list to track the waker.
+Additionally, there is no future associated with the operation which means it is
+not possible to cancel interest in the readiness events.
+
+To implement `AsyncRead` and `AsyncWrite`, `ScheduledIo` includes dedicated
+waker values for the read direction and the write direction. These values are
+used to store the waker. Specific `interest` is not tracked for `AsyncRead` and
+`AsyncWrite` implementations. It is assumed that only events of interest are:
+
+* Read ready
+* Read closed
+* Write ready
+* Write closed
+
+Note that "read closed" and "write closed" are only available with Mio 0.7. With
+Mio 0.6, things were a bit messy.
+
+It is only possible to implement `AsyncRead` and `AsyncWrite` for resource types
+themselves and not for `&Resource`. Implementing the traits for `&Resource`
+would permit concurrent operations to the resource. Because only a single waker
+is stored per direction, any concurrent usage would result in deadlocks. An
+alternate implementation would call for a `Vec<Waker>` but this would result in
+memory leaks.
+
+## Enabling reads and writes for `&TcpStream`
+
+Instead of implementing `AsyncRead` and `AsyncWrite` for `&TcpStream`, a new
+function is added to `TcpStream`.
+
+```rust
+impl TcpStream {
+    /// Naming TBD
+    fn by_ref(&self) -> TcpStreamRef<'_>;
+}
+
+struct TcpStreamRef<'a> {
+    stream: &'a TcpStream,
+
+    // `Waiter` is the node in the intrusive waiter linked-list
+    read_waiter: Waiter,
+    write_waiter: Waiter,
+}
+```
+
+Now, `AsyncRead` and `AsyncWrite` can be implemented on `TcpStreamRef<'a>`. When
+the `TcpStreamRef` is dropped, all associated waker resources are cleaned up.
+
+### Removing all the `split()` functions
+
+With `TcpStream::by_ref()`, `TcpStream::split()` is no longer needed. Instead,
+it is possible to do something as follows.
+
+```rust
+let rd = my_stream.by_ref();
+let wr = my_stream.by_ref();
+
+select! {
+    // use `rd` and `wr` in separate branches.
+}
+```
+
+It is also possible to store a `TcpStream` in an `Arc`.
+
+```rust
+let arc_stream = Arc::new(my_tcp_stream);
+let n = arc_stream.by_ref().read(buf).await?;
+```

data/ext/cargo-vendor/tokio-1.24.2/src/fs/read_dir.rs

@@ -0,0 +1,350 @@
+use crate::fs::asyncify;
+
+use std::collections::VecDeque;
+use std::ffi::OsString;
+use std::fs::{FileType, Metadata};
+use std::future::Future;
+use std::io;
+use std::path::{Path, PathBuf};
+use std::pin::Pin;
+use std::sync::Arc;
+use std::task::Context;
+use std::task::Poll;
+
+#[cfg(test)]
+use super::mocks::spawn_blocking;
+#[cfg(test)]
+use super::mocks::JoinHandle;
+#[cfg(not(test))]
+use crate::blocking::spawn_blocking;
+#[cfg(not(test))]
+use crate::blocking::JoinHandle;
+
+const CHUNK_SIZE: usize = 32;
+
+/// Returns a stream over the entries within a directory.
+///
+/// This is an async version of [`std::fs::read_dir`](std::fs::read_dir)
+///
+/// This operation is implemented by running the equivalent blocking
+/// operation on a separate thread pool using [`spawn_blocking`].
+///
+/// [`spawn_blocking`]: crate::task::spawn_blocking
+pub async fn read_dir(path: impl AsRef<Path>) -> io::Result<ReadDir> {
+    let path = path.as_ref().to_owned();
+    asyncify(|| -> io::Result<ReadDir> {
+        let mut std = std::fs::read_dir(path)?;
+        let mut buf = VecDeque::with_capacity(CHUNK_SIZE);
+        ReadDir::next_chunk(&mut buf, &mut std);
+
+        Ok(ReadDir(State::Idle(Some((buf, std)))))
+    })
+    .await
+}
+
+/// Reads the entries in a directory.
+///
+/// This struct is returned from the [`read_dir`] function of this module and
+/// will yield instances of [`DirEntry`]. Through a [`DirEntry`] information
+/// like the entry's path and possibly other metadata can be learned.
+///
+/// A `ReadDir` can be turned into a `Stream` with [`ReadDirStream`].
+///
+/// [`ReadDirStream`]: https://docs.rs/tokio-stream/0.1/tokio_stream/wrappers/struct.ReadDirStream.html
+///
+/// # Errors
+///
+/// This stream will return an [`Err`] if there's some sort of intermittent
+/// IO error during iteration.
+///
+/// [`read_dir`]: read_dir
+/// [`DirEntry`]: DirEntry
+/// [`Err`]: std::result::Result::Err
+#[derive(Debug)]
+#[must_use = "streams do nothing unless polled"]
+pub struct ReadDir(State);
+
+#[derive(Debug)]
+enum State {
+    Idle(Option<(VecDeque<io::Result<DirEntry>>, std::fs::ReadDir)>),
+    Pending(JoinHandle<(VecDeque<io::Result<DirEntry>>, std::fs::ReadDir)>),
+}
+
+impl ReadDir {
+    /// Returns the next entry in the directory stream.
+    ///
+    /// # Cancel safety
+    ///
+    /// This method is cancellation safe.
+    pub async fn next_entry(&mut self) -> io::Result<Option<DirEntry>> {
+        use crate::future::poll_fn;
+        poll_fn(|cx| self.poll_next_entry(cx)).await
+    }
+
+    /// Polls for the next directory entry in the stream.
+    ///
+    /// This method returns:
+    ///
+    ///  * `Poll::Pending` if the next directory entry is not yet available.
+    ///  * `Poll::Ready(Ok(Some(entry)))` if the next directory entry is available.
+    ///  * `Poll::Ready(Ok(None))` if there are no more directory entries in this
+    ///    stream.
+    ///  * `Poll::Ready(Err(err))` if an IO error occurred while reading the next
+    ///    directory entry.
+    ///
+    /// When the method returns `Poll::Pending`, the `Waker` in the provided
+    /// `Context` is scheduled to receive a wakeup when the next directory entry
+    /// becomes available on the underlying IO resource.
+    ///
+    /// Note that on multiple calls to `poll_next_entry`, only the `Waker` from
+    /// the `Context` passed to the most recent call is scheduled to receive a
+    /// wakeup.
+    pub fn poll_next_entry(&mut self, cx: &mut Context<'_>) -> Poll<io::Result<Option<DirEntry>>> {
+        loop {
+            match self.0 {
+                State::Idle(ref mut data) => {
+                    let (buf, _) = data.as_mut().unwrap();
+
+                    if let Some(ent) = buf.pop_front() {
+                        return Poll::Ready(ent.map(Some));
+                    };
+
+                    let (mut buf, mut std) = data.take().unwrap();
+
+                    self.0 = State::Pending(spawn_blocking(move || {
+                        ReadDir::next_chunk(&mut buf, &mut std);
+                        (buf, std)
+                    }));
+                }
+                State::Pending(ref mut rx) => {
+                    let (mut buf, std) = ready!(Pin::new(rx).poll(cx))?;
+
+                    let ret = match buf.pop_front() {
+                        Some(Ok(x)) => Ok(Some(x)),
+                        Some(Err(e)) => Err(e),
+                        None => Ok(None),
+                    };
+
+                    self.0 = State::Idle(Some((buf, std)));
+
+                    return Poll::Ready(ret);
+                }
+            }
+        }
+    }
+
+    fn next_chunk(buf: &mut VecDeque<io::Result<DirEntry>>, std: &mut std::fs::ReadDir) {
+        for ret in std.by_ref().take(CHUNK_SIZE) {
+            let success = ret.is_ok();
+
+            buf.push_back(ret.map(|std| DirEntry {
+                #[cfg(not(any(
+                    target_os = "solaris",
+                    target_os = "illumos",
+                    target_os = "haiku",
+                    target_os = "vxworks"
+                )))]
+                file_type: std.file_type().ok(),
+                std: Arc::new(std),
+            }));
+
+            if !success {
+                break;
+            }
+        }
+    }
+}
+
+feature! {
+    #![unix]
+
+    use std::os::unix::fs::DirEntryExt;
+
+    impl DirEntry {
+        /// Returns the underlying `d_ino` field in the contained `dirent`
+        /// structure.
+        ///
+        /// # Examples
+        ///
+        /// ```
+        /// use tokio::fs;
+        ///
+        /// # #[tokio::main]
+        /// # async fn main() -> std::io::Result<()> {
+        /// let mut entries = fs::read_dir(".").await?;
+        /// while let Some(entry) = entries.next_entry().await? {
+        ///     // Here, `entry` is a `DirEntry`.
+        ///     println!("{:?}: {}", entry.file_name(), entry.ino());
+        /// }
+        /// # Ok(())
+        /// # }
+        /// ```
+        pub fn ino(&self) -> u64 {
+            self.as_inner().ino()
+        }
+    }
+}
+
+/// Entries returned by the [`ReadDir`] stream.
+///
+/// [`ReadDir`]: struct@ReadDir
+///
+/// This is a specialized version of [`std::fs::DirEntry`] for usage from the
+/// Tokio runtime.
+///
+/// An instance of `DirEntry` represents an entry inside of a directory on the
+/// filesystem. Each entry can be inspected via methods to learn about the full
+/// path or possibly other metadata through per-platform extension traits.
+#[derive(Debug)]
+pub struct DirEntry {
+    #[cfg(not(any(
+        target_os = "solaris",
+        target_os = "illumos",
+        target_os = "haiku",
+        target_os = "vxworks"
+    )))]
+    file_type: Option<FileType>,
+    std: Arc<std::fs::DirEntry>,
+}
+
+impl DirEntry {
+    /// Returns the full path to the file that this entry represents.
+    ///
+    /// The full path is created by joining the original path to `read_dir`
+    /// with the filename of this entry.
+    ///
+    /// # Examples
+    ///
+    /// ```no_run
+    /// use tokio::fs;
+    ///
+    /// # async fn dox() -> std::io::Result<()> {
+    /// let mut entries = fs::read_dir(".").await?;
+    ///
+    /// while let Some(entry) = entries.next_entry().await? {
+    ///     println!("{:?}", entry.path());
+    /// }
+    /// # Ok(())
+    /// # }
+    /// ```
+    ///
+    /// This prints output like:
+    ///
+    /// ```text
+    /// "./whatever.txt"
+    /// "./foo.html"
+    /// "./hello_world.rs"
+    /// ```
+    ///
+    /// The exact text, of course, depends on what files you have in `.`.
+    pub fn path(&self) -> PathBuf {
+        self.std.path()
+    }
+
+    /// Returns the bare file name of this directory entry without any other
+    /// leading path component.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use tokio::fs;
+    ///
+    /// # async fn dox() -> std::io::Result<()> {
+    /// let mut entries = fs::read_dir(".").await?;
+    ///
+    /// while let Some(entry) = entries.next_entry().await? {
+    ///     println!("{:?}", entry.file_name());
+    /// }
+    /// # Ok(())
+    /// # }
+    /// ```
+    pub fn file_name(&self) -> OsString {
+        self.std.file_name()
+    }
+
+    /// Returns the metadata for the file that this entry points at.
+    ///
+    /// This function will not traverse symlinks if this entry points at a
+    /// symlink.
+    ///
+    /// # Platform-specific behavior
+    ///
+    /// On Windows this function is cheap to call (no extra system calls
+    /// needed), but on Unix platforms this function is the equivalent of
+    /// calling `symlink_metadata` on the path.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use tokio::fs;
+    ///
+    /// # async fn dox() -> std::io::Result<()> {
+    /// let mut entries = fs::read_dir(".").await?;
+    ///
+    /// while let Some(entry) = entries.next_entry().await? {
+    ///     if let Ok(metadata) = entry.metadata().await {
+    ///         // Now let's show our entry's permissions!
+    ///         println!("{:?}: {:?}", entry.path(), metadata.permissions());
+    ///     } else {
+    ///         println!("Couldn't get file type for {:?}", entry.path());
+    ///     }
+    /// }
+    /// # Ok(())
+    /// # }
+    /// ```
+    pub async fn metadata(&self) -> io::Result<Metadata> {
+        let std = self.std.clone();
+        asyncify(move || std.metadata()).await
+    }
+
+    /// Returns the file type for the file that this entry points at.
+    ///
+    /// This function will not traverse symlinks if this entry points at a
+    /// symlink.
+    ///
+    /// # Platform-specific behavior
+    ///
+    /// On Windows and most Unix platforms this function is free (no extra
+    /// system calls needed), but some Unix platforms may require the equivalent
+    /// call to `symlink_metadata` to learn about the target file type.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use tokio::fs;
+    ///
+    /// # async fn dox() -> std::io::Result<()> {
+    /// let mut entries = fs::read_dir(".").await?;
+    ///
+    /// while let Some(entry) = entries.next_entry().await? {
+    ///     if let Ok(file_type) = entry.file_type().await {
+    ///         // Now let's show our entry's file type!
+    ///         println!("{:?}: {:?}", entry.path(), file_type);
+    ///     } else {
+    ///         println!("Couldn't get file type for {:?}", entry.path());
+    ///     }
+    /// }
+    /// # Ok(())
+    /// # }
+    /// ```
+    pub async fn file_type(&self) -> io::Result<FileType> {
+        #[cfg(not(any(
+            target_os = "solaris",
+            target_os = "illumos",
+            target_os = "haiku",
+            target_os = "vxworks"
+        )))]
+        if let Some(file_type) = self.file_type {
+            return Ok(file_type);
+        }
+
+        let std = self.std.clone();
+        asyncify(move || std.file_type()).await
+    }
+
+    /// Returns a reference to the underlying `std::fs::DirEntry`.
+    #[cfg(unix)]
+    pub(super) fn as_inner(&self) -> &std::fs::DirEntry {
+        &self.std
+    }
+}