opendal 0.1.6.pre.rc.1-aarch64-linux

data/core/src/docs/rfcs/4382_range_based_read.md

@@ -0,0 +1,213 @@
+- Proposal Name: `range_based_read`
+- Start Date: 2024-03-20
+- RFC PR: [apache/opendal#4382](https://github.com/apache/opendal/pull/4382)
+- Tracking Issue: [apache/opendal#4383](https://github.com/apache/opendal/issues/4383)
+
+# Summary
+
+Convert `oio::Read` into a stateless, range-based reading pattern.
+
+# Motivation
+
+The current `oio::Read` API is stateful:
+
+```rust
+pub trait Read: Unpin + Send + Sync {
+    fn read(&mut self, limit: usize) -> impl Future<Output = Result<Bytes>> + Send;
+    fn seek(&mut self, pos: io::SeekFrom) -> impl Future<Output = Result<u64>> + Send;
+}
+```
+
+Users use `read` to retrieve data from storage and can use `seek` to navigate to specific positions. OpenDAL manages the underlying state. This design is good for users from `std::io::Read`, `futures::AsyncRead` and `tokio::io::AsyncRead`.
+
+OpenDAL also provides `range` option at the `Operator` level for users to read a specific range of data. The most common usage will be like:
+
+```rust
+let r: Reader = op.reader_with(path).range(1024..2048).await?;
+```
+
+However, after observing our users, we found that:
+
+- `AsyncSeek` in `Reader` is prone to misuse.
+- `Reader` does not support concurrent reading.
+- `Reader` can't adopt Completion-based IO
+
+## Misuse of `AsyncSeek`
+
+When designing `Reader`, I expected users to check the `read_can_seek` capability to determine if the underlying storage services natively support `seek`. However, many users are unaware of this and directly use `seek`, leading to suboptimal performance.
+
+For example, `s3` storage does not support `seek` natively. When users call `seek`, opendal will drop current reader and sending a new request. This behavior is hidden from users and can lead to unexpected performance issues like [What's going on in my parquet stream](https://github.com/apache/opendal/issues/3725).
+
+## Lack of concurrent reading
+
+`oio::Read` complicates supporting concurrent reading. Users must implement a feature similar to merge IO, as discussed in [support merge io read api by settings](https://github.com/apache/opendal/issues/3675).
+
+There is no way for opendal to support this feature.
+
+## Can't adopt Completion-based IO
+
+Completion-based IO requires take the buffer's owner ship. But API that take `&mut [u8]` can't do that.
+
+# Guide-level explanation
+
+So I propose to convert `Reader` into a stateless, range-based reading pattern.
+
+We will remove the following `impl` from `Reader`:
+
+- `futures::AsyncRead`
+- `futures::AsyncSeek`
+- `futures::Stream`
+- `tokio::AsyncRead`
+- `tokio::AsyncSeek`
+
+We will add the following new APIs to `Reader`:
+
+```rust
+impl Reader {
+    /// Read data from the storage at the specified offset.
+     pub async fn read(&self, buf: &mut impl BufMut, offset: u64, limit: usize) -> Result<usize>;
+
+    /// Read data from the storage at the specified range.
+    pub async fn read_range(
+        &self,
+        buf: &mut impl BufMut,
+        range: impl RangeBounds<u64>,
+    ) -> Result<usize>;
+
+    /// Read all data from the storage into given buf.
+    pub async fn read_to_end(&self, buf: &mut impl BufMut) -> Result<usize>;
+
+    /// Copy data from the storage into given writer.
+    pub async fn copy(&mut self, write_into: &mut impl futures::AsyncWrite) -> Result<u64>;
+
+    /// Sink date from the storage into given sink.
+    pub async fn sink<S, T>(&mut self, sink_from: &mut S) -> Result<u64>
+    where
+        S: futures::Sink<T, Error = Error>,
+        T: Into<Bytes>,
+}
+```
+
+Apart from `Reader`'s own API, we will also provide convert to existing IO APIs like:
+
+```rust
+impl Reader {
+    /// Convert Reader into `futures::AsyncRead`  
+    pub fn into_futures_io_async_read(self, range: Range<u64>) -> FuturesIoAsyncReader;
+    
+    /// Convert Reader into `futures::Stream`
+    pub fn into_futures_bytes_stream(self, range: Range<u64>) -> FuturesBytesStream;
+}
+```
+
+After this change, users will be able to use `Reader` to read data from storage in a stateless, range-based pattern. Users can also convert `Reader` into `futures::AsyncRead`, `futures::AsyncSeek` and `futures::Stream` as needed.
+
+# Reference-level explanation
+
+The new raw API will be:
+
+```rust
+pub trait Read: Unpin + Send + Sync {
+    fn read_at(
+        &self,
+        offset: u64,
+        limit: usize,
+    ) -> impl Future<Output = Result<oio::Buffer>> + Send;
+}
+```
+
+The API is similar to [`ReadAt`](https://doc.rust-lang.org/std/fs/struct.File.html#method.read_at), but with following changes:
+
+```diff
+- async fn read_at(&self, buf: &mut [u8], offset: u64) -> Result<usize>
++ async fn read_at(&self, offset: u64, limit: usize) -> Result<oio::Buffer>
+```
+
+- opendal chooses to use `oio::Buffer` instead of `&mut [u8]` to avoid lifetime issues.
+- opendal chooses to return `oio::Buffer` to let services itself manage the buffer.
+
+For example, http based storage services like `s3` is a stream that generating data on the fly.
+
+# Drawbacks
+
+## Breaking changes to `Reader`
+
+This change will break the existing `Reader` API. Users will need to update their code to use the new `Reader` API.
+
+Users wishing to migrate to the new range-based API will need to update their code. Those who simply want to use `futures::AsyncRead` can instead utilize `Reader::into_futures_read`.
+  
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+## `object_store`'s API design  
+
+Current API design inspired from `object_store`'s `ObjectStore` a lot:
+    
+```rust
+#[async_trait]
+pub trait ObjectStore: std::fmt::Display + Send + Sync + Debug + 'static {
+    /// Return the bytes that are stored at the specified location.
+    async fn get(&self, location: &Path) -> Result<GetResult> {
+        self.get_opts(location, GetOptions::default()).await
+    }
+
+    /// Perform a get request with options
+    async fn get_opts(&self, location: &Path, options: GetOptions) -> Result<GetResult>;
+
+    /// Return the bytes that are stored at the specified location
+    /// in the given byte range.
+    ///
+    /// See [`GetRange::Bounded`] for more details on how `range` gets interpreted
+    async fn get_range(&self, location: &Path, range: Range<usize>) -> Result<Bytes> {
+        let options = GetOptions {
+            range: Some(range.into()),
+            ..Default::default()
+        };
+        self.get_opts(location, options).await?.bytes().await
+    }
+    
+    /// Return the bytes that are stored at the specified location
+    /// in the given byte ranges
+    async fn get_ranges(&self, location: &Path, ranges: &[Range<usize>]) -> Result<Vec<Bytes>> {
+        coalesce_ranges(
+            ranges,
+            |range| self.get_range(location, range),
+            OBJECT_STORE_COALESCE_DEFAULT,
+        )
+        .await
+    }
+}
+```
+
+We can add support that similar to `get_ranges` in the future.
+  
+OpenDAL opts to return a `Reader` rather than directly implementing `read` to allow for optimization with storage services like `fs` to reduce the extra `open` syscall. 
+
+# Unresolved questions
+
+## Buffer
+
+After switching to range-based reading, we can no longer keep a buffer within the reader. As of writing this proposal, users should use `into_async_buf_read` instead.
+
+# Future possibilities
+
+## Read Ranges
+
+We can implement `read_ranges` support in the future. This will allow users to read multiple ranges of data in less requests.
+
+## Native `read_at` for fs and hdfs
+
+We can reduce unnecessary `open` and `seek` syscalls by using the `read_at` API across different platforms.
+
+## Auto Range Read
+
+We can implement [Auto ranged read support](https://github.com/apache/opendal/issues/1105) like AWS S3 Crt Client. For examples, split the range into multiple ranges and read them concurrently. 
+
+Services can define the preferred io size as default, and users can override it. For example, s3 can use `8 MiB` as preferred io size, while fs can use `4 KiB` instead.
+
+## Completion-based IO
+
+`oio::Read` is designed with Completion-based IO in mind. We can add IOCP/io_uring support in the future.

data/core/src/docs/rfcs/4638_executor.md

@@ -0,0 +1,215 @@
+- Proposal Name: `executor`
+- Start Date: 2024-05-23
+- RFC PR: [apache/opendal#4638](https://github.com/apache/opendal/pull/4638)
+- Tracking Issue: [apache/opendal#4639](https://github.com/apache/opendal/issues/4639)
+
+# Summary
+
+Add executor in opendal to allow running tasks concurrently in background.
+
+# Motivation
+
+OpenDAL offers top-tier support for concurrent execution, allowing tasks to run simultaneously in the background. Users can easily enable concurrent file read/write operations with just one line of code:
+
+```diff
+ let mut w = op
+     .writer_with(path)
+     .chunk(8 * 1024 * 1024) // 8 MiB per chunk
++    .concurrent(16) // 16 concurrent tasks
+     .await?;
+     
+ w.write(bs).await?;
+ w.write(bs).await?; // The submitted tasks only be executed while user calling `write`.
+ ...
+ sleep(Duration::from_secs(10)).await; // The submitted tasks make no progress during `sleep`.
+ ...
+ w.close().await?;
+```
+
+However, the execution of those tasks relies on users continuously calling `write`. They cannot run tasks concurrently in the background. (I explained the technical details in the `Rationale and alternatives` section.)
+
+This can result in the following issues:
+
+- Task latency may increase as tasks are not executed until the task queue is full.
+- Memory usage may be high because all chunks must be held in memory until the task is completed.
+
+I propose introducing an executor abstraction in OpenDAL to enable concurrent background task execution. The executor will automatically manage the tasks in the background without requiring users to drive the progress manually.
+
+# Guide-level explanation
+
+OpenDAL will add a new `Executor` struct to manage concurrent tasks.
+
+```rust
+pub struct Executor {
+    ...
+}
+
+pub struct Task {
+    ...
+}
+
+impl Executor {
+    /// Create a new tokio based executor.
+    pub fn new() -> Self { ... }
+    
+    /// Create a new executor with given execute impl.
+    pub fn with(exec: Arc<dyn Execute>) -> Self { ... }
+
+    /// Run given future in background immediately.
+    pub fn execute<F>(&self, f: F) -> Task<F::Output>
+    where
+        F: Future + Send + 'static,
+    {
+        ...
+    }
+}
+```
+
+The `Executor` uses the `tokio` runtime by default but users can also provide their own runtime by:
+
+```rust
+pub trait Execute {
+    fn execute(&self, f: BoxedFuture<()>) -> Result<()>;
+}
+```
+
+Users can set executor in `OpWrite` / `OpRead` to enable concurrent background task execution:
+
+```rust
++ let exec = Executor::new();
+  let w = op
+      .writer_with(path)
+      .chunk(8 * 1024 * 1024) // 8 MiB per chunk
+      .concurrent(16) // 16 concurrent tasks
++     .executor(exec) // Use specified executor
+      .await?;
+```
+
+Specifying an executor every time is cumbersome. Users can also set a global executor for given operator:
+
+```rust
++ let exec = Executor::new();
++ let op = op.with_default_executor(exec);
+
+  let w = op
+      .writer_with(path)
+      .chunk(8 * 1024 * 1024) // 8 MiB per chunk
+      .concurrent(16) // 16 concurrent tasks
+      .await?;
+```
+
+# Reference-level explanation
+
+As mentioned in the `Guide-level explanation`, the `Executor` struct will manage concurrent tasks in the background. `Executor` will be powered by trait `Execute` to support different underlying runtimes. To make trait `Execute` object safe, we only accept `BoxedFuture<()>` as input. `Executor` will handle the future output and return the result to the caller.
+
+Operations that supporting concurrent execution will add a new field:
+
+```rust
+pub struct OpXxx {
+    ...
+    executor: Option<Executor>,
+}
+```
+
+Operator will add a new field to store the default executor:
+
+```rust
+pub struct Operator {
+    ...
+    default_executor: Option<Executor>,
+}
+```
+
+The `Task` spawned by `Executor` will be a future that can be awaited to fetch the result:
+
+```rust
+let res = task.await;
+```
+
+The task will be executed immediately after calling `execute`. Users can also cancel the task by dropping the `Task` object. Users don't need to poll those `Task` object to make progress.
+
+# Drawbacks
+
+## Complexity
+
+To support concurrent execution, we need to introduce:
+
+- a new `Executor` struct 
+- a new `Task` struct
+- a new `Execute` trait
+
+This may increase the complexity of the codebase.
+
+# Rationale and alternatives
+
+## Why introducing so many new abstractions?
+
+We need to introduce new abstractions to support concurrent execution across different runtimes. Unfortunately, this is the current reality of async rust.
+
+Supporting just one or two runtimes by adding features is much easier. Supporting only Tokio is extremely simple, requiring about 10 lines of changes. However, this violates our vision of free data access.
+
+Firstly, we don't want to force our users to use Tokio. We aim to support all runtimes, including async-std, smol, and others.
+
+Secondly, OpenDAL should be capable of running in any environment, including embedded systems. We don’t want to restrict our users to a specific runtime.
+
+Finally, users may have their own preferences for observability and performance in their runtime. We intend to accommodate these needs effortlessly.
+
+## Why `ConcurrentFutures` doesn't work?
+
+`ConcurrentFutures` is a `Vec<impl Future>`, users need to keep calling `poll_next` to make progress. This is not suitable for our use case. We need a way to run tasks in the background without user intervention.
+
+> I've heard that futures will wake up when they're ready, and it's the runtime's job to poll them, right?
+
+No, it's partially correct. The runtime will wake up the future when it's ready, but it's the user's job to poll the future. The runtime will not poll the future automatically unless it's managed by the runtime.
+
+For tokio, that means all futures provided by tokio, like `tokio::time::Sleep`, will be polled by tokio runtime. However, if you create a future by yourself, you need to poll it manually.
+
+I have an example to explain this:
+
+*Try it at [playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=628e67adef90128151e175d22c87808e)*
+
+```rust
+use futures::stream::FuturesUnordered;
+use futures::StreamExt;
+use std::time::Duration;
+use tokio::time::{sleep, Instant};
+
+#[tokio::main]
+async fn main() {
+    let now = Instant::now();
+    let mut cf = FuturesUnordered::new();
+
+    // cf.push(Box::pin(sleep(Duration::from_secs(3))));
+    cf.push(Box::pin(async move {
+        sleep(Duration::from_secs(3)).await;
+        println!("async task finished at {}s", now.elapsed().as_secs_f64());
+    }));
+    sleep(Duration::from_secs(4)).await;
+    println!("outer sleep finished at {}s", now.elapsed().as_secs_f64());
+
+    let _: Vec<()> = cf.collect().await;
+    println!("consumed: {}s", now.elapsed().as_secs_f64())
+}
+```
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Blocking Executor
+
+This proposal mainly focuses on async tasks. However, we can also consider adding blocking support to `Executor`. Users can use concurrent tasks in blocking context too:
+
+```rust
+ let w = op
+     .writer_with(path)
+     .chunk(8 * 1024 * 1024) // 8 MiB per chunk
++    .concurrent(16) // 16 concurrent tasks
+     .do()?;
+```

data/core/src/docs/rfcs/5314_remove_metakey.md

@@ -0,0 +1,120 @@
+- Proposal Name: `remove_metakey`
+- Start Date: 2024-11-12
+- RFC PR: [apache/opendal#5313](https://github.com/apache/opendal/pull/5313)
+- Tracking Issue: [apache/opendal#5314](https://github.com/apache/opendal/issues/5314)
+
+# Summary
+
+Remove the `Metakey` concept from OpenDAL and replace it with a simpler and more predictable metadata handling mechanism.
+
+# Motivation
+
+The current `Metakey` design has several issues:
+
+1. Performance Impact: Users often initiate costly operations unintentionally, such as using `Metakey::Full`, which results in extra stat calls
+2. Usability Issues: Users often try to access metadata that hasn't been explicitly requested
+3. API Confusion: There's a conflict between `Metakey::Version` and the new `version(bool)` parameter
+4. Implementation Complexity: Service developers struggle to implement `Metakey` correctly
+
+The goal is a simpler, more intuitive API that prevents common mistakes and improves performance as standard.
+
+# Guide-level explanation
+
+Instead of using `Metakey` to specify which metadata fields to fetch, services will now declare their metadata capabilities upfront through a new `MetadataCapability` struct:
+
+```rust
+let entries = op.list("path").await?;
+for entry in entries {
+    if op.metadata_capability().content_type {
+        println!("Content-Type: {}", entry.metadata().content_type());
+    }
+}
+```
+
+If users need additional metadata not provided by `list`:
+
+```rust
+let entries = op.list("path").await?;
+for entry in entries {
+    let mut meta = entry.metadata();
+    if !op.metadata_capability().etag {
+        meta = op.stat(&entry.path()).await?;
+    }
+    println!("Content-Type: {}", meta.etag());
+}
+```
+
+For existing OpenDAL users, the main changes are:
+
+- Remove all `metakey()` calls from their code
+- Use `metadata_capability()` to check available metadata
+- Explicitly call `stat()` when needed
+
+# Reference-level explanation
+
+The implementation involves:
+
+1. Remove the `Metakey` enum
+2. Add new `MetadataCapability` struct:
+```rust
+pub struct MetadataCapability {
+    pub content_length: bool,
+    pub content_type: bool,
+    pub last_modified: bool,
+    pub etag: bool,
+    pub mode: bool,
+    pub version: bool,
+    ...
+}
+```
+
+3. Add method to Operator to query capabilities:
+```rust
+impl Operator {
+    pub fn metadata_capability(&self) -> MetadataCapability;
+}
+```
+
+4. Modify list operation to avoid implicit stat calls
+5. Update all service implementations to declare their metadata capabilities
+
+Each service implementation will need to:
+- Remove `Metakey` handling logic
+- Implement `metadata_capability()` to accurately indicate the metadata provided by default
+- Ensure list operations return metadata that's always available without extra API calls
+
+# Drawbacks
+
+- Breaking change for existing users
+- Loss of fine-grained control over metadata fetching
+- Potential increased API calls if users need multiple metadata fields
+
+# Rationale and alternatives
+
+This design is superior because:
+- Prevents performance pitfalls by default
+- Makes metadata availability explicitly
+- Simplifies service implementation
+- Provides clearer mental model
+
+Alternatives considered:
+1. Keep `Metakey` but make it more restrictive
+2. Add warnings for potentially costly operations
+3. Make stat calls async/lazy
+
+Not making this change would continue the current issues of performance problems and API misuse.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- Add metadata prefetching optimization
+- Add metadata caching layer
+- Support for custom metadata fields
+- Automated capability detection