opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/docs/rfcs/0501_new_builder.md

@@ -0,0 +1,111 @@
+- Proposal Name: `new_builder`
+- Start Date: 2022-08-03
+- RFC PR: [apache/opendal#501](https://github.com/apache/opendal/pull/501)
+- Tracking Issue: [apache/opendal#502](https://github.com/apache/opendal/issues/502)
+
+# Summary
+
+Allow users to build services without async.
+
+# Motivation
+
+Most services share a similar builder API to construct backends.
+
+```rust
+impl Builder {
+    pub async fn finish(&mut self) -> Result<Arc<dyn Accessor>> {}
+}
+```
+
+We have `async` here so that every user who wants to build services backend must go through an async runtime. Even for `memory` backend:
+
+```rust
+impl Builder {
+    /// Consume builder to build a memory backend.
+    pub async fn finish(&mut self) -> Result<Arc<dyn Accessor>> {
+        Ok(Arc::new(Backend::default()))
+    }
+}
+```
+
+Only `s3` services need to call async functions `detect_region` to get the correct region.
+
+So, we can provide blocking `Builder` APIs and move async-related logic out for users to call out. This way, our users can build services without playing with async runtime.
+
+# Guide-level explanation
+
+After this change, all our services builder will add a new API:
+
+```rust
+impl Builder {
+    pub fn build(&mut self) -> Result<Backend> {}
+}
+```
+
+Along with this change, our `Operator` will accept `impl Accessor + 'static` instead of `Arc<dyn Accessor>` anymore:
+
+```rust
+impl Operator {
+    pub fn new(accessor: impl Accessor + 'static) -> Self {}
+}
+```
+
+Also, we will implement `From<impl Accessor + 'static>` for `Operator`:
+
+```rust
+impl<A> From<A> for Operator
+where
+    A: Accessor + 'static,
+{
+    fn from(acc: A) -> Self {
+        Operator::newx(acc)
+    }
+}
+```
+
+We can initiate an operator quicker:
+
+```diff
+- let op: Operator = Operator::new(fs::Backend::build().finish().await?);
++ let op: Operator = fs::Builder::new().build()?.into();
+```
+
+# Reference-level explanation
+
+We will add the following APIs:
+
+- All builders will add `build(&mut self) -> Result<Backend>`
+- `impl<A> From<A> for Operator where A: Accessor + 'static`
+
+We will deprecate the following APIs:
+
+- All builders `finish()` API (should be replaced by `build()`)
+- All services `build()` API (should be replaced by `Builder::new()` or `Builder::default()`)
+
+We will change the following APIs:
+
+- Operator: `new(accessor: Arc<dyn Accessor>)` -> `fn new(accessor: impl dyn Accessor + 'static)`
+- Operator: `async fn from_iter()` -> `fn from_iter()`
+- Operator: `async fn from_env()` -> `fn from_env()`
+
+Most services will work the same, except for `s3`: `s3` depends on `detect_region` to check the correct region if the user doesn't input. After this change, `s3::Builder.build()` will return error if `region` is missing. Users should call `detect_region` by themselves to get the region.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/0554_write_refactor.md

@@ -0,0 +1,96 @@
+- Proposal Name: `write_refactor`
+- Start Date: 2022-08-22
+- RFC PR: [apache/opendal#554](https://github.com/apache/opendal/pull/554)
+- Tracking Issue: [apache/opendal#555](https://github.com/apache/opendal/issues/555)
+
+# Summary
+
+Refactor `write` operation to accept a `BytesReader` instead.
+
+# Motivation
+
+To simulate the similar operation like POSIX fs, OpenDAL returns `BytesWriter` for users to write, flush and close:
+
+```rust
+pub trait Accessor {
+    async fn write(&self, args: &OpWrite) -> Result<BytesWriter> {}
+}
+```
+
+`Operator` builds the high level APIs upon this:
+
+```rust
+impl Object {
+    pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
+    
+    pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}
+}
+```
+
+However, we are meeting the following problems:
+
+- Performance: HTTP body channel is mush slower than read from Reader directly.
+- Complicity: Service implementer have to deal with APIs like `new_http_channel`.
+- Extensibility: Current design can't be extended to multipart APIs.
+
+# Guide-level explanation
+
+Underlying `write` implementations will be replaced by:
+
+```rust
+pub trait Accessor {
+    async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
+}
+```
+
+Existing API will have no changes, and we will add a new API:
+
+```rust
+impl Object {
+    pub async fn write_from(&self, size: u64, r: impl BytesRead) -> Result<u64> {}
+}
+```
+
+# Reference-level explanation
+
+`Accessor`'s `write` API will be changed to accept a `BytesReader`:
+
+```rust
+pub trait Accessor {
+    async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
+}
+```
+
+We will provide `Operator::writer` based on this new API instead.
+
+[RFC-0438: Multipart](./0438-multipart.md) will also be updated to:
+
+```rust
+pub trait Accessor {
+    async fn write_multipart(&self, args: &OpWriteMultipart, r: BytesReader) -> Result<u64> {}
+}
+```
+
+In this way, we don't need to introduce a `PartWriter`.
+
+# Drawbacks
+
+## Layer API breakage
+
+This change will introduce break changes to layers.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+- [RFC-0191: Async Streaming IO](./0191-async-streaming-io.md)
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/0561_list_metadata_reuse.md

@@ -0,0 +1,210 @@
+- Proposal Name: `list_metadata_reuse`
+- Start Date: 2022-08-23
+- RFC PR: [apache/opendal#561](https://github.com/apache/opendal/pull/561)
+- Tracking Issue: [apache/opendal#570](https://github.com/apache/opendal/pull/570)
+
+# Summary
+
+Reuse metadata returned during listing, by extending `DirEntry` with some metadata fields.
+
+# Motivation
+
+Users may expect to browse metadata of some directories' child files and directories. Using `walk()` of `BatchOperator` seems to be an ideal way to complete this job. 
+
+Thus, they start iterating on it, but soon they realized the `DirEntry`, could only offer the name (or path, more precisely) and access mode of the object, and it's not enough.
+
+So they have to call `metadata()` for each name they extracted from the iterator.
+
+The final example looks like:
+
+```rust
+let op = Operator::from_env(Scheme::Gcs)?.batch();
+
+// here is a network request
+let mut dir_stream = op.walk("/dir/to/walk")?;
+
+while let Some(Ok(file)) = dir_stream.next().await {
+    let path = file.path();
+
+    // here is another network request
+    let size = file.metadata().await?.content_length();
+    println!("size of file {} is {}B", path, size);
+}
+```
+
+But...wait! many storage-services returns object metadata when listing, like HDFS, AWS and GCS. The rust standard library returns metadata when listing local file systems, too.
+
+In the previous versions of OpenDAL those fields were just get ignored. This wastes users' time on requesting on metadata.
+
+# Guide-level explanation
+
+The loop in main will be changed to the following code with this RFC:
+
+```rust
+while let Some(Ok(file)) = dir_stream.next().await {
+    let size = if let Some(len) = file.content_length() {
+        len
+    } else {
+        file.metadata().await?.content_length();
+    };
+    let name = file.path();
+    println!("size of file {} is {}B", path, size);
+}
+```
+
+# Reference-level explanation
+
+Extend `DirEntry` with metadata fields:
+
+```rust
+pub struct DirEntry {
+    acc: Arc<dyn Accessor>,
+
+    mode: ObjectMode,
+    path: String,
+
+    // newly add metadata fields
+    content_length: Option<u64>,  // size of file
+    content_md5: Option<String>,
+    last_modified: Option<OffsetDateTime>,
+}
+
+impl DirEntry {
+    pub fn content_length(&self) -> Option<u64> {
+        self.content_length
+    }
+    pub fn last_modified(&self) -> Option<OffsetDateTime> {
+        self.last_modified
+    }
+    pub fn content_md5(&self) -> Option<OffsetDateTime> {
+        self.content_md5
+    }
+}
+```
+
+For all services that supplies metadata during listing, like AWS, GCS and HDFS. Those optional fields will be filled up; Meanwhile for those services doesn't return metadata during listing, like in memory storages, just left them as `None`.
+
+As you can see, for those services returning metadata when listing, the operation of listing metadata will save many unnecessary requests.
+
+# Drawbacks
+ 
+Add complexity to `DirEntry`. To use the improved features of `DirEntry`, users have to explicitly check the existence of metadata fields.
+
+The size of `DirEntry` increased from 40 bytes to 80 bytes, a 100% percent growth requires more memory.
+
+# Rational and alternatives
+
+The largest drawback of performance usually comes from network or hard disk operations. By letting `DirEntry` storing some metadata, many redundant requests could be avoided.
+
+## Embed a Structure Containing Metadata
+
+Define a `MetaLite` structure containing some metadata fields, and embed it in `DirEntry`
+
+```rust
+struct MetaLite {
+    pub content_length: u64,  // size of file
+    pub content_md5: String,
+    pub last_modified: OffsetDateTime,
+}
+
+pub struct DirEntry {
+    acc: Arc<dyn Accessor>,
+    
+    mode: ObjectMode,
+    path: String,
+    
+    // newly add metadata struct
+    metadata: Option<MetaLite>,
+}
+
+impl DirEntry {
+    // get size of file
+    pub fn content_length(&self) -> Option<u64> {
+        self.metadata.as_ref().map(|m| m.content_length)
+    }
+    // get the last modified time
+    pub fn last_modified(&self) -> Option<OffsetDateTime> {
+        self.metadata.as_ref().map(|m| m.last_modified)
+    }
+    // get md5 message digest
+    pub fn content_md5(&self) -> Option<String> {
+        self.metadata.as_ref().map(|m| m.content_md5)
+    }
+}
+```
+
+The existence of those newly added metadata fields is highly correlated. If one field does not exist, the others neither.
+
+By wrapping them together in an embedded structure, 8 bytes of space for each `DirEntry` object could be saved. In the future, more metadata fields may be added to `DirEntry`, then a lot more space could be saved.
+
+This approach could be slower because some intermediate functions are involved. But it's worth sacrificing rarely used features' performance to save memory.
+
+## Embed a `ObjectMetadata` into `DirEntry`
+
+- Embed a `ObjectMetadata` struct into `DirEntry`
+- Remove the `ObjectMode` field in `DirEntry`
+- Change `ObjectMetadata`'s `content_length` field's type to `Option<u64>`.
+
+```rust
+pub struct DirEntry {
+    acc: Arc<dyn Accessor>,
+
+    // - mode: ObjectMode, removed
+    path: String,
+
+    // newly add metadata struct
+    metadata: ObjectMetadata,
+}
+
+impl DirEntry {
+    pub fn mode(&self) -> ObjectMode {
+        self.metadata.mode()
+    }
+    pub fn content_length(&self) -> Option<u64> {
+        self.metadata.content_length()
+    }
+    pub fn content_md5(&self) -> Option<&str> {
+        self.metadata.content_md5()
+    }
+    // other metadata getters...
+}
+```
+
+In the degree of memory layout, it's the same as proposed way in this RFC. This approach offers more metadata fields and fewer changes to code.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Switch to Alternative Implement Approaches
+
+As the growing of metadata fields, someday the alternatives could be better. And other RFCs will be raised then.
+
+## More Fields
+
+Add more metadata fields to DirEntry, like:
+
+- accessed: the last access timestamp of object
+
+## Simplified Get
+
+Users have to explicitly check if those metadata fields actual present in the DirEntry. This may be done inside the getter itself.
+
+```rust
+let path = file.path();
+
+// if content_length is not exist
+// this getter will automatically fetch from the storage service.
+let size = file.content_length().await?;
+
+// the previous getter can cache metadata fetched from service
+// so this function could return instantly.
+let md5 = file.content_md5().await?;
+println!("size of file {} is {}B, md5 outcome of file is {}", path, size, md5);
+```

data/core/src/docs/rfcs/0599_blocking_api.md

@@ -0,0 +1,157 @@
+- Proposal Name: `blocking_api`
+- Start Date: 2022-08-30
+- RFC PR: [apache/opendal#599](https://github.com/apache/opendal/pull/599)
+- Tracking Issue: [apache/opendal#601](https://github.com/apache/opendal/issues/601)
+
+# Summary
+
+We are adding a blocking API for OpenDAL.
+
+# Motivation
+
+Blocking API is the most requested feature inside the OpenDAL community: [Opendal support sync read/write API](https://github.com/apache/opendal/discussions/68)
+
+Our users want blocking API for:
+
+- Higher performance for local IO
+- Using OpenDAL in a non-async environment
+
+However, supporting sync and async API in current Rust is a painful job, especially for an IO library like OpenDAL. For example:
+
+```rust
+impl Object {
+    pub async fn reader(&self) -> Result<impl BytesRead> {}
+}
+```
+
+Supporting blocking API doesn't mean removing the `async` from the function. We should also handle the returning `Reader`:
+
+```rust
+impl Object {
+    pub fn reader(&self) -> Result<impl Read> {}
+}
+```
+
+Until now, I still don't know how to handle them correctly. But we need to have a start: not perfect, but enough for our users to have a try.
+
+So this RFC is an **experiment** try to introduce blocking API support. I expect the OpenDAL community will evaluate those APIs and keep improving them. And finally, we will pick up the best one for stabilizing.
+
+# Guide-level explanation
+
+With this RFC, we can call blocking API with the `blocking_` prefix:
+
+```rust
+fn main() -> Result<()> {
+    // Init Operator
+    let op = Operator::from_env(Scheme::Fs)?;
+
+    // Create object handler.
+    let o = op.object("test_file");
+
+    // Write data info object;
+    o.blocking_write("Hello, World!")?;
+
+    // Read data from object;
+    let bs = o.blocking_read()?;
+
+    // Read range from the object;
+    let bs = o.blocking_range_read(1..=11)?;
+
+    // Get the object's path
+    let name = o.name();
+    let path = o.path();
+
+    // Fetch more meta about the object.
+    let meta = o.blocking_metadata()?;
+    let mode = meta.mode();
+    let length = meta.content_length();
+    let content_md5 = meta.content_md5();
+    let etag = meta.etag();
+
+    // Delete object.
+    o.blocking_delete()?;
+
+    // List dir object.
+    let o = op.object("test_dir/");
+    let mut ds = o.blocking_list()?;
+    while let Some(entry) = ds.try_next()? {
+        let path = entry.path();
+        let mode = entry.mode();
+    }
+
+    Ok(())
+}
+```
+
+All async public APIs of `Object` and `Operator` will have a sync version with `blocking_` prefix. And they will share precisely the same semantics.
+
+The differences are:
+
+- They will be executed and blocked on the current thread.
+- Input and output's `Reader` will become the blocking version like `std::io::Read`.
+- Output's `DirStreamer` will become the blocking version like `Iterator`.
+
+Thanks to [RFC-0501: New Builder](./0501-new-builder.md), all our builder-related APIs have been transformed into blocking APIs, so we don't change our initiation logic.
+
+# Reference-level explanation
+
+Under the hood, we will add the following APIs in `Accessor`:
+
+```rust
+trait Accessor {
+    fn blocking_create(&self, args: &OpCreate) -> Result<()>;
+    
+    fn blocking_read(&self, args: &OpRead) -> Result<BlockingBytesReader>;
+    
+    fn blocking_write(&self, args: &OpWrite, r: BlockingBytesReader) -> Result<u64>;
+    
+    fn blocking_stat(&self, args: &OpStat) -> Result<ObjectMetadata>;
+    
+    fn blocking_delete(&self, args: &OpDelete) -> Result<()>;
+    
+    fn blocking_list(&self, args: &OpList) -> Result<DirIterator>;
+}
+```
+
+Notes:
+
+- `BlockingBytesReader` is a boxed `std::io::Read`.
+- All blocking operations are happening on the current thread.
+- Blocking operation is implemented natively, no `futures::block_on`.
+
+# Drawbacks
+
+## Two sets of APIs
+
+This RFC will add a new set of APIs, adding complicity for OpenDAL.
+
+And users may misuse them. For example: using `blocking_read` in an async context could block the entire thread.
+
+# Rationale and alternatives
+
+## Use features to switch `async` and `sync`
+
+Some crates provide features to switch the `async` and `sync` versions of API.
+
+In this way:
+
+- We can't provide two kinds of API at the same time.
+- Users must decide to use `async` or `sync` at compile time.
+
+## Use blocking IO functions in local fs services
+
+> Can we use blocking IO functions in local fs services to implement Accessor's asynchronous functions directly? What is the drawback of our current non-blocking API?
+
+We can't run blocking IO functions inside the `async` context. We need to let the local thread pool execute them and use `mio` to listen to the events. If we do so, congrats, we are building `tokio::fs` again!
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None