opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/docs/rfcs/2133_append_api.md

@@ -0,0 +1,88 @@
+- Proposal Name: `append_api`
+- Start Date: 2023-04-26
+- RFC PR: [apache/opendal#2133](https://github.com/apache/opendal/pull/2133)
+- Tracking Issue: [apache/opendal#2163](https://github.com/apache/opendal/issues/2163)
+
+# Summary
+
+Introduce append operations for OpenDAL which allow users to add data to a file.
+
+# Motivation
+
+OpenDAL has the write operation used to create a file and upload in parts. This is implemented based on multipart API. However, current approach has some limitations:
+
+- Data could be lost and not readable before w.close() returned Ok(())
+- File can't be appended again after w.close() returned Ok(())
+
+To address these issues, I propose adding an append operation. Users can create an appender that provides a reentrant append operation. Each append operation will add data to the end of the file, which can be read immediately after the operation.
+
+# Guide-level explanation
+
+The files created by the append operation can be appended via append API.
+
+```rust
+async fn append_test(op: Operation) -> Result<()> {
+  // create writer
+  let append = op.append("path_to_file").await?;
+
+  let bs = read_from_file();
+  append.append(bs).await?;
+
+  let bs = read_from_another_file();
+  append.append(bs).await?;
+
+  // close the file
+  append.close().await?;
+}
+```
+
+Difference between the write and append operation:
+
+- write: Always create a new file, not readable until close.
+- append: Can append existing appendable file, readable after append return.
+
+# Reference-level explanation
+
+For underlay API, we will make these changes in Accessor:
+
+```rust
+trait Accessor {
+    type Appender: oio::Append;
+
+    async fn append(&self, path: &str, args: OpAppend) -> Result<Self::Append>;
+}
+```
+
+To implement this feature, we need to add a new API `append` into `oio::Append`.
+
+```rust
+#[async_trait]
+pub trait Append: Unpin + Send + Sync {
+    /// Append data to the end of file.
+    /// Users will call `append` multiple times. Please make sure `append` is safe to re-enter.
+    async fn append(&mut self, bs: Bytes) -> Result<()>;
+
+    /// Seal the file to mark it as unmodifiable.
+    async fn close(&mut self) -> Result<()>;
+}
+```
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+We can use append API to implement for services that natively support append, such as [Azure blob](https://learn.microsoft.com/en-us/rest/api/storageservices/append-block?tabs=azure-ad) and [Alibaba cloud OSS](https://www.alibabacloud.com/help/en/object-storage-service/latest/appendobject). This will improve the performance and reliability of append operation.

data/core/src/docs/rfcs/2299_chain_based_operator_api.md

@@ -0,0 +1,99 @@
+- Proposal Name: `chain_based_operator_api`
+- Start Date: 2023-05-23
+- RFC PR: [apache/opendal#2299](https://github.com/apache/opendal/pull/2299)
+- Tracking Issue: [apache/opendal#2300](https://github.com/apache/opendal/issues/2300)
+
+# Summary
+
+Add chain based Operator API to replace `OpXxx`.
+
+# Motivation
+
+OpenDAL provides `xxx_with` API for users to add more options for requests:
+
+```rust
+let bs = op.read_with("path/to/file", OpRead::new().with_range(0..=1024)).await?;
+```
+
+However, the API's usability is hindered as users are required to create a new `OpXxx` struct. The API call can be excessively verbose:
+
+```rust
+let bs = op.read_with(
+  "path/to/file",
+  OpRead::new()
+    .with_range(0..=1024)
+    .with_if_match("<etag>")
+    .with_if_none_match("<etag>")
+    .with_override_cache_control("<cache_control>")
+    .with_override_content_disposition("<content_disposition>")
+  ).await?;
+```
+
+
+# Guide-level explanation
+
+In this proposal, I plan to introduce chain based `Operator` API to make them more friendly to use:
+
+```rust
+let bs = op.read_with("path/to/file")
+  .range(0..=1024)
+  .if_match("<etag>")
+  .if_none_match("<etag>")
+  .override_cache_control("<cache_control>")
+  .override_content_disposition("<content_disposition>")
+  .await?;
+```
+
+By eliminating the usage of `OpXxx`, our users can write code that is more readable.
+
+# Reference-level explanation
+
+To implement chain based API, we will change `read_with` as following:
+
+```diff
+- pub async fn read_with(&self, path: &str, args: OpRead) -> Result<Vec<u8>>
++ pub fn read_with(&self, path: &str) -> FutureRead
+```
+
+`FutureRead` will implement `Future<Output=Result<Vec<u8>>>`, so that users can still call `read_with` like the following:
+
+```rust
+let bs = op.read_with("path/to/file").await?;
+```
+
+For blocking operations, we will change `read_with` as following:
+
+```diff
+- pub fn read_with(&self, path: &str, args: OpRead) -> Result<Vec<u8>>
++ pub fn read_with(&self, path: &str) -> FunctionRead
+```
+
+`FunctionRead` will implement `call(self) -> Result<Vec<u8>>`, so that users can call `read_with` like the following:
+
+```rust
+let bs = op.read_with("path/to/file").call()?;
+```
+
+After this change, all `OpXxx` will be moved as raw API.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+## Change API after fn_traits stabilized
+
+After [fn_traits](https://github.com/rust-lang/rust/issues/29625) get stabilized, we will implement `FnOnce` for `FunctionXxx` instead of `call`.

data/core/src/docs/rfcs/2602_object_versioning.md

@@ -0,0 +1,138 @@
+- Proposal Name: object_versioning
+- Start Date: 2023-07-06
+- RFC PR: [apache/opendal#2602](https://github.com/apache/opendal/pull/2602)
+- Tracking Issue: [apache/opendal#2611](https://github.com/apache/opendal/issues/2611)
+
+# Summary
+
+This proposal describes the object versioning (or object version control) feature of OpenDAL.
+
+# Motivation
+
+There is a kind of storage service, which is called object storage service, 
+provides a simple and scalable way to store, organize, and access unstructured data. 
+These services store data as objects within buckets. 
+And an object is a file and any metadata that describes that file, a bucket is a container for objects. 
+
+The object versioning provided by these services is a very useful feature. 
+It allows users to keep multiple versions of an object in the same bucket. 
+If users enable object versioning, each object will have a history of versions. 
+Each version will have a unique version ID, which is a string that is unique for each version of an object.
+
+(The object, bucket,
+and version ID mentioned here are all concepts of object storage services,
+they could be called differently in different services, 
+but they are the same thing.)
+
+OpenDAL provides support for some of those services, such as S3, GCS, Azure Blob Storage, etc.
+Now we want to add support for object versioning to OpenDAL.
+
+# Guide-level explanation
+
+When object versioning is enabled, the following operations will be supported:
+
+- `stat`: Get the metadata of an object with specific version ID.
+- `read`: Read a specific version of an object.
+- `delete`: Delete a specific version of an object.
+
+Code example:
+
+```rust
+// To get the current version ID of a file
+let meta = op.stat("path/to/file").await?;
+let version_id = meta.version().expect("just for example");
+
+// To fetch the metadata of specific version of a file
+let meta = op.stat_with("path/to/file").version("version_id").await?;
+let version_id = meta.version().expect("just for example"); // get the version ID
+
+// To read an file with specific version ID
+let content = op.read_with("path/to/file").version("version_id").await?;
+
+// To delete an file with specific version ID
+op.delete_with("path/to/file").version("version_id").await?;
+```
+
+# Reference-level explanation
+
+Those operations with object version are different from the normal operations:
+
+- `stat`: when getting the metadata of a file, it will always get the metadata of the latest version of the file if no version ID is specified. And there will be a new field `version` in the metadata to indicate the version ID of the file.
+- `read`: when reading a file, it will always read the latest version of the file if no version ID is specified.
+- `delete`: when deleting a file, it will always delete the latest version of the file if no version ID is specified. And users will not be able to read this file without specifying the version ID, unless they specify a version not be deleted.
+
+And with object versioning, when writing an object, 
+it will always create a new version of the object than overwrite the old version. 
+But here it is imperceptible to the user. 
+Because the version id is generated by the service itself, it cannot be specified by the user and user cannot override the historical version.
+
+To implement object versioning, we will do the following:
+
+- Add a new field `version` to `OpStat`, `OpRead` and `OpDelete` struct.
+- Add a new field `version` to `ObjectMetadata` struct.
+- Add a new property(setter) `version` to the return value of `stat_with`, `read_with` method.
+- Add a new method `delete_with` and add a new property(setter) `version` to the return value of `delete_with` method.
+
+For service backend, it should support the following operations:
+
+- `stat`: Get the metadata of an object with specific version ID.
+- `read`: Read a specific version of an object.
+- `delete`: Delete a specific version of an object.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+## What is object versioning?
+
+Object versioning is a feature that allows users to keep multiple versions of an object in the same bucket.
+
+It's a way to preserve, retrieve, and restore every version of every object stored in a bucket.
+
+With object versioning, users can easily recover from both unintended user actions and application failures.
+
+## How does object versioning work?
+
+When object versioning is enabled, each object will have a history of versions. Each version will have a unique version ID, which is a string that is unique for each version of an object.
+
+The version ID is not a timestamp.
+It is not guaranteed to be sequential.
+Many object storage services produce object version IDs by themselves, using their own algorithms.
+Users cannot specify the version ID when writing an object.
+
+## Will object versioning affect the existing code?
+
+There is no difference between whether object versioning is enabled or not when writing an object.
+The storage service will always create a new version of the object than overwrite the old version when writing an object.
+But here it is imperceptible to the user.
+
+## What are the benefits of object versioning?
+
+With object versioning, users can:
+
+- Track the history of a file.
+- Implement optimistic concurrency control.
+- Implement a simple backup system.
+
+## reference
+
+- [AWS S3 Object Versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html)
+- [How does AWS S3 object versioning work?](https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html)
+- [How to enable object versioning for a bucket in AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html)
+- [Google Cloud Storage Object Versioning](https://cloud.google.com/storage/docs/object-versioning)
+- [Azure Blob Storage Object Versioning](https://docs.microsoft.com/en-us/azure/storage/blobs/versioning-overview)
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+Impl a new method `list_versions`(list all versions of an object).
+

data/core/src/docs/rfcs/2758_merge_append_into_write.md

@@ -0,0 +1,79 @@
+- Proposal Name: `merge_append_into_write`
+- Start Date: 2023-08-02
+- RFC PR: [apache/opendal#2758](https://github.com/apache/opendal/pull/2758)
+- Tracking Issue: [apache/opendal#2760](https://github.com/apache/opendal/issues/2760)
+
+# Summary
+
+Merge the `appender` API into `writer` by introducing a new `writer_with.append(true)` method to enable append mode.
+
+# Motivation
+
+Currently OpenDAL has separate `appender` and `writer` APIs:
+
+```rust
+let mut appender = op.appender_with("file.txt").await?; 
+
+appender.append(bs).await?;
+appender.append(bs).await?;
+```
+
+This duplication forces users to learn two different APIs for writing data.
+
+By adding this change, we can:
+
+- Simpler API surface - users only need to learn one writing API.
+- Reduce code duplication between append and write implementations.
+- Atomic append semantics are handled internally in `writer`.
+- Reuse the `sink` api for both `overwrite` and `append` mode.
+
+# Guide-level explanation
+
+The new approach is to enable append mode on `writer`:
+
+```rust 
+let mut writer = op.writer_with("file.txt").append(true).await?;
+
+writer.write(bs).await?; // appends to file
+writer.write(bs).await?; // appends again
+```
+
+Calling `writer_with.append(true)` will start the writer in append mode. Subsequent `write()` calls will append rather than overwrite.
+
+There is no longer a separate `appender` API.
+
+# Reference-level explanation
+
+We will add an `append` flag into `OpWrite`:
+
+```rust
+impl OpWrite {   
+    pub fn with_append(mut self, append: bool) -> Self {
+        self.append = append;
+        self
+    }
+}
+```
+
+All services need to check `append` flag and handle append mode accordingly. Services that not support append should return an `Unsupported` error instead.
+
+# Drawbacks
+
+- `writer` API is more complex with the append mode flag.
+- Internal implementation must handle both overwrite and append logic.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+Python's file open() supports an `"a"` mode flag to enable append-only writing.
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/2774_lister_api.md

@@ -0,0 +1,66 @@
+- Proposal Name: `lister_api`
+- Start Date: 2023-08-04
+- RFC PR: [apache/opendal#2774](https://github.com/apache/opendal/pull/2774)
+- Tracking Issue: [apache/opendal#2775](https://github.com/apache/opendal/issues/2775)
+
+# Summary
+
+Add `lister` API to align with other OpenDAL APIs like `read`/`reader`.
+
+# Motivation
+
+Currently OpenDAL has `list` APIs like:
+
+```rust
+let lister = op.list().await?;
+```
+
+This is inconsistent with APIs like `read`/`reader` and can confuse users.
+
+We should add a new `lister` API and change the `list` to:
+
+- Align with other OpenDAL APIs
+- Simplify usage
+
+# Guide-level explanation
+
+The new APIs will be:
+
+```rust
+let entries = op.list().await?; // Get entries directly
+
+let lister = op.lister().await?; // Get lister
+```
+
+- `op.list()` returns entries directly.
+- `op.lister()` returns a lister that users can list entries on demand.
+
+# Reference-level explanation
+
+We will:
+
+- Rename existing `list` to `lister`
+- Add new `list` method to call `lister` and return all entries
+- Merge `scan` into `list_with` with `delimiter("")`
+
+This keeps the pagination logic encapsulated in `lister`.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/2779_list_with_metakey.md

@@ -0,0 +1,143 @@
+- Proposal Name: `list_with_metakey`
+- Start Date: 2023-08-04
+- RFC PR: [apache/opendal#2779](https://github.com/apache/opendal/pull/2779)
+- Tracking Issue: [apache/opendal#2802](https://github.com/apache/opendal/issues/2802)
+
+# Summary
+
+Move `Operator` `metadata` API to `list_with().metakey()` to simplify the usage.
+
+# Motivation
+
+The current `Entry` metadata API is:
+
+```rust
+use opendal::Entry;
+use opendal::Metakey;
+
+let meta = op
+    .metadata(&entry, Metakey::ContentLength | Metakey::ContentType)
+    .await?;
+```
+
+This API is difficult to understand and rarely used correctly. And in reality, users always fetch the same set of metadata during listing.
+
+Take one of our users code as an example:
+
+```rust
+let stream = self
+    .inner
+    .scan(&path)
+    .await
+    .map_err(|err| format_object_store_error(err, &path))?;
+
+let stream = stream.then(|res| async {
+    let entry = res.map_err(|err| format_object_store_error(err, ""))?;
+    let meta = self
+        .inner
+        .metadata(&entry, Metakey::ContentLength | Metakey::LastModified)
+        .await
+        .map_err(|err| format_object_store_error(err, entry.path()))?;
+
+    Ok(format_object_meta(entry.path(), &meta))
+});
+
+Ok(stream.boxed())
+```
+
+By moving metadata to `lister`, our user code can be simplified to:
+
+```rust
+let stream = self
+    .inner
+    .scan_with(&path)
+    .metakey(Metakey::ContentLength | Metakey::LastModified)
+    .await
+    .map_err(|err| format_object_store_error(err, &path))?;
+
+let stream = stream.then(|res| async {
+    let entry = res.map_err(|err| format_object_store_error(err, ""))?;
+    let meta = entry.into_metadata()
+
+    Ok(format_object_meta(entry.path(), &meta))
+});
+
+Ok(stream.boxed())
+```
+
+By introducing this change:
+
+- Users don't need to capture `Operator` in the closure.
+- Users don't need to do async call like `metadata()` again.
+
+If we don't have this change:
+
+- every place that could receive a `fn()` must use `Fn()` instead which enforce users to have a generic parameter in their code.
+- It's harder for other languages binding to implement `op.metadata()` right.
+
+# Guide-level explanation
+
+The new API will be:
+
+```rust
+let entries: Vec<Entry> = op
+  .list_with("dir")
+  .metakey(Metakey::ContentLength | Metakey::ContentType).await?;
+
+let meta: &Metadata = entries[0].metadata();
+```
+
+Metadata can be queried directly when listing entries via `metadata()`, and later extracted via `into_parts()`.
+
+# Reference-level explanation
+
+## How metakey works
+
+For every services, `stat` will return the full set of it's metadata. For example, `s3` will return `ContentLength | ContentType | LastModified | ...`, and `fs` will return `ContentLength | LastModified`. And most services will return part of those metadata during `list`. `s3` will return `ContentLength`, `LastModified`, but `fs` returns none of them.
+
+So when users use `list` to list entries, they will get a list of entries with incomplete metadata. The metadata could be in three states:
+
+- Filled: the metadata is returned in `list`
+- NotExist: the metadata is not supported by service.
+- Unknown: the metadata is supported by service but not returned in `list`.
+
+By accept `metakey`, we can compare the returning entry's metadata with metakey:
+
+- Return the entry if metakey already met by `Filled` and `NotExist`.
+- Send `stat` call to fetch the metadata if metadata is `Unknown`.
+
+## Changes
+
+We will add `metakey` into `OpList`. Underlying services can use those information to try their best to fetch the metadata.
+
+There are following possibilities:
+
+- The entry metadata is met: `Lister` return the entry directly
+- The entry metadata is not met and not fully filled: `Lister` will try to send `stat` call to fetch the metadata
+- The entry metadata is not met and fully filled: `Lister` will return the entry directly.
+
+To make sure we can handle all metadata correctly, we will add a new capability called `stat_complete_metakey`. This capability will be used to indicate the complete set of metadata that can be fetched via `stat` call. For example, `s3` can set this capability to `ContentLength | ContentType | LastModified | ...`, and `fs` only have `ContentLength | LastModified`. `Lister` can use this capability to decide whether to send `stat` call or not.
+
+Services' lister implementation should not changed.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+Keeping the complex standalone API has limited benefit given low usage.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+## Add glob and regex support for Lister
+
+We can add `glob` and `regex` support for `Lister` to make it more powerful.

data/core/src/docs/rfcs/2852_native_capability.md

@@ -0,0 +1,58 @@
+- Proposal Name: `native_capability`
+- Start Date: 2023-08-11
+- RFC PR: [apache/opendal#2852](https://github.com/apache/opendal/pull/2852)
+- Tracking Issue: [apache/opendal#2859](https://github.com/apache/opendal/issues/2859)
+
+# Summary
+
+Add `native_capability` and `full_capability` to `Operator` so that users can make more informed decisions.
+
+# Motivation
+
+OpenDAL adds `Capability` to inform users whether a service supports a specific feature. However, this is not enough for users to make decisions. OpenDAL doesn't simply expose the services' API directly; instead, it simulates the behavior to make it more useful.
+
+For example, `s3` doesn't support seek operations like a local file system. But it's a quite common operation for users. So OpenDAL will try to simulate the behavior by calculating the correct offset and reading the data from that offset instead. After this simulation, the `s3` service has the `read_can_seek` capability now.
+
+As another example, most services like `s3` don't support blocking operations. OpenDAL implements a `BlockingLayer` to make it possible. After this implementation, the `s3` service has the `blocking` capability now.
+
+However, these capabilities alone are insufficient for users to make informed decisions. Take the `s3` service's `blocking` capability as an example. Users are unable to determine whether it is a native capability or not, which may result in them unknowingly utilizing this feature in performance-sensitive scenarios, leading to significantly poor performance.
+
+So this proposal intends to address this issue by adding `native_capability` and `full_capability` to `OperatorInfo`. Users can use `native_capability` to determine whether a capability is native or not.
+
+# Guide-level explanation
+
+We will add two new APIs `native_capability()` and `full_capability()` in `OperatorInfo`, and remove the `capability()` and related `can_xxx()` API.
+
+```diff
++ pub fn native_capability(&self) -> Capability
++ pub fn full_capability(&self) -> Capability
+- pub fn capability(&self) -> Capability
+```
+
+# Reference-level explanation
+
+We will add two new fields `native_capability` and `full_capability` in `AccessorInfo`:
+
+- Services SHOULD only set `native_capability`, and `full_capability` will be the same as `native_capability`.
+- Layers MAY change `full_capability` and MUST NOT modify `native_capability`.
+- `OperatorInfo` should forward `native_capability()` and `full_capability()` to `AccessorInfo`.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None