opendal 0.1.6.pre.rc.1-aarch64-linux

data/core/src/docs/rfcs/2884_merge_range_read_into_read.md

@@ -0,0 +1,80 @@
+- Proposal Name: `merge_range_read_into_read`
+- Start Date: 2023-08-20
+- RFC PR: [apache/opendal#2884](https://github.com/apache/opendal/pull/2884)
+- Tracking Issue: [apache/opendal#2885](https://github.com/apache/opendal/issues/2885)
+
+# Summary
+
+Merge the `range_read` API into `read` by deleting the `op.range_reader(path, range)` and `op.range_read(path, range)` method.
+
+# Motivation
+
+Currently OpenDAL has separate `range_read` and `read` APIs:
+
+```rust
+let bs = op.range_read("path/to/file", 1024..2048).await?;
+
+let bs = op.read("path/to/file").await?;
+```
+
+As same as `range_reader` and `reader` APIs:
+```rust
+let reader = op.range_reader("path/to/file", 1024..2048).await?;
+
+let reader = op.reader("path/to/file").await?;
+```
+
+
+This duplication forces users to learn two different APIs for reading data.
+
+By adding this change, we can:
+
+- Simpler API surface - users only need to learn one writing API.
+- Reduce code duplication between read and range_read implementations.
+- Atomic read semantics are handled internally in `reader`.
+
+# Guide-level explanation
+
+There is no new approach to read data from file. The `read` and `reader` API supported range read by default.
+
+Calling `read_with("path/to/file").range(range)` will return a `reader` that supports range read.
+
+```rust
+let bs = op.read_with("path/to/file").range(1024..2048).await?;
+```
+
+Calling `reader_with("path/to/file").range(range)` will return a `reader` that supports range read.
+
+```rust
+let rs = op.reader_with("path/to/file").range(1024..2048).await?;
+```
+
+There is no longer a separate `range_read` and `range_reader` API.
+
+# Reference-level explanation
+
+None
+
+# Drawbacks
+
+None
+
+## Breaking Changes
+
+This RFC  has removed the `range_read` and `range_reader` APIs. If you have been using these APIs, you will need to reimplement your code by `op.read("path/to/file").range(1024..2048)` or `op.reader("path/to/file").range(1024..2048)`.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/3017_remove_write_copy_from.md

@@ -0,0 +1,94 @@
+- Proposal Name: `remove_write_copy_from`
+- Start Date: 2023-09-06
+- RFC PR: [apache/opendal#3017](https://github.com/apache/opendal/pull/3017)
+- Tracking Issue: [apache/opendal#3017](https://github.com/apache/opendal/issues/3017)
+
+# Summary
+
+Remove the `oio::Write::copy_from()` API pending a more thoughtful design.
+
+# Motivation
+
+In [RFC-2083: Writer Sink API](./2083_writer_sink_api.md), we launched an API, initially named `sink` and changed to `copy_from`, that enables data writing from a `Reader` to a `Writer` object.
+
+The current API signature is:
+```rust
+pub trait Write: Unpin + Send + Sync {
+    /// Copies data from the given reader to the writer.
+    ///
+    /// # Behavior
+    ///
+    /// - `Ok(n)` indicates successful writing of `n` bytes.
+    /// - `Err(err)` indicates a failure, resulting in zero bytes written.
+    ///
+    /// A situation where `n < size` may arise; the caller should then transmit the remaining bytes until the full amount is written.
+    async fn copy_from(&mut self, size: u64, src: oio::Reader) -> Result<u64>;
+}
+```
+
+The API has the following limitations:
+
+- Incompatibility with existing buffering and retry mechanisms.
+- Imposes ownership requirements on the reader, complicating its use. The reader must be recreated for every write operation.
+
+Due to restrictions in both Rust and Hyper's APIs, the following ideal implementation is currently unattainable:
+
+```rust
+pub trait Write: Unpin + Send + Sync {
+    async fn copy_from(&mut self, size: u64, src: &mut impl oio::Read) -> Result<u64>;
+}
+```
+
+- Rust doesn't allow us to have `impl oio::Read` in trait method if we want object safe.
+- hyper doesn't allow us to use reference here, it requires `impl Stream + 'static`.
+
+Given these constraints, the proposal is to remove `oio::Write::copy_from` until a more fitting design becomes feasible.
+
+# Guide-level explanation
+
+The `Writer::sink()` and `Writer::copy()` methods will be kept, but it's internal implementation will be changed to use `AsyncWrite` instead. For example:
+
+```diff
+pub async fn copy<R>(&mut self, size: u64, read_from: R) -> Result<u64>
+where
+    R: futures::AsyncRead + Send + Sync + Unpin + 'static,
+{
+    if let State::Idle(Some(w)) = &mut self.state {
+        let r = Box::new(oio::into_streamable_read(
+            oio::into_read_from_file(read_from, 0, size),
+            64 * 1024,
+        ));
+-       w.copy_from(size, r).await
++       futures::io::copy(&mut r, w).await
+    } else {
+        unreachable!(
+            "writer state invalid while copy, expect Idle, actual {}",
+            self.state
+        );
+    }
+}
+```
+
+# Reference-level explanation
+
+The method `oio::Write::copy_from` will be removed.
+
+# Drawbacks
+
+The deprecation eliminates the ability to stream data uploads. A viable alternative is to directly use `AsyncWrite` offered by `Writer`.
+
+# Rationale and Alternatives
+
+N/A
+
+# Prior Art
+
+N/A
+
+# Unresolved Questions
+
+N/A
+
+# Future Possibilities
+
+Introduce utility functions such as `Writer::copy_from(r: &dyn AsyncRead)` when possible.

data/core/src/docs/rfcs/3197_config.md

@@ -0,0 +1,237 @@
+- Proposal Name: `config`
+- Start Date: 2023-09-27
+- RFC PR: [apache/opendal#3197](https://github.com/apache/opendal/pull/3197)
+- Tracking Issue: [apache/opendal#3240](https://github.com/apache/opendal/issues/3240)
+
+# Summary
+
+Expose services config to the user. 
+
+# Motivation
+
+OpenDAL provides two ways to configure services: through a builder pattern and via a map.
+
+The `Builder` allows users to configure services using the builder pattern:
+
+```rust
+// Create fs backend builder.
+let mut builder = Fs::default();
+// Set the root for fs, all operations will happen under this root.
+builder.root("/tmp");
+
+// Build an `Operator` to start operating the storage.
+let op: Operator = Operator::new(builder)?.finish();
+```
+
+The benefit of builder is that it is type safe and easy to use. However, it is not flexible enough to configure services. Users must create a new builder for each service they wish to configure, translating user input into the API calls for each respective builder.
+
+Consider the following real-world example from one of our users:
+
+```rust
+let mut builder = services::S3::default();
+
+// Credential.
+builder.access_key_id(&cfg.access_key_id);
+builder.secret_access_key(&cfg.secret_access_key);
+builder.session_token(&cfg.session_token);
+builder.role_arn(&cfg.role_arn);
+builder.external_id(&cfg.external_id);
+
+// Root.
+builder.root(&cfg.root);
+
+// Disable credential loader
+if cfg.disable_credential_loader {
+    builder.disable_config_load();
+    builder.disable_ec2_metadata();
+}
+
+// Enable virtual host style
+if cfg.enable_virtual_host_style {
+    builder.enable_virtual_host_style();
+}
+```
+
+The `Map` approach allows users to configure services using a string-based HashMap:
+
+```rust
+let map = HashMap::from([
+  // Set the root for fs, all operations will happen under this root.
+  ("root".to_string(), "/tmp".to_string()),
+]);
+
+// Build an `Operator` to start operating the storage.
+let op: Operator = Operator::via_map(Scheme::Fs, map)?;
+```
+
+This approach is simpler since it allows users to configure all services within a single map. However, it is not type safe and not easy to use. Users will need to convert their input to string and make sure the key is correct. And breaking changes could happen silently.
+
+This is one of our limitations: We need a way to configure services that is type safe, easy to use and flexible. The other one is that there is no way for users to fetch the config of a service after it's built. This limitation complicates the dynamic modification of a service's root path for the user.
+
+Our users have to wrap all our configs into an enum and store it in their own struct:
+
+```rust
+pub enum StorageParams {
+    Azblob(StorageAzblobConfig),
+    Fs(StorageFsConfig),
+    Ftp(StorageFtpConfig),
+    Gcs(StorageGcsConfig),
+    Hdfs(StorageHdfsConfig),
+    Http(StorageHttpConfig),
+    Ipfs(StorageIpfsConfig),
+    Memory,
+    Moka(StorageMokaConfig),
+    Obs(StorageObsConfig),
+    Oss(StorageOssConfig),
+    S3(StorageS3Config),
+    Redis(StorageRedisConfig),
+    Webhdfs(StorageWebhdfsConfig),
+    Cos(StorageCosConfig),
+}
+```
+
+So I propose to expose services config to the users, allowing them to work on config structs directly and fetch the config at runtime.
+
+# Guide-level explanation
+
+First of all, we will add config struct for each service. For example, `Fs` will have a `FsConfig` struct and `S3` will have a `S3Config`. The fields within the config struct are public and marked as non-exhaustive.
+
+```rust
+#[non_exhaustive]
+pub struct S3Config {
+  pub root: Option<String>,
+  pub bucket: String,
+  pub endpoint: Option<String>,
+  pub region: Option<String>,
+  ...
+}
+```
+
+Then, we will add a `Config` enum that contains all the config structs. The enum is public and non-exhaustive too.
+
+```rust
+#[non_exhaustive]
+pub enum Config {
+  Fs(FsConfig)
+  S3(S3Config),
+  Custom(&'static str, HashMap<String, String>),
+}
+```
+
+Notably, a `Custom` variant will be added to the enum. This variant aligns with `Scheme::Custom(name)` and allows users to configure custom services.
+
+At `Operator` level, we will add `from_config` and `via_config` methods.
+
+```rust
+impl Operator {
+  pub fn via_config(cfg: impl Into<Config>) -> Result<Operator> {}
+}
+```
+
+Additionally, `OperatorInfo` will introduce a new API method, `config()`:
+
+```rust
+impl OperatorInfo {
+  pub fn config(&self) -> Config {}
+}
+```
+
+Users can use `config()` to fetch the config of a service at runtime and construct a new operator based on needs.
+
+# Reference-level explanation
+
+Every services will have a `XxxConfig` struct.
+
+`XxxConfig` will implement the following things:
+
+- `Default` trait: All config fields will have a default value.
+- `Deserialize` trait: Allow users to deserialize a config from a string.
+- `Into<Config>` trait: All service config can be converted to `Config` enum.
+
+Internally, `XxxConfig` will have the following traits:
+
+- `FromMap` trait: Allow users to build a config via hashmap which will replace existing `from_map` API in `Builder`.
+- `Into<XxxBuilder>` trait: Config can convert into corresponding builder with zero cost. 
+
+All config fields will be public and non-exhaustive, allowing users to build config this way:
+
+```rust
+let s3 = S3Config {
+  bucket: "test".to_string(),
+  endpoint: Some("http://localhost:9000".to_string()),
+  ..Default::default()
+}
+```
+
+The public API of existing builders will remain unchanged, although their internal implementations will be modified to utilize `XxxConfig`. Type that can't be represents as `String` like `Box<dyn AwsCredentialLoad>` and `HttpClient` will be kept in `Builder` as before.
+
+For example:
+
+```rust
+#[non_exhaustive]
+pub struct S3Config {
+  pub root: Option<String>,
+  pub bucket: String,
+  pub endpoint: Option<String>,
+  pub region: Option<String>,
+  ...
+}
+
+pub struct S3Builder {
+    config: S3Config,
+    
+    customized_credential_load: Option<Box<dyn AwsCredentialLoad>>,
+    http_client: Option<HttpClient>,
+}
+```
+
+# Drawbacks
+
+## API Surface
+
+This modification will significantly expand OpenDAL's public API surface, makes it harder to maintain and increases the risk of breaking changes. Also, this change will add much more work for bindings which need to implement `XxxConfig` for each service.
+
+## Secrets Leakage
+
+After our config supports `Serialize`, it's possible that users will serialize the config and log it. This will lead to secrets leakage. We should add a warning in the docs to prevent this. And we should also encourage uses of services like AWS IAM instead of static secrets.
+
+# Rationale and alternatives
+
+## Move `root` out of service config to operator level
+
+There is another way to solve the problem: [Move `root` out of service config to operator level](https://github.com/apache/opendal/issues/3151).
+
+We can move `root` out of the service config and put it in `Operator` level. This way, users can configure `root` for all services in one place. However, this is a large breaking changes and users will need to maintain the `root` logic everywhere.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Implement `FromStr` for `Config`
+
+We can implement `FromStr` for `Config` so that users can parse a config from a string.
+
+```rust
+let cfg = Config::from_str("s3://bucket/path/to/file?access_key_id=xxx&secret_access_key=xxx")?;
+```
+
+## Implement `Serialize` for `Config`
+
+We can implement `Serialize` for `Config` so that users can serialize a config.
+
+```rust
+// Serialize
+let bs = serde_json::to_vec(&cfg)?;
+// Deserialize
+let cfg: Config = serde_json::from_slice(&bs)?;
+```
+
+## Implement `check` for `Config`
+
+Implement check for config so that users can check if a config is valid before `build`.

data/core/src/docs/rfcs/3232_align_list_api.md

@@ -0,0 +1,69 @@
+- Proposal Name: `align_list_api`
+- Start Date: 2023-10-07
+- RFC PR: [apache/opendal#3232](https://github.com/apache/opendal/pull/3232)
+- Tracking Issue: [apache/opendal#3236](https://github.com/apache/opendal/issues/3236)
+
+# Summary
+
+Refactor internal `Page` API to `List` API.
+
+# Motivation
+
+OpenDAL's `Lister` is implemented by `Page`:
+
+```rust
+#[async_trait]
+pub trait Page: Send + Sync + 'static {
+    /// Fetch a new page of [`Entry`]
+    ///
+    /// `Ok(None)` means all pages have been returned. Any following call
+    /// to `next` will always get the same result.
+    async fn next(&mut self) -> Result<Option<Vec<Entry>>>;
+}
+```
+
+Each call to `next` will retrieve a page of `Entry` objects. This design is modeled after the `list_object` API used in object storage services. However, this design has several drawbacks:
+
+- Services like `fs`, `hdfs` needs to buffer the whole page in memory before returning it to the caller.
+- `Page` is not aligned with `opendal::Lister` make it hard to understand the code.
+- `Page` is not aligned with `Read` & `Write` which is poll based.
+
+# Guide-level explanation
+
+No user-facing changes.
+
+# Reference-level explanation
+
+We will rename `Page` to `List` and change the API to:
+
+```rust
+pub trait List: Send + Sync + 'static {
+    /// Fetch a new [`Entry`]
+    ///
+    /// `Ok(None)` means all entries have been returned. Any following call
+    /// to `next` will always get the same result.
+    fn poll_next(&mut self, cx: &mut Context<'_>) -> Result<Option<Entry>>;
+}
+```
+
+All `page` related code will be replaced by `list`.
+
+# Drawbacks
+
+Breaking changes for raw API.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/3243_list_prefix.md

@@ -0,0 +1,128 @@
+- Proposal Name: `list_prefix`
+- Start Date: 2023-10-08
+- RFC PR: [apache/opendal#3243](https://github.com/apache/opendal/pull/3243)
+- Tracking Issue: [apache/opendal#3247](https://github.com/apache/opendal/issues/3247)
+
+# Summary
+
+Allow users to specify a prefix and remove the requirement that the path must end with `/`.
+
+# Motivation
+
+OpenDAL uses `/` to distinguish between a file and a directory. This design is necessary for object storage services such as S3 and GCS, where both `abc` (file) and `abc/` (directory) can coexist. We require users to provide the correct path to the API. For instance, when using `read("abc/")`, it returns `IsADirectory`, whereas with `list("abc/")` it returns `NotADirectory`. This behavior may be perplexing for users.
+
+As a side-effect of this design, OpenDAL always return exist for `stat("not_exist/")` since there is no way for OpenDAL to check if `not_exist/file_example` is exist via `HeadObject` call.
+
+There are some issues and pull requests related to those issues.
+
+- [Invalid metadata for dir objects in s3](https://github.com/apache/opendal/issues/3199)
+- [`is_exist` always return true for key end with '/', in S3 service](https://github.com/apache/opendal/issues/2086)
+
+POSIX-like file systems also have their own issues, as they lack native support for listing a prefix.
+
+Give file tree like the following:
+
+```shell
+abc/
+abc/def
+abc/xyz/
+```
+
+Calling `list("ab")` will return `NotFound` after we removing the requirement that the path must end with `/`.
+
+So I propose the following changes of OpenDAL API behaviors:
+
+- Remove the requirement that the path for `list` must end with `/`.
+- Object storage services will use `list_object` API to check if a dir is exist.
+- Simulate the list prefix behavior for POSIX-like file systems.
+
+# Guide-level explanation
+
+Given the following file tree:
+
+```shell
+abc/
+abc/def_file
+abc/def_dir/
+abc/def_dir/xyz_file
+abc/def_dir/xyz_dir/
+```
+
+While listing a path:
+
+| Case                    | Path            | Result                              | Description                             |
+|-------------------------|-----------------|-------------------------------------|-----------------------------------------|
+| list dir                | `abc/`          | `abc/def_file` <br/> `abc/def_dir/` | children that matches the dir           |
+| list prefix             | `abc/def`       | `abc/def_file` <br/> `abc/def_dir/` | children that matches the prefix        |
+| list file               | `abc/def_file`  | `abc/def_file`                      | the only children that matches the path |
+| list dir without `/`    | `abc/def_dir`   | `abc/def_dir/`                      | the only children that matches the path |
+| list file ends with `/` | `abc/def_file/` | EMPTY                               | no children matches the dir             |
+| list not exist dir      | `def/`          | EMPTY                               | no children found matches the dir       |
+| list not exist file     | `def`           | EMPTY                               | no children found matches the prefix    |
+
+While listing a path with `delimiter` set to `""`:
+
+| Case                    | Path            | Result                                                                                        | Description                             |
+|-------------------------|-----------------|-----------------------------------------------------------------------------------------------|-----------------------------------------|
+| list dir                | `abc/`          | `abc/def_file` <br/> `abc/def_dir/` <br/> `abc/def_dir/xyz_file` <br/> `abc/def_dir/xyz_dir/` | children that matches the dir           |
+| list prefix             | `abc/def`       | `abc/def_file` <br/> `abc/def_dir/` <br/> `abc/def_dir/xyz_file` <br/> `abc/def_dir/xyz_dir/` | children that matches the prefix        |
+| list file               | `abc/def_file`  | `abc/def_file`                                                                                | the only children that matches the path |
+| list dir without `/`    | `abc/def_dir`   | `abc/def_dir/` <br/> `abc/def_dir/xyz_file` <br/> `abc/def_dir/xyz_dir/`                      | children that matches the path          |
+| list file ends with `/` | `abc/def_file/` | EMPTY                                                                                         | no children matches the dir             |
+| list not exist dir      | `def/`          | EMPTY                                                                                         | no children found matches the dir       |
+| list not exist file     | `def`           | EMPTY                                                                                         | no children found matches the prefix    |
+
+While stat a path:
+
+| Case                   | Path            | Result                                     |
+|------------------------|-----------------|--------------------------------------------|
+| stat existing dir      | `abc/`          | Metadata with dir mode                     | 
+| stat existing file     | `abc/def_file`  | Metadata with file mode                    | 
+| stat dir without `/`   | `abc/def_dir`   | Error `NotFound` or metadata with dir mode | 
+| stat file with `/`     | `abc/def_file/` | Error `NotFound`                           |
+| stat not existing path | `xyz`           | Error `NotFound`                           |
+
+While create dir on a path:
+
+| Case                        | Path   | Result                     |
+|-----------------------------|--------|----------------------------|
+| create dir on existing dir  | `abc/` | Ok                         |
+| create dir on existing file | `abc`  | Error with `NotADirectory` |
+| create dir with `/`         | `xyz/` | Ok                         |
+| create dir without `/`      | `xyz`  | Ok with `xyz/` created     |
+
+# Reference-level explanation
+
+For POSIX-like services, we will:
+
+- Simulate the list prefix behavior by listing the parent dir and filter the children that matches the prefix.
+- Return `NotFound` while stat an existing file with `/`
+
+For object storage services, we will:
+
+- Use `list_object` API while stat a path ends with `/`.
+  - Return dir metadata if the dir is exist or there is at least a children.
+  - Return `NotFound` if the dir is not exist and there is no children.
+- Check path before create dir with a path not ends with `/`.
+  - Return `NotADirectory` if the path is exist.
+  - Create the dir with `/` appended.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None