opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/docs/rfcs/0041_object_native_api.md

@@ -0,0 +1,185 @@
+- Proposal Name: `object_native_api`
+- Start Date: 2022-02-18
+- RFC PR: [apache/opendal#41](https://github.com/apache/opendal/pull/41)
+- Tracking Issue: [apache/opendal#35](https://github.com/apache/opendal/pull/35)
+
+# Summary
+
+Refactor API in object native way to make it easier to user.
+
+# Motivation
+
+`opendal` is not easy to use.
+
+In our early adoption project `databend`, we can see a lot of code looks like:
+
+```rust
+let data_accessor = self.data_accessor.clone();
+let path = self.path.clone();
+let reader = SeekableReader::new(data_accessor, path.as_str(), stream_len);
+let reader = BufReader::with_capacity(read_buffer_size as usize, reader);
+Self::read_column(reader, &col_meta, data_type.clone(), arrow_type.clone()).await
+```
+
+And
+
+```rust
+op.stat(&path).run().await
+```
+
+## Conclusion
+
+So in this proposal, I expect to address those problems. After implementing this proposal, we have a faster and easier-to-use `opendal`.
+
+# Guide-level explanation
+
+To operate on an object, we will use `Operator::object()` to create a new handler:
+
+```rust
+let o = op.object("path/to/file");
+```
+
+All operations that are available for `Object` for now includes:
+
+- `metadata`: get object metadata (return an error if not exist).
+- `delete`: delete an object.
+- `reader`: create a new reader to read data from this object.
+- `writer`: create a new writer to write data into this object.
+
+Here is an example:
+
+```rust
+use anyhow::Result;
+use futures::AsyncReadExt;
+
+use opendal::services::fs;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let op = Operator::new(fs::Backend::build().root("/tmp").finish().await?);
+
+    let o = op.object("test_file");
+
+    // Write data info file;
+    let w = o.writer();
+    let n = w
+        .write_bytes("Hello, World!".to_string().into_bytes())
+        .await?;
+    assert_eq!(n, 13);
+
+    // Read data from file;
+    let mut r = o.reader();
+    let mut buf = vec![];
+    let n = r.read_to_end(&mut buf).await?;
+    assert_eq!(n, 13);
+    assert_eq!(String::from_utf8_lossy(&buf), "Hello, World!");
+
+    // Get file's Metadata
+    let meta = o.metadata().await?;
+    assert_eq!(meta.content_length(), 13);
+
+    // Delete file.
+    o.delete().await?;
+
+    Ok(())
+}
+```
+
+# Reference-level explanation
+
+## Native Reader support
+
+We will provide a `Reader` (which implement both `AsyncRead + AsyncSeek`) for user instead of just a `AsyncRead`. In this `Reader`, we will:
+
+- Not maintain internal buffer: caller can decide to wrap into `BufReader`.
+- Only rely on accessor's `read` and `stat` operations.
+
+To avoid the extra cost for `stat`, we will:
+
+- Allow user specify total_size for `Reader`.
+- Lazily Send `stat` while the first time `SeekFrom::End()`
+
+To avoid the extra cost for `poll_read`, we will:
+
+- Keep the underlying `BoxedAsyncRead` open, so that we can reuse the same connection/fd.
+
+With these change, we can improve the `Reader` performance both on local fs and remote storage:
+
+- fs, before
+
+```shell
+Benchmarking fs/bench_read/64226295-b7a7-416e-94ce-666ac3ab037b:
+                        time:   [16.060 ms 17.109 ms 18.124 ms]
+                        thrpt:  [882.82 MiB/s 935.20 MiB/s 996.24 MiB/s]
+
+Benchmarking fs/bench_buf_read/64226295-b7a7-416e-94ce-666ac3ab037b:
+                        time:   [14.779 ms 14.857 ms 14.938 ms]
+                        thrpt:  [1.0460 GiB/s 1.0517 GiB/s 1.0572 GiB/s]
+```
+
+- fs, after
+
+```shell
+Benchmarking fs/bench_read/df531bc7-54c8-43b6-b412-e4f7b9589876:
+                        time:   [14.654 ms 15.452 ms 16.273 ms]
+                        thrpt:  [983.20 MiB/s 1.0112 GiB/s 1.0663 GiB/s]
+
+Benchmarking fs/bench_buf_read/df531bc7-54c8-43b6-b412-e4f7b9589876:
+                        time:   [5.5589 ms 5.5825 ms 5.6076 ms]
+                        thrpt:  [2.7864 GiB/s 2.7989 GiB/s 2.8108 GiB/s]
+```
+
+- s3, before
+
+```shell
+Benchmarking s3/bench_read/72025a81-a4b6-46dc-b485-8d875d23c3a5:
+                        time:   [4.8315 ms 4.9331 ms 5.0403 ms]
+                        thrpt:  [3.1000 GiB/s 3.1674 GiB/s 3.2340 GiB/s]
+
+Benchmarking s3/bench_buf_read/72025a81-a4b6-46dc-b485-8d875d23c3a5:
+                        time:   [16.246 ms 16.539 ms 16.833 ms]
+                        thrpt:  [950.52 MiB/s 967.39 MiB/s 984.84 MiB/s]
+```
+
+- s3, after
+
+```shell
+Benchmarking s3/bench_read/6971c464-15f7-48d6-b69c-c8abc7774802:
+                        time:   [4.4222 ms 4.5685 ms 4.7181 ms]
+                        thrpt:  [3.3117 GiB/s 3.4202 GiB/s 3.5333 GiB/s]
+
+Benchmarking s3/bench_buf_read/6971c464-15f7-48d6-b69c-c8abc7774802:
+                        time:   [5.5598 ms 5.7174 ms 5.8691 ms]
+                        thrpt:  [2.6622 GiB/s 2.7329 GiB/s 2.8103 GiB/s]
+```
+
+## Object API
+
+Other changes are just a re-order of APIs.
+
+- `Operator::read() -> BoxedAsyncRead` => `Object::reader() -> Reader`
+- `Operator::write(r: BoxedAsyncRead, size: u64)` => `Object::writer() -> Writer`
+- `Operator::stat() -> Object` => `Object::stat() -> Metadata`
+- `Operator::delete()` => `Object::delete()`
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- Implement `AsyncWrite` for `Writer` so that we can use `Writer` easier.
+- Implement `Operator::objects()` to return an object iterator.

data/core/src/docs/rfcs/0044_error_handle.md

@@ -0,0 +1,198 @@
+- Proposal Name: `error_handle`
+- Start Date: 2022-02-23
+- RFC PR: [apache/opendal#44](https://github.com/apache/opendal/pull/44)
+- Tracking Issue: [apache/opendal#43](https://github.com/apache/opendal/pull/43)
+
+# Summary
+
+Enhanced error handling for OpenDAL.
+
+# Motivation
+
+OpenDAL didn't handle errors correctly.
+
+```rust
+fn parse_unexpected_error<E>(_: SdkError<E>, path: &str) -> Error {
+    Error::Unexpected(path.to_string())
+}
+```
+
+Most time, we return a path that is meaningless for debugging.
+
+There are two issues about this shortcoming:
+
+- [error: Split ErrorKind and Context for error check easier](https://github.com/apache/opendal/issues/24)
+- [Improvement: provides more information about the cause of DalTransportError](https://github.com/apache/opendal/issues/29)
+
+First, we can't check `ErrorKind` quickly. We have to use `matches` for the help:
+
+```rust
+assert!(
+    matches!(
+        result.err().unwrap(),
+        opendal::error::Error::ObjectNotExist(_)
+    ),
+);
+```
+
+Then, we didn't bring enough information for users to debug what happened inside OpenDAL.
+
+So we must handle errors correctly, so that:
+
+- We can check the `Kind` to know what error happened.
+- We can read `context` to know more details.
+- We can get the source of this error to know more details.
+
+# Guide-level explanation
+
+Now we are trying to get an object's metadata:
+
+```rust
+let meta = o.metadata().await;
+```
+
+Unfortunately, the `Object` does not exist, so we can check out what happened.
+
+```rust
+if let Err(e) = meta {
+    if e.kind() == Kind::ObjectNotExist {
+        // Handle this error
+    }
+}
+```
+
+It's possible that we don't care about other errors. It's OK to log it out:
+
+```rust
+if let Err(e) = meta {
+    if e.kind() == Kind::ObjectNotExist {
+        // Handle this error
+    } else {
+        error!("{e}");
+    }
+}
+```
+
+For a backend implementer, we can provide as much information as possible. For example, we can return `bucket is empty` to let the user know:
+
+```rust
+return Err(Error::Backend {
+    kind: Kind::BackendConfigurationInvalid,
+    context: HashMap::from([("bucket".to_string(), "".to_string())]),
+    source: anyhow!("bucket is empty"),
+});
+```
+
+Or, we can return an underlying error to let users figure out:
+
+```rust
+Error::Object {
+    kind: Kind::Unexpected,
+    op,
+    path: path.to_string(),
+    source: anyhow::Error::from(err),
+}
+```
+
+So our application users will get enough information now:
+
+```shell
+Object { kind: ObjectNotExist, op: "stat", path: "/tmp/998e4dec-c84b-4164-a7a1-1f140654934f", source: No such file or directory (os error 2) }
+```
+
+
+# Reference-level explanation
+
+We will split `Error` into `Error` and `Kind`.
+
+`Kind` is an enum organized by different categories.
+
+Every error will map to a kind, which will be in the error message.
+
+```rust
+pub enum Kind {
+    #[error("backend not supported")]
+    BackendNotSupported,
+    #[error("backend configuration invalid")]
+    BackendConfigurationInvalid,
+
+    #[error("object not exist")]
+    ObjectNotExist,
+    #[error("object permission denied")]
+    ObjectPermissionDenied,
+
+    #[error("unexpected")]
+    Unexpected,
+}
+```
+
+In `Error`, we will have different struct to carry different contexts:
+
+```rust
+pub enum Error {
+    #[error("{kind}: (context: {context:?}, source: {source})")]
+    Backend {
+        kind: Kind,
+        context: HashMap<String, String>,
+        source: anyhow::Error,
+    },
+
+    #[error("{kind}: (op: {op}, path: {path}, source: {source})")]
+    Object {
+        kind: Kind,
+        op: &'static str,
+        path: String,
+        source: anyhow::Error,
+    },
+
+    #[error("unexpected: (source: {0})")]
+    Unexpected(#[from] anyhow::Error),
+}
+```
+
+Every one of them will carry a source: `anyhow::Error` so that users can get the complete picture of this error. We have implemented `Error::kind()`, other helper functions are possible, but they are out of this RFC's scope.
+
+```rust
+pub fn kind(&self) -> Kind {
+    match self {
+        Error::Backend { kind, .. } => *kind,
+        Error::Object { kind, .. } => *kind,
+        Error::Unexpected(_) => Kind::Unexpected,
+    }
+}
+```
+
+The implementer should do their best to carry as much context as possible. Such as, they should return `Error::Object` to carry the `op` and `path`, instead of just returns `Error::Unexpected(anyhow::Error::from(err))`.
+
+```rust
+Error::Object {
+    kind: Kind::Unexpected,
+    op,
+    path: path.to_string(),
+    source: anyhow::Error::from(err),
+}
+```
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+## Why don't we implement `backtrace`?
+
+`backtrace` is not stable yet, and `OpenDAL` must be compilable on stable Rust.
+
+This proposal doesn't erase the possibility to add support once `backtrace` is stable.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- `Backtrace` support.

data/core/src/docs/rfcs/0057_auto_region.md

@@ -0,0 +1,160 @@
+- Proposal Name: `auto_region`
+- Start Date: 2022-02-24
+- RFC PR: [apache/opendal#57](https://github.com/apache/opendal/pull/57)
+- Tracking Issue: [apache/opendal#58](https://github.com/apache/opendal/issues/58)
+
+# Summary
+
+Automatically detecting user's s3 region.
+
+# Motivation
+
+Current behavior for `region` and `endpoint` is buggy. `endpoint=https://s3.amazonaws.com` and `endpoint=""` are expected to be the same, because `endpoint=""` means take the default value `https://s3.amazonaws.com`. However, they aren't.
+
+S3 SDK has a mechanism to construct the correct API endpoint. It works like `format!("s3.{}.amazonaws.com", region)` internally. But if we specify the endpoint to `https://s3.amazonaws.com`, SDK will take this endpoint static.
+
+So users could meet errors like:
+
+```shell
+attempting to access must be addressed using the specified endpoint
+```
+
+Automatically detecting the user's s3 region will help resolve this problem. Users don't need to care about the region anymore, `OpenDAL` will figure it out. Everything works regardless of whether the input is `s3.amazonaws.com` or `s3.us-east-1.amazonaws.com`.
+
+# Guide-level explanation
+
+`OpenDAL` will remove `region` option, and users only need to set the `endpoint` now.
+
+Valid input including:
+
+- `https://s3.amazonaws.com`
+- `https://s3.us-east-1.amazonaws.com`
+- `https://oss-ap-northeast-1.aliyuncs.com`
+- `http://127.0.0.1:9000`
+
+`OpenDAL` will handle the `region` internally and automatically.
+
+# Reference-level explanation
+
+S3 services support mechanism to indicate the correct region on itself.
+
+Sending a `HEAD` request to `<endpoint>/<bucket>` will get a response like:
+
+```shell
+:) curl -I https://s3.amazonaws.com/databend-shared
+HTTP/1.1 301 Moved Permanently
+x-amz-bucket-region: us-east-2
+x-amz-request-id: NPYSWK7WXJD1KQG7
+x-amz-id-2: 3FJSJ5HACKqLbeeXBUUE3GoPL1IGDjLl6SZx/fw2MS+k0GND0UwDib5YQXE6CThiQxpYBWZjgxs=
+Content-Type: application/xml
+Date: Thu, 24 Feb 2022 05:15:13 GMT
+Server: AmazonS3
+```
+
+`x-amz-bucket-region: us-east-2` will be returned, and we can use this region to construct the correct endpoint for this bucket:
+
+```shell
+:) curl -I https://s3.us-east-2.amazonaws.com/databend-shared
+HTTP/1.1 403 Forbidden
+x-amz-bucket-region: us-east-2
+x-amz-request-id: 98CN5MYV3GQ1XMPY
+x-amz-id-2: Tdxy36bRRP21Oip18KMQ7FG63MTeXOpXdd5/N3izFH0oalPODVaRlpCkDU3oUN0HIE24/ezX5Dc=
+Content-Type: application/xml
+Date: Thu, 24 Feb 2022 05:16:57 GMT
+Server: AmazonS3
+```
+
+It also works for S3 compilable services like minio:
+
+```shell
+# Start minio with `MINIO_SITE_REGION` configured
+:) MINIO_SITE_REGION=test minio server .
+# Sending request to minio bucket
+:) curl -I 127.0.0.1:9900/databend
+HTTP/1.1 403 Forbidden
+Accept-Ranges: bytes
+Content-Length: 0
+Content-Security-Policy: block-all-mixed-content
+Server: MinIO
+Strict-Transport-Security: max-age=31536000; includeSubDomains
+Vary: Origin
+Vary: Accept-Encoding
+X-Amz-Bucket-Region: test
+X-Amz-Request-Id: 16D6A12DCA57E0FA
+X-Content-Type-Options: nosniff
+X-Xss-Protection: 1; mode=block
+Date: Thu, 24 Feb 2022 05:18:51 GMT
+```
+
+We can use this mechanism to detect `region` automatically. The algorithm works as follows:
+
+- If `endpoint` is empty, fill it will `https://s3.amazonaws.com` and the corresponding template: `https://s3.{region}.amazonaws.com`.
+- Sending a `HEAD` request to `<endpoint>/<bucket>`.
+- If got `200` or `403` response, the endpoint works.
+  - Use this endpoint directly without filling the template.
+  - Take the header `x-amz-bucket-region` as the region to fill the endpoint.
+  - Use the fallback value `us-east-1` to make SDK happy if the header not exists.
+- If got a `301` response, the endpoint needs construction.
+  - Take the header `x-amz-bucket-region` as the region to fill the endpoint.
+  - Return an error to the user if not exist.
+- If got `404`, the bucket could not exist, or the endpoint is incorrect.
+  - Return an error to the user.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+## Use virtual style `<bucket>.<endpoint>`?
+
+The virtual style works too. But not all services support this kind of API endpoint. For example, using `http://testbucket.127.0.0.1` is wrong, and we need to do extra checks.
+
+Using `<endpoint>/<bucket>` makes everything easier.
+
+## Use `ListBuckets` API?
+
+`ListBuckets` requires higher permission than normal bucket read and write operations. It's better to finish the job without requesting more permission. 
+
+## Misbehavior S3 Compilable Services
+
+Many services didn't implement S3 API correctly.
+
+Aliyun OSS will return `404` for every bucket:
+
+```shell
+:) curl -I https://aliyuncs.com/<my-existing-bucket>
+HTTP/2 404
+date: Thu, 24 Feb 2022 05:32:57 GMT
+content-type: text/html
+content-length: 690
+ufe-result: A6
+set-cookie: thw=cn; Path=/; Domain=.taobao.com; Expires=Fri, 24-Feb-23 05:32:57 GMT;
+server: Tengine/Aserver
+```
+
+QingStor Object Storage will return `307` with the `Location` header:
+
+```shell
+:) curl -I https://s3.qingstor.com/community
+HTTP/1.1 301 Moved Permanently
+Server: nginx/1.13.6
+Date: Thu, 24 Feb 2022 05:33:55 GMT
+Connection: keep-alive
+Location: https://pek3a.s3.qingstor.com/community
+X-Qs-Request-Id: 05b83b615c801a3d
+```
+
+In this proposal, we will not figure them out. It's easier for the user to fill the correct endpoint instead of automatically detecting them.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/0069_object_stream.md

@@ -0,0 +1,145 @@
+- Proposal Name: `object_stream`
+- Start Date: 2022-02-25
+- RFC PR: [apache/opendal#69](https://github.com/apache/opendal/pull/69)
+- Tracking Issue: [apache/opendal#69](https://github.com/apache/opendal/issues/69)
+
+# Summary
+
+Allow user to read dir via `ObjectStream`.
+
+# Motivation
+
+Users need `readdir` support in `OpenDAL`: [Implement List support](https://github.com/apache/opendal/issues/12). Take [databend] for example, with `List` support, we can implement copy from `s3://bucket/path/to/dir` instead of only `s3://bucket/path/to/file`.
+
+# Guide-level explanation
+
+`Operator` supports new action called `objects("path/to/dir")` which returns a `ObjectStream`, we can iterator current dir like `std::fs::ReadDir`:
+
+```rust
+let mut obs = op.objects("").map(|o| o.expect("list object"));
+while let Some(o) = obs.next().await {
+    // Do something upon `Object`.
+}
+```
+
+To better support different file modes, there is a new object meta called `ObjectMode`:
+
+```rust
+let meta = o.metadata().await?;
+let mode = meta.mode();
+if mode.contains(ObjectMode::FILE) {
+    // Do something on a file object.
+} else if mode.contains(ObjectMode::DIR) {
+    // Do something on a dir object.
+}
+```
+
+We will try to cache some object metadata so that users can reduce `stat` calls:
+
+```rust
+let meta = o.metadata_cached().await?;
+```
+
+`o.metadata_cached()` will return local cached metadata if available.
+
+# Reference-level explanation
+
+First, we will add a new API in `Accessor`:
+
+```rust
+pub type BoxedObjectStream = Box<dyn futures::Stream<Item = Result<Object>> + Unpin + Send>;
+
+async fn list(&self, args: &OpList) -> Result<BoxedObjectStream> {
+    let _ = args;
+    unimplemented!()
+}
+```
+
+To support options in the future, we will wrap this call via `ObjectStream`:
+
+```rust
+pub struct ObjectStream {
+    acc: Arc<dyn Accessor>,
+    path: String,
+
+    state: State,
+}
+
+enum State {
+    Idle,
+    Sending(BoxFuture<'static, Result<BoxedObjectStream>>),
+    Listing(BoxedObjectStream),
+}
+```
+
+So the public API to end-users will be:
+
+```rust
+impl Operator {
+    pub fn objects(&self, path: &str) -> ObjectStream {
+        ObjectStream::new(self.inner(), path)
+    }
+}
+```
+
+For cached metadata support, we will add a flag in `Metadata`:
+
+```rust
+#[derive(Debug, Clone, Default)]
+pub struct Metadata {
+    complete: bool,
+
+    path: String,
+    mode: Option<ObjectMode>,
+
+    content_length: Option<u64>,
+}
+```
+
+And add new API `Objbct::metadata_cached()`:
+
+```rust
+pub async fn metadata_cached(&mut self) -> Result<&Metadata> {
+    if self.meta.complete() {
+        return Ok(&self.meta);
+    }
+
+    let op = &OpStat::new(self.meta.path());
+    self.meta = self.acc.stat(op).await?;
+
+    Ok(&self.meta)
+}
+```
+
+The backend implementer must make sure `complete` is correctly set.
+
+`Metadata` will be immutable outsides, so all `set_xxx` APIs will be set to crate public only:
+
+```rust
+pub(crate) fn set_content_length(&mut self, content_length: u64) -> &mut Self {
+    self.content_length = Some(content_length);
+    self
+}
+```
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- More precise field-level metadata cache so that user can send `stat` only when needed.
+
+[databend]: https://github.com/datafuselabs/databend