opendal 0.1.6.pre.rc.1

data/core/src/docs/rfcs/0221_create_dir.md

@@ -0,0 +1,89 @@
+- Proposal Name: `create-dir`
+- Start Date: 2022-04-06
+- RFC PR: [apache/opendal#221](https://github.com/apache/opendal/pull/221)
+- Tracking Issue: [apache/opendal#222](https://github.com/apache/opendal/issues/222)
+
+# Summary
+
+Add creating dir support for OpenDAL.
+
+# Motivation
+
+Interoperability between OpenDAL services requires dir support. The object storage system will simulate dir operations with `/` via object ends. But we can't share the same behavior with `fs`, as `mkdir` is a separate syscall.
+
+So we need to unify the behavior about dir across different services.
+
+# Guide-level explanation
+
+After this proposal got merged, we will treat all paths that end with `/` as a dir.
+
+For example:
+
+- `read("abc/")` will return an `IsDir` error.
+- `write("abc/")` will return an `IsDir` error.
+- `stat("abc/")` will be guaranteed to return a dir or a `NotDir` error.
+- `delete("abc/")` will be guaranteed to delete a dir or `NotDir` / `NotEmpty` error.
+- `list("abc/")` will be guaranteed to list a dir or a `NotDir` error.
+
+And we will support create an empty object:
+
+```rust
+// create a dir object "abc/"
+let _ = op.object("abc/").create().await?;
+// create a file object "abc"
+let _ = op.object("abc").create().await?;
+```
+
+# Reference-level explanation
+
+And we will add a new API called `create` to create an empty object.
+
+```rust
+struct OpCreate {
+    path: String,
+    mode: ObjectMode,
+}
+
+pub trait Accessor: Send + Sync + Debug {
+    async fn create(&self, args: &OpCreate) -> Result<Metadata>;
+}
+```
+
+`Object` will expose API like `create` which will call `Accessor::create()` internally.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+When writing this proposal, [io_error_more](https://github.com/rust-lang/rust/issues/86442) is not stabilized yet. We can't use `NotADirectory` nor `IsADirectory` directly.
+
+Using `from_raw_os_error` is unacceptable because we can't carry our error context.
+
+```rust
+use std::io;
+
+let error = io::Error::from_raw_os_error(22);
+assert_eq!(error.kind(), io::ErrorKind::InvalidInput);
+```
+
+So we will use `ErrorKind::Other` for now, which means our users can't check the following errors:
+
+- `IsADirectory`
+- `DirectoryNotEmpty`
+- `NotADirectory`
+
+Until they get stabilized.
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/0247_retryable_error.md

@@ -0,0 +1,87 @@
+- Proposal Name: `retryable_error`
+- Start Date: 2022-04-12
+- RFC PR: [apache/opendal#247](https://github.com/apache/opendal/pull/247)
+- Tracking Issue: [apache/opendal#248](https://github.com/apache/opendal/issues/248)
+
+# Summary
+
+Treat `io::ErrorKind::Interrupt` as retryable error.
+
+# Motivation
+
+Supports retry make our users' lives easier:
+
+> [Feature request: Custom retries for the s3 backend](https://github.com/apache/opendal/issues/196)
+>
+> While the reading/writing from/to s3, AWS occasionally returns errors that could be retried (at least 5xx?). Currently, in the databend, this will fail the whole execution of the statement (which may have been running for an extended time).
+
+Most users may need this retry feature, like `decompress`. Implementing it in OpenDAL will make users no bother, no backoff logic.
+
+# Guide-level explanation
+
+With the `retry` feature enabled:
+
+```toml
+opendal = {version="0.5.2", features=["retry"]}
+```
+
+Users can configure the retry behavior easily:
+
+```rust
+let backoff = ExponentialBackoff::default();
+let op = op.with_backoff(backoff);
+```
+
+All requests sent by `op` will be automatically retried.
+
+# Reference-level explanation
+
+We will implement retry features via adding a new `Layer`.
+
+In the retry layer, we will support retrying all operations. To do our best to keep retrying read & write, we will implement `RetryableReader` and `RetryableWriter`, which will support retry while no actual IO happens.
+
+## Retry operations
+
+Most operations are safe to retry, like `list`, `stat`, `delete` and `create`.
+
+We will retry those operations via input backoff.
+
+## Retry IO operations
+
+Retry IO operations are a bit complex because IO operations have side effects, especially for HTTP-based services like s3. We can't resume an operation during the reading process without sending new requests.
+
+This proposal will do the best we can: retry the operation if no actual IO happens.
+
+If we meet an internal error before reading/writing the user's buffer, it's safe and cheap to retry it with precisely the same argument.
+
+## Retryable Error
+
+- Operator MAY retry `io::ErrorKind::Interrupt` errors.
+- Services SHOULD return `io::ErrorKind::Interrupt` kind if the error is retryable.
+
+# Drawbacks
+
+## Write operation can't be retried
+
+As we return `Writer` to users, there is no way for OpenDAL to get the input data again.
+
+# Rationale and alternatives
+
+## Implement retry at operator level
+
+We need to implement retry logic for every operator function, and can't address the same problem:
+
+- `Reader` / `Writer` can't be retired.
+- Intrusive design that users cannot expand on their own
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+- `read` and `write` can't be retried during IO.
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/0293_object_id.md

@@ -0,0 +1,67 @@
+- Proposal Name: `object_id`
+- Start Date: 2022-05-27
+- RFC PR: [apache/opendal#293](https://github.com/apache/opendal/pull/293)
+- Tracking Issue: [apache/opendal#294](https://github.com/apache/opendal/issues/294)
+
+# Summary
+
+Allow getting id from an object.
+
+# Motivation
+
+Allow get id from an object will make it possible to operate across different operators. Users can store objects' IDs locally and refer to them with different settings. This proposal will make tasks like backup, restore, and migration possible.
+
+# Guide-level explanation
+
+Users can fetch an object id via:
+
+```rust
+let o = op.object("test_object");
+let id = o.id();
+```
+
+The id is unique and permanent inside the underlying storage.
+
+For example, if we have an s3 bucket with the root `/workdir/`, the object's id `test_object` will be `/workdir/test_object`.
+
+# Reference-level explanation
+
+`id()` and `path()` will be added as functions of `object`:
+
+```rust
+impl Object {
+    pub fn id(&self) -> String {}
+    pub fn path(&self) -> String {}
+}
+```
+
+- `path` is a re-export of call to `Metadata::path()`.
+- `id` will be generated by Operator's root and `Metadata::path()`.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+## Why not add a new field in `Metadata`?
+
+Adding a new field inside `Metadata` requires every service to handle the id separately. And every metadata will need to store a complete id with the operators' root.
+
+## Why not provide a full URI like `s3://path/to/object`?
+
+Because we can't.
+
+A full and functional URI towards an object will need the operator's endpoint and credentials. It's better to provide the mechanism and allow users to construct them based on their own business.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/0337_dir_entry.md

@@ -0,0 +1,191 @@
+- Proposal Name: `dir_entry`
+- Start Date: 2022-06-08
+- RFC PR: [apache/opendal#337](https://github.com/apache/opendal/pull/337)
+- Tracking Issue: [apache/opendal#338](https://github.com/apache/opendal/issues/338)
+
+# Summary
+
+Returning `DirEntry` instead of `Object` in list.
+
+# Motivation
+
+In [Object Stream](./0069-object-stream.md), we introduce read_dir support via:
+
+```rust
+pub trait ObjectStream: futures::Stream<Item = Result<Object>> + Unpin + Send {}
+impl<T> ObjectStream for T where T: futures::Stream<Item = Result<Object>> + Unpin + Send {}
+
+pub struct Object {
+    acc: Arc<dyn Accessor>,
+    meta: Metadata,
+}
+```
+
+However, the `meta` inside `Object` is not well-used:
+
+```rust
+pub(crate) fn metadata_ref(&self) -> &Metadata {}
+pub(crate) fn metadata_mut(&mut self) -> &mut Metadata {}
+pub async fn metadata_cached(&mut self) -> Result<&Metadata> {}
+```
+
+Users can't know an object's mode after the list, so they have to send `metadata` every time they get an object:
+
+```rust
+let o = op.object("path/to/dir/");
+let mut obs = o.list().await?;
+// ObjectStream implements `futures::Stream`
+while let Some(o) = obs.next().await {
+    let mut o = o?;
+    // It's highly possible that OpenDAL already did metadata during list.
+    // Use `Object::metadata_cached()` to get cached metadata at first.
+    let meta = o.metadata_cached().await?;
+    match meta.mode() {
+        ObjectMode::FILE => {
+            println!("Handling file")
+        }
+        ObjectMode::DIR => {
+            println!("Handling dir like start a new list via meta.path()")
+        }
+        ObjectMode::Unknown => continue,
+    }
+}
+```
+
+This behavior doesn't make sense as we already know the object's mode after the list.
+
+Introducing a separate `DirEntry` could reduce an extra call for metadata most of the time.
+
+```rust
+let o = op.object("path/to/dir/");
+let mut ds = o.list().await?;
+// ObjectStream implements `futures::Stream`
+while let Some(de) = ds.try_next().await {
+    match de.mode() {
+        ObjectMode::FILE => {
+            println!("Handling file")
+        }
+        ObjectMode::DIR => {
+            println!("Handling dir like start a new list via meta.path()")
+        }
+        ObjectMode::Unknown => continue,
+    }
+}
+```
+
+# Guide-level explanation
+
+Within this RFC, `Object::list()` will return `DirStreamer` instead.
+
+```rust
+pub trait DirStream: futures::Stream<Item = Result<DirEntry>> + Unpin + Send {}
+pub type DirStreamer = Box<dyn DirStream>;
+```
+
+`DirStreamer` will stream `DirEntry`, which carries information already known during the list. So we can:
+
+```rust
+let id = de.id();
+let path = de.path();
+let name = de.name();
+let mode = de.mode();
+let meta = de.metadata().await?;
+```
+
+With `DirEntry` support, we can reduce an extra `metadata` call if we only want to know the object's mode:
+
+```rust
+let o = op.object("path/to/dir/");
+let mut ds = o.list().await?;
+// ObjectStream implements `futures::Stream`
+while let Some(de) = ds.try_next().await {
+    match de.mode() {
+        ObjectMode::FILE => {
+            println!("Handling file")
+        }
+        ObjectMode::DIR => {
+            println!("Handling dir like start a new list via meta.path()")
+        }
+        ObjectMode::Unknown => continue,
+    }
+}
+```
+
+We can convert this `DirEntry` into `Object` without overhead:
+
+```rust
+let o = de.into();
+```
+
+# Reference-level explanation
+
+This proposal will introduce a new struct, `DirEntry`:
+
+```rust
+struct DirEntry {}
+
+impl DirEntry {
+    pub fn id() -> String {}
+    pub fn path() -> &str {}
+    pub fn name() -> &str {}
+    pub fn mode() -> ObjectMode {}
+    pub async fn metadata() -> ObjectMetadata {}
+}
+
+impl From<DirEntry> for Object {}
+```
+
+And use `DirStream` to replace `ObjectStream`:
+
+```rust
+pub trait DirStream: futures::Stream<Item = Result<DirEntry>> + Unpin + Send {}
+pub type DirStreamer = Box<dyn DirStream>;
+```
+
+With the addition of `DirEntry`, we will remove `meta` from `Object`:
+
+```rust
+#[derive(Clone, Debug)]
+pub struct Object {
+    acc: Arc<dyn Accessor>,
+    path: String,
+}
+```
+
+After this change, `Object` will become a thin wrapper of `Accessor` with path. And metadata related APIs like `metadata_ref()` and `metadata_mut()` will also be removed.
+
+# Drawbacks
+
+We are adding a new concept to our core logic.
+
+# Rationale and alternatives
+
+## Rust fs API design
+
+Rust also provides abstractions like `File` and `DirEntry`:
+
+```rust
+use std::fs;
+
+fn main() -> std::io::Result<()> {
+    for entry in fs::read_dir(".")? {
+        let dir = entry?;
+        println!("{:?}", dir.path());
+    }
+    Ok(())
+}
+```
+
+Users can open a file with `entry.path()`.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/0409_accessor_capabilities.md

@@ -0,0 +1,67 @@
+- Proposal Name: `accessor_capabilities`
+- Start Date: 2022-06-29
+- RFC PR: [apache/opendal#409](https://github.com/apache/opendal/pull/409)
+- Tracking Issue: [apache/opendal#410](https://github.com/apache/opendal/issues/410)
+
+# Summary
+
+Add support for accessor capabilities so that users can check if a given accessor is capable of a given ability.
+
+# Motivation
+
+Users of OpenDAL are requesting advanced features like the following:
+
+- [Support parallel upload object](https://github.com/apache/opendal/issues/256)
+- [Add presign url support](https://github.com/apache/opendal/issues/394)
+
+It's meaningful for OpenDAL to support them in a unified way. Of course, not all storage services have the same feature sets. OpenDAL needs to provide a way for users to check if a given accessor is capable of a given capability.
+
+# Guide-level explanation
+
+Users can check an `Accessor`'s capability via `Operator::metadata()`.
+
+```rust
+let meta = op.metadata();
+let _: bool = meta.can_presign();
+let _: bool = meta.can_multipart(); 
+```
+
+`Accessor` will return [`io::ErrorKind::Unsupported`](https://doc.rust-lang.org/stable/std/io/enum.ErrorKind.html#variant.Unsupported) for not supported operations instead of panic as `unimplemented()`.
+
+Users can check before operations or the `Unsupported` error kind after operations.
+
+# Reference-level explanation
+
+We will introduce a new enum called `AccessorCapability`, which includes `AccessorMetadata`.
+
+This enum is private and only accessible inside OpenDAL, so it's not part of our public API. We will expose the check API via `AccessorMetadata`:
+
+```rust
+impl AccessorMetadata {
+    pub fn can_presign(&self) -> bool { .. }
+    pub fn can_multipart(&self) -> bool { .. }
+}
+```
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+## go-storage
+
+- [GSP-109: Redesign Features](https://github.com/beyondstorage/go-storage/blob/master/docs/rfcs/109-redesign-features.md)
+- [GSP-837: Support Feature Flag](https://github.com/beyondstorage/go-storage/blob/master/docs/rfcs/837-support-feature-flag.md)
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/0413_presign.md

@@ -0,0 +1,154 @@
+- Proposal Name: `presign`
+- Start Date: 2022-06-30
+- RFC PR: [apache/opendal#0413](https://github.com/apache/opendal/pull/413)
+- Tracking Issue: [apache/opendal#394](https://github.com/apache/opendal/issues/394)
+
+# Summary
+
+Add presign support in OpenDAL so users can generate a pre-signed URL without leaking `serect_key`.
+
+# Motivation
+
+> By default, all S3 objects are private. Only the object owner has permission to access them. However, the object owner can optionally share objects with others by creating a presigned URL, using their own security credentials, to grant time-limited permission to download the objects.
+>
+> From [Sharing objects using presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html)
+
+We can use this presigned URL for:
+
+- Download the object within the expired time from a bucket directly
+- Upload content to the bucket on client-side
+
+Adding this feature in OpenDAL will make users' lives easier to generate presigned URLs across different storage services.
+
+The whole process would be:
+
+```text
+            ┌────────────┐
+            │    User    ├─────────────────────┐
+            └──┬─────▲───┘                     │
+               │     │      4. Send Request to S3 Directly
+1. Request Resource  │                         │
+               │     │                         │
+               │     │                         │
+               │     │                         ▼
+               │  3. Return Request     ┌────────────┐
+               │     │                  │            │
+               │     │                  │     S3     │
+            ┌──▼─────┴───┐              │            │
+            │            │              └────────────┘
+            │     App    │
+            │ ┌────────┐ │
+            │ │ OpenDAL│ │
+            │ ├────────┤ │
+            └─┴──┼─────┴─┘
+                 │      ▲
+                 └──────┘
+          2. Generate Request
+```
+
+# Guide-level explanation
+
+With this feature, our users can:
+
+## Generate presigned URL for downloading
+
+```rust
+let req = op.presign_read("path/to/file")?;
+// req.method: GET
+// req.url: https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
+```
+
+Users can download this object directly from the s3 bucket. For example:
+
+```shell
+curl <generated_url> -O test.txt
+```
+
+## Generate presigned URL for uploading
+
+```rust
+let req = op.presign_write("path/to/file")?;
+// req.method: PUT
+// req.url: https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
+```
+
+Users can upload content directly to the s3 bucket. For example:
+
+```shell
+curl -X PUT <generated_url> -T "/tmp/test.txt"
+```
+
+# Reference-level explanation
+
+`Accessor` will add a new API `presign`:
+
+```rust
+pub trait Accessor {
+    fn presign(&self, args: &OpPresign) -> Result<PresignedRequest> {..}
+}
+```
+
+`presign` accepts `OpPresign` and returns `Result<PresignedRequest>`:
+
+```rust
+struct OpPresign {
+    path: String,
+    op: Operation,
+    expire: time::Duration,
+}
+
+struct PresignedRequest {}
+
+impl PresignedRequest {
+    pub fn method(&self) -> &http::Method {..}
+    pub fn url(&self) -> &http::Uri {..}
+}
+```
+
+We are building a new struct to avoid leaking underlying implementations like `hyper::Request<T>` to users.
+
+This feature will be a new capability in `AccessorCapability` as described in [RFC-0409: Accessor Capabilities](./0409-accessor-capabilities.md)
+
+Based on `Accessor::presign`, we will export public APIs in `Operator`:
+
+```rust
+impl Operator {
+    fn presign_read(&self, path: &str) -> Result<PresignedRequest> {}
+    fn presign_write(&self, path: &str) -> Result<PresignedRequest> {}
+}
+```
+
+Although it's possible to generate URLs for `create`, `delete`, `stat`, and `list`, there are no obvious use-cases. So we will not add them to this proposal.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+## Query Sign Support Status
+
+- s3: [Authenticating Requests: Using Query Parameters (AWS Signature Version 4)](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html)
+- azblob: [Delegate access with a shared access signature](https://docs.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature)
+- gcs: [Signed URLs](https://cloud.google.com/storage/docs/access-control/signed-urls) (Only for XML API)
+
+# Prior art
+
+## awscli presign
+
+AWS CLI has native presign support
+
+```shell
+> aws s3 presign s3://DOC-EXAMPLE-BUCKET/test2.txt
+https://DOC-EXAMPLE-BUCKET.s3.us-west-2.amazonaws.com/key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAEXAMPLE123456789%2F20210621%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210621T041609Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=EXAMBLE1234494d5fba3fed607f98018e1dfc62e2529ae96d844123456
+```
+
+Refer to [AWS CLI Command Reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html) for more information.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+- Add `stat`/`list`/`delete` support