opendal 0.1.6.pre.rc.1

data/core/src/docs/rfcs/1398_query_based_metadata.md

@@ -0,0 +1,125 @@
+- Proposal Name: `query_based_metadata`
+- Start Date: 2022-02-22
+- RFC PR: [apache/opendal#1398](https://github.com/apache/opendal/pull/1398)
+- Tracking Issue: [apache/opendal#1398](https://github.com/apache/opendal/pull/1398)
+
+# Summary
+
+Read cached metadata based on users query.
+
+# Motivation
+
+OpenDAL has native metadata cache for now:
+
+```rust
+let _ = o.metadata().await?;
+// This call doesn't need to send a request.
+let _ = o.metadata().await?;
+```
+
+Also, OpenDAL can reuse metadata from `list` or `scan`:
+
+```rust
+let mut ds = o.scan().await?;
+while let Some(de) = ds.try_next().await? {
+    // This call doesn't need to send a request (if we are lucky enough).
+    let _ = de.metadata().await?;
+}
+```
+
+By reusing metadata from `list` or `scan` we can reduce the extra `stat` call for each object. In our real use cases, we can reduce the total time to calculate the total length inside a dir with 6k files from 4 minutes to 2 seconds.
+
+However, metadata can only be cached as a whole. If services could return more metadata in `stat` than in `list`, we wouldn't be able to mark the metadata as cacheable. If services add more metadata, we could inadvertently introduce the performance degradation.
+
+RFC [Object Metadataer](./rfc_1391_object_metadataer) intends to add `ObjectMetadataer` to address this issue. But it sooner to be proved that a failure: it's hard to design a correct API.
+
+Users have to write code like the following:
+
+```rust
+let om = o.metadata().await?;
+let _ = om.content_length().await?;
+let _ = om.content_md5().await?;
+```
+
+In this RFC, we will add a query based metadata.
+
+# Guide-level explanation
+
+After this RFC, `o.metadata()` will accept a query composed by `ObjectMetadataKey`.
+
+To query already cached metadata:
+
+```rust
+let meta = op.object("test").metadata(None).await?;
+let _ = meta.content_length();
+let _ = meta.content_type();
+```
+
+To query content length and content type:
+
+```rust
+let meta = op
+    .object("test")
+    .metadata({
+        use ObjectMetadataKey::*;
+        ContentLength | ContentType
+    })
+    .await?;
+let _ = meta.content_length();
+let _ = meta.content_type();
+```
+
+To query all metadata about this object:
+
+```rust
+let meta = op
+    .object("test")
+    .metadata({ ObjectMetadataKey::Complete })
+    .await?;
+let _ = meta.content_length();
+let _ = meta.content_type();
+```
+
+# Reference-level explanation
+
+We will store bits in `ObjectMetadata` to store which fields have been set. And we can compare the bits to decide whether we need to query from storage again:
+
+```rust
+pub async fn metadata(
+    &mut self,
+    flags: impl Into<FlagSet<ObjectMetadataKey>>,
+) -> Result<Arc<ObjectMetadata>> {
+    if let Some(meta) = &self.meta {
+        if meta.bit().contains(flags) || meta.bit().contains(ObjectMetadataKey::Complete) {
+            return Ok(meta.clone());
+        }
+    }
+
+    let meta = Arc::new(self.stat().await?);
+    self.meta = Some(meta.clone());
+
+    Ok(meta)
+}
+```
+
+# Drawbacks
+
+## Breaking changes
+
+After this RFC, `Object::metadata()` will accept a query. And all existing users need to adapt their code for that.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/1420_object_writer.md

@@ -0,0 +1,147 @@
+- Proposal Name: `object_writer`
+- Start Date: 2023-02-27
+- RFC PR: [apache/opendal#1420](https://github.com/apache/opendal/pull/1420)
+- Tracking Issue: [apache/opendal#1421](https://github.com/apache/opendal/issues/1421)
+
+# Summary
+
+Adding `ObjectWriter` to improve support for multipart uploads, as well as enable retry options for write operations.
+
+# Motivation
+
+OpenDAL works well for `read` operations:
+
+- OpenDAL can seek over content even on services like S3.
+- OpenDAL can retry read from the failing point without extra read cost.
+
+However, OpenDAL is not good at `write`:
+
+## Complex multipart operations
+
+OpenDAL supports multipart operations but it's very hard to use:
+
+```ignore
+let object_multipart = o.create_multipart().await?;
+let part_0 = object_multipart.write(0, content_0).await?;
+...
+let part_x = object_multipart.write(x, content_x).await?;
+let new_object = object_multipart.complete(vec![part_0,...,part_x]).await?;
+```
+
+Users should possess the knowledge of the multipart API to effectively use it.
+
+To exacerbate the situation, the multipart API is not standardized and only some object storage services offer support for it. Unfortunately, we cannot even provide support for it on the local file system.
+
+## Lack of retry support
+
+OpenDAL can't retry `write` operations because we accept an `Box<dyn AsyncRead>`. Once we pass this read into other functions, we consumed it.
+
+```rust
+async fn write(&self, path: &str, args: OpWrite, r: input::Reader) -> Result<RpWrite> {
+    self.inner.write(path, args, r).await
+}
+```
+
+By introducing the `ObjectWriter` feature, we anticipate resolving all the associated inquiries simultaneously.
+
+# Guide-level explanation
+
+`ObjectWriter` will provide the following APIs:
+
+```rust
+impl ObjectWriter {
+    pub async write(&mut self, bs: Bytes) -> Result<()>;
+    pub async close(&mut self) -> Result<()>;
+}
+```
+
+After `ObjectWriter` has been constructed, users can use it as a normal writer:
+
+```rust
+let mut w = o.writer().await?;
+w.write(bs1).await?;
+w.write(bs2).await?;
+w.close().await?;
+```
+
+`ObjectWriter` also implements `AsyncWrite` trait which will allow users to use `io::copy` as well:
+
+```rust
+let mut w = o.writer().await?;
+let _ = io::copy_buf(r, o).await?;
+```
+
+# Reference-level explanation
+
+OpenDAL will add a new trait called `output::Writer`:
+
+```rust
+pub trait Write: Unpin + Send + Sync {
+    pub async write(&mut self, bs: Bytes) -> Result<()>;
+
+    pub async initiate(&mut self) -> Result<()>;
+    pub async append(&mut self, bs: Bytes) -> Result<()>;
+
+    pub async close(&mut self) -> Result<()>;
+}
+```
+
+- `write` is used to write full content.
+- `initiate` is used to initiate a multipart writer.
+- `append` is used to append more content into this writer.
+- `close` is used to close and construct the final file.
+
+And `Accessor` will change the `write` API into:
+
+```diff
+pub trait Accessor {
++    type Writer: output::Write;
+
+-    async fn write(&self, path: &str, args: OpWrite, r: input::Reader) -> Result<RpWrite>;
++    async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)>
+}
+```
+
+# Drawbacks
+
+More heavy work for service implementers.
+
+# Rationale and alternatives
+
+## Why accept `Bytes`?
+
+OpenDAL's write is similar to `io::Write::write_all` which will always consume the whole input and return errors if something is wrong. By accepting `Bytes`, we can reduce the extra `Clone` between user land to OpenDAL's services/layers.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Vectored Write
+
+We can add `write_vectored` support in the future:
+
+```rust
+pub trait Write: Unpin + Send + Sync {
+    pub async write_vectored(&mut self, bs: &[Bytes]) -> Result<()>;
+}
+```
+
+Take `s3` services as an example, we can upload different parts at the same time.
+
+## Write From Stream
+
+We can add `write_from` support in the future:
+
+```rust
+pub trait Write: Unpin + Send + Sync {
+    pub async write_from(&mut self, r: BytesStream, size: u64) -> Result<()>;
+}
+```
+
+By implementing this feature, users don't need to hold a large buffer inside memory.

data/core/src/docs/rfcs/1477_remove_object_concept.md

@@ -0,0 +1,159 @@
+- Proposal Name: `remove_object_concept`
+- Start Date: `2023-03-05`
+- RFC PR: [apache/opendal#1477](https://github.com/apache/opendal/pull/1477)
+- Tracking Issue: [apache/opendal#0000](https://github.com/apache/opendal/issues/0000)
+
+# Summary
+
+Eliminating the Object concept to enhance the readability of OpenDAL.
+
+# Motivation
+
+OpenDAL introduces [Object Native API][crate::docs::rfcs::rfc_0041_object_native_api] to resolve the problem of not being easy to use:
+
+```diff
+- let reader = SeekableReader::new(op, path, stream_len);
++ let reader = op.object(&path).reader().await?;
+
+- op.stat(&path).run().await
++ op.object(&path).stat().await
+```
+
+However, times are changing. After the list operation has been moved to the `Object` level, `Object` is now more like a wrapper for `Operator`. The only meaningful API of `Operator` now is `Operator::object`.
+
+Writing `op.object(&path)` repeatedly is boring. Let's take real example from databend as an example:
+
+```rust
+if let Some(dir) = dir_path {
+    match op.object(&dir).stat().await {
+        Ok(_) => {
+            let mut ds = op.object(&dir).scan().await?;
+            while let Some(de) = ds.try_next().await? {
+                if let Some(fi) = stat_file(de).await? {
+                    files.push(fi)
+                }
+            }
+        }
+        Err(e) => warn!("ignore listing {path}/, because: {:?}", e),
+    };
+}
+```
+
+We designed `Object` to make users can reuse the same `Object`. However, nearly no users use our API this way. Most users just build a new `Object` every time. There are two problems:
+
+## Extra cost
+
+`Object::new()` is not zero cost:
+
+```rust
+pub(crate) fn with(op: Operator, path: &str, meta: Option<ObjectMetadata>) -> Self {
+    Self {
+        acc: op.inner(),
+        path: Arc::new(normalize_path(path)),
+        meta: meta.map(Arc::new),
+    }
+}
+```
+
+The `Object` must contain an `Operator` and an `Arc` of strings. With the introduction of [Query Based Metadata][crate::docs::rfcs::rfc_1398_query_based_metadata], there is no longer a need to perform operations on the object.
+
+## Complex concepts
+
+The term `Object` is applied in various fields, making it difficult to provide a concise definition of `opendal::Object`. Moreover, this could potentially confuse our users who may assume that `opendal::Object` is primarily intended for object storage services.
+
+I propose eliminating the intermediate API layer of `Object` and enabling users to directly utilize `Operator`.
+
+# Guide-level explanation
+
+After this RFC is implemented, our users can:
+
+```rust
+# read all content of the file
+op.read("file").await?;
+# read part content of the file
+op.range_read("file", 0..1024).await?;
+# create a reader
+op.reader("file").await?;
+
+# write all content into file
+op.write("file", bs).await?;
+# create a writer
+op.writer("file").await?;
+
+# get metadata of a path
+op.stat("path").await?;
+
+# delete a path
+op.delete("path").await?;
+# remove paths
+op.remove(vec!["path_a"]).await?;
+# remove path recursively
+op.remove_all("path").await?;
+
+# create a dir
+op.create_dir("dir/").await?;
+
+# list a dir
+op.list("dir/").await?;
+# scan a dir
+op.scan("dir/").await?;
+```
+
+We will include the `blocking::Operator` for enhanced ease of use while performing blocking operations.
+
+```rust
+# this is a cheap call without allocation
+let bop = op.blocking();
+
+# read all content
+bop.read("file")?;
+# write all content
+bop.write("file", bs)?;
+```
+
+The scan/list result will be an `Entry` or `BlockingEntry` that contains the same fields as Object, but is only used for scan/list entries.
+
+The public API should look like:
+
+```rust
+impl Entry {
+    pub fn mode(&self) -> EntryType;
+    pub fn path(&self) -> &str;
+    pub async fn stat(&self) -> Result<Metadata>;
+    pub async fn metadata(&self, key: impl Into<MetaKey>) -> Result<Metadata>;
+    ...
+}
+```
+
+# Reference-level explanation
+
+We will remove `Object` entirely and move all `Object` APIs to `Operator` instead:
+
+```rust
+- op.object("path").read().await
++ op.read("path").await
+```
+
+Along with this change, we should also rename the `ObjectXxx` structs, such as `ObjectReader` to `Reader`.
+
+# Drawbacks
+
+## Breaking Changes
+
+This RFC proposes a major breaking change that will require almost all current usage to be rewritten.
+
+# Rationale and alternatives
+
+None
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/1735_operation_extension.md

@@ -0,0 +1,117 @@
+- Proposal Name: `operation_extension`
+- Start Date: 2023-03-23
+- RFC PR: [apache/opendal#1735](https://github.com/apache/opendal/pull/1735)
+- Tracking Issue: [apache/opendal#1738](https://github.com/apache/opendal/issues/1738)
+
+# Summary
+
+Extend operation capabilities to support additional native features.
+
+# Motivation
+
+OpenDAL only supports a limited set of capabilities for operations.
+
+- `read`/`stat`: only supports `range`
+- `write`: only supports `content_type` and `content_disposition`
+
+Our community has a strong need for more additional native features. For example:
+
+- [opendal#892](https://github.com/apache/opendal/issues/892) wants [Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control): Allow users to specify the cache control headers for the uploaded files.
+- [opendal#825](https://github.com/apache/opendal/issues/825) wants [If-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Match) and [If-None-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match): Allow users to makes a request conditional.
+- [opendal#1726](https://github.com/apache/opendal/issues/1726) wants [response-content-disposition](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html): Allow users to specify the content disposition for the downloaded files.
+
+All of these feature requests are essentially asking for the same thing: the capability to define supplementary arguments for operations, particularly about HTTP services.
+
+# Guide-level explanation
+
+In this RFC, we will allow users to specify the standard HTTP headers like cache_control/if_match:
+
+```rust
+let op = OpRead::default().
+    with_cache_control("max-age=3600").
+    with_if_match("\"bfc13a64729c4290ef5b2c2730249c88ca92d82d\"");
+let bs = o.read_with(op).await?;
+```
+
+Also, we will support some non-standard but widely used features like `response-content-disposition`:
+
+```rust
+let op = OpRead::default().
+    with_override_content_disposition("attachment; filename=\"foo.txt\"");
+let req = op.presign_read_with("filename", Duration::hours(1), op)?;
+```
+
+And, finally, we will support allow specify the default headers for services. Take `s3`'s `storage_class` as an example:
+
+```rust
+let mut builder = S3::default();
+builder.default_storage_class("STANDARD_IA");
+builder.default_cache_control("max-age=3600");
+```
+
+In general, we will support the following features:
+
+- Allow users to specify the (non-)standard HTTP headers for operations.
+- Allow users to specify the default HTTP headers for services.
+
+# Reference-level explanation
+
+We will make the following changes in OpenDAL:
+
+For `OpRead` & `OpStat`:
+
+```diff
+pub struct OpRead {
+    br: BytesRange,
++   cache_control: Option<String>,
++   if_match: Option<String>,
++   if_none_match: Option<String>,
++   override_content_disposition: Option<String>,
+}
+```
+
+For `OpWrite`:
+
+```diff
+pub struct OpWrite {
+    append: bool,
+
+    content_type: Option<String>,
+    content_disposition: Option<String>,
++   cache_control: Option<String>,
+}
+```
+
+We will provide different default options for each service. For example, we can add `default_storage_class` for `s3`.
+
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+## Why using `override_content_disposition` instead of `response_content_disposition`?
+
+`response_content_disposition` is not a part of HTTP standard, it's the private API provided by `s3`.
+
+- `azblob` will use `x-ms-blob-content-disposition` header
+- `ocios` will use `httpResponseContentDisposition` query
+
+OpenDAL does not accept the query as is. Instead, we have created a more readable name `override_content_disposition` to clarify its purpose.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+## Strict Mode
+
+Additionally, we have implemented a `strict` option for the `Operator`. If users enable this option, OpenDAL will return an error message for unsupported options. Otherwise, it will ignore them.
+
+For instance, if users rely on the `if_match` behavior but services like `fs` and `hdfs` do not support it natively, they can use the `op.with_strict()` function to prompt OpenDAL to return an error.

data/core/src/docs/rfcs/2083_writer_sink_api.md

@@ -0,0 +1,106 @@
+- Proposal Name: `writer_sink_api`
+- Start Date: 2023-04-23
+- RFC PR: [apache/opendal#2083](https://github.com/apache/opendal/pull/2083)
+- Tracking Issue: [apache/opendal#2084](https://github.com/apache/opendal/issues/2084)
+
+# Summary
+
+Include a `sink` API within the `Writer` to enable streaming writing.
+
+# Motivation
+
+OpenDAL does not support streaming data uploads. Users must first load the data into memory and then send it to the `writer`.
+
+```rust
+let bs = balabala();
+w.write(bs).await?;
+let bs = daladala();
+w.write(bs).await?;
+...
+w.close().await?;
+```
+
+There are two main drawbacks to OpenDAL:
+
+- high memory usage, as reported in issue #1821 on GitHub
+- low performance due to the need to buffer user data before sending it over the network.
+
+To address this issue, it would be beneficial for OpenDAL to provide an API that allows users to pass a stream or reader directly into the writer.
+
+# Guide-level explanation
+
+I propose to add the following API to `Writer`:
+
+```rust
+impl Writer {
+    pub async fn copy_from<R>(&mut self, size: u64, r: R) -> Result<()>
+    where
+        R: futures::AsyncRead + Send + Sync + 'static;
+
+    pub async fn pipe_from<S>(&mut self, size: u64, s: S) -> Result<()>
+    where
+        S: futures::TryStream + Send + Sync + 'static
+        Bytes: From<S::Ok>;
+}
+```
+
+Users can now upload data in a streaming way:
+
+```rust
+// Start writing the 5 TiB file.
+let w = op.writer_with(
+    OpWrite::default()
+        .with_content_length(5 * 1024 * 1024 * 1024 * 1024));
+
+let r = balabala();
+// Send to network directly without in-memory buffer.
+w.copy_from(size, r).await?;
+// repeat...
+...
+
+// Close the write once we are ready!
+w.close().await?;
+```
+
+The underlying services will handle this stream in the most efficient way possible.
+
+# Reference-level explanation
+
+To support `Wrtier::copy_from` and `Writer::pipe_from`, we will add a new API called `sink` inside `oio::Writer`:
+
+```rust
+#[async_trait]
+pub trait Write: Unpin + Send + Sync {
+    async fn sink(&mut self, size: u64, s: Box<dyn futures::TryStream<Ok=Bytes> + Send + Sync>) -> Result<()>;
+}
+```
+
+OpenDAL converts the user input reader and stream into a byte stream for `oio::Write`. Services that support streaming upload natively can directly pass the stream. If not, they can use `write` repeatedly to write the entire stream.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+## What's the different of `OpWrite::content_length` and `sink` size?
+
+The `OpWrite::content_length` parameter specifies the total length of the file to be written, while the `size` argument in the `sink` API indicates the size of the reader or stream provided. Certain services may optimize by writing all content in a single request if `content_length` is the same with given `size`.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+## Retry for the `sink` API
+
+It's impossible to retry the `sink` API itself, but we can provide a wrapper to retry the stream's call of `next`. If we met a retryable error, we can call `next` again by crate like `backon`.
+
+## Blocking support for sink
+
+We will add async support first.