opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/docs/rfcs/0793_generic_kv_services.md

@@ -0,0 +1,209 @@
+- Proposal Name: `generic-kv-services`
+- Start Date: 2022-10-03
+- RFC PR: [apache/opendal#793](https://github.com/apache/opendal/pull/793)
+- Tracking Issue: [apache/opendal#794](https://github.com/apache/opendal/issues/794)
+
+# Summary
+
+Add generic kv services support OpenDAL.
+
+# Motivation
+
+OpenDAL now has some kv services support:
+
+- memory
+- redis
+
+However, maintaining them is complex and very easy to be wrong. We don't want to implement similar logic for every kv
+service. This RFC intends to introduce a generic kv service so that we can:
+
+- Implement OpenDAL Accessor on this generic kv service
+- Add new kv service support via generic kv API.
+
+# Guide-level explanation
+
+No user-side changes.
+
+# Reference-level explanation
+
+OpenDAL will introduce a generic kv service:
+
+```rust
+trait KeyValueAccessor {
+    async fn get(&self, key: &[u8]) -> Result<Option<Vec<u8>>>;
+    async fn set(&self, key: &[u8], value: &[u8]) -> Result<()>;
+}
+```
+
+We will implement the OpenDAL service on `KeyValueAccessor`. To add new kv service support, users only need to implement
+it against `KeyValueAccessor`.
+
+## Spec
+
+This RFC is mainly inspired
+by [TiFS: FUSE based on TiKV](https://github.com/Hexilee/tifs/blob/main/contribution/design.md). We will use the
+same `ScopedKey` idea in `TiFS`.
+
+```rust
+pub enum ScopedKey {
+    Meta,
+    Inode(u64),
+    Block {
+        ino: u64,
+        block: u64,
+    },
+    Entry {
+        parent: u64,
+        name: String,
+    },
+}
+```
+
+We can encode a scoped key into a byte array as a key. Following is the common layout of an encoded key.
+
+```text
++ 1byte +<----------------------------+ dynamic size +------------------------------------>+
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       v                                                                                  v
++------------------------------------------------------------------------------------------+
+|       |                                                                                  |
+| scope |                                       body                                       |
+|       |                                                                                  |
++-------+----------------------------------------------------------------------------------+
+```
+
+### Meta
+
+There is only one key in the meta scope. The meta key is designed to store metadata of our filesystem. Following is the
+layout of an encoded meta key.
+
+```text
++ 1byte +
+|       |
+|       |
+|       |
+|       |
+|       |
+|       |
+|       v
++-------+
+|       |
+|   0   |
+|       |
++-------+
+```
+
+This key will store data:
+
+```rust
+pub struct Meta {
+    inode_next: u64,
+}
+```
+
+The meta-structure contains only an auto-increasing counter `inode_next`, designed to generate an inode number.
+
+### Inode
+
+Keys in the inode scope are designed to store attributes of files. Following is the layout of an encoded inode key.
+
+```text
++ 1byte +<-------------------------------+ 8bytes +--------------------------------------->+
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       |                                                                                  |
+|       v                                                                                  v
++------------------------------------------------------------------------------------------+
+|       |                                                                                  |
+|   1   |                               inode number                                       |
+|       |                                                                                  |
++-------+----------------------------------------------------------------------------------+
+```
+
+This key will store data:
+
+```rust
+pub struct Inode {
+    meta: Metadata,
+    blocks: HashMap<u64, u32>,
+}
+```
+
+blocks is the map from `block_id` -> `size`. We will use this map to calculate the correct blocks to read.
+
+### Block
+
+Keys in the block scope are designed to store blocks of a file. Following is the layout of an encoded block key.
+
+```text
++ 1byte +<----------------- 8bytes ---------------->+<------------------- 8bytes ----------------->+
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       v                                           v                                              v
++--------------------------------------------------------------------------------------------------+
+|       |                                           |                                              |
+|   2   |              inode number                 |                  block index                 |
+|       |                                           |                                              |
++-------+-------------------------------------------+----------------------------------------------+
+```
+
+### Entry
+
+Keys in the file index scope are designed to store the entry of the file. Following is the layout of an encoded file
+entry key.
+
+```text
++ 1byte +<----------------- 8bytes ---------------->+<-------------- dynamic size ---------------->+
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       |                                           |                                              |
+|       v                                           v                                              v
++--------------------------------------------------------------------------------------------------+
+|       |                                           |                                              |
+|   3   |     inode number of parent directory      |         file name in utf-8 encoding          |
+|       |                                           |                                              |
++-------+-------------------------------------------+----------------------------------------------+
+```
+
+Store the correct inode number for this file.
+
+```rust
+pub struct Index {
+    pub ino: u64,
+}
+```
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/0926_object_reader.md

@@ -0,0 +1,93 @@
+- Proposal Name: `object_reader`
+- Start Date: 2022-11-13
+- RFC PR: [apache/opendal#926](https://github.com/apache/opendal/pull/926)
+- Tracking Issue: [apache/opendal#927](https://github.com/apache/opendal/issues/927)
+
+# Summary
+
+Returning reading related object meta in the reader.
+
+# Motivation
+
+Some services like s3 could return object meta while issuing reading requests.
+
+In `GetObject`, we could get:
+
+- Last-Modified
+- Content-Length
+- ETag
+- Content-Range
+- Content-Type
+- Expires
+
+We can avoid extra `HeadObject` calls by reusing that meta wisely, which could take 50ms. For example, `Content-Range` returns the content range of this read in the whole object: `<unit> <range-start>-<range-end>/<size>`. By using the content range, we can avoid `HeadObject` to get this object's total size, which means a lot for the content cache.
+
+# Guide-level explanation
+
+`reader` and all its related API will return `ObjectReader` instead:
+
+```diff
+- pub async fn reader(&self) -> Result<impl BytesRead> {}
++ pub async fn reader(&self) -> Result<ObjectReader> {}
+```
+
+`ObjectReader` impls `BytesRead` too, so existing code will keep working. And `ObjectReader` will provide similar APIs to `Entry`, for example:
+
+```rust
+pub async fn content_length(&self) -> Option<u64> {}
+pub async fn last_modified(&self) -> Option<OffsetDateTime> {}
+pub async fn etag(&self) -> Option<String> {}
+```
+
+Note:
+
+- All fields are optional, as services like fs could not return them.
+- `content_length` here is this read request's length, not the object's length.
+
+# Reference-level explanation
+
+We will change the API signature of `Accessor`:
+
+```diff
+- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
++ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}
+```
+
+`ObjectReader` is a wrapper of `BytesReader` and `ObjectMeta`:
+
+```rust
+pub struct ObjectReader {
+    inner: BytesReader
+    meta: ObjectMetadata,
+}
+
+impl ObjectReader {
+    pub async fn content_length(&self) -> Option<u64> {}
+    pub async fn last_modified(&self) -> Option<OffsetDateTime> {}
+    pub async fn etag(&self) -> Option<String> {}
+}
+```
+
+Services can decide whether or not to fill them.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## Add content-range support
+
+We can add `content-range` in `ObjectMeta` so that users can fetch and use them.

data/core/src/docs/rfcs/0977_refactor_error.md

@@ -0,0 +1,151 @@
+- Proposal Name: `refactor-error`
+- Start Date: 2022-11-21
+- RFC PR: [apache/opendal#977](https://github.com/apache/opendal/pull/977)
+- Tracking Issue: [apache/opendal#976](https://github.com/apache/opendal/pull/976)
+
+# Summary
+
+Use a separate error instead of `std::io::Error`.
+
+# Motivation
+
+OpenDAL is used to use `std::io::Error` for all functions. This design is natural and easy to use. But there are many problems with the usage:
+
+## Not friendly for retry
+
+`io::Error` can't carry retry-related information. In [RFC-0247: Retryable Error](./0247-retryable-error.md), we use `io::ErrorKind::Interrupt` to indicate this error is retryable. But this change will hide the real error kind from the underlying. To mark this error has been retried, we have to add another new error wrapper:
+
+```rust
+#[derive(thiserror::Error, Debug)]
+#[error("permanent error: still failing after retry, source: {source}")]
+struct PermanentError {
+    source: Error,
+}
+```
+
+## ErrorKind is inaccurate
+
+`std::io::ErrorKind` is used to represent errors returned from system io, which is unsuitable for mistakes that have business semantics. For example, users can't distinguish `ObjectNotFound` or `BucketNotFound` from `ErrorKind::NotFound`.
+
+## ErrorKind is incomplete
+
+OpenDAL has been waiting for features [`io_error_more`](https://github.com/rust-lang/rust/issues/86442) to be stabilized for a long time. But there is no progress so far, which makes it impossible to return `ErrorKind::IsADirectory` or `ErrorKind::NotADirectory` on stable rust.
+
+## Error is not easy to carry context
+
+To carry context inside `std::io::Error`, we have to check and make sure all functions are constructed `ObjectError` or `BackendError`:
+
+```rust
+#[derive(Error, Debug)]
+#[error("object error: (op: {op}, path: {path}, source: {source})")]
+pub struct ObjectError {
+    op: Operation,
+    path: String,
+    source: anyhow::Error,
+}
+```
+
+To make everything worse, we can't prevent opendal returns raw io errors directly. For example, in `Object::range_read`:
+
+```rust
+pub async fn range_read(&self, range: impl RangeBounds<u64>) -> Result<Vec<u8>> {
+    ...
+
+    io::copy(s, &mut bs).await?;
+
+    Ok(bs.into_inner())
+}
+```
+
+We leaked the `io::Error` without any context.
+
+# Guide-level explanation
+
+Thus, I propose to add `opendal::Error` back with everything improved.
+
+Users will have similar usage as before:
+
+```rust
+if let Err(e) = op.object("test_file").metadata().await {
+    if e.kind() == ErrorKind::ObjectNotFound {
+        println!("object not exist")
+    }
+}
+```
+
+Users can check if this error a `temporary`:
+
+```rust
+if err.is_temporary() {
+    // retry the operation
+}
+```
+
+Users can print error messages via `Display`:
+
+```rust
+> println!("{}", err);
+
+ObjectNotFound (permanent) at read, context: { service: S3, path: path/to/file } => status code: 404, headers: {"x-amz-request-id": "GCYDTQX51YRSF4ZF", "x-amz-id-2": "EH0vV6lTwWk+lFXqCMCBSk1oovqhG4bzALU9+sUudyw7TEVrfWm2o/AFJKhYKpdGqOoBZGgMTC0=", "content-type": "application/xml", "date": "Mon, 21 Nov 2022 05:26:37 GMT", "server": "AmazonS3"}, body: ""
+```
+
+Also, users can choose to print the more verbose message via `Debug`:
+
+```rust
+> println!("{:?}", err);
+
+ObjectNotFound (permanent) at read => status code: 404, headers: {"x-amz-request-id": "GCYDTQX51YRSF4ZF", "x-amz-id-2": "EH0vV6lTwWk+lFXqCMCBSk1oovqhG4bzALU9+sUudyw7TEVrfWm2o/AFJKhYKpdGqOoBZGgMTC0=", "content-type": "application/xml", "date": "Mon, 21 Nov 2022 05:26:37 GMT", "server": "AmazonS3"}, body: ""
+
+Context:
+    service: S3
+    path: path/to/file
+
+Source: <source error>
+
+Backtrace: <backtrace if we have>
+```
+
+# Reference-level explanation
+
+We will add new `Error` and `ErrorKind` in opendal:
+
+```rust
+pub struct Error {
+    kind: ErrorKind,
+    message: String,
+
+    status: ErrorStatus,
+    operation: &'static str,
+    context: Vec<(&'static str, String)>,
+    source: Option<anyhow::Error>,
+}
+```
+
+- status: the status of this error, which indicates if this error is temporary
+- operation: the operation which generates this error
+- context: the context related to this error
+- source: the underlying source error
+
+# Drawbacks
+
+## Breaking changes
+
+This RFC will lead to a breaking at user side.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+## More ErrorKind
+
+We can add more error kinds to make it possible for users to check.

data/core/src/docs/rfcs/1085_object_handler.md

@@ -0,0 +1,73 @@
+- Proposal Name: `object_handler`
+- Start Date: 2022-12-19
+- RFC PR: [apache/opendal#1085](https://github.com/apache/opendal/pull/1085)
+- Tracking Issue: [apache/opendal#1085](https://github.com/apache/opendal/issues/1085)
+
+# Summary
+
+Returning a `file description` to users for native seek support.
+
+# Motivation
+
+OpenDAL's goal is to `access data freely, painlessly, and efficiently`, so we build an operation first API which means we provide operation instead of the file description. Users don't need to call `open` before `read`; OpenDAL will handle all the open and close functions.
+
+However, our users do want to control the complex behavior of that:
+
+- Some storage backends have native `seek` support, but OpenDAL can't fully use them.
+- Users want to improve performance by reusing the same file description without `open` and `close` for every read operation.
+
+This RFC will fill this gap.
+
+
+# Guide-level explanation
+
+Users can get an object handler like:
+
+```rust
+let oh: ObjectHandler = op.object("path/to/file").open().await?;
+```
+
+`ObjectHandler` will implement `AsyncRead` and `AsyncSeek` so it can be used like `tokio::fs::File`. If the backend supports native seek operation, we will take the native process; otherwise, we will fall back to simulation implementations.
+
+The blocking version will be provided by:
+
+```rust
+let boh: BlockingObjectHandler = op.object("path/to/file").blocking_open()?;
+```
+
+And `BlockingObjectHandler` will implement `Read` and `Seek` so it can be used like `std::fs::File`. If the backend supports native seek operation, we will take the native process; otherwise, we will fall back to simulation implementations.
+
+# Reference-level explanation
+
+This RFC will add a new API `open` in `Accessor`:
+
+```rust
+pub trait Accessor {
+    async fn open(&self, path: &str, args: OpOpen) -> Result<(RpOpen, BytesHandler)>;
+}
+```
+
+Only services that support native `seek` operations can implement this API, like `fs` and `hdfs`. For services that do not support native `seek` operations like `s3` and `azblob`, we will fall back to the simulation implementations: maintaining an in-memory index instead.
+
+# Drawbacks
+
+None
+
+# Rationale and alternatives
+
+## How about writing operations?
+
+Ideally, writing on `ObjectHandler` should also be supported. But we still don't know how this API will be used. Let's apply this API for `read` first.
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- Add write support
+- Adopt native `pread`

data/core/src/docs/rfcs/1391_object_metadataer.md

@@ -0,0 +1,110 @@
+- Proposal Name: `object_metadataer`
+- Start Date: 2023-02-21
+- RFC PR: [apache/opendal#1391](https://github.com/apache/opendal/pull/1391)
+- Tracking Issue: [apache/opendal#1393](https://github.com/apache/opendal/issues/1393)
+
+# Summary
+
+Add object metadataer to avoid unneeded extra metadata call.
+
+# Motivation
+
+OpenDAL has native metadata cache for now:
+
+```rust
+let _ = o.metadata().await?;
+// This call doesn't need to send a request.
+let _ = o.metadata().await?;
+```
+
+Also, OpenDAL can reuse metadata from `list` or `scan`:
+
+```rust
+let mut ds = o.scan().await?;
+while let Some(de) = ds.try_next().await? {
+    // This call doesn't need to send a request (if we are lucky enough).
+    let _ = de.metadata().await?;
+}
+```
+
+By reusing metadata from `list` or `scan` we can reduce the extra `stat` call for each object. In our real use cases, we can reduce the total time to calculate the total length inside a dir with 6k files from 4 minutes to 2 seconds.
+
+However, metadata can only be cached as a whole. If services could return more metadata in `stat` than in `list`, we wouldn't be able to mark the metadata as cacheable. If services add more metadata, we could inadvertently introduce the performance degradation.
+
+This RFC aims to address this problem by hiding `ObjectMetadata` and adding `ObjectMetadataer` instead. All object metadata values will be cached separately and all user calls to object metadata will go to the cache.
+
+# Guide-level explanation
+
+This RFC will add `ObjectMetadataer` and `BlockingObjectMetadataer` for users:
+
+Users call to `o.metadata()` will return `ObjectMetadataer` instead:
+
+```rust
+let om: ObjectMetadataer = o.metadata().await?;
+```
+
+And users can query more metadata over it:
+
+```rust
+let content_length = om.content_length().await;
+let etag = om.etag().await;
+```
+
+During the whole lifetime of the corresponding `Object` or `ObjectMetadataer`, we make sure that at most one `stat` call is sent. After this change, users will never get an `ObjectMetadata` anymore.
+
+# Reference-level explanation
+
+We will introduce a bitmap to store the state of all object metadata fields separately. Everytime users call `key` on metadata, we will check as following:
+
+- If `bitmap` is set, return directly.
+- If `bitmap` is not set, but is complete, return directly.
+- If both `bitmap` is not set and not `complete`, call `stat` to get the meta.
+
+`Object` will return `ObjectMetadataer` instead of `ObjectMetadata`:
+
+```diff
+- pub async fn metadata(&self) -> Result<ObjectMetadata> {}
++ pub async fn metadata(&self) -> Result<ObjectMetadataer> {}
+```
+
+And `ObjectMetadataer` will provide the following API:
+
+```rust
+impl ObjectMetadataer {
+    pub async fn mode(&self) -> Result<ObjectMode>;
+    pub async fn content_length(&self) -> Result<u64>;
+    pub async fn content_md5(&self) -> Result<Option<String>>;
+    pub async fn last_modified(&self) -> Result<Option<OffsetDateTime>>;
+    pub async fn etag(&self) -> Result<Option<String>>;
+}
+
+impl BlockingObjectMetadataer {
+    pub fn mode(&self) -> Result<ObjectMode>
+    pub fn content_length(&self) -> Result<u64>;
+    pub fn content_md5(&self) -> Result<Option<String>>;
+    pub fn last_modified(&self) -> Result<Option<OffsetDateTime>>;
+    pub fn etag(&self) -> Result<Option<String>>;
+}
+```
+
+# Drawbacks
+
+## Breaking changes
+
+This RFC will introduce breaking changes for `Object::metadata`. And users can't do `serde::Serialize` or `serde::Deserialize` on object metadata any more. All metadata related API calls will be removed from `Object`.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.