opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/docs/rfcs/5444_operator_from_uri.md

@@ -0,0 +1,162 @@
+- Proposal Name: `operator_from_uri`
+- Start Date: 2024-12-23
+- RFC PR: [apache/opendal#5444](https://github.com/apache/opendal/pull/5444)
+- Tracking Issue: [apache/opendal#5445](https://github.com/apache/opendal/issues/5445)
+
+# Summary
+
+This RFC proposes adding URI-based configuration support to OpenDAL, allowing users to create operators directly from URIs. The proposal introduces a new `from_uri` API in both the `Operator` and `Configurator` traits, along with an `OperatorRegistry` to manage operator factories. As part of this change, we will also transition from the `Scheme` enum to string-based scheme identifiers, enabling better modularity and support for service crate splitting.
+
+# Motivation
+
+Currently, creating an operator in OpenDAL requires explicit configuration through builder patterns. While this approach provides type safety and clear documentation, it can be verbose and inflexible for simple use cases. Many storage systems are naturally identified by URIs (e.g., `s3://bucket/path`, `fs:///path/to/dir`).
+
+Adding URI-based configuration would:
+
+- Simplify operator creation for common use cases
+- Enable configuration via connection strings (common in many applications)
+- Make OpenDAL more approachable for new users
+- Allow dynamic operator creation based on runtime configuration
+
+# Guide-level explanation
+
+The new API allows creating operators directly from URIs:
+
+```rust
+// Create an operator using URI
+let op = Operator::from_uri("s3://my-bucket/path", vec![
+    ("endpoint".to_string(), "http://localhost:8080"to_string()),
+])?;
+
+// Users can pass options through the URI along with additional key-value pairs
+// The extra options will override identical options specified in the URI
+let op = Operator::from_uri("s3://my-bucket/path?region=us-east-1", vec![
+    ("endpoint".to_string(), "http://localhost:8080"to_string()),
+])?;
+
+// Create a file system operator
+let op = Operator::from_uri("fs:///tmp/test", vec![])?;
+```
+
+OpenDAL will, by default, register services enabled by features in a global `OperatorRegistry`. Users can also create custom operator registries to support their own schemes or additional options.
+
+```rust
+// Using a custom registry
+let registry = OperatorRegistry::new();
+
+// Register builtin builders under desired schemes
+registry.register::<services::S3>(services::S3_SCHEME);
+registry.register::<services::S3>("minio");  // MinIO is S3-compatible
+registry.register::<services::S3>("r2");     // Cloudflare R2 is S3-compatible
+
+// Users can define their own scheme names for internal use
+registry.register::<services::S3>("company-storage");
+registry.register::<services::Azblob>("backup-storage");
+
+let op = registry.load("company-storage://bucket/path", [])?;
+```
+
+# Reference-level explanation
+
+The implementation consists of three main components:
+
+1. The `OperatorFactory` and `OperatorRegistry`:
+
+`OperatorFactory` is a function type that takes a URI string plus options and returns an `Operator`. `OperatorRegistry` manages factories registered under different schemes.
+
+`OperatorFactory` is a function type that takes a URI string plus options and returns an `Operator`. `OperatorRegistry` manages factories registered under different schemes.
+
+```rust
+type OperatorFactory = fn(&str, Vec<(String, String)>) -> Result<Operator>;
+
+pub struct OperatorRegistry { ... }
+
+impl OperatorRegistry {
+    fn register<B: Builder>(&self, scheme: &str) {
+        ...
+    }
+
+    fn load(&self, uri: &str, options: impl IntoIterator<Item = (String, String)>) -> Result<Operator> {
+        ...
+    }
+}
+```
+
+2. The `Configurator` trait extension:
+
+`Configurator` will add a new API to create a configuration from a URI and options. Services should only parse the URI components relevant to their configuration (host, path, query parameters) without concerning themselves with the scheme portion.
+
+```rust
+impl Configurator for S3Config {
+    fn from_uri(uri: &Uri, options: &HashMap<String, String>) -> Result<Self> {
+        ...
+    }
+}
+```
+
+This design allows the same S3 implementation to work whether accessed via `s3://`, `minio://`, or any other user-defined scheme.
+
+3. The `Operator` `from_uri` method:
+
+The `Operator` trait will add a new `from_uri` method to create an operator from a URI and options. This method will use the global `OperatorRegistry` to find the appropriate factory for the scheme.
+
+```rust
+impl Operator {
+    pub fn from_uri(
+        uri: &str,
+        options: impl IntoIterator<Item = (String, String)>,
+    ) -> Result<Self> {
+        ...
+    }
+}
+```
+
+## Scheme Enum Removal
+
+As part of this RFC, we will transition from the `Scheme` enum to string-based identifiers (`&'static str`). This change is necessary because:
+
+1. **Modularity**: Services in separate crates cannot add variants to a core enum
+2. **Extensibility**: Users and third-party crates can define custom schemes without modifying OpenDAL
+3. **Simplicity**: Services don't need to know their scheme identifier
+
+# Drawbacks
+
+- Increases API surface area
+- Less type safety compared to builder patterns
+- Potential for confusing error messages with invalid URIs
+- Need to maintain backwards compatibility
+
+# Rationale and alternatives
+
+Alternatives considered:
+
+1. Connection string format instead of URIs
+2. Builder pattern with URI parsing
+3. Macro-based configuration
+
+URI-based configuration was chosen because:
+
+- URIs are widely understood
+- Natural fit for storage locations
+- Extensible through custom schemes
+- Common in similar tools
+
+# Prior art
+
+Similar patterns exist in:
+
+- Database connection strings (PostgreSQL, MongoDB)
+- [`object_store::parse_url`](https://docs.rs/object_store/latest/object_store/fn.parse_url.html)
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- Support for connection string format
+- Configuration presets like `r2` and `s3` with directory bucket enabled
+- Service crate splitting: Each service can live in its own crate and register itself with the core
+- Plugin system: Allow dynamic loading of service implementations at runtime
+- Service discovery: Automatically register available services based on feature flags or runtime detection
+- Scheme validation and conventions: Provide utilities to validate scheme naming conventions

data/core/src/docs/rfcs/5479_context.md

@@ -0,0 +1,140 @@
+- Proposal Name: `context`
+- Start Date: 2024-12-30
+- RFC PR: [apache/opendal#5480](https://github.com/apache/opendal/pull/5480)
+- Tracking Issue: [apache/opendal#5479](https://github.com/apache/opendal/issues/5479)
+
+# Summary
+
+Add `Context` in opendal to distribute global resources like http client, runtime, etc.
+
+# Motivation
+
+OpenDAL now includes two global resources, the `http client` and `runtime`, which are utilized by the specified service across all enabled layers.
+
+However, it's a bit challenging for layers to interact with these global resources.
+
+## For http client
+
+Layers cannot directly access the HTTP client. The only way to interact with the HTTP client is through the service builder, such as [`S3::http_client()`](https://docs.rs/opendal/latest/opendal/services/struct.S3.html#method.http_client). Layers like logging and metrics do not have direct access to the HTTP client.
+
+Users need to implement the `HttpFetcher` trait to interact with the HTTP client. However, the drawback is that users lack context for the given requests; they do not know which service the request originates from or which operation it is performing.
+
+## For runtime
+
+OpenDAL has the [`Execute`](https://docs.rs/opendal/latest/opendal/trait.Execute.html) for users to implement so that they can interact with the runtime. However, the API is difficult to use, as layers need to extract and construct the `Executor` for every request.
+
+For example:
+
+```rust
+async fn read(&self, path: &str, mut args: OpRead) -> Result<(RpRead, Self::Reader)> {
+    if let Some(exec) = args.executor().cloned() {
+        args = args.with_executor(Executor::with(TimeoutExecutor::new(
+            exec.into_inner(),
+            self.io_timeout,
+        )));
+    }
+
+    self.io_timeout(Operation::Read, self.inner.read(path, args))
+        .await
+        .map(|(rp, r)| (rp, TimeoutWrapper::new(r, self.io_timeout)))
+}
+```
+
+# Guide-level explanation
+
+So I propose to add a `Context` to OpenDAL to distribute global resources like the HTTP client and runtime.
+
+The `Context` is a struct that contains the global resources, such as the HTTP client and runtime. It is passed to the service builder and layers so that they can interact with the global resources.
+
+```rust
+let mut ctx = Context::default();
+ctx.set_http_client(my_http_client);
+ctx.set_executor(my_executor);
+
+let op = op.with_context(ctx);
+```
+
+The following API will be added:
+
+- new struct `Context`
+- `Context::default()`
+- `Context::load_http_client(&self) -> HttpClient`
+- `Context::load_executor(&self) -> Executor`
+- `Context::update_http_client(&self, f: impl FnOnce(HttpClient) -> HttpClient)`
+- `Context::update_executor(&self, f: impl FnOnce(Executor) -> Executor)`
+- `Operator::with_context(ctx: Context) -> Operator`
+
+The following API will be deprecated:
+
+- `Operator::default_executor`
+- `Operator::with_default_executor`
+- `OpRead::with_executor`
+- `OpRead::executor`
+- `OpWrite::with_executor`
+- `OpWrite::executor`
+- All services builders' `http_client` API
+
+# Reference-level explanation
+
+We will add `Context` struct in `AccessInfo`. Every service must use `Context::default()` for `AccessInfo` and stores the same instance of `Context` in the service core. All the following usage of http client or runtime should be through the `Context` instead.
+
+The `Context` itself is a struct wrapped by something like `ArcSwap<T>`, allowing us to update it atomically.
+
+The layers will switch to `Context` to get the global resources instead of `OpRead`.
+
+We no longer need to hijack the read operation.
+
+```rust
+- async fn read(&self, path: &str, mut args: OpRead) -> Result<(RpRead, Self::Reader)> {
+-    if let Some(exec) = args.executor().cloned() {
+-        args = args.with_executor(Executor::with(TimeoutExecutor::new(
+-            exec.into_inner(),
+-            self.io_timeout,
+-        )));
+-    }
+-    
+-    ...
+- }
+```
+
+Instead, we can directly get the executor from the `Context` during `layer`.
+
+```rust
+impl<A: Access> Layer<A> for TimeoutLayer {
+    type LayeredAccess = TimeoutAccessor<A>;
+
+    fn layer(&self, inner: A) -> Self::LayeredAccess {
+        inner
+            .info()
+            .context()
+            .update_executor(|exec| Executor::with(TimeoutExecutor::new(exec, self.io_timeout)));
+
+        TimeoutAccessor {
+            inner,
+
+            timeout: self.timeout,
+            io_timeout: self.io_timeout,
+        }
+    }
+}
+```
+
+# Drawbacks
+
+A bit cost (`50ns`) for every operation that `load_http_client`.
+
+# Rationale and alternatives
+
+None.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/5485_conditional_reader.md

@@ -0,0 +1,112 @@
+- Proposal Name: `conditional_reader`
+- Start Date: 2024-12-31
+- RFC PR: [apache/opendal#5485](https://github.com/apache/opendal/pull/5485)
+- Tracking Issue: [apache/opendal#5486](https://github.com/apache/opendal/issues/5486)
+
+# Summary
+
+Add `if_match`, `if_none_match`, `if_modified_since` and `if_unmodified_since` options to OpenDAL's `reader_with` API.
+
+# Motivation
+
+OpenDAL currently supports conditional `reader_with` operations based only on `version`. However, many storage services 
+also support conditional operations based on Etag and/or modification time.
+
+Adding these options will:
+
+- Provide more granular control over read operations.
+- Align OpenDAL with features provided by modern storage services, meeting broader use cases.
+
+# Guide-level explanation
+
+Four new options will be added to the `reader_with` API:
+
+## `if_match`
+
+Return the content only if its Etag matches the specified Etag; otherwise,
+an error kind `ErrorKind::ConditionNotMatch` will be returned:
+
+```rust
+let reader = op.reader_with("path/to/file")
+    .if_match(etag)
+    .await?;
+```
+
+## `if_none_match`
+
+Return the content only if its Etag does NOT match the specified Etag; otherwise,
+an error kind `ErrorKind::ConditionNotMatch` will be returned:
+
+```rust
+let reader = op.reader_with("path/to/file")
+    .if_none_match(etag)
+    .await?;
+```
+
+## `if_modified_since`
+
+Return the content if it has been modified since the specified time; otherwise, 
+an error kind `ErrorKind::ConditionNotMatch` will be returned:
+
+```rust
+use chrono::{Duration, Utc};
+
+let last_check = Utc::now() - Duration::seconds(3600); // 1 hour ago
+let reader = op.reader_with("path/to/file")
+    .if_modified_since(last_check)
+    .await?;
+```
+
+
+## `if_unmodified_since` 
+
+Return the content if it has NOT been modified since the specified time; otherwise, 
+an error kind `ErrorKind::ConditionNotMatch` will be returned:
+
+```rust
+use chrono::{Duration, Utc};
+
+let timestamp = Utc::now() - Duration::seconds(86400); // 24 hours ago
+let reader = op.reader_with("path/to/file")
+    .if_unmodified_since(timestamp)
+    .await?;
+```
+
+
+# Reference-level explanation
+
+The main implementation will include:
+
+1. Add new fields(`if_modified_since`, `if_unmodified_since`) and related functions to `OpRead`.
+
+2. Add the related functions to `FutureReader`
+
+3. Add new capability flags:
+```rust
+pub struct Capability {
+    // ... other fields
+    pub read_with_if_modified_since: bool,
+    pub read_with_if_unmodified_since: bool,
+}
+```
+4. implement `if_modified_since`, `if_unmodified_since` for the underlying storage service.
+
+# Drawbacks
+
+- Add complexity to the API
+
+# Rationale and alternatives
+
+- Follows existing OpenDAL patterns for conditional operations
+
+# Prior art
+
+None
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/5495_list_with_deleted.md

@@ -0,0 +1,81 @@
+- Proposal Name: `list_with_deleted`
+- Start Date: 2025-01-02
+- RFC PR: [apache/opendal#5495](https://github.com/apache/opendal/pull/0000)
+- Tracking Issue: [apache/opendal#5496](https://github.com/apache/opendal/issues/5496)
+
+# Summary
+
+Add `list_with(path).deleted(true)` to enable users to list deleted files from storage services.
+
+# Motivation
+
+OpenDAL is currently working on adding support for file versions, allowing users to read, list, and delete them.
+
+```rust
+// Read given version
+op.read_with(path).version(version_id).await;
+// Fetch the metadata of given version.
+op.stat_with(path).version(version_id).await;
+// Delete the given version.
+op.delete_with(path).version(version_id).await;
+// List the path's versions.
+op.list_with(path).versions().await;
+```
+
+However, to implement the complete data recovery workflow, we should also include support for recovering deleted files from storage services. This feature is referred to as `DeleteMarker` in S3 and `Soft Deleted` in Azure Blob Storage or Google Cloud Storage. Users can utilize these deleted files (or versions) to restore files that may have been accidentally deleted.
+
+# Guide-level explanation
+
+I suggest adding `list_with(path).deleted(true)` to allow users to list deleted files from storage services.
+
+```rust
+let entries = op.list_with(path).deleted(true).await;
+```
+
+Please note that `deleted` here means "including deleted files" rather than "only deleted files." Therefore, `list_with(path).deleted(true)` will list both current files and deleted ones.
+
+At the same time, we will add an `is_deleted` field to the `Metadata` struct to indicate whether the file has been deleted. Together with the existing `is_current` field, we will have the following matrix:
+
+| `is_current`  | `is_deleted` | Description                                                                                                                                                                                                                                                                                                                                                                          |
+|---------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `Some(true)`  | `false`      | **The metadata's associated version is the latest, current version.** This is the normal state, indicating that this version is the most up-to-date and accessible version.                                                                                                                                                                                                          |
+| `Some(true)`  | `true`       | **The metadata's associated version is the latest, deleted version (Latest Delete Marker or Soft Deleted).** This is particularly important in object storage systems like S3. It signifies that this version is the **most recent delete marker**, indicating the object has been deleted. Subsequent GET requests will return 404 errors unless a specific version ID is provided. |
+| `Some(false)` | `false`      | **The metadata's associated version is neither the latest version nor deleted.** This indicates that this version is a previous version, still accessible by specifying its version ID.                                                                                                                                                                                              |
+| `Some(false)` | `true`       | **The metadata's associated version is not the latest version and is deleted.** This represents a historical version that has been marked for deletion. Users will need to specify the version ID to access it, and accessing it may be subject to specific delete marker behavior (e.g., in S3, it might not return actual data but a specific delete marker response).             |
+| `None`        | `false`      | **The metadata's associated file is not deleted, but its version status is either unknown or it is not the latest version.** This likely indicates that versioning is not enabled for this file, or versioning information is unavailable.                                                                                                                                           |
+| `None`        | `true`       | **The metadata's associated file is deleted, but its version status is either unknown or it is not the latest version.** This typically means the file was deleted without versioning enabled, or its versioning information is unavailable. This may represent an actual data deletion operation rather than an S3 delete marker.                                                   |
+
+
+# Reference-level explanation
+
+- Implement the `list_with(path).deleted(true)` API for the `Operator`.
+- Add an `is_deleted` field to `Metadata`.
+- Integrate logic for including deleted files into the `list` method of the storage service.
+
+# Drawbacks
+
+None.
+
+# Rationale and alternatives
+
+## Why "including deleted files" rather than "only deleted files"?
+
+Most storage services are designed to list files along with deleted files, rather than exclusively listing deleted files. For example:
+
+- S3's [ListObjectVersions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html) API lists all versions of an object, including delete markers.
+- GCS's [list](https://cloud.google.com/storage/docs/json_api/v1/objects/list) API includes a parameter `softDeleted` to display soft-deleted files.
+- AzBlob's [List Blobs](https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs) API supports the parameter `include=deleted` to list soft-deleted blobs.
+
+So, it is more natural to list files along with deleted files, rather than only listing deleted files.
+
+# Prior art
+
+None.
+
+# Unresolved questions
+
+None.
+
+# Future possibilities
+
+None.

data/core/src/docs/rfcs/5556_write_returns_metadata.md

@@ -0,0 +1,121 @@
+- Proposal Name: `write_returns_metadata`
+- Start Date: 2025-01-16
+- RFC PR: [apache/opendal#5556](https://github.com/apache/opendal/pull/5556)
+- Tracking Issue: [apache/opendal#5557](https://github.com/apache/opendal/issues/5557)
+
+# Summary
+
+Enhance write operations by returning metadata after successful writes.
+
+# Motivation
+
+Currently, write operations (`write`, `write_with`, `writer`, `writer_with`) only return `Result<()>` or `Result<Writer>`. 
+Users who need metadata after writing (like `ETag` or `version_id`) must make an additional `stat()` call. This is inefficient 
+and can lead to race conditions if the file is modified between the write and stat operations.
+
+Many storage services (like S3, GCS, Azure Blob) return metadata in their write responses. We should expose this information 
+to users directly after write operations.
+
+# Guide-level explanation
+
+The write operations will be enhanced to return metadata:
+
+```rust
+// Before
+op.write("path/to/file", data).await?;
+let meta = op.stat("path/to/file").await?;
+if Some(etag) = meta.etag() {
+    println!("File ETag: {}", etag);
+}
+
+// After
+let meta = op.write("path/to/file", data).await?;
+if Some(etag) = meta.etag() {
+    println!("File ETag: {}", etag);
+}
+```
+
+For writer operations:
+
+```rust
+// Before
+let mut writer = op.writer("path/to/file").await?;
+writer.write(data).await?;
+writer.close().await?;
+let meta = op.stat("path/to/file").await?;
+if Some(etag) = meta.etag() {
+    println!("File ETag: {}", etag);
+}
+
+// After
+let mut writer = op.writer("path/to/file").await?;
+writer.write(data).await?;
+let meta = writer.close().await?;
+if Some(etag) = meta.etag() {
+    println!("File ETag: {}", etag);
+}
+```
+
+The behavior remains unchanged if users don't need the metadata - they can simply ignore the return value.
+
+# Reference-level explanation
+
+## Changes to `Operator` API
+
+The following functions will be modified to return `Result<Metadata>` instead of `Result<()>`:
+
+- `write()`
+- `write_with()`
+
+The `writer()` and `writer_with()` return types remain unchanged as they return `Result<Writer>`.
+
+## Changes to struct `Writer`
+
+The `Writer` struct will be modified to return `Result<Metadata>` instead of `Result<()>` for the `close()` function.
+
+## Changes to trait `oio::Write` and trait `oio::MultipartWrite`
+
+The `Write` trait will be modified to return `Result<Metadata>` instead of `Result<()>` for the `close()` function.
+
+The `MultipartWrite` trait will be modified to return `Result<Metadata>` instead of `Result<()>` for the `complete_part()` 
+and `write_once` functions.
+
+## Implementation Details
+
+For services that return metadata in their write responses:
+- The metadata will be captured from the service response
+- All available fields (etag, version_id, etc.) will be populated
+
+For services that don't return metadata in write responses:
+- for `fs`: we can use `stat` to retrieve the metadata before returning. since the metadata is cached by the kernel, 
+this won't cause a performance issue.
+- for other services: A default metadata object will be returned.
+
+
+# Drawbacks
+
+- Minor breaking change for users who explicitly type the return value of write operations
+- Additional complexity in the Writer implementation
+
+# Rationale and alternatives
+
+- Provides a clean, consistent API
+- Maintains backward compatibility for users who ignore the return value
+- Improves performance by avoiding additional stat calls when possible
+
+# Prior art
+
+Similar patterns exist in other storage SDKs:
+
+- `object_store` crate returns metadata in `PutResult` after calling `put_opts`
+- AWS SDK returns metadata in `PutObjectOutput`
+- Azure SDK returns `UploadFileResponse` after uploads
+
+# Unresolved questions
+
+- None
+
+
+# Future possibilities
+
+- None