opendal 0.1.6.pre.rc.1

data/core/src/docs/rfcs/5871_read_returns_metadata.md

@@ -0,0 +1,112 @@
+- Proposal Name: `read_returns_metadata`
+- Start Date: 2025-03-24
+- RFC PR: [apache/opendal#5871](https://github.com/apache/opendal/pull/5871)
+- Tracking Issue: [apache/opendal#5872](https://github.com/apache/opendal/issues/5872)
+
+# Summary
+
+Enhance read operations by returning metadata along with data in read operations.
+
+# Motivation
+
+Currently, read operations (`read`, `read_with`, `reader`, `reader_with`) only return the data content. Users who need metadata 
+during reads (like `Content-Type`, `ETag`, `version_id`, etc.) must make an additional `stat()` call. This is inefficient and 
+can lead to race conditions if the file is modified between the read and stat operations.
+
+Many storage services (like S3, GCS, Azure Blob) return metadata in their read responses. For example, S3's GetObject API returns 
+important metadata like `ContentType`, `ETag`, `VersionId`, `LastModified`, etc. We should expose this information to users 
+directly during read operations.
+
+# Guide-level explanation
+
+For `reader` API, we will introduce a new method `metadata()` that returns metadata:
+
+```rust
+// Before
+let data = op.reader("path/to/file").await?.read(..).await?;
+let meta = op.stat("path/to/file").await?;
+if let Some(etag) = meta.etag() {
+    println!("ETag: {}", etag);
+}
+
+// After
+let reader = op.reader("path/to/file").await?;
+let meta = reader.metadata();
+if let Some(etag) = meta.etag() {
+    println!("ETag: {}", etag);
+}
+let data = reader.read(..).await?;
+```
+The new API will be provided alongside existing functionality, allowing users to continue using current `reader` methods without modification.
+
+For backward compatibility and to minimize migration costs, We won't change the existing `read` API. Anyone who wants 
+to obtain metadata during reading can use the new reader operations instead.
+
+# Reference-level explanation
+
+## Changes to `Reader` API
+
+The `impl Reader` will be modified to include a new function `metadata()` that returns metadata.
+
+```rust
+impl Reader {
+    // Existing fields...
+    
+    fn metadata(&self) -> &Metadata {}
+}
+```
+
+## Changes to struct `raw::RpRead`
+
+The `raw::RpRead` struct will be modified to include a new field `metadata` that stores the metadata returned by the read operation.
+Existing fields will be evaluated and potentially removed if they become redundant.
+
+```rust
+pub struct RpRead {
+    // New field to store metadata 
+    metadata: Metadata,
+}
+```
+
+
+## Implementation Details
+
+For services that return metadata in their read responses:
+- The metadata will be captured from the service response.
+- All available fields (content_type, etag, version_id, last_modified, etc.) will be populated
+
+For services that don't return metadata in read responses:
+- We'll make an additional `stat` call to fetch the metadata and populate the `metadata` field in `raw::RpRead`.
+
+Special considerations:
+- We should always return total object size in the metadata, even if it's not part of the read response
+- For range reads, the metadata should reflect the full object's properties (like total size) rather than the range
+- For versioned objects, the metadata should include version information if available
+
+# Drawbacks
+
+- Additional memory overhead for storing metadata during reads
+- Potential complexity in handling metadata for range reads
+
+# Rationale and alternatives
+
+- Maintains full backward compatibility with existing read operations
+- Improves performance by avoiding additional stat calls
+- Aligns with common storage service APIs (S3, GCS, Azure)
+
+# Prior art
+
+Similar patterns exist in other storage SDKs:
+
+- `object_store` crate returns metadata in `GetResult` after calling `get_opts`
+- AWS S3 SDK returns comprehensive metadata in `GetObjectOutput`
+- Azure Blob SDK returns properties and metadata in `DownloadResponse`
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- Once we return metadata during reader initialization, we can optimize `ReadContext::parse_into_range` by using the 
+`content_length` from `metadata` directly, eliminating the need for an additional `stat` call

data/core/src/docs/rfcs/6189_remove_native_blocking.md

@@ -0,0 +1,106 @@
+- Proposal Name: `remove_native_blocking`
+- Start Date: 2025-05-15
+- RFC PR: [apache/opendal#6189](https://github.com/apache/opendal/pull/6189)
+- Tracking Issue: [apache/opendal#6190](https://github.com/apache/opendal/issues/6190)
+
+# Summary
+
+Remove native blocking support from OpenDAL and use `block_on` in the async runtime instead.
+
+# Motivation
+
+OpenDAL projects offer both native async and blocking support. This means we can provide true blocking functionality when services are capable of it. The advantage is that users who only require blocking operations can enjoy optimal performance. For example, `fs` offers native blocking support, allowing users to utilize `fs` without any additional overhead or performance loss compared to using `tokio_fs`.
+
+However, this approach has several drawbacks:
+
+- Huge maintenance burden: We need to maintain two versions of the same code, which is a significant overhead.
+- Increased complexity: The codebase becomes more complex, making it harder to understand and maintain.
+- API inconsistency: The API for blocking and async operations may differ, leading to confusion for users, for example, concurrent reading support is not available in blocking APIs.
+
+Apart from the above drawbacks, we also find that the following facts:
+
+- Only a few services have native blocking support, such as `fs`, `memory` and `moka`.
+- Only a few users are using `blocking::Operator` (based on [search results from github](https://github.com/search?q=opendal%3A%3Ablocking::Operator&type=code))
+
+So I propose to remove the native blocking support from OpenDAL and use `block_on` in the async runtime instead. This will simplify the codebase, reduce maintenance overhead, and provide a more consistent API for users.
+
+# Guide-level explanation
+
+I plan to move `blocking::Operator` to the `opendal::blocking` module, which will offer a blocking interface for users. The `blocking::Operator` will utilize `block_on` within an async runtime to execute asynchronous operations in a blocking way.
+
+- `opendal::blocking::Operator` will become `opendal::blocking::Operator`
+- `opendal::BlockingReader` will become `opendal::blocking::Reader`
+- `opendal::BlockingWriter` will become `opendal::blocking::Writer`
+- `opendal::BlockingLister` will become `opendal::blocking::Lister`
+- `opendal::BlockingDeleter` will become `opendal::blocking::Deleter`
+
+The public API will remain unchanged. The only impacts for users are as follows:
+
+- `blocking::Operator` can no longer be used in an async context
+- An async runtime is now required to use the blocking APIs
+
+# Reference-level explanation
+
+I plan to remove all blocking APIs in `oio::Access`:
+
+```diff
+pub trait Access: Send + Sync + Debug + Unpin + 'static { 
+  ...
+
+  - fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result<RpCreateDir> {
+  -     self.as_ref().blocking_create_dir(path, args)
+  - }
+  - 
+  - fn blocking_stat(&self, path: &str, args: OpStat) -> Result<RpStat> {
+  -     self.as_ref().blocking_stat(path, args)
+  - }
+  - 
+  - fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> {
+  -     self.as_ref().blocking_read(path, args)
+  - }
+  - 
+  - fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> {
+  -     self.as_ref().blocking_write(path, args)
+  - }
+  - 
+  - fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> {
+  -     self.as_ref().blocking_delete()
+  - }
+  - 
+  - fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> {
+  -     self.as_ref().blocking_list(path, args)
+  - }
+  - 
+  - fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result<RpCopy> {
+  -     self.as_ref().blocking_copy(from, to, args)
+  - }
+  - 
+  - fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result<RpRename> {
+  -     self.as_ref().blocking_rename(from, to, args)
+  - }
+}
+```
+
+Due to this change, we will remove all blocking implementations in services and layers. All existing public APIs that start with `Blocking` will be moved into the `blocking` module. For example, `opendal::blocking::Operator` will be relocated to `opendal::blocking::Operator`.
+
+# Drawbacks
+
+Users who rely on blocking APIs will need to adapt their code to use the new `blocking` module. Users might lost the performance benefits of using native blocking APIs, as they will now be using `block_on` in an async runtime.
+
+# Rationale and alternatives
+
+- We could keep the native blocking support, but this would increase the maintenance burden and complexity of the codebase.
+- We could provide a separate crate for blocking APIs, but this would lead to API inconsistency and confusion for users.
+
+# Prior art
+
+`reqwest` is using the same approach. It provides a blocking client in `reqwest::blocking` module, which is a wrapper around the async client. This allows users to use blocking APIs without the need for native blocking support.
+
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/6209_glob_support.md

@@ -0,0 +1,132 @@
+- Proposal Name: `glob_support`
+- Start Date: 2025-05-21
+- RFC PR: [apache/opendal#6209](https://github.com/apache/opendal/pull/6209)
+- Tracking Issue: [apache/opendal#6210](https://github.com/apache/opendal/issues/6210)
+
+# Summary
+
+Add support for matching file paths against Unix shell style patterns (glob) in OpenDAL.
+
+# Motivation
+
+Glob patterns are a widely used way to filter files based on their paths. They provide a simple and intuitive syntax for matching multiple files with similar path patterns. Adding glob support to OpenDAL would enable users to easily filter and process files that match certain patterns without having to implement this functionality themselves.
+
+Currently, users who want to filter objects based on patterns have to list all objects and then apply filters manually, which is verbose and not very intuitive. By providing native glob support, we can make this common operation more convenient and efficient.
+
+# Guide-level explanation
+
+With glob support, users can easily match files based on patterns. The API would be available as an option on the `list_with` and `lister_with` methods, allowing users to filter entries that match the provided glob pattern.
+
+For example:
+
+```rust
+// Get all jpeg files in the media directory and its subdirectories
+let entries: Vec<Entry> = op.list_with("media/").glob("**/*.jpg").await?;
+
+// Process entries
+for entry in entries {
+   do_something(&entry);
+}
+
+// Or use a lister for streaming access
+let mut lister = op.lister_with("media/").glob("**/*.jpg").await?;
+
+while let Some(entry) = lister.next().await? {
+   do_something(&entry);
+}
+```
+
+The glob syntax would support common patterns like:
+
+- `*` - Match any sequence of non-separator characters
+- `?` - Match any single non-separator character
+- `**` - Match any sequence of characters including separators
+- `{a,b}` - Match either a or b
+- `[ab]` - Match either a or b
+- `[a-z]` - Match any character in range a-z
+
+The API would be integrated into the existing builder pattern.
+
+# Reference-level explanation
+
+The implementation would involve:
+
+1. Implementing a pattern matching logic for glob expressions. This can be a simplified version focusing on common use cases like `*`, `?`, and `**`.
+
+2. Modifying the `FunctionLister` and `FutureLister` to accept a glob pattern and filter entries accordingly.
+
+The `GlobMatcher` struct would be an internal implementation detail that encapsulates the parsed glob pattern and the matching logic.
+
+```rust
+// This is an internal implementation detail, not exposed in the public API
+struct GlobMatcher {
+    // internal representation of the pattern
+}
+
+impl GlobMatcher {
+    fn new(pattern: &str) -> Result<Self> {
+        // Parse the pattern string
+        // ...
+    }
+
+    fn matches(&self, path: &str) -> bool {
+        // Perform the matching logic
+        // ...
+    }
+}
+```
+
+The implementation would be built on top of the existing listing capabilities. Pattern matching will primarily occur client-side. However, for services with native glob/pattern support (e.g., GCS `matchGlob`, Redis `SCAN` with `MATCH`), OpenDAL will delegate the pattern matching to the service where possible to improve efficiency.
+
+# Drawbacks
+
+- While the API surface change is minimized by integrating with the existing builder pattern, it still introduces a new concept (glob patterns) for users to learn.
+- Implementing server-side delegation adds complexity, as OpenDAL needs to identify services with native support and translate glob patterns to their specific syntax.
+- For services without native glob support, client-side matching still requires listing all potentially relevant entries first, which might be inefficient for very large directories or complex patterns.
+- Ensuring consistent behavior between client-side and various server-side implementations of glob matching can be challenging.
+
+# Rationale and alternatives
+
+This design integrates glob filtering into the existing builder pattern API, providing a natural extension to current functionality. We will implement our own pattern matching logic, focusing on commonly used glob syntax (e.g., `*`, `?`, `**`, `*.parquet`) to avoid the complexity of full-featured glob libraries designed for local file systems. This approach allows for a lean implementation tailored to object storage path matching.
+
+Where services offer native pattern matching capabilities, OpenDAL will delegate to them. This leverages server-side efficiencies. For other services, client-side filtering will be applied after listing entries.
+
+Alternatives considered:
+
+1. Not implementing this feature and letting users implement filtering manually
+   - This puts the burden on users and leads to repetitive code.
+   - Users might implement inefficient or buggy filtering.
+
+2. Relying entirely on an external glob library
+    - Most glob libraries include complex logic for local file systems (e.g., directory traversal, symlink handling) which is not needed for OpenDAL's path matching.
+    - This can introduce unnecessary dependencies and overhead.
+
+3. Implementing server-side filtering *only* for services that support it, without a client-side fallback.
+   - This would lead to inconsistent feature availability across services.
+   - A client-side fallback ensures glob functionality is universally available.
+
+4. Adding a more general filtering API instead of specifically glob patterns
+   - While potentially more flexible, this would be more complex to design and implement.
+   - Glob patterns are a well-understood and widely used standard for this type of path matching, covering the majority of use cases.
+
+Not providing a unified glob capability means users continue to write verbose code for a common operation, or face inconsistencies if trying to leverage service-specific features directly. OpenDAL aims to provide a consistent and ergonomic interface for such common tasks.
+
+# Prior art
+
+Many file system and storage APIs provide glob or similar pattern matching capabilities:
+
+- The [glob](https://crates.io/crates/glob) crate in Rust
+- Python's [glob](https://docs.python.org/3/library/glob.html) module
+- Node.js [glob](https://www.npmjs.com/package/glob) package
+- Unix shells like bash with built-in glob support
+
+Most implementations provide similar syntax, though there are some variations. We should align with established Rust patterns.
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+- If services add native support for glob filtering, we could optimize by pushing the filtering to the server side
+- We could extend the API to support more advanced pattern matching like regex

data/core/src/docs/rfcs/6213_options_api.md

@@ -0,0 +1,142 @@
+- Proposal Name: `options_api`
+- Start Date: 2025-05-22
+- RFC PR: [apache/opendal#6213](https://github.com/apache/opendal/pull/6213)
+- Tracking Issue: [apache/opendal#6214](https://github.com/apache/opendal/issues/6214)
+
+# Summary
+
+Introduce a new options-based API pattern for OpenDAL to simplify the construction of operations.
+
+# Motivation
+
+OpenDAL currently offers two API patterns. 
+
+The first is a simple pattern that provides the most basic operations, such as `read`, `write`, and `delete`. These operations do not accept any options and serve as quick shortcuts for users to interact with OpenDAL. 
+
+```rust
+let data = op.read("example.txt").await?;
+let _ = op.write("example.txt", data).await?;
+```
+
+The second is the builder pattern API, which includes methods like `read_with`, `write_with`, and `delete_with`. This approach uses a builder pattern to construct operations with customizable options, offering greater flexibility and allowing users to tailor operations to their specific needs.
+
+```rust
+let data = op
+    .read_with("example.txt")
+    .if_match(etag).await?;
+let _ = op.write_with("example.txt", data)
+    .chunk(1024 * 1024)
+    .concurrent(8)
+    .await?;
+```
+
+The builder pattern is designed to make the OpenDAL API more flexible and extensible, but it can be cumbersome for users who are trying to construct complex operations that require tuning options at runtime.
+
+For examples, we have seen users write code like:
+
+```rust
+let mut write = self
+    .core
+    .write_with(&path, bs)
+    .append(kwargs.append.unwrap_or(false));
+if let Some(chunk) = kwargs.chunk {
+    write = write.chunk(chunk);
+}
+if let Some(content_type) = &kwargs.content_type {
+    write = write.content_type(content_type);
+}
+if let Some(content_disposition) = &kwargs.content_disposition {
+    write = write.content_disposition(content_disposition);
+}
+if let Some(cache_control) = &kwargs.cache_control {
+    write = write.cache_control(cache_control);
+}
+if let Some(user_metadata) = kwargs.user_metadata.take() {
+    write = write.user_metadata(user_metadata);
+}
+write.call().map(|_| ()).map_err(format_pyerr)
+```
+
+This makes the code hard to read and maintain, especially when there are many options to set.
+
+So I propose to introduce an option based API pattern that allows users to set options in a more structured and readable way. 
+
+This new API will allow users to create structs like `ReadOptions`  which contains all the options they want to set for an operation. They can then pass this struct to the operation, making the code cleaner and easier to understand.
+
+# Guide-level explanation
+
+We will have the following new structs:
+
+```rust
+pub struct ReadOptions {
+  pub if_match: Option<String>,
+  pub if_none_match: Option<String>,
+}
+```
+
+And we will expose APIs like:
+
+```rust
+impl Operator {
+    pub async fn read_options(&self, path: &str, opts: ReadOptions) -> Result<Buffer> {
+        ...
+    }
+}
+```
+
+Users can then use this API like:
+
+```rust
+let opts = ReadOptions {
+    version: Some(version_id),
+    ...ReadOptions::default(),
+};
+let data = op.read_options(path, opts).await?;
+```
+
+# Reference-level explanation
+
+We will introduce an options API by creating new structs for each operation, with each struct containing all configurable options for that specific operation.
+
+Before calling APIs on `oio::Access`, we will convert `ReadOptions` to `OpRead` and `OpReader`.
+
+# Drawbacks
+
+This RFC will add a new API patterns to OpenDAL, which may increase the complexity of the codebase and the learning curve for new users.
+
+# Rationale and alternatives
+
+## Expose `OpXxx` API AS-IS
+
+It’s a valid alternative to expose the `OpXxx` API as is. However, OpenDAL’s public API is not a direct one-to-one mapping from `Operator` to `oio::Access`. For instance, concurrency and chunk handling for read operations are implemented within the public API, which is why we have a separate `OpReader` struct.
+
+Therefore, we need a new struct that can handle all of these aspects simultaneously.
+
+# Prior art
+
+`object_store` has APIs like `read_opts`:
+
+```rust
+trait ObjectStore {
+    async fn get_opts(&self, path: &Path, options: GetOptions) {..}
+}
+
+pub struct GetOptions {
+    pub if_match: Option<String>,
+    pub if_none_match: Option<String>,
+    pub if_modified_since: Option<DateTime<Utc>>,
+    pub if_unmodified_since: Option<DateTime<Utc>>,
+    pub range: Option<GetRange>,
+    pub version: Option<String>,
+    pub head: bool,
+    pub extensions: Extensions,
+}
+```
+
+# Unresolved questions
+
+None
+
+# Future possibilities
+
+None

data/core/src/docs/rfcs/README.md

@@ -0,0 +1,62 @@
+# RFCs - OpenDAL Active RFC List
+
+RFCs power OpenDAL's development.
+
+The "RFC" (request for comments) process is intended to provide a consistent and controlled path for changes to OpenDAL (such as new features) so that all stakeholders can be confident about the direction of the project.
+
+Many changes, including bug fixes and documentation improvements, can be implemented and reviewed via the normal GitHub pull request workflow.
+
+Some changes, though, are "substantial" and we ask that these be put through a bit of a design process and produce a consensus among the OpenDAL community.
+
+### Which kinds of changes require an RFC?
+
+Any substantial change or addition to the project that would require a significant amount of work to implement should generally be an RFC. 
+
+Some examples include:
+
+- A new feature that creates a new public API or raw API. 
+- The removal of features that already shipped as part of the release.
+- A big refactor of existing code or reorganization of code into new modules.
+
+Those are just a few examples. Ultimately, the judgment call of what constitutes a big enough change to warrant an RFC is left to the project maintainers.
+
+If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first.
+
+## Before creating the RFC
+
+Preparing in advance before submitting an RFC hastily can increase its chances of being accepted. If you have proposals to make, it is advisable to engage in some preliminary groundwork to facilitate a smoother process.
+
+It is great to seek feedback from other project developers first, as this can help validate the viability of the RFC. To ensure a sustained impact on the project, it is important to work together and reach a consensus.
+
+Common preparatory steps include presenting your idea on platforms such as GitHub [issues](https://github.com/apache/opendal/issues/) or [discussions](https://github.com/apache/opendal/discussions/categories/ideas), or engaging in discussions through our [email list](https://opendal.apache.org/community/#mailing-list) or [Discord server](https://opendal.apache.org/discord). 
+
+## The RFC process
+
+- Fork the [OpenDAL repo](https://github.com/apache/opendal) and create your branch from `main`.
+- Copy [`0000_example.md`] to `0000-my-feature.md` (where "my-feature" is descriptive). Don't assign an RFC number yet; This is going to be the PR number, and we'll rename the file accordingly if the RFC is accepted.
+- Submit a pull request. As a pull request, the RFC will receive design feedback from the larger community, and the author should be prepared to revise it in response.
+- Now that your RFC has an open pull request, use the issue number of this PR to update your `0000-` prefix to that number.
+- Build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don't receive any comments. Feel free to reach OpenDAL maintainers for help.
+- RFCs rarely go through this process unchanged, especially as alternatives and drawbacks are shown. You can make edits, big and small, to the RFC to clarify or change the design, but make changes as new commits to the pull request, and leave a comment on the pull request explaining your changes. Specifically, do not squash or rebase commits after they are visible on the pull request.
+- The RFC pull request lasts for three days after the last update. After that, the RFC will be accepted or declined based on the consensus reached in the discussion.
+- For the accepting of an RFC, we will require approval from at least three maintainers.
+- Once the RFC is accepted, please create a tracking issue and update links in RFC. And then the PR will be merged and the RFC will become 'active' status.
+
+## Implementing an RFC
+
+An active RFC does not indicate the priority assigned to its implementation,
+nor does it imply that a developer has been specifically assigned the task of implementing the feature.
+
+The RFC author is encouraged to submit an implementation after the RFC has been accepted.
+Nevertheless, it is not obligatory for them to do so.
+
+Accepted RFCs may represent features that can wait until a developer chooses to work on them.
+Each accepted RFC is associated with an issue in the OpenDAL repository, which tracks its implementation.
+
+If you are interested in implementing an RFC but are unsure if someone else is already working on it,
+feel free to inquire by leaving a comment on the associated issue.
+
+## Some useful tips
+
+- The author of an RFC may not be the same one as the implementer. Therefore, when submitting an RFC, it is advisable to include sufficient information.
+- If modifications are needed for an accepted RFC, please submit a new pull request or create a new RFC to propose changes.