opendal 0.1.9 → 0.1.11

data/core/core/src/docs/concepts.rs

@@ -15,101 +15,104 @@
 // specific language governing permissions and limitations
 // under the License.
 
-//! The core concepts of OpenDAL's public API.
+//! The core concepts of OpenDAL's Rust API.
 //!
-//! OpenDAL provides a unified abstraction for accessing storage services.
+//! OpenDAL gives applications one storage API across services. Rust
+//! applications use four concepts: a service describes the storage backend, a
+//! builder configures it, an operator exposes operations, and layers add
+//! behavior around those operations.
 //!
-//! Public users mostly work with three concepts:
+//! For the language-independent model shared by every OpenDAL binding, see the
+//! [OpenDAL concepts guide](https://opendal.apache.org/docs/concepts/).
 //!
-//! - [`Builder`]: configures and builds a storage service.
-//! - [`Operator`]: owns a service stack and exposes storage operations such as
-//!   `read`, `write`, and `list`.
-//! - [`Layer`][crate::raw::Layer]: wraps an operator's service stack or runtime
-//!   resources to add behavior such as retry, timeout, tracing, and metrics.
+//! # Service and builder
 //!
-//! If you are interested in internal implementation details, please have a look at [`internals`][super::internals].
+//! A **service** is a storage backend such as S3, Google Cloud Storage, a
+//! local filesystem, or an in-memory store. Each service supports a different
+//! set of operations and optional features.
 //!
-//! # Builder
-//!
-//! A [`Builder`] configures and creates the underlying storage service. Service
-//! crates expose builders through [`services`][crate::services], for example
-//! [`services::Memory`][crate::services::Memory].
+//! A [`Builder`] collects one service's configuration and constructs its
+//! implementation. The [`services`][crate::services] module exposes concrete
+//! builders, while [`Operator::new`] turns a builder into a ready-to-use
+//! operator.
 //!
 //! ```text
-//! ┌─────────┐   build()   ┌─────────┐
-//! │ Builder ├────────────►│ Service │
-//! └─────────┘             └─────────┘
+//! configuration -> builder -> service -> operator
 //! ```
 //!
 //! ```no_run
+//! # use opendal_core::Result;
+//! use opendal_core::Operator;
 //! use opendal_core::services::Memory;
 //!
+//! # fn test() -> Result<()> {
 //! let builder = Memory::default();
+//! let op = Operator::new(builder)?;
+//! # let _ = op;
+//! # Ok(())
+//! # }
 //! ```
 //!
+//! OpenDAL does not make every service support every operation. Applications
+//! can inspect the operator's effective [`Capability`] before using an
+//! optional operation or option.
+//!
 //! # Operator
 //!
-//! An [`Operator`] is the public handle for a storage service stack. It stores:
+//! An [`Operator`] is the public handle for one configured service and root.
+//! It normalizes paths, validates options against effective capabilities, runs
+//! the configured layers, and dispatches each operation to the service.
 //!
-//! - a base service created from the builder.
-//! - a base [`OperationContext`] with runtime resources such as HTTP transport
-//!   and executor.
-//! - an ordered list of layers.
-//! - the composed service and context used by storage operations.
+//! Operators are cheap to clone, contain no caller-visible lifetime or service
+//! type parameter, and can be shared across threads. A clone refers to the
+//! same composed service stack. Methods that add a layer or replace runtime
+//! resources return a new operator; existing clones and in-flight operations
+//! keep their current stack.
 //!
-//! ```text
-//! ┌─────────┐   Operator::new   ┌────────────────────────────┐
-//! │ Builder ├──────────────────►│ Operator                   │
-//! └─────────┘                   │ - base service             │
-//!                               │ - base OperationContext    │
-//!                               │ - layers                   │
-//!                               │ - composed service/context │
-//!                               └────────────────────────────┘
-//! ```
+//! # Operation
 //!
-//! `Operator::new` returns a ready-to-use operator. There is no separate
-//! `finish` step.
+//! Operations are storage actions such as `read`, `write`, `stat`, `list`,
+//! `delete`, `copy`, and `rename`. Convenience methods use default options,
+//! while the corresponding `_with` methods expose operation-specific options.
 //!
 //! ```no_run
 //! # use opendal_core::Result;
-//! use opendal_core::services::Memory;
 //! use opendal_core::Operator;
+//! use opendal_core::services::Memory;
 //!
-//! # fn test() -> Result<()> {
-//! let builder = Memory::default();
-//!
-//! let op = Operator::new(builder)?;
+//! # async fn test() -> Result<()> {
+//! let op = Operator::new(Memory::default())?;
+//! let bs = op.read("abc").await?;
+//! # let _ = bs;
 //! # Ok(())
 //! # }
 //! ```
 //!
-//! - `Operator` is cheap to clone because it stores shared handles internally.
-//! - `Operator` has no generic parameters or lifetimes, so it is easy to pass
-//!   through application code.
-//! - `Operator` is `Send` and `Sync`, so it can be shared across threads.
-//! - Methods that change layers or runtime resources return a new operator;
-//!   existing clones and in-flight operations keep using their current composed
-//!   service and context.
+//! OpenDAL normalizes every path relative to the operator's root. `/`
+//! represents the root, a trailing `/` represents a directory, and any other
+//! normalized path represents a file. Operation documentation defines the
+//! observable behavior and errors. [Specifications][super::specs] define
+//! portable contracts that span multiple operations and services.
+//!
+//! # Layer and operation context
 //!
-//! # Runtime resources and layers
+//! A [`Layer`][crate::raw::Layer] adds cross-cutting behavior such as retry,
+//! timeout, tracing, or metrics. Layers form an ordered stack around a
+//! service.
 //!
-//! [`OperationContext`] carries runtime resources from the operator to services,
-//! such as [`HttpTransporter`][crate::HttpTransporter] and [`Executor`][crate::Executor].
-//! Operation arguments such as ranges, versions, and concurrency limits stay in
-//! the `Op*` argument structs used by each operation.
+//! An [`OperationContext`] carries runtime resources such as the HTTP
+//! transport and executor from the operator to the service.
+//! Operation-specific values such as ranges, versions, conditions, and
+//! concurrency remain in that operation's options.
 //!
-//! Use [`Operator::with_context`] to replace the base context. Use
-//! [`Operator::layer`] to append a layer. In both cases, OpenDAL replays the full
-//! layer list from the base service and base context to produce a fresh composed
-//! service and context.
+//! Adding a layer with [`Operator::layer`] or replacing the base context with
+//! [`Operator::with_context`] rebuilds both the service stack and the composed
+//! context from the same ordered layer list:
 //!
 //! ```text
-//! base service ─┐
-//!               ├─ layer replay ─► composed service ─┐
-//! base context ─┘                                     ├─► operation dispatch
-//!                                                     │    srv.read(&ctx, ...)
-//!                                                     ▼
-//!                                             composed context
+//! base service -----+                    +-> composed service --+
+//!                   +-> ordered layers --+                      +-> operation
+//! base context -----+                    +-> composed context --+
 //! ```
 //!
 //! ```no_run
@@ -129,40 +132,13 @@
 //! # }
 //! ```
 //!
-//! # Operations
-//!
-//! After creating an operator, use it to run operations on normalized paths.
-//!
-//! ```text
-//!                            ┌──────────────┐
-//!                  ┌────────►│ read("abc")  │
-//!                  │         └──────────────┘
-//! ┌───────────┐    │
-//! │ Operator  │    │         ┌──────────────┐
-//! │ ┌───────┐ ├────┼────────►│ write("def") │
-//! │ │Service│ │    │         └──────────────┘
-//! └─┴───────┴─┘    │
-//!                  │         ┌──────────────┐
-//!                  └────────►│ list("ghi/") │
-//!                            └──────────────┘
-//! ```
-//!
-//! We can read data with given path in this way:
-//!
-//! ```no_run
-//! # use opendal_core::Result;
-//! use opendal_core::services::Memory;
-//! use opendal_core::Operator;
-//!
-//! # async fn test() -> Result<()> {
-//! let builder = Memory::default();
-//!
-//! let op = Operator::new(builder)?;
-//! let bs: Vec<u8> = op.read("abc").await?;
-//! # Ok(())
-//! # }
-//! ```
+//! Most applications only need the public [`Operator`] API. Service and layer
+//! authors should continue with the [internals][super::internals] guide.
 //!
 //! [`Builder`]: crate::Builder
 //! [`Operator`]: crate::Operator
+//! [`Operator::new`]: crate::Operator::new
+//! [`Operator::layer`]: crate::Operator::layer
+//! [`Operator::with_context`]: crate::Operator::with_context
+//! [`Capability`]: crate::Capability
 //! [`OperationContext`]: crate::OperationContext

data/core/core/src/docs/internals/accessor.rs

@@ -15,318 +15,87 @@
 // specific language governing permissions and limitations
 // under the License.
 
-//! The internal implementation details of [`Service`].
+//! Implementing a service.
 //!
-//! [`Service`] is the core trait of OpenDAL's raw API. We operate
-//! underlying storage services via APIs provided by [`Service`].
+//! Every OpenDAL backend implements the raw [`Service`] trait. A service owns
+//! protocol-specific request construction and response handling. The operator
+//! owns path normalization, option validation, layer composition, and
+//! dispatch.
 //!
-//! # Introduction
+//! Service crates live under `core/services/<name>/`. Their public builder and
+//! configuration types construct a typed backend; the backend then implements
+//! [`Service`].
 //!
-//! [`Service`] can be split in the following parts:
+//! # Service identity and capabilities
 //!
-//! ```ignore
-//! //                  <----------Trait Bound-------------->
-//! pub trait Service: Send + Sync + Debug + Unpin + 'static {
-//!     fn info(&self) -> ServiceInfo;
-//!     fn capability(&self) -> Capability;
+//! [`Service::info`] returns immutable identity such as the scheme, root, and
+//! namespace name. Runtime resources do not belong in [`ServiceInfo`];
+//! services read them from [`OperationContext`].
 //!
-//!     type Reader: oio::Read;
-//!     type Writer: oio::Write;
-//!     type Lister: oio::List;
-//!     type Deleter: oio::Delete;
-//!     type Copier: oio::Copy;
+//! [`Service::capability`] reports the behavior implemented by that service
+//! stack. Set an operation or option capability only when the implementation
+//! satisfies the corresponding public contract. Operators reject options whose
+//! required capability is absent.
 //!
-//!     fn read(
-//!         &self,
-//!         ctx: &OperationContext,
-//!         path: &str,
-//!         args: OpRead,
-//!     ) -> impl Future<Output = Result<Self::Reader>> + MaybeSend;
-//! }
-//! ```
-//!
-//! Let's go deep into [`Service`] line by line.
-//!
-//! ## Trait Bound
-//!
-//! First we will read the declaration of [`Service`] trait:
-//!
-//! ```ignore
-//! pub trait Service: Send + Sync + Debug + Unpin + 'static {}
-//! ```
-//!
-//! There are many trait boundings here. For now, [`Service`] requires the following bound:
-//!
-//! - [`Send`]: Allow user to send between threads without extra wrapper.
-//! - [`Sync`]: Allow user to sync between threads without extra lock.
-//! - [`Debug`][std::fmt::Debug]: Allow users to print underlying debug information of service.
-//! - [`Unpin`]: Make sure `Service` can be safely moved after being pinned, so users don't need to `Pin<Box<A>>`.
-//! - `'static`: Make sure `Service` is not a short-time reference, allow users to use `Service` in closures and futures without playing with lifetime.
-//!
-//! Implementer of `Service` should take care of the following things:
-//!
-//! - Implement `Debug` for backend, but don't leak credentials.
-//! - Make sure the backend is `Send` and `Sync`, wrap the internal struct with `Arc<Mutex<T>>` if necessary.
-//!
-//! ## Associated Type
-//!
-//! The middle block of [`Service`] is our associated types. We require
-//! implementers to specify concrete operation body types, avoiding extra
-//! dynamic dispatch inside the backend and typed service wrappers.
-//!
-//! [`Service`] has five associated types so far:
-//!
-//! - `Reader`: reader returned by `read` operation.
-//! - `Writer`: writer returned by `write` operation.
-//! - `Lister`: lister returned by `list` operation.
-//! - `Deleter`: deleter returned by `delete` operation.
-//! - `Copier`: copier returned by `copy` operation.
-//!
-//! Backend implementers should take care of the following things:
-//!
-//! - Return concrete operation body types instead of dynamic trait objects like `oio::Reader`.
-//! - Use `()` as the type if the operation is not supported.
-//! - Implement every operation method; unsupported operations should return
-//!   [`ErrorKind::Unsupported`] explicitly.
-//!
-//! Runtime service wrappers that already receive a [`Servicer`] can forward the
-//! erased `oio::*` body types, but the original backend should not erase itself.
-//!
-//! OpenDAL erases these associated types at the [`ServiceDyn`] boundary.
-//! [`Servicer`] is `Arc<dyn ServiceDyn>` and is the handle used by
-//! [`Operator`] and runtime layer composition. This keeps backend and typed
-//! layer implementations concrete, while the composed operator stack can be
-//! stored and cloned through one object-safe service handle.
-//!
-//! ## API Style
-//!
-//! Every API of [`Service`] follows the same style:
-//!
-//! - All APIs have a unique [`Operation`] and [`Capability`]
-//! - All APIs are orthogonal and do not overlap with each other
-//! - Most APIs accept [`OperationContext`], `path` and `OpXxx`, and returns `RpXxx` with concrete operation bodies.
-//! - Most APIs have `async` and `blocking` variants, they share the same semantics but may have different underlying implementations.
-//!
-//! [`OperationContext`] carries operator resources composed by operator layers,
-//! such as HTTP client and executor. Backend implementations should use this
-//! context instead of storing mutable operator resources in [`ServiceInfo`].
-//!
-//! [`Service`] declares immutable identity facts via [`ServiceInfo`] and
-//! operation capability via [`Service::capability`]:
-//!
-//! ```ignore
-//! impl Service for MyBackend {
-//!     type Reader = MyReader;
-//!     type Writer = ();
-//!     type Lister = ();
-//!     type Deleter = ();
-//!     type Copier = ();
-//!
-//!     fn info(&self) -> ServiceInfo {
-//!         ServiceInfo::new(MY_SCHEME, "/", "")
-//!     }
-//!
-//!     fn capability(&self) -> Capability {
-//!         Capability {
-//!             read: true,
-//!             write: true,
-//!             ..Default::default()
-//!         }
-//!     }
-//! }
-//! ```
-//!
-//! Now that you have mastered [`Service`], let's go and implement our own backend!
-//!
-//! # Tutorial
+//! # Operation body types
 //!
-//! This tutorial implements a `duck` storage service that sends API
-//! requests to a super-powered duck. Gagaga!
+//! [`Service`] uses associated types for operation bodies:
 //!
-//! ## Scheme
-//!
-//! First of all, let's pick a good scheme for our duck service. The
-//! scheme should be unique and easy to understand. Normally we should
-//! use its formal name.
-//!
-//! For example, we will use `s3` for AWS S3 Compatible Storage Service
-//! instead of `aws` or `awss3`. This is because there are many storage
-//! vendors that provide s3-like RESTful APIs, and our s3 service is
-//! implemented to support all of them, not just AWS S3.
-//!
-//! Obviously, we can use `duck` as scheme, let's add a new constant string:
-//!
-//! ```ignore
-//! pub const DUCK_SCHEME: &str = "duck";
+//! ```text
+//! type Reader: oio::Read;
+//! type Writer: oio::Write;
+//! type Lister: oio::List;
+//! type Deleter: oio::Delete;
+//! type Copier: oio::Copy;
 //! ```
 //!
-//! ## Builder
+//! A backend returns concrete body types so its implementation and typed
+//! wrappers do not pay for dynamic dispatch. Use `()` for an unsupported body
+//! type and return [`ErrorKind::Unsupported`] from the corresponding operation
+//! entry point.
 //!
-//! Then we can implement a builder for the duck service. The [`Builder`]
-//! will provide APIs for users to configure, and they will create an
-//! instance of a particular service.
+//! OpenDAL erases these types once, at [`ServiceDyn`]. [`Servicer`] is
+//! `Arc<dyn ServiceDyn>` and is the handle used by operators and runtime layer
+//! composition. A wrapper that receives a [`Servicer`] may forward erased
+//! `oio::*` bodies, but a backend should keep its own bodies concrete.
 //!
-//! Let's create a `backend` mod under `services/duck` directory, and adding the following code.
+//! # Operation methods
 //!
-//! ```ignore
-//! use crate::raw::*;
-//! use crate::*;
+//! Each operation method receives normalized paths, an [`OperationContext`],
+//! and operation-specific arguments. The context supplies layer-composed
+//! runtime resources such as the HTTP transport and executor. Options such as
+//! ranges, versions, conditions, and concurrency remain in the operation
+//! arguments.
 //!
-//! /// Duck Storage Service support. Gagaga!
-//! ///
-//! /// # Capabilities
-//! ///
-//! /// This service can be used to:
-//! ///
-//! /// - [x] read
-//! /// - [ ] write
-//! /// - [ ] list
-//! /// - [ ] presign
-//! ///
-//! /// # Configuration
-//! ///
-//! /// - `root`: Set the work dir for backend.
-//! ///
-//! /// ## Via Builder
-//! ///
-//! /// ```no_run
-//! /// use std::sync::Arc;
-//! ///
-//! /// use anyhow::Result;
-//! /// use opendal_core::services::Duck;
-//! /// use opendal_core::Operator;
-//! ///
-//! /// #[tokio::main]
-//! /// async fn main() -> Result<()> {
-//! ///     // Create Duck backend builder.
-//! ///     let mut builder = DuckBuilder::default();
-//! ///     // Set the root for duck, all operations will happen under this root.
-//! ///     //
-//! ///     // NOTE: the root must be absolute path.
-//! ///     builder.root("/path/to/dir");
-//! ///
-//! ///     let op: Operator = Operator::new(builder)?;
-//! ///
-//! ///     Ok(())
-//! /// }
-//! /// ```
-//! #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
-//! #[serde(default)]
-//! #[non_exhaustive]
-//! pub struct DuckConfig {
-//!     pub root: Option<String>,
-//! }
+//! An implementation must:
 //!
-//! #[derive(Default, Clone)]
-//! pub struct DuckBuilder {
-//!     config: DuckConfig,
-//! }
-//! ```
-//!
-//! Now let's implement the required APIs for `DuckConfig`:
+//! - Map every advertised option to the native request without silently
+//!   dropping it.
+//! - Preserve the operation's public success, error, and atomicity contract.
+//! - Return structured OpenDAL errors with the correct [`ErrorKind`] and
+//!   useful context.
+//! - Keep credentials and other secrets out of `Debug` output and errors.
+//! - Forward cancellation and cleanup to protocol-specific readers, writers,
+//!   deleters, and copiers.
 //!
-//! ```ignore
-//! impl Configurator for DuckConfig {
-//!     type Builder = DuckBuilder;
-//!
-//!     fn into_builder(self) -> Self::Builder {
-//!         DuckBuilder { config: self }
-//!     }
-//! }
-//! ```
+//! # Adding or changing a service
 //!
-//! Note that `DuckBuilder` is part of our public API, so it needs to be
-//! documented. And any changes you make will directly affect users, so
-//! please take it seriously. Otherwise, you will be hunted down by many
-//! angry ducks.
+//! Keep configuration, request construction, operation bodies, and error
+//! parsing at their existing service boundaries. Update the facade feature and
+//! service registration when the service must be available through the
+//! `opendal` crate.
 //!
-//! Then, we can implement required APIs for `DuckBuilder`:
-//!
-//! ```ignore
-//! impl DuckBuilder {
-//!     /// Set root of this backend.
-//!     ///
-//!     /// All operations will happen under this root.
-//!     pub fn root(&mut self, root: &str) -> &mut Self {
-//!         self.config.root = if root.is_empty() {
-//!             None
-//!         } else {
-//!             Some(root.to_string())
-//!         };
-//!
-//!         self
-//!     }
-//! }
-//!
-//! impl Builder for DuckBuilder {
-//!     type Config = DuckConfig;
-//!
-//!     fn build(self) -> Result<impl Service>  {
-//!         debug!("backend build started: {:?}", &self);
-//!
-//!         let root = normalize_root(&self.config.root.clone().unwrap_or_default());
-//!         debug!("backend use root {}", &root);
-//!
-//!         Ok(DuckBackend { root })
-//!     }
-//! }
-//! ```
-//!
-//! `DuckBuilder` is ready now, let's try to play with real ducks!
-//!
-//! ## Backend
-//!
-//! I'm sure you can see it already: `DuckBuilder` will build a
-//! `DuckBackend` that implements [`Service`]. The backend is what we used
-//! to communicate with the super-powered ducks!
-//!
-//! Let's keep adding more code under `backend.rs`:
-//!
-//! ```ignore
-//! /// Duck storage service backend
-//! #[derive(Clone, Debug)]
-//! pub struct DuckBackend {
-//!     root: String,
-//! }
-//!
-//! impl Service for DuckBackend {
-//!     type Reader = DuckReader;
-//!     type Writer = ();
-//!     type Lister = ();
-//!     type Deleter = ();
-//!     type Copier = ();
-//!
-//!     fn info(&self) -> ServiceInfo {
-//!         ServiceInfo::new(DUCK_SCHEME, &self.root, "")
-//!     }
-//!
-//!     fn capability(&self) -> Capability {
-//!         Capability {
-//!             read: true,
-//!             ..Default::default()
-//!         }
-//!     }
-//!
-//!     fn read(&self, ctx: &OperationContext, path: &str, args: OpRead) -> Result<Self::Reader> {
-//!         gagaga!()
-//!     }
-//! }
-//! ```
-//!
-//! Congratulations, we have implemented a [`Service`] that can talk to
-//! Super Power Ducks!
-//!
-//! What!? There are no Super Power Ducks? So sad, but never mind, we have
-//! really powerful storage services [here](https://github.com/apache/opendal/issues/5). Welcome to pick one to implement. I promise you won't
-//! have to `gagaga!()` this time.
+//! Before advertising new behavior, reproduce it against the actual service
+//! and run the matching capability-gated behavior tests. Protocol
+//! documentation, emulators, and fabricated responses can explain an
+//! implementation, but they do not establish real-service conformance.
 //!
 //! [`Service`]: crate::raw::Service
+//! [`Service::info`]: crate::raw::Service::info
+//! [`Service::capability`]: crate::raw::Service::capability
+//! [`ServiceInfo`]: crate::raw::ServiceInfo
 //! [`ServiceDyn`]: crate::raw::ServiceDyn
 //! [`Servicer`]: crate::raw::Servicer
-//! [`Operation`]: crate::raw::Operation
-//! [`Capability`]: crate::Capability
-//! [`ServiceInfo`]: crate::raw::ServiceInfo
-//! [`OperationContext`]: crate::raw::OperationContext
-//! [`Operator`]: crate::Operator
-//! [`Builder`]: crate::Builder
-//! [`Configurator`]: crate::Configurator
+//! [`OperationContext`]: crate::OperationContext
+//! [`ErrorKind`]: crate::ErrorKind
+//! [`ErrorKind::Unsupported`]: crate::ErrorKind::Unsupported