opendal 0.1.6.pre.rc.1-arm64-darwin-23

data/core/src/services/vercel_artifacts/docs.md

@@ -0,0 +1,40 @@
+## Capabilities
+
+This service can be used to:
+
+- [ ] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] ~~copy~~
+- [ ] ~~rename~~
+- [ ] ~~list~~
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Configuration
+
+- `access_token`: set the access_token for Rest API
+
+You can refer to [`VercelArtifactsBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```no_run
+use anyhow::Result;
+use opendal::services::VercelArtifacts;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = VercelArtifacts::default()
+        .access_token("xxx");
+
+    let op: Operator = Operator::new(builder)?.finish();
+    Ok(())
+}
+```

data/core/src/services/vercel_blob/docs.md

@@ -0,0 +1,45 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [ ] rename
+- [x] list
+- [ ] presign
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the work directory for backend
+- `token`: VercelBlob token, environment var `BLOB_READ_WRITE_TOKEN`
+
+You can refer to [`VercelBlobBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::VercelBlob;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = VercelBlob::default()
+        // set the storage bucket for OpenDAL
+        .root("/")
+        // set the token for OpenDAL
+        .token("you_token");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/webdav/docs.md

@@ -0,0 +1,49 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [x] list
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Notes
+
+Bazel Remote Caching and Ccache HTTP Storage is also part of this service.
+Users can use `webdav` to connect those services.
+
+## Configuration
+
+- `endpoint`: set the endpoint for webdav
+- `root`: Set the work directory for backend
+
+You can refer to [`WebdavBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::Webdav;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = Webdav::default()
+        .endpoint("127.0.0.1")
+        .username("xxx")
+        .password("xxx");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/webhdfs/docs.md

@@ -0,0 +1,90 @@
+There two implementations of WebHDFS REST API:
+
+- Native via HDFS Namenode and Datanode, data are transferred between nodes directly.
+- [HttpFS](https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/index.html) is a gateway before hdfs nodes, data are proxied.
+
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [ ] copy
+- [ ] rename
+- [x] list
+- [ ] ~~presign~~
+- [ ] blocking
+
+## Differences with HDFS
+
+[Hdfs][crate::services::Hdfs] is powered by HDFS's native java client. Users need to set up the HDFS services correctly. But webhdfs can access from HTTP API and no extra setup needed.
+
+## WebHDFS Compatibility Guidelines
+
+### File Creation and Write
+
+For [File creation and write](https://hadoop.apache.org/docs/r3.1.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File) operations,
+OpenDAL WebHDFS is optimized for Hadoop Distributed File System (HDFS) versions 2.9 and later. 
+This involves two API calls in webhdfs, where the initial `put` call to the namenode is redirected to the datanode handling the file data.
+The optional `noredirect` flag can be set to prevent redirection. If used, the API response body contains the datanode URL, which is then utilized for the subsequent `put` call with the actual file data.
+OpenDAL automatically sets the `noredirect` flag with the first `put` call. This feature is supported starting from HDFS version 2.9.
+
+### Multi-Write Support
+
+OpenDAL WebHDFS supports multi-write operations by creating temporary files in the specified `atomic_write_dir`.
+The final concatenation of these temporary files occurs when the writer is closed.
+However, it's essential to be aware of HDFS concat restrictions for earlier versions,
+where the target file must not be empty, and its last block must be full. Due to these constraints, the concat operation might fail for HDFS 2.6.
+This issue, identified as [HDFS-6641](https://issues.apache.org/jira/browse/HDFS-6641), has been addressed in later versions of HDFS.
+
+In summary, OpenDAL WebHDFS is designed for optimal compatibility with HDFS, specifically versions 2.9 and later.
+
+
+
+## Configurations
+
+- `root`: The root path of the WebHDFS service.
+- `endpoint`: The endpoint of the WebHDFS service.
+- `delegation`: The delegation token for WebHDFS.
+- `atomic_write_dir`: The tmp write dir of multi write for WebHDFS.Needs to be configured for multi write support.
+
+Refer to [`Builder`]'s public API docs for more information.
+
+## Examples
+
+### Via Builder
+
+```rust,no_run
+use std::sync::Arc;
+
+use anyhow::Result;
+use opendal::services::Webhdfs;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let mut builder = Webhdfs::default()
+        // set the root for WebHDFS, all operations will happen under this root
+        //
+        // Note:
+        // if the root exists, the builder will automatically create the
+        // root directory for you
+        // if the root exists and is a directory, the builder will continue working
+        // if the root exists and is a file, the builder will fail on building backend
+        .root("/path/to/dir")
+        // set the endpoint of webhdfs namenode, controlled by dfs.namenode.http-address
+        // default is http://127.0.0.1:9870
+        .endpoint("http://127.0.0.1:9870")
+        // set the delegation_token for builder
+        .delegation("delegation_token")
+        // set atomic_write_dir for builder
+        .atomic_write_dir(".opendal_tmp/");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/src/services/yandex_disk/docs.md

@@ -0,0 +1,45 @@
+## Capabilities
+
+This service can be used to:
+
+- [x] stat
+- [x] read
+- [x] write
+- [x] create_dir
+- [x] delete
+- [x] copy
+- [x] rename
+- [x] list
+- [ ] presign
+- [ ] blocking
+
+## Configuration
+
+- `root`: Set the work directory for backend
+- `access_token` YandexDisk oauth access_token
+
+You can refer to [`YandexDiskBuilder`]'s docs for more information
+
+## Example
+
+### Via Builder
+
+```rust,no_run
+use anyhow::Result;
+use opendal::services::YandexDisk;
+use opendal::Operator;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // create backend builder
+    let mut builder = YandexDisk::default()
+        // set the storage bucket for OpenDAL
+        .root("/")
+        // set the access_token for OpenDAL
+        .access_token("test");
+
+    let op: Operator = Operator::new(builder)?.finish();
+
+    Ok(())
+}
+```

data/core/tests/behavior/README.md

@@ -0,0 +1,77 @@
+# Behavior Test for OpenDAL
+
+Behavior tests are used to make sure every service works correctly.
+
+To support different testing backends simultaneously, we use `environment value` to carry the backend config.
+
+## Setup
+
+### Install Docker
+
+Currently, opendal uses Docker to set up environment for behaviour tests. If you are a MacOS user, The docker does not work properly by default(Docker Desktop) in the MacOS environment. It is strongly recommended to use **OrbStack** instead of Docker Desktop. You can install orbstack using Homebrew.
+
+```shell
+brew install orbstack
+```
+
+### Prepare Env
+
+To run the behavior tests, please copy the `.env.example`, which is at project root, to `.env` and change the values on need.
+
+Take `fs` for example, we need to change to enable behavior test on `fs` on `/tmp`.
+
+```dotenv
+OPENDAL_FS_ROOT=/path/to/dir/
+```
+
+into
+
+```doten
+OPENDAL_FS_ROOT=/tmp/
+```
+
+Notice: If the env variables are not set, all behavior tests will be skipped by default.
+
+## Run
+
+Use `OPENDAL_TEST` to control which service to test:
+
+```shell
+OPENDAL_TEST=fs cargo test behavior --features tests
+```
+
+To run certain types of tests(such as `write`), we use `behavior::test_write`.
+
+```shell
+OPENDAL_TEST=fs cargo test behavior::test_write --features tests
+```
+
+You can also run specific test(such as `test_stat_dir`) for specific backend.
+
+```shell
+OPENDAL_TEST=fs cargo test behavior::test_stat_dir --features tests
+```
+
+## Debug
+
+To debug a behavior test, you can:
+
+- Add env `RUST_LOG=debug` to enable logging
+- Add env `RUST_BACKTRACE=full` to enable the full backtrace
+- Add `--show-output` to show the whole output even when test succeeded.
+
+Take `memory` service as an example, the full command will be:
+
+```shell
+RUST_LOG=debug RUST_BACKTRACE=full OPENDAL_TEST=memory cargo test behavior --features tests -- --show-output
+```
+
+You use `export` to avoid set env every time:
+
+```shell
+export RUST_LOG=debug 
+export RUST_BACKTRACE=full 
+OPENDAL_TEST=memory cargo test behavior --features tests -- --show-output
+```
+
+For more details, please visit [cargo test](https://doc.rust-lang.org/cargo/commands/cargo-test.html) or run the command `cargo test --help`.