hashserver 1.0__tar.gz

hashserver-1.0/LICENSE.txt

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Author: Sjoerd de Vries, MBI platform.
+Copyright (c) 2023-2026 CNRS. 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hashserver-1.0/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.4
+Name: hashserver
+Version: 1.0
+Summary: Simple FastAPI-based hash server
+Author: Sjoerd de Vries
+License-Expression: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: fastapi
+Requires-Dist: uvicorn[standard]
+Requires-Dist: typing-extensions
+Dynamic: license-file
+
+# Hashserver
+
+A lightweight, content-addressed file server over HTTP.
+
+Hashserver stores and serves opaque binary buffers keyed by their cryptographic checksum. You PUT a buffer with its checksum in the URL; you GET it back by the same checksum. There are no filenames, no directories, no metadata — just content and its hash.
+
+The hash algorithm is configurable: SHA-256 (default) or SHA3-256.
+
+## Why content-addressed storage?
+
+Content-addressed storage (CAS) is a well-established pattern used by Git, IPFS, Docker registries, and many other systems. Identifying data by its cryptographic hash gives you automatic deduplication, trivially verifiable integrity, and strong reproducibility guarantees.
+
+Hashserver brings these benefits to any project that needs a simple HTTP-based buffer store. It is intentionally minimal: a single ASGI application backed by a directory of files, designed to be easy to deploy, easy to integrate, and easy to reason about.
+
+## Relationship to Seamless
+
+Hashserver was originally developed as the buffer-serving component of [Seamless](https://github.com/sjdv1982/seamless), a framework for reproducible, reactive computational workflows. In Seamless, all data — inputs, source code, and results — is represented as a tree of checksums, and hashserver provides the storage layer that maps those checksums back to actual data.
+
+However, **hashserver has no dependency on Seamless** and no knowledge of it. It is a generic content-addressed file server that is useful in any context where you need to store and retrieve buffers by hash — caching layers, artifact stores, reproducible pipelines, or your own CAS-backed application. It is published as an independent PyPI package for exactly this reason.
+
+## Features
+
+- **Content-addressed**: buffers are stored and retrieved by their cryptographic checksum.
+- **Configurable hash algorithm**: SHA-256 (default) or SHA3-256, selected at startup.
+- **Integrity-verified reads**: every buffer is re-checksummed on GET to detect corruption.
+- **Prefix directory layout**: by default, buffers are stored under a two-character prefix subdirectory (e.g. `ab/ab3f7c...`) to avoid filesystem performance problems with large flat directories. A flat layout is also supported.
+- **Extra read-only directories**: additional buffer directories can be mounted as fallback read sources.
+- **Promises**: a client can announce that a buffer will be uploaded soon via `PUT /promise/{checksum}`. Other clients reading that checksum will wait for the upload rather than getting a 404.
+- **Concurrent-safe**: in-flight PUT requests are tracked so concurrent GETs and batch queries return consistent results. Lock files are respected for external writers.
+- **Multiple instances**: several hashserver processes can safely share the same buffer directory.
+- **Lightweight**: built on FastAPI/Starlette — no database, no external services.
+- **Flexible deployment**: run as a CLI tool, under any ASGI server, or via Docker Compose.
+
+## Installation
+
+```bash
+pip install hashserver
+```
+
+Or with conda:
+
+```bash
+mamba env create --file environment.yml
+conda activate hashserver
+```
+
+## Quick start
+
+Serve buffers from a local directory:
+
+```bash
+hashserver ./my-buffers
+```
+
+This starts the server under uvicorn on port 8000. Run `hashserver -h` for all options.
+
+### Storing and retrieving a buffer
+
+```bash
+# Start a writable server
+hashserver ./my-buffers --writable
+
+# Compute the SHA-256 checksum and upload
+CHECKSUM=$(python3 -c "
+import hashlib, sys
+print(hashlib.sha256(open(sys.argv[1],'rb').read()).hexdigest())
+" myfile.bin)
+curl -X PUT --data-binary @myfile.bin http://localhost:8000/$CHECKSUM
+
+# Download
+curl -O http://localhost:8000/$CHECKSUM
+```
+
+To use SHA3-256 instead, start the server with `--hash-algorithm sha3-256` and hash your files with `hashlib.sha3_256`.
+
+## API
+
+### Retrieving buffers
+
+**`GET /{checksum}`** — Retrieve a buffer by its hex checksum. The server verifies the checksum before sending the response. Returns the raw buffer (200), or 404 if not found.
+
+### Storing buffers
+
+Requires `--writable`.
+
+**`PUT /{checksum}`** — Upload a buffer. The request body is the raw data; the server verifies that its checksum matches the URL. Returns 200 on success, 201 if the buffer already existed, or 400 on checksum mismatch.
+
+**`PUT /promise/{checksum}`** — Announce that a buffer will be uploaded soon. Returns 202 with the promise TTL. While a promise is active, GET requests for that checksum will wait rather than returning 404, and `/has` queries will report the checksum as present.
+
+### Querying availability
+
+**`GET /has`** — Batch existence check. Send a JSON list of checksums in the request body. Returns a JSON list of booleans. Includes both on-disk buffers and active promises.
+
+**`GET /has-now`** — Same as `/has`, but excludes promises — only reports buffers that are already on disk.
+
+**`GET /buffer-length`** — Batch size query. Send a JSON list of checksums in the request body. Returns a JSON list of integers: the buffer size in bytes, or 0 if not present. Promised checksums are reported as `true`.
+
+### Health
+
+**`GET /healthcheck`** — Returns "OK". Useful for load balancer probes.
+
+## Configuration
+
+### CLI flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `directory` | Buffer storage directory (positional, required) | — |
+| `--writable` | Enable PUT endpoints | off |
+| `--hash-algorithm` | Hash algorithm: `sha3-256` or `sha-256` | `sha-256` |
+| `--layout` | Directory layout: `prefix` or `flat` | `prefix` |
+| `--extra-dirs` | Semicolon-separated list of extra read-only buffer directories | — |
+| `--host` | Listen address | `127.0.0.1` |
+| `--port` | Listen port | `8000` |
+| `--port-range START END` | Pick a random free port in range (mutually exclusive with `--port`) | — |
+| `--status-file` | JSON file for reporting server status | — |
+| `--timeout` | Shut down after this many seconds of inactivity | — |
+
+### Environment variables
+
+When running under an external ASGI server (e.g. `uvicorn hashserver:app`), configure via environment variables instead:
+
+| Variable | Equivalent flag |
+|----------|----------------|
+| `HASHSERVER_DIRECTORY` | `directory` |
+| `HASHSERVER_WRITABLE` | `--writable` (set to `1` or `true`) |
+| `HASHSERVER_HASH_ALGORITHM` | `--hash-algorithm` |
+| `HASHSERVER_LAYOUT` | `--layout` |
+| `HASHSERVER_EXTRA_DIRS` | `--extra-dirs` |
+
+### Docker Compose
+
+```bash
+export HASHSERVER_PORT=8000
+export HASHSERVER_HOST=0.0.0.0
+export HASHSERVER_DIRECTORY=./buffers
+export HASHSERVER_WRITABLE=1
+docker compose up -d
+```
+
+Container user/group ID can be set with `HASHSERVER_USER_ID` and `HASHSERVER_GROUP_ID` (both default to 0).
+
+## Directory layouts
+
+In **prefix** layout (the default), a buffer with checksum `ab3f7c...` is stored as `<directory>/ab/ab3f7c...`. A sentinel file `.HASHSERVER_PREFIX` is written to the directory. This avoids performance issues when storing large numbers of buffers.
+
+In **flat** layout, the same buffer is stored as `<directory>/ab3f7c...`.
+
+Extra directories auto-detect their layout by checking for the `.HASHSERVER_PREFIX` sentinel.
+
+## Running tests
+
+```bash
+pip install requests
+pytest tests/
+```
+
+## License
+
+See [LICENSE.txt](LICENSE.txt).

hashserver-1.0/README.md

@@ -0,0 +1,160 @@
+# Hashserver
+
+A lightweight, content-addressed file server over HTTP.
+
+Hashserver stores and serves opaque binary buffers keyed by their cryptographic checksum. You PUT a buffer with its checksum in the URL; you GET it back by the same checksum. There are no filenames, no directories, no metadata — just content and its hash.
+
+The hash algorithm is configurable: SHA-256 (default) or SHA3-256.
+
+## Why content-addressed storage?
+
+Content-addressed storage (CAS) is a well-established pattern used by Git, IPFS, Docker registries, and many other systems. Identifying data by its cryptographic hash gives you automatic deduplication, trivially verifiable integrity, and strong reproducibility guarantees.
+
+Hashserver brings these benefits to any project that needs a simple HTTP-based buffer store. It is intentionally minimal: a single ASGI application backed by a directory of files, designed to be easy to deploy, easy to integrate, and easy to reason about.
+
+## Relationship to Seamless
+
+Hashserver was originally developed as the buffer-serving component of [Seamless](https://github.com/sjdv1982/seamless), a framework for reproducible, reactive computational workflows. In Seamless, all data — inputs, source code, and results — is represented as a tree of checksums, and hashserver provides the storage layer that maps those checksums back to actual data.
+
+However, **hashserver has no dependency on Seamless** and no knowledge of it. It is a generic content-addressed file server that is useful in any context where you need to store and retrieve buffers by hash — caching layers, artifact stores, reproducible pipelines, or your own CAS-backed application. It is published as an independent PyPI package for exactly this reason.
+
+## Features
+
+- **Content-addressed**: buffers are stored and retrieved by their cryptographic checksum.
+- **Configurable hash algorithm**: SHA-256 (default) or SHA3-256, selected at startup.
+- **Integrity-verified reads**: every buffer is re-checksummed on GET to detect corruption.
+- **Prefix directory layout**: by default, buffers are stored under a two-character prefix subdirectory (e.g. `ab/ab3f7c...`) to avoid filesystem performance problems with large flat directories. A flat layout is also supported.
+- **Extra read-only directories**: additional buffer directories can be mounted as fallback read sources.
+- **Promises**: a client can announce that a buffer will be uploaded soon via `PUT /promise/{checksum}`. Other clients reading that checksum will wait for the upload rather than getting a 404.
+- **Concurrent-safe**: in-flight PUT requests are tracked so concurrent GETs and batch queries return consistent results. Lock files are respected for external writers.
+- **Multiple instances**: several hashserver processes can safely share the same buffer directory.
+- **Lightweight**: built on FastAPI/Starlette — no database, no external services.
+- **Flexible deployment**: run as a CLI tool, under any ASGI server, or via Docker Compose.
+
+## Installation
+
+```bash
+pip install hashserver
+```
+
+Or with conda:
+
+```bash
+mamba env create --file environment.yml
+conda activate hashserver
+```
+
+## Quick start
+
+Serve buffers from a local directory:
+
+```bash
+hashserver ./my-buffers
+```
+
+This starts the server under uvicorn on port 8000. Run `hashserver -h` for all options.
+
+### Storing and retrieving a buffer
+
+```bash
+# Start a writable server
+hashserver ./my-buffers --writable
+
+# Compute the SHA-256 checksum and upload
+CHECKSUM=$(python3 -c "
+import hashlib, sys
+print(hashlib.sha256(open(sys.argv[1],'rb').read()).hexdigest())
+" myfile.bin)
+curl -X PUT --data-binary @myfile.bin http://localhost:8000/$CHECKSUM
+
+# Download
+curl -O http://localhost:8000/$CHECKSUM
+```
+
+To use SHA3-256 instead, start the server with `--hash-algorithm sha3-256` and hash your files with `hashlib.sha3_256`.
+
+## API
+
+### Retrieving buffers
+
+**`GET /{checksum}`** — Retrieve a buffer by its hex checksum. The server verifies the checksum before sending the response. Returns the raw buffer (200), or 404 if not found.
+
+### Storing buffers
+
+Requires `--writable`.
+
+**`PUT /{checksum}`** — Upload a buffer. The request body is the raw data; the server verifies that its checksum matches the URL. Returns 200 on success, 201 if the buffer already existed, or 400 on checksum mismatch.
+
+**`PUT /promise/{checksum}`** — Announce that a buffer will be uploaded soon. Returns 202 with the promise TTL. While a promise is active, GET requests for that checksum will wait rather than returning 404, and `/has` queries will report the checksum as present.
+
+### Querying availability
+
+**`GET /has`** — Batch existence check. Send a JSON list of checksums in the request body. Returns a JSON list of booleans. Includes both on-disk buffers and active promises.
+
+**`GET /has-now`** — Same as `/has`, but excludes promises — only reports buffers that are already on disk.
+
+**`GET /buffer-length`** — Batch size query. Send a JSON list of checksums in the request body. Returns a JSON list of integers: the buffer size in bytes, or 0 if not present. Promised checksums are reported as `true`.
+
+### Health
+
+**`GET /healthcheck`** — Returns "OK". Useful for load balancer probes.
+
+## Configuration
+
+### CLI flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `directory` | Buffer storage directory (positional, required) | — |
+| `--writable` | Enable PUT endpoints | off |
+| `--hash-algorithm` | Hash algorithm: `sha3-256` or `sha-256` | `sha-256` |
+| `--layout` | Directory layout: `prefix` or `flat` | `prefix` |
+| `--extra-dirs` | Semicolon-separated list of extra read-only buffer directories | — |
+| `--host` | Listen address | `127.0.0.1` |
+| `--port` | Listen port | `8000` |
+| `--port-range START END` | Pick a random free port in range (mutually exclusive with `--port`) | — |
+| `--status-file` | JSON file for reporting server status | — |
+| `--timeout` | Shut down after this many seconds of inactivity | — |
+
+### Environment variables
+
+When running under an external ASGI server (e.g. `uvicorn hashserver:app`), configure via environment variables instead:
+
+| Variable | Equivalent flag |
+|----------|----------------|
+| `HASHSERVER_DIRECTORY` | `directory` |
+| `HASHSERVER_WRITABLE` | `--writable` (set to `1` or `true`) |
+| `HASHSERVER_HASH_ALGORITHM` | `--hash-algorithm` |
+| `HASHSERVER_LAYOUT` | `--layout` |
+| `HASHSERVER_EXTRA_DIRS` | `--extra-dirs` |
+
+### Docker Compose
+
+```bash
+export HASHSERVER_PORT=8000
+export HASHSERVER_HOST=0.0.0.0
+export HASHSERVER_DIRECTORY=./buffers
+export HASHSERVER_WRITABLE=1
+docker compose up -d
+```
+
+Container user/group ID can be set with `HASHSERVER_USER_ID` and `HASHSERVER_GROUP_ID` (both default to 0).
+
+## Directory layouts
+
+In **prefix** layout (the default), a buffer with checksum `ab3f7c...` is stored as `<directory>/ab/ab3f7c...`. A sentinel file `.HASHSERVER_PREFIX` is written to the directory. This avoids performance issues when storing large numbers of buffers.
+
+In **flat** layout, the same buffer is stored as `<directory>/ab3f7c...`.
+
+Extra directories auto-detect their layout by checking for the `.HASHSERVER_PREFIX` sentinel.
+
+## Running tests
+
+```bash
+pip install requests
+pytest tests/
+```
+
+## License
+
+See [LICENSE.txt](LICENSE.txt).

hashserver-1.0/hash_file_response.py

@@ -0,0 +1,237 @@
+import os
+import stat
+import time
+import typing
+from hashlib import sha3_256, sha256
+
+import anyio
+
+from starlette.background import BackgroundTask
+from starlette.types import Receive, Scope, Send
+from starlette.responses import FileResponse
+
+HASH_ALGORITHMS = {
+    "sha3-256": sha3_256,
+    "sha-256": sha256,
+}
+DEFAULT_HASH_ALGORITHM = "sha-256"
+_current_hash_algorithm = DEFAULT_HASH_ALGORITHM
+_hash_constructor = HASH_ALGORITHMS[DEFAULT_HASH_ALGORITHM]
+
+
+def set_hash_algorithm(algorithm: str) -> None:
+    global _current_hash_algorithm, _hash_constructor
+    try:
+        _hash_constructor = HASH_ALGORITHMS[algorithm]
+    except KeyError as exc:
+        raise ValueError(
+            f"Unsupported hash algorithm '{algorithm}'. "
+            f"Choose one of: {', '.join(HASH_ALGORITHMS)}"
+        ) from exc
+    _current_hash_algorithm = algorithm
+
+
+def get_hash_algorithm() -> str:
+    return _current_hash_algorithm
+
+
+def parse_checksum(checksum) -> str:
+    """Parses checksum and returns it as string.
+
+    Adapted from the Seamless source code (fair use)"""
+    if isinstance(checksum, bytes):
+        checksum = checksum.hex()
+    if isinstance(checksum, str):
+        if len(checksum) % 2:
+            raise ValueError("Wrong length")
+        checksum = bytes.fromhex(checksum)
+
+    if isinstance(checksum, bytes):
+        if len(checksum) != 32:
+            raise ValueError("Wrong length")
+        return checksum.hex()
+
+    if checksum is None:
+        return
+    raise TypeError(type(checksum))
+
+
+class HashFileResponse(FileResponse):
+    """FileResponse that validates files against their checksum-derived filename."""
+
+    _PREFIX = False
+
+    lock_timeout = 120
+    chunk_size = 640 * 1024
+
+    def __init__(
+        self,
+        checksum: str,
+        directory: str,
+        status_code: int = 200,
+        headers: typing.Optional[typing.Mapping[str, str]] = None,
+        media_type: typing.Optional[str] = None,
+        background: typing.Optional[BackgroundTask] = None,
+        stat_result: typing.Optional[os.stat_result] = None,
+        method: typing.Optional[str] = None,
+        content_disposition_type: str = "attachment",
+        extra_dirs: typing.Optional[typing.List[str]] = None,
+    ) -> None:
+        filename = parse_checksum(checksum)
+        self.prefix = filename[:2]
+        stat_result = None
+        if self._PREFIX:
+            path = os.path.join(directory, self.prefix, filename)
+        else:
+            path = os.path.join(directory, filename)
+        super().__init__(
+            path=path,
+            status_code=status_code,
+            headers=headers,
+            media_type=media_type,
+            background=background,
+            filename=filename,
+            stat_result=stat_result,
+            method=method,
+            content_disposition_type=content_disposition_type,
+        )
+        self.directory = directory
+        self.extra_dirs = extra_dirs
+        extra_dirs_layout = {}
+        for extra_dir in extra_dirs:
+            prefix_file = os.path.join(extra_dir, ".HASHSERVER_PREFIX")
+            if os.path.exists(prefix_file):
+                layout = "prefix"
+            else:
+                layout = "flat"
+            extra_dirs_layout[extra_dir] = layout
+        self.extra_dirs_layout = extra_dirs_layout
+
+    async def refresh_stat_headers(self):
+        if self.extra_dirs and not await anyio.Path(self.path).exists():
+            for extra_dir in self.extra_dirs:
+                layout = self.extra_dirs_layout[extra_dir]
+                if layout == "prefix":
+                    path0 = os.path.join(extra_dir, self.prefix, self.filename)
+                else:
+                    path0 = os.path.join(extra_dir, self.filename)
+                if await anyio.Path(path0).exists():
+                    self.path = path0
+                    break
+
+        try:
+            stat_result = await anyio.to_thread.run_sync(os.stat, self.path)
+            del self.headers["content-length"]
+            del self.headers["last-modified"]
+            del self.headers["etag"]
+
+            self.set_stat_headers(stat_result)
+        except FileNotFoundError:
+            raise FileNotFoundError(
+                f"File at path {self.path} does not exist."
+            ) from None
+        else:
+            mode = stat_result.st_mode
+            if not stat.S_ISREG(mode):
+                raise RuntimeError(f"File at path {self.path} is not a file.")
+        return stat_result
+
+    async def _until_no_lock(self, lockpaths):
+        for lockpath in lockpaths:
+            while 1:
+                try:
+                    lock_stat_result = await anyio.to_thread.run_sync(os.stat, lockpath)
+                except FileNotFoundError:
+                    break
+                lock_mtime = lock_stat_result.st_mtime
+                if time.time() - lock_mtime > self.lock_timeout:
+                    break
+                await anyio.sleep(1)
+
+    async def until_no_lock(self):
+        lockpaths = [os.path.join(self.directory, ".LOCK")]
+        if self.path is not None:
+            lockpaths.append(self.path + ".LOCK")
+        return await self._until_no_lock(lockpaths)
+
+    async def calculate_checksum(self):
+        """Return checksum for the configured algorithm."""
+        checksum = _hash_constructor()
+        async with await anyio.open_file(self.path, mode="rb") as file:
+            more_body = True
+            while more_body:
+                chunk = await file.read(self.chunk_size)
+                checksum.update(chunk)
+                more_body = len(chunk) == self.chunk_size
+
+            checksum = checksum.digest().hex()
+        return checksum
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if self.stat_result is None:
+            try:
+                stat_result = await self.refresh_stat_headers()
+            except FileNotFoundError:
+                await self.until_no_lock()
+                stat_result = await self.refresh_stat_headers()
+            self.stat_result = stat_result
+
+        checksum = await self.calculate_checksum()
+        if checksum != self.filename:
+            await self.until_no_lock()
+            stat_result = await self.refresh_stat_headers()
+            self.stat_result = stat_result
+            checksum2 = await self.calculate_checksum()
+            if checksum2 != self.filename:
+                raise RuntimeError(
+                    f"File corruption: file at path {self.path} does not have the correct {_current_hash_algorithm} checksum."
+                )
+
+        await super().__call__(scope=scope, receive=receive, send=send)
+
+
+class PrefixHashFileResponse(HashFileResponse):
+    """Same as HashFileResponse but files are stored under a two-character prefix.
+
+    File has the same name as checksum.
+    File is stored as $PREFIX/$CHECKSUM, where $PREFIX is the first two
+      characters of $CHECKSUM
+    """
+
+    _PREFIX = True
+
+    def __init__(
+        self,
+        checksum: str,
+        directory: str,
+        status_code: int = 200,
+        headers: typing.Optional[typing.Mapping[str, str]] = None,
+        media_type: typing.Optional[str] = None,
+        background: typing.Optional[BackgroundTask] = None,
+        stat_result: typing.Optional[os.stat_result] = None,
+        method: typing.Optional[str] = None,
+        content_disposition_type: str = "attachment",
+        extra_dirs: typing.Optional[typing.List[str]] = None,
+    ) -> None:
+
+        super().__init__(
+            checksum=checksum,
+            directory=directory,
+            status_code=status_code,
+            headers=headers,
+            media_type=media_type,
+            background=background,
+            stat_result=stat_result,
+            method=method,
+            content_disposition_type=content_disposition_type,
+            extra_dirs=extra_dirs,
+        )
+        prefix_file = os.path.join(directory, ".HASHSERVER_PREFIX")
+        with open(prefix_file, mode="wb") as f:
+            f.write(b"1\n")
+
+    async def until_no_lock(self):
+        lockpaths = [os.path.join(self.directory, self.prefix, ".LOCK")]
+        if self.path is not None:
+            lockpaths.append(self.path + ".LOCK")
+        return await self._until_no_lock(lockpaths)