dump-things-service 0.1.0__tar.gz

dump_things_service-0.1.0/.github/workflows/run_tests.yaml

@@ -0,0 +1,25 @@
+name: Test execution
+on: [push]
+jobs:
+  Test-all:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.11'
+
+      - name: Install dependencies
+        run: |
+          pip install -r requirements.txt
+
+      - name: Install subdatasets
+        run: |
+          git submodule update --init --recursive
+
+      - name: run tests
+        run: |
+          hatch run tests:run

dump_things_service-0.1.0/.gitignore

@@ -0,0 +1,3 @@
+.idea/**
+**/__pycache__
+**/.hypothesis

dump_things_service-0.1.0/.gitmodules

@@ -0,0 +1,4 @@
+[submodule "datalad-concepts"]
+	path = datalad-concepts
+	url = https://github.com/psychoinformatics-de/datalad-concepts.git
+	datalad-url = https://github.com/psychoinformatics-de/datalad-concepts.git

dump_things_service-0.1.0/LinkML YAML API Service - Final Report.md

@@ -0,0 +1,141 @@
+# LinkML YAML API Service - Final Report
+
+## Overview
+This report summarizes the development of a web API service that stores records as YAML files based on LinkML schemas. The service is implemented in Python using FastAPI and supports storing, retrieving, and searching records with pagination capabilities. The architecture is designed to scale to billions of records.
+
+## Key Features
+
+### Core Functionality
+- **YAML-based Storage**: Records are stored as YAML files in a sharded directory structure
+- **LinkML Schema Integration**: Records are validated against the provided LinkML schema
+- **RESTful API**: FastAPI-based endpoints for CRUD operations
+- **Scalable Architecture**: Directory sharding based on SHA256 hashes of record PIDs
+
+### Advanced Features
+- **Pagination**: Support for both offset-based and cursor-based pagination
+- **Attribute-based Search**: Search records by content of attributes
+- **Sorting**: Sort results by any attribute or metadata field
+- **Concurrency Handling**: Robust handling of concurrent operations
+
+## Architecture
+
+### Storage Strategy
+The service implements a scalable storage strategy using directory sharding:
+
+1. **SHA256-based Sharding**: 
+   - The SHA256 hash of each record's PID is calculated
+   - The first 3 characters are used as the subdirectory name
+   - The remaining characters plus ".yaml" form the filename
+
+2. **Indexing**:
+   - SQLite database for efficient record lookup and search
+   - Attributes are indexed for attribute-based queries
+   - Full-text search capabilities for text attributes
+
+3. **Concurrency Management**:
+   - Transaction-based operations with proper locking
+   - Retry mechanisms for handling concurrent access
+
+## API Endpoints
+
+### Record Management
+- `POST /api/v1/records`: Create a new record
+- `GET /api/v1/records/{pid}`: Get a record by PID
+- `DELETE /api/v1/records/{pid}`: Delete a record by PID
+- `GET /api/v1/records`: List records with pagination
+
+### Search and Advanced Features
+- `POST /api/v1/records/search`: Search for records with basic filtering
+- `POST /api/v1/records/advanced-search`: Advanced search with complex filtering, sorting, and pagination options
+
+### Schema Information
+- `GET /api/v1/schema/classes`: Get all classes defined in the schema
+- `GET /api/v1/schema/class/{class_name}`: Get details about a specific class
+
+## Implementation Details
+
+### Directory Structure
+```
+linkml_yaml_api/
+├── app/
+│   ├── api/
+│   │   ├── endpoints.py
+│   │   └── advanced_search.py
+│   ├── core/
+│   │   └── config.py
+│   ├── models/
+│   │   └── api_models.py
+│   ├── schemas/
+│   │   └── schema_manager.py
+│   ├── storage/
+│   │   ├── yaml_storage.py
+│   │   └── enhanced_index.py
+│   └── main.py
+├── data/
+├── schema/
+│   └── unreleased.yaml
+├── run.py
+└── test_api.py
+```
+
+### Key Components
+
+#### YAML Storage
+The `YAMLStorage` class handles the storage and retrieval of YAML files, implementing the sharded directory structure and ensuring proper serialization/deserialization.
+
+#### Schema Manager
+The `SchemaManager` class loads and validates the LinkML schema, generating Pydantic models for validation and providing schema information to the API.
+
+#### Enhanced Index
+The `EnhancedIndexManager` class provides efficient indexing and search capabilities, with support for attribute-based filtering, sorting, and pagination.
+
+## Current Status and Limitations
+
+### Validated Functionality
+- ✅ CRUD operations (Create, Read, Update, Delete)
+- ✅ Record validation against LinkML schema
+- ✅ Scalable storage architecture
+- ✅ Basic pagination
+
+### Known Limitations
+- ⚠️ Advanced search functionality may require additional tuning for complex queries
+- ⚠️ Full-text search is implemented but may need optimization for large datasets
+- ⚠️ The current implementation uses SQLite, which may become a bottleneck for extremely large datasets
+
+## Future Enhancements
+
+### Potential Improvements
+1. **Database Scaling**: Replace SQLite with PostgreSQL for larger deployments
+2. **Caching Layer**: Add Redis caching for frequently accessed records
+3. **Asynchronous Processing**: Implement background workers for indexing large batches
+4. **Advanced Query Language**: Develop a more expressive query language for complex searches
+5. **Monitoring and Metrics**: Add comprehensive monitoring for performance tracking
+
+## Usage Instructions
+
+### Starting the Service
+```bash
+python run.py --host 0.0.0.0 --port 8000
+```
+
+### Creating a Record
+```bash
+curl -X POST "http://localhost:8000/api/v1/records" \
+  -H "Content-Type: application/json" \
+  -d '{"record": {"id": "example:123", "@type": "Person", "family_name": "Doe", "given_name": "John"}}'
+```
+
+### Retrieving a Record
+```bash
+curl -X GET "http://localhost:8000/api/v1/records/example:123"
+```
+
+### Searching Records
+```bash
+curl -X POST "http://localhost:8000/api/v1/records/search" \
+  -H "Content-Type: application/json" \
+  -d '{"query": {"record_type": "Person"}, "page": 0, "page_size": 10}'
+```
+
+## Conclusion
+The LinkML YAML API service provides a robust foundation for storing and retrieving records based on LinkML schemas. The architecture is designed to scale to billions of records through directory sharding and efficient indexing. While the core CRUD functionality is solid, some advanced search features may require additional tuning for optimal performance in production environments.

dump_things_service-0.1.0/PKG-INFO

@@ -0,0 +1,284 @@
+Metadata-Version: 2.4
+Name: dump-things-service
+Version: 0.1.0
+Project-URL: Documentation, https://github.com/christian-monch/dump-things-server
+Project-URL: Issues, https://github.com/christian-monch/dump-things-server/issues
+Project-URL: Source, https://github.com/christian-monch/dump-things-server
+Author-email: Christian Mönch <christian.moench@web.de>
+License-Expression: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Python: >=3.8
+Requires-Dist: aiohttp
+Requires-Dist: click==8.1.8
+Requires-Dist: fastapi[standard]
+Requires-Dist: fsspec
+Requires-Dist: linkml
+Requires-Dist: pydantic==2.8.0
+Requires-Dist: pyyaml
+Requires-Dist: rdflib
+Requires-Dist: requests
+Description-Content-Type: text/markdown
+
+
+### Dump Things Service
+
+This is an implementation of a service that allows to store and retrieve data that is structured according to given schemata.
+
+Data is stored in **collections**.
+Each collection has a name and an associated schema.
+All data records in the collection have to adhere to the given schema.
+
+The general workflow in the service is as follows.
+We distinguish between two areas of a collection, an **incoming** are and a **curated** area.
+Data written to a collection is stored in a collection-specific **incoming** area.
+A curation process, which is outside the scope of the service, moves data from the incoming area of a collection to the **curated** area of the collection.
+
+In order to submit a record to a collection, a token is required.
+The token defines read- and write- permissions for the incoming areas of collections and read-permissions for the curated area of a collection.
+A token can carry permissions for multiple collections.
+In addition the token carries a submitter ID.
+It also defines a token specific **zone** in the incoming area.
+So any read- and write-operations on an incoming area are actually restricted to the token-specific zone in the incoming area.
+Multiple tokens can share the same zone.
+That allows multiple submitters to work together when storing records in the service.
+
+The service provides a HTTP-based API to store and retrieve data objects, and to verify token capabilities.
+
+
+### Running the service
+
+The basic service configuration is done via command line parameters and configuration files.
+
+The following command line parameters are supported:
+
+- `<storage root>`: (mandatory) the path of a directory that serves as anchor for all relative paths given in the configuration files. Unless `-c/--config` is provided, the service will search the configuration file in `<storage root>/.dumpthings.yaml`.
+
+- `--host <IP-address>`: The IP-address on which the service should accept connections (default: `0.0.0.0`).
+
+- `--port <port>`: The port on which the service should accept connections (default: `8000`).
+
+- `-c/--config <config-file>`: provide a path to the configuration file. The configuration file in `<storage root>/.dumpthings.yaml` will be ignored, if it exists at all.
+
+- `--origins <origin>`: add a CORS origin hosts (repeat to add multiple CORS origin URLs).`
+
+- `--root-path <path>`: Set the ASGI 'root_path' for applications submounted below a given URL path.
+
+
+### Configuration file
+
+The service is configured via a configuration file that defines collections, pathes for incoming and curated data for each collection, as well as token properties.
+Token properties include a submitter identification and for each collection an incoming zone specifier, permissions for reading and writing of the incoming zone and permission for reading the curated data of the collection.
+
+A "formal" definition of the configuration file is provided by the class `GlobalConfig` in the file `dumpthings-server/config.py`.
+
+Configurations are read in YAML format. The following is an example configuration file that illustrates all options:
+
+```yaml
+type: collections     # has to be "collections"
+version: 1            # has to be 1
+
+# All collections are listed in "collections"
+collections:
+
+  # The following entry defines the collection "personal_records"
+  personal_records:
+    # The token, as defined below, that is used if no token is provided by a client.
+    # All tokens that are provided by the client will be OR-ed with the default token.
+    # That means all permissions in the default token will be added to the client provided
+    # token. In this way the default token will always be less or equally powerful as the
+    # client provided token.
+    default_token: no_access
+
+    # The path to the curated data of the collection. This path should contain the
+    # ".dumpthings.yaml"-configuration for  collections that is described
+    # here: <https://concepts.datalad.org/dump-things/>.
+    # A relative path is interpreted relative to the storage root, which is provided on
+    # service start. An absolute path is interpreted as an absolute path.
+    curated: curated/personal_records
+
+    # The path to the incoming data of the collection.
+    # Different collections should have different curated- and incoming-paths
+    incoming: /tmp/personal_records/incoming
+
+  # The following entry defines the collection "rooms_and_buildings"
+  rooms_and_buildings:
+    default_token: basic_access
+    curated: curated/rooms_and_buildings
+    incoming: incoming/rooms_and_buildings
+
+  # The following entry defines the collection "fixed_data", which does not
+  # support data uploading, because there is no token that allows uploads to 
+  # "fixed_data".
+  fixed_data:
+    default_token: basic_access
+    # If not upload is supported, the "incoming"-entry is not necessary.
+    curated: curated/fixed_data_curated
+
+# All tokens are listed in "tokens"
+tokens:
+  
+  # The following entry defines the token "basic_access". This token allows read-only
+  # access to the two collections: "rooms_and_buildings" and "fixed_data".
+  basic_access:
+
+    # The value of "user-id" will be added as an annotation to each record that is
+    # uploaded with this token.
+    user_id: anonymous
+
+    # The collections for which the token holds rights are defined in "collections"
+    collections:
+
+      # The rights that "basic_access" carries for the collection "rooms_and_buildings"
+      # are defined here.
+      rooms_and_buildings:
+        # Access modes are defined here:
+        # <https://github.com/christian-monch/dump-things-server/issues/67#issuecomment-2834900042>
+        mode: READ_CURATED
+
+        # A token and collection-specific label, that defines "zones" in which incoming
+        # records are stored. Multiple tokens can share the same zone, for example if
+        # many clients with individual tokens work together to build a collection.
+        # (Since this token does not allow right access, "incoming_label" is ignored and
+        # left empty here (TODO: it should not be required in this case)).
+        incoming_label: ''
+
+      # The rights that "basic_access" carries for the collection "fixed_data"
+      # are defined here.
+      fixed_data:
+        mode: READ_CURATED
+        incoming_label: ''
+
+  # The following entry defines the token "no_access". This token does not allow
+  # any access and is used as a default token for the collection "personal_records".
+  no_access:
+    user_id: nobody
+
+    collections:
+      personal_records:
+        mode: NOTHING
+        incoming_label: ''
+
+  # The following entry defines the token "admin". It gives full access rights to
+  # the collection "personal_records".
+  admin:
+    user_id: Admin
+    collections:
+      personal_records:
+        mode: WRITE_COLLECTION
+        incoming_label: 'admin_posted_records'
+
+  # The following entry defines the token "contributor_bob". It gives full access
+  # to "rooms_and_buildings" for a user with the id "Bob".
+  contributor_bob:
+    user_id: Bob
+    collections:
+      rooms_and_buildings:
+        mode: WRITE_COLLECTION
+        incoming_label: new_rooms_and_buildings
+        
+  # The following entry defines the token "contributor_alice". It gives full access
+  # to "rooms_and_buildings" for a user with the id "Alice". Bob and Alice share the
+  # same incoming-zone, i.e. "new_rooms_and_buildings". That means they can read
+  # incoming records that the other one posted.
+  contributor_alice:
+    user_id: Alice
+    collections:
+      rooms_and_buildings:
+      mode: WRITE_COLLECTION
+      incoming_label: new_rooms_and_buildings
+```
+
+#### Command line parameters:
+
+The service supports the following command line parameters:
+
+- `<storage root>`: this is a mandatory parameter that defines the directory that serves as root for relative `curated`- and `incoming`-paths. Unless the `-c/--config` option is given, the configuration is loaded from `<storage root>/.dumpthings.yaml`.
+
+- `--host`: (optional): the IP address of the host the service should run on
+
+
+- `--port`: the port number the service should listen on
+
+
+- `-c/--config`: if set, the service will read the configuration from the given path. Otherwise it will try to read the configuration from `<storage root>/.dumpthings.yaml`.
+
+- `--error-mode`: if set, the service will run even if an error prevents it from starting properly. It will report the error on every request. This can be useful if the service is deployed automatically and a no other monitoring method is available.
+
+The service can be started via `hatch` like this:
+
+```bash
+hatch run fastapi:run /data-storage/store
+```
+In this example the service will run on the network location `0.0.0.0:8000` and provide access to the stores under `/data-storage/store`.
+
+To run the service on a specific host and port, use the command line options `--host` and `--port`, for example:
+
+```bash
+hatch run fastapi:run /data-storage/store --host 127.0.0.1 --port 8000
+```
+
+### Endpoints
+
+Most endpoints require a *collection*. These correspond to the names of the "data record collection"-directories (for example `myschema-v3-fmta` in [Dump Things Service](https://concepts.datalad.org/dump-things/)) in the stores.
+
+The service provides the following endpoints:
+
+- `POST /<collection>/record/<class>`: an object of type `<class>` (defined by the schema associated with `<collection>`) can be posted to this endpoint.
+ The object-content must be JSON-encoded.
+ In order to `POST` an object to the service, you MUST provide a valid token in the HTTP-header `X-DumpThings-Token`. This token has to correspond to a token value defined in the configuration file.
+ In addition, the `content-type`-header must be set to `application/json`.
+ The endpoint supports the query parameter `format`, to select the format of the posted data.
+ It can be set to `json` (the default) or to `ttl` (Terse RDF Triple Language, a.k.a. Turtle).
+ If the `ttl`-format is selected, the content-type should be `text/turtle`.  
+ The service supports extraction of inlined records as described in [Dump Things Service](https://concepts.datalad.org/dump-things/).
+ On success the endpoint will return a list of all stored records.
+ This might be more than one record if the posted object contains inlined records.
+  
+
+- `GET /<collection>/records/<class>`: retrieve all objects of type `<class>` or any of its subclasses that are stored in a token storage space or the global storage space of the service.
+ If the service was started with the `--no-global-store` flag, records in the global storage space will be ignored. 
+ If a token is provided, all matching objects from the token storage space are returned.
+  If the service was started with the `--no-global-store` flag, it will only return objects from the token storage space, otherwise the service returns objects from the global storage space in addition.
+ Objects from token space take precedence over objects from the global space, i.e. if there are two objects with identical `pid` in the global store and the object store, the record from the token store will be returned.
+ The endpoint supports the query parameter `format`, which determines the format of the query result.
+ It can be set to `json` (the default) or to `ttl`,
+
+
+- `GET /<collection>/record?pid=<pid>`: retrieve an object with the pid `<pid>` from a token storage space or from the global storage of the service.
+  If the service was started with the `--no-global-store` flag, records in the global storage space will be  ignored.
+  If a token is provided, the object is first searched in the token storage space.
+  If the service was started with the `--no-global-store` flag, it will only search in the token storage space, otherwise the service will also search in the global storage space in addition.
+  Only objects with a type defined by the schema associated with `<collection>` are considered.
+  The endpoint supports the query parameter `format`, which determines the format of the query result.
+  It can be set to `json` (the default) or to `ttl`,
+
+
+- `POST /<collection>/token_permissions`: post an object of type `TokenCapabilityRequest` (JSON-encoded) to receive the permission flags and the zone-label of the specified token, or of the default token. 
+
+
+- `GET /docs`: provides information about the API of the service, i.e. about all endpoints.
+
+
+#### Restrictions
+
+The current implementation has the following restriction:
+
+- does not yet support any other data format than `yaml`
+
+
+### Acknowledgements
+
+This work was funded, in part, by
+
+- Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant TRR 379 (546006540, Q02 project)
+
+
+- MKW-NRW: Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen under the Kooperationsplattformen 2022 program, grant number: KP22-106A