linkml-store 0.1.9__tar.gz → 0.1.11__tar.gz

linkml_store-0.1.11/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.1
+Name: linkml-store
+Version: 0.1.11
+Summary: linkml-store
+License: MIT
+Author: Author 1
+Author-email: author@org.org
+Requires-Python: >=3.9, !=2.7.*, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.*, !=3.8.*
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Provides-Extra: analytics
+Provides-Extra: app
+Provides-Extra: chromadb
+Provides-Extra: fastapi
+Provides-Extra: frictionless
+Provides-Extra: h5py
+Provides-Extra: llm
+Provides-Extra: map
+Provides-Extra: mongodb
+Provides-Extra: pyarrow
+Provides-Extra: renderer
+Provides-Extra: tests
+Provides-Extra: validation
+Requires-Dist: black (>=24.0.0) ; extra == "tests"
+Requires-Dist: chromadb ; extra == "chromadb"
+Requires-Dist: click
+Requires-Dist: duckdb (>=0.10.1,<0.11.0)
+Requires-Dist: duckdb-engine (>=0.11.2)
+Requires-Dist: fastapi ; extra == "fastapi"
+Requires-Dist: frictionless ; extra == "frictionless"
+Requires-Dist: h5py ; extra == "h5py"
+Requires-Dist: jinja2 (>=3.1.4,<4.0.0)
+Requires-Dist: jsonlines (>=4.0.0,<5.0.0)
+Requires-Dist: linkml (>=1.8.0) ; extra == "validation"
+Requires-Dist: linkml-runtime (>=1.8.0)
+Requires-Dist: linkml_map ; extra == "map"
+Requires-Dist: linkml_renderer ; extra == "renderer"
+Requires-Dist: llm ; extra == "llm"
+Requires-Dist: matplotlib ; extra == "analytics"
+Requires-Dist: pandas (>=2.2.1) ; extra == "analytics"
+Requires-Dist: plotly ; extra == "analytics"
+Requires-Dist: pyarrow ; extra == "pyarrow"
+Requires-Dist: pydantic (>=2.0.0,<3.0.0)
+Requires-Dist: pymongo ; extra == "mongodb"
+Requires-Dist: pystow (>=0.5.4,<0.6.0)
+Requires-Dist: seaborn ; extra == "analytics"
+Requires-Dist: sqlalchemy
+Requires-Dist: streamlit (>=1.32.2,<2.0.0) ; extra == "app"
+Requires-Dist: uvicorn ; extra == "fastapi"
+Description-Content-Type: text/markdown
+
+# linkml-store
+
+An AI-ready data management and integration platform. LinkML-Store
+provides an abstraction layer over multiple different backends
+(including DuckDB, MongoDB, and local filesystems), allowing for
+common query, index, and storage operations.
+
+For full documentation, see [https://linkml.io/linkml-store/](https://linkml.io/linkml-store/)
+
+__Warning__ LinkML-Store is still undergoing changes and refactoring,
+APIs and command line options are subject to change!
+
+## Quick Start
+
+Install, add data, query it:
+
+```
+pip install linkml-store[all]
+linkml-store -d duckdb:///db/my.db -c persons insert data/*.json
+linkml-store -d duckdb:///db/my.db -c persons query -w "occupation: Bricklayer"
+```
+
+Index it, search it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons index -t llm
+linkml-store -d duckdb:///db/my.db -c persons search "all persons employed in construction"
+```
+
+Validate it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons validate
+```
+
+## Basic usage
+
+* [Command Line](https://linkml.io/linkml-store/tutorials/Command-Line-Tutorial.html)
+* [Python](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* API
+* Streamlit applications
+
+## Features
+
+### Multiple Adapters
+
+LinkML-Store is designed to work with multiple backends, giving a common abstraction layer
+
+* [MongoDB](https://linkml.io/linkml-store/how-to/Use-MongoDB.html)
+* [DuckDB](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* [Solr](https://linkml.io/linkml-store/how-to/Query-Solr-using-CLI.html)
+* Filesystem
+
+Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...
+
+The intent is to give a union of all features of each backend. For
+example, analytic faceted queries are provided for *all* backends, not
+just Solr.
+
+### Composable indexes
+
+Many backends come with their own indexing and search
+schemes. Classically this was Lucene-based indexes, now it is semantic
+search using LLM embeddings.
+
+LinkML store treats indexing as an orthogonal concern - you can
+compose different indexing schemes with different backends. You don't
+need to have a vector database to run embedding search!
+
+See [How to Use-Semantic-Search](https://linkml.io/linkml-store/how-to/Use-Semantic-Search.html)
+
+### Use with LLMs
+
+TODO - docs
+
+### Validation
+
+LinkML-Store is backed by [LinkML](https://linkml.io), which allows
+for powerful expressive structural and semantic constraints.
+
+See [Indexing JSON](https://linkml.io/linkml-store/how-to/Index-Phenopackets.html)
+
+and [Referential Integrity](https://linkml.io/linkml-store/how-to/Check-Referential-Integrity.html)
+
+## Web API
+
+There is a preliminary API following HATEOAS principles implemented using FastAPI.
+
+To start you should first create a config file, e.g. `db/conf.yaml`:
+
+Then run:
+
+```
+export LINKML_STORE_CONFIG=./db/conf.yaml
+make api
+```
+
+The API returns links as well as data objects, it's recommended to use a Chrome plugin for JSON viewing
+for exploring the API. TODO: add docs here.
+
+The main endpoints are:
+
+* `http://localhost:8000/` - the root of the API
+* `http://localhost:8000/pages/` - browse the API via HTML
+* `http://localhost:8000/docs` - the Swagger UI
+
+## Streamlit app
+
+```
+make app
+```
+
+## Background
+
+See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for more details
+

linkml_store-0.1.11/README.md

@@ -0,0 +1,115 @@
+# linkml-store
+
+An AI-ready data management and integration platform. LinkML-Store
+provides an abstraction layer over multiple different backends
+(including DuckDB, MongoDB, and local filesystems), allowing for
+common query, index, and storage operations.
+
+For full documentation, see [https://linkml.io/linkml-store/](https://linkml.io/linkml-store/)
+
+__Warning__ LinkML-Store is still undergoing changes and refactoring,
+APIs and command line options are subject to change!
+
+## Quick Start
+
+Install, add data, query it:
+
+```
+pip install linkml-store[all]
+linkml-store -d duckdb:///db/my.db -c persons insert data/*.json
+linkml-store -d duckdb:///db/my.db -c persons query -w "occupation: Bricklayer"
+```
+
+Index it, search it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons index -t llm
+linkml-store -d duckdb:///db/my.db -c persons search "all persons employed in construction"
+```
+
+Validate it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons validate
+```
+
+## Basic usage
+
+* [Command Line](https://linkml.io/linkml-store/tutorials/Command-Line-Tutorial.html)
+* [Python](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* API
+* Streamlit applications
+
+## Features
+
+### Multiple Adapters
+
+LinkML-Store is designed to work with multiple backends, giving a common abstraction layer
+
+* [MongoDB](https://linkml.io/linkml-store/how-to/Use-MongoDB.html)
+* [DuckDB](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* [Solr](https://linkml.io/linkml-store/how-to/Query-Solr-using-CLI.html)
+* Filesystem
+
+Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...
+
+The intent is to give a union of all features of each backend. For
+example, analytic faceted queries are provided for *all* backends, not
+just Solr.
+
+### Composable indexes
+
+Many backends come with their own indexing and search
+schemes. Classically this was Lucene-based indexes, now it is semantic
+search using LLM embeddings.
+
+LinkML store treats indexing as an orthogonal concern - you can
+compose different indexing schemes with different backends. You don't
+need to have a vector database to run embedding search!
+
+See [How to Use-Semantic-Search](https://linkml.io/linkml-store/how-to/Use-Semantic-Search.html)
+
+### Use with LLMs
+
+TODO - docs
+
+### Validation
+
+LinkML-Store is backed by [LinkML](https://linkml.io), which allows
+for powerful expressive structural and semantic constraints.
+
+See [Indexing JSON](https://linkml.io/linkml-store/how-to/Index-Phenopackets.html)
+
+and [Referential Integrity](https://linkml.io/linkml-store/how-to/Check-Referential-Integrity.html)
+
+## Web API
+
+There is a preliminary API following HATEOAS principles implemented using FastAPI.
+
+To start you should first create a config file, e.g. `db/conf.yaml`:
+
+Then run:
+
+```
+export LINKML_STORE_CONFIG=./db/conf.yaml
+make api
+```
+
+The API returns links as well as data objects, it's recommended to use a Chrome plugin for JSON viewing
+for exploring the API. TODO: add docs here.
+
+The main endpoints are:
+
+* `http://localhost:8000/` - the root of the API
+* `http://localhost:8000/pages/` - browse the API via HTML
+* `http://localhost:8000/docs` - the Swagger UI
+
+## Streamlit app
+
+```
+make app
+```
+
+## Background
+
+See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for more details

{linkml_store-0.1.9 → linkml_store-0.1.11}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "linkml-store"
-version = "0.1.9"
+version = "0.1.11"
 description = "linkml-store"
 authors = ["Author 1 <author@org.org>"]
 license = "MIT"
@@ -10,7 +10,7 @@ readme = "README.md"
 python = "^3.9, !=3.9.7"
 click = "*"
 pydantic = "^2.0.0"
-linkml-runtime = "^1.7.5"
+linkml-runtime = ">=1.8.0"
 streamlit = { version = "^1.32.2", optional = true }
 sqlalchemy = "*"
 duckdb = "^0.10.1"
@@ -25,11 +25,15 @@ pymongo = { version="*", optional = true }
 chromadb = { version="*", optional = true }
 pyarrow = { version="*", optional = true }
 h5py = { version="*", optional = true }
-linkml = { version="*", optional = true }
+linkml = { version=">=1.8.0", optional = true }
 linkml_map = { version="*", optional = true }
+linkml_renderer = { version="*", optional = true }
+frictionless = { version="*", optional = true }
 pandas = ">=2.2.1"
 jinja2 = "^3.1.4"
 jsonlines = "^4.0.0"
+fastapi = { version="*", optional = true }
+uvicorn = { version="*", optional = true }
 
 [tool.poetry.group.dev.dependencies]
 pytest = {version = ">=7.1.2"}
@@ -46,6 +50,7 @@ nbsphinx = "*"
 jupyter = "*"
 jupysql = "*"
 papermill = "*"
+nbdime = "*"
 
 [tool.poetry.group.tests.dependencies]
 pytest = "^7.4.0"
@@ -66,9 +71,13 @@ h5py = ["h5py"]
 pyarrow = ["pyarrow"]
 validation = ["linkml"]
 map = ["linkml_map"]
+renderer = ["linkml_renderer"]
+fastapi = ["fastapi", "uvicorn"]
+frictionless = ["frictionless"]
 
 [tool.poetry.scripts]
 linkml-store = "linkml_store.cli:cli"
+linkml-store-api = "linkml_store.webapi.main:start"
 
 [tool.poetry-dynamic-versioning]
 enable = false

{linkml_store-0.1.9 → linkml_store-0.1.11}/src/linkml_store/api/client.py

@@ -98,7 +98,7 @@ class Client:
         """
         return self.metadata.base_dir
 
-    def from_config(self, config: Union[ClientConfig, str, Path], base_dir=None, **kwargs):
+    def from_config(self, config: Union[ClientConfig, dict, str, Path], base_dir=None, **kwargs):
         """
         Create a client from a configuration.
 
@@ -118,11 +118,13 @@ class Client:
         :return:
 
         """
+        if isinstance(config, dict):
+            config = ClientConfig(**config)
         if isinstance(config, Path):
             config = str(config)
         if isinstance(config, str):
-            if not base_dir:
-                base_dir = Path(config).parent
+            # if not base_dir:
+            #    base_dir = Path(config).parent
             parsed_obj = yaml.safe_load(open(config))
             config = ClientConfig(**parsed_obj)
         self.metadata = config
@@ -133,8 +135,15 @@ class Client:
 
     def _initialize_databases(self, **kwargs):
         for name, db_config in self.metadata.databases.items():
-            handle = db_config.handle.format(base_dir=self.base_dir)
+            base_dir = self.base_dir
+            logger.info(f"Initializing database: {name}, base_dir: {base_dir}")
+            if not base_dir:
+                base_dir = Path.cwd()
+                logger.info(f"Using current working directory: {base_dir}")
+            handle = db_config.handle.format(base_dir=base_dir)
             db_config.handle = handle
+            if db_config.schema_location:
+                db_config.schema_location = db_config.schema_location.format(base_dir=base_dir)
             db = self.attach_database(handle, alias=name, **kwargs)
             db.from_config(db_config)
 
@@ -233,7 +242,7 @@ class Client:
         Return all attached databases
 
         Examples
-        --------
+
         >>> client = Client()
         >>> _ = client.attach_database("duckdb", alias="test1")
         >>> _ = client.attach_database("duckdb", alias="test2")
@@ -259,25 +268,81 @@ class Client:
         """
         Drop a database.
 
+        Example (in-memory):
+
+        >>> client = Client()
+        >>> db1 = client.attach_database("duckdb", alias="test1")
+        >>> db2 = client.attach_database("duckdb", alias="test2")
+        >>> len(client.databases)
+        2
+        >>> client.drop_database("test1")
+        >>> len(client.databases)
+        1
+
+        Databases that persist on disk:
+
+        >>> client = Client()
+        >>> path = Path("tmp/test.db")
+        >>> path.parent.mkdir(parents=True, exist_ok=True)
+        >>> db = client.attach_database(f"duckdb:///{path}", alias="test")
+        >>> len(client.databases)
+        1
+        >>> db.store({"persons": [{"id": "P1", "name": "John"}]})
+        >>> db.commit()
+        >>> Path("tmp/test.db").exists()
+        True
+        >>> client.drop_database("test")
+        >>> len(client.databases)
+        0
+        >>> Path("tmp/test.db").exists()
+        False
+
+        Dropping a non-existent database:
+
+        >>> client = Client()
+        >>> client.drop_database("duckdb:///tmp/made-up1", missing_ok=True)
+        >>> client.drop_database("duckdb:///tmp/made-up2", missing_ok=False)
+        Traceback (most recent call last):
+        ...
+        ValueError: Database duckdb:///tmp/made-up2 not found
+
         :param name:
         :param missing_ok:
         :return:
         """
-        if name in self._databases:
-            db = self._databases[name]
-            db.drop(**kwargs)
-            del self._databases[name]
+        if self._databases:
+            if name in self._databases:
+                db = self._databases[name]
+                db.drop(**kwargs)
+                del self._databases[name]
+            else:
+                if not missing_ok:
+                    raise ValueError(f"Database {name} not found")
         else:
-            if not missing_ok:
-                raise ValueError(f"Database {name} not found")
+            db = self.get_database(name, create_if_not_exists=True)
+            db.drop(**kwargs)
 
     def drop_all_databases(self, **kwargs):
         """
         Drop all databases.
 
+        Example (in-memory):
+
+        >>> client = Client()
+        >>> db1 = client.attach_database("duckdb", alias="test1")
+        >>> assert "test1" in client.databases
+        >>> db2 = client.attach_database("duckdb", alias="test2")
+        >>> assert "test2" in client.databases
+        >>> client.drop_all_databases()
+        >>> len(client.databases)
+        0
+
+
         :param missing_ok:
         :return:
         """
+        if not self._databases:
+            return
         for name in list(self._databases.keys()):
             self.drop_database(name, missing_ok=False, **kwargs)
         self._databases = {}