linkml-store 0.1.9__tar.gz → 0.1.10__tar.gz

linkml_store-0.1.10/PKG-INFO

@@ -0,0 +1,138 @@
+Metadata-Version: 2.1
+Name: linkml-store
+Version: 0.1.10
+Summary: linkml-store
+License: MIT
+Author: Author 1
+Author-email: author@org.org
+Requires-Python: >=3.9, !=2.7.*, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*, !=3.7.*, !=3.8.*
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Provides-Extra: analytics
+Provides-Extra: app
+Provides-Extra: chromadb
+Provides-Extra: fastapi
+Provides-Extra: h5py
+Provides-Extra: llm
+Provides-Extra: map
+Provides-Extra: mongodb
+Provides-Extra: pyarrow
+Provides-Extra: renderer
+Provides-Extra: tests
+Provides-Extra: validation
+Requires-Dist: black (>=24.0.0) ; extra == "tests"
+Requires-Dist: chromadb ; extra == "chromadb"
+Requires-Dist: click
+Requires-Dist: duckdb (>=0.10.1,<0.11.0)
+Requires-Dist: duckdb-engine (>=0.11.2)
+Requires-Dist: fastapi ; extra == "fastapi"
+Requires-Dist: h5py ; extra == "h5py"
+Requires-Dist: jinja2 (>=3.1.4,<4.0.0)
+Requires-Dist: jsonlines (>=4.0.0,<5.0.0)
+Requires-Dist: linkml ; extra == "validation"
+Requires-Dist: linkml-runtime (>=1.8.0rc2)
+Requires-Dist: linkml_map ; extra == "map"
+Requires-Dist: linkml_renderer ; extra == "renderer"
+Requires-Dist: llm ; extra == "llm"
+Requires-Dist: matplotlib ; extra == "analytics"
+Requires-Dist: pandas (>=2.2.1) ; extra == "analytics"
+Requires-Dist: plotly ; extra == "analytics"
+Requires-Dist: pyarrow ; extra == "pyarrow"
+Requires-Dist: pydantic (>=2.0.0,<3.0.0)
+Requires-Dist: pymongo ; extra == "mongodb"
+Requires-Dist: pystow (>=0.5.4,<0.6.0)
+Requires-Dist: seaborn ; extra == "analytics"
+Requires-Dist: sqlalchemy
+Requires-Dist: streamlit (>=1.32.2,<2.0.0) ; extra == "app"
+Requires-Dist: uvicorn ; extra == "fastapi"
+Description-Content-Type: text/markdown
+
+# linkml-store
+
+An AI-ready data management and integration platform. LinkML-Store
+provides an abstraction layer over multiple different backends
+(including DuckDB, MongoDB, and local filesystems), allowing for
+common query, index, and storage operations.
+
+For full documentation, see [https://linkml.io/linkml-store/](https://linkml.io/linkml-store/)
+
+__Warning__ LinkML-Store is still undergoing changes and refactoring,
+APIs and command line options are subject to change!
+
+## Quick Start
+
+Install, add data, query it:
+
+```
+pip install linkml-store[all]
+linkml-store -d duckdb:///db/my.db -c persons insert data/*.json
+linkml-store -d duckdb:///db/my.db -c persons query -w "occupation: Bricklayer"
+```
+
+Index it, search it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons index -t llm
+linkml-store -d duckdb:///db/my.db -c persons search "all persons employed in construction"
+```
+
+Validate it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons validate
+```
+
+## Basic usage
+
+* [Command Line](https://linkml.io/linkml-store/tutorials/Command-Line-Tutorial.html)
+* [Python](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* API
+* Streamlit applications
+
+## Features
+
+### Multiple Adapters
+
+LinkML-Store is designed to work with multiple backends, giving a common abstraction layer
+
+* [MongoDB](https://linkml.io/linkml-store/how-to/Use-MongoDB.html)
+* [DuckDB](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* [Solr](https://linkml.io/linkml-store/how-to/Query-Solr-using-CLI.html)
+* Filesystem
+
+Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...
+
+The intent is to give a union of all features of each backend. For
+example, analytic faceted queries are provided for *all* backends, not
+just Solr.
+
+### Composable indexes
+
+Many backends come with their own indexing and search
+schemes. Classically this was Lucene-based indexes, now it is semantic
+search using LLM embeddings.
+
+LinkML store treats indexing as an orthogonal concern - you can
+compose different indexing schemes with different backends. You don't
+need to have a vector database to run embedding search!
+
+See [How to Use-Semantic-Search](https://linkml.io/linkml-store/how-to/Use-Semantic-Search.html)
+
+### Validation
+
+LinkML-Store is backed by [LinkML](https://linkml.io), which allows
+for powerful expressive structural and semantic constraints.
+
+See [Indexing JSON](https://linkml.io/linkml-store/how-to/Index-Phenopackets.html)
+
+and [Referential Integrity](https://linkml.io/linkml-store/how-to/Check-Referential-Integrity.html)
+
+
+## Background
+
+See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for more details
+

linkml_store-0.1.10/README.md

@@ -0,0 +1,84 @@
+# linkml-store
+
+An AI-ready data management and integration platform. LinkML-Store
+provides an abstraction layer over multiple different backends
+(including DuckDB, MongoDB, and local filesystems), allowing for
+common query, index, and storage operations.
+
+For full documentation, see [https://linkml.io/linkml-store/](https://linkml.io/linkml-store/)
+
+__Warning__ LinkML-Store is still undergoing changes and refactoring,
+APIs and command line options are subject to change!
+
+## Quick Start
+
+Install, add data, query it:
+
+```
+pip install linkml-store[all]
+linkml-store -d duckdb:///db/my.db -c persons insert data/*.json
+linkml-store -d duckdb:///db/my.db -c persons query -w "occupation: Bricklayer"
+```
+
+Index it, search it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons index -t llm
+linkml-store -d duckdb:///db/my.db -c persons search "all persons employed in construction"
+```
+
+Validate it:
+
+```
+linkml-store -d duckdb:///db/my.db -c persons validate
+```
+
+## Basic usage
+
+* [Command Line](https://linkml.io/linkml-store/tutorials/Command-Line-Tutorial.html)
+* [Python](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* API
+* Streamlit applications
+
+## Features
+
+### Multiple Adapters
+
+LinkML-Store is designed to work with multiple backends, giving a common abstraction layer
+
+* [MongoDB](https://linkml.io/linkml-store/how-to/Use-MongoDB.html)
+* [DuckDB](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
+* [Solr](https://linkml.io/linkml-store/how-to/Query-Solr-using-CLI.html)
+* Filesystem
+
+Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...
+
+The intent is to give a union of all features of each backend. For
+example, analytic faceted queries are provided for *all* backends, not
+just Solr.
+
+### Composable indexes
+
+Many backends come with their own indexing and search
+schemes. Classically this was Lucene-based indexes, now it is semantic
+search using LLM embeddings.
+
+LinkML store treats indexing as an orthogonal concern - you can
+compose different indexing schemes with different backends. You don't
+need to have a vector database to run embedding search!
+
+See [How to Use-Semantic-Search](https://linkml.io/linkml-store/how-to/Use-Semantic-Search.html)
+
+### Validation
+
+LinkML-Store is backed by [LinkML](https://linkml.io), which allows
+for powerful expressive structural and semantic constraints.
+
+See [Indexing JSON](https://linkml.io/linkml-store/how-to/Index-Phenopackets.html)
+
+and [Referential Integrity](https://linkml.io/linkml-store/how-to/Check-Referential-Integrity.html)
+
+
+## Background
+
+See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for more details

{linkml_store-0.1.9 → linkml_store-0.1.10}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "linkml-store"
-version = "0.1.9"
+version = "0.1.10"
 description = "linkml-store"
 authors = ["Author 1 <author@org.org>"]
 license = "MIT"
@@ -10,7 +10,7 @@ readme = "README.md"
 python = "^3.9, !=3.9.7"
 click = "*"
 pydantic = "^2.0.0"
-linkml-runtime = "^1.7.5"
+linkml-runtime = ">=1.8.0rc2"
 streamlit = { version = "^1.32.2", optional = true }
 sqlalchemy = "*"
 duckdb = "^0.10.1"
@@ -27,9 +27,12 @@ pyarrow = { version="*", optional = true }
 h5py = { version="*", optional = true }
 linkml = { version="*", optional = true }
 linkml_map = { version="*", optional = true }
+linkml_renderer = { version="*", optional = true }
 pandas = ">=2.2.1"
 jinja2 = "^3.1.4"
 jsonlines = "^4.0.0"
+fastapi = { version="*", optional = true }
+uvicorn = { version="*", optional = true }
 
 [tool.poetry.group.dev.dependencies]
 pytest = {version = ">=7.1.2"}
@@ -66,9 +69,12 @@ h5py = ["h5py"]
 pyarrow = ["pyarrow"]
 validation = ["linkml"]
 map = ["linkml_map"]
+renderer = ["linkml_renderer"]
+fastapi = ["fastapi", "uvicorn"]
 
 [tool.poetry.scripts]
 linkml-store = "linkml_store.cli:cli"
+linkml-store-api = "linkml_store.webapi.main:start"
 
 [tool.poetry-dynamic-versioning]
 enable = false

{linkml_store-0.1.9 → linkml_store-0.1.10}/src/linkml_store/api/client.py

@@ -98,7 +98,7 @@ class Client:
         """
         return self.metadata.base_dir
 
-    def from_config(self, config: Union[ClientConfig, str, Path], base_dir=None, **kwargs):
+    def from_config(self, config: Union[ClientConfig, dict, str, Path], base_dir=None, **kwargs):
         """
         Create a client from a configuration.
 
@@ -118,11 +118,13 @@ class Client:
         :return:
 
         """
+        if isinstance(config, dict):
+            config = ClientConfig(**config)
         if isinstance(config, Path):
             config = str(config)
         if isinstance(config, str):
-            if not base_dir:
-                base_dir = Path(config).parent
+            # if not base_dir:
+            #    base_dir = Path(config).parent
             parsed_obj = yaml.safe_load(open(config))
             config = ClientConfig(**parsed_obj)
         self.metadata = config
@@ -133,8 +135,15 @@ class Client:
 
     def _initialize_databases(self, **kwargs):
         for name, db_config in self.metadata.databases.items():
-            handle = db_config.handle.format(base_dir=self.base_dir)
+            base_dir = self.base_dir
+            logger.info(f"Initializing database: {name}, base_dir: {base_dir}")
+            if not base_dir:
+                base_dir = Path.cwd()
+                logger.info(f"Using current working directory: {base_dir}")
+            handle = db_config.handle.format(base_dir=base_dir)
             db_config.handle = handle
+            if db_config.schema_location:
+                db_config.schema_location = db_config.schema_location.format(base_dir=base_dir)
             db = self.attach_database(handle, alias=name, **kwargs)
             db.from_config(db_config)
 

{linkml_store-0.1.9 → linkml_store-0.1.10}/src/linkml_store/api/collection.py

@@ -4,7 +4,7 @@ import hashlib
 import logging
 from collections import defaultdict
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Dict, Generic, Iterator, List, Optional, TextIO, Tuple, Type, Union
+from typing import TYPE_CHECKING, Any, ClassVar, Dict, Generic, Iterator, List, Optional, TextIO, Tuple, Type, Union
 
 import numpy as np
 from linkml_runtime import SchemaView
@@ -64,6 +64,7 @@ class Collection(Generic[DatabaseType]):
     # hidden: Optional[bool] = False
 
     metadata: Optional[CollectionConfig] = None
+    default_index_name: ClassVar[str] = "simple"
 
     def __init__(
         self, name: str, parent: Optional["Database"] = None, metadata: Optional[CollectionConfig] = None, **kwargs
@@ -421,7 +422,30 @@ class Collection(Generic[DatabaseType]):
         **kwargs,
     ) -> QueryResult:
         """
-        Search the collection using a full-text search index.
+        Search the collection using a text-based index index.
+
+        Example:
+
+        >>> from linkml_store import Client
+        >>> from linkml_store.utils.format_utils import load_objects
+        >>> client = Client()
+        >>> db = client.attach_database("duckdb")
+        >>> collection = db.create_collection("Country")
+        >>> objs = load_objects("tests/input/countries/countries.jsonl")
+        >>> collection.insert(objs)
+
+        Now let's index, using the simple trigram-based index
+
+        >>> index = get_indexer("simple")
+        >>> collection.attach_indexer(index)
+
+        Now let's find all objects:
+
+        >>> qr = collection.search("France")
+        >>> score, top_obj = qr.ranked_rows[0]
+        >>> assert score > 0.1
+        >>> top_obj["code"]
+        'FR'
 
         :param query:
         :param where:
@@ -431,12 +455,18 @@ class Collection(Generic[DatabaseType]):
         :return:
         """
         if index_name is None:
-            if len(self._indexers) == 1:
-                index_name = list(self._indexers.keys())[0]
+            if len(self.indexers) == 1:
+                index_name = list(self.indexers.keys())[0]
             else:
-                raise ValueError("Multiple indexes found. Please specify an index name.")
+                logger.warning("Multiple indexes found. Using default index.")
+                index_name = self.default_index_name
         ix_coll = self.parent.get_collection(self._index_collection_name(index_name))
-        ix = self._indexers.get(index_name)
+        if index_name not in self.indexers:
+            ix = get_indexer(index_name)
+            if not self._indexers:
+                self._indexers = {}
+            self._indexers[index_name] = ix
+        ix = self.indexers.get(index_name)
         if not ix:
             raise ValueError(f"No index named {index_name}")
         qr = ix_coll.find(where=where, limit=-1, **kwargs)
@@ -453,7 +483,10 @@ class Collection(Generic[DatabaseType]):
     @property
     def is_internal(self) -> bool:
         """
-        Check if the collection is internal
+        Check if the collection is internal.
+
+        Internal collections are hidden by default. Examples of internal collections
+        include shadow "index" collections
 
         :return:
         """
@@ -469,6 +502,45 @@ class Collection(Generic[DatabaseType]):
         """
         Attach an index to the collection.
 
+        As an example, first let's create a collection in a database:
+
+        >>> from linkml_store import Client
+        >>> from linkml_store.utils.format_utils import load_objects
+        >>> client = Client()
+        >>> db = client.attach_database("duckdb")
+        >>> collection = db.create_collection("Country")
+        >>> objs = load_objects("tests/input/countries/countries.jsonl")
+        >>> collection.insert(objs)
+
+        We will create two indexes - one that indexes the whole object
+        (default behavior), the other one indexes the name only
+
+        >>> full_index = get_indexer("simple")
+        >>> full_index.name = "full"
+        >>> name_index = get_indexer("simple", text_template="{name}")
+        >>> name_index.name = "name"
+        >>> collection.attach_indexer(full_index)
+        >>> collection.attach_indexer(name_index)
+
+        Now let's find objects using the full index, using the string "France".
+        We expect the country France to be the top hit, but the score will
+        be less than zero because we did not match all fields in the object.
+
+        >>> qr = collection.search("France", index_name="full")
+        >>> score, top_obj = qr.ranked_rows[0]
+        >>> assert score > 0.1
+        >>> assert score < 0.5
+        >>> top_obj["code"]
+        'FR'
+
+        Now using the name index
+
+        >>> qr = collection.search("France", index_name="name")
+        >>> score, top_obj = qr.ranked_rows[0]
+        >>> assert score > 0.99
+        >>> top_obj["code"]
+        'FR'
+
         :param index:
         :param name:
         :param auto_index: Automatically index all objects in the collection
@@ -504,15 +576,18 @@ class Collection(Generic[DatabaseType]):
 
     def index_objects(self, objs: List[OBJECT], index_name: str, replace=False, **kwargs):
         """
-        Index a list of objects
+        Index a list of objects using a specified index.
+
+        By default, the indexed objects will be stored in a shadow
+        collection in the same database, with additional fields for the index vector
 
         :param objs:
-        :param index_name:
+        :param index_name: e.g. simple, llm
         :param replace:
         :param kwargs:
         :return:
         """
-        ix = self._indexers.get(index_name)
+        ix = self._indexers.get(index_name, None)
         if not ix:
             raise ValueError(f"No index named {index_name}")
         ix_coll_name = self._index_collection_name(index_name)

{linkml_store-0.1.9 → linkml_store-0.1.10}/src/linkml_store/api/config.py

@@ -3,7 +3,11 @@ from typing import Any, Dict, List, Optional
 from pydantic import BaseModel, Field
 
 
-class CollectionConfig(BaseModel):
+class ConfiguredBaseModel(BaseModel, extra="forbid"):
+    pass
+
+
+class CollectionConfig(ConfiguredBaseModel):
     name: Optional[str] = Field(
         default=None,
         description="An optional name for the collection",
@@ -42,7 +46,7 @@ class CollectionConfig(BaseModel):
     )
 
 
-class DatabaseConfig(BaseModel):
+class DatabaseConfig(ConfiguredBaseModel):
     handle: str = Field(
         default="duckdb:///:memory:",
         description="The database handle, e.g., 'duckdb:///:memory:' or 'mongodb://localhost:27017'",
@@ -86,7 +90,7 @@ class DatabaseConfig(BaseModel):
     )
 
 
-class ClientConfig(BaseModel):
+class ClientConfig(ConfiguredBaseModel):
     handle: Optional[str] = Field(
         default=None,
         description="The client handle",
@@ -95,6 +99,10 @@ class ClientConfig(BaseModel):
         default={},
         description="A dictionary of database configurations",
     )
+    default_database: Optional[str] = Field(
+        default=None,
+        description="The default database",
+    )
     schema_path: Optional[str] = Field(
         default=None,
         description="The path to the LinkML schema file",

{linkml_store-0.1.9 → linkml_store-0.1.10}/src/linkml_store/api/stores/duckdb/duckdb_collection.py

@@ -90,7 +90,9 @@ class DuckDBCollection(Collection):
         cd = self.class_definition()
         with self.parent.engine.connect() as conn:
             if not facet_columns:
-                facet_columns = list(self.class_definition().attributes.keys())
+                if not cd:
+                    raise ValueError(f"No class definition found for {self.target_class_name}")
+                facet_columns = list(cd.attributes.keys())
             for col in facet_columns:
                 logger.debug(f"Faceting on {col}")
                 if isinstance(col, tuple):
@@ -101,7 +103,7 @@ class DuckDBCollection(Collection):
                 facet_query_str = facet_count_sql(facet_query, col, multivalued=sd.multivalued)
                 logger.debug(f"Facet query: {facet_query_str}")
                 rows = list(conn.execute(text(facet_query_str)))
-                results[col] = rows
+                results[col] = [tuple(row) for row in rows]
             return results
 
     def _sqla_table(self, cd: ClassDefinition) -> Table:
@@ -110,7 +112,7 @@ class DuckDBCollection(Collection):
         cols = []
         for att in schema_view.class_induced_slots(cd.name):
             typ = TMAP.get(att.range, sqla.String)
-            if att.inlined:
+            if att.inlined or att.inlined_as_list:
                 typ = sqla.JSON
             if att.multivalued:
                 typ = sqla.ARRAY(typ, dimensions=1)

{linkml_store-0.1.9 → linkml_store-0.1.10}/src/linkml_store/api/stores/duckdb/duckdb_database.py

@@ -31,6 +31,18 @@ logger = logging.getLogger(__name__)
 
 
 class DuckDBDatabase(Database):
+    """
+    An adapter for DuckDB databases.
+
+    Note that this adapter does not make use of a LinkML relational model transformation and
+    SQL Alchemy ORM layer. Instead, it attempts to map each collection (which is of type
+    some LinkML class) to a *single* DuckDB table. New tables are not created for nested references,
+    and linking tables are not created for many-to-many relationships.
+
+    Instead the native DuckDB ARRAY type is used to store multivalued attributes, and DuckDB JSON
+    types are used for nested inlined objects.
+    """
+
     _connection: DuckDBPyConnection = None
     _engine: sqlalchemy.Engine = None
     collection_class = DuckDBCollection
@@ -103,7 +115,14 @@ class DuckDBDatabase(Database):
                         if row[col]:
                             if isinstance(row[col], list):
                                 for i in range(len(row[col])):
-                                    row[col][i] = json.loads(row[col][i])
+                                    try:
+                                        parsed_val = json.loads(row[col][i])
+                                    except json.JSONDecodeError as e:
+                                        logger.error(f"Failed to parse col {col}[{i}] == {row[col][i]}")
+                                        raise e
+                                    row[col][i] = parsed_val
+                            elif isinstance(row[col], dict):
+                                pass
                             else:
                                 row[col] = json.loads(row[col])
             qr.set_rows(pd.DataFrame(rows))

{linkml_store-0.1.9 → linkml_store-0.1.10}/src/linkml_store/api/stores/filesystem/filesystem_collection.py

@@ -170,6 +170,8 @@ class FileSystemCollection(Collection[DatabaseType]):
             for fc in facet_columns:
                 if fc in row:
                     v = row[fc]
+                    if not isinstance(v, str):
+                        v = str(v)
                     if v not in facet_results[fc]:
                         facet_results[fc][v] = 1
                     else: