linkml-store 0.1.13__tar.gz → 0.2.0__tar.gz

{linkml_store-0.1.13 → linkml_store-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: linkml-store
-Version: 0.1.13
+Version: 0.2.0
 Summary: linkml-store
 License: MIT
 Author: Author 1
@@ -24,6 +24,7 @@ Provides-Extra: mongodb
 Provides-Extra: neo4j
 Provides-Extra: pyarrow
 Provides-Extra: renderer
+Provides-Extra: scipy
 Provides-Extra: tests
 Provides-Extra: validation
 Requires-Dist: black (>=24.0.0) ; extra == "tests"
@@ -51,9 +52,12 @@ Requires-Dist: pyarrow ; extra == "pyarrow"
 Requires-Dist: pydantic (>=2.0.0,<3.0.0)
 Requires-Dist: pymongo ; extra == "mongodb"
 Requires-Dist: pystow (>=0.5.4,<0.6.0)
+Requires-Dist: scikit-learn ; extra == "scipy"
+Requires-Dist: scipy ; extra == "scipy"
 Requires-Dist: seaborn ; extra == "analytics"
 Requires-Dist: sqlalchemy
 Requires-Dist: streamlit (>=1.32.2,<2.0.0) ; extra == "app"
+Requires-Dist: tiktoken ; extra == "llm"
 Requires-Dist: uvicorn ; extra == "fastapi"
 Description-Content-Type: text/markdown
 
@@ -61,11 +65,13 @@ Description-Content-Type: text/markdown
 
 An AI-ready data management and integration platform. LinkML-Store
 provides an abstraction layer over multiple different backends
-(including DuckDB, MongoDB, and local filesystems), allowing for
+(including DuckDB, MongoDB, Neo4j, and local filesystems), allowing for
 common query, index, and storage operations.
 
 For full documentation, see [https://linkml.io/linkml-store/](https://linkml.io/linkml-store/)
 
+See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for a high level overview.
+
 __Warning__ LinkML-Store is still undergoing changes and refactoring,
 APIs and command line options are subject to change!
 
@@ -99,6 +105,23 @@ linkml-store -d duckdb:///db/my.db -c persons validate
 * API
 * Streamlit applications
 
+## The CRUDSI pattern
+
+Most database APIs implement the **CRUD** pattern: Create, Read, Update, Delete.
+LinkML-Store adds **Search** and **Inference** to this pattern, making it **CRUDSI**.
+
+The notion of "Search" and "Inference" is intended to be flexible and extensible,
+including:
+
+* Search
+   * Traditional keyword search
+   * Search using LLM Vector embeddings (*without* a dedicated vector database)
+   * Pluggable specialized search, e.g. genomic sequence (not yet implemented)
+* Inference (encompassing  *validation*, *repair*, and inference of missing data)
+   * Classic rule-based inference
+   * Inference using LLM Retrieval Augmented Generation (RAG)
+   * Statistical/ML inference
+
 ## Features
 
 ### Multiple Adapters
@@ -108,6 +131,8 @@ LinkML-Store is designed to work with multiple backends, giving a common abstrac
 * [MongoDB](https://linkml.io/linkml-store/how-to/Use-MongoDB.html)
 * [DuckDB](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
 * [Solr](https://linkml.io/linkml-store/how-to/Query-Solr-using-CLI.html)
+* [Neo4j](https://linkml.io/linkml-store/how-to/Use-Neo4j.html)
+
 * Filesystem
 
 Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...
@@ -173,3 +198,4 @@ make app
 
 See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for more details
 
+

{linkml_store-0.1.13 → linkml_store-0.2.0}/README.md

@@ -2,11 +2,13 @@
 
 An AI-ready data management and integration platform. LinkML-Store
 provides an abstraction layer over multiple different backends
-(including DuckDB, MongoDB, and local filesystems), allowing for
+(including DuckDB, MongoDB, Neo4j, and local filesystems), allowing for
 common query, index, and storage operations.
 
 For full documentation, see [https://linkml.io/linkml-store/](https://linkml.io/linkml-store/)
 
+See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for a high level overview.
+
 __Warning__ LinkML-Store is still undergoing changes and refactoring,
 APIs and command line options are subject to change!
 
@@ -40,6 +42,23 @@ linkml-store -d duckdb:///db/my.db -c persons validate
 * API
 * Streamlit applications
 
+## The CRUDSI pattern
+
+Most database APIs implement the **CRUD** pattern: Create, Read, Update, Delete.
+LinkML-Store adds **Search** and **Inference** to this pattern, making it **CRUDSI**.
+
+The notion of "Search" and "Inference" is intended to be flexible and extensible,
+including:
+
+* Search
+   * Traditional keyword search
+   * Search using LLM Vector embeddings (*without* a dedicated vector database)
+   * Pluggable specialized search, e.g. genomic sequence (not yet implemented)
+* Inference (encompassing  *validation*, *repair*, and inference of missing data)
+   * Classic rule-based inference
+   * Inference using LLM Retrieval Augmented Generation (RAG)
+   * Statistical/ML inference
+
 ## Features
 
 ### Multiple Adapters
@@ -49,6 +68,8 @@ LinkML-Store is designed to work with multiple backends, giving a common abstrac
 * [MongoDB](https://linkml.io/linkml-store/how-to/Use-MongoDB.html)
 * [DuckDB](https://linkml.io/linkml-store/tutorials/Python-Tutorial.html)
 * [Solr](https://linkml.io/linkml-store/how-to/Query-Solr-using-CLI.html)
+* [Neo4j](https://linkml.io/linkml-store/how-to/Use-Neo4j.html)
+
 * Filesystem
 
 Coming soon: any RDBMS, any triplestore, Neo4J, HDF5-based stores, ChromaDB/Vector dbs ...
@@ -113,3 +134,4 @@ make app
 ## Background
 
 See [these slides](https://docs.google.com/presentation/d/e/2PACX-1vSgtWUNUW0qNO_ZhMAGQ6fYhlXZJjBNMYT0OiZz8DDx8oj7iG9KofRs6SeaMXBBOICGknoyMG2zaHnm/embed?start=false&loop=false&delayms=3000) for more details
+

{linkml_store-0.1.13 → linkml_store-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "linkml-store"
-version = "0.1.13"
+version = "0.2.0"
 description = "linkml-store"
 authors = ["Author 1 <author@org.org>"]
 license = "MIT"
@@ -21,6 +21,7 @@ plotly = { version = "*", optional = true }
 pystow = "^0.5.4"
 black = { version=">=24.0.0", optional = true }
 llm = { version="*", optional = true }
+tiktoken = { version="*", optional = true }
 pymongo = { version="*", optional = true }
 neo4j = { version="*", optional = true }
 py2neo = { version="*", optional = true }
@@ -28,6 +29,8 @@ networkx = { version="*", optional = true }
 chromadb = { version="*", optional = true }
 pyarrow = { version="*", optional = true }
 h5py = { version="*", optional = true }
+scipy = { version="*", optional = true }
+scikit-learn = { version="*", optional = true }
 linkml = { version=">=1.8.0", optional = true }
 linkml_map = { version="*", optional = true }
 linkml_renderer = { version="*", optional = true }
@@ -67,7 +70,7 @@ numpy = [
 analytics = ["pandas", "matplotlib", "seaborn", "plotly"]
 app = ["streamlit"]
 tests = ["black"]
-llm = ["llm"]
+llm = ["llm", "tiktoken"]
 mongodb = ["pymongo"]
 neo4j = ["neo4j", "py2neo", "networkx"]
 chromadb = ["chromadb"]
@@ -78,6 +81,7 @@ map = ["linkml_map"]
 renderer = ["linkml_renderer"]
 fastapi = ["fastapi", "uvicorn"]
 frictionless = ["frictionless"]
+scipy = ["scipy", "scikit-learn"]
 
 [tool.poetry.scripts]
 linkml-store = "linkml_store.cli:cli"

{linkml_store-0.1.13 → linkml_store-0.2.0}/src/linkml_store/api/client.py

@@ -100,7 +100,7 @@ class Client:
         """
         return self.metadata.base_dir
 
-    def from_config(self, config: Union[ClientConfig, dict, str, Path], base_dir=None, **kwargs):
+    def from_config(self, config: Union[ClientConfig, dict, str, Path], base_dir=None, auto_attach=False, **kwargs):
         """
         Create a client from a configuration.
 
@@ -109,6 +109,10 @@ class Client:
         >>> from linkml_store.api.config import ClientConfig
         >>> client = Client().from_config(ClientConfig(databases={"test": {"handle": "duckdb:///:memory:"}}))
         >>> len(client.databases)
+        0
+        >>> client = Client().from_config(ClientConfig(databases={"test": {"handle": "duckdb:///:memory:"}}),
+        ...                                auto_attach=True)
+        >>> len(client.databases)
         1
         >>> "test" in client.databases
         True
@@ -116,6 +120,8 @@ class Client:
         'duckdb:///:memory:'
 
         :param config:
+        :param base_dir:
+        :param auto_attach:
         :param kwargs:
         :return:
 
@@ -125,17 +131,17 @@ class Client:
         if isinstance(config, Path):
             config = str(config)
         if isinstance(config, str):
-            # if not base_dir:
-            #    base_dir = Path(config).parent
+            if not base_dir:
+                base_dir = Path(config).parent
             parsed_obj = yaml.safe_load(open(config))
             config = ClientConfig(**parsed_obj)
         self.metadata = config
         if base_dir:
             self.metadata.base_dir = base_dir
-        self._initialize_databases(**kwargs)
+        self._initialize_databases(auto_attach=auto_attach, **kwargs)
         return self
 
-    def _initialize_databases(self, **kwargs):
+    def _initialize_databases(self, auto_attach=False, **kwargs):
         for name, db_config in self.metadata.databases.items():
             base_dir = self.base_dir
             logger.info(f"Initializing database: {name}, base_dir: {base_dir}")
@@ -146,8 +152,22 @@ class Client:
             db_config.handle = handle
             if db_config.schema_location:
                 db_config.schema_location = db_config.schema_location.format(base_dir=base_dir)
-            db = self.attach_database(handle, alias=name, **kwargs)
-            db.from_config(db_config)
+            if auto_attach:
+                db = self.attach_database(handle, alias=name, **kwargs)
+                db.from_config(db_config)
+
+    def _set_database_config(self, db: Database):
+        """
+        Set the configuration for a database.
+
+        :param name:
+        :param config:
+        :return:
+        """
+        if not self.metadata:
+            return
+        if db.alias in self.metadata.databases:
+            db.from_config(self.metadata.databases[db.alias])
 
     def attach_database(
         self,
@@ -202,6 +222,7 @@ class Client:
                 raise AssertionError(f"Inconsistent alias: {db.alias} != {alias}")
         else:
             db.metadata.alias = alias
+        self._set_database_config(db)
         return db
 
     def get_database(self, name: Optional[str] = None, create_if_not_exists=True, **kwargs) -> Database:
@@ -230,13 +251,19 @@ class Client:
             return list(self._databases.values())[0]
         if not self._databases:
             self._databases = {}
+        if name not in self._databases and name in self.metadata.databases:
+            db_config = self.metadata.databases[name]
+            db = self.attach_database(db_config.handle, alias=name, **kwargs)
+            self._databases[name] = db
         if name not in self._databases:
             if create_if_not_exists:
                 logger.info(f"Creating database: {name}")
                 self.attach_database(name, **kwargs)
             else:
                 raise ValueError(f"Database {name} does not exist")
-        return self._databases[name]
+        db = self._databases[name]
+        self._set_database_config(db)
+        return db
 
     @property
     def databases(self) -> Dict[str, Database]:

{linkml_store-0.1.13 → linkml_store-0.2.0}/src/linkml_store/api/collection.py

@@ -502,6 +502,7 @@ class Collection(Generic[DatabaseType]):
                 index_name = self.default_index_name
         ix_coll = self.parent.get_collection(self._index_collection_name(index_name))
         if index_name not in self.indexers:
+            logger.debug(f"Indexer not found: {index_name} -- creating")
             ix = get_indexer(index_name)
             if not self._indexers:
                 self._indexers = {}
@@ -509,6 +510,11 @@ class Collection(Generic[DatabaseType]):
         ix = self.indexers.get(index_name)
         if not ix:
             raise ValueError(f"No index named {index_name}")
+        logger.debug(f"Using indexer {type(ix)} with name {index_name}")
+        if ix_coll.size() == 0:
+            logger.info(f"Index {index_name} is empty; indexing all objects")
+            all_objs = self.find(limit=-1).rows
+            self.index_objects(all_objs, index_name, replace=True, **kwargs)
         qr = ix_coll.find(where=where, limit=-1, **kwargs)
         index_col = ix.index_field
         # TODO: optimize this for large indexes
@@ -518,6 +524,7 @@ class Collection(Generic[DatabaseType]):
             del r[1][index_col]
         new_qr = QueryResult(num_rows=len(results))
         new_qr.ranked_rows = results
+        new_qr.rows = [r[1] for r in results]
         return new_qr
 
     @property
@@ -562,6 +569,7 @@ class Collection(Generic[DatabaseType]):
                     format=source.format,
                     expected_type=source.expected_type,
                     compression=source.compression,
+                    select_query=source.select_query,
                     **kwargs,
                 )
             elif metadata.source.url:
@@ -570,9 +578,12 @@ class Collection(Generic[DatabaseType]):
                     format=source.format,
                     expected_type=source.expected_type,
                     compression=source.compression,
+                    select_query=source.select_query,
                     **kwargs,
                 )
-        self.insert(objects)
+            else:
+                raise ValueError("No source local_path or url provided")
+            self.insert(objects)
 
     def _check_if_initialized(self) -> bool:
         return self._initialized
@@ -629,6 +640,14 @@ class Collection(Generic[DatabaseType]):
             self.insert(tr_objs)
             self.commit()
 
+    def size(self) -> int:
+        """
+        Return the number of objects in the collection.
+
+        :return: The number of objects in the collection.
+        """
+        return self.find({}, limit=1).num_rows
+
     def attach_indexer(self, index: Union[Indexer, str], name: Optional[str] = None, auto_index=True, **kwargs):
         """
         Attach an index to the collection.
@@ -777,6 +796,8 @@ class Collection(Generic[DatabaseType]):
         sv: SchemaView = self.parent.schema_view
         if sv:
             cls = sv.get_class(self.target_class_name)
+            # if not cls:
+            #     logger.warning(f"{self.target_class_name} not in {sv.all_classes().keys()} ")
             # cls = sv.schema.classes[self.target_class_name]
             if cls and not cls.attributes:
                 if not sv.class_induced_slots(cls.name):
@@ -900,11 +921,14 @@ class Collection(Generic[DatabaseType]):
                     exact_dimensions_list.append(v.shape)
                     break
                 if isinstance(v, list):
+                    # sample first item. TODO: more robust strategy
                     v = v[0] if v else None
                     multivalueds.append(True)
                 elif isinstance(v, dict):
-                    v = list(v.values())[0]
-                    multivalueds.append(True)
+                    pass
+                    # TODO: check if this is a nested object or key-value list
+                    # v = list(v.values())[0]
+                    # multivalueds.append(True)
                 else:
                     multivalueds.append(False)
                 if not v:
@@ -933,10 +957,21 @@ class Collection(Generic[DatabaseType]):
             #    raise AssertionError(f"Empty rngs for {k} = {vs}")
             rng = rngs[0] if rngs else None
             for other_rng in rngs:
+                coercions = {
+                    ("integer", "float"): "float",
+                }
                 if rng != other_rng:
-                    raise ValueError(f"Conflict: {rng} != {other_rng} for {vs}")
+                    if (rng, other_rng) in coercions:
+                        rng = coercions[(rng, other_rng)]
+                    elif (other_rng, rng) in coercions:
+                        rng = coercions[(other_rng, rng)]
+                    else:
+                        raise ValueError(f"Conflict: {rng} != {other_rng} for {vs}")
             logger.debug(f"Inducing {k} as {rng} {multivalued} {inlined}")
-            cd.attributes[k] = SlotDefinition(k, range=rng, multivalued=multivalued, inlined=inlined)
+            inlined_as_list = inlined and multivalued
+            cd.attributes[k] = SlotDefinition(
+                k, range=rng, multivalued=multivalued, inlined=inlined, inlined_as_list=inlined_as_list
+            )
             if exact_dimensions_list:
                 array_expr = ArrayExpression(exact_number_dimensions=len(exact_dimensions_list[0]))
                 cd.attributes[k].array = array_expr

{linkml_store-0.1.13 → linkml_store-0.2.0}/src/linkml_store/api/config.py

@@ -1,8 +1,8 @@
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Union
 
 from pydantic import BaseModel, Field
 
-from linkml_store.graphs.graph_map import GraphProjection
+from linkml_store.graphs.graph_map import EdgeProjection, NodeProjection
 
 
 class ConfiguredBaseModel(BaseModel, extra="forbid"):
@@ -30,13 +30,30 @@ class CollectionSource(ConfiguredBaseModel):
     """
 
     url: Optional[str] = None
+    """Remote URL to fetch data from"""
+
     local_path: Optional[str] = None
+    """Local path to fetch data from"""
+
     source_location: Optional[str] = None
+
     refresh_interval_days: Optional[float] = None
+    """How often to refresh the data, in days"""
+
     expected_type: Optional[str] = None
+    """The expected type of the data, e.g list"""
+
     format: Optional[str] = None
+    """The format of the data, e.g., json, yaml, csv"""
+
     compression: Optional[str] = None
+    """The compression of the data, e.g., tgz, gzip, zip"""
+
+    select_query: Optional[str] = None
+    """A jsonpath query to preprocess the objects with"""
+
     arguments: Optional[Dict[str, Any]] = None
+    """Optional arguments to pass to the source"""
 
 
 class CollectionConfig(ConfiguredBaseModel):
@@ -81,7 +98,7 @@ class CollectionConfig(ConfiguredBaseModel):
         description="LinkML-Map derivations",
     )
     page_size: Optional[int] = Field(default=None, description="Suggested page size (items per page) in apps and APIs")
-    graph_projection: Optional[GraphProjection] = Field(
+    graph_projection: Optional[Union[EdgeProjection, NodeProjection]] = Field(
         default=None,
         description="Optional graph projection configuration",
     )

{linkml_store-0.1.13 → linkml_store-0.2.0}/src/linkml_store/api/database.py

@@ -707,12 +707,29 @@ class Database(ABC, Generic[CollectionType]):
         """
         raise NotImplementedError()
 
-    def import_database(self, location: str, source_format: Optional[Union[str, Format]] = None, **kwargs):
+    def import_database(
+        self,
+        location: str,
+        source_format: Optional[Union[str, Format]] = None,
+        collection_name: Optional[str] = None,
+        **kwargs,
+    ):
         """
         Import a database from a file or location.
 
+        >>> from linkml_store.api.client import Client
+        >>> client = Client()
+        >>> db = client.attach_database("duckdb", alias="test")
+        >>> db.import_database("tests/input/iris.csv", Format.CSV, collection_name="iris")
+        >>> db.list_collection_names()
+        ['iris']
+        >>> collection = db.get_collection("iris")
+        >>> collection.find({}).num_rows
+        150
+
         :param location: location of the file
         :param source_format: source format
+        :param collection_name: (Optional) name of the collection, for data that is flat
         :param kwargs: additional arguments
         """
         if isinstance(source_format, str):
@@ -732,8 +749,12 @@ class Database(ABC, Generic[CollectionType]):
                 self.store(obj)
                 return
         objects = load_objects(location, format=source_format)
-        for obj in objects:
-            self.store(obj)
+        if collection_name:
+            collection = self.get_collection(collection_name, create_if_not_exists=True)
+            collection.insert(objects)
+        else:
+            for obj in objects:
+                self.store(obj)
 
     def export_database(self, location: str, target_format: Optional[Union[str, Format]] = None, **kwargs):
         """

{linkml_store-0.1.13 → linkml_store-0.2.0}/src/linkml_store/api/stores/duckdb/duckdb_collection.py

@@ -36,6 +36,9 @@ class DuckDBCollection(Collection):
         logger.info(f"Inserting into: {self.alias} // T={table.name}")
         engine = self.parent.engine
         col_names = [c.name for c in table.columns]
+        bad_objs = [obj for obj in objs if not isinstance(obj, dict)]
+        if bad_objs:
+            logger.error(f"Bad objects: {bad_objs}")
         objs = [{k: obj.get(k, None) for k in col_names} for obj in objs]
         with engine.connect() as conn:
             with conn.begin():

{linkml_store-0.1.13 → linkml_store-0.2.0}/src/linkml_store/api/stores/mongodb/mongodb_collection.py

@@ -51,9 +51,13 @@ class MongoDBCollection(Collection):
         if offset and offset >= 0:
             cursor = cursor.skip(offset)
 
+        select_cols = query.select_cols
+
         def _as_row(row: dict):
             row = copy(row)
             del row["_id"]
+            if select_cols:
+                row = {k: row[k] for k in select_cols if k in row}
             return row
 
         rows = [_as_row(row) for row in cursor]