pyneoinstance 2.0.2__tar.gz → 3.0.0__tar.gz

{pyneoinstance-2.0.2/pyneoinstance.egg-info → pyneoinstance-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: pyneoinstance
-Version: 2.0.2
+Version: 3.0.0
 Summary: Handle Neo4j read and write transactions and provide methos for performing graph EDA
 Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
 License: GNU GENERAL PUBLIC LICENSE
@@ -689,6 +689,7 @@ Requires-Dist: neo4j>=5.0.0
 Requires-Dist: pandas
 Requires-Dist: PyYAML
 Requires-Dist: pyvis
+Dynamic: license-file
 
 # PyNeoInstance Package
 
@@ -829,7 +830,7 @@ We can update a Neo4j database with data contained in a Pandas Data Frame using
 
 - _data_: Pandas Data Frame containing data to process.
 - _database_: Optional, database on which to execute the queries.
-- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
+- _batchSize_: Optional, number of records per batch use to break the Data Frame in smaller chucks.
 - _parallel_: Whether to execute the query using multiple Python process.
 - _workers_: The number of processes to spawn to execute the queries in parallel.
 - _parameters_: Extra arguments containing optional cypher parameters.
@@ -909,25 +910,25 @@ node_load_queries = config['loading_queries']['nodes']
 person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
 ```
 
-If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
+If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _batchSize_ argument the number of records per data batch to break this Data Frame into smaller data batches. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
 
 ```python
 person_load_result = graph.execute_write_query_with_data(
     node_load_queries['person'],
     person_data,
     db_info['database'],
-    partitions = 5
+    batchSize = 100000
     )
 ```
 ## Updating a database with data in a Data Frame: Parallel  
 
-To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
+To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _batchSize_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
 
 ```python
 person_load_result = graph.execute_write_query_with_data(
     node_load_queries['person'],
     person_data,
-    partitions = 5,
+    batchSize = 100000,
     parallel = True,
     workers = 5
     )

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/README.md

@@ -137,7 +137,7 @@ We can update a Neo4j database with data contained in a Pandas Data Frame using
 
 - _data_: Pandas Data Frame containing data to process.
 - _database_: Optional, database on which to execute the queries.
-- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
+- _batchSize_: Optional, number of records per batch use to break the Data Frame in smaller chucks.
 - _parallel_: Whether to execute the query using multiple Python process.
 - _workers_: The number of processes to spawn to execute the queries in parallel.
 - _parameters_: Extra arguments containing optional cypher parameters.
@@ -217,25 +217,25 @@ node_load_queries = config['loading_queries']['nodes']
 person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
 ```
 
-If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
+If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _batchSize_ argument the number of records per data batch to break this Data Frame into smaller data batches. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
 
 ```python
 person_load_result = graph.execute_write_query_with_data(
     node_load_queries['person'],
     person_data,
     db_info['database'],
-    partitions = 5
+    batchSize = 100000
     )
 ```
 ## Updating a database with data in a Data Frame: Parallel  
 
-To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
+To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _batchSize_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
 
 ```python
 person_load_result = graph.execute_write_query_with_data(
     node_load_queries['person'],
     person_data,
-    partitions = 5,
+    batchSize = 100000,
     parallel = True,
     workers = 5
     )
@@ -305,4 +305,4 @@ schema.show('schema.html',notebook=False)
 
 For more detailed examples please refer to the following articles:
 * Data ingestion: [PyNeoInstance: A User-Friendly Python Library for Neo4j](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
-* Graph EDA: [Performing graph EDA with the PyNeoInstance Python Package](https://medium.com/@jose.alvarado-guzman/0c0f6f230aa9)
+* Graph EDA: [Performing graph EDA with the PyNeoInstance Python Package](https://medium.com/@jose.alvarado-guzman/0c0f6f230aa9)

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyneoinstance/__init__.py

@@ -1,4 +1,4 @@
 from .database.neo4jdbms import Neo4jInstance
 from .fileload.loadyaml import load_yaml_file
 
-__version__ = '2.0.2'
+__version__ = '3.0.0'

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyneoinstance/database/context.py

@@ -3,4 +3,4 @@ import os
 import sys
 sys.path.insert(0, os.path.abspath(
     os.path.join(os.path.dirname(__file__), '..')))
-from pyneoinstance.util.functions import get_logger
+from pyneoinstance.util.functions import get_logger,get_batches,get_columns_diff

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyneoinstance/database/neo4jdbms.py

@@ -22,6 +22,7 @@ from neo4j.exceptions import ServiceUnavailable
 from neo4j.exceptions import AuthError
 from neo4j.exceptions import ClientError
 from neo4j.exceptions import ConfigurationError
+from .context import get_logger, get_batches, get_columns_diff
 
 def _get_driver(neo_info : Dict[str, str]):
     try:
@@ -59,54 +60,6 @@ def _read_transaction_function(transaction, query, **kwargs):
     data = DataFrame(results.values(), columns=results.keys())
     return data
 
-def _execute_write(session, query,
-                   rows: Optional[Dict[str, Any]] = None,
-                   parameters: Optional[Dict[str, Any]] = None
-                  ):
-    if parameters:
-        params = parameters
-    else:
-        params = {}
-    try:
-        if rows:
-            results = session.execute_write(
-                _write_transaction_function, query,
-                rows = rows, **params).counters.__dict__
-        else:
-            results = session.execute_write(
-                _write_transaction_function, query,
-                **params).counters.__dict__
-    except ServiceUnavailable as exception:
-        raise ServiceUnavailable() from exception
-    except ClientError as exception:
-        raise ClientError() from exception
-    return results
-
-def _execute_write_parallel(neo_info, database, query,
-                   rows: Optional[Dict[str, Any]] = None,
-                   parameters: Optional[Dict[str, Any]] = None
-                  ):
-    if parameters:
-        params = parameters
-    else:
-        params = {}
-    with _get_driver(neo_info) as driver:
-        with _get_session(driver, database) as session:
-            try:
-                if rows:
-                    results = session.execute_write(
-                        _write_transaction_function, query,
-                        rows = rows, **params).counters.__dict__
-                else:
-                    results = session.execute_write(
-                        _write_transaction_function, query,
-                        **params).counters.__dict__
-            except ServiceUnavailable as exception:
-                raise ServiceUnavailable() from exception
-            except ClientError as exception:
-                raise ClientError() from exception
-    return results
-
 class Neo4jInstance:
     """Class use to handle Neo4j requests.
 
@@ -169,7 +122,12 @@ class Neo4jInstance:
         self.neo_info['password'] = password
         self.neo_info['encrypted'] = kwargs.get('encrypted') or ''
         self.__results = defaultdict(int)
-
+        self.__logger = get_logger('pyneoinstance', logging.WARNING)
+        stream_handler = logging.StreamHandler()
+        formatter = logging.Formatter(
+            '%(levelname)s - %(message)s')
+        stream_handler.setFormatter(formatter)
+        self.__logger.addHandler(stream_handler)
     def execute_read_query(self, query: str,
                            database: Optional[str] = None,
                            parameters: Optional[Dict[str, Any]] = None
@@ -251,7 +209,7 @@ class Neo4jInstance:
         with _get_driver(self.neo_info) as driver:
             with _get_session(driver, database) as session:
                 for query in queries:
-                    result = _execute_write(session, query, parameters=params)
+                    result = self._execute_write(session, query, parameters=params)
                     for key, value in result.items():
                         if key != '_contains_updates':
                             results[key] += value
@@ -289,9 +247,10 @@ class Neo4jInstance:
         result = self.execute_write_queries([query], database, parameters)
         return  result
 
-    def execute_write_queries_with_data(self, queries: List[str], data: DataFrame,
+    def execute_write_queries_with_data(self, queries: List[str],
+                                        data: DataFrame,
                                         database: Optional[str] = None,
-                                        partitions: Optional[int] = 1,
+                                        batchSize: Optional[int] = 100_000,
                                         parallel: Optional[bool] = False,
                                         workers: Optional[int] = None,
                                         parameters: Optional[Dict[str, Any]] = None
@@ -307,8 +266,8 @@ class Neo4jInstance:
             database : str, optional
                 Name of the Neo4j database of which to execute the transactions.
                 If not provided the default database is going to be use.
-            partitions : int, optional
-                The number of partitions in which to split the data frame.
+            batchSize : int, optional
+                The number of records per batch or partitions of the data frame.
             parallel : bool, optional
                 Wheather to execute the load in parallel.
             workers : int, optional
@@ -332,10 +291,13 @@ class Neo4jInstance:
                 When the number of batch sizes to split the DataFrame on
                 is larger than the number of rows in it.
         """
-        if partitions > data.shape[0]:
-            raise ValueError(
-                "The batch size cannot be greater than the number of rows in the data.")
-        data_chunks = np.array_split(data,partitions)
+        row_num = data.shape[0]
+        batchSize = batchSize if batchSize <= row_num else row_num
+        msg = f'Partitioning the data in batches of size {batchSize}'
+        self.__logger.info(msg)
+        data_chunks = get_batches(data, batchSize)
+        partitions = len(data_chunks)
+        parallel = parallel if len(data_chunks)>1 else False
         if parameters:
             params = parameters
         else:
@@ -345,21 +307,33 @@ class Neo4jInstance:
                 workers_num = workers
             else:
                 workers_num = mp.cpu_count()
+            msg = f'Loading {partitions:,.0f} data chunks using {workers_num} thread(s)'
+            self.__logger.info(msg)
             for query in queries:
+                col_diff = get_columns_diff(query,data.columns)
+                msg = f'This columns are not in your data: {col_diff}'
+                if len(col_diff)!=0:
+                    self.__logger.warning(msg)
                 with mp.Pool(workers_num) as worker_pool:
                     results = worker_pool.starmap(
-                        _execute_write_parallel,
-                        [(self.neo_info, database,query, d.fillna(value='').to_dict('records'),
+                        self._execute_write_parallel,
+                        [(self.neo_info, database,query, data.iloc[d].to_dict('records'),
                           params) for d in data_chunks]
                     )
         else:
+            msg = f'Loading {partitions:,.0f} data chunks sequentially'
+            self.__logger.info(msg)
             results = []
             with _get_driver(self.neo_info) as driver:
                 with _get_session(driver, database) as session:
-                    for rows in data_chunks:
-                        rows_dict = {'rows': rows.fillna(value="").to_dict('records')}
+                    for idx in data_chunks:
+                        rows_dict = {'rows': data.iloc[idx].to_dict('records')}
                         for query in queries:
-                            results.append(_execute_write(session,
+                            col_diff = get_columns_diff(query,data.columns)
+                            msg = f'This columns are not in your data: {col_diff}'
+                            if len(col_diff)!=0:
+                                self.__logger.warning(msg)
+                            results.append(self._execute_write(session,
                                                           query,
                                                           rows_dict['rows'],
                                                           params))
@@ -369,12 +343,14 @@ class Neo4jInstance:
                     self.__results[key] += value
         results = dict(self.__results.copy())
         self.__results.clear()
+        msg = f'Completed load with results {results}'
+        self.__logger.info(msg)
         return results
 
     def execute_write_query_with_data(self,
                                       query: str, data: DataFrame,
                                       database: Optional[str] = None,
-                                      partitions: Optional[int] = 1,
+                                      batchSize: Optional[int] = 100_000,
                                       parallel: Optional[bool] = False,
                                       workers: Optional[int] = None,
                                       parameters: Optional[Dict[str, Any]] = None
@@ -390,8 +366,8 @@ class Neo4jInstance:
             database : str, optional
                 Name of the Neo4j database of which to execute the transaction.
                 If not provided the default database is going to be use.
-            partitions : int, optional
-                The number of partitions in which to split the data frame.
+            batchSize : int, optional
+                The number of records per batch or partitions of the data frame.
             parallel : bool, optional
                 Wheather to execute the load in parallel.
             workers : int, optional
@@ -420,7 +396,7 @@ class Neo4jInstance:
         else:
             params = {}
         result = self.execute_write_queries_with_data(
-            [query], data,database, partitions, parallel, workers, params)
+            [query], data,database, batchSize, parallel, workers, params)
         return result
 
     def get_node_label_freq(self, database: Optional[str] = None) -> DataFrame:
@@ -806,3 +782,55 @@ class Neo4jInstance:
             rela_parts = relationship.split("|")
             network.add_edge(rela_parts[0],rela_parts[2],title=rela_parts[1])
         return network
+
+    def _execute_write_parallel(self, neo_info, database, query,
+                       rows: Optional[Dict[str, Any]] = None,
+                       parameters: Optional[Dict[str, Any]] = None
+                      ):
+        if parameters:
+            params = parameters
+        else:
+            params = {}
+        with _get_driver(neo_info) as driver:
+            with _get_session(driver, database) as session:
+                try:
+                    if rows:
+                        results = session.execute_write(
+                            _write_transaction_function, query,
+                            rows = rows, **params).counters.__dict__
+                    else:
+                        results = session.execute_write(
+                            _write_transaction_function, query,
+                            **params).counters.__dict__
+                except ServiceUnavailable as exception:
+                    raise ServiceUnavailable() from exception
+                except ClientError as exception:
+                    raise ClientError() from exception
+        msg = f'Successfully loaded partition with resulting in {results}'
+        self.__logger.info(msg)
+        return results
+
+    def _execute_write(self, session, query,
+                       rows: Optional[Dict[str, Any]] = None,
+                       parameters: Optional[Dict[str, Any]] = None
+                      ):
+        if parameters:
+            params = parameters
+        else:
+            params = {}
+        try:
+            if rows:
+                results = session.execute_write(
+                    _write_transaction_function, query,
+                    rows = rows, **params).counters.__dict__
+            else:
+                results = session.execute_write(
+                    _write_transaction_function, query,
+                    **params).counters.__dict__
+        except ServiceUnavailable as exception:
+            raise ServiceUnavailable() from exception
+        except ClientError as exception:
+            raise ClientError() from exception
+        msg = f'Successfully loaded partition with resulting in {results}'
+        self.__logger.info(msg)
+        return results

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyneoinstance/util/__init__.py

@@ -1,3 +1,5 @@
 from .functions import get_file_name
 from .functions import is_reachable_url
 from .functions import get_logger
+from .functions import get_batches
+from .functions import get_columns_diff

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyneoinstance/util/functions.py

@@ -4,6 +4,9 @@
 import functools
 import time
 import logging
+import re
+import numpy as np
+from pandas import DataFrame
 from datetime import datetime
 from typing import Tuple, Any, Callable, List
 from urllib.request import urlopen
@@ -94,3 +97,51 @@ def timing(function : Callable) -> Tuple[Any, float]:
         elapsed_time = end - start * 1.0
         return result, elapsed_time
     return wrap
+
+def get_batches(data: DataFrame, batch_size) -> List[List[int]]:
+    """Partition the data indexes in batches of the specified size.
+
+    Parameters
+    ----------
+    data : DataFrame
+        Pandas DataFrame containing the data
+    batch_size : int
+        Number of records per partition
+
+    Returns
+    -------
+    List[List[int, bool]]
+        List containing the list of indexes of the DataFrame per partition.
+    """
+    records = data.shape[0]
+    indexes = data.index
+    partitions = records // batch_size
+    if partitions < 2:
+        batches = [list(indexes)]
+    else:
+        remainder = records % batch_size
+        array = np.array(
+            indexes[:records-remainder]).reshape(partitions,-1)
+        batches = array.tolist()
+        if remainder > 0:
+            batches.append(indexes[-remainder:])
+    return batches
+
+def get_columns_diff(query: str, columns: List[str]) -> List[str]:
+    """Get a list of the columns that a Cypher query is trying to access that are
+    not in the list of columns provided.
+
+    Parameters
+    ----------
+    query : str
+       Cypher query
+    columns : List[str]
+        List containing the column names of a Pandas Data Frame
+    Returns
+    -------
+    List[str]
+       List containing the columns that Cypher is trying to access that are not
+       included in the columns names of the data frame.
+    """
+    cypher_attributes = set([w.split('.')[1] for w in re.findall(r'\brow\.\w+',query)])
+    return [a for a in cypher_attributes if a not in columns]

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0/pyneoinstance.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: pyneoinstance
-Version: 2.0.2
+Version: 3.0.0
 Summary: Handle Neo4j read and write transactions and provide methos for performing graph EDA
 Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
 License: GNU GENERAL PUBLIC LICENSE
@@ -689,6 +689,7 @@ Requires-Dist: neo4j>=5.0.0
 Requires-Dist: pandas
 Requires-Dist: PyYAML
 Requires-Dist: pyvis
+Dynamic: license-file
 
 # PyNeoInstance Package
 
@@ -829,7 +830,7 @@ We can update a Neo4j database with data contained in a Pandas Data Frame using
 
 - _data_: Pandas Data Frame containing data to process.
 - _database_: Optional, database on which to execute the queries.
-- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
+- _batchSize_: Optional, number of records per batch use to break the Data Frame in smaller chucks.
 - _parallel_: Whether to execute the query using multiple Python process.
 - _workers_: The number of processes to spawn to execute the queries in parallel.
 - _parameters_: Extra arguments containing optional cypher parameters.
@@ -909,25 +910,25 @@ node_load_queries = config['loading_queries']['nodes']
 person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
 ```
 
-If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
+If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _batchSize_ argument the number of records per data batch to break this Data Frame into smaller data batches. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
 
 ```python
 person_load_result = graph.execute_write_query_with_data(
     node_load_queries['person'],
     person_data,
     db_info['database'],
-    partitions = 5
+    batchSize = 100000
     )
 ```
 ## Updating a database with data in a Data Frame: Parallel  
 
-To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
+To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _batchSize_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
 
 ```python
 person_load_result = graph.execute_write_query_with_data(
     node_load_queries['person'],
     person_data,
-    partitions = 5,
+    batchSize = 100000,
     parallel = True,
     workers = 5
     )

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyneoinstance.egg-info/SOURCES.txt

@@ -19,6 +19,7 @@ pyneoinstance/fileload/loadyaml.py
 pyneoinstance/util/__init__.py
 pyneoinstance/util/functions.py
 test/__init__.py
+test/age_data.csv
 test/context.py
 test/cypher_queries.yaml
 test/load_files.yaml

{pyneoinstance-2.0.2 → pyneoinstance-3.0.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pyneoinstance"
-version = "2.0.2"
+version = "3.0.0"
 description = "Handle Neo4j read and write transactions and provide methos for performing graph EDA"
 readme = "README.md"
 authors = [{ name = "Jose A. Alvarado-Guzman",email = "jose.alvarado-guzman@neo4j.com" }]