pyneoinstance 1.0.0__tar.gz → 1.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pyneoinstance-1.0.0/pyneoinstance.egg-info → pyneoinstance-1.1.2}/PKG-INFO +71 -37
- pyneoinstance-1.1.2/README.md +236 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/__init__.py +1 -1
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/database/neo4jdbms.py +26 -21
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2/pyneoinstance.egg-info}/PKG-INFO +71 -37
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/requires.txt +1 -1
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyproject.toml +3 -3
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/cypher_queries.yaml +11 -8
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/test_neo4jdb.py +12 -2
- pyneoinstance-1.0.0/README.md +0 -205
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/LICENSE +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/MANIFEST.in +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/database/__init__.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/database/context.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/__init__.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/context.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/helpers.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/loadyaml.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/util/__init__.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance/util/functions.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/SOURCES.txt +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/dependency_links.txt +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/top_level.txt +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/setup.cfg +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/setup.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/load_files.yaml +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/movies.csv +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/people.csv +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/roles.csv +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/test.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/test_fileload.py +0 -0
- {pyneoinstance-1.0.0 → pyneoinstance-1.1.2}/test/test_util_functions.py +0 -0
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: pyneoinstance
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.1.2
|
|
4
4
|
Summary: Handle Neo4j read and write transactions
|
|
5
|
-
Author-email: Jose A Alvarado-
|
|
5
|
+
Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
|
|
6
6
|
License: GNU GENERAL PUBLIC LICENSE
|
|
7
7
|
Version 3, 29 June 2007
|
|
8
8
|
|
|
@@ -685,16 +685,22 @@ Classifier: Development Status :: 4 - Beta
|
|
|
685
685
|
Requires-Python: >=3.8
|
|
686
686
|
Description-Content-Type: text/markdown
|
|
687
687
|
License-File: LICENSE
|
|
688
|
+
Requires-Dist: neo4j>=5.0.0
|
|
689
|
+
Requires-Dist: pandas
|
|
690
|
+
Requires-Dist: PyYAML
|
|
688
691
|
|
|
689
692
|
# PyNeoInstance Package
|
|
690
693
|
|
|
691
694
|
## What is it?
|
|
692
|
-
|
|
695
|
+
|
|
696
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
693
697
|
It supports the following features:
|
|
694
698
|
|
|
695
|
-
- __Read query__: Submit a Cypher query and get
|
|
696
|
-
- __Write query__:
|
|
697
|
-
- __Write
|
|
699
|
+
- __Read query__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
700
|
+
- __Write query__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
701
|
+
- __Write queries__: Submit a list of Cypher queries to update an existing database.
|
|
702
|
+
- __Write query with data__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
703
|
+
- __Write queries with data__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
698
704
|
|
|
699
705
|
## How to install it?
|
|
700
706
|
|
|
@@ -706,16 +712,16 @@ pip install pyneoinstance
|
|
|
706
712
|
|
|
707
713
|
### Instantiating the Neo4jInstance class
|
|
708
714
|
|
|
709
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
715
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
710
716
|
|
|
711
717
|
```python
|
|
712
|
-
from
|
|
718
|
+
from pyneoinstance import Neo4jInstance
|
|
713
719
|
```
|
|
714
720
|
|
|
715
721
|
Once imported, you need to instantiate this class by providing the following arguments:
|
|
716
722
|
|
|
717
723
|
- _uri_: The Neo4j instance unique resource identifier.
|
|
718
|
-
- _user_: The user name
|
|
724
|
+
- _user_: The user name used to authenticate.
|
|
719
725
|
- _password_: The user name password.
|
|
720
726
|
|
|
721
727
|
```python
|
|
@@ -736,7 +742,7 @@ user = os.environ['NEO4J_USER']
|
|
|
736
742
|
password = os.environ['NEO4J_PASSWORD']
|
|
737
743
|
graph = Neo4jInstance(uri,user,password)
|
|
738
744
|
```
|
|
739
|
-
Another alternative is to create a YAML
|
|
745
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
740
746
|
|
|
741
747
|
```yaml
|
|
742
748
|
# configuration.yaml
|
|
@@ -744,9 +750,10 @@ db_info:
|
|
|
744
750
|
uri: neo4j_uri
|
|
745
751
|
user: neo4j_user
|
|
746
752
|
password: user_password
|
|
753
|
+
database: neo4j
|
|
747
754
|
```
|
|
748
755
|
|
|
749
|
-
This YAML file can then be loaded in Python and
|
|
756
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
750
757
|
|
|
751
758
|
```python
|
|
752
759
|
from pyneoinstance import load_yaml_file
|
|
@@ -757,7 +764,7 @@ graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
|
757
764
|
```
|
|
758
765
|
### Submitting a Cypher read query
|
|
759
766
|
|
|
760
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method
|
|
767
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
761
768
|
|
|
762
769
|
```python
|
|
763
770
|
query = """
|
|
@@ -767,35 +774,56 @@ query = """
|
|
|
767
774
|
m.tagline AS tagline;
|
|
768
775
|
"""
|
|
769
776
|
|
|
770
|
-
data = graph.execute_read_query(query)
|
|
777
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
771
778
|
```
|
|
779
|
+
|
|
780
|
+
### Submitting a Cypher read query with parameters
|
|
781
|
+
|
|
782
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
783
|
+
|
|
784
|
+
```python
|
|
785
|
+
query = """
|
|
786
|
+
MATCH (m:Movie {title: $movie_title})
|
|
787
|
+
RETURN m.tile AS title,
|
|
788
|
+
m.released AS releasedYear,
|
|
789
|
+
m.tagline AS tagline;
|
|
790
|
+
"""
|
|
791
|
+
|
|
792
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
793
|
+
```
|
|
794
|
+
|
|
772
795
|
### Submitting a Cypher write query
|
|
773
796
|
|
|
774
|
-
|
|
797
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
775
798
|
|
|
776
799
|
```python
|
|
777
800
|
query = """
|
|
778
|
-
MERGE(m:Movie {title
|
|
801
|
+
MERGE(m:Movie {title:$movie_title})
|
|
779
802
|
ON CREATE SET
|
|
780
|
-
m.release=
|
|
781
|
-
m.tagline =
|
|
803
|
+
m.release = $year,
|
|
804
|
+
m.tagline = $tagline
|
|
782
805
|
"""
|
|
783
806
|
|
|
784
|
-
results = graph.execute_write_query(query
|
|
807
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
808
|
+
parameters={
|
|
809
|
+
'movie_title':'Wakanda Forever',
|
|
810
|
+
'year'=2022,
|
|
811
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
812
|
+
})
|
|
785
813
|
```
|
|
786
814
|
|
|
787
815
|
### Updating a database with data in a DataFrame: Sequentially
|
|
788
816
|
|
|
789
|
-
We can update a Neo4j database with data contained in a Pandas
|
|
817
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
790
818
|
|
|
791
|
-
- _data_: Pandas
|
|
792
|
-
- _database_: Optional, database on which to execute the
|
|
793
|
-
- _partitions_: Optional, number of partitions use to break the
|
|
794
|
-
- _parallel_:
|
|
819
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
820
|
+
- _database_: Optional, database on which to execute the queries.
|
|
821
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
822
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
795
823
|
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
796
824
|
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
797
825
|
|
|
798
|
-
To illustrates how to use
|
|
826
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
799
827
|
|
|
800
828
|
```python
|
|
801
829
|
# people.csv
|
|
@@ -811,11 +839,11 @@ personId,name,birthYear,deathYear
|
|
|
811
839
|
3482,Chus Lampreave,1930,
|
|
812
840
|
```
|
|
813
841
|
|
|
814
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to
|
|
842
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
815
843
|
|
|
816
|
-
1. Load the csv to Python as a
|
|
817
|
-
2. Develop the Cypher query that is going to be use to load the
|
|
818
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the
|
|
844
|
+
1. Load the csv to Python as a Data Frame.
|
|
845
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
846
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
819
847
|
|
|
820
848
|
```python
|
|
821
849
|
import pandas as pd
|
|
@@ -833,10 +861,10 @@ person_load_query = """
|
|
|
833
861
|
p.deathYear = toInteger(row.deathYear)
|
|
834
862
|
"""
|
|
835
863
|
|
|
836
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
864
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
837
865
|
```
|
|
838
866
|
|
|
839
|
-
|
|
867
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
840
868
|
|
|
841
869
|
```yaml
|
|
842
870
|
# configuration.yaml
|
|
@@ -844,9 +872,10 @@ db_info:
|
|
|
844
872
|
uri: neo4j_uri
|
|
845
873
|
user: neo4j_user
|
|
846
874
|
password: user_password
|
|
875
|
+
database: neo4j
|
|
847
876
|
loading_queries:
|
|
848
877
|
nodes:
|
|
849
|
-
|
|
878
|
+
Person: |
|
|
850
879
|
WITH $rows AS rows
|
|
851
880
|
UNWIND rows AS row
|
|
852
881
|
MERGE(p:Person {id:row.personId})
|
|
@@ -866,28 +895,33 @@ person_data = pd.read_csv('people.csv')
|
|
|
866
895
|
|
|
867
896
|
node_load_queries = config['loading_queries']['nodes']
|
|
868
897
|
|
|
869
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
898
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
870
899
|
```
|
|
871
900
|
|
|
872
|
-
If the number of rows in your
|
|
901
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
873
902
|
|
|
874
903
|
```python
|
|
875
904
|
person_load_result = graph.execute_write_query_with_data(
|
|
876
|
-
person_data,
|
|
877
905
|
node_load_queries['person'],
|
|
906
|
+
person_data,
|
|
907
|
+
db_info['database'],
|
|
878
908
|
partitions = 5
|
|
879
909
|
)
|
|
880
910
|
```
|
|
881
|
-
## Updating a database with data in a
|
|
911
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
882
912
|
|
|
883
|
-
To load the data in the
|
|
913
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
884
914
|
|
|
885
915
|
```python
|
|
886
916
|
person_load_result = graph.execute_write_query_with_data(
|
|
887
|
-
person_data,
|
|
888
917
|
node_load_queries['person'],
|
|
918
|
+
person_data,
|
|
889
919
|
partitions = 5,
|
|
890
920
|
parallel = True,
|
|
891
921
|
workers = 5
|
|
892
922
|
)
|
|
893
923
|
```
|
|
924
|
+
|
|
925
|
+
## More detailed examples
|
|
926
|
+
|
|
927
|
+
For more detailed examples please referred to this [article](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
# PyNeoInstance Package
|
|
2
|
+
|
|
3
|
+
## What is it?
|
|
4
|
+
|
|
5
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
6
|
+
It supports the following features:
|
|
7
|
+
|
|
8
|
+
- __Read query__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
9
|
+
- __Write query__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
10
|
+
- __Write queries__: Submit a list of Cypher queries to update an existing database.
|
|
11
|
+
- __Write query with data__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
12
|
+
- __Write queries with data__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
13
|
+
|
|
14
|
+
## How to install it?
|
|
15
|
+
|
|
16
|
+
```python
|
|
17
|
+
pip install pyneoinstance
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## How to use it?
|
|
21
|
+
|
|
22
|
+
### Instantiating the Neo4jInstance class
|
|
23
|
+
|
|
24
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
from pyneoinstance import Neo4jInstance
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Once imported, you need to instantiate this class by providing the following arguments:
|
|
31
|
+
|
|
32
|
+
- _uri_: The Neo4j instance unique resource identifier.
|
|
33
|
+
- _user_: The user name used to authenticate.
|
|
34
|
+
- _password_: The user name password.
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
uri = 'bolt://localhost:7687'
|
|
38
|
+
user = 'neo4j'
|
|
39
|
+
password = 'password'
|
|
40
|
+
|
|
41
|
+
graph = Neo4jInstance(uri, user, password)
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Although the previous code is valid, it is not recommended to hard code your user credentials in your application. As an alternative you can store this information as part of your OS user environment variables and then retrieve them in Python.
|
|
45
|
+
|
|
46
|
+
```python
|
|
47
|
+
import os
|
|
48
|
+
|
|
49
|
+
uri = os.environ['NEO4J_URI']
|
|
50
|
+
user = os.environ['NEO4J_USER']
|
|
51
|
+
password = os.environ['NEO4J_PASSWORD']
|
|
52
|
+
graph = Neo4jInstance(uri,user,password)
|
|
53
|
+
```
|
|
54
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
55
|
+
|
|
56
|
+
```yaml
|
|
57
|
+
# configuration.yaml
|
|
58
|
+
db_info:
|
|
59
|
+
uri: neo4j_uri
|
|
60
|
+
user: neo4j_user
|
|
61
|
+
password: user_password
|
|
62
|
+
database: neo4j
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
66
|
+
|
|
67
|
+
```python
|
|
68
|
+
from pyneoinstance import load_yaml_file
|
|
69
|
+
|
|
70
|
+
config = load_yaml_file(configuration.yaml)
|
|
71
|
+
db_info = config['db_info']
|
|
72
|
+
graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
73
|
+
```
|
|
74
|
+
### Submitting a Cypher read query
|
|
75
|
+
|
|
76
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
77
|
+
|
|
78
|
+
```python
|
|
79
|
+
query = """
|
|
80
|
+
MATCH (m:Movie {title: "The Matrix"})
|
|
81
|
+
RETURN m.tile AS title,
|
|
82
|
+
m.released AS releasedYear,
|
|
83
|
+
m.tagline AS tagline;
|
|
84
|
+
"""
|
|
85
|
+
|
|
86
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
### Submitting a Cypher read query with parameters
|
|
90
|
+
|
|
91
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
query = """
|
|
95
|
+
MATCH (m:Movie {title: $movie_title})
|
|
96
|
+
RETURN m.tile AS title,
|
|
97
|
+
m.released AS releasedYear,
|
|
98
|
+
m.tagline AS tagline;
|
|
99
|
+
"""
|
|
100
|
+
|
|
101
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Submitting a Cypher write query
|
|
105
|
+
|
|
106
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
107
|
+
|
|
108
|
+
```python
|
|
109
|
+
query = """
|
|
110
|
+
MERGE(m:Movie {title:$movie_title})
|
|
111
|
+
ON CREATE SET
|
|
112
|
+
m.release = $year,
|
|
113
|
+
m.tagline = $tagline
|
|
114
|
+
"""
|
|
115
|
+
|
|
116
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
117
|
+
parameters={
|
|
118
|
+
'movie_title':'Wakanda Forever',
|
|
119
|
+
'year'=2022,
|
|
120
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
121
|
+
})
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### Updating a database with data in a DataFrame: Sequentially
|
|
125
|
+
|
|
126
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
127
|
+
|
|
128
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
129
|
+
- _database_: Optional, database on which to execute the queries.
|
|
130
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
131
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
132
|
+
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
133
|
+
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
134
|
+
|
|
135
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
136
|
+
|
|
137
|
+
```python
|
|
138
|
+
# people.csv
|
|
139
|
+
personId,name,birthYear,deathYear
|
|
140
|
+
23945,Gérard Pirès,1942,
|
|
141
|
+
553509,Helen Reddy,1941,
|
|
142
|
+
113934,Susan Flannery,1939,
|
|
143
|
+
26706,David Rintoul,1948,
|
|
144
|
+
237537,Rita Marley,1946,
|
|
145
|
+
11502,Harry Davenport,1866,1949
|
|
146
|
+
11792,George Dickerson,1933,
|
|
147
|
+
7242,Temuera Morrison,1960,
|
|
148
|
+
3482,Chus Lampreave,1930,
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
152
|
+
|
|
153
|
+
1. Load the csv to Python as a Data Frame.
|
|
154
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
155
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
156
|
+
|
|
157
|
+
```python
|
|
158
|
+
import pandas as pd
|
|
159
|
+
|
|
160
|
+
person_data = pd.read_csv('people.csv')
|
|
161
|
+
|
|
162
|
+
person_load_query = """
|
|
163
|
+
WITH $rows AS rows
|
|
164
|
+
UNWIND rows AS row
|
|
165
|
+
MERGE(p:Person {id:row.personId})
|
|
166
|
+
ON CREATE
|
|
167
|
+
SET p.id = row.personId,
|
|
168
|
+
p.name = row.name,
|
|
169
|
+
p.birthYear=toInteger(row.birthYear),
|
|
170
|
+
p.deathYear = toInteger(row.deathYear)
|
|
171
|
+
"""
|
|
172
|
+
|
|
173
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
177
|
+
|
|
178
|
+
```yaml
|
|
179
|
+
# configuration.yaml
|
|
180
|
+
db_info:
|
|
181
|
+
uri: neo4j_uri
|
|
182
|
+
user: neo4j_user
|
|
183
|
+
password: user_password
|
|
184
|
+
database: neo4j
|
|
185
|
+
loading_queries:
|
|
186
|
+
nodes:
|
|
187
|
+
Person: |
|
|
188
|
+
WITH $rows AS rows
|
|
189
|
+
UNWIND rows AS row
|
|
190
|
+
MERGE(p:Person {id:row.personId})
|
|
191
|
+
ON CREATE
|
|
192
|
+
SET p.id = row.personId,
|
|
193
|
+
p.name = row.name,
|
|
194
|
+
p.birthYear=toInteger(row.birthYear),
|
|
195
|
+
p.deathYear = toInteger(row.deathYear)
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Now that we added the loading query to our configuration file, lets re-write our Python script to get this query from the configuration file.
|
|
199
|
+
|
|
200
|
+
```python
|
|
201
|
+
import pandas as pd
|
|
202
|
+
|
|
203
|
+
person_data = pd.read_csv('people.csv')
|
|
204
|
+
|
|
205
|
+
node_load_queries = config['loading_queries']['nodes']
|
|
206
|
+
|
|
207
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
211
|
+
|
|
212
|
+
```python
|
|
213
|
+
person_load_result = graph.execute_write_query_with_data(
|
|
214
|
+
node_load_queries['person'],
|
|
215
|
+
person_data,
|
|
216
|
+
db_info['database'],
|
|
217
|
+
partitions = 5
|
|
218
|
+
)
|
|
219
|
+
```
|
|
220
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
221
|
+
|
|
222
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
223
|
+
|
|
224
|
+
```python
|
|
225
|
+
person_load_result = graph.execute_write_query_with_data(
|
|
226
|
+
node_load_queries['person'],
|
|
227
|
+
person_data,
|
|
228
|
+
partitions = 5,
|
|
229
|
+
parallel = True,
|
|
230
|
+
workers = 5
|
|
231
|
+
)
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## More detailed examples
|
|
235
|
+
|
|
236
|
+
For more detailed examples please referred to this [article](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
@@ -20,7 +20,6 @@ from neo4j.exceptions import ServiceUnavailable
|
|
|
20
20
|
from neo4j.exceptions import AuthError
|
|
21
21
|
from neo4j.exceptions import ClientError
|
|
22
22
|
from neo4j.exceptions import ConfigurationError
|
|
23
|
-
from .context import get_logger
|
|
24
23
|
|
|
25
24
|
def _get_driver(neo_info : Dict[str, str]):
|
|
26
25
|
try:
|
|
@@ -34,7 +33,8 @@ def _get_driver(neo_info : Dict[str, str]):
|
|
|
34
33
|
neo_info['uri'],
|
|
35
34
|
auth=(neo_info['user'], neo_info['password']))
|
|
36
35
|
except ServiceUnavailable as exception:
|
|
37
|
-
raise ServiceUnavailable()
|
|
36
|
+
raise ServiceUnavailable()
|
|
37
|
+
sys.exit()
|
|
38
38
|
except AuthError as exception:
|
|
39
39
|
raise AuthError() from exception
|
|
40
40
|
except ConfigurationError as exception:
|
|
@@ -67,14 +67,14 @@ def _execute_write(session, query,
|
|
|
67
67
|
params = {}
|
|
68
68
|
try:
|
|
69
69
|
if rows:
|
|
70
|
-
results = session.
|
|
70
|
+
results = session.execute_write(
|
|
71
71
|
_write_transaction_function, query,
|
|
72
72
|
rows = rows, **params).counters.__dict__
|
|
73
73
|
else:
|
|
74
|
-
results = session.
|
|
74
|
+
results = session.execute_write(
|
|
75
75
|
_write_transaction_function, query,
|
|
76
76
|
**params).counters.__dict__
|
|
77
|
-
except ServiceUnavailable
|
|
77
|
+
except ServiceUnavailable as exception:
|
|
78
78
|
raise ServiceUnavailable() from exception
|
|
79
79
|
except ClientError as exception:
|
|
80
80
|
raise ClientError() from exception
|
|
@@ -92,14 +92,14 @@ def _execute_write_parallel(neo_info, database, query,
|
|
|
92
92
|
with _get_session(driver, database) as session:
|
|
93
93
|
try:
|
|
94
94
|
if rows:
|
|
95
|
-
results = session.
|
|
95
|
+
results = session.execute_write(
|
|
96
96
|
_write_transaction_function, query,
|
|
97
97
|
rows = rows, **params).counters.__dict__
|
|
98
98
|
else:
|
|
99
|
-
results = session.
|
|
99
|
+
results = session.execute_write(
|
|
100
100
|
_write_transaction_function, query,
|
|
101
101
|
**params).counters.__dict__
|
|
102
|
-
except ServiceUnavailable
|
|
102
|
+
except ServiceUnavailable as exception:
|
|
103
103
|
raise ServiceUnavailable() from exception
|
|
104
104
|
except ClientError as exception:
|
|
105
105
|
raise ClientError() from exception
|
|
@@ -167,12 +167,11 @@ class Neo4jInstance:
|
|
|
167
167
|
self.neo_info['password'] = password
|
|
168
168
|
self.neo_info['encrypted'] = kwargs.get('encrypted') or ''
|
|
169
169
|
self.__results = defaultdict(int)
|
|
170
|
-
self.__logger = get_logger('pyneoingest', logging.INFO)
|
|
171
170
|
|
|
172
171
|
|
|
173
172
|
def execute_read_query(self, query: str,
|
|
174
173
|
database: Optional[str] = None,
|
|
175
|
-
|
|
174
|
+
parameters: Optional[Dict[str, Any]] = None
|
|
176
175
|
) -> DataFrame:
|
|
177
176
|
"""Execute a read transaction to a specific database.
|
|
178
177
|
|
|
@@ -184,7 +183,7 @@ class Neo4jInstance:
|
|
|
184
183
|
Name of the Neo4j database of which to execute the
|
|
185
184
|
transaction. If not provided the default database is going to
|
|
186
185
|
be use.
|
|
187
|
-
|
|
186
|
+
parameters: Dict[str, Any], optional
|
|
188
187
|
Extra arguments containing optional cypher parameters.
|
|
189
188
|
|
|
190
189
|
Return
|
|
@@ -199,11 +198,15 @@ class Neo4jInstance:
|
|
|
199
198
|
ClientError
|
|
200
199
|
When there is a Cypher syntax error or datatype error.
|
|
201
200
|
"""
|
|
201
|
+
if parameters:
|
|
202
|
+
params = parameters
|
|
203
|
+
else:
|
|
204
|
+
params = {}
|
|
202
205
|
with _get_driver(self.neo_info) as driver:
|
|
203
206
|
with _get_session(driver, database) as session:
|
|
204
207
|
try:
|
|
205
|
-
result = session.
|
|
206
|
-
_read_transaction_function,query=query,**
|
|
208
|
+
result = session.execute_read(
|
|
209
|
+
_read_transaction_function,query=query,**params)
|
|
207
210
|
except ServiceUnavailable as exception:
|
|
208
211
|
raise ServiceUnavailable() from exception
|
|
209
212
|
except ClientError as exception:
|
|
@@ -212,7 +215,7 @@ class Neo4jInstance:
|
|
|
212
215
|
|
|
213
216
|
def execute_write_queries(self, queries: List[str],
|
|
214
217
|
database: Optional[str] = None,
|
|
215
|
-
|
|
218
|
+
parameters: Optional[Dict[str, Any]] = None
|
|
216
219
|
) -> Dict[str, int]:
|
|
217
220
|
"""Execute a write query to a specific database.
|
|
218
221
|
|
|
@@ -223,7 +226,7 @@ class Neo4jInstance:
|
|
|
223
226
|
database : str, optional
|
|
224
227
|
Name of the Neo4j database of which to execute the transaction.
|
|
225
228
|
If not provided the default database is going to be use.
|
|
226
|
-
|
|
229
|
+
parameters : Dict[str, Any], optional
|
|
227
230
|
Extra arguments containing optional cypher parameters.
|
|
228
231
|
|
|
229
232
|
Returns
|
|
@@ -239,20 +242,23 @@ class Neo4jInstance:
|
|
|
239
242
|
ClientError
|
|
240
243
|
When their is a Cypher syntax error or datatype error.
|
|
241
244
|
"""
|
|
245
|
+
if parameters:
|
|
246
|
+
params = parameters
|
|
247
|
+
else:
|
|
248
|
+
params = {}
|
|
242
249
|
results = defaultdict(int)
|
|
243
250
|
with _get_driver(self.neo_info) as driver:
|
|
244
251
|
with _get_session(driver, database) as session:
|
|
245
252
|
for query in queries:
|
|
246
|
-
result = _execute_write(session, query,
|
|
253
|
+
result = _execute_write(session, query, parameters=params)
|
|
247
254
|
for key, value in result.items():
|
|
248
255
|
if key != '_contains_updates':
|
|
249
256
|
results[key] += value
|
|
250
|
-
self.__logger.info(f'Loading stats: {dict(results)}')
|
|
251
257
|
return dict(results)
|
|
252
258
|
|
|
253
259
|
def execute_write_query(self, query: str,
|
|
254
260
|
database: Optional[str] = None,
|
|
255
|
-
|
|
261
|
+
parameters: Optional[Dict['str',Any]] = None
|
|
256
262
|
) -> Dict[str, Any]:
|
|
257
263
|
"""Execute a write query to a specific database.
|
|
258
264
|
|
|
@@ -263,7 +269,7 @@ class Neo4jInstance:
|
|
|
263
269
|
database : str, optional
|
|
264
270
|
Name of the Neo4j database of which to execute the transaction.
|
|
265
271
|
If not provided the default database is going to be use.
|
|
266
|
-
|
|
272
|
+
parameters : Dict[str, Any], optional
|
|
267
273
|
Extra arguments containing optional cypher parameters.
|
|
268
274
|
|
|
269
275
|
Returns
|
|
@@ -279,7 +285,7 @@ class Neo4jInstance:
|
|
|
279
285
|
ClientError
|
|
280
286
|
When their is a Cypher syntax error or datatype error.
|
|
281
287
|
"""
|
|
282
|
-
result = self.execute_write_queries([query], database,
|
|
288
|
+
result = self.execute_write_queries([query], database, parameters)
|
|
283
289
|
return result
|
|
284
290
|
|
|
285
291
|
def execute_write_queries_with_data(self, queries: List[str], data: DataFrame,
|
|
@@ -362,7 +368,6 @@ class Neo4jInstance:
|
|
|
362
368
|
self.__results[key] += value
|
|
363
369
|
results = dict(self.__results.copy())
|
|
364
370
|
self.__results.clear()
|
|
365
|
-
self.__logger.info(f'Loading stats: {results}')
|
|
366
371
|
return results
|
|
367
372
|
|
|
368
373
|
def execute_write_query_with_data(self,
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: pyneoinstance
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.1.2
|
|
4
4
|
Summary: Handle Neo4j read and write transactions
|
|
5
|
-
Author-email: Jose A Alvarado-
|
|
5
|
+
Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
|
|
6
6
|
License: GNU GENERAL PUBLIC LICENSE
|
|
7
7
|
Version 3, 29 June 2007
|
|
8
8
|
|
|
@@ -685,16 +685,22 @@ Classifier: Development Status :: 4 - Beta
|
|
|
685
685
|
Requires-Python: >=3.8
|
|
686
686
|
Description-Content-Type: text/markdown
|
|
687
687
|
License-File: LICENSE
|
|
688
|
+
Requires-Dist: neo4j>=5.0.0
|
|
689
|
+
Requires-Dist: pandas
|
|
690
|
+
Requires-Dist: PyYAML
|
|
688
691
|
|
|
689
692
|
# PyNeoInstance Package
|
|
690
693
|
|
|
691
694
|
## What is it?
|
|
692
|
-
|
|
695
|
+
|
|
696
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
693
697
|
It supports the following features:
|
|
694
698
|
|
|
695
|
-
- __Read query__: Submit a Cypher query and get
|
|
696
|
-
- __Write query__:
|
|
697
|
-
- __Write
|
|
699
|
+
- __Read query__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
700
|
+
- __Write query__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
701
|
+
- __Write queries__: Submit a list of Cypher queries to update an existing database.
|
|
702
|
+
- __Write query with data__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
703
|
+
- __Write queries with data__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
698
704
|
|
|
699
705
|
## How to install it?
|
|
700
706
|
|
|
@@ -706,16 +712,16 @@ pip install pyneoinstance
|
|
|
706
712
|
|
|
707
713
|
### Instantiating the Neo4jInstance class
|
|
708
714
|
|
|
709
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
715
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
710
716
|
|
|
711
717
|
```python
|
|
712
|
-
from
|
|
718
|
+
from pyneoinstance import Neo4jInstance
|
|
713
719
|
```
|
|
714
720
|
|
|
715
721
|
Once imported, you need to instantiate this class by providing the following arguments:
|
|
716
722
|
|
|
717
723
|
- _uri_: The Neo4j instance unique resource identifier.
|
|
718
|
-
- _user_: The user name
|
|
724
|
+
- _user_: The user name used to authenticate.
|
|
719
725
|
- _password_: The user name password.
|
|
720
726
|
|
|
721
727
|
```python
|
|
@@ -736,7 +742,7 @@ user = os.environ['NEO4J_USER']
|
|
|
736
742
|
password = os.environ['NEO4J_PASSWORD']
|
|
737
743
|
graph = Neo4jInstance(uri,user,password)
|
|
738
744
|
```
|
|
739
|
-
Another alternative is to create a YAML
|
|
745
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
740
746
|
|
|
741
747
|
```yaml
|
|
742
748
|
# configuration.yaml
|
|
@@ -744,9 +750,10 @@ db_info:
|
|
|
744
750
|
uri: neo4j_uri
|
|
745
751
|
user: neo4j_user
|
|
746
752
|
password: user_password
|
|
753
|
+
database: neo4j
|
|
747
754
|
```
|
|
748
755
|
|
|
749
|
-
This YAML file can then be loaded in Python and
|
|
756
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
750
757
|
|
|
751
758
|
```python
|
|
752
759
|
from pyneoinstance import load_yaml_file
|
|
@@ -757,7 +764,7 @@ graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
|
757
764
|
```
|
|
758
765
|
### Submitting a Cypher read query
|
|
759
766
|
|
|
760
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method
|
|
767
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
761
768
|
|
|
762
769
|
```python
|
|
763
770
|
query = """
|
|
@@ -767,35 +774,56 @@ query = """
|
|
|
767
774
|
m.tagline AS tagline;
|
|
768
775
|
"""
|
|
769
776
|
|
|
770
|
-
data = graph.execute_read_query(query)
|
|
777
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
771
778
|
```
|
|
779
|
+
|
|
780
|
+
### Submitting a Cypher read query with parameters
|
|
781
|
+
|
|
782
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
783
|
+
|
|
784
|
+
```python
|
|
785
|
+
query = """
|
|
786
|
+
MATCH (m:Movie {title: $movie_title})
|
|
787
|
+
RETURN m.tile AS title,
|
|
788
|
+
m.released AS releasedYear,
|
|
789
|
+
m.tagline AS tagline;
|
|
790
|
+
"""
|
|
791
|
+
|
|
792
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
793
|
+
```
|
|
794
|
+
|
|
772
795
|
### Submitting a Cypher write query
|
|
773
796
|
|
|
774
|
-
|
|
797
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
775
798
|
|
|
776
799
|
```python
|
|
777
800
|
query = """
|
|
778
|
-
MERGE(m:Movie {title
|
|
801
|
+
MERGE(m:Movie {title:$movie_title})
|
|
779
802
|
ON CREATE SET
|
|
780
|
-
m.release=
|
|
781
|
-
m.tagline =
|
|
803
|
+
m.release = $year,
|
|
804
|
+
m.tagline = $tagline
|
|
782
805
|
"""
|
|
783
806
|
|
|
784
|
-
results = graph.execute_write_query(query
|
|
807
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
808
|
+
parameters={
|
|
809
|
+
'movie_title':'Wakanda Forever',
|
|
810
|
+
'year'=2022,
|
|
811
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
812
|
+
})
|
|
785
813
|
```
|
|
786
814
|
|
|
787
815
|
### Updating a database with data in a DataFrame: Sequentially
|
|
788
816
|
|
|
789
|
-
We can update a Neo4j database with data contained in a Pandas
|
|
817
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
790
818
|
|
|
791
|
-
- _data_: Pandas
|
|
792
|
-
- _database_: Optional, database on which to execute the
|
|
793
|
-
- _partitions_: Optional, number of partitions use to break the
|
|
794
|
-
- _parallel_:
|
|
819
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
820
|
+
- _database_: Optional, database on which to execute the queries.
|
|
821
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
822
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
795
823
|
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
796
824
|
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
797
825
|
|
|
798
|
-
To illustrates how to use
|
|
826
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
799
827
|
|
|
800
828
|
```python
|
|
801
829
|
# people.csv
|
|
@@ -811,11 +839,11 @@ personId,name,birthYear,deathYear
|
|
|
811
839
|
3482,Chus Lampreave,1930,
|
|
812
840
|
```
|
|
813
841
|
|
|
814
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to
|
|
842
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
815
843
|
|
|
816
|
-
1. Load the csv to Python as a
|
|
817
|
-
2. Develop the Cypher query that is going to be use to load the
|
|
818
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the
|
|
844
|
+
1. Load the csv to Python as a Data Frame.
|
|
845
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
846
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
819
847
|
|
|
820
848
|
```python
|
|
821
849
|
import pandas as pd
|
|
@@ -833,10 +861,10 @@ person_load_query = """
|
|
|
833
861
|
p.deathYear = toInteger(row.deathYear)
|
|
834
862
|
"""
|
|
835
863
|
|
|
836
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
864
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
837
865
|
```
|
|
838
866
|
|
|
839
|
-
|
|
867
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
840
868
|
|
|
841
869
|
```yaml
|
|
842
870
|
# configuration.yaml
|
|
@@ -844,9 +872,10 @@ db_info:
|
|
|
844
872
|
uri: neo4j_uri
|
|
845
873
|
user: neo4j_user
|
|
846
874
|
password: user_password
|
|
875
|
+
database: neo4j
|
|
847
876
|
loading_queries:
|
|
848
877
|
nodes:
|
|
849
|
-
|
|
878
|
+
Person: |
|
|
850
879
|
WITH $rows AS rows
|
|
851
880
|
UNWIND rows AS row
|
|
852
881
|
MERGE(p:Person {id:row.personId})
|
|
@@ -866,28 +895,33 @@ person_data = pd.read_csv('people.csv')
|
|
|
866
895
|
|
|
867
896
|
node_load_queries = config['loading_queries']['nodes']
|
|
868
897
|
|
|
869
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
898
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
870
899
|
```
|
|
871
900
|
|
|
872
|
-
If the number of rows in your
|
|
901
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
873
902
|
|
|
874
903
|
```python
|
|
875
904
|
person_load_result = graph.execute_write_query_with_data(
|
|
876
|
-
person_data,
|
|
877
905
|
node_load_queries['person'],
|
|
906
|
+
person_data,
|
|
907
|
+
db_info['database'],
|
|
878
908
|
partitions = 5
|
|
879
909
|
)
|
|
880
910
|
```
|
|
881
|
-
## Updating a database with data in a
|
|
911
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
882
912
|
|
|
883
|
-
To load the data in the
|
|
913
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
884
914
|
|
|
885
915
|
```python
|
|
886
916
|
person_load_result = graph.execute_write_query_with_data(
|
|
887
|
-
person_data,
|
|
888
917
|
node_load_queries['person'],
|
|
918
|
+
person_data,
|
|
889
919
|
partitions = 5,
|
|
890
920
|
parallel = True,
|
|
891
921
|
workers = 5
|
|
892
922
|
)
|
|
893
923
|
```
|
|
924
|
+
|
|
925
|
+
## More detailed examples
|
|
926
|
+
|
|
927
|
+
For more detailed examples please referred to this [article](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
@@ -6,10 +6,10 @@ build-backend = "setuptools.build_meta"
|
|
|
6
6
|
|
|
7
7
|
[project]
|
|
8
8
|
name = "pyneoinstance"
|
|
9
|
-
version = "1.
|
|
9
|
+
version = "1.1.2"
|
|
10
10
|
description = "Handle Neo4j read and write transactions"
|
|
11
11
|
readme = "README.md"
|
|
12
|
-
authors = [{ name = "Jose A Alvarado-
|
|
12
|
+
authors = [{ name = "Jose A. Alvarado-Guzman",email = "jose.alvarado-guzman@neo4j.com" }]
|
|
13
13
|
license = { file = "LICENSE" }
|
|
14
14
|
classifiers = [
|
|
15
15
|
"License :: OSI Approved :: GNU General Public License (GPL)",
|
|
@@ -18,7 +18,7 @@ classifiers = [
|
|
|
18
18
|
]
|
|
19
19
|
keywords = ["neo4j", "graph", "cypher"]
|
|
20
20
|
dependencies = [
|
|
21
|
-
"neo4j
|
|
21
|
+
"neo4j>=5.0.0",
|
|
22
22
|
"pandas",
|
|
23
23
|
"PyYAML"
|
|
24
24
|
]
|
|
@@ -1,11 +1,10 @@
|
|
|
1
1
|
database: test
|
|
2
2
|
pre_ingest:
|
|
3
|
-
- CREATE CONSTRAINT nkPerson IF NOT EXISTS
|
|
4
|
-
- CREATE CONSTRAINT nkMovie IF NOT EXISTS
|
|
3
|
+
- CREATE CONSTRAINT nkPerson IF NOT EXISTS FOR(n:Person) REQUIRE (n.personId) IS NODE KEY;
|
|
4
|
+
- CREATE CONSTRAINT nkMovie IF NOT EXISTS FOR(n:Movie) REQUIRE (n.movieId) IS NODE KEY;
|
|
5
5
|
|
|
6
6
|
queries:
|
|
7
|
-
load_person:
|
|
8
|
-
|
|
|
7
|
+
load_person: |
|
|
9
8
|
WITH $rows AS rows
|
|
10
9
|
UNWIND rows AS row
|
|
11
10
|
MERGE(p:Person {personId:row.personId})
|
|
@@ -13,8 +12,7 @@ queries:
|
|
|
13
12
|
SET p.personId=row.personId,
|
|
14
13
|
p.name=row.name,
|
|
15
14
|
p.birthYear=toInteger(row.birthYear)
|
|
16
|
-
load_movie:
|
|
17
|
-
|
|
|
15
|
+
load_movie: |
|
|
18
16
|
WITH $rows AS rows
|
|
19
17
|
UNWIND rows AS row
|
|
20
18
|
MERGE(m:Movie {movieId:row.movieId})
|
|
@@ -24,11 +22,16 @@ queries:
|
|
|
24
22
|
m.averageVote=toFloat(row.avgVote),
|
|
25
23
|
m.releaseYear=toInteger(row.releaseYear),
|
|
26
24
|
m.genres=split(row.genres,':')
|
|
27
|
-
load_role:
|
|
28
|
-
|
|
|
25
|
+
load_role: |
|
|
29
26
|
WITH $rows AS rows
|
|
30
27
|
UNWIND rows AS row
|
|
31
28
|
MATCH(p:Person {personId:row.personId}),(m:Movie {movieId:row.movieId})
|
|
32
29
|
MERGE(p)-[r:ACTED_IN]->(m)
|
|
33
30
|
ON CREATE
|
|
34
31
|
SET r.character=split(row.character,':')
|
|
32
|
+
create_actor: |
|
|
33
|
+
MERGE(p:Person:Actor {personId:$id})
|
|
34
|
+
ON CREATE
|
|
35
|
+
SET p.personId=$id,
|
|
36
|
+
p.name=$name,
|
|
37
|
+
p.birthYear=$year
|
|
@@ -93,11 +93,21 @@ class TestNeo4jInstance(unittest.TestCase):
|
|
|
93
93
|
# Test the execute_read_query_using cypher parameters
|
|
94
94
|
movie = "The Matrix"
|
|
95
95
|
query = "MATCH(m:Movie) WHERE m.title=$movie_name RETURN m.title AS movie"
|
|
96
|
-
print('Parameters')
|
|
96
|
+
print('Read Parameters')
|
|
97
97
|
result = self.graph.execute_read_query(query,
|
|
98
98
|
self.queries['database'],
|
|
99
|
-
movie_name
|
|
99
|
+
{'movie_name':movie})
|
|
100
100
|
self.assertEqual(movie,result['movie'][0])
|
|
101
101
|
|
|
102
|
+
# Test the execute_write_query_using cypher parameters
|
|
103
|
+
print('Write Parameters')
|
|
104
|
+
result = self.graph.execute_write_query(
|
|
105
|
+
self.queries['queries']['create_actor'],
|
|
106
|
+
self.queries['database'],
|
|
107
|
+
parameters={'id':-1,'name':'Darth Vader','year':'1977'})
|
|
108
|
+
solution = {'nodes_created':1,'labels_added':2,
|
|
109
|
+
'properties_set':4}
|
|
110
|
+
self.assertEqual(solution, result)
|
|
111
|
+
|
|
102
112
|
if __name__ == '__main__':
|
|
103
113
|
unittest.main()
|
pyneoinstance-1.0.0/README.md
DELETED
|
@@ -1,205 +0,0 @@
|
|
|
1
|
-
# PyNeoInstance Package
|
|
2
|
-
|
|
3
|
-
## What is it?
|
|
4
|
-
Is a Python package that provides a user friendly interface for submitting Cypher queries to Neo4j 4.x.
|
|
5
|
-
It supports the following features:
|
|
6
|
-
|
|
7
|
-
- __Read query__: Submit a Cypher query and get the result of it as a Pandas DataFrame.
|
|
8
|
-
- __Write query__: Sumbmit a Cypher query to update an existing database.
|
|
9
|
-
- __Write query with data__: Update a database based on a Pandas DataFrame and one or more Cypher queries.
|
|
10
|
-
|
|
11
|
-
## How to install it?
|
|
12
|
-
|
|
13
|
-
```python
|
|
14
|
-
pip install pyneoinstance
|
|
15
|
-
```
|
|
16
|
-
|
|
17
|
-
## How to use it?
|
|
18
|
-
|
|
19
|
-
### Instantiating the Neo4jInstance class
|
|
20
|
-
|
|
21
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
22
|
-
|
|
23
|
-
```python
|
|
24
|
-
from pyneoingest import Neo4jInstance
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
Once imported, you need to instantiate this class by providing the following arguments:
|
|
28
|
-
|
|
29
|
-
- _uri_: The Neo4j instance unique resource identifier.
|
|
30
|
-
- _user_: The user name use to authenticate.
|
|
31
|
-
- _password_: The user name password.
|
|
32
|
-
|
|
33
|
-
```python
|
|
34
|
-
uri = 'bolt://localhost:7687'
|
|
35
|
-
user = 'neo4j'
|
|
36
|
-
password = 'password'
|
|
37
|
-
|
|
38
|
-
graph = Neo4jInstance(uri, user, password)
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
Although the previous code is valid, it is not recommended to hard code your user credentials in your application. As an alternative you can store this information as part of your OS user environment variables and then retrieve them in Python.
|
|
42
|
-
|
|
43
|
-
```python
|
|
44
|
-
import os
|
|
45
|
-
|
|
46
|
-
uri = os.environ['NEO4J_URI']
|
|
47
|
-
user = os.environ['NEO4J_USER']
|
|
48
|
-
password = os.environ['NEO4J_PASSWORD']
|
|
49
|
-
graph = Neo4jInstance(uri,user,password)
|
|
50
|
-
```
|
|
51
|
-
Another alternative is to create a YAML configutation file to store this infromation.
|
|
52
|
-
|
|
53
|
-
```yaml
|
|
54
|
-
# configuration.yaml
|
|
55
|
-
db_info:
|
|
56
|
-
uri: neo4j_uri
|
|
57
|
-
user: neo4j_user
|
|
58
|
-
password: user_password
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
This YAML file can then be loaded in Python and these information can be retrieve.
|
|
62
|
-
|
|
63
|
-
```python
|
|
64
|
-
from pyneoinstance import load_yaml_file
|
|
65
|
-
|
|
66
|
-
config = load_yaml_file(configuration.yaml)
|
|
67
|
-
db_info = config['db_info']
|
|
68
|
-
graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
69
|
-
```
|
|
70
|
-
### Submitting a Cypher read query
|
|
71
|
-
|
|
72
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receive the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query is return as a a Pandas DataFrame. To avoid unwanted column names in the resulting DataFrame, always provide aliases for all return properties.
|
|
73
|
-
|
|
74
|
-
```python
|
|
75
|
-
query = """
|
|
76
|
-
MATCH (m:Movie {title: "The Matrix"})
|
|
77
|
-
RETURN m.tile AS title,
|
|
78
|
-
m.released AS releasedYear,
|
|
79
|
-
m.tagline AS tagline;
|
|
80
|
-
"""
|
|
81
|
-
|
|
82
|
-
data = graph.execute_read_query(query)
|
|
83
|
-
```
|
|
84
|
-
### Submitting a Cypher write query
|
|
85
|
-
|
|
86
|
-
We submit a Cypher write query using the method execute\_write\_query. The signature of this method is identical to the execute\_read\_query. This method returns a dictionary containing the updates made to the database.
|
|
87
|
-
|
|
88
|
-
```python
|
|
89
|
-
query = """
|
|
90
|
-
MERGE(m:Movie {title:'Wakanda Forever'})
|
|
91
|
-
ON CREATE SET
|
|
92
|
-
m.release=2022,
|
|
93
|
-
m.tagline = 'Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!'
|
|
94
|
-
"""
|
|
95
|
-
|
|
96
|
-
results = graph.execute_write_query(query)
|
|
97
|
-
```
|
|
98
|
-
|
|
99
|
-
### Updating a database with data in a DataFrame: Sequentially
|
|
100
|
-
|
|
101
|
-
We can update a Neo4j database with data contained in a Pandas DataFrame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methos return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
102
|
-
|
|
103
|
-
- _data_: Pandas DataFrame containing data to process.
|
|
104
|
-
- _database_: Optional, database on which to execute the quieries.
|
|
105
|
-
- _partitions_: Optional, number of partitions use to break the DataFrame in smaller chucks.
|
|
106
|
-
- _parallel_: Wheather to execute the query using multiple Python process.
|
|
107
|
-
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
108
|
-
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
109
|
-
|
|
110
|
-
To illustrates how to use this methods lets imagine we have a csv file that looks like this:
|
|
111
|
-
|
|
112
|
-
```python
|
|
113
|
-
# people.csv
|
|
114
|
-
personId,name,birthYear,deathYear
|
|
115
|
-
23945,Gérard Pirès,1942,
|
|
116
|
-
553509,Helen Reddy,1941,
|
|
117
|
-
113934,Susan Flannery,1939,
|
|
118
|
-
26706,David Rintoul,1948,
|
|
119
|
-
237537,Rita Marley,1946,
|
|
120
|
-
11502,Harry Davenport,1866,1949
|
|
121
|
-
11792,George Dickerson,1933,
|
|
122
|
-
7242,Temuera Morrison,1960,
|
|
123
|
-
3482,Chus Lampreave,1930,
|
|
124
|
-
```
|
|
125
|
-
|
|
126
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performe the following steps:
|
|
127
|
-
|
|
128
|
-
1. Load the csv to Python as a DataFrame.
|
|
129
|
-
2. Develop the Cypher query that is going to be use to load the DataFrame into Neo4j.
|
|
130
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the DataFrame, Cypher query and any other optinal arguments to this method.
|
|
131
|
-
|
|
132
|
-
```python
|
|
133
|
-
import pandas as pd
|
|
134
|
-
|
|
135
|
-
person_data = pd.read_csv('people.csv')
|
|
136
|
-
|
|
137
|
-
person_load_query = """
|
|
138
|
-
WITH $rows AS rows
|
|
139
|
-
UNWIND rows AS row
|
|
140
|
-
MERGE(p:Person {id:row.personId})
|
|
141
|
-
ON CREATE
|
|
142
|
-
SET p.id = row.personId,
|
|
143
|
-
p.name = row.name,
|
|
144
|
-
p.birthYear=toInteger(row.birthYear),
|
|
145
|
-
p.deathYear = toInteger(row.deathYear)
|
|
146
|
-
"""
|
|
147
|
-
|
|
148
|
-
person_load_result = graph.execute_write_query_with_data(person_data, person_load_query)
|
|
149
|
-
```
|
|
150
|
-
|
|
151
|
-
Is not a good practice to store your Cypher queries as strings within your Python modules. To solve this we can extend our YAML configuration file to store all our Cypher queries.
|
|
152
|
-
|
|
153
|
-
```yaml
|
|
154
|
-
# configuration.yaml
|
|
155
|
-
db_info:
|
|
156
|
-
uri: neo4j_uri
|
|
157
|
-
user: neo4j_user
|
|
158
|
-
password: user_password
|
|
159
|
-
loading_queries:
|
|
160
|
-
nodes:
|
|
161
|
-
person: |
|
|
162
|
-
WITH $rows AS rows
|
|
163
|
-
UNWIND rows AS row
|
|
164
|
-
MERGE(p:Person {id:row.personId})
|
|
165
|
-
ON CREATE
|
|
166
|
-
SET p.id = row.personId,
|
|
167
|
-
p.name = row.name,
|
|
168
|
-
p.birthYear=toInteger(row.birthYear),
|
|
169
|
-
p.deathYear = toInteger(row.deathYear)
|
|
170
|
-
```
|
|
171
|
-
|
|
172
|
-
Now that we added the loading query to our configuration file, lets re-write our Python script to get this query from the configuration file.
|
|
173
|
-
|
|
174
|
-
```python
|
|
175
|
-
import pandas as pd
|
|
176
|
-
|
|
177
|
-
person_data = pd.read_csv('people.csv')
|
|
178
|
-
|
|
179
|
-
node_load_queries = config['loading_queries']['nodes']
|
|
180
|
-
|
|
181
|
-
person_load_result = graph.execute_write_query_with_data(person_data, node_load_queries['person'])
|
|
182
|
-
```
|
|
183
|
-
|
|
184
|
-
If the number of rows in your DataFrame is 100K or more it is reccomended to break the DataFrame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this DataFrame into. This partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
185
|
-
|
|
186
|
-
```python
|
|
187
|
-
person_load_result = graph.execute_write_query_with_data(
|
|
188
|
-
person_data,
|
|
189
|
-
node_load_queries['person'],
|
|
190
|
-
partitions = 5
|
|
191
|
-
)
|
|
192
|
-
```
|
|
193
|
-
## Updating a database with data in a DataFrame: Parallel
|
|
194
|
-
|
|
195
|
-
To load the data in the DataFrame in parallel, we need to partition the DataFrame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default this will use all the available virtual CPUs in the machine. If you need to use less, you can specified how many to use by setting the _workers_ argument.
|
|
196
|
-
|
|
197
|
-
```python
|
|
198
|
-
person_load_result = graph.execute_write_query_with_data(
|
|
199
|
-
person_data,
|
|
200
|
-
node_load_queries['person'],
|
|
201
|
-
partitions = 5,
|
|
202
|
-
parallel = True,
|
|
203
|
-
workers = 5
|
|
204
|
-
)
|
|
205
|
-
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|